Commit Graph
4080 Commits
Author SHA1 Message Date
Devin FoleyandClaude Opus 5.5 2756be2f32 fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent setup lists the models that an OpenRouter key can select. The
server gets this list from the public OpenRouter catalog.
> - The catalog request has its own 10 second deadline. When the
deadline expired, fetch rejected with a raw `TimeoutError` (a
`DOMException`).
> - The error handler answered 500 "Internal server error" and sent a
crash report with no first-party stack frames. The user did not see the
existing retry guidance.
> - A slow or unavailable third-party catalog is not a fault in the
Paperclip server. It must not create a crash report.
> - This pull request maps catalog outages to a 503 response with the
existing retry message and a stable code. Only those errors skip crash
reporting.
> - The benefit: users see "Retry or enter a model ID manually", and
Sentry does not get noise from a provider outage.

## Linked Issues or Issue Description

**What happened?**

A staging server answered `GET
/api/companies/:companyId/adapters/opencode_local/models` with `500 —
The operation was aborted due to timeout`. At the same time, other
outbound requests from the host were slow. Sentry recorded
`TimeoutError: The operation was aborted due to timeout`. The event had
only Node internal frames (`AbortSignal.timeout`).

**Expected behavior**

The route returns a clear "catalog unavailable" response. The UI shows
the retry guidance. A third-party outage does not create a crash report.

**Steps to reproduce**

1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more
than 10 seconds), or make it unreachable.
2. Request the OpenCode model list with `provider=openrouter`.
3. See a 500 response and a Sentry `TimeoutError` event.

**Paperclip version or commit**

`e23549cd9c631e4595a3213e03d0c960360452c6`

I searched for duplicate PRs. #17083, #17961 and #10290 change the
OpenRouter catalog contents or model validation. They do not change how
the server handles a catalog timeout.

## What Changed

- `server/src/services/openrouter-models.ts`: The catalog now maps three
failure types to `HttpError(503)` with code
`openrouter_catalog_unavailable` and the existing retry message: its own
deadline, a classified transport failure (uses
`classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A
module-local `WeakSet` records only the errors that it creates for these
outages. `isOpenRouterCatalogUnavailable()` reads that record.
- Other non-OK statuses become a 502 with code
`openrouter_catalog_error`. These are still reported. Malformed JSON and
an invalid catalog shape are also still reported.
- `server/src/middleware/error-handler.ts`: The handler does not send a
crash report for an error from `isOpenRouterCatalogUnavailable()`. It
still answers 503 with the message and code. The new check is next to
the existing MCP connection-failure exclusion.
- Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for
the failures that stay reportable. A route-level test sends a stalled
catalog request through the real error handler. It checks for a 503
response and no crash report. It also checks that a different 503 is
still reported.

## Verification

- `vitest run server/src/services/openrouter-models.test.ts`: 7 tests
pass. I ran this file alone because the full workspace install did not
fit in the agent sandbox disk.
- Red check: with the original `openrouter-models.ts`, 5 of the 7 tests
fail. The deadline test receives `DOMException TimeoutError`, which is
the same error as the production event.
- `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs
the full workspace dependencies. CI runs it.

## Risks

- Low risk. Successful responses do not change. The cache and in-flight
deduplication do not change.
- The failure status changes from 500 to 503 (or 502) for the three
routes that read this catalog: adapter models, fast-response models and
prompt-injection-prevention models. The UI already shows
`error.message`, so users now see the retry guidance in place of
"Internal server error".
- The crash-report exclusion is narrow. It applies only to error objects
that this module created after an actual outage. A copied message or
code does not match.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry, Railway and
GitHub connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:00:24 -08:00
DottaandClaude Fable 5.2 1a96247480 fix(recovery): continue a failed continuation again instead of escalating it to the board (#21394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic task recovery continues a task when its run ends for a
reason that is not a person's to fix
> - A continuation or recovery run can itself fail before it does any
work: a sandbox create timeout, a full sandbox disk, a lease mismatch, a
provider pack check, or a database deadlock
> - Today the stranded reconciler treats such a failure as non-retryable
and blocks the task for the board. The assigned-work sweep then leaves
the board hold alone, because its list of temporary causes is a short
allow-list. The held backlog pass uses a similar allow-list. Nothing
continues the task, and it shows "failed, Retry"
> - This pull request makes the cause test "not owned by a person" on
these paths, and routes a failed continuation run back through the
fresh-continuation fallback
> - The benefit is that such a task continues from its saved work, under
the existing hourly cap, and nobody needs to press Retry

## Linked Issues or Issue Description

**What happened?**
A task was continued automatically after an interruption. The
continuation run failed during setup, before any provider turn (for
example `setup_failed` with a sandbox HTTP timeout). The stranded
reconciler then set the task to `blocked` with a board-owned recovery
hold. The assigned-work sweep left the task to that hold, and the
fresh-continuation fallback never owns that hold kind. The task stayed
blocked and showed a Retry button. Settled holds whose run was a
recovery run that failed with `adapter_failed`, `setup_failed` or a
provider pack error stayed in the same state, because the held backlog
pass accepted only restart-class codes.

**Expected behavior**
A run that ends for a reason a person does not own is continued again
from saved work. The continuation goes through the fresh-continuation
fallback, so the old workspace epoch is fenced or a new isolated
workspace is used. The per-task limit of 6 automatic continuations per
hour still applies. At the limit the task waits for the next window and
is not blocked.

**Steps to reproduce**
1. Record a fresh continuation for an interrupted run of an
agent-assigned task.
2. Let the continuation run fail with `setup_failed` before provider
work.
3. Run the stranded reconciler. The task becomes `blocked` with a board
hold.
4. Run the assigned-work sweep. It returns `left_to_hold_owner`. No
continuation follows.

**Paperclip version or commit**
master at 6917d0568d

**Deployment mode**
Hosted (two servers can run at once during a deploy)

## What Changed

- New `continuation-cause.ts` holds the cause classification:
`PERSON_OWNED_FAILURE_CODES` (moved, unchanged), a new short list
`PERMANENT_FAILURE_CODES` (low-trust boundary codes,
`adapter_engine_unavailable`, `agent_not_invokable`, `agent_not_found`,
`agent_interaction_delivery_denied`), `isSystemOwnedFailureCode`, and
`isAutomaticContinuationRun` (a retry, scheduled retry, fresh
continuation, sweep or recovery run; never a person's Retry).
`fresh-continuation-fallback.ts` re-exports what callers use.
- `stranded-hold-continuation.ts`: the sweep continues a recent (24
hours) board hold when the failure cause is system-owned, not only for a
short allow-list. A reusable sandbox resume failure still needs a
transient cause.
- `recovery/service.ts`: the stranded reconciler does not escalate an
agent-assigned `todo` or `in_progress` task to the board when its latest
run was an automatic continuation or recovery run that ended within 24
hours with a system-owned cause. The assigned-work sweep owns that task.
A person's Retry, a person-owned cause, a permanent cause, or an older
failure escalates as before.
- `assigned-work-sweep.ts`: for a failed automatic continuation run, the
sweep asks the fresh-continuation fallback first. A recorded
continuation is the repair (fenced epoch or new isolated workspace). On
the fallback's hourly backoff the sweep records "not started" with the
retry time and keeps the task unblocked. While exact recovery still
waits, it records the retry time. A person-owned reason from the
fallback is written on a refused attempt.
- `task-automatic-attempts.ts`: the shared 6-per-hour ledger also counts
`server_restart` and `interrupted_preparation` scheduled retries, so the
sweep cannot add attempts on top of those lanes in the same hour.
- `fresh-continuation-fallback.ts`: the held backlog pass also accepts a
source run that was an automatic continuation or recovery run with a
system-owned code (24 hours, 5 per pass, 10 per minute unchanged). With
no hold to carry a pending check, a wait is measured from the source
run's end, so it is bounded.
- `fresh-continuation-fallback.ts`: rechecks the cause with the shared
classifier and applies the shared hourly cap inside the task lock. A
continuation keeps the old writer's workspace unless a wait was actually
cut short (bound measured from a recorded check, or a wait skipped).
- New test file `failed-continuation-continues-again.test.ts` (15
tests).

### Changed test expectations

- `interrupted-preparation-hold-settle.test.ts`, "bounds repeated lost
leases before provider work" (both cases): the lane cap still holds (2
and 12 successors). The end state is no longer `blocked`: the task keeps
its status, no further run exists, and the sweep returns a deferral with
a future retry time (`deferred_cap` at 12 ledger attempts,
`deferred_rate_limit` spacing at 2). The fixture now settles the
scheduled wakes and retry times the loop leaves behind, so the sweep
reads the real state. A person's Retry is still accepted.
- Same file, "blocks the task when a replaced preparation's lane is
spent" is renamed "leaves the task to the assigned-work sweep, not
blocked, when a replaced preparation's lane is spent". It still expects
no successor run, and now expects `todo` with no active hold.
- `heartbeat-process-recovery.test.ts` and `task-retry.test.ts`: no
change. Their runs either ended months before (outside the 24-hour
window) or were a person's Retry, so they escalate as before. The bound,
no-duplicate, and redaction tests keep their assertions.

## The one classifier

`continuation-cause.ts` `continuesAutomatically(run)` (and its SQL twin
`continuesAutomaticallySql`) decides whether a failed run continues on
its own:

- the cause is system-owned: not in `PERSON_OWNED_FAILURE_CODES`
(unchanged), not in the new `PERMANENT_FAILURE_CODES`, not an external
execution whose stop cannot be confirmed (Dot and Muse, the same rule
the retry scheduler uses to decline them permanently), and a reusable
sandbox resume failure only with a transient cause; and
- the run was an automatic continuation or recovery run (a scheduled
retry, a recovery wake, a fresh continuation; never a person's Retry),
or a first run that ended with a restart-class code or a transient setup
failure before provider work (the pre-existing allow-list for the
backlog and the board-hold takeover).

A person's failed Retry keeps master's outcome at every entry point: the
classifier never treats it as an automatic run. Whether it should
continue on its own is a separate product decision.

## Every entry point and the cap it uses

Shared cap = `sharedContinuationAttempts` in
`recovery/continuation-ledger.ts`: 6 per task per rolling hour, counting
fresh continuation records, settled sweep wakes, and transient, restart
and preparation retries. Only attempts that started or created a run
count.

- Disposition pass (`settleUnrecoverableExecutions` to
`continueFreshInTransaction`): cause test of the classifier and the
shared cap, inside the fallback under the task lock. Run kind is not
required on this path: it handles fresh failures of any run.
- Held backlog (`continueHeldExecutions`): `continuesAutomaticallySql`
in the listing; then the fallback's cause test and shared cap. Pacing 5
per pass, 10 per minute, 24 hours.
- System continuation wake in the queue (`heartbeat/queue.ts`,
unchanged) to the fallback: cause test and shared cap.
- Dispatch refusal (`run-dispatch` Postgres adapter, unchanged) to the
fallback: cause test and shared cap.
- Sweep board-hold takeover (`strandedHoldContinuesAutomatically`):
`continuesAutomatically`, 24 hours.
- Sweep route to the fallback (trigger `assigned_work`): recovery run
with a system-owned cause, then the fallback's cause test and shared
cap; the sweep claim also applies its ledger.
- Old reconciler deferral (`assignedWorkSweepOwnsFailedRun`): automatic
run with a system-owned cause (`canContinueRecoveryRunAutomatically`),
24 hours. First runs keep their existing escalation and budgets.
- Sweep plain wake (unchanged path, for first runs): the sweep claim
ledger (`taskAutomaticAttempts`: started sweep attempts, scheduler
retries, and fresh records from other paths), spacing from every
attempt.

## Self-review: places where the rule still differs, on purpose

- The fallback's fresh path continues a first run with any system-owned
cause, while the backlog limits first runs to restart-class and
transient setup codes. The backlog limit is the existing herd rule for
records older releases left behind.
- The fallback treats a pending question as a person's wait; the sweep
treats a question asked before a person's Retry as superseded. In that
one shape the sweep sends its ordinary wake, as before.
- The scheduler lanes (transient, restart, preparation) keep their own
bounds and do not read the shared cap before scheduling; their runs
count in it, so every other path stops once they fill the hour. Changing
the scheduler is outside these files.
- The sweep's plain wake path does not test the cause; it relies on the
invariant's waiting reasons, as before.

## Waits and their bounds

- Every exact-recovery wait in the fallback (settling execution, native
finalizer, live controller lease, preparation lane, saved-work recovery,
workspace restore owner, retained-source check, previous local process):
it starts at the first recorded pending check on the run's hold. When no
check is recorded (no hold, a hold nobody checks, or a replaced record),
it starts at the source run's end (`finished_at`, a database row that
every server reads the same). Bound: 10 minutes idle, 15 minutes while
the owner shows progress, 30 minutes while saved-work recovery advances.
Then the fallback continues from saved work.
- A source run that ended more than 24 hours ago with no recorded check
does not wait: the fallback answers `retry_needed` (the existing named
state that leaves an old hold for a person's Retry), so nothing waits
silently and the 24-hour rule holds.
- The sweep on a fallback backoff: waits until the oldest continuation
leaves the hour (6 per task per hour), task stays unblocked.
- The sweep while exact recovery waits: rechecks at the fallback's retry
time (at most 4 minutes), until the bound above.

## Decisions on permanent causes

- `PERSON_OWNED_FAILURE_CODES` is not changed. A separate
`PERMANENT_FAILURE_CODES` list keeps causes that cannot heal by
themselves out of the new automatic paths, so they still block and tell
a person. `repository_catalog_unavailable` was transient in the old
allow-list too; in the failing Retry test it was a person's Retry, which
the reconciler still escalates.

## Verification

- `failed-continuation-continues-again.test.ts`: 28 tests, including a
12-case table (person's Retry, automatic continuation, first assignment;
a person's Retry gets master's outcome everywhere) x (setup failure,
restart code, permanent code, person-owned code) through the backlog,
the sweep takeover and route, and the reconciler, and a test that
waiting checks spend no attempt (red before the fix for the first-run
case and for the cap); earlier: 15 tests (the latest three added
red-first: permanent codes and a person's Retry stay blocked on the
backlog; no entry point continues a person-owned or permanent code; six
mixed attempts stop the backlog until the next window; plus a pin that
the ledger uses the restart lane's window, cap and reason). Earlier:
Before the fix, the 10 behavior tests failed for the expected reasons
(board escalation, `left_to_hold_owner`, no backlog continuation,
endless wait with no hold, endless wait on an unchecked hold at 20
minutes and 30 hours, checked alternately by two sweep instances). The
person-owned guard passed before and after. After: 11 of 11 pass. After
the last change, alone: `fresh-continuation-fallback.test.ts` 72/72,
`fresh-continuation-move-forward-offer.test.ts` 14/14,
`assigned-work-invariant.test.ts` 31/31,
`interrupted-preparation-hold-settle.test.ts` 29/29.
- Locally after the change, one file at a time:
`assigned-work-invariant.test.ts` 31/31,
`recovery/assigned-work-invariant.test.ts` 50/50,
`interrupted-preparation-hold-settle.test.ts` 29/29,
`task-retry.test.ts` 117/117, the 10 previously failing
`heartbeat-process-recovery.test.ts` tests 12/12 (filtered run),
`fresh-continuation-fallback.test.ts` 66/72 in a loaded run with 6
timeouts at 30 to 90 s that pass alone (8/8 filtered).
- `tsc --noEmit` for `server`: clean.
- CI supplies the full suite.

## Risks

- Behavior change: a recent failure with a cause outside the
person-owned list is no longer escalated to the board by the stranded
reconciler. The sweep continues it instead, at most 6 times per hour per
task, then once per window. Some codes outside that list may be
configuration a person must change (for example `low_trust_*`,
`adapter_engine_unavailable`, `agent_not_found`). Such tasks now retry
at the capped pace instead of blocking. The person-owned list is not
changed in this pull request.
- Upgrade: tasks blocked by an older release with a board hold for such
a cause, within 24 hours, are continued by the sweep on its first
passes. The sweep's 10 repairs per minute per process, the delivery
pacing, and the hourly cap bound the burst.
- Two servers: the sweep's claim and the fallback each run under the
task row lock, and the fallback is idempotent per source run.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:49:16 -06:00
DottaandClaude Opus 5.5 b5adf35119 test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) that the
server launches in a sandbox.
> - A deploy can stop the server after the launch command reached the
sandbox but before the server read back the Runner's process identity (a
lost acknowledgement).
> - The database then has no PID for the run, so the successor server
launches the Runner again on the same state folder.
> - Two Runners must never serve the same run's state. #20852 proposed a
large launch-intent design for this window. Since then, #21079 added a
fence that stops a leftover Runner on the same state folder before each
launch.
> - This pull request adds a Linux regression test that drives the real
launcher through the lost-acknowledgement window.
> - The benefit is that a future change to the launch path cannot bring
back a second Runner without failing this test.

## Linked Issues or Issue Description

Refs #20852 (superseded: current master already prevents the duplicate
Runner through #21079). Refs #21079.

**What happened?**
#20852 claimed that a relaunch after a lost launch acknowledgement could
start a second Runner for the same run. Nothing proved whether current
master still has this problem.

**Expected behavior**
After a lost acknowledgement, the successor's relaunch stops the
orphaned Runner. Exactly one Runner serves the run's state folder, and
the process marker names it.

**Steps to reproduce**
1. Launch a remote Runner, and lose the launch RPC's acknowledgement
before the identity read (the old server stops).
2. Relaunch the same run on the same sandbox from a new launcher (the
successor).
3. Count the live `paperclip-runnerd` processes that serve the run's
`--state-dir`.

## What Changed

- New
`server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`.
It is a test only and changes no product code.
- It runs only where `/proc` exists (Linux CI), like the existing #21079
fence tests.
- It drives the exported `createRemoteRunnerProcessLauncher` with a
local command runner. The old launcher's runner runs the launch command,
then throws `remote_launch_ack_lost`. After that, every command fails,
because the old server is gone.
- A small bash stand-in for `paperclip-runnerd` keeps the Runner's
argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty
lineage.
- The test checks these results: the orphan was alive and had no PID on
the handle; the successor's Runner has a new PID; the orphan stops;
exactly one Runner serves the state folder; the marker names the
successor's PID.

## Verification

- Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact
outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I
extracted from this head:
- master scripts: `runners after successor relaunch [61], orphan alive
false`. Result: one Runner (PASS).
- the same scripts without the #21079 stale-state fence
(counterfactual): `runners after successor relaunch [24, 56], orphan
alive true`. Result: duplicate Runner (FAIL).
- macOS: `npx vitest run
src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`
reports the file as skipped (1 skipped), as intended, because there is
no `/proc`.
- The test runs for real only on Linux CI. CI on this head is the first
real run of the vitest file.

## Risks

- Low risk. The change is a test only.
- The test starts real local processes. Its teardown kills only the PIDs
it found on its own temporary state folder, and it never signals PID 0.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, Docker, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:24 -06:00
DottaandClaude Opus 5.5 4e2c149f2f test(decisions): let the concurrent admission test survive a legitimate serialization retry (#21416)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Decision models let features ask a company AI connection short
questions. Each decision reserves budget before it calls the provider.
> - The server test "admits concurrent work from the same budget
observation" fails or hangs intermittently in CI. This blocks Core image
publication from master.
> - The test assumed that both admission snapshots always survive. They
do not. A legitimate serialization retry can make one admission see the
other's committed hold.
> - This pull request changes the test so that it proves what the
product guarantees, with the same strength, and without the timing
assumption.
> - The benefit is a deterministic test and unblocked image publication.

## Linked Issues or Issue Description

Refs #21387

**What happened?**
The test "company decision service > admits concurrent work from the
same budget observation and records both obligations" in
`server/src/__tests__/decision-models.test.ts` hangs intermittently. CI
run 43451649374 (attempt 2, job "verify / General tests (server
(4/10))") failed with `Test timed out in 15000ms` at line 371. Before
#21387 the same race failed with `expected "vi.fn()" to be called 2
times, but got 1 times`.

**Root cause**
- `decide()` resolves the credential before admission.
`secrets.resolveSecretValue` writes `company_secrets.last_resolved_at`
and `updated_at` in its own statement
(`server/src/services/secrets.ts:1417-1421`).
- Admission runs in a REPEATABLE READ
`withOperationAccountingTransaction`
(`server/src/services/accounting-transaction.ts:31-53`). Its snapshot
starts before it waits on the test's lock.
- When decision B's `last_resolved_at` write commits after decision A's
snapshot, A's `SELECT ... FOR SHARE` on the same secret row
(`server/src/services/decision-models.ts:263-264`) raises 40001.
- The retry loop takes a fresh snapshot. If B has committed its 1c hold
by then, A sees 1c held against a 1c hard stop and correctly returns
`budget_blocked`. A never reaches the provider, so the test waits
forever.

**Expected behavior**
The test passes every time. It still fails if admission becomes a
company-wide writer mutex or if the hard stop stops blocking.

## What Changed

- Set the test budget to 2c with 1c reservations. Both decisions are now
admitted whether or not one admission retries.
- Kept the proof that both admissions are in flight at once (both wait
on the same row lock at the same time). This is the no-mutex guarantee.
- Kept the checks that both providers run concurrently, both holds are
`held` at 1c, both decisions succeed, and two cost events exist.
- Added a check that a third decision returns `budget_blocked` while
both holds are held.
- Renamed the test to "admits concurrent work without a company writer
mutex and records both obligations".
- No product code changed. The retry and the `budget_blocked` result
after the retry are correct hard-stop behavior.

## Verification

- Root cause reproduced with temporary instrumentation (not committed).
Starting decision B after decision A was parked gave 40001 on the
`company_secrets` FOR SHARE in 4/4 runs. With the retry delay forced to
300 ms, the old test failed with the exact CI message `expected
"vi.fn()" to be called 2 times, but got 1 times`, and the log showed
`admitted` then `budget_blocked`.
- The new test passes in that forced worst case (staggered starts plus a
300 ms retry delay): both admitted, the third `budget_blocked`.
- 50 repetitions (vitest `repeats: 49`, temporary) with 16 extra CPU
busy loops on an 18-core machine: 50/50 pass.
- Red check: changing the company read in
`withOperationAccountingTransaction` to `FOR NO KEY UPDATE` (a company
writer mutex) makes the test fail with `expected 1 to be 2`.
- `npx vitest run src/__tests__/decision-models.test.ts`: 37/37 in 2 of
3 runs. One run had a failure in a different test ("returns answers and
a committed charge before slow budget stop delivery"). That test passes
alone and is not changed here.

## Risks

- Low risk. Only one test changes.
- The test no longer proves that two admissions from the same snapshot
both pass a 1c hard stop. That overshoot is not a product guarantee: any
concurrent committed write to a share-locked row restarts the snapshot.
- Each concurrent decision on the same credential can force a sibling
admission to retry because of the `last_resolved_at` write. This is
extra latency, not a correctness problem. A follow-up can remove that
write from the decision hot path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:14 -06:00
DottaandClaude Opus 5.5 c1ea9304f7 fix(recovery): continue Runner preparations whose remote workspace was handed on unchanged (#21481)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.

**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.

**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.

**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.

## What Changed

- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.

## Verification

- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:05 -06:00
DottaandClaude Opus 5.5 a48eaaef57 perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent keeps its own files (AGENT_HOME). A warm native session
saves these files at the end of each turn (an agent-file checkpoint)
> - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on
the small file-storage pool, also when the run changed no file
> - A checkpoint that changed files also uploaded the new bytes to
object storage while it held that lock. Instruction edits and other runs
of the same agent waited for the upload
> - With many agents, these long transactions use the 2-connection
file-storage pool and slow every agent
> - This pull request removes the lock from an unchanged checkpoint and
moves the uploads of a changed checkpoint before the lock
> - The benefit is that an unchanged checkpoint takes no row lock, and
the row lock is no longer held across object-storage I/O

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm agent-file checkpoint: `checkpoint()` in
`server/src/services/agent-directory-working-copies.ts`,
`agentFileStore.locked` and `apply` in `agent-file-store.ts`, and
`persistDurableAgentFiles` in `agent-file-durability.ts`.

**Current behavior**
- When the run changed no file, the checkpoint still ran
`store.locked(..., write = true)`. That transaction locks the agent row
(`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the
local projection, walks the canonical tree, and compares the durable
head. Its only result is the authorization check.
- When the run changed files, `persistDurableAgentFiles` uploaded each
new file to object storage inside the transaction that holds the agent
row lock.

**Proposed behavior**
- An unchanged checkpoint (`saveUnchanged`) first reads, without a lock,
whether the locked file-tree step would change anything
(`agentFileProjectionCurrent`): the durable head exists and the local
projection is marked at its generation, or durable files are off, no
head exists and the local adoption is recorded. If so, it runs only the
write authorization (`authorizeSave`): the agent interaction policy and
`authorizeInstructionCommit`, in a short transaction without row locks
that always rolls back, so it writes nothing. If not, it runs the locked
save as on `origin/master`, which adopts, restores or heals the durable
head. A de-authorized agent still gets the same error (the existing test
"rechecks authorization even for an unchanged warm checkpoint" still
passes).
- A changed checkpoint runs the same read-only authorization (it
consumes no accepted change confirmation), then uploads the run's
changed files before it takes the lock (`uploadAgentFilesBeforeCommit`).
A run that lost its save permission uploads nothing. The locked commit
runs the authorization again, and only that commit consumes a
confirmation. The locked commit uses an uploaded object only for a file
with the same SHA-256 and size. It uploads any other file itself, as
before.

**Reason and benefit**
The target is 1,000 agents in one controller process. Most warm turns
change no agent file. These turns now take no row lock and send fewer
statements. A changed turn holds the row lock for less time, because
object-storage latency is outside the lock.

## What Changed

- `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)`
and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a
checkpoint, then uploads the run's changed files before `locked`, and
passes the uploaded objects to `persistDurableAgentFiles`. `locked`
takes an optional map of uploaded objects.
- `agent-file-durability.ts`: move the verified upload stream into
`uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the
durable head without a lock, skips files the head already has at the
same path and digest, and uploads the others. If an upload fails, it
stops and the locked commit uploads the rest, so errors are the same as
before. `persistDurableAgentFiles` uses an uploaded object when the
canonical file has the same digest and size.
- `agent-directory-working-copies.ts`: an unchanged checkpoint calls
`saveUnchanged` instead of `store.locked(..., true)`.
- `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head
read and two file checks, no lock).
- The commit is still inside the agent row lock. The head cannot change
between the read and the write of the commit, so it needs no separate
compare-and-set. The upload before the lock does not need one either: an
object is immutable and verified by digest, and the commit records it
only for a file with that digest.

## Verification

- New `server/src/__tests__/agent-file-checkpoint-locking.test.ts`
(embedded PostgreSQL, object-storage fixture). All tests fail on the
code before the change and pass with it:
- An unchanged checkpoint completes while another transaction holds the
agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR
SHARE` statement and only one unlocked read of `agent_file_roots`.
- Durable files on, the agent has local files and no head: an unchanged
checkpoint creates the head, as on `origin/master` (added after review;
failed without the fallback).
- The local projection is not marked at the head: an unchanged
checkpoint takes the locked path and marks it again (added after review;
failed without the fallback).
- A changed checkpoint uploads its file while the agent row is not
locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from
another connection during each upload). The head then records that
object.
- A run loses its save permission before a changed checkpoint. The
checkpoint is refused and uploads nothing (added after review; I saw it
fail without the authorization before the upload).
- An agent with `agents:suggest-changes` and one accepted change
confirmation saves a changed checkpoint. An unchanged checkpoint first
leaves the confirmation unconsumed. The changed save succeeds once and
consumes the confirmation once; the next change is refused before it
uploads (added after review; I saw it fail when the preview committed).
- Two warm runs of the same agent save at the same time. Both files are
in the canonical tree and in the head, and every head object has the
correct digest.
- An instruction edit commits while a checkpoint waits in its upload,
without a wait for the checkpoint. Then the checkpoint commits. Both
changes are in the head.
- A save stops between upload and commit (the instruction entry changes
while the bytes upload, so the commit refuses). The head does not change
and stays restorable. The next save commits the same file. The unused
object is never authoritative; the existing object collector removes it
after its grace period.
- Existing `agent-directory-working-copies.test.ts`,
`agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts`
pass.
- Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`,
test "measures checkpoint statements and agent row lock time"): one warm
copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that
each change one file, with 250 ms object-store latency. Wall times are
noisy (load average about 40). Statement counts are exact.

| | Before (`origin/master`) | After (final head) |
| --- | --- | --- |
| Unchanged checkpoint: statements | 26 | 25 (one unlocked head read,
and the read-only authorization) |
| Unchanged checkpoint: row-lock statements | 2 | 0 |
| Unchanged checkpoint: median time | 214 ms | 26–126 ms |
| Changed checkpoint: statements | 29 | 47 (the read-only authorization
before the upload and one head read) |
| Changed checkpoint: median time the agent row is locked | 924 ms |
51–227 ms |
| Changed checkpoint: uploads while the agent row is locked | 1 | 0 |

The "after" ranges come from two runs at different machine load (load
average between about 18 and 40).

## Risks

- Master side effects of the locked unchanged save, and what this change
does with each:
- Kept: the write authorization (`assertAgentInteraction`,
`authorizeInstructionCommit`), so a run that lost its permission still
gets the same error.
- Kept through the fallback: one-time adoption (no head yet), restore of
a projection that is not marked at the head, re-mark after a failed
persist (a write path removes the marker before it changes files), and
heal of a head that lags the root after such a failure.
`agentFileProjectionCurrent` returns false in each of these cases, so
the locked save runs.
- Dropped when the projection is current: the re-mark at the same
generation (no change), and the compare of the local tree with the head.
With the marker at the head generation, every writer that changed the
tree went through the locked save and moved the marker, so the compare
finds no difference. An edit made outside Paperclip directly in the
canonical directory is not picked up by an unchanged checkpoint any
more; the next save that goes through the locked path still records it.
- Dropped when the projection is current: the storage-limit check of the
canonical tree in `persistDurableAgentFiles`. An over-quota agent whose
run changed nothing no longer gets an error from that check; any save
that changes files still enforces the limits.
- Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction`
takes on the actor and target agents (`lock: "update"`). Those locks
hold permission changes off a write until it commits. The read-only
authorization writes nothing, so there is nothing to hold them off. The
locked commit of a changed save still takes them. An unchanged save has
no mention texts, so master locked no other agents for it.
- Effects of `authorizeSave` outside the rolled-back transaction: none.
I read `assertAgentInteraction` (`agentInteractionDenial`: reads only),
`authorizeInstructionCommit`, `resolveInstructionActor` (reads only),
`authorizationService.decide` (reads only; its memo maps live on the
bound actor and on the service instance, both created for this call and
dropped after it), and `changeConsentGateService.assertConsented` (one
`UPDATE` of the confirmation row, rolled back). None of them publishes
live events, queues notifications or wakes, writes activity through
another connection, makes external calls, or changes an in-process cache
that outlives the call.
- Behavior change: on `origin/master`, an unchanged checkpoint of a
consent-gated agent (`agents:suggest-changes`) consumed an accepted
change confirmation, because the locked transaction committed. Now the
fast path does not; only the commit of a changed save consumes it. When
the fast path cannot be used (adoption, restore or heal), the locked
save runs as on master and can still consume a confirmation.

- Not in this change: the manifest is still stored twice in the receipt
(`baseline` and `checkpointManifest`). Servers of the previous version
read `receipt.baseline` during a deploy overlap, and the next capture
needs the stamps in `checkpointManifest`, so neither can be removed
safely now.
- Not in this change: on the volume-free server, `serial()` still opens
a dedicated connection with an advisory-lock transaction for each
checkpoint. That lock serializes controllers and is not a row lock;
changing it is a separate design decision.
- A changed checkpoint can now leave an unused object in storage when
its commit is refused after the upload (for example, the instruction
entry changes during the upload) or the process stops before the commit.
This was already possible for a failed commit. Such an object is never
in the durable head, so no reader can reach it, and the existing
`collectAgentFileObjects` maintenance removes it after
`AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the
save stops the save before any upload.
- Old and new servers can run at the same time: the head format and the
receipt format do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:41:46 -06:00
DottaandPaperclip 8f105e743a fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 14:37:13 -06:00
DottaandClaude Opus 5.5 11f4a76743 perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every native agent session. Each session
gets the app tools (MCP tools) that are assigned to its agent.
> - Each live or warm session kept two private copies of its assigned
tool catalog: the gateway descriptors and a canonical copy of every
input schema.
> - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap
for each session. With 800 tools, it was about 15 MB. Warm sessions stay
for 5 minutes after the last turn, so idle agents also hold this memory.
> - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the
server cannot get there.
> - This pull request keeps one deeply frozen catalog for all sessions
of the same company that get the same admitted tools and schemas.
> - The benefit is that each additional session on the same catalog adds
about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is
faster.

## Linked Issues or Issue Description

Refs #21415 (releases a finished session's tool authority). Refs #21440
(adds the catalog memory benchmark that this change extends).

**What existing behavior does this improve?**
`createAssignedMcpTools` in
`server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call
kept every gateway descriptor (with its provider metadata) and a
separate canonical clone of each input schema. Each `definitions()` call
also made a new `structuredClone` of every schema.

**Subsystem affected**
Native runtime: assigned app tools (MCP) for native agent sessions.

**Current behavior**
Retained heap grows by one full catalog for each session. Measured with
the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB
per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100
sessions. Each `definitions()` call takes 14 to 41 ms because it clones
every schema.

**Proposed behavior**
Sessions of the same company that get identical admitted tools share one
frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB
at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms.

**Reason and benefit**
The server must hold 1,000 concurrent agents in one process. The catalog
was the largest per-session heap cost. Revocation and authorization do
not change: every call is still authorized again by the gateway, and
every new session still reads a fresh listing.

**Breaking changes**
None in the API or data. The direct definitions that `definitions()`
returns are now frozen. A consumer that writes to a returned definition
or schema gets a `TypeError` and does not change other sessions. No
production consumer writes to them (see Risks).

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`:
- A session keeps only a small frozen record for each tool (gateway
name, display name, description, risk), plus the frozen schema and the
frozen definition. It does not keep the gateway descriptors.
- New per-process index of shared catalogs. The key is the SHA-256 of an
identity string: `[companyId, [[assignedName, gatewayName, displayName,
description, risk, sha256(JSON of the input schema)] for each listed
tool, in listing order]]`. On a hit, the code also compares the full
identity string, so a hash collision cannot give a wrong catalog.
- The identity is computed from the fresh listing that the gateway
returns for this session. The gateway applies all per-agent, per-run,
per-profile, and policy filtering, and the approval description suffix,
before it returns the listing. Thus two sessions share only if they were
given exactly the same tools, names, descriptions, risks, and schemas in
the same company. Without `companyId`, nothing is shared.
- Schemas, definitions, tool records, and the definition arrays are
deeply frozen. Search results still return cloned copies.
- Sessions that start at the same time with the same catalog wait for
one build and share it.
- The index holds each catalog through a `WeakRef`, so it never keeps a
catalog alive. The index has at most 512 entries and 256 MiB (accounted
schema and description bytes). The oldest entries are removed first. A
catalog that is larger than the byte limit is not indexed.
- New `close()` on the assigned tools. When the last session that holds
a catalog closes, the index entry is removed. A `FinalizationRegistry`
also removes entries whose catalog was collected.
- `definitions()` returns the shared frozen definitions for the
session's work mode. It no longer clones every schema on each call.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
`close()` also closes the assigned tools. The session executor already
closes the authority when it revokes a session's tool authority.
- `server/src/services/native-runtime/native-session-executor.ts`: one
line. It passes the run binding's `companyId` to
`createAssignedMcpTools`.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes
`companyId`, adds `--companies`, measures session start and
`definitions()` time, and returns fresh copies of fixed rows (like
database reads).
- Tests in `assigned-mcp-tools.test.ts` and
`paperclip-runner-tool-authority.test.ts`.

## Verification

Red/green tests in
`server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared
assigned tool catalogs"):

- Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema
catalog. The heap for the other 19 sessions must be less than two
catalogs. Red on master: `expected 14025976 to be less than 3788112`.
Green with this change.
- Shared deeply frozen schemas, also for sessions that start together.
Red on master (`Object.is` equality fails).
- Mutation isolation: writes to a returned definition, schema property,
and `required` array, and a delete, throw `TypeError`. Changes to the
gateway's source rows and to a search result do not change any session.
Red on master (no throw).
- Release and bounds: the entry stays while one session holds it and
goes when the last session closes. `close()` is idempotent. A new
session after release builds a new catalog. 517 companies keep the index
at 512 entries or fewer. Red on master (no `close` or stats).
- Guard tests: two companies with identical catalogs do not share.
Sessions without a company do not share. Fewer tools, an edited schema,
an edited description, or a different risk do not share. A revoked or
edited tool is served from fresh discovery. On master these pass,
because master never shares. To prove they test the key, I removed
`companyId` and the schema digest from the identity: three guard tests
failed, and they pass with the full key.
- An existing test changed: "does not project gateway secrets..." now
expects a write to a returned schema to throw `TypeError` (before, the
write changed only a private clone). The isolation check after the write
is the same.
- `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes
the assigned tools.

Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one
company). Command: `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N>
--schema-bytes 5700 --sessions <S> --turns 5`.

| Tools | Sessions | Total before | Per session before | Total after |
Per session after |
| --- | --- | --- | --- | --- | --- |
| 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB |
| 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB |
| 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB |
| 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB |
| 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB |
| 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB |

The first session also uses about half the heap of before, because the
session no longer keeps the gateway descriptors. When all sessions are
dropped without `close()`, the catalog is collected (0 bytes held after
release).

Session start, interleaved A/B in one process (old and new
`createAssignedMcpTools` alternate, 40 starts each, median of the last
35, includes a fresh copy of the listing; the machine had a load average
of 25 to 65):

| Tools | Listing copy only | Before | After |
| --- | --- | --- | --- |
| 400 | 22.6 ms | 68.0 ms | 37.6 ms |
| 800 | 30.5 ms | 72.5 ms | 41.2 ms |

A start that reuses a catalog does no schema clone and no admission. It
still serializes and hashes each schema to compute the key. The first
start of a new catalog does the same admission work as before (the cold
first start was about 600 to 750 ms in both versions on this busy
machine).

`definitions()` median: 14 to 41 ms before, 0.01 ms after.

Other checks:

- `cd server && npx vitest run
src/services/native-runtime/assigned-mcp-tools.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-tool-results-prp.integration.test.ts
src/__tests__/deployment-connections.test.ts`: 204 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"assigned|tool authority|released|MCP"`: 31 passed.
- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10
passed.
- `cd server && npx tsc --noEmit`: pass.
- `pnpm check:module-boundaries`: pass.
- `pnpm verify` selects a broad plan (the benchmark script is a broad
input), so CI supplies the full suite.

## Risks

- Frozen definitions: the direct definitions now go to the runner
drivers as frozen objects. I checked the consumers.
`PaperclipRunnerToolAuthority.definitions()` filters and spreads them.
The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them,
spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP
bridge), or serialize them. Their types are `readonly
Readonly<Record<string, unknown>>[]`. The first-party semantic tool
definitions are already shared module constants that go through the same
path without clones. If a consumer that I did not find writes to a
definition, it gets a `TypeError` instead of a silent change. Catalogs
over 256 tools or over the listing budget use the two on-demand tools
and are not affected.
- Authorization: the shared data is only the result of a listing that
the gateway already authorized for that session, keyed on the full
content plus the company. Execution still goes through the gateway with
the session's own token on every call. Search still intersects with a
fresh listing. A new session still reads a fresh listing, so a revoked
or edited tool takes effect for new sessions and for existing sessions
at the same points as before.
- Several server processes: the index is a per-process memo of the
listing content. Each process builds its own catalog from its own
listing. Nothing is stored or shared outside the process.
- The index is bounded (512 entries, 256 MiB) and holds catalogs only
weakly. Index eviction only stops future sharing. It never frees memory
that a session uses.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap and timing benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:30 -06:00
DottaandClaude Opus 5.5 533d02ff1a perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live-run recovery uses the shared scoped deadline worker
(`scoped-deadline-work.ts`). Each queue keeps its entries (one for each
running run) and the scopes that writes marked dirty (for example one
company and one run)
> - To schedule and to pick due work, the worker tested every entry
against every dirty scope. It also decoded each scope hint with
`JSON.parse` again inside those loops, and each reconstructed scope
walked all entries to drop vanished ones
> - With many runs and many dirty run scopes, this work grows with
entries × scopes on the event loop of the one server process that
controls all runs
> - This pull request lets an owner declare a partition key for its
scopes. Live-run recovery uses the company. The worker then tests an
entry only against the scopes of its own company, and decodes each hint
once
> - The benefit is that matching cost grows with entries + scopes, with
the same results

## Linked Issues or Issue Description

Refs #21444 (the lease renewal change from the same load model).

**What existing behavior does this improve?**
The scoped deadline worker in
`server/src/services/scoped-deadline-work.ts`, which live-run recovery
(`server/src/services/heartbeat/live-recovery-work.ts`) uses for its
seven queues.

**Subsystem affected**
Server: background work scheduling for live-run and resource recovery.

**Current behavior**
For 400 entries in 40 companies and 40 dirty run scopes, one schedule
computation and one pass call `matches` 27,830 times and decode each
hint many times.

**Proposed behavior**
The same case calls `matches` fewer than 3,200 times and decodes each
hint at most once. Owners that do not declare a key behave exactly as
before.

**Reason and benefit**
The plan for 1,000 concurrent agents needs per-run background work that
does not grow with the product of runs and changes. The work runs on the
main thread of the controlling process.

**Breaking changes**
None. `scopeKey` is optional.

## What Changed

- `scoped-deadline-work.ts`: new optional `scopeKey` option
(`entry(entry)` and `scope(scope)`). The contract: a scope with a
defined key matches only entries with the same key. A scope without a
key (for example `*`) is tested against every entry, as before.
- The worker indexes the dirty scopes by key, so `next()` and the due
filter test each entry only against the scopes of its key and the
unkeyed scopes.
- When a reconstructed keyed scope drops vanished entries, it visits
only the entries of its key (an index built once for each pass).
- Scope hints are decoded once and the decoded value is reused while the
hint is dirty or scanning.
- `live-recovery-work.ts`: declares the company as the key. Its
`matches` already requires the same company for every defined scope.

## Verification

- New test in `scoped-deadline-work.test.ts`: 400 entries in 40
companies, then 40 run-scoped invalidations. It requires fewer than
3,200 `matches` calls and at most 40 decodes, and the next deadline is
unchanged.
- Red first: with `origin/master` `scoped-deadline-work.ts`, the test
failed with 27,830 `matches` calls.
- `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22
passed.
- The live-run recovery matrix (`live-recovery-work.test.ts`) is covered
by CI.

## Risks

- If an owner declares a key that its `matches` does not respect, the
worker can miss a match. Only live-run recovery declares a key, and its
`matches` returns false for any defined scope of another company.
- Low risk otherwise: owners without a key keep the old code path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:21 -06:00
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00
DottaandClaude Opus 5.5 969cf246ae perf(run-events): keep each native source's cursor on the run row (#21435)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Each native source event append counted all retained events of its
source (`count(*)`, `max(source_seq)`) to compute the acknowledgement
cursor
> - So the cost of each append grew with the length of the run, and long
runs made every text delta slower
> - This pull request keeps the gap-free cursor of each source on the
run row and moves it in the same UPDATE that reserves the event sequence
> - The benefit is that an in-order append stops reading other events,
so its cost does not grow with run length

## Linked Issues or Issue Description

Refs #21131 (earlier reduction of statements per event). Item 1b of the
agent scaling plan in #21418.

**What existing behavior does this improve?**
The append of a native source event (`appendNativeSourceEvent` in
`server/src/services/heartbeat-run-events.ts`). It ran a `retained` CTE
that counted every event of the source on every append.

**Subsystem affected**
Server run event ingest (`heartbeat_run_events`, `heartbeat_runs`).

**Current behavior**
On a run that already holds 50,000 events, one in-order append takes
about 10–13 ms (p50) on a local PostgreSQL. On a run with 400 events it
takes about 3.3 ms.

**Proposed behavior**
The append reads the stored cursor in the run lock statement that it
already sends. An in-order event moves the cursor and does not count
events. On the same 50,000-event run an append takes about 1.7–3.3 ms
(p50).

**Reason and benefit**
Text deltas are about 85% of the server's statements in the load model.
Their cost must not grow with run length.

**Breaking changes**
None. The new column is nullable. The previous release ignores it.

## What Changed

- Add the nullable column `heartbeat_runs.native_source_cursors`
(`jsonb`, `{"<source instance id>": <seq>}`). Migration `0479` is `ADD
COLUMN` only and classifies as `online_compatible`.
- The run lock read of a native append also reads the cursor of its
source.
- When the event sequence is at most cursor + 1, the append statement
has no `retained` CTE. It moves the cursor in the UPDATE that reserves
the sequence, and the receipt uses the cursor.
- Every other case (no cursor, a gap, or a cursor behind the stored
events) runs the earlier recount. In the same UPDATE it stores the
cursor when the source has no gap, and removes it when the source has a
gap. So the event that fills a gap recounts.
- The cursor is a lower bound. A writer that does not keep it (the
previous release during a deploy overlap, or
`NativeRunCoordinatorStore`) can only make it lag. The next out-of-order
event then recounts and corrects it.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts` passes (19 tests).
- Red/green: before the change, 3 tests failed: the two new tests and
the updated lock-read statement test. After the change, all pass.
- New: "keeps the source cursor on the run so an in-order append does
not count the run's events" checks that the append statement has no
`count(` and that a replay returns the cursor.
- New: "recounts a source whose run has no cursor or a cursor behind its
stored events" covers rows saved before the change, rows from a writer
that does not keep the cursor, a gap, and the event that fills the gap.
- Focused related tests: `paperclip-control-plane-port.test.ts`,
`native-observer-replay`, `native-event-boundary-golden`,
`chat-native-progress-work`, `live-recovery-work`,
`native-interaction-bridge`.
- `pnpm check:module-boundaries` passes. `pnpm verify` returned exit 2
(broader CI coverage required) because of local benchmark scripts that
are not in this pull request. Required CI supplies the broad coverage.
- Append cost on a run with many events (local embedded PostgreSQL,
1,000 in-order appends, busy machine, load average about 20–40):

  | Events already in run | Before p50 / p90 | After p50 / p90 |
  | --- | --- | --- |
  | 50,000 (run 1) | 10.4 / 15.8 ms | 1.7 / 3.7 ms |
  | 50,000 (run 2) | 12.9 / 21.6 ms | 3.3 / 5.9 ms |
  | 400 | 3.3 / 6.2 ms | 3.7 / 7.7 ms (noise) |

- Statements per event do not change: BEGIN, run lock read, one append
statement, COMMIT (4). The append statement no longer scans the source's
events.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`): the machine was too busy for a stable result.
Deltas only: before 20,000 of 20,000 done at 3,999 statements/s. After:
13,796 and 14,817 of 20,000 in two runs. All sources: before 6,346 of
20,000 deltas done. After: 20,000 and 2,083 in two runs. A 20-second
window gives only 400 events per run, so it cannot show this change. The
long-run figures above are the evidence.

## Risks

- Low. The column is nullable and the previous release does not read it.
Runs that are in flight during a deploy have no cursor. Their first
append recounts once and stores the cursor.
- A stale cursor can only make the acknowledgement lag, never pass the
stored events. Event rows are deleted only with their company or agent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:08 -06:00
DottaandClaude Opus 5.5 0d832a02d3 perf(ci): shorten the general server and chat test lanes (#21393)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the `PR` CI workflow before it can merge
> - The `General tests (server (N/12))` and `General tests (chat (N/3))`
jobs finish last, so they set the wall-clock time of CI
> - Before this change, server shards took 9.5 to 11.7 minutes and chat
shards took 6.4 to 9.6 minutes; the target is 7 minutes or less
> - Measurement shows that the chat lane was unbalanced and that the
server lane is limited by CPU on the 4-vCPU runners
> - This pull request balances the chat lane by recorded test time, runs
three chat partitions per job, and takes general server weights from PR
runs
> - The benefit is a chat lane under 7 minutes and measured evidence for
the shard count that the server lane needs

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The PR CI general server and chat test jobs. They are the slowest jobs
of every pull request.

**Subsystem affected**

CI test scheduling: `scripts/run-vitest-stable.mjs`,
`scripts/test-line-shard.mjs` and
`scripts/ci/refresh-test-durations.mjs`.

**Current behavior**

Baseline: green `PR` runs
[43437558012](https://github.com/paperclipai/paperclip/actions/runs/43437558012)
and
[43441184316](https://github.com/paperclipai/paperclip/actions/runs/43441184316),
all on m7i-flex.xlarge (4 vCPU, 16 GB).

- Server shards: 9.5 to 11.7 job minutes, 502 to 654 s test step,
effective file concurrency 2.5 to 2.9 of 3 workers. Vitest reports
`tests 61-68%, import 25-31%, transform 6-7%, setup 1%`. Each shard sums
1,410 to 1,765 s of file time. The longest file is 303 s
(`deferred-execution-receipt-work.test.ts`), so the floor is total work
divided by workers, not one file.
- Chat shards: 9.3 to 9.6, 7.6 to 7.8 and 6.4 to 6.5 job minutes. Each
job ran two partitions of `chat-channels.integration.test.ts`. The
partitions were balanced by collected-case count, but one `it.each` line
can run many cases. One of six partitions recorded 532 s of test time
against a 377 s mean.

**Proposed behavior**

- Chat partitions keep each source line whole and balance by recorded
time per collected entry.
- Each chat job runs three partitions in place of two.
- General server weights come from the server jobs of merged PR runs,
which run files with the same concurrency as the lane they place.

**Reason and benefit**

The slowest job sets the CI wall-clock time for every pull request. The
chat lane goes from a 9.3-9.6 minute maximum to 6.5-6.8 minutes.

## What Changed

- `scripts/test-line-shard.mjs`: `partitionTestLines` takes optional
recorded durations. With durations, it balances source lines by recorded
time; entries without a record take the median. Without durations, the
behavior is unchanged.
- `scripts/chat-shard-durations.json` (new): recorded time per collected
chat entry, from the CI profile records of PR run
[43444854171](https://github.com/paperclipai/paperclip/actions/runs/43444854171).
926 of 928 collected entries are measured; the other two are the bogus
collection entries that the launcher already drops.
- `scripts/ci/refresh-chat-shard-durations.mjs` (new): builds that
manifest from downloaded `ci-profile-*` artifacts. Each executed case
maps to the collected `it`/`it.each` entry that registers it. A case
without an entry fails the refresh.
- `scripts/run-vitest-stable.mjs`: a sharded chat job runs three
partitions in place of two, and the chat launcher passes the recorded
durations. A chat job collects the suite once with `vitest list` in
place of once per partition, and the per-partition filter checks run at
the same time. Before, each partition ran two blocking collections of
the 86,000-line file one after another. The server worker count stays at
three.
- `scripts/ci/refresh-test-durations.mjs` and
`scripts/ci/pr-run-durations.mjs`: the weekly refresh reads general
server weights from the two newest merged-PR runs. Each run must have
the full server shard count from `.github/ci-task-matrix.json` and cover
at least 90% of the lane. Serialized weights still come from master
Cloud readiness runs.
- `scripts/general-server-shard-durations.json`: refreshed with the new
source (PR runs 43439145752 and 43439279537).
- Tests: `scripts/__tests__/run-vitest-stable-shard.test.mjs`,
`scripts/ci/test-selection-pr-timings.test.mjs` and
`server/src/__tests__/vitest-chat-shards.test.ts` cover the weighted
partition, the entry mapping, the manifest format, the PR-run collection
rules and the three-partition chat launcher.
- `doc/DEVELOPING.md`: describes the three chat partitions, the chat
weights and how to refresh them, and the new source of the server
weights.

## Verification

Experiments ran in this PR's own CI with a temporary resource sampler
(removed before review).

| Setting (run) | Server step vs. same shard's baseline | CPU busy | Max
memory used | Result |
|---|---|---|---|---|
| 3 workers (43444854171, shards 1-4) | +8% (profiling overhead) |
62-73% | 6.8 GB | 1 flaky retry |
| 4 workers (43444854171, shards 5-8) | -4% | 82-86% | 8.2 GB | clean |
| 5 workers (43444854171, shards 9-12) | -10% | 70-93% | 9.5 GB | 1 hard
failure: `decision-models.test.ts` timed out at 15 s twice |
| 4 workers (43450469416) | mixed m5a/m5n/m6i runners | 78-89% | 7.7 GB
| 1 hard failure: `workspace-recovery-warm-handoff.test.ts` timed out at
30 s on m5a |

The server lane is CPU bound: at four workers the CPU is 82-89% busy and
wall time improves by only about 10%. Five workers fail tests. Memory is
not the limit (no swap, no memory pressure, at least 6 GB free). Node's
compile cache and Postgres `fsync=off` showed no attributable gain
(import share stayed 29-34%; `fsync=off` cut iowait from 2-4% to
0.2-0.7% of CPU time), so neither is in this PR.

Final configuration (server: 3 workers with refreshed weights; chat: 3
weighted partitions per job), job minutes / test-step seconds /
effective concurrency:

| Shard | Baseline 43437558012 | Baseline 43441184316 | A: 43465156503
attempt 1 | B: attempt 2 | C: attempt 3 |
|---|---|---|---|---|---|
| server 1/12 | 11.0 / 609 / 2.81 | 10.2 / 562 / 2.82 | 10.1 / 535 /
2.88 | 11.6 / 615 / 2.86 | 11.3 / 600 / 2.86 |
| server 2/12 | 10.2 / 567 / 2.71 | 10.9 / 607 / 2.79 | 9.7 / 510 / 2.85
| 8.9 / 466 / 2.86 | 11.9 / 635 / 2.83 |
| server 3/12 | 10.9 / 609 / 2.88 | 10.9 / 601 / 2.86 | 9.4 / 498 / 2.81
| 11.7 / 623 / 2.80 | 20.3 / 1109 / 0.00 |
| server 4/12 | 11.0 / 613 / 2.77 | 11.2 / 622 / 2.84 | 11.3 / 602 /
2.84 | 9.3 / 487 / 2.83 | 12.5 / 676 / 2.84 |
| server 5/12 | 10.1 / 555 / 2.83 | 9.9 / 544 / 2.84 | 10.2 / 542 / 2.85
| 11.6 / 618 / 2.82 | 11.9 / 640 / 2.84 |
| server 6/12 | 9.9 / 551 / 2.85 | 10.4 / 579 / 2.85 | 9.8 / 516 / 2.69
| 9.1 / 480 / 2.68 | 9.2 / 480 / 2.67 |
| server 7/12 | 10.8 / 596 / 2.84 | 10.1 / 558 / 2.86 | 10.8 / 575 /
2.80 | 11.8 / 631 / 2.80 | 20.8 / 1129 / 2.69 |
| server 8/12 | 11.7 / 654 / 2.62 | 11.2 / 624 / 2.56 | 9.8 / 513 / 2.83
| 11.2 / 589 / 2.82 | 19.4 / 1059 / 2.40 |
| server 9/12 | 9.7 / 533 / 2.85 | 9.7 / 533 / 2.84 | 11.5 / 615 / 2.68
| 9.8 / 515 / 2.66 | 13.0 / 703 / 2.68 |
| server 10/12 | 9.8 / 540 / 2.79 | 9.5 / 502 / 2.81 | 10.6 / 561 / 2.84
| 11.8 / 625 / 2.83 | 9.3 / 488 / 2.83 |
| server 11/12 | 10.9 / 605 / 2.83 | 11.3 / 627 / 2.75 | 10.3 / 545 /
2.75 | 9.5 / 498 / 2.72 | 20.3 / 1107 / 0.00 |
| server 12/12 | 10.6 / 592 / 2.50 | 11.7 / 645 / 2.53 | 9.6 / 503 /
2.81 | 8.9 / 468 / 2.82 | 19.7 / 1065 / 2.70 |
| chat 1/3 | 9.6 / 528 / 1.56 | 9.3 / 511 / 1.58 | 6.6 / 310 / 2.69 |
5.7 / 272 / 2.65 | 6.8 / 331 / 2.69 |
| chat 2/3 | 7.8 / 418 / 1.74 | 7.6 / 408 / 1.75 | 6.3 / 305 / 2.57 |
6.5 / 311 / 2.58 | 5.5 / 260 / 2.58 |
| chat 3/3 | 6.5 / 337 / 1.74 | 6.4 / 333 / 1.73 | 6.0 / 285 / 2.65 |
5.4 / 258 / 2.68 | 5.8 / 262 / 2.69 |

Summary: chat maximum 9.3-9.6 min before, 6.6 / 6.5 / 6.8 min in A / B /
C. Server maximum 11.7 min before, 11.5 / 11.8 min in A / B; the server
worker setting is unchanged, so its run-to-run spread is mostly runner
type (m8i.xlarge shards took 8.9-9.8 min, m6a.xlarge 11.9-13.0 min).

Runners: A all m7i.xlarge; B m8i/m6i; C m6a/m6in/m8i/m5a. In C, the five
server shards on m5a.xlarge summed 2,545-3,037 s of file time against
about 1,700 s elsewhere; three reached the 20-minute job limit and two
failed tests on 15 s timeouts. The server lane settings in this PR are
the same as master's, so this is a fleet finding: m5a.xlarge cannot run
a server shard inside the job limit (follow-up for the fleet
configuration, outside this PR).

Flakiness of the final setting: the nine chat jobs in A, B and C had no
failure and no retried test. Server shards in A and B had no failure and
no retried test. Before the last commit (single chat collection), three
runs of 43456289623 had chat at 6.7-7.5 min with no chat retries; server
retries there (2 in one run, 1 in another) were on the unchanged
three-worker server setting.

Local checks (correctness only):

- `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/ci/test-selection-timings.test.mjs`: 68 passed.
- `pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/vitest-chat-shards.test.ts`: passed.
- `pnpm verify` exits 2 (global CI input changed), so the required CI
checks supply the broad coverage.

## Risks

- Three chat partitions per job use more memory: the peak was 11.4 GB of
16 GB, with at least 4.2 GB free and no memory pressure. CPU was 55-63%
busy.
- Stale chat weights only change balance, never which tests run. Shard
membership is still checked with `vitest list` before each run.
- The weekly refresh now downloads general server logs from merged-PR
runs. Runs that selected a subset of files are skipped before download
because their shard count is smaller.
- The server lane is still above 7 minutes. See the remaining floor
below.

**Remaining floor and shard-count recommendation (for a follow-up under
`.github/`)**

A server shard sums about 1,600 s of file time, and 3 workers on 4 vCPUs
give at most about 2.8 effective concurrency, so the test step is about
560-640 s. Job overhead is about 50 s. A job of 7 minutes needs a test
step of about 360 s, so about 1,000 s of file time per shard. The
current 19,200 s of general server file time therefore needs about 19-20
shards of 4 vCPUs at 3 workers. More workers on the same runners do not
help, because the CPU is already the limit.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
GitHub CLI) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:03 -06:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
DottaandClaude Opus 5.5 e9d7895648 fix(recovery): start an overdue saved-work recovery instead of looping on no pending path (#21478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run stops before its sandbox files are saved, Paperclip holds
the task and recovers the files first. A recovery queue does this work.
> - The queue runs two attempts at a time for the whole instance. A
remote attempt can take many minutes.
> - On staging, 29 recoveries waited with 0 attempts. The oldest waited
since 15:01Z. One task waited 3 h 33 min for its single attempt.
> - During that time the task said "Recovering saved work…", but the
assigned-work invariant saw no pending path. The sweep sent a
continuation wake every few minutes. The hold refused each wake, and the
activity log showed "could not start yet".
> - This pull request makes the queued recovery the task's pending path
for a bound. After the bound, the sweep starts the recovery itself
instead of sending a wake that the hold refuses.
> - The benefit: a held task gets its files recovered and continues
without a person, and the sweep stops its refused-wake loop.

## Linked Issues or Issue Description

**What happened?**
A scheduled task ran a legacy retry on a Daytona sandbox. Setup failed
with `setup_failed` ("remote_workspace_operation_timed_out: capture
(configured timeout 300000ms)"). Paperclip held the task for saved-work
recovery (`legacy_execution_requires_reconciliation`, resolved, replay
blocked, system recovery `scheduled`, 0 attempts). After 45 minutes the
recovery had not started. The task showed "Recovering saved work…" with
no Retry. The assigned-work sweep logged `issue.continued_automatically`
with outcome `not_started` and violation `no_pending_path` at 17:58,
18:02, 18:10 and 18:25 UTC.

**Expected behavior**
The recovery proceeds by itself and the task continues. If the recovery
cannot finish, the task shows a Retry that works. "Recovering…" never
shows with nothing behind it.

**Steps to reproduce**
1. A legacy run with a retained remote workspace fails during setup, and
its saved-work hold is adopted by system recovery.
2. The recovery queue does not reach the hold (it has two slots, and
they are busy).
3. The assigned-work sweep checks the task 10 or more minutes later. It
finds `no_pending_path` and sends a wake. The hold refuses the wake.
This repeats until the continuation cap.

The regression test
`server/src/services/saved-work-recovery-overdue.test.ts` reproduces
this state on real rows.

**Paperclip version or commit**
`master` at `c6301c92ff` (staging, 2026-11-13).

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with a Daytona sandbox on the
legacy runtime.

## What Changed

- `assigned-work-invariant.ts`: new pending intent
`saved_work_recovery`. A saved-work recovery that the queue still owes
counts as the task's pending path while an attempt runs, while its next
attempt is scheduled, or for `SAVED_WORK_RECOVERY_START_BOUND_MS` (10
minutes) after it became due.
- `assigned-work-invariant.ts`: new violation
`saved_work_recovery_overdue`, used when that bound passes. It comes
before the `no_pending_path`, `retry_offered` and blocked checks, so the
sweep never sends a wake that the hold refuses.
- `assigned-work-sweep.ts`: the sweep loads the fact with the same
eligibility predicate the recovery worker uses. For the overdue
violation it calls `startSavedWorkRecovery` and records no continuation
activity.
- `workspace-recovery.ts`: `startOverdueWorkspaceRecovery` starts one
attempt outside the queue's two slots, with at most two such attempts
per process. It skips runs that are already in flight and uses the same
ownership lock and attempt bookkeeping as the queue. The queue skips
runs that this path started. `recoverRunAction` moved to module scope.
Its "Retained workspace remains pending" warning now logs the error.
- `heartbeat.ts` and `recovery/service.ts`: wire the dependency.
- The sweep returns the end of the bound as the task's re-check
deadline, so the overdue start does not depend on another notification.

## Verification

- New `server/src/services/saved-work-recovery-overdue.test.ts` (real
Postgres, recovery fixture with a mocked sandbox):
- Overdue hold: the sweep starts the recovery and does not call
`enqueueWakeup`. No continuation activity is written. The system
recovery reaches `complete`. The next sweep records a fresh
continuation, and `deliverReconciledExecutions` starts a successor run.
- Hold within the bound: verdict `ok / pending_intent /
saved_work_recovery`. No wake is sent and no attempt starts.
- A second start for the same run returns `in_flight`. Exactly one
attempt is recorded, and a completed hold returns `not_eligible`.
- Red on the old invariant and sweep, with `no_pending_path` and
`enqueue_refused`, the same as staging. Green with this change.
- `server/src/services/recovery/assigned-work-invariant.test.ts`: 54
pass, with new cases for the bound, a running or scheduled attempt, a
stale attempt, and a pending question.
- `server/src/__tests__/assigned-work-invariant.test.ts`: 31 pass.
- `server/src/services/workspace-recovery-passive-fairness.test.ts`:
passes.
- `tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- A sweep-started attempt can run beside the queue's two attempts. That
adds up to two more concurrent recovery attempts per process, each with
its own database connection while it holds the ownership lock. Each
attempt for a run still takes that run's advisory lock, so two servers
cannot run the same recovery at once.
- The 10-minute bound is new. A queue that is slow but working now gets
help after 10 minutes. It does not get the hours it took on staging.
- Out of scope, reported for follow-up: why the queue was so slow on
staging (two slots for the whole instance, attempts that take minutes),
and why the setup capture hit its 300 s timeout. Server logs were not
available from this session.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser reads of the staging API, local test
runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:26:26 -06:00
d018a9d1cd fix(queue): retain chat bindings on resolved card continuations (#20825)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task can receive a card response while its agent runs.
> - The queue saves this response for the next turn.
> - Chat authorization requires the saved inbound message binding.
> - Queue promotion removed this binding when a prior turn received the
message.
> - This pull request retains live bindings for resolved card responses.
> - The next turn can check current chat access and continue the
approved work.

## Linked Issues or Issue Description

**What happened?**

An accepted plan card queued behind a chat run could start with a chat
source and no message IDs. Reviewed chat authorization then failed with
context_mismatch before provider work started.

**Expected behavior**

A resolved card must retain its live chat binding through queue
promotion. The new run must check current access. A consumed
message-only wake must not start another turn.

**Steps to reproduce**

1. Start a chat run with an inbound message.
2. Queue a card response with that message binding.
3. Complete the first run and release its queue.
4. Inspect the promoted context and check chat authorization.

Related work: #20689 cancels consumed message-only wakes. #19156
protects a successor checkout. This change retains the binding for an
independent card response.

## What Changed

- Retain delivered message bindings for resolved chat card responses
during queue promotion.
- Keep deleted, foreign-task, and self-authored message filters and
current chat authorization.
- Add a persisted queue regression and Slack/Discord
promotion-to-authorization checks, including access revocation.
- Document the queue behavior.

## Verification

- Before the fix, the persisted queue test failed because the promoted
context had no wakeCommentIds. Three existing tests passed.
- The first focused run passed 108 tests. The added Slack and Discord
checks passed after promotion and refused revoked access.
- Final head a38bcb9bc5: 110/110 focused
tests pass. Low-memory server typecheck passes. pnpm verify exits 2 and
requires broad hosted CI. CI typecheck, build, all
E2E/general/serialized tests and Runner checks pass except the Runner
support suite and its aggregate gate. The support suite fails on
inherited stale SDK lock pins (source-sdk-public-type,
source-sdk-evidence and source-sdk-public-inputs); existing PR #20623
owns that repair and has passing CI. No retry is appropriate for this
deterministic failure. Greptile reviewed this exact head at 5/5 with no
actionable findings; there are no inline or unresolved review threads.
- The hosted failed task independently resumed. No source task replay or
deployment was performed. Fixed-release hosted provider acceptance
remains unverified.
- Preflight installed dependencies but could not build Rust binaries
because Cargo is absent. The first server typecheck ended without a
diagnostic; the low-memory retry passed.

## Risks

An overly broad exception could deliver consumed input twice. The
exception applies only to resolved card responses with a chat source.
Message-only wakes still cancel. Message existence, task scope, author
filters, execution ownership, and current chat access still apply.

## Model Used

OpenAI GPT-6 Codex agent with reasoning, repository tools, and code
execution. The runtime did not expose a more specific model build ID.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:12:02 -06:00
Devin Foley b4861d5e37 fix(diagnostics): report Daytona acquisition state for runs retried as provider outages (#21476)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts on a sandbox environment, the host gets an
environment lease. On Daytona, this creates a sandbox.
> - When the create does not finish in time, the Daytona plugin records
the failed phase, the elapsed time, the budget, and the Daytona sandbox
state. #15736 and #15946 added these fields to the run failure report,
so that operators can see where the acquisition stopped.
> - #20857 changed the error code of these failures from `setup_failed`
to `environment_provider_unavailable`, so that the host retries them.
> - The run failure diagnostics read the acquisition fields only when
the error code is `setup_failed`. Thus every Daytona create timeout now
reaches the error report without its phase or provider state.
> - On staging, more than 270 Daytona lease timeouts occurred in three
days, and none of them shows the provider state.
> - This pull request also reads the acquisition fields when the error
code is `environment_provider_unavailable`.
> - The benefit is that the error report shows again whether the sandbox
was stuck in `creating`, in a snapshot state, or absent. This evidence
is necessary to find the cause of the timeouts.

## Linked Issues or Issue Description

Refs #20857, Refs #20828, Refs #15946, Refs #15736

**What happened?**
A Daytona lease acquisition timeout is now classified as
`environment_provider_unavailable` (#20857). The run failure report for
this failure does not contain `environmentAcquisitionPhase`,
`environmentAcquisitionElapsedMs`, `environmentAcquisitionBudgetMs` or
`environmentAcquisitionProviderState`.

**What did you expect?**
The report contains the acquisition phase and the provider state, as it
did for `setup_failed` runs before #20857.

**Steps to reproduce**
1. Let a Daytona sandbox create pass its acquisition deadline.
2. The plugin returns "Daytona lease acquisition timed out; allocation
cleanup is pending" with an acquisition diagnostic.
3. The host fails the run with `environment_provider_unavailable`.
4. Look at the `run_execution` context of the error report. The
acquisition fields are not there.

## What Changed

- `collectRunFailureDiagnostics` reads the private acquisition
diagnostic for a setup failure with the error code `setup_failed` or
`environment_provider_unavailable`. The other conditions do not change:
the phase must be `setup`, and the diagnostic must belong to the same
run and company.
- A new test builds the real wrapper chain (lease failure, then provider
failure, then the RPC error that holds the diagnostic). It checks that
the fields are reported for the failed run, and that they are not
reported for another run or for the execute phase.

## Verification

- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts`
fails on master (the report has only `failurePhase`). It passes with
this change.
- `cd server && npx vitest run
src/services/__tests__/run-failure-diagnostics-provider-unavailable.test.ts
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts`: 3 files, 139
tests pass.

## Risks

Low risk. The change adds four bounded scalar fields to an error report.
These fields already exist for `setup_failed` runs. The change does not
alter retries, the error grouping, or the run result.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in
an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-11-13 12:04:25 -08:00
Devin FoleyandClaude Opus 5.5 89ab7e51ff fix(recovery): retain a failed run's sandbox in one lease write (#21474)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute in remote sandboxes. Each run holds an
environment lease row that points to its environment, task, workspace
and company
> - When a run fails before it copies its files back, Paperclip keeps
the sandbox for a file repair. It writes a stop-and-retain intent and a
repair-hold policy on the lease
> - The recorder wrote the same lease row two times in one transaction.
Postgres runs the foreign-key checks again on a row that the current
transaction already wrote, and these checks take `FOR KEY SHARE` locks
on the referenced rows
> - Other runs lock the shared environment row `FOR UPDATE` when they
acquire a lease. Then the second write waits, meets the 1 second lock
timeout, and the run fails with "Database query failed (55P03)"
> - This pull request writes the stop intent and the repair hold in one
update
> - The benefit is that a busy shared environment no longer fails runs,
and the file repair hold is recorded

## Linked Issues or Issue Description

**What happened?**
On staging, native OpenCode runs failed with `adapter_failed` and the
error "Database query failed (55P03)". The failed statement was `update
"environment_leases" set "lease_policy" = 'retain_on_failure',
"metadata" = "metadata" || {workspaceRestoreRecovery...} where ...
"status" = 'pending_cleanup'` in
`preserveLegacyWorkspaceRestoreSources`. The cause was `canceling
statement due to lock timeout`. The statement does not wait on the lease
row, because the transaction already holds it `FOR UPDATE`. It waits on
the foreign-key check `SELECT 1 FROM ONLY "public"."environments" x
WHERE "id" = $1 FOR KEY SHARE OF x`. Postgres runs this check because
the transaction wrote the same row one statement before, in
`prepareSandboxStopAndRetain`.

**What did you expect?**
The run records its workspace restore failure and keeps its sandbox for
repair, also when another run holds the shared environment row.

**Steps to reproduce**
1. Start a run with an active, ephemeral sandbox lease.
2. In another session, lock the lease's `environments` row with `SELECT
... FOR UPDATE` and keep the transaction open.
3. Record a workspace restore failure for the run
(`terminalizeLegacyExecution`, legacy or native
`recordRestoreFailureOnly`).
4. Before this change, the second lease write fails after 1 second with
55P03. The new test
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`
shows this for legacy and native runs.

No public issue exists. I searched open and recent pull requests for
this lock timeout and found no duplicate. #21071 bounds a different lock
wait (workspace source locks in run preparation) and does not overlap.

## What Changed

- `server/src/services/sandbox-stop-and-retain.ts`:
`prepareSandboxStopAndRetain` accepts an optional `retention` argument
(`leasePolicy` and extra `metadata`). The stop-and-retain update sets
these values in the same statement. Other callers do not change.
- `server/src/services/legacy-workspace-restore-recovery.ts`: the
active-source path passes the repair hold (`retain_on_failure` and
`workspaceRestoreRecovery` with `originalLeasePolicy`) to that update.
It no longer does a second update on the same row.
-
`server/src/__tests__/legacy-workspace-restore-lock-contention.test.ts`:
a regression test for legacy and native runs. Another session holds the
environment row `FOR UPDATE` while the run records its restore failure.
The test checks the run's recovery evidence and the lease policy,
status, stop intent and repair metadata.

## Verification

- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-lock-contention.test.ts`: both
cases fail on `master` with `canceling statement due to lock timeout` on
the `environments` `FOR KEY SHARE` check. Both cases pass with this
change.
- `cd server && npx vitest run
src/__tests__/legacy-workspace-restore-recovery.test.ts`: 73 existing
tests pass.
- Related suites pass (10 files, 485 tests):
`heartbeat-observed-workspace-preservation`,
`heartbeat-native-remote-failure-retention`, `live-recovery-work`,
`workspace-recovery-stop-proof`, `native-failed-workspace-recovery`,
`native-cancelled-workspace-preservation`,
`workspace-recovery-admission`, `native-workspace-reconstruction`,
`environment-retained-orphan-sandbox`, `abandoned-workspace-files-hold`.
- I did not complete a local server typecheck. The process ran out of
memory in the 10 GB sandbox. CI runs the typecheck on this head.

## Risks

- Low risk. The lease ends with the same status, policy and metadata as
before. The values are written in one statement instead of two. The
update keeps its ownership guards (status, provider, provider lease ID
and exact metadata).
- Other code paths can still write a lease row two times in one
transaction (for example the shutdown settlement path, which sets a
lease back to `active` before it records the restore failure). That path
handles 55P03 as a lock timeout and retries. This pull request does not
change it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, code execution, embedded Postgres tests, and
GitHub tools).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:58:00 -08:00
DottaandClaude Opus 5.5 89f486cdab fix(heartbeat): run one dispatch pass per agent, back off refused claims, and enforce the run limit in the claim (#21433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. Each trigger (a
run finishes, a wake arrives, a sweep runs) asks for one dispatch pass
of that agent
> - An agent could run more runs than its `maxConcurrentRuns`, dispatch
passes could stack up without a limit, and every pass offered the same
unclaimable runs again
> - On hosted staging (e23549cd9c), one agent with limit 20 had 20–35
runs `running` and 170–270 queued. The logs showed 375 "offered a free
slot but could not be claimed" lines and 117 "claim was rejected" lines
in 10 minutes
> - This pull request runs one pass per agent at a time, folds triggers
into one follow-up pass, backs off refused claims, enforces the limit in
the claim transaction, and reads only the columns the pass needs
> - The benefit is that an agent never runs past its limit (also across
two servers during a deploy), and a burst of triggers costs one or two
passes, not one pass per trigger

## Linked Issues or Issue Description

Refs #21391, #21242, #21234, #21277, #21324, #21349

**What happened?**

On hosted staging (Core e23549cd9c), one agent ("Codie",
`maxConcurrentRuns` 20) had 170–270 queued runs and 20–35 runs
`running`, so it was over its limit. The logs showed a refused claim
offered again on every pass. This pull request is about correctness and
load. It is not the cause of the staging out-of-memory crashes: a
separate leak (`sessionToolAuthorityEpochs` in
`native-session-executor.ts`, about 3 MB per finished native run)
explains those, and another change fixes it.

Live again at 17:02Z on staging (Core 04f12fd2be, which includes
#21391): the same agent had 52 runs `running` against
`maxConcurrentRuns` 40. All 52 started between 16:55Z and 17:02Z, right
after a deploy released the backlog, with 112 runs still queued. So
overlapping passes exceed the limit with claimable runs, not only with
unclaimable ones.

**Why did it happen?**

1. `withAgentStartLock` chained each trigger behind the previous one and
measured its 30 s stale limit from when the predecessor was chained, not
from when it started. When triggers arrived faster than one per pass,
every trigger after about 30 s of backlog started its own pass next to
the others.
2. Each pass counted free slots by itself, outside any lock that other
passes or another server see. Overlapping passes and two servers during
a deploy swap each claimed up to the limit.
3. Each pass read every queued run with `select *`, including the large
`contextSnapshot`, and offered every refused run again with no backoff.

A local reproduction of the staging shape by another session (250
unclaimable queued runs with 8 KB snapshots, one trigger per second, on
e23549cd9c and again on master e7e28ba241): passes that run at the same
time went 1 → 12 → 39 → 79 → 139 at t = 30/45/75/116/178 s, only 4
passes finished in 178 s, and post-GC heap grew by about 2.8 MB per
stacked pass (519 MB → 1,061 MB at 190 passes, mostly the parsed
`heartbeat_runs` rows and the dispatcher facts each pass holds). 242 s
after the triggers stopped, 216 passes were still outstanding.

**Expected behavior**

At most one dispatch pass per agent runs at a time. Triggers that arrive
during a pass cause one follow-up pass. The `running` count never goes
above `maxConcurrentRuns`. A refused run is offered again after a
bounded wait, and no queued run is stranded.

**Steps to reproduce**

1. Give an agent a limit of 4, 12 aged queued runs that the chat-control
gate refuses, and 12 claimable runs.
2. Start a pass. While its first claim waits, send 40 triggers, each
after more than 30 s of the old lock's clock.
3. Count the passes, the refused offers, and the highest `running`
count.

## What Changed

- `server/src/services/agent-start-lock.ts`: replaces the stale-bypass
lock with `dispatchForAgent`. One pass runs per agent. Triggers during a
pass merge into one follow-up pass that all their callers share. A
trigger from inside the agent's own pass does not wait on itself. A pass
has a 60 s deadline on a monotonic clock, measured from when the pass
started (not from when a waiter was chained). At the deadline the pass
stops claiming (an abort signal) and the follow-up may start. A trigger
that arrives after the deadline also ends the pass, so a stalled timer
cannot keep an agent blocked. Post-pass effects run after the agent is
free, as before.
- `server/src/services/heartbeat/run-slot-admission.ts` (new):
`assertAgentRunSlotFree` counts the agent's `running` runs and compares
them with `maxConcurrentRunsSql()` inside the agent-scoped advisory lock
that every claim already takes (`withExternalAdmissionGuard`). It throws
`AgentRunSlotsFull` when the agent is full. The count leaves out the run
that the claimed run replaces (`retryOfRunId` or `previousRunId`), so a
successor that starts while its predecessor finishes is not refused.
- `server/src/services/heartbeat/queue.ts`:
- Every claim path (ordinary, queued comment, native review through
`native-review-dispatch.ts`) checks the slot inside its claim
transaction. A full agent leaves the run queued, and the pass stops.
- The pass reads only the columns the dispatcher needs, with `issueId`,
`wakeReason` and `previousRunId` taken from `contextSnapshot` in SQL. It
reads the full row only for the run it is about to claim.
- A refused claim (declined or 4xx) of a run that has waited more than
an hour (the existing aging limit) is not offered again for 5 s, then
10, 20, 40, and at most 60 s. A person's run of that age waits 2 s. A
run queued for less than an hour is never backed off, so a successor or
retry that waits for its predecessor starts as soon as the predecessor
ends. A backoff also ends when the run's task changes (status,
`updatedAt` or execution run) or when a run of that task ends in this
process. A backed-off run does not take the oldest-run share. The
queued-run sweep (`resumeQueuedRuns`, `resumeQueuedRunsAfterDrain`,
which the queued-run redrive calls) offers refused runs again regardless
of the backoff.
- A pass makes at most 24 claim attempts. When it reaches that limit it
asks for one follow-up pass that skips every run the chain already
tried, so the follow-up reaches runs further down the line. The chain
ends when a pass does not reach the limit, so refused runs cannot cause
a continuous chain of passes. Only this automatic follow-up carries the
tried list. A wake, a completion or another fresh trigger that merges
into it resets the list, so a run whose blocker cleared during the chain
is offered again.
- Admission, suppression and cutoff are read once per pass, not once per
trigger.
- Seam for a skip before the claim: in `runQueuedDispatchPass`
(`claimQueuedRunsForAgent`), right after the skip line (`if
(triedRunIds.has(candidate.id) || (!retryRefused &&
claimBackedOff(candidate.id, new Date(), taskStamp))) continue;`),
before the claim-budget check, `claimAttempts += 1` and the full-row
read. A `continue` there spends no claim attempt, reads no row, does not
enter the claim backoff or the aged-decline note, and does not call
`dispatcher.claimed`, so the run keeps its place and its oldest-run
share. The dispatcher exposes `dispatcher.personOrigin(run)` (the
person-origin fact it already derives) and `dispatcher.personRun(run)`.
The candidate rows keep `id`, `companyId`, `agentId`, `status`,
`invocationSource`, `retryOfRunId`, `wakeupRequestId`,
`scheduledRetryAt`, `scheduledRetryAttempt`, `scheduledRetryReason`,
`createdAt`, `startedAt`, and
`contextSnapshot.issueId/wakeReason/previousRunId`.
- Each pass is registered with the execution drain
(`drainActiveRunExecutions`) until it ends or reaches its deadline.
- Tests: new `server/src/services/agent-start-lock.test.ts` and
`server/src/__tests__/queued-run-dispatch-stacking.test.ts`.
`server/src/__tests__/heartbeat-start-lock.test.ts` is removed: it
asserted the 30 s stale bypass that this pull request removes.
`workspace-recovery-preparation-dispatch.test.ts` holds and counts
passes through the new function.

The chat-control fallback lookup from #21391 (`issueId = X OR
nativeIssueId = X OR contextSnapshot->>'issueId' = X`) was measured on
embedded Postgres with 200,000 runs in one company. Postgres uses a
BitmapOr of three existing indexes
(`heartbeat_runs_company_issue_created_idx`,
`heartbeat_runs_company_native_issue_contract_id_uq`,
`heartbeat_runs_company_ctx_issue_created_idx`), not a sequential scan:
0.33–0.45 ms per call for a recent previous run, a 30-day-old previous
run, and no previous run. It is not changed.

## Verification

Red/green, `server/src/__tests__/queued-run-dispatch-stacking.test.ts`
(embedded Postgres, real heartbeat queue, limit 4, 12 refused aged runs,
12 claimable runs, 41 triggers):

| Measure | master | this pull request |
|---|---|---|
| Passes | 42 | 2 |
| Peak passes at the same time | 40 | 1 |
| Peak claims in flight | 9 | 1 |
| Refused-claim offers per trigger | 12 | 0.29 (12 in total) |
| Peak `running` (limit 4) | 8 | 4 |
| Passes outstanding after the burst | — | 0 |

- "waits a bounded time before offering a declined/rejected run again":
on master 10 triggers offer the refused run 10 times. With the fix they
offer it once, and once the backoff passes it is offered again. The
rejected case uses the 422 "Unable to resolve responsible user" path
from the staging reproduction.
- "starts a run that becomes ready during a chain of passes when a fresh
trigger arrives": red at 3d48c4bed2, green now.
- "stops chaining passes once every refused run was tried": 60 refused
runs and one trigger cause exactly 3 passes, then no more.
- "starts runnable work queued behind more refused person runs than one
pass tries": 30 refused person runs ahead of 3 runnable runs, with free
slots. The runnable runs start after one trigger, in at most 3 passes.
At 68e64a2de9 the same 24 person runs were offered on every pass and the
runnable runs never started.
- `server/src/services/agent-start-lock.test.ts`: one pass for a burst
of 50 triggers plus one follow-up, no second pass after 45 s, a hung
pass stopped at its deadline without overlap, no self-wait, settle after
release, and errors passed to the caller.
- `node server/scripts/typecheck-layers.mjs`: pass.
- Vitest (embedded Postgres): `heartbeat/queue.test.ts` (102),
`recovery-wake-herd.test.ts` (35),
`heartbeat-queued-run-claim-isolation.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`issue-reassignment-queued-run.test.ts`,
`queued-comment-consumption.test.ts`,
`issue-queued-comment-queue.test.ts`, `chat-run-milestone-work.test.ts`.

## Risks

- Refusals of runs queued for less than an hour are not backed off. A
backoff there delayed successors that wait for in-process predecessor
ownership, which can end without a task change (two CI tests failed with
it). Per trigger, such a run costs at most one claim attempt.
- A refused run that has waited more than an hour now waits up to 60 s
(a person's run 2 s) before an event-driven pass offers it again, unless
its task changes. The sweep and the redrive ignore the backoff.
- A successor may start while the run it replaces is still `running`.
The limit check leaves that one predecessor out of the count.
- Claims of an agent that is at its limit now fail inside the
transaction. A run that `executeRun` claims directly (for example a
native resume) stays queued when the agent is full, and the next pass
starts it.
- A pass that passes its 60 s deadline can still finish one in-flight
claim next to the follow-up pass. The limit check in the transaction
keeps the `running` count within the limit.
- No schema or migration change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:55:38 -06:00
aa4a06653c fix(native): name the recorded turn failure when a native run fails (#21235)
Failed native runs now report the runner's recorded turn-failure code instead of the generic unclassified fallback, and the Sentry run-failure fingerprint includes runner:<code>.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:22:55 -08:00
DottaandPaperclip 08341ea134 fix(queue): let board users discard agent-authored queued messages (#21458)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The task composer holds messages that an agent has not consumed yet.
> - A board operator needs to remove an agent-authored pending message
before it starts more work.
> - The queue currently grants discard only to the human author, so
these messages have a disabled trash button and the API rejects the
board user.
> - This change derives the board discard permission in the queue
snapshot and uses that same permission under the existing issue lock.
> - The route checks task access before it changes the queue. The
benefit is a working discard action with the existing company, privacy,
revision, and dispatch guards.

## Linked Issues or Issue Description

Fixes #15464.

Supersedes #15465 with an independent implementation and red/green
regression evidence. Thanks to @MMTE for identifying the gap and
proposing a fix. Related: #18958 addresses deletion of comments that
have already been delivered; this change covers pending queue entries.

## What Changed

- Queue responses grant board users discard permission for
agent-authored messages. Editing and other human authors' messages
retain their existing permissions.
- The discard use case honors the actor-specific permission rebuilt
under the queue lock. Author agents keep their existing cancellation
path.
- The board discard route checks private-task access before deletion.
The denial for a non-author agent now points to the board action instead
of saying that no board action exists.
- Tests cover queue states, cancellation of an empty continuation,
persistence, queue linkage, audit attribution, private-task denial, and
cross-company denial. The API reference describes the supported action.

## Verification

- Red before the fix: the two snapshot regressions returned `canDiscard:
false`; the discard regression threw
`QueuedCommentMutationForbiddenError`. The route regressions also caught
the disabled permission, missing private-task guard, and stale agent
denial text.
- Green after the fix: 30 queue snapshot and use-case tests; all five
focused route regressions with real disposable Postgres; six Postgres
adapter tests; 42 existing queue UI tests.
- Server typecheck: `cd server && node scripts/typecheck-layers.mjs`
passes. `node scripts/check-module-boundaries.mjs` and `git diff
--check` pass.
- The [complete PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43484735280)
passes on head `164200602828e23f01a8b209ff954821c5e14e9a`. [Greptile
reports
5/5](https://github.com/paperclipai/paperclip/pull/21458#issuecomment-6861824451)
on that same head with no open review threads.
- Local scope: the 83 focused snapshot, use-case, route, adapter, and UI
checks pass. The existing queue UI suite has 42 tests. The new route
cases use real disposable Postgres and verify the resulting database
state after reload.
- Additional local full-suite attempt: 174 tests passed and 13
approval-related tests failed with cleanup timeouts, Postgres deadlocks,
and duplicate-prefix errors after cleanup failed. The first failure,
`queues request_confirmation resolution while its legacy source run is
still running`, also fails on unchanged base code with the same
30-second `afterEach` and `afterAll` timeouts. This broader local run is
not counted as a pass. No test timeout or assertion was weakened. CI
supplies the broad passing result for the committed head.
- `node scripts/local-verify.mjs --explain` returned exit 2 because the
API documentation is a global CI input; all of its required CI lanes
passed. The initial adapter attempt could not initialize local Postgres
under the sandbox; the same six checks passed with local-process access.
The initial cross-company fixture was corrected to use a distinct
company prefix.

## Risks

- This grants board operators a new action on agent-authored pending
messages. The route must pass company membership and private-task access
checks; the mutation retains its transaction, revision, and dispatch
protections.
- No schema or migration is required. The UI uses its existing discard
control and error handling.
- The staging task was inspected to confirm the cause. No staging data
was deleted, and a deployed UI retest is not claimed by these local and
CI checks.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and code execution. The exact runtime model ID and context limit are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — the focused regression,
adapter, and UI checks listed above; the additional broad local cleanup
failure is documented separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 13:03:46 -06:00
commitperclip[bot] 8b7b2a0701 fix: Recognize Git mount-boundary absence diagnostics in workspace checks (#21048)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

> [!WARNING]
> This change crosses a trust boundary. See Risks.

## Thinking Path

- Issue #20525 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Both callers recognized only
the ordinary Git absence diagnostic. Preserve the earlier exact boundary
matcher and all existing exit, signal, environment, and filesystem
checks.
- This pull request: Recognize Git mount-boundary absence diagnostics in
workspace checks

## What Changed

- `server/src/services/git-mount-boundary-message.ts`: Retain the shared
complete-diagnostic matcher, which rejects partial and appended output.
- `server/src/services/managed-repository-checkout.ts`: Retain boundary
recognition without changing ordinary-diagnostic behavior or fallback
guards.
- `server/src/services/heartbeat/workspaces.ts`: Retain boundary
recognition and ancestor metadata inspection.
- `server/src/__tests__/git-mount-boundary.test.ts`: Add root mount,
CRLF, and embedded-line-break cases through both callers. Confirmed six
failures with boundary recognition disabled; restored implementation
passes this suite, both neighboring suites, and typecheck.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/git-mount-boundary.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- The repository's compile command passes with the fix applied.
- 10 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.
- It crosses a trust boundary. Check each crossing against what the
issue asked for, and against who asked:
- validation (server/src/services/heartbeat/workspaces.ts:860): Broadens
the accepted Git absence diagnostics to include the mount-boundary
message when classifying a checkout failure as an expected
non-repository result.
- validation (server/src/services/managed-repository-checkout.ts:265):
Allows the mount-boundary diagnostic to bypass a previously thrown Git
error when plainTaskDirectory and the existing exit, signal, and
environment conditions hold.

## Model Used

- opus (Anthropic Claude, via Claude Code), writing and checking the
change in an isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #20525

#20525 also mentions, without asking this fix for it:
- Non-Git directories in close readiness. (The report explicitly
identifies merged #19553 as handling 'a different caller'; this patch
does not modify close readiness.)
- Initial managed-clone readiness. (The report says '#2977 concerns
initial managed-clone readiness, rather than this diagnostic matcher.')
- CI jobs 131047332629 and 131040975860 fail; fix the change only if it
causes the failures. (This earlier-round conditional comment supplies no
demonstrated matcher defect. .github/workflows/pr-trusted.yml:1801-1804
asserts aggregate shard status, while
server/src/services/tool-access/catalog/health.ts:222 reports MCP health
failures; neither changed here. Historical job causation cannot be)

A search of paperclipai/paperclip's open and closed pull requests found
no other pull request that names #20525.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=20525 base=156453fe7c67
report=513d758246cb2ac5 -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:59:22 -06:00
commitperclip[bot] 6f2ea3b4d5 fix: Complete run cancellation when lifecycle event append fails (#20215)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

> [!WARNING]
> This change crosses a trust boundary. See Risks.

## Thinking Path

- Issue #20116 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Cancellation awaited the
lifecycle event append after cancelling the run and wakeup. An append
failure rejected the stop before issue release, agent finalization, and
queue advancement.
- This pull request: Complete run cancellation when lifecycle event
append fails

## What Changed

- `server/src/services/heartbeat/run-control.ts`: Log cancellation event
append failures and continue essential cleanup, matching lease-release
terminalization.
- `server/src/services/heartbeat/run-control-event-failure.test.ts`:
Cover queued and running cancellation with an injected append failure
and the real issue-release adapter. Confirmed regression failures before
the fix; regression tests, neighboring tests, and final typecheck pass.

## Verification

- Reproducing test(s), added in this PR:
`server/src/services/heartbeat/run-control-event-failure.test.ts`,
`server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts`,
`server/src/services/heartbeat/run-control.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 9 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.
- It crosses a trust boundary. Check each crossing against what the
issue asked for, and against who asked:
- logging (server/src/services/heartbeat/run-control.ts:1665): Logs the
event-append error and run ID to the server log when cancellation event
recording fails.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #20116

The reviewer found the base already handles:
- A closed request socket produces a misleading PATCH 200 log that hides
the failure. (server/src/middleware/logger.ts:89–91 detects aborted or
unfinished responses and formats 'request aborted'; lines 150 and
157–158 log these at warn level with that message instead of the default
status-based success message.)

A search of paperclipai/paperclip's open and closed pull requests found
no other pull request that names #20116.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=20116 base=ca34bed25797
report=e2609f9803f7cc0c -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:58:29 -06:00
commitperclip[bot] a6608bd735 fix: Preserve plugin UI UUID validation and test trailing line terminators (#20058)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

## Thinking Path

- Issue #17332 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: Plugin keys reached a
UUID-column query, and Drizzle's nested PostgreSQL error bypassed the
original error handler. The existing UUID guard fixes this; its
non-multiline JavaScript regex also rejects trailing newlines.
- This pull request: Preserve plugin UI UUID validation and test
trailing line terminators

## What Changed

- `server/src/routes/plugin-ui-static.ts`: Retain the existing guard,
PostgreSQL UUID forms, and key fallback.
- `server/src/__tests__/plugin-ui-static.test.ts`: Add route tests for
four trailing line terminators and accepted PostgreSQL UUID forms.
Confirmed the original lookup fails; all 58 focused tests and typecheck
pass with the guard.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/plugin-ui-static.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- 3 independent attempts wrote 3 reproductions; this fix passes every
one of them.
- The repository's compile command passes with the fix applied.
- 2 existing test file(s) near the change passed.
- The full test suite passes except for the 16 file(s) that already fail
on the base commit.
- Revised in 1 round(s) of automated code review.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.

## Model Used

- codex:gpt-6.1-sol, opus, sonnet (Anthropic Claude, via Claude Code;
OpenAI, via the Codex CLI), writing and checking the change in an
isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #17332

Other pull requests that name #17332:
- #17433 (open): fix(plugins): serve a plugin UI bundle addressed by
plugin key
- #17335 (open): fix(server): serve plugin UI assets requested by plugin
key

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=17332 base=ca34bed25797
report=b2e95a170c3be729 -->

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:58:11 -06:00
commitperclip[bot] 0eeee6ae96 fix: Defer chat milestone result reads and fix regression test transaction typing (#19875)
> [!NOTE]
> Opened by Prospector's automated issue-fix pipeline. A maintainer
reviews and decides; nothing here merges on its own.

## Thinking Path

- Issue #16728 reports a defect in paperclipai/paperclip.
- Root cause, as the fixing agent read it: PostgreSQL could evaluate
both result_json predicates before publication filtering, detoasting
already-published runs. The regression test also passed transaction
handles to helpers typed for full database handles.
- This pull request: Defer chat milestone result reads and fix
regression test transaction typing

## What Changed

- `server/src/services/chat-run-publications.ts`: Retain
publication-dependent guards around both result JSON predicates in the
shared milestone query.
- `server/src/__tests__/chat-run-milestone-toast.test.ts`: Use the
neighboring tests' transaction cast convention. Confirmed both cases
fail without the guards and pass with them; all seven selected milestone
test files and typecheck passed.

## Verification

- Reproducing test(s), added in this PR:
`server/src/__tests__/chat-run-milestone-candidates.test.ts`,
`server/src/__tests__/chat-run-milestone-toast.test.ts`
- They fail on the base commit and pass with the fix, each twice in a
fresh sandbox.
- The repository's compile command passes with the fix applied.
- 8 existing test file(s) near the change failed, and fail on the base
too, so they cannot show whether the change breaks them.
- The full test suite passes except for the 15 file(s) that already fail
on the base commit.

## Risks

- Written by an automated pipeline, not a person: review it as you would
an outside contribution.
- Risk tier of the touched paths: 2.

## Model Used

- opus (Anthropic Claude, via Claude Code), writing and checking the
change in an isolated sandbox.
- opus (Anthropic Claude, via Claude Code), reviewing it independently.

## Linked Issues

Fixes #16728

The reviewer found the base already handles:
- Milestone projection still runs on a one-second cadence.
(server/src/services/chat-run-milestone-work.ts:12 describes and
implements discovery on startup/committed changes with retries for
failures or unfinished batches; server/src/app.ts:1269 registers this
work owner. This code exists independently of the proposed predicate
change.)

Other pull requests that name #16728:
- #18686 (open): perf(runs): read result_json once per run in run lists
and index agent runs by created_at

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with 'Fixes: #' / 'Closes
#' / 'Refs #' OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub '#NNN' / 'github.com/paperclipai/paperclip'
URLs)
- [ ] My branch name describes the change (e.g. 'docs/...', 'fix/...')
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

<!-- prospector:issue-fix v1 issue=16728 base=d1ee25d0f722
report=5924b3e8dd6d2fe4 -->

---------

Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com>
2026-11-13 12:57:45 -06:00
DottaandPaperclip 5859e7cfa6 perf(apps): avoid repeated writes during permission saves (#21033)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps give agents access to tools through saved profiles and approval
rules.
> - A save replaces the full set of action entries and access bindings.
> - It also updates each Ask first rule and each retired rule in a
separate query.
> - Adding one agent can therefore cause hundreds of database round
trips.
> - This pull request keeps unchanged rows and batches the required
changes.
> - The benefit is faster saves with the same action permissions.

## Linked Issues or Issue Description

**What happened?**

An app permission save repeats database work for actions and agents that
did not change. With 100 actions and 500 retired rules, one agent
addition makes 636 queries. Of these, 599 update approval policies.
Related boundary coverage: Refs #19973. This change keeps that issue's
Off and agent-selection protections.

**Expected behavior**

Adding an agent applies the saved Allowed, Ask first, and Off choices.
The save changes only the rows that need a change. Independent approval
rules remain in force.

**Steps to reproduce**

1. Create an app with 100 active actions and 99 Ask first rules.
2. Add 500 disabled app approval rules.
3. Save access for three agents. Then add a fourth agent.
4. Count database queries during the save.

**Paperclip version or commit**

Baseline: `51d77648bc62`.

**Deployment mode**

Built from source. The reproduction uses disposable local PostgreSQL.

## What Changed

- Keep action entries and access bindings when their saved values match.
- Insert new bindings in one query. Remove only bindings and entries
that no longer apply.
- Read managed approval rules for the selected app. Skip unchanged rules
and rules that are already disabled.
- Batch new or changed Ask first rules and disable stale enabled rules
in one query.
- Keep existing policy conditions, priority, author, and creation time
when a rule changes.
- Add real PostgreSQL regression tests and a repeatable benchmark.
Document independent approval rules.

## Verification

- Red: the original regression made 2,995 policy writes across five
saves and failed the zero-write check.
- Green: 636 focused service and runtime tests passed after the rebase.
CI supplies the broad checks required by `pnpm verify` (exit 2).
- Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62
--output=/tmp/app-permission-save.json`.
- Local benchmark: two warmups and six samples in alternating order.
Restore three agents outside each timed sample, then add the fourth
agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times
faster). Query count fell from 636 to 32. Unchanged policy writes fell
from 599 to zero.
- Compare all saved profile, entry, binding, and policy values after
mapping fixture IDs and excluding generated row IDs and write times.
Check Allowed and Ask first decisions for the added agent. Tests also
cover Off, an unassigned agent, additive access, removal, independent
policies, company boundaries, and re-enabled rules.
- CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023).
This includes typecheck, build, server tests, and all eight browser
shards.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829)
gives the final head 5/5. Its one finding is resolved. The separate
Paperclip Review check is still queued. Its review task stopped after a
Runner failure.
- Hosted request timing and the specific reported approval prompt are
not verified. No live permissions changed.

## Risks

- The save now retains unchanged row IDs and timestamps. Consumers must
use permission values to determine access.
- Existing transaction locks and permission checks remain in place. The
runtime concurrency tests exercise these paths.
- The benchmark measures a local service with synthetic data. It does
not establish hosted endpoint latency.
- No migration or startup change is required.

## Model Used

OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this
session. Used reasoning, tool calls, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:57:03 -06:00
DottaandPaperclip 6ad4529c5a fix(native): preserve pack proof across interrupted reattachment (#21246)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can connect again to a live Runner after a restart.
> - Each new controller claim changes provider-pack admission to
pending.
> - The claim kept the previous verified proof only in local memory.
> - If recovery failed before its callback, the next claim lost that
proof.
> - This pull request saves a small reference to the verified proof with
the claim.
> - The next claim can check the same receipt without weakening
admission checks.

## Linked Issues or Issue Description

**What happened?**
A restart recovery attempt timed out before its admission callback. The
next attempt refused the saved pack with `source_state_pending`. It then
failed with `native_provider_artifacts_admission_unverified`.

**Expected behavior**
The next attempt can reuse the previous verified proof for the same
receipt, session, Runner, and lease. Current admission must remain
pending until its normal callback succeeds.

**Steps to reproduce**
1. Save verified provider-pack admission for a run.
2. Claim that run again. Commit the pending admission state.
3. Interrupt recovery before the session admission callback.
4. Claim the same run again and reconnect to the same Runner.
5. Observe that the old code refuses the saved receipt.

Related work: #20682 and #20850 address lock waits. #21085 adds refusal
logs. #21161 addresses command waits after restart. This change
addresses lost admission proof across interrupted claims. The original
transport timeout remains a separate investigation.

## What Changed

- The admission module prepares the claim patch and its recovery source.
- The claim saves a versioned reference with the source run, receipt
digest, and previous verified state. It adds no copy of provider files
or credentials.
- A later pending claim can use that reference only if the receipt is
unchanged. Normal company, agent, session, Runner, and lease checks
still apply.
- Missing proof, unknown state, a changed receipt, or a different Runner
stays refused. The callback clears the reference when it records its
decision.
- The executor uses this helper under its existing claim lock. Existing
artifact-retention and ownership checks stay in place.
- The execution guide states the previous-release behavior. Old verified
records seed the reference. Old pending records without a reference
remain refused. Older readers ignore the field and keep their existing
refusal behavior. No migration is needed.

## Verification

- The new interrupted-claim regression failed before the fix: admission
returned false instead of true.
- Focused admission, PostgreSQL fencing, and executor lease tests passed
after the rebase: 61 tests on commit c8123cc35d.
- The database case saves two interrupted claims, reads their persisted
state, checks a different Runner is refused, and publishes verified
admission for the original Runner.
- Negative cases cover missing proof, unknown state, a changed receipt,
a different source run, a different Runner, and a different lease.
Existing company and claim fences also pass.
- `pnpm verify` returned exit 2 because the execution guide requires
broader CI. Required CI must cover the final commit.
- Local server typecheck was killed with SIGKILL, including its
single-thread retry. This is an environment gap. Required CI must
provide typecheck and build results.
- Repository preflight installed dependencies and built shared packages.
It could not finish Runner preparation because cargo is absent.
- A live OpenCode restart workflow and recovery of the original task are
not verified. No task replay or deployment was performed.

## Risks

- The added profile field carries historical evidence. It does not mark
the current claim verified. Its use is limited to an unchanged receipt
with the normal identity checks.
- Existing pending or unknown incidents without this evidence cannot
gain proof from this fix. They keep their normal recovery path.
- No provider attempt limit, approval gate, workspace hold, or company
boundary changes.

## Model Used

OpenAI GPT-6, through Codex, with code search, tool use, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:57 -06:00
DottaandPaperclip b0447f3c76 fix: restrict chat task sponsors to self or admins (#21218)
## Thinking Path

> - Paperclip manages AI agents and their authority to do work.
> - External chat channels can create tasks for those agents.
> - A task sponsor is the person responsible for that work.
> - GitHub bot settings accepted another member as a sponsor from any
connection manager.
> - Connection management must not grant authority to act as another
person.
> - This change checks each sponsor on the server and limits the picker
choices.
> - The benefit is that ordinary members cannot assign work to another
person's identity.

## Linked Issues or Issue Description

**What happened?**

A connection manager could save another member as a GitHub guest sponsor
or automatic-task responsible member. The server checked membership but
did not check authority to select that person.

**Expected behavior**

An ordinary manager can select only themselves. A company owner, company
admin, or instance admin can select another active non-viewer member of
the same company. The picker shows member avatars.

**Steps to reproduce**

1. Sign in as an ordinary member with connection-management permission.
2. Open a GitHub bot's Access settings.
3. Set another member as a guest sponsor or responsible member.
4. Save. The old server accepts the request. This change returns 403.

**Paperclip version or commit**

Base: `9a9cd20d91edc93dba4e2295e42f0f1f9fcd4862`.

**Deployment mode**

Built from source. The checks use local PostgreSQL and an assembled
authenticated API server.

Related work: #17737 adds delegated connection configuration. #16163
separates author selection from guest access. This fix preserves both
contracts. A search found no duplicate sponsor-permission fix.

## What Changed

- Check every submitted sponsor in the configuration transaction. Lock
the actor's membership and any required admin role through the save.
- Require self-selection unless the actor has a current admin role.
Check all target members in the same company.
- Keep delegated connection checks, authority notifications, tool
updates, and audit attribution.
- Use avatar pickers for guest sponsorship and automatic-task
responsibility. Default new guests to the signed-in member.
- Test service and HTTP denials, admin selection, role removal, invalid
targets, saved settings, and picker choices.
- Test forged sponsor changes with real agent API keys and run
credentials. Update the existing access-map policy.

## Verification

- 67 sponsor, HTTP, picker, and settings tests pass. The picker test
includes 600 members and bounds member lookups.
- 93 existing GitHub management, event, policy, and tool tests pass.
- The assembled connection-management fixture passes. Self-save
succeeds. Both credential types receive 403 for forged guest and
responsible-member sponsors. The saved revision stays unchanged.
- UI typecheck and production build pass. Token gates, map checks,
query-error policy, migration checks, and whitespace checks pass.
- Two broad chat-test shards pass: 720 cases. The local full-suite run
was then stopped; the complete local suite has no final verdict.
Current-head CI remains the full verification path.
- Full local typecheck and build reach Rust but cannot finish because
Cargo is absent.
- CI found the explicit retry override in the access query. Removed it;
all 45 affected UI tests pass and the CI policy gate passed on the next
commit.
- Historical head `4146c89f9345efd38a69739d17e3194a7cf91980` received
Greptile 5/5 and passed full CI:
https://github.com/paperclipai/paperclip/actions/runs/43330028028 .
Those results apply to that earlier head.
- Current head `890df0322f382bb8ccc0fd3d59b25ba1e0299c58` merges master
without conflicts. All eight changed files match the earlier reviewed
fix. The 67 focused sponsor, HTTP, picker, and settings tests pass
again. The production-bearer fixture also passes for both agent-key and
run-token requests. UI typecheck, token gates, map checks, query-error
policy, and whitespace checks pass.
- Greptile gives current head `890df0322f382bb8ccc0fd3d59b25ba1e0299c58`
5/5 with no open review threads. All checks for this head are complete
with no failures. The PR workflow passed:
https://github.com/paperclipai/paperclip/actions/runs/43363400247 . This
includes build, typecheck, general tests, serialized server suites,
Runner checks, policy checks, and all eight E2E shards. The Snyk status
also passed.
- `pnpm verify --explain` returns exit 2 because the access-map change
selects broad CI. The focused local checks pass. Broad checks are
assigned to the CI run above.
- The first broad checks inspected upstream migrations before the merge
commit existed and stopped at migration classification. After the merge
commit, the migration checks pass with zero new migrations.
- Earlier saved-session broad checks stopped because Cargo was absent or
memory was exhausted. These gaps are retained here. No live
external-provider delivery or staging deployment is claimed.

The configuration service owns sponsor authorization because both board
and delegated API requests use it. No provider protocol or execution
contract changes.

## Risks

- An ordinary manager cannot save a configuration that names another
sponsor, even when that sponsor is unchanged. They must select
themselves or have an admin save it.
- Membership and admin rows stay locked through the configuration write.
A concurrent role change can delay the save.
- Existing configurations are not rewritten. Each new save must pass the
current checks.
- Generic channel and AgentMail creation derive the sponsor from the
authenticated user. Their strict input schemas do not accept sponsor
overrides.

## Model Used

OpenAI Codex, based on GPT-6. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository inspection, tool calls, code execution, and tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this fix does not
duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue with the bug-template field labels
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My published branch name describes the change and contains no
internal task details
- [x] I have run the focused tests locally and they pass
- [x] I have added relevant tests
- [x] I have updated the relevant access policy documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green for the current head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:56:13 -06:00
DottaandPaperclip 08be8be414 fix(recovery): report exact retained sandbox resumes as resumed (#21259)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A controller restart must keep the same remote execution owner.
> - A retained sandbox lease stores the result of its first acquisition.
> - That result can say replacement after the same allocation resumes.
> - The Runner then rejects a valid resume as a changed sandbox.
> - This pull request reports the verified operation as resumed.
> - The saved acquisition history and identity checks stay intact.

## Linked Issues or Issue Description

Refs #18934. That change makes the retained sandbox ready before
preparation. This change corrects the acquisition receipt after that
resume.

**What happened?**
A remote native run resumed the exact retained sandbox after a
controller restart. The lease still carried a historical replacement
receipt. Runner verification rejected it with
`native_remote_recovery_lease_mismatch`, stage `runner_verification`,
field `replacement_sandbox`.

**What did you expect to happen?**
Report the verified exact-allocation operation as resumed. Preserve the
original saved history. Reject a changed, absent or expired allocation.

**Steps to reproduce**
Save an active lease with historical replacement metadata. Resume that
exact provider allocation through `acquireForRun`. Resolve its execution
target. Before the fix, the target still reports replacement and Runner
verification refuses it.

**Deployment mode**
Native Runner on a remote sandbox. The reported provider was Daytona.

## What Changed

- Return a separate operation receipt after the exact retained
allocation passes the existing resume checks.
- Pass the receipt through heartbeat preparation and apply it only to
the execution target.
- Keep the saved lease metadata and workspace unchanged.
- Test historical created, replacement and resumed receipts. Test
ordinary and remote-authority realization through the real execution
target resolver. Check every saved metadata write.
- Document the difference between saved acquisition history and the
current resume operation.

## Verification

- Head: `74d03e8ddadd1274891bea99a10455795c500557`.
- Red: three historical receipt cases fail without the resume receipt.
- Review regression red: realization writes resumed over the historical
replacement receipt. The saved metadata loses the previous allocation
and cause.
- Green: all 69 orchestrator tests pass. Ordinary and remote-authority
realization preserve history in every metadata write and return resumed
in the target.
- Green: 135 focused orchestrator and remote controller adoption cases
pass. Identity and changed-allocation refusals remain effective.
- Server typecheck passes, including its TypeScript build dependencies
and test layers.
- `git diff --check` passes.
- `pnpm verify` exits 2 because the execution-semantics document needs
broad CI. Required CI supplies that coverage.
- Linux x64, Node 25.9.0. Provider boundaries are simulated.
- The first post-merge attempt could not collect tests because the
compiled plugin SDK lacked the new taskLease schema. Shared and plugin
SDK builds passed, then the unchanged selection passed.
- Full preflight stopped because Cargo is absent. Dependency
installation completed. The later server TypeScript build and typecheck
passed.
- Live retained Runner reattachment is not verified. This workspace has
no provider credentials. No rollout or original-task recovery by this
patch is claimed.
- Required current-head CI and automated review remain pending.

## Risks

A wrong receipt could hide an actual replacement. The projection occurs
only after the existing company, task, run, agent, workspace, provider
allocation, expiry and directory checks pass. The provider must resume
the exact allocation. Runner process and identity verification still
runs. No replacement is acquired and no saved acquisition history is
changed. The heartbeat change only carries the typed receipt between the
owning orchestrator stages. Callers without a receipt keep their
existing behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and tool use. The exact backend snapshot and context window
are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:55 -06:00
DottaandPaperclip 2b1d0fc825 fix(connectors): keep result classification alive during run progress (#20763)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - App connectors screen returned content before they give it to an
agent.
> - The model screen checks run identity and reserves its budget in one
transaction.
> - That transaction took a repeatable-read snapshot before it locked
the run.
> - A run event could change the row first and cause all four attempts
to fail.
> - This pull request uses READ COMMITTED for this admission under the
existing locks.
> - The benefit is that connector results can pass screening while run
progress continues.

## Linked Issues or Issue Description

Refs #20385 and #20484. These changes fixed other run accounting paths.
They did not change classifier admission.

**What happened?**

A connector read can return the accounting contention error before the
model screen starts. A real gateway test reproduces SQLSTATE `40001` at
`prompt-injection-prevention.ts` when it reads `heartbeat_runs FOR
SHARE`. Each attempt commits a run event after the transaction starts
its snapshot.

**Expected behavior**

Return screened connector content and save its classifier charge while
run events continue. Refuse paid classification if the run stops, its
identity changes, or access or budget checks fail.

**Steps to reproduce**

Run this test with the production source from `9574065a28`:

`cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts -t 'returns a screened
connector result'`

The test fails after four `40001` errors. It passes with this change.

## What Changed

- Use READ COMMITTED only for prompt-injection admission. Keep company,
project, configuration, agent, run and secret share locks.
- Keep classifier settlement at repeatable read. Keep provider calls
outside transaction retries.
- Test screened results, stored charges and settled holds under run
progress. Test four concurrent reads and refusal after a run stops or
changes identity.
- Document the transaction boundary.

### Hypothesis and thesis for a fix

**Hypothesis:** the classifier admission snapshot becomes stale before
its run share lock. The gateway regression confirms this cause in
current source. The historical connector error alone does not identify
its SQL statement.

**Thesis for a fix:** read current run state under the existing locks at
READ COMMITTED. This removes stale snapshots caused by run events. The
change belongs to classifier admission because every screened connector
result uses this service.

Investigation source: [Braintrust
pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/d5369c32-4c0f-43a1-af94-a01fef0a697b).
Access is required to read the traces.

**Proof:** baseline `9574065a28` failed with `40001` after four
attempts. The same single-read regression passed after the change. The
full focused suites then passed, including four concurrent reads, stored
charges and refusal controls.

## Verification

- Real PostgreSQL gateway regression: baseline red in 28.07 seconds;
candidate green in 23.16 seconds.
- `cd server && pnpm exec vitest run
src/__tests__/prompt-injection-gateway.test.ts
src/__tests__/prompt-injection-prevention.test.ts`: 54 tests passed.
After sync with master `1ac8b6a711`, all 54 passed again in 34.46
seconds.
- The tests use the real gateway, content guards, classifier service and
database. The MCP server and paid model are simulated. This is not a
live deployment test.
- Existing controls cover company boundaries, budgets, credentials,
configuration changes, interruption and receipt recovery.
- Staging health reported `16024ffa2`. Its classifier and accounting
source matches the reproduced baseline and includes #20385 and #20484.
The revision at the earlier failure times remains unknown.
- `pnpm verify` selected 2,512 of 3,303 test files because its
dependency policy retained 2,510 files conservatively. The broad local
run was stopped after the selection became clear. It is not a pass. The
focused 54-test result above is the local proof; required CI supplies
broad coverage. An earlier selection was also invalidated by a
concurrent commit.
- Full preflight could not finish because this host has no Cargo. Pinned
dependencies and shared/plugin SDK builds completed. The focused tests
passed without Rust.
- [Greptile reviewed this head at
5/5](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790238737).
No inline thread is open. [Paperclip completed its assessment at 5/5
with no
findings](https://github.com/paperclipai/paperclip/pull/20763#issuecomment-6790312753).
[Required CI passed for head
`643bbe51faa70dcb17f372b9e85e3cbbcb724b9d`](https://github.com/paperclipai/paperclip/actions/runs/42981473514),
including typecheck, build, 12 server test shards, nine serialized
suites, eight E2E shards and Runner checks. Snyk and Socket passed. The
full inventory contains 58 completed check runs; the neutral Paperclip
check is a completed 5/5 assessment, not an incomplete review.

## Risks

- READ COMMITTED uses a new snapshot for each statement. Existing
admission locks still protect the run status, identity, configuration
and secret checks. Budget checks and reservation writes still commit
together.
- Concurrent independent admissions retain existing budget behavior.
This change adds no retry of paid provider work.
- No schema, API or saved receipt format changes.
- Historical failing SQL is unavailable. The source regression proves
this failure path; it does not prove that every historical example has
the same cause.
- The change is unmerged and not deployed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment model ID and context-window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:55:37 -06:00
DottaandClaude Opus 5.5 cfa959add7 refactor(slack): correct stale status comment and add production acceptance checklist (#20840)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Slack chat connector lets people talk to their agents in Slack
> - The connector is close to production, but its live evidence is out
of date
> - The September live results use old revisions, and the October setup
run did not test follow-up, reauthorization, or removal
> - Cloud managed setup has never run against real Slack, and its docs
say not to enable it until it has
> - This pull request adds one checklist for a fresh live pass on
current master, and corrects one stale code comment
> - The benefit is a clear finish line for Slack production readiness on
self-hosted and Cloud

## Linked Issues or Issue Description

**Issue type**
Missing documentation.

**Where is the issue?**
`doc/connections/SLACK-AUTOMATIC-SETUP.md`,
`doc/plans/chat-adapters/2026-09-05-slack-live-qualification-result.md`,
`doc/testing/slack-cloud-qualification.md`, and
`server/src/services/chat-channels.ts`.

**What's wrong?**
The Slack live qualification is spread over three documents. Their
evidence uses older revisions. No document lists what a production pass
must cover for Cloud managed setup. A code comment says the manifest
does not request `assistant:write`, but the manifest does request it.

**Suggested fix**
Add one dated acceptance checklist with pass conditions per row. Correct
the comment.

## What Changed

- Added `doc/plans/2026-11-11-slack-production-acceptance.md`. It has
sections for self-hosted automatic setup, manual and existing apps,
conversation behavior, Cloud managed setup on staging, and distribution
questions (Marketplace-only workspaces, non-Marketplace rate limits,
scope justification, zero-copy).
- Corrected the comment in `server/src/services/chat-channels.ts`. It
now says that Slack shows the working state through
`agents.sessions.setStatus` and the lifecycle reply. No code behavior
changed.

## Verification

- Documentation and a comment only. No runtime behavior changed.
- I checked the doc links and the referenced files against current
master.

## Risks

- Low risk. The change is documentation and one comment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
web search.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:54:50 -06:00
DottaandClaude Opus 5.5 c6b497d831 perf(auth): authenticate a run's agent JWT in three statements (#21434)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each running agent calls the Paperclip API with a signed run token
(an agent JWT)
> - The server authenticates each of these requests before the route
handler starts
> - This authentication sent 13 statements to the database for each
request, and none of them were cached
> - One server process starts to struggle at about 50 concurrent agents,
and the target is 1,000
> - This pull request cuts the work before the route handler to 3
statements, with no transaction and no row locks in the usual case
> - The benefit is less database work and less event-loop work for each
agent request, with the same security checks on each request

## Linked Issues or Issue Description

Refs #16577 (trimmed saved run JSON from agent authentication)
Refs #17931 (scoped run identity reads to the needs of each operation)

**What existing behavior does this improve?**
The authentication of an API request that an agent makes with its run
JWT. Before the route handler starts, `actorMiddleware` did 13 database
statements in sequence: a board key lookup, an agent key lookup, a
`select *` of the agent, the run read, a 7-statement locked identity
capture (BEGIN, run task read, task row lock, run row lock and wide row
read, pending steering read, context read, COMMIT), and two reads for
the responsible user.

**Subsystem affected**
Server: `server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts` and
`server/src/services/run-identity.ts`.

**Current behavior**
Each agent request costs 13 statements. The identity capture is a
transaction that holds a pooled connection and takes `FOR NO KEY UPDATE`
locks on the task row and the run row. These are the same rows that
event appends and steering lock, so agent requests wait behind them.

**Proposed behavior**
Each agent request costs 3 statements and no transaction: the agent
status columns, one run statement, and one membership statement. A run
with a pending steering context still uses the locked capture, so
steering acceptance does not change.

**Reason and benefit**
Item 1c of the thousand-concurrent-agents plan. The load model at 50
agents (API requests only) shows the database statements for this path
go from 165.7/s to 30.1/s, transactions from 15.1/s to 0, and event-loop
utilization from 0.74 to 0.25.

**Breaking changes**
None. The actor shape that routes get does not change. No route handler
changes. No schema change.

## What Changed

- **Skip key lookups for a verified run JWT.** The middleware verifies
the JWT first (HMAC only, no database). Every stored board key and agent
key is `pcp_` followed by letters, `_` and hex (`createBoardApiToken`,
`createToken`). A verified JWT has three parts with dots. So its SHA-256
cannot equal a stored key hash, and both lookups always returned no row
for it. A token that does not verify still tries the board key, then the
agent key, in the same order as before.
- **Read only the agent columns that authentication checks.**
`companyId`, `status` and `lifecycleHolds` replace `select *`. The agent
configuration JSON stays in the database. The agent-key path uses the
same columns.
- **Read revocation and identity in one statement.**
`loadAgentRunAuthentication` now also returns the run task ids, a
`pendingIdentityContext` flag (`exists` on a pending context of this
run), and the active context `id` and `responsibleUserId` (only when
that context belongs to this run). The new `resolveRequestRunIdentity`
in `run-identity.ts` uses these columns:
- The run is not running, or it has no active context: it keeps the run
row values, as before.
- A pending context exists: it calls `captureRunIdentity(...,
"request_authentication")`, as before. This keeps the locked
accept-or-refuse step and the 409 for an unacknowledged steer.
- Otherwise: it checks the run task ids with the same rule as
`lockIdentityTask` (now `assertRunTaskIdentity`), and returns the
context from the statement.
- **Reuse the run row for the legacy responsible user.** The legacy
lookup had the same filter as the run read. The middleware now uses the
`responsibleUserId` from that read, with the same non-UUID guard.
- **One statement for memberships.** The user existence check is an
`exists` condition in the membership statement, in place of a second
parallel statement.
- No in-process memo and no time-based cache. Every request reads the
current run row.

### Why the request-authentication read does not need the locks or the
transaction

The capture locks are released at COMMIT, before the route handler
starts. So they do not protect anything that the route does. Inside
authentication, they have two effects:

1. **They protect the acceptance write when a pending context exists.**
This PR keeps this case on the locked capture.
2. **They make the read wait for a steering transaction that is in
progress.** In PostgreSQL READ COMMITTED, one statement reads one
snapshot. So the run status, the cancellation fields, the active
context, its responsible user and the pending flag are all from the same
committed state. Each interleaving with a steering writer gives a result
that a serial order also gives:

| Writer state when the statement runs | What the statement sees |
Result | Same as a serial order? |
| --- | --- | --- | --- |
| Reservation (`reserveSteeredIdentity`) not committed | Old active
context, no pending | Old identity, 200 | Yes: request before the
reservation. The old identity stays active until the steer is accepted.
|
| Reservation committed | Pending flag | Locked capture: 409, or accept
if acknowledged | Same code path as before |
| Acceptance in progress (pending committed) | Pending flag | Locked
capture waits for the acceptance, then returns the new identity | Same
code path as before |
| Acceptance committed | New active context, no pending | New identity |
Yes: request after the acceptance |
| Cancel or supersede committed | `cancelled`,
`executionCancellation.state = requested`, or
`freshContinuationSupersededAt` | 403 on a write | Yes, on the very next
request |

`activate()` writes `activeIdentityContextId` and `responsibleUserId` in
one update, and the statement joins the context by id and run id. So a
mix of an old context with a new user is not possible. The tests show
the first and third rows on real PostgreSQL.

### Notes for reviewers (moved from code comments)

- The two correlated subqueries in `requestRunIdentityFields` name
`"heartbeat_runs"` and an alias for `run_identity_contexts` in raw SQL.
Drizzle writes the columns of a one-table select without the table name,
so a `${heartbeatRuns.id}` inside the subquery would resolve to the
inner table. The first test run showed this (the identity came back
empty), and the statement-count test catches it.
- The `exists` condition in `loadResponsibleUserMemberships` keeps the
old rule: a membership counts only while its user exists.
- The legacy responsible-user value comes from the run read, because
that read has the same filter (`id`, `company_id`, `agent_id`) as the
removed legacy lookup.

## Verification

- New `server/src/__tests__/agent-jwt-request-authentication.test.ts`
(embedded PostgreSQL, real `actorMiddleware`,
`meterDatabaseStatements`):
- Statement count: 3 statements, 0 transactions, 3 waits for a running
agent with an active identity and a responsible user. Before: 13
statements. The test also checks that fewer than 8 KiB come back, with a
120 KB agent configuration.
- Revocation: a write that passes is refused on the very next request
after `status = cancelled`, after a Stop request
(`executionCancellation.state = requested`), and after a
fresh-continuation supersede (`agent_run_superseded`).
- Identity change between two requests: the next request uses the new
context and the new responsible user.
- Uncommitted steering reservation: the request does not wait, and keeps
the committed identity. After commit, the next request gets 409. After
the acknowledgement, the next request gets the new identity.
- Acceptance in progress: the request waits for it (locked capture),
then returns the new identity.
- Wrong claims: an agent token signed for another company gets 401. A
token that names another agent's run gets no identity from that run. An
active context id of another run is not used.
  - Pending steering without an acknowledgement still gets 409.
- Red/green: on `origin/master` code, the statement-count test failed
with 13 statements, and the uncommitted-reservation test failed because
the request was blocked on the row lock. Both pass with this change.
- `agent-auth-identity-transfer.test.ts`: the heartbeat_runs read count
is now exactly 1 (it was a lower bound of 3).
- These pass: `agent-auth-middleware.test.ts`,
`agent-auth-identity-transfer.test.ts`,
`agent-run-authentication.test.ts`, `run-identity.test.ts`.
- `pnpm verify`: it selected 2,639 of 3,459 Vitest files, because
`auth.ts` is imported by most route tests. That is close to the full
suite, so I stopped it at 30 minutes on this busy machine (exit 124, no
failures before the stop). CI supplies the broad coverage. The focused
checks above pass on the final revision (41 tests).
- `pnpm check:module-boundaries`: passed. Server `tsc --noEmit`: passed.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from the plan
branch, not committed), `AGENTS=50 SECONDS=20`, one run at a time. The
machine had a load average of 25 to 33 from other work, so the timings
are noisy. The statement and transaction counts are not affected by the
noise.

| `SOURCES=api` | Before | After |
| --- | --- | --- |
| Statements/s (15 requests/s) | 165.7 | 30.1 |
| Statements per request (this model has no responsible user) | 11 | 2 |
| Transactions/s | 15.1 | 0 |
| Event-loop utilization | 0.741 | 0.252 |
| Event-loop delay p99 | 267.8 ms | 19.0 ms |
| API lateness p50 / p99 | 1104 / 5631 ms | 6.7 / 79 ms |

| All sources | Before | After |
| --- | --- | --- |
| API requests done | 300/300 | 300/300 |
| API lateness p50 / p99 | 291 / 1130 ms | 230 / 2681 ms |
| Delta appends done | 14977/20000 | 8260/20000 |

The all-sources run is limited by delta appends, which this PR does not
change. Its results move a lot between runs on this busy machine (the
delta count is lower after, with no change to that path), so it does not
show an effect of this PR in either direction.

## Risks

- The usual path no longer waits for a steering reservation that is in
progress. The request then uses the identity that is committed and
active. This is the identity that the run acts under until the steer is
accepted. The table above shows that each interleaving matches a serial
order.
- `loadAgentRunAuthentication` returns more columns (two small
correlated subqueries on `run_identity_contexts`, by primary key and by
the `(run_id, ...)` index). Other callers get extra fields and ignore
them.
- The JWT-first order depends on the stored key format. If a new key
format can contain three dot-separated base64url parts that verify as a
JWT, this order must be checked again.
- Low risk for routes: the actor object is the same, and no route
handler changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:38:01 -06:00
DottaandClaude Opus 5.5 16f00d62fb fix(connections): answer setup questions with a setup link in ranked results too (#21428)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21400 told the agent to answer a setup question ("What is the
easiest way to connect Notion here?") with a setup-page link, because
`connection_request` ends the turn before any answer text is shown
> - That rule covered only a single search match
> - In a paid eval run the agent searched "Notion integration connect
workspace", got ranked matches, raised the card and ended with no answer
> - This pull request adds the same rule to the ranked-matches
instruction
> - The benefit is that a setup question gets a written answer with a
working setup link, whichever search the agent runs

## Linked Issues or Issue Description

Refs #21400. This commit was pushed to #21400 after it merged, so it is
a follow-up.

**What happened?**
Paid run 43455387724, case `connection-setup.choose-provider`: the agent
called `connections_search` with a multi-word query, got the
ranked-matches instruction ("Use connection_request when human setup or
access is required"), called `connection_request`, and the turn ended as
"Waiting for Connect Notion" with no text reply.

**Expected behavior**
For a setup question, the agent answers in plain words and links
`/<prefix>/apps/connect?source=<app>`. For a data request, it raises the
setup card.

**Steps to reproduce**
1. Have no Notion connection.
2. Ask "What is the easiest way to connect Notion here?" and let the
agent search with a multi-word query.

## What Changed

- `server/src/services/connection-intents.ts`: the ranked-matches
instruction says to use `connection_request` for data or actions the
person asked for, and to answer a setup question with the app's
setup-page link instead. It also says not to create a connection or
draft unless the person asked for setup.
- Test: the ranked instruction includes the setup-page rule.

## Verification

- `cd server && npx vitest run
src/__tests__/connection-intents-service.test.ts -t "own-server"`.
- Paid run 43460401711 on a branch with this change:
`connection-setup.choose-provider` passed (answer with `[Connect
Notion](/COM/apps/connect?source=notion)`),
`connection-setup.notion-declined` passed.

## Risks

- About 60 more words in one tool result. No effect on agent boot or
time to first token.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:36:59 -06:00
DottaandClaude Opus 5.5 89e6575418 test(native-runtime): cover released session tool authority and add a catalog memory benchmark (#21440)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs get their tools from a session tool authority,
which holds the run's assigned app (MCP) tool catalog
> - #21415 releases a finished session's authority, so the server heap
no longer grows with each finished session
> - Two behaviors of that fix had no test: a released session fails
closed, and the release when a follow-up run retires its warm
predecessor and then fails before it admits its own authority
> - The memory saving was measured once, by hand, with no script in the
repository
> - This pull request adds the two tests and a benchmark script. It
changes no product code
> - The benefit is that a later change cannot silently bring back the
leak or let a released session reach its tools

## Linked Issues or Issue Description

Refs #21415 (the fix these tests cover). Refs #21430 (closed duplicate;
a reviewer comment there named the follow-up failure path).

**What is missing?**

#21415 tests that finished sessions return the authority map to its
starting size. No test checks that a released session refuses late tool
calls. No test checks the follow-up path, where a follow-up run retires
the previous warm session (so the release at forget time is skipped
while the follow-up owns the scope) and then fails before it admits its
own authority.

**Expected behavior**

A late tool call or completion feedback for a released session fails
with `native_session_tool_authority_unavailable`. The next run of the
same scope gets a new authority with only its own assignments. When a
follow-up fails after it retires its predecessor, the authority map is
back at its starting size after that run settles.

**Paperclip version or commit**

master `edb6945ff3` (after #21415, `85f4a91e70`).

## What Changed

- Test "refuses a released session's late tool call and completion
feedback and admits the next run with only its own assignments".
- Test "releases a retired warm predecessor's tool authority when the
follow-up fails before admitting its own". A follow-up of the same warm
scope with changed configuration retires the predecessor. Then its
assigned tool listing fails. The test asserts that the predecessor was
closed, that the follow-up failed with that error, and that
`sessionToolAuthorityCount()` is back at its starting value.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: retained
bytes of one assigned tool catalog per session, and schema copies per
`definitions()` call.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "released
native session tool authority|retired warm predecessor"`: 2 passed.
- Red check: with the release at settle removed from
`native-session-executor.ts`, both tests fail (`expected 1 to be +0`,
and the late tool call is not refused).
- Retained bytes per finished per-turn session were measured through
`executePaperclipNativeSession` with the real `createAssignedMcpTools`
(80 tools, 2 KB schemas, 100 sessions, forced GC, `heapUsed` delta). On
`85f4a91e70^` (before #21415): 604,566 bytes. On `85f4a91e70`: 19,794
bytes (test-harness noise), and the authority map stayed at its starting
size.
- `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools 80 --schema-bytes
2048 --sessions 200 --turns 20` reports about 546 to 567 KB retained per
assigned tool catalog. Each `definitions()` call makes 80 new schema
copies (about 296 KB), and none are kept.
- Full `native-session-executor.test.ts` locally, with a load average
near 78 from other work on the machine: 1,483 passed and 5 failed. All 5
failures were timeouts in tests this change does not touch (the slowest
took 163 s). CI gives the clean full-suite result.

## Risks

- Low risk. Tests and one benchmark script only. No product code
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:17:25 -06:00
DottaandClaude Opus 5.5 df1b12ad8e fix(runner): advertise flat tool schemas and stop repeated runner-side input rejections (#21398)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools. The runner shows each tool to
the model with an input schema and checks every call against that schema
before it runs
> - Some tool schemas had a root `oneOf`, `anyOf`, `allOf` or
`if`/`then`. `handoff_case` had no top-level `properties` at all
> - Some model hosts convert such a schema into an empty argument list.
The model then calls the tool with no arguments, and the runner rejects
the call as "Invalid tool input"
> - The repeated-failure guard from #21360 does not see these
rejections, because they happen in the runner before the call reaches
the control plane. The model sent the same rejected call again and again
> - This pull request gives every agent-facing tool a plain object
schema with named properties, checks the old cross-field rules with
named errors, and adds a runner-side guard for identical "Invalid tool
input" rejections
> - The benefit is that models on these hosts can see and send the
arguments, and a dropped argument ends in a clear correction or a stop,
not a long loop

## Linked Issues or Issue Description

**What happened?**
In a chat eval cohort with OpenCode and
`openrouter/deepseek/deepseek-v4.1-flash-1017`, the hosts Wafer and
DigitalOcean served most model calls. In the case handoff case,
`handoff_case` arrived with no arguments on all 10 calls across two
cases. The runner rejected each one with "must have required property
idempotencyKey, case, toAgentId, ...". The model's reasoning said that
the tool showed an empty property list. The model fell back to
`call_api`, but the reply waiter had already expired, so the case
failed. In the preserve-draft case, `request_human_input` was rejected 9
times in a row with the same root `then` error.

**Expected behavior**
Every tool that an agent sees has a plain object schema with named
properties, so every host can show its arguments. When the same call is
rejected twice with the same arguments, the model gets a correction that
lists the expected properties. On the third identical call, the model is
told to change the arguments or stop.

**Paperclip version or commit**
`master` at f7ddab8e94.

**Steps to reproduce**
1. Run an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, where OpenRouter routes
to Wafer or DigitalOcean.
2. Ask the agent to hand a case to another agent with `handoff_case`.
3. See `handoff_case` arrive with no arguments and get rejected on every
call.

## What Changed

- Add `providerToolSchemaShapeIssues`
(`packages/paperclip-runner/src/contracts/provider-tool-schema-shape.ts`).
It reports a root that is not `type: "object"`, has no `properties`, or
uses `oneOf`, `anyOf`, `allOf`, `not`, `if`, `then`, `else`,
`dependent*` or `$ref`.
- Add inventory tests that run this check on every agent-facing tool:
the live semantic catalog, the semantic action catalog, the Codex tool
specs, `paperclip_finish` and `paperclip_block` (runner), and the
server-only tool definitions (server). Before this change, the runner
test listed 16 entries for 8 tools: `handoff_case`,
`request_human_input`, `read_case`, `update_case`, `create_case`,
`read_workflow`, `update_workflow` and `paperclip_finish`.
- Flatten these 8 schemas. `handoff_case` is now one object with `to`
(optional, `"done"`), and `case` is one object with optional `id` and
`create`. Descriptions on the properties state which fields each variant
needs.
- Add `toolInputVariantError`
(`packages/paperclip-runner/src/contracts/tool-input-variants.ts`). It
checks the former cross-field rules and returns an error that names what
is wrong and how to send the call correctly. For example: `Invalid tool
input for handoff_case: An agent handoff requires toAgentId, phase. To
finish an existing case instead, send to: "done" with
expectedHandoffVersion and expectedIssueStatusVersion.`
- The server runs this check first in
`PaperclipRunnerToolAuthority.execute` and returns `422`. The #21360
guard therefore counts these errors too. Assigned MCP tools and
connector tools are not checked. The runner semantic dispatchers run the
same check after schema validation. The runner MCP bridges run it for
`paperclip_finish`, which never reaches the control plane.
- Add `RepeatedToolInputRejectionGuard`
(`packages/paperclip-runner/src/drivers/repeated-tool-input-rejection.ts`)
and use it in the OpenCode MCP bridge and the runner tool bridge, where
runner-side validation happens. The key is the tool, the canonical
arguments and the error text, separated per MCP session. The second
identical rejection adds the arguments that arrived and the expected
required and optional properties. The third and later identical calls
say that the call was not accepted and that the model must change the
arguments or stop and report the blocker. State is in memory, per
bridge, and capped at 64 entries.
- Regenerate `semantic-tool-contracts.json`,
`semantic-action-catalog.json`, `protocol/manifest.json` and the seeded
eval fixture.
- Add `spec/fixtures/pre-flat-tool-input-schemas.json`: the 8 strict
schemas from the previous release, for the upgrade test.

## Verification

- `cd packages/paperclip-runner && npx vitest run src/contracts
src/catalog src/protocol-actions src/semantic-tools
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/runner-tool-bridge.test.ts`: all pass.
- `cd server && npx vitest run
src/services/native-runtime/tool-input-variant-admission.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts`: 9
passed.
- Red/green:
- The runner inventory test fails on `master` and lists the 16 entries.
It passes after the change.
- The guard test replays the recorded empty `handoff_case` call 4 times
through the real OpenCode bridge. Without the guard, the second call has
no correction and the test fails. With the guard, the second call lists
`required idempotencyKey, case, instructions; optional to, toAgentId,
phase, ...`, and the third says the call was not accepted.
- Negative controls: changed arguments get a fresh error, and a valid
call still runs once.
- Upgrade safety: `tool-input-variants.test.ts` compiles the previous
release's strict schemas from the fixture. For a corpus of valid and
invalid inputs per tool (27 for `handoff_case`), it checks that "flat
schema accepts and `toolInputVariantError` is null" gives exactly the
same result as the old schema. Each tool has accepted and rejected
inputs.
- Generated outputs: `check:semantic-contracts`,
`check:capability-contract`, `check:protocol-coverage`,
`check:capability-inventory`, `check:protocol-manifest` and the replay
golden check pass.
- Rust tests that read the generated contract: see the CI `runner`
checks.
- Paid narrow run: see the comment below.

## Risks

- Previous-release story: tool schemas are not saved run identity. The
OpenCode bridge lists tools again when a session resumes. A durable Rust
run that is already attached keeps the tool catalog it saved, because
re-attaching the same run does not replace the catalog. Every input the
old strict schemas accepted is still accepted, and every input they
rejected is still rejected, so saved calls, receipts and replays carry
forward. No session is reset, and no profile version or provider
`commandDigest` changes. The pack `contractDigest` changes because
`generated/` changed, as it does for any tool contract change.
- A rejected variant now arrives later for the semantic tools: the
server returns `422` instead of the runner bridge rejecting it. The
error text names the problem.
- For `paperclip_finish` on Codex, the advertised schema no longer has
the root conditional. The Rust bridge accepts a `done` result with a
continuation object, and canonical result validation still rejects it,
as before.
- Latency: the advertised schemas of the 8 tools are 1,876 bytes smaller
(23,785 to 21,909), about 470 fewer prompt tokens when all are listed.
Compiling their validators in the bridge takes 164 ms instead of 214 ms
(median of 15). The guard adds one SHA-256 of the canonical arguments
for each rejected call only. Nothing is added before turn start.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:08:58 -06:00
Nicky LeachandClaude Fable 5.2 84a4d345b3 refactor(docker): take the application image's Runner from the runner-artifacts stage (#21332)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner is the part that executes agent work. `master` publishes
it alone as the image `ghcr.io/paperclipai/paperclip-runner`, built by
the `runner-artifacts` stages of the `Dockerfile`
> - The application image does not use those stages. It has five stages
of its own that compile the daemon, it builds the Runner again inside
the server build, and it builds a second provider pack
> - So one `Dockerfile` holds two recipes for the Runner, and the two
results differ: another base image and another daemon binary
> - This pull request removes the second recipe. The application image
copies the Runner out of the `runner-artifacts` stage, which the same
build runs once
> - The benefit is one Runner recipe, and an application image whose
Runner is the Runner the contract describes. A later pull request can
replace the stage with the published image

## Linked Issues or Issue Description

This is the second part of a chain. The first part is #21331, which
removes the cache check that cannot run on this `Dockerfile`. Refs
#21204, #21229, #21230 (the Runner image), #21306 (the no-rebuild mode
of the vendor step), #21307 (the pull-request check of the application
image) and #21312. No issue exists for this change.

**Problem or motivation**

The stages `rust-toolchain`, `rust-chef`, `runner-plan`, `runner-deps`
and `runner-build` compile the daemon on Debian trixie. The server build
then runs the full build of the Runner package. The
`runner-provider-pack` stage builds the provider pack. The
`runner-artifacts` stages do all of that again for the Runner image, on
Debian bookworm, with a stripped daemon.

**Proposed solution**

Read only the `runner-artifacts` stage from the application stages.
Build it inline in every build. Do not pass a published image yet.

**Alternatives considered**

- Pass the published Runner image to the build in this pull request.
That needs the resolver, labels and a rule for commits without an image.
It is the next step, and this one must be safe alone.
- Keep the old stages as a fallback. Two recipes is the problem.

## What Changed

- `Dockerfile`: remove the five stages. `build` starts from `base`. It
copies `/runner-package/dist` and the build stamp from
`runner-artifacts`, sets `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` and
runs the vendor check before the first application build.
- `Dockerfile`: `runner-provider-pack` starts from `base`. One `RUN`
with a bind mount of `runner-artifacts` copies the pack, the receipt and
the archives, and moves the large files to their own tree. `build` and
`runner-provider-pack` now come after the stages they read.
- `Dockerfile`: the three `COPY` instructions of `production` that take
the pack are byte-identical. A test pins all three.
- `Dockerfile`: `build` declares the build argument
`PAPERCLIP_RUNNER_EXPECT_INPUTS_DIGEST`, empty by default, and one step
runs the new `scripts/check-runner-image-expectation.mjs` on
`/runner-image.json` of `runner-artifacts`. **This pull request declares
the argument. No build sets it. The next pull request, which gives a
published Runner image to the build, is its first user.** With the
argument set, the build fails unless `runner-artifacts` is the published
image of exactly those inputs. `docker-app-check.yml` also runs when the
script changes.
- Comments: the `Dockerfile` has 220 comment lines, down from 247. The
comments of the two moved stages moved with them, unchanged, so the diff
shows them as added lines. Eight comments that named a removed stage or
a step that moved get the smallest correction. Two comments in
`prepare-runner-vendor.mjs` and `build-inputs.mjs` said that the
`Dockerfile` relies on the standalone rebuild; that clause is removed.
- Tests: replace the two cargo-chef tests of
`docker-build-stamp.test.ts` with three tests of the new shape. Update
the pack test, `docker-image-layer-reuse.test.ts`,
`vite-build-commit.test.ts`, the stage rules of
`runner-image-identity.test.ts`, and the include-layout test of
`runner-release-crate-cache.test.mjs`, which now reads
`runner-artifacts-runnerd`.
- Docs: `doc/DOCKER.md`, `doc/DEVELOPING.md`, and a new section "The
application image" in `doc/runner-image-contract.md`.

## Verification

All builds: `linux/arm64`, local, `--build-arg
PAPERCLIP_BUILD_COMMIT=<sha>` with one commit for every build. The
builds ran before the rebase onto `master`, on the head of #21310 as it
was before it merged. `master` changed neither the `Dockerfile` nor an
input of the Runner in between.

- `docker buildx build --target production`, cold, before and after: 506
s and 410 s. Compressed size: 2,971 MB and 2,900 MB.
- The image config and the text of all 33 history rows are identical.
The row `COPY /provider-pack-transport/ /opt/paperclip-runner/ #
buildkit` matches the pattern of `retained-provider-pack-oci.ts` exactly
once.
- A comparison of every file of the two images (path, type, size, mode,
link target, and SHA-256 under `/app` and `/opt`). The differences are
in "Risks". All other files are byte-identical, the root `node_modules`
included.
- From the image after: the four `/app/packages/paperclip-runner/...`
paths and the contents of `/opt/paperclip-runner/` exist.
`paperclip-runnerd --build-metadata` runs. `provider-pack.json` has
`runnerSourceRevision` equal to the build commit and no
`runnerInputsDigest`. The entry points `index.js`, `live/index.js` and
`testing.js` of the Runner import.
- The vendor step logged `reusing the current Runner build` in both of
its runs. Every `Compiling` line of the log is in
`runner-artifacts-runnerd`.
- `--target build`, `--target runner-provider-pack` and `--target cloud`
built. The `build` stage has no `gcc`, `cc`, `cargo`, `rustc` or
`pkg-config`.
- `--target runner-artifacts` and `runner-artifacts-slim`, before and
after: `runner-image-layout.mjs check` and `runner-image-oci.mjs
check-layers` pass on both, in `node:24.22.3-bookworm`. The daemon,
`provider-pack.json` and the two stable layers have the same digests
before and after.
- Without a build commit: `build` and `runner-provider-pack` built, and
the pack was skipped with the same message. This run was on the next
branch of this chain, after the rebase. The last steps of `production`
were not run for this case.
- The expectation argument, on this `Dockerfile`, `--target build
--output type=cacheonly`, after the rebase:
- No argument, inline Runner: the build passes (233 s). The step passes.
- Argument, inline Runner: the build fails at the step. The message says
that the `runner-artifacts` of the build are not a published Runner
image.
- Argument `d4df566f…` and the published image of those inputs as
`--build-context
runner-artifacts=docker-image://ghcr.io/paperclipai/paperclip-runner@sha256:0cd8250b…`:
the step passes, and the build runs no Runner stage. The vendor step
then fails with `inputs changed since the recorded build`, because this
tree differs from the tree of that image by one comment line in
`build-inputs.mjs`. With that line put back, the same build completes,
and the vendor step reuses the published build.
- Another digest with the same context: the build fails at the step. The
message names the inputs the image has.
- A tree whose lockfile is behind the manifests of the Runner fails at
the frozen install of the Runner stages, after 169 s, with the message
that stage already had.
- `docker buildx build --check`: no warning, before and after.
`scripts/check-docker-deps-stage.mjs` and
`scripts/check-node-version-policy.mjs` pass.
- Tests, on macOS and in `node:24.22.3-bookworm`, after the rebase: the
changed server test files with the other tests that read the
`Dockerfile` or the image workflows (231 pass in 10 files on Linux),
`vite-build-commit.test.ts` (10 pass),
`runner-release-crate-cache.test.mjs` (11 pass),
`docker-deps-postinstall.test.mjs`. The changed server test files
typecheck.
- Not run: the full test suite, any `linux/amd64` build, `podman`, the
workflows and the release pipeline. Only CI can prove amd64,
`docker.yml`, `docker-merge-group.yml`, `release.yml`,
`docker-app-check.yml` and `runner-image.yml`.

## Risks

- **The digest of the Runner image changes.** Its identity covers the
whole `Dockerfile`. The next `master` push publishes a new
`inputs-<digest>`. The contents do not change: see the comparison above.
- **A build that is given a Runner of other inputs fails; it does not
ship that Runner.** With the argument, it fails at the expectation step.
Without the argument, it fails at the vendor step when the build inputs
of the Runner differ. One comment line in `build-inputs.mjs` changes
those build inputs in this pull request, so the image published before
this pull request cannot be vendored into a tree after it.
- **What a consumer of the application image sees** (measured on
`linux/arm64`):
- The daemon is the stripped binary compiled on Debian bookworm: 23.2
MB, was 36.4 MB, at
`/app/packages/paperclip-runner/dist/bin/paperclip-runnerd` and at the
vendored copy in `server/dist`. Its hash changes. It has no symbols for
a crash backtrace.
- `/app/packages/paperclip-eval-kernel/dist` is absent. Only
`dist/eval/workflow-harness.js` of the Runner imports that package. Only
`eval/index.js` exports the harness again, and no module imports that
file. The server runtime bundle does not name the package.
- The provider pack: `provider-pack.json` is byte-identical. Two
provider executables and one licence file are hard links to their other
copy in the pack, where they were separate files. One empty directory is
absent. The pack is 195 MB smaller, and the layer of its large files
gets a new digest once.
- The transport archives have other bytes, because the Runner build
gives large files a fixed timestamp. The receipt names the same files
with the same digests.
- Paths, modes, owners, the image config and the layer order do not
change.
- **What a contributor and a maintainer of the image build see:**
- **The image build does not run the drift checks of the Runner.**
Before, the server build ran the full `build` script of the Runner, so a
failed drift check failed the image build on `master`. Now the Runner
stages run only the output steps of that script. Repository CI runs the
checks on every pull request. `.github/docker-context-checks.Dockerfile`
and three exceptions in `.dockerignore` exist for those checks in the
image build; a separate follow-up removes them, and this chain does not
edit them.
- **The dependency-caching layer of the daemon is gone.** Before,
cargo-chef compiled the Rust dependencies in a layer of their own, and a
change to the Rust source compiled only the Runner's own crates. Now
such a change compiles the dependencies again (146 s here for the whole
daemon). A change to the server, the UI or the Runner's TypeScript still
reuses the compiled daemon.
- **A lockfile behind the manifests of the Runner fails the build.**
Before, `docker build` resolved it inside the image. The image workflows
refresh the lockfile first. A local build prints the command to run. The
next pull request makes this failure early and clear.
- **The build needs more disk**: about 25 GB of builder storage here,
was about 20 GB, with the image export.
- Rollback: revert this pull request. Images that exist do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude
Code. The model used tools: it edited files, ran the tests and ran the
Docker builds that this description reports.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 09:48:49 -08:00
DottaandClaude Opus 5.5 426e8a35dc perf(runs): read live runs in a fixed number of statements (#21417)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board shows the runs that are live now. The sidebar and the
active agents panel read `GET /api/companies/:companyId/live-runs`
> - That route checked read access and built the output-silence summary
one run at a time, with 8 statements per run
> - On a hosted instance this route ran about 6 times per minute with an
average of 2.1 s of server time, and its statements and allocations add
load to a busy main thread
> - This pull request reads the access decisions and the silence state
for all runs of the request in one batch
> - The benefit is a fixed number of statements per request (15 instead
of 8 + 8 per run) and about one quarter of the allocation

## Linked Issues or Issue Description

No public issue exists. This is an enhancement.

**What existing behavior does this improve?**
The live-runs read on the board (`GET
/api/companies/:companyId/live-runs`).

**Subsystem affected**
Server: agent routes, run read authorization, active-run watchdog.

**Current behavior**
For each run, the route calls `canActorReadHeartbeatRun` (task
existence, instance-admin flag, membership, run policy, task privacy)
and `buildRunOutputSilence` (two watchdog decision reads and one
evaluation-task read). The instance-admin and membership reads repeat
the same answer for every run. Measured locally with a statement and
allocation harness:

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 88 | 10.0 MB |
| 50 | 408 | 27.9 MB |
| 200 (route returns 50) | 408 | 28.6 MB |

**Proposed behavior**
The route reads every access decision in one batch and every silence
summary in one batch, so the statement count does not grow with the
number of runs.

**Reason and benefit**
Fewer statements and less allocation on a route that the board calls
often.

**Breaking changes**
None. The response is the same.

## What Changed

- `server/src/routes/agents.ts`: the live-runs handler decides reads
with the existing `getHeartbeatRunReadDecisions` (the run history list
already uses it) and builds silence summaries with one
`buildRunOutputSilences` call for the readable runs. Unreadable runs are
still redacted with `redactHeartbeatRunListRow`, and no silence state is
read for them.
- Active-run watchdog Postgres adapter: new `findLatestDecisions` and
`findOpenStaleRunEvaluations` read all runs of one company in one
statement each (`DISTINCT ON`). The single-run `findLatestDecision` and
`findOpenStaleRunEvaluation` now call the batch with one run, so both
paths use the same query.
- Watchdog application: new `createBuildRunOutputSilences` use case
groups runs by company. `createBuildRunOutputSilence` now calls it with
one run, so the single and batch summaries use one rule.
- `recoveryService` and `heartbeatService` expose
`buildRunOutputSilences` as pass-throughs.
- Tests: new `live-runs-batched-reads.test.ts`; updated watchdog
use-case tests and the live-run route unit test.

## Verification

- `cd server && npx vitest run
src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres):
- The batched read decision equals one `canActorReadHeartbeatRun` call
per run for 24 actor kinds (owner, admin, operator, viewer, member,
board key, cloud tenant, instance admin, local implicit board, board
without membership, agent keys, agent runs on behalf of users, low-trust
agents, agent of another company, unauthenticated). The runs mix open
tasks, private tasks readable by grant, creation or assignment, private
tasks nobody may read, private-project tasks, a hidden task,
company-scoped runs, a run whose task was deleted, runs of another
company, and runs of another company asked for as this company's.
- For every actor that passes the company gate, each live-runs row is
either the full row or exactly `redactHeartbeatRunListRow` of it (field
by field), as one check per run decides.
- Output silence per run equals the single-run reader for: no output
yet, active snooze plus a later continue, expired snooze, dismissed
false positive, recent output, open evaluation task, done evaluation
task, queued run, an evaluation task of another company, and a run that
has no rows at all.
- The statement count is the same with 19 and 49 live runs for 6 actor
kinds. Before the fix this test fails (for example 347 against 107
statements).
- `npx vitest run src/modules/active-run-watchdog
src/__tests__/heartbeat-active-run-output-watchdog.test.ts
src/__tests__/agent-live-run-routes.test.ts
src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass.
- `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck
of the changed tests: pass. `pnpm check:module-boundaries`: pass.
- After the fix, measured with the same harness (embedded Postgres, 400
tasks, 40 agents, 5 requests each):

| Live runs | Statements per request | Allocated per request |
| --- | --- | --- |
| 10 | 15 | 6.4 MB |
| 50 | 15 | 7.3 MB |
| 200 (route returns 50) | 15 | 7.8 MB |

## Risks

- This changes an authorization read path. The route now uses the batch
helper that the run history list and the live events socket already use.
The test compares it with the per-run check for every actor kind in the
visibility fixture.
- If several open evaluation tasks point at the same run, the batch
picks one of them. The single-run read also picked an unordered one
before.
- No schema, migration, locking or transaction change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:10:30 -06:00
DottaandClaude Opus 5.5 85f4a91e70 fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.

## Linked Issues or Issue Description

**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).

**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.

**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.

**Paperclip version or commit**
`master` (and the hosted build that crashed).

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
  - after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.

## Verification

Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.

Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.

Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:

| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |

Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.

Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`

## Risks

- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 11:00:09 -06:00
DottaandClaude Opus 5.5 24302a09ad fix(environments): stop warm sandboxes once their warm window ends (#21410)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in remote sandboxes (for example Daytona) through
reusable environment leases
> - A warm run ends with its lease `retained` and its sandbox still
running, so the next turn on the task starts warm
> - After the warm window, only the provider's own idle auto-stop
stopped that sandbox. Paperclip never stopped it
> - On staging, idle Daytona sandboxes get provider activity about every
10 minutes. With a 12 minute auto-stop, they never stop and keep billing
> - This pull request makes Paperclip stop a warm sandbox itself when
its warm window ends and no live run holds it
> - The benefit is that idle sandboxes stop on time, while warm turns
inside the window stay warm

## Linked Issues or Issue Description

**What happened?**
Daytona sandboxes for ended warm runs stay in state `started` long after
their warm window. On 2026-11-13 the staging Daytona org had 111 started
sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new
provider activity exactly 10 minutes after their last use, so Daytona
never auto-stopped them.

**Expected behavior**
Paperclip stops a sandbox when its warm window ends and no live run
holds it. Daytona auto-stop stays only a backstop.

**Steps to reproduce**
1. Run a warm `paperclip_runner` task on a reusable sandbox environment.
2. Let the run succeed. The lease becomes `retained` and the sandbox
keeps running.
3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it
runs until the provider auto-stop fires, or forever if activity resets
the provider idle clock.

**Deployment mode**
Hosted (Paperclip Cloud staging) with the Daytona sandbox provider
plugin.

**Additional context**
Plugin Runner task leases (`driver: "plugin"`) are out of scope: their
release path does not use the stop-only contract, so this change leaves
them to their own cleanup.

## What Changed

- New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox
expiry:
- The warm window is the run's resolved runner idle timeout
(`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease
release (`lastUsedAt`).
- Plugin Runner task leases (`driver: "plugin"`) are excluded. Their
cleanup releases through the plugin, not the verified stop-only
contract, so expiry could free the task's files.
- A lease saved before this change has no recorded window. It uses the
longest window a run can configure (24h), so it is never stopped early.
- A candidate is a `retained` `reuse_by_environment` lease whose run has
ended. It is held (never stopped, never a deadline) when another lease
holds the same sandbox, a run on the task is `running` (it may be
resuming this sandbox), or an operation is in progress on the lease
(stop intent, export resume, restore recovery, cleanup claim,
successor/supersede markers).
- Due sandboxes go through the existing stop-only contract
(`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact
sandbox at the provider. The lease then ends `released` as stopped and
kept, which the next run on the task resumes.
- Acquisition (plugin and built-in sandbox paths) records the run's warm
window on the lease (`warmSandboxIdleTimeoutMs`).
- `nextEnvironmentCleanupAt` includes the earliest warm window end. The
environment cleanup worker sleeps until that deadline. It also wakes on
every lease release and runs at boot, so a sandbox left running across a
restart or deploy is reconciled. There is no interval scan.
- `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at
once, so the first stop does not wait for the 5 minute retry backoff.
- The pending-cleanup sweep hands due warm sandboxes to the stop-only
cleanup before it reads its rows, so one pass stops them.
- Only this instance's own lease rows are judged. Nothing lists provider
resources, so a sandbox of another company or environment is never
touched.

## Verification

- `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts`
(server): 9 passed.
- Inside the warm window: kept, and the worker deadline is exactly the
window end.
  - Past the window: stop intent recorded, first stop due now.
- Through the cleanup sweep: the provider stop is dispatched once; the
lease ends `released`, stopped and kept.
  - A live run's lease on the same sandbox: kept, and no deadline.
  - A run on the task is starting: kept.
  - The lease's own run is still running: kept.
- A plugin Runner task lease past its window: left to its own cleanup,
and no deadline.
- Orphan after a restart: a lease retained 35h ago without a recorded
window is stopped on the first (boot) pass; one retained 1h ago is kept
with a deadline at 24h.
- Red/green: with the wiring removed (sweep step, deadline,
first-attempt marker), 4 of the 8 tests fail (window deadline,
past-window stop, sweep stop, restart reconcile). With the change, all 8
pass.
- CI on this PR covers the broad suites for the final revision. A local
`pnpm verify` selected a very wide set (the change touches
`heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under
machine load, so I stopped it and rely on CI.
- After deploy, I check the staging Daytona org: started counts for
ended warm runs drop, and live or in-window sandboxes stay started.

## Risks

- A run that starts on the task in the moment between the deadline check
and the stop could resume a sandbox that is being stopped. A `running`
run on the task holds the sandbox, and the stop intent is a
compare-and-swap on the exact retained row, so this window is small. The
same race already exists for the retained-orphan settle.
- The first boot after deploy stops every retained warm sandbox that is
older than 24h (or past its recorded window). This is the intended
cleanup. It makes one stop call per lease, 25 leases per pass.
- The stop-only contract disables Daytona auto-delete on the stopped
sandbox (existing behavior of that contract). Daytona auto-archive still
moves it to cold storage. The retained-orphan settle destroys it when
the task no longer uses it.
- The deadline query reads `retained` leases without a dedicated index,
the same as the existing pending-cleanup deadline. It runs only when the
cleanup worker wakes.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Daytona API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:47:43 -06:00
DottaandClaude Opus 5.5 e7e28ba241 feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns`
tasks at once. Other runs wait in the agent's queue with status `queued`
> - On hosted staging, one agent had all 20 slots full and 295 queued
runs. Some tasks waited 3 to 7 hours and never started
> - The UI showed these tasks as live or working: a pulsing "Live" badge
in the task header and in task rows, a spinning "Working … for 4 hours"
in the task chat, and "N live" counts that included every queued run
> - People could not tell that work was waiting, why it waited, or what
to change
> - This pull request makes a deep queue an honest, visible state:
"Waiting for Codie · 20 slots max · 41 of 295 in line · Configure"
> - The benefit is that operators see the real state and the one setting
that changes it

## Linked Issues or Issue Description

Refs #21391, #21349, #21242

**What happened?**

A queued run that never started looked the same as a running run. The
task header, the task rows in lists and the inbox, the task chat and the
sidebar counts all showed queued runs as live or working.

**Expected behavior**

A run that waits for one of its agent's concurrency slots shows that it
waits. It names the agent, the agent's limit and the run's place in
line, and it links to where the limit is set. The place in line is the
order in which the server starts the agent's queued runs.

**Why does it matter?**

With a deep queue, the UI said "working" for hours while nothing ran.
Operators could not see the cause or the fix.

## What Changed

- Server: a new module
`server/src/services/heartbeat/queued-run-line.ts` now holds the facts
that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The
dispatch path in `queue.ts` and the new "place in line" read use this
one builder, so the line the UI shows is the dispatch order. The logic
moved without change, including the recovery-share fix from #21391.
- Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit,
running count, position, queued count) and the label "Waiting for
<agent>" with cause `agent_at_capacity` to a queued, never-started run
when its agent has no free slot. It reads one batch of agents and
running counts. Then it builds one line per full agent (at most 8 agents
per read), using only the three context keys that ordering needs. No new
polling.
- Server: the place in line uses `queuedRunDispatchLine`, an O(n log n)
replay of the dispatcher. The plain replay rescans the queue after each
step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000,
600 ms at 10,000, all on the event loop). Dispatch itself is unchanged:
it stops after the free slots. Each agent's line is memoized for 5 s and
shared by concurrent reads. In a preview worktree, the line applies the
same creation-time cutoff as dispatch.
- Server: the task list summary adds `runningDescendantCount`
(descendants whose run started) next to `liveDescendantCount`.
`liveDescendantCount` still decides whether a blocked parent is covered.
The "n live below" chip uses the new count.
- Shared: the new `ExecutionAgentQueue` type and the optional field
`ExecutionProjection.agentQueue`.
- Server: `/companies/:companyId/live-runs` lists running runs before
queued runs, so a deep queue cannot push running work out of the 50-row
limit.
- UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20
slots max · 41 of 295 in line · Configure", with no spinner and no
elapsed timer. Configure opens the agent's Advanced view, where the run
policy (`maxConcurrentRuns`) is edited. The task chat (both renderers),
the run ledger and the Storybook story use it.
- UI: the task header and the task rows (task list, inbox, legacy views)
show a quiet "Waiting" instead of the pulsing "Live" when every live run
of the task has not started. The full sentence shows on hover. "n live
below" counts only started sub-tasks. A waiting sub-task still covers
its blocked parent.
- UI: the sidebar "N live" counts (Dashboard, agents) count only started
runs.
- UI: the task chat pill now names other server holds on a queued run
(for example "Waiting for maintenance") instead of "Working".
- Docs: DESIGN.md records the waiting-for-slot hold.
- Storybook: `runs-agent-queue-wait` stories (full line, task chat, task
row, one slot, mobile), with play checks.

## Verification

- Red/green server test:
`server/src/services/execution-projection-agent-queue.test.ts` (embedded
Postgres). Before this change, the projection has no `agentQueue`. Now
it checks the agent name, limit, running count, positions 1 to 4 in
dispatch order (a person's retry first, then priority, not creation
order), no wait while a slot is free, and isolation from other
companies' runs. It also checks that the positions equal the line that
the dispatch builder produces.
- `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`,
`recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and
`agent-live-run-routes` in `server/`.
- Runtime cost (rule `runtime-dependency-boundaries`): the assumption is
that the line stays cheap for a deep queue. `queued-run-line.test.ts`
checks that `queuedRunDispatchLine` equals the dispatcher's
claim-every-run order over 400 random queues (ages around the aging
boundary, recovery and person runs, blocked runs, deferred share runs,
all share counters). It also orders 10,000 runs in under 250 ms (the
quadratic replay takes about 600 ms). The DB test projects a 300-run
queue end to end with correct positions 1 to 300. Remaining limit: the
line is up to 5 s stale.
- Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts
`runningDescendantCount: 0` for a parent whose only live descendant is
queued and never started.
- UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep
"Live", no "running below" count for waiting sub-tasks) and
`TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the
Configure link, other holds named). The suites for the run ledger,
sidebar, sidebar agents, task detail, task columns and task chat also
pass.
- The Storybook stories render in headless Chromium and their play
checks pass.
- After deploy: open a queued task on staging and confirm the waiting
state.

## Risks

- Cost on read: a projection read that includes a queued run of a full
agent now reads that agent's queued runs (only small columns) and the
dispatch facts. This is at most one line per full agent every 5 s, for
at most 8 agents per read. Reads with no queued, never-started runs do
no more work.
- The place in line is the server's prediction of dispatch order. A
claim that is refused at start time (a gate, a blocker) can let the next
run start first. The position updates on the next refresh.
- The process-local "declined aged claim" hint that the dispatcher uses
is not visible to other servers, so the projection does not use it. This
can change only the aged-oldest share step.
- Configure opens the agent's Advanced view (the YAML run policy). The
agent's Runtime tab hides the run-policy fields today.
- Low risk otherwise: no schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser) and subagents for the UI survey.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:24:20 -06:00
DottaandClaude Opus 5.5 b0e99dfcfa fix(connections): show the setup card unless the person asked to set up the connection (#21400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents find and request app connections with `connections_search`
and `connection_request`
> - For one unconnected match, the search told the agent "The user asked
to connect" and said to configure the connection with `call_api`
> - That text was given for every search, also when the person only
asked a question or asked for app data
> - The agent then created a draft connection itself and sent an
absolute `http://127.0.0.1:<port>` link, and the person got no setup
card to accept or decline
> - This pull request gives the configure instruction only when the
person's own message asks to set the connection up, and makes the
agent's sign-in handoff link a relative app route
> - The benefit is that a question or a data request shows the setup
card, and setup links open on the host the person uses

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12 (OpenCode with DeepSeek V4.1 Flash), the person
wrote "Read my Notion and tell me the latest on the launch".
`connections_search("Notion")` returned one result with the instruction
"The user asked to connect. If method.management.allowed, use
search_api/call_api ... to configure it". The agent called `POST
tools/apps/connect` twice and replied with
`http://127.0.0.1:<port>/COM/apps/connect?resume=...`. No setup card was
shown, so the person could not decline. The same happened for "What is
the easiest way to connect Notion here?".

**Expected behavior**
When the person asks about an app or needs its data, the agent shows the
setup card (`connection_request`) and answers in plain words. The agent
configures a connection itself only when the person asks for that setup.
Links the agent gives are relative app routes.

**Steps to reproduce**
1. Have no Notion connection.
2. In chat, write "Read my Notion and tell me the latest on the launch".
3. See that the agent creates a draft connection and links an absolute
URL, with no setup card.

## What Changed

- `server/src/services/connection-intents.ts`: for a single unconnected
tool match, read the latest human message on the task. Give the
configure instruction only when its first clause opens with a setup verb
and names the matched app ("Connect Notion, then ...", "Can you set up
Notion?"). "Connect Slack, then read my Notion" does not count for
Notion. Otherwise tell the agent not to create a connection or draft.
For a setup question, the agent answers in plain words and links the
catalog setup page `/<prefix>/apps/connect?source=<app>`, because
`connection_request` ends the turn before any answer text is shown. For
a data request, the agent calls `connection_request` to show the setup
card. A saved custom connection always uses the card. The check is
conservative: a miss never configures anything.
- `server/src/services/tool-access/oauth/authorization-start.ts`: the
agent handoff link (`authorizationUrl`, also
`requiredUserAction.setupPath`) is now a relative app route, not built
from the server's redirect origin.
- Tests: a red/green service test for the three messages, and the
handoff test now requires a relative link.

## Verification

- `cd server && npx vitest run
src/__tests__/connection-intents-service.test.ts -t "configure a single
match"`: passes for a data request, a setup question, a mixed-app
request and an explicit setup request. With the old
`connection-intents.ts` it fails (`expected 'The user asked to connect.
...' to contain 'connection_request with service notion'`).
- `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t "generic app
authentication"`: passes.
- A narrow paid eval run of `connection-setup.notion-declined` and
`connection-setup.choose-provider` follows on this branch.

## Risks

- A person who asks for setup with other words ("I'd like Notion hooked
in") now gets the setup card instead of agent-side configuration. The
card still completes the setup.
- One small indexed read of the latest human message per single-result
search. There is no change to agent boot or time to first token.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:02:36 -06:00
DottaandClaude Opus 5.5 e40a16e5c5 fix(heartbeat): let Stop take a task out of the agent's queue (#21407)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs a limited number of tasks at once. Other tasks wait
in line as queued runs.
> - An agent can have hundreds of waiting tasks. People need one click
to take a task out of line.
> - Stop already cancelled a queued run, but a task that was in progress
stayed in progress. It still looked busy.
> - The classic task thread also hid Stop until the run started.
> - This pull request makes Stop on a waiting task free the task and
show it as not started, in both thread views.
> - The benefit is that people can clear a long queue one task at a
time, and the agent moves on to its next task.

## Linked Issues or Issue Description

**What happened?**

An agent had many queued runs. A person pressed Stop on a task that
waited in line. The run was cancelled, but the task stayed "in
progress". In the classic thread, Stop did not show until the run
started.

**Expected behavior**

One Stop click takes the task out of line. The queued run and its wake
are cancelled. The task lock is released. The task does not look in
progress. No new run starts for that task. The agent's next waiting task
still starts. An activity entry records the change.

**Steps to reproduce**

1. Give an agent one run slot and keep it busy.
2. Assign two more in-progress tasks to the agent, so their runs wait in
line.
3. Press Stop on one waiting task.
4. See that the task still shows "in progress".

**Paperclip version or commit**

`master` at 30c718e04b.

**Agent adapter(s) involved**

Not adapter-specific (core bug).

## What Changed

- `POST /heartbeat-runs/:runId/cancel`: when the run was queued, the
route calls the new `returnStoppedWaitingTaskToTodo` after the cancel.
- New `server/src/services/heartbeat/stopped-waiting-task.ts`: one
guarded update moves an `in_progress` task to `todo` and logs
`issue.updated` (reason `stopped_while_waiting`). It changes the task
only if the run never started, the task is still assigned to that agent
and not to a person, the task lock is free or held by that run, and no
other live run exists for the task.
- The route keeps its company and run-read checks and still logs
`heartbeat.cancelled`. Recovery already stands down for an operator
stop, so no new run starts for the task.
- `IssueChatThread` (classic view): the composer shows Stop for a queued
run too, in the same place as a running task's Stop. The default
`TaskChatThread` already did this; a test now covers it. The waiting
banner is unchanged.

## Verification

- `server/src/__tests__/heartbeat-stop-queued-task.test.ts` (new,
embedded Postgres, real heartbeat): one busy slot, two waiting tasks.
Stop one waiting task. The run and wake are cancelled, the lock is
released, the task is `todo`, and an activity entry exists for the
in-progress case. After the slot frees, the next task runs. The stopped
task gets no new run, also after `resumeQueuedRuns` and
`reconcileStrandedAssignedIssues`. The in-progress case was red before
the fix.
- `server/src/__tests__/heartbeat-run-privacy-routes.test.ts`: a real
route test. Another user gets 404 and the task does not change. The
task's operator gets 200, the task becomes `todo`, and both activity
entries are recorded. Red without the route change, green with it. The
full file passes (30/30).
- The same route test sends a correlated Stop (`cancellationRequestId`)
to the waiting legacy run. The server returns 409 and the run and task
do not change. Correlated Stop is for native runs only.
`ui/src/api/heartbeats.test.ts` checks that task Stop sends the plain
request (`{}`).
- `ui/src/components/IssueChatThread.test.tsx` and
`TaskChatThread.test.tsx`: Stop shows for a queued run and calls the
stop action. The classic test was red before the fix.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/ui typecheck` pass.

## Risks

- Low risk. The status write is one guarded `UPDATE` and changes nothing
if any guard fails. Only the board Stop route calls it, and only for a
run that was queued.
- A task stopped while waiting shows `todo` instead of `in_progress`.
This is intended. A Stop on a running task does not change its status.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:58:54 -06:00
DottaandClaude Opus 5.5 b7774a01cf fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent

## Linked Issues or Issue Description

**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.

**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.

**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.

## What Changed

- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.

## Risks

- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:53:25 -06:00
DottaandClaude Opus 5.5 1529bca5c1 fix(chat): do not post a completion report that an earlier reply already gave (#21401)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a delegated task that finishes wakes the chat agent
with `chat_task_completed`, so it tells the person in the same
conversation
> - A turn that runs for another reason, such as the person's form
answer, can read the task when it is already done and report it
> - The completion turn then starts after it and reports the same news
again
> - The person gets two completion messages for one task
> - This pull request records when the chat agent reads its handed-off
task in the done state, and skips the completion turn when an earlier
reply already reported that completion
> - The benefit is one completion report per task, and no extra model
call for it

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12, case `questions-and-approvals.duplicate-answer`:
the delegated draft task finished at 13:48:23 while the chat agent's
turn (woken by the form answer at 13:48:16) was running. That turn read
the task, saw it done and replied "Acorn is confirmed ... the draft is
finished" with a task link (13:48:50). The `chat_task_completed` turn
started at 13:48:53 and replied "The Acorn draft is done" (13:49:22).

**Expected behavior**
The person gets one completion report per finished task in the
conversation.

**Steps to reproduce**
1. In chat, delegate a short task.
2. While it runs, send a message that wakes the chat agent.
3. The task finishes during that turn, and the turn reads and reports
it.
4. See a second completion report from the `chat_task_completed` turn.

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`recordCompletedHandoffRead` appends a `chat_handoff.completed_read` run
event when the conversation agent reads its own handed-off task in the
done state (with the task's status version).
- `prepareChatCompletionTurn`: a delivery is marked delivered (with the
earlier reply as its response) when an earlier reply of the same agent
in the conversation was posted after completion, links the task, and
comes from a run that read the task at that exact done status. Both
conditions are structural (run event and task reference), not reply
wording. A reply that links the task without such a read (for example
"Morgan is working on it", from a read before completion) does not
count.
- `server/src/services/heartbeat.ts`: when every delivery of the
completion turn was already reported, the run ends as succeeded before
the conversation turn starts, with no model call.
- `server/src/routes/issues.ts`: task reads by an agent run call the
recorder (one primary-key read of the handoff, and an event only for the
chat agent's own done handoffs).

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts`: 53 pass, including two
new tests: the already-reported completion is delivered without a turn,
and a stale linking reply still gets the completion turn.
- Red: with the earlier-reply check disabled, "does not report a
completion again ..." fails (`expected false to be true`).
- A narrow paid eval run of `questions-and-approvals.duplicate-answer`
follows on this branch.

## Risks

- If the earlier reply linked the task but did not say it was finished,
the person does not get a second message. The read-at-done-status
condition makes this unlikely.
- The skipped completion run is a short succeeded run with no model
call. Boot and time to first token are unchanged for all other turns.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:49:22 -06:00
DottaandClaude Opus 5.5 5c5a2fcebe fix(dispatch): start the assigned-work sweep's continuations and count only system recovery for the recovery share (#21391)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. #21234, #21277
and #21349 order them: person work first, a recovery share, aging, and a
share for the longest-waiting run
> - On hosted staging, one agent had about 250 queued runs. For 90
minutes after #21349 deployed, every run that started was a retry or
continuation with `retryOfRunId`. The oldest queued runs, from 15 hours
earlier, never started
> - The dispatch order was not bypassed. The oldest-run share offered
those runs, but the claim refused them every time, without a log at info
level or a visible state
> - The refusal came from the chat-control recovery gate. It needs
`retryOfRunId` to trace an automatic continuation. The assigned-work
invariant sweep queues continuations that name their source only in
context, so the gate returned "unresolved", and an unresolved claim
stays queued forever
> - This pull request lets the gate check such a continuation against
the task's previous run, and makes the recovery share count only system
recovery starts
> - The benefit is that the sweep's continuations start (it already told
people "you don't need to press Retry"), and system recovery gets its
share even when a person's recovery restarts keep starting

## Linked Issues or Issue Description

Refs #21234, #21277, #21324, #21349, #21242

**What happened?**

On hosted staging (build e23549cd9c), one agent with `maxConcurrentRuns`
20 had about 250 queued runs. From 12:06Z to 13:30Z, all 30 runs that
started had `retryOfRunId` set. The oldest queued runs were automatic
continuations (`issue_continuation_needed`) created from 22:53Z the day
before. Each task's activity shows "Continued automatically after an
interruption ... you don't need to press Retry" from the assigned-work
invariant sweep, at the time the run was queued. None of these runs
started. Assignments created at 07:51Z and later did start, in creation
order. Those starts came through the oldest-run share after the share
passed over the refused continuations.

**Why did it happen?**

1. `assignedWorkSweep` queues the continuation with `previousRunId` in
its context and no `retryOfRunId`. Its wake is a system wake
(`requestedByActorId` null) with reason `issue_continuation_needed`.
2. At claim, `withChatControlRecoveryGate` calls
`readChatControlRecoveryStop`. For an automatic continuation without
`retryOfRunId`, it returned `unresolved`. At the claim stage,
`unresolved` returns null: the run stays queued and nothing changes. So
every dispatch pass offered these runs and every claim refused them.
Retries and continuations with `retryOfRunId` pass the gate, so they
were the only recovery work that started.
3. Separately, the recovery share (#21234) read the agent's recent
starts and counted every start with a recovery wake reason as a recovery
start. A person's `issue_recovery_action_restored` restart ranks as
person work in the dispatcher, but it still reset the share. So the
recovery share almost never fired while those restarts kept starting.

**Free slots.** The "14 of 20 running" figure came from a capped run
list. A full read at 14:03Z showed 22 runs `running` on that agent, so
there were no free slots. The agent was over its limit, not under it.
Overlapping dispatch passes after the 30 s stale agent start lock are a
likely cause (each pass re-tried every refused run, so passes were
slow). This pull request removes the refused runs that made passes slow.
#21242 changes the claim-then-execute timing.

**Expected behavior**

An automatic continuation that the system queued starts in its turn. A
chat close of the task's previous run still stops it. The recovery share
goes to system recovery after three starts that were not system
recovery.

**Steps to reproduce**

1. Give an agent one slot. Interrupt a run on an in-progress task so
that the assigned-work invariant finds a violation.
2. Run the assigned-work check. It queues an `issue_continuation_needed`
continuation.
3. Queue fresher recovery runs on other tasks and dispatch.
4. See that the continuation stays queued on every pass.

## What Changed

- `server/src/services/chat-control-recovery-stop.ts`: for an automatic
continuation that has no `retryOfRunId`, the gate now walks to the
task's latest earlier run (by `issueId`, `nativeIssueId` or the context
task id, in the same company) and checks that run. If the task has no
earlier run, the result is `clear`. A continuation with `retryOfRunId`
is unchanged. A chat close of the previous run still stops the
continuation.
- `server/src/services/heartbeat/queue.ts`: the recovery-share counter
reads which recent recovery-reason starts were person-originated, with
the same source-chain query that the dispatcher already uses
(`listPersonOriginSourceRunIds`, at most 4 ids). Only system recovery
starts reset the share. This matches the in-pass rule
(`isSystemRecoveryRun`).

## Verification

- Red/green, embedded Postgres, real heartbeat queue
(`server/src/__tests__/recovery-wake-herd.test.ts`):
- "starts the assigned-work sweep's continuation of a task whose run was
interrupted, ahead of fresher retries": on master the continuation stays
queued for 60 s of dispatch passes; with the fix it starts.
- "gives system recovery its share when the agent's recent starts were a
person's recovery restarts": on master a person run starts first; with
the fix the system recovery run starts.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: "checks an
automatic continuation that names no retry source against the task's
previous run" proves that a chat close of the previous run still stops
the continuation (`stopped`) and that it is `clear` when the close is
not published.
- `node server/scripts/typecheck-layers.mjs`: passes.
- Related suites: `recovery-wake-herd`, `queued-run-dispatch-order`,
`assigned-work-invariant`, `run-authority-read-projections`,
`chat-run-milestone-work`, `legacy-preparation-retry-hold`,
`explicit-native-continuation`, `saved-work-retry-eligibility`,
`native-failed-turn-continuation.integration`,
`heartbeat-process-recovery` pass, except:
- `heartbeat-process-recovery` "schedules bounded retries for failed
accepted interaction continuation wakes" is flaky on master too (1 of 3
runs failed on master source, same line).
- `recovery-wake-herd` "delivers the continuation after a restart..."
failed once with a hook timeout under parallel load and passes alone.
- `runtime-selection.test.ts` fails locally in a nested worktree because
a bundled skill file path does not exist there (environment, not this
change).
- After deploy: the queued `issue_continuation_needed` runs on the
affected staging agent should start within the oldest-run share, and no
run should stay queued longer than the aging bound plus a few starts.

## Risks

- Behavior change: an automatic continuation without `retryOfRunId` was
refused at claim and failed at dispatch with
`chat_control_recovery_proof_unresolved`. It now inherits the chat-close
check of the task's previous run instead. A chat close still stops it.
- Already-queued stuck continuations on hosted instances will start
after deploy. On the staging agent that is about 30 runs. They go
through the normal order and shares, so they do not take all slots at
once.
- One extra indexed read per gate check of this shape, and one small
recursive query per dispatch pass when recovery runs are queued. No
schema change. Mixed versions are safe.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, built-in browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:51:33 -06:00
DottaandClaude Opus 5.5 30c718e04b test(decisions): wait for both provider calls instead of a 1-second poll in the concurrent admission test (#21387)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Company decision models admit model calls against a budget, and the
server test suite checks this admission
> - The test "admits concurrent work from the same budget observation
and records both obligations" fails sometimes on loaded CI runners
> - Cloud readiness runs this test, and any Cloud readiness failure
stops the Core image from being published
> - On 2026-11-13 this test failed in Cloud readiness for two master
commits and stopped image publication
> - This pull request makes the test wait for the real condition instead
of a 1-second poll
> - The benefit is that a slow runner no longer fails the test, so the
release gate does not stop on a timing flake

## Linked Issues or Issue Description

No public issue exists. This is a test flake.

**What happened?**
The test failed in Cloud readiness "verify / General tests (server
(4/10))" on master. The failure was `AssertionError: expected "vi.fn()"
to be called 2 times, but got 1 times` at the second `vi.waitFor`. It
happened for master commit 11840f9904 (run 43435593630, attempt 3,
14:03Z) and for e3faf579a1 (run 43427997745, 13:00Z). The other 3661
tests in that run passed.

**Expected behavior**
The test passes on a slow runner. After the lock is released, both
decisions commit their reservations and call the provider. On a loaded
machine this can take more than 1 second.

**Steps to reproduce**
Run `server/src/__tests__/decision-models.test.ts` on a heavily loaded
runner. The second `vi.waitFor` uses the default 1000 ms timeout. The
line before it uses 3000 ms. If both provider calls do not start within
1 second of the unlock, the test fails.

**Paperclip version or commit**
master at 6917d0568d

## What Changed

- The test provider mock now counts the calls that start. It resolves a
promise when the second call starts.
- The test awaits that promise instead of polling
`toHaveBeenCalledTimes(2)` with a 1-second timeout. The test timeout
still applies if the calls never start.
- No product code changed.

## Verification

- `cd server && npx vitest run src/__tests__/decision-models.test.ts`:
37 passed locally.
- A red/green proof is not practical. The failure depends on runner
load, and the old code passes on an idle machine. The new code has no
timer between the unlock and the check, so it cannot time out early.

## Risks

- Low risk. This is a test-only change.
- If a regression stops the second provider call, the test now fails at
the test timeout instead of after 1 second. The failure is still
reported.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, GitHub CLI, AWS CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (not
applicable: test-only)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:38:43 -06:00
DottaandClaude Opus 5.5 944b1c64d7 feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connected apps through Paperclip's MCP client in the tool
gateway
> - Some provider actions run for minutes: CI runs, deploys, batch jobs
> - Today such a call is lost if it outlives the tool timeout or if
Paperclip restarts or deploys mid-call, and a retry can start the work a
second time
> - Recovery must just work, and the target rate of runs that need a
manual retry is 0%
> - This pull request implements the MCP Tasks extension
(`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client
> - The benefit is that a long provider call becomes a durable task: it
survives timeouts, restarts, and deploys, and it is never started twice

## Linked Issues or Issue Description

**Problem**
A long provider tool call holds an HTTP request open. When it passes the
tool timeout, or when Paperclip restarts or deploys during the call, the
result is lost. The agent then often repeats the call, and the provider
does the work again.

**Proposed solution**
Implement the client side of the MCP Tasks extension. Declare it on
stateless tool calls. Store each returned task durably, wait on it
during the call, keep polling it in the background after the call ends,
and let the same call attach to the stored task instead of sending
`tools/call` again.

**Why it matters**
CI, deploy, and batch actions are where agents lose the most work today.
A durable task handle makes these actions safe across timeouts,
restarts, and deploys, without a person.

Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP
client, and MCP Skills work this builds on).

## What Changed

- **Off by default:** MCP Tasks are behind the new instance experimental
setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default
off. When it is off, Paperclip never declares the extension, nothing
attaches to or polls tasks, pending rows do not block idle sleep, and
behavior matches master. A test covers the off state.

- `mcp-http.ts`: the tasks extension is declared per request, only on
HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers
do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`.
- New `mcp-tasks.ts`: parses task results, bounds the server's timing
hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals
task IDs at rest (AES-256-GCM, bound to company and connection; a keyed
hash gives uniqueness).
- New company-scoped table `tool_mcp_tasks` (migration `0475`,
online-compatible). It links the task to the call, run, task,
connection, and grant. It stores no provider result.
- `tool-gateway.ts`:
- A returned task is recorded, then polled up to the normal tool
timeout. Its result goes through the same validation, redaction, and
recording as a direct result.
- Past the timeout, or when the caller disconnects, the agent gets
`provider_task_pending` ("call again with the same arguments"). The task
stays durable.
- A call with the same endpoint, tool, full arguments, grant, agent,
task, and actor attaches to the undelivered task. A retry with the same
idempotency key also attaches, and it can never start different work
under that key.
- Input: URL requests show the provider link (Paperclip never answers
them). Form requests become a question card; only the person's answer is
sent with `tasks/update`. Other requests stop with a clear "needs input"
error.
- Stopping the run sends `tasks/cancel` and marks the task cancelled. A
caller that only disconnects keeps the task.
- `failed` becomes a tool error with the provider's message. An unknown
task or a passed TTL becomes `provider_task_expired`. Every task stops
being tracked at its TTL, or after 7 days at most.
- Audit entries: `tool_gateway.task_started`, `task_input_sent`,
`task_cancelled`.
- New `mcp-task-work.ts`, registered at startup with the shared deadline
scheduler: it resumes polling pending tasks after a restart, cancels
tasks whose task closed or run was stopped, and wakes the assignee once
when a task settles after its run ended.
- Docs: `doc/connections/MCP-TASKS.md`, linked from
`doc/connections/README.md`.

Background work (per `rules/sleep-aware-background-work.mdc`): trigger
is a task row write or release (`mcp-tasks` queue) and one startup pass.
The deadline is the earliest `next_poll_at`. Cadence is the server's
`pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting
for a person. It is quiet when no row has a next poll. This is the
narrow external-source exception: providers give no wake Paperclip can
receive, and polling stops with the task.

## Verification

- `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake
2026-07-28 server that returns tasks only to a client that declares the
extension and checks `Mcp-Name`. They cover:
- polling to completion, with the result identical to a synchronous
call;
  - timeout, then an attaching call with no second `tools/call`;
  - a keyed retry, and a reused key with different arguments (refused);
- restart mid-task: a fresh gateway with no in-memory state polls the
row, wakes the agent once, and the recovered run gets the result once
with no second `tools/call`;
  - no wake while the run is still going;
- URL input (link, never answered), and form input (question card, then
`tasks/update` with the answer);
- cancellation by the run, by closing the task, and no cancellation when
only the caller leaves;
- failure, an unknown task, and TTL expiry (ends in under 3 s, no hang);
  - another agent never attaches;
- synchronous calls unchanged and older servers never see the extension.
- Red/green: with attach disabled, the timeout, keyed-retry, and restart
tests fail (3). With `tasks/cancel` disabled, both cancellation tests
fail (2). Before `tools/call` declared the extension, 13 end-to-end
tests failed. All pass with the change.
- Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`,
`mcp-skills`, `server-startup-feedback-export`, `tool-gateway`,
`tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests).
- `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db
check:migrations` passes (the migration is classified
online-compatible). Snapshots pruned to five.
- Agent boot and time-to-first-token: no change in the boot or
first-turn path. The only new per-call cost is one indexed lookup on
stateless tool calls.
- Not run: a live provider that returns tasks. No public MCP server that
returns tasks was available without new credentials.

## Risks

- Shipped off by default. Review findings that only affect the
setting-on path after this merge are tracked as follow-ups.

- Migration: a new table with indexes and foreign keys to `companies`,
`tool_invocations`, and `tool_connections`. Online-compatible.
- Behaviour change only for servers that return tasks, which they may do
only to clients that declare the extension. Older servers and stdio
servers are unchanged.
- A task that settles while nobody is waiting stays attachable for 1
hour, so a deliberate repeat of the same call within that hour collects
the earlier result instead of starting new work.
- The wake after a background task settles uses the generic task wake;
the agent must call the tool again to collect the result.
- Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling
and roots input requests; showing a long action's progress in the task
chat.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:32:06 -06:00
DottaandClaude Opus 5.5 5026093d0f feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, Paperclip records a fresh continuation
and delivers it later as a new run. Since #21353 the task reads as
continuing (`working`, no Retry) while it waits
> - #21373 added "Move it forward now" (`retryAction: "move_forward"`).
It is offered only after the delivered continuation run waited 15
minutes in the agent's queue
> - Before delivery (`continuationDelivery: "pending"`) the task showed
no button. On hosted staging such continuations waited 30 to 110 minutes
behind a deep queue, and the person had no control
> - #21378 and #21379 made the click safe in that state: a
`move_forward` request with a recorded, undelivered continuation is
accepted and delivers that continuation as the one run, with its
recorded isolation
> - This pull request also offers "Move it forward now" while the
continuation waits for delivery, once it has waited 15 minutes since it
was recorded
> - The benefit is one action, with the same label and copy, for both
waiting states. The click never starts a second run

## Linked Issues or Issue Description

Refs #21353, #21373, #21378, #21379

**What happened?**

A task has a fresh continuation that is recorded but not delivered
(`continuationDelivery: "pending"`). It waits behind a deep agent queue
for 15 minutes or more. The task shows "continuing" with no action.

**Expected behavior**

After 15 minutes the task still reads as continuing (`working`,
responsibility `system`, never `failed`). It also offers "Move it
forward now". The click delivers the recorded continuation as the one
run.

**Steps to reproduce**

1. Record a fresh continuation for a task while its agent is busy. Do
not run the delivery pass.
2. Wait 15 minutes.
3. Read the task presentation. It shows `canRetry: false` and no
`retryAction`.

## What Changed

- `server/src/services/task-interruption-presentation.ts`: the
fresh-continuation read also loads the record status, schema,
`recordedAt` and reconciled run. A `continuing` state can now carry
`pendingSince` when the primary record is `resolved`,
`continuationDelivery` is `pending`, the schema is the
fresh-continuation schema, and the reconciled run is the observed run.
These are the same facts that `undeliveredContinuation` checks at click
time.
- The projection offers `canRetry: true, retryAction: "move_forward"`
(state `working`) when `pendingSince` is 15 minutes or older, no person
Retry for that run already waits, and the run's agent is the current
assignee. Under 15 minutes, nothing changes.
- No button when the system gave up delivery (`invalidated`; the normal
Retry rules apply), when the delivery is `not_requested`, when a person
already asked, or for a held restore (the existing execution-blocker
post-filter removes the offer).
- New integration test
`server/src/services/fresh-continuation-move-forward-offer.test.ts`. It
uses the real click path for the three recorded kinds
(`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`):
presentation, `requestTaskRetry` with the shown `observedRunId` and
`retryAction`, `resumeTaskRetry`, then the delivery pass.
- Updated guards in `task-interruption-automatic-path.test.ts` and
`recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a
pending continuation shows no Retry at 30, 45, 110 and 240 minutes now
expect the move-forward offer. The `working` / `system` / never-failed
assertions stay. New cases cover 14 minutes, `not_requested`, a waiting
person Retry and reassignment.
- `doc/execution-semantics.md`: describes the new offer and its races.
- No shared type, validator, UI or API client change. The action and its
UI copy ("Waiting for the agent to finish other work. Paperclip
continues this task by itself." / "Move it forward now") already exist
and do not depend on the state. No runner-e2e SDK public-input pin
covers the changed files.

## Verification

- Red: with the presentation change reverted, 16 tests fail (`expected {
intentId: null, …(7) } to match object { responsibility: 'system', …
retryAction: 'move_forward' }`).
- Green: `npx vitest run
src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14
pass. For each kind:
- The click gives exactly one run. It is the recorded continuation
(`recoveryActionId`, `previousRunId`, `retryOfRunId`,
`forceFreshSession`). The fresh isolated workspace is kept
(`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The
record reads `delivered` with that run. Two later delivery passes and a
repeated resume create no second run, and the offer then ends.
  - The click and two concurrent delivery passes give one run.
- Delivered between the offer and the click: the click is refused
(`accepted: false`, no wake) and the only run is the delivered
continuation.
- The system gave up the delivery between the offer and the click:
`alreadyMovedOn: true`, no run.
- The task moved to another run between the offer and the click:
refused, the only run is that other run.
  - Under 15 minutes: no button.
- Guards: `task-interruption-automatic-path.test.ts` (37/37),
`recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`,
`task-interruption-query.test.ts`,
`fresh-continuation-retry-delivery.test.ts`,
`fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass.
Also the saved-work, workspace-recovery, native restart and
closed-session retry suites that read the presentation.
- `npx tsc --noEmit -p .` in `server/`: no errors.

## Risks

Low risk. The change is presentation only. It adds four selected JSON
fields to a read that already exists and one conditional module import.
The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is
unchanged and already holds the task row lock. A record without
`recordedAt` or with an older schema gets no button, as before. People
now see a button on tasks that showed none after 15 minutes. The button
can only deliver the recorded continuation or do nothing.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:24:12 -06:00
DottaandClaude Opus 5.5 f7ddab8e94 fix(runner): give agent task and workspace reads the workspace page href (#21384)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People ask a chat agent where the files of a task are.
> - The answer must link the page of the workspace that holds the task's
files, when that page exists.
> - The agent reads the task with `get_task`. The task's
`currentExecutionWorkspace` had no `href`.
> - The chat directive tells the agent that a workspace without an
`href` has no page. So the agent said the page did not exist, although
isolated workspaces were on.
> - This pull request adds the workspace page `href` to the agent's task
and workspace reads, as `get_workspace` already does.
> - The benefit is that the agent links the real workspace page and does
not make a false claim.

## Linked Issues or Issue Description

**What happened?**
A person asked the CEO chat agent: "Where are the files for the website
task actually being worked on?" The agent named the task workspace and
linked the file. It did not link the workspace page. It also said "the
workspace has no separate page on this instance". The instance had
isolated workspaces on, so the page existed at
`/<prefix>/execution-workspaces/<id>`. The chat eval
`chat-features.runner-opencode.local.workspaces.locate-files` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed its
`bound-workspace-page-link` check in four cohorts and in one more paid
run for this reason.

**Expected behavior**
The agent reads the task, gets the workspace page link from the tool
result, and links that page in its reply.

**Steps to reproduce**

1. Turn on isolated workspaces (`enableIsolatedWorkspaces`).
2. Let an agent complete a projectless task, so the task has a bound
execution workspace.
3. In a different chat, ask the CEO where the files of that task are.
4. The agent calls `get_task` for the task.
`data.currentExecutionWorkspace` has no `href`. The agent reply names
the workspace and says it has no page.

**Additional context**
`get_workspace` and `list_workspaces` add the page `href` only for the
current task. The agent reads another task with `get_task` (`GET
/api/issues/{id}`) or with `call_api`. Those results did not get the
`href`. The run transcript shows the agent's reasoning: "The task API
includes currentExecutionWorkspace with no href", followed by the
directive rule "a workspace without an href has no page on this
instance".

## What Changed

- `server/src/services/task-workspace-tools.ts`: add
`withWorkspacePageHref` and `apiResultMayHaveWorkspace`. They add the
page `href` to the workspace in successful agent reads of `GET
/api/issues/{id}` (`currentExecutionWorkspace`), `GET
/api/issues/{id}/workspace` (`workspace`) and `GET
/api/execution-workspaces/{id}`. The function uses the same `href`
builder as `get_workspace`. It adds no `href` when the page is off, when
the read failed, or when the workspace is in a different company.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
apply the function to the results of the task bridge tools (`get_task`)
and `call_api`. The tool authority reads the page setting only for these
three operations.
- Tests: unit tests for the function, and an integration test through
the real tool authority and API. The integration test checks `get_task`
and `call_api` with workspace pages on (the `href` is present) and off
(no `href`).

## Verification

- Red/green: with the base tool authority, the new integration test
fails (`undefined` instead of `/<prefix>/execution-workspaces/<id>` for
both reads). With the fix, it passes. The unit tests fail on the base (4
failures) and pass with the fix.
- `cd server && npx vitest run src/__tests__/chat-project-tools.test.ts
src/__tests__/task-workspace-tools.test.ts
src/__tests__/agent-conversations.test.ts`: 110 passed.
- `cd server && npx tsc --noEmit -p .`: passes.
- Latency: the change runs only when the agent calls one of the three
reads. It does not touch run boot or time to first token. Local
`get_task` timing through the tool authority, 40 calls per sample, pages
on: base p50 51 ms and 74 ms; fix p50 49 ms and 54 ms. The difference is
in the noise.
- Paid eval: one narrow run of
`chat-features.runner-opencode.local.workspaces.locate-files` on this
branch:
https://github.com/paperclipai/paperclip/actions/runs/43436040405
(result in a comment below).

## Risks

- Low risk. The change only adds an `href` field to three agent read
results. It does not change REST responses for the board UI.
- The `href` appears only while isolated workspaces are on, because the
page route exists only then. This is the same rule `get_workspace` uses.
- An agent can now link a workspace page that it read through
`get_task`. The page itself still applies normal board access checks.

## Model Used

- Claude, Anthropic, model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:16:44 -06:00
DottaandClaude Opus 5.5 6917d0568d fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:08:38 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00