Commit Graph
3197 Commits
Author SHA1 Message Date
Dotta ca7d42ba71 fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path

> - Paperclip runs agent work for a company.
> - A task drain holds new run starts while maintenance or idle handoff
is in progress.
> - A queued dispatcher can keep an unreconciled request and a 30-second
retry while the drain is active.
> - Ending the drain can resume domain work without refreshing that
dispatcher's saved deadline.
> - This change refreshes the pending queued owner when a drain ends or
its expiry is observed.
> - Normal admission checks and real retry deadlines still apply.

## Linked Issues or Issue Description

Related: #20640. That change refreshes deferred receipt workers. This
change refreshes the separate queued-run dispatcher.

**What happened?**

A queued-work notification received during a drain could leave a
30-second fallback in the dispatcher after the drain ended. The owner
stayed unreconciled after useful work had settled. This delayed its next
pass and the instance's idle state.

**Expected behavior**

Refresh an unreconciled queued owner when normal dispatch becomes
allowed. Keep other drain holds, admission rules and scheduled retry
deadlines.

**Steps to reproduce**

1. Let the normal dispatcher complete its first pass.
2. Start an idle task drain and commit a company-scoped queued-work
notification.
3. Stop the drain, or observe its expiry, before the saved fallback is
due.
4. Check that the scoped queued pass runs before that fallback. The two
regression cases fail on the unchanged owner and pass with the fix.

**Paperclip version or commit**

The main-specific patch is based on
`f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in
assembled sleep acceptance. The affected queued-dispatch, drain and
notification modules match main.

## What Changed

- Listen for normal drain changes and observed expiry in the queued-run
owner.
- Wake that owner only when dispatch is allowed and its reconciliation
is pending.
- Forward the deadline worker's optional stop callback after its stop
fence.
- Remove both subscriptions when the owner or coordinator stops.
- Add release, expiry, remaining-gate, shutdown, retry-deadline and idle
controls.

## Verification

Local checks used the integrated feature branch that exposed the issue.
The main-specific port preserves main's existing `canRun` logic and
excludes the private branch's extra cancellation hook.

- Both new release and expiry cases failed at the intended scoped-resume
assertion before the product change.
- Six dispatch cases pass with the normal package import.
- Fourteen existing deadline controls and nineteen queue-order controls
pass. Their inputs did not change during the test-import correction.
- The complete normal Server typecheck and its three affected-file
program checks pass.
- The first typecheck caught a test import outside the Server root. The
import now uses the existing public package wildcard. The corrected
six-case suite and normal typecheck pass.
- Normal Source SDK input verification passes.
- Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed:
54 passing checks, four expected skips and one neutral result; Snyk
passed. Greptile reviewed this exact head at 5/5 with no actionable
findings. All review threads were read; none remain open. A full local
repository build and suite were not run in this checkout.

The existing admission-drain contract in `doc/execution-semantics.md`
already requires prompt admission after drain release. This fix restores
that contract. No configuration or command documentation changes are
needed.

## Risks

Drain events can request an earlier pass for a pending owner. The normal
permission, reconciliation, coalescing and earliest-time checks remain
active. The scheduled retry owner has no new listener. Shutdown must
remove both listeners; the coordinator-stop control covers that path.

This PR does not claim deployed suspension or complete transport sleep
acceptance.

## Model Used

OpenAI `gpt-6.1-sol` assisted with implementation and independent
review. The Codex managing agent performed technical review and
verification. The host does not expose its exact runtime model ID or
context-window size. Tools used include source inspection, code edits
and local test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (not
required: the existing contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-13 22:45:26 -06:00
DottaandClaude Opus 5.5 9824277f65 perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Most runner events that are not text deltas
(`tool.execution.started/progressed/completed`, `item.completed`) wake
the chat progress projection
> - That wake is registered inside the append's transaction, so each
such event takes 4 statements (BEGIN, the run lock read, the append,
COMMIT), where an event without a wake takes 1 (#21511)
> - The chat progress projection rebuilds itself from the stored events;
the wake only tells it when to look
> - This pull request writes an event whose only wake is chat progress
with the one-statement append, and sends the wake after that statement
commits
> - The benefit is 1 statement instead of 4 for most non-delta runner
events, with response, reassessment, retention and recovery wakes
unchanged

## Linked Issues or Issue Description

Refs #21511 (one-statement append). Refs #21528 (lock read returns the
transaction id). Refs #21549 (load model at recorded rates).

**What existing behavior does this improve?**
`appendHeartbeatRunEvent` for native source events whose only wake is
chat progress.

**Subsystem affected**
Server run event ingest (`heartbeat-run-events.ts`), the chat progress
notification helper, and two additive exports in
`packages/db/src/work-signals.ts`.

**Current behavior**
A progress event opens a transaction, reads the run lock with
`runtime_mode`, the bound task and the transaction id, probes for a
replay, registers the chat-progress wake as a tracked intent, appends,
and commits.

**Proposed behavior**
When chat progress is the event's only wake, the append takes the
one-statement path of #21511. That statement also returns the run's
`runtime_mode` and bound task id from the row it locks, and, where
commit notifications are on, calls `pg_notify` inside the same statement
for runs that are native and bound to a task, so other server processes
get the wake at commit. After the statement returns, this process
publishes the wake as settled to its own listeners. A replay, a gap, a
changed run or any other fallback takes the existing transaction, which
registers the wake as before.

**Reason and benefit**
At recorded rates these events are about 61% of the non-delta events
(#21549), so this cuts about 3 statements per such event.

**Breaking changes**
None. No schema change.

## What Changed

- `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other
wake clear) also qualifies for the one-statement path.
- `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement
append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`,
rendered once) from the locked row, plus a conditional `pg_notify`.
- `chat-native-progress-notifications.ts`:
`chatNativeProgressCommitNotification` and
`publishChatNativeProgressSettled`.
- `work-signals.ts` (additive): `databaseWorkCommitNotification(owner,
topic)` returns the cross-process payload when notifications are on;
`publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled
wake to this process's listeners.

## Verification

- Red/green on embedded PostgreSQL. Each new test failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "appends an event whose
only wake is chat progress in one statement and wakes progress after it
commits": 1 statement (before: a transaction), the wake is published
once as settled with no intent, and a replay publishes nothing.
- "keeps the transaction for the last events of a turn and a run, and
one statement for progress-only events":
`tool.execution.started/progressed/completed` and a non-final
`item.completed` take one statement; a final `item.completed`,
`run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled`
and `turn.interrupted` keep the transaction.
- "sends the progress wake to other server processes with the one
statement's commit": with commit notifications on, another connection
that listens on the work channel receives exactly one chat-progress
notification, and a replay sends none.
- `chat-native-progress-work.test.ts` "wakes the projection from a
one-statement progress event after the statement commits": the real chat
progress worker publishes the progress for an event appended in one
statement.
- Mutation check: without the settled publish after the statement, that
test fails.
- Lost-wake repair (condition b): an event row stored without any wake
models a lost hint.
- "repairs a lost progress wake with the run's next progress event":
nothing is published for the stored event; the run's next progress event
publishes the progress.
- "repairs a lost progress wake with the startup reconcile": a worker
that starts after the stored event publishes it.
- The existing response-wake test now uses a final reply and still shows
the intent before the write, and the race test of #21528 uses a final
reply so it still covers the transaction path.
- Other focused suites, one file at a time: the port suite with
conformance (105), `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `heartbeat-native-restart-suspension` (115), `live-recovery-work`
(51), `work-signals` (25), atomic-append (31),
`chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run
typecheck` and `pnpm check:module-boundaries` pass.

- CPU per operation (a local script, not committed: 800
`tool.execution.progressed` appends to one native task run on embedded
PostgreSQL, one connection, three rounds, `origin/master` code against
this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms
after in the warm rounds; wall time per event 0.84–1.04 ms before and
0.52–0.53 ms after.
- Inside a caller's transaction: new test "keeps a progress-only event
inside a caller's transaction on the locked path, and settles its wake
only when that transaction ends". `publishDatabaseWorkSettled` publishes
at once only outside a transaction; inside one it waits for the
outermost commit and drops the wake on rollback (new
`work-signals.test.ts` test, which failed with the unguarded publish).
The one-statement path is only taken when the owner is not a transaction
(`!is(db, PgTransaction)`, #21511), so the two guards agree.

## Risks

- **Lost wake.** If the statement's outcome is unknown (the connection
drops after the server ran it), this process sends no wake for that
event. In a chat channel (Slack, Telegram) the progress line for that
task can then stay one step behind (for example it does not move from
"working" to the latest tool step) until the run's next runner event, a
run milestone, or a server restart. A turn normally ends with an event
that keeps the transaction
(`turn.completed/failed/cancelled/interrupted`), and the run's
completion notifies chat run milestones, which include chat progress
(`notifyChatRunMilestones` in `run-completion.ts` and
`native-run-finalizer.ts`). So progress cannot stay stale after a run
ends; in the worst case it lags one step for the rest of a turn. The
projection rebuilds from stored events, so nothing is lost or
duplicated.
- The chat-progress wake no longer publishes an intent before the write.
Intents let the coordinator fence a pass against an in-flight write; the
settled wake after the commit makes the projection run again after the
row is visible.
- `packages/db/src/work-signals.ts` gets two additive exports and no
change to existing behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:37:35 -06:00
DottaandClaude Opus 5.5 54c3616ad9 fix(recovery): ignore the non-assignee comment wake marker in the failed-preparation proof (#21550)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a Runner run fails in setup before any provider work, a
person's Retry must start a new run
> - Admission proves "no provider work" for a failed preparation by
checking that the run record holds only known, harmless fields
> - The run profile allowlist did not include
`aiConnectionNonAssigneeCommentWake`, a boolean that dispatch writes
when it admits a run from a non-assignee comment
> - So a run with that marker never passed the proof. Its Retry waited
on `previous_execution_unverified` forever, even after saved-work
recovery finished and the workspace was handed on unchanged
> - This pull request accepts the marker (boolean only) in the proof,
and in the related workspace-wait parent proof
> - The benefit is that these tasks get a working Retry, and a Retry
that was saved earlier starts by itself once the proof passes

## Linked Issues or Issue Description

Refs #21481, #21497.

**What happened?**
On a hosted staging instance, a Runner run failed in setup ("RPC call
environmentExecute timed out") while it prepared. Saved-work recovery
finished ("Saved work recovered."). The workspace was handed on
unchanged for this run. A person pressed Retry two times. Both Retries
waited on `previous_execution_unverified`, and no run started.

**Expected behavior**
A run that failed in setup before any provider work
(`executionRecovery.providerWorkStarted: false`, no process, no
execution events, stopped sandbox) gets a working Retry. A saved Retry
starts by itself when its gate clears.

**Steps to reproduce**
1. Let dispatch admit a Runner run from a non-assignee comment, so the
run profile has `aiConnectionNonAssigneeCommentWake: true`.
2. Make the run fail in setup during preparation.
3. Press Retry. See that the request waits on
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `e82c4f0a31`; still present on `master` at
`6b631dd477`.

## What Changed

- `server/src/services/cancelled-native-startup.ts`: the
failed-preparation proof accepts `aiConnectionNonAssigneeCommentWake`
when its value is a boolean. The workspace-wait parent proof ignores the
same key.
- `server/src/services/legacy-preparation-retry-hold.test.ts`:
staging-shape tests for a recovered setup failure (failed restore, a
workspace that still names its location, the marker, all together), and
a Retry that attaches to an earlier refused request and then starts.

## Verification

- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`: 12 passed. Without
the fix, the `comment_wake_marker`, `all` and `attached_after_refusal`
cases fail (red), and the other cases pass.
- Focused test file only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The marker only records why dispatch admitted the run. Every
other proof clause (no process, no execution events, no native result,
stopped leases, settled preparation evidence, workspace repair) is
unchanged. A non-boolean value still fails the proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:02:55 -06:00
DottaandClaude Opus 5.5 6b9e45787f fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, automatic recovery continues
the task, through the fresh-continuation fallback, in a new run that
starts from the saved work
> - A follow-up run (for example one started by a new comment or by
finished subtasks) can fail before any provider work and leave a
saved-work hold. The system then queues its own retry, and the dispatch
gate cancels that retry at claim with
`execution_reconciliation_required` because the hold is still in place
> - The fallback takes over the hold when that happens, but it continued
only "recent interruptions". That test required the failed run to be a
recovery run itself or to fail with a restart code. A first follow-up
run that failed with `adapter_failed` passed neither test, even though
the system had already decided to retry it. So the fallback answered "a
person must press Retry". The backlog pass used the same test, and the
assigned-work sweep never listed the task. The task stayed blocked with
no run, no pending path and no Retry button
> - This pull request treats a system retry refused at dispatch as proof
that the system continues the task. It also stops cancelled-at-dispatch
attempts from counting against the hourly budget, lets the sweep
continue a blocked task whose latest run was refused at dispatch, and
keeps recovery notices from asking for a Retry that the task does not
offer
> - The benefit is that such a task continues by itself in exactly one
fresh run from the saved work, whether the saved work was recovered
before or after the refusal, and a person's own failed Retry keeps
today's behaviour

## Linked Issues or Issue Description

Refs #21394, #21348, #21363, #21478

**What happened?**
A task's follow-up run, a legacy run that was not a recovery run, failed
before any provider work with `adapter_failed`. Its files could not be
restored, so it left a saved-work hold
(`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure:
restore_failed`). The system queued a retry of that run. The dispatch
gate cancelled the retry at claim with
`execution_reconciliation_required`, and the "older run hold" path
handed the hold to the fresh-continuation fallback
(`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback
found the refused run and asked `interruptionEligibility` whether the
interruption was recent. That predicate required
`continuesAutomaticallySql(source)` (a recovery run, or a restart,
deploy or bootstrap code), so it returned `person_owned` /
`retry_needed`. The stranded reconciler then blocked the task with
"Press Retry", but the saved-work hold reports that Retry is not
available. This happened twice in a local restart-chaos wave. In one
case the saved work was recovered two minutes before the claim. In the
other it was recovered thirty seconds after the claim.

**Expected behavior**
A task never rests blocked with no live run, no pending path and no
Retry. While the saved-work recovery runs, that recovery is the pending
path. When it finishes, one continuation starts from the saved work
through the fallback (isolation, the shared hourly cap). A run that was
cancelled at dispatch before it started does not use up the automatic
budget. A person's own failed Retry is still left for that person.

**Steps to reproduce**
1. Fail a follow-up run (wake reason `issue_children_completed`) before
provider work with `adapter_failed` and `workspaceRestoreFailure:
restore_failed`.
2. Queue the system's retry of that run (`retryOfRunId`, wake reason
`issue_continuation_needed`) and claim it. The dispatch gate cancels it
with `execution_reconciliation_required`.
3. Before this change, no continuation is recorded, the stranded
reconciler blocks the task, and nothing ever continues it.

**Paperclip version or commit**
master at 2756be2f32

## What Changed

- `fresh-continuation-fallback.ts`, `interruptionEligibility`: the
interruption counts as one the system continues when the failed run's
cause is system-owned and a later run that the dispatch gate refused
before it started is a system recovery run (`recoveryRunSql`, which
excludes a person's Retry). A failed run that was itself a person's
Retry (`task_retry` or `explicitUserContinuation`) never qualifies this
way. The 24-hour recency bound is unchanged. The claim-gate hand-off,
the backlog pass and its deadline query all use this one predicate.
- `recovery/continuation-ledger.ts`,
`recovery/task-automatic-attempts.ts`: one shared rule,
`startedOrPendingAutomaticRetry`. An automatic retry that the dispatch
gate cancelled before it started (`cancelled`, no `startedAt`,
`execution_reconciliation_required`) does not count toward the hourly
cap in either ledger.
- `recovery/assigned-work-sweep.ts`:
- When the latest run was refused at dispatch, the sweep continues from
the run it was refused for through the fallback. Before, it gave up.
- A blocked task with an active hold and a recent refused-at-dispatch
latest run is now listed as a suspect. The check hands it to the
fallback before it answers `left_to_hold_owner`. A pending recovery
becomes `waiting_on_recovery` with its recheck time.
- Holds that wait for a person's configuration or confirmation are still
excluded.
- Review follow-up (`recovery/task-automatic-attempts.ts`,
`recovery/assigned-work-sweep.ts`):
- The task ledger counts a sweep's fresh-continuation record (`trigger:
assigned_work`) itself. It no longer depends on a companion activity
entry, and an activity entry marked `continuedFromSavedWork` is not
counted a second time. Every sweep continuation counts once, whatever
path recorded it.
- When the sweep claims a continuation, it marks its claim entry
`continuedFromSavedWork` inside the transaction that records the fresh
continuation. A restart between the two writes can no longer leave one
attempt counted twice.
- The sweep's continuation behind a refused run spends the per-minute
repair budget (`ASSIGNED_WORK_REPAIRS_PER_MINUTE`). The excess is
deferred with a `retryAt`.
- `recovery/stranded-notice.ts`, `recovery/service.ts`: when the task's
execution blocker offers no Retry, the escalation notice and the
recovery action's next action leave out every Retry instruction. They
say that Paperclip is holding the task until its saved work is settled.
- No change to the dispatch gate (`heartbeat/queue.ts`, the restore-hold
predicate in `run-dispatch/adapters/postgres.ts`, or
`execution-blocker.ts`). The recorded continuation settles the hold, so
the delivered run is not refused again.

## Verification

Red before the change, green after (embedded Postgres, real services:
`terminalizeLegacyExecution`, the dispatch adapter's
`cancelStaleQueuedRun`, `reconcileStrandedAssignedIssues`,
`deliverReconciledExecutions`, `continueHeldExecutions`,
`recoveryService().assignedWork`):
- `server/src/services/fresh-continuation-fallback.test.ts`
- "continues a follow-up … when the saved work was recovered before the
system retry was refused at claim": red (`continuationDelivery:
"invalidated"`, no continuation). Green: the continuation is recorded at
the claim, and the stranded reconciler does not block the task. The
delivered run is queued and passes the claim gate (`not_stale`).
- "… recovered after the system retry was refused at claim": red (no
pending check). Green: the claim records `exact_recovery_pending` /
`saved_work_recovery_running`. When the recovery completes, the backlog
pass continues the task, and the delivered run passes the claim gate.
- "continues a follow-up … with its files in place, after its system
retry was refused at claim": red, then green. This is the control
without a restore failure.
- "still leaves a person's failed Retry of a follow-up for that person
…": stays `person_owned` / `retry_needed`, before and after.
- "counts only automatic retries that started, or still may, against the
task's hourly budget": new.
- `server/src/__tests__/failed-continuation-continues-again.test.ts`,
"the sweep continues a blocked task whose system retry was refused at
claim behind a recovered saved-work hold": red (the task was not a
suspect). Green: `repaired`, a pending continuation in the hold, the
task back in `todo`, and no wake enqueued. The next check reports a
pending intent.
- Same file, "a continuation the sweep records behind a refused run
counts once toward the task's hourly cap": red (5 counted after 5
attempts plus the new continuation), green (6, and the transient-retry
cap check refuses the next attempt).
- Same file, "continuations behind refused runs spend the sweep's
per-minute repair budget": 20 tasks. Red (20 continued in one pass),
green (10 continue, 10 are deferred with a `retryAt`, and the 10
continue in the next pass).
- Same file, "a sweep continuation interrupted right after it was
recorded still counts once": the continuation commits, then the
post-commit activity write fails (a simulated server stop). Red: 6
counted for 5 real attempts. Green: 5 counted, and the next automatic
retry is still allowed.
- `server/src/services/recovery/stranded-notice.test.ts`, "never tells a
person to press Retry when the task offers no Retry": new.

Local runs on this branch, one file at a time:
- `fresh-continuation-fallback.test.ts`: 77 passed. The suite's
`afterAll` teardown hit the 30 s hook timeout on a heavily loaded
machine.
- `failed-continuation-continues-again.test.ts`: 32 passed
- `assigned-work-invariant.test.ts`: 31 passed
- `stranded-notice.test.ts`: 33 passed
- CI on the first head (ce4431410b): every required check passed
- `server` `tsc --noEmit`: clean
- `pnpm verify --explain` selects most of the repository (2666 files),
so required CI supplies the broad coverage.

## Risks

- More tasks continue automatically. Each continuation goes through the
existing fallback. The 24-hour recency bound, the per-task hourly cap
(6), isolation of the next run, and every person-owned gate
(person-owned failure codes, a person's Stop, review, a person-owned
hold, a person's failed Retry) are unchanged.
- The hourly cap now ignores automatic retries that never started. A
task whose retries are refused at dispatch again and again is still
bounded: the refusal hands the hold to the fallback, and the fallback
records one continuation per source run.
- Known separate gap, not changed here: after an interrupted
preparation's holds are resolved (`settleInterruptedNativeBootstrap`),
`scheduleBoundedRetry` can report `exhausted` or `blocked`, and nothing
follows that up. The assigned-work sweep may now pick such a task up
through the refused-at-dispatch path, but this pull request does not
prove that.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:01:39 -06:00
Devin FoleyandPaperclip b7922570e1 fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.

Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.

Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on 843cd9945e. Greptile is 5/5 with no
unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 19:41:17 -08:00
DottaandClaude Fable 5.2 689f3b190e fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic recovery continues interrupted and failed tasks without a
person, with a per-task limit of 6 automatic attempts per hour
> - There is no company-level limit. When the sandbox provider refuses
to create sandboxes, every task fails the same way, and recovery keeps
creating new runs for each task
> - On a hosted instance, one hour of a provider outage made 510
automatic runs, 587 failures and 0 successes across 289 tasks. No task
had more than 4 runs, so the per-task limit did not engage. Each start
asked a refusing provider for a sandbox and used server memory, which
made the outage longer
> - Recovery must still continue every task by itself. The answer is to
wait while the infrastructure fails, not to give the task to a person
> - This pull request adds a company-scoped pause for automatic work: it
opens when recent automatic runs fail before the agent starts, the
recovery passes stop creating new automatic runs while it is open, and
it closes by itself when runs start normally again
> - The benefit is that an infrastructure outage no longer multiplies
itself through recovery, and recovery resumes with no person acting

## Linked Issues or Issue Description

**What happened?**
During a sandbox provider outage, automatic recovery created hundreds of
new runs per hour across hundreds of tasks. All of them failed before
the agent did any work. The per-task limit did not engage because no
single task reached it.

**Expected behavior**
When most recent automatic runs of a company fail before they start,
recovery waits instead of creating more runs. It tries again at a small
rate. It resumes at full rate when runs start normally again. A person's
work is never held.

**Steps to reproduce**
1. Make the sandbox provider refuse lease creation (HTTP 503) for a
company with many assigned tasks.
2. Let recovery continue the failed tasks.
3. Count automatic runs created per hour: it grows with the number of
tasks, and every run fails at sandbox creation.

Refs #21394 (continuation classification this change will reuse when it
lands), #21433 (start-pass rewrite that the start hook will build on).

## What Changed

- New table `automatic_run_brakes` (one row per company, cascade delete
with the company; migration `0480`). Migration `0481` builds
`heartbeat_runs_company_finished_idx (company_id, finished_at desc)
where finished_at is not null` concurrently, so the signal reads the
runs that finished in the window directly. It stores the state (closed,
open, half_open), the reason with the window numbers, and the
compare-and-set fields that two servers share. A company with no row is
closed.
- New module `server/src/services/recovery/automatic-run-brake.ts`:
- Signal: on the database clock, the automatic runs of the company that
finished in the last 10 minutes (and since the last state change),
newest 500 by finish time, plus up to 200 running runs. A run counts as
a failure only when it ended in its own start phase: no provider work
(`executionRecovery.providerWorkStarted = false` or the native
`nativePreDispatchFailure` record), a legacy run still preparing, or a
transient setup code. A run interrupted by a server stop, restart or
lease loss is neutral, so a release swap does not open the pause. A run
that succeeded, or ended for a person-owned or task reason, counts as
healthy.
- Opens at 12 or more counted runs with 80% or more failures, or on 5
sandbox refusals (`environment_provider_unavailable`) in a row.
- Each outcome counts once, at its own time: a finished run at its
finish time, a running run that got past setup at its start time. Read
time never orders evidence.
- Closes in two steps: open moves to half_open after at least 2 minutes
when the newest 2 counted runs are healthy or the newest one succeeded;
half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64)
and then closes. It reopens from half_open if the signal trips again. It
closes by itself after 15 minutes with no automatic activity.
- Reads are one indexed query per company, cached for 5 seconds in a
bounded in-process map (256 companies). Each state change writes one
structured log line
(`automatic_run_brake.opened|half_open|closed|reopened`) with the
numbers.
- It also contains the start gate `holdAutomaticRunStart` (one probe at
a time, at most one new probe per 60 s, release budget per window by
compare-and-set, no I/O while the cached "held until" time has not
passed, person origin taken as an argument with a cached fallback). This
pull request does not call it from the start pass; see Risks.
- Recovery passes defer creating runs while the pause is open. The gate
is only where a run is created:
- `deliverReconciledExecutions`: after the agent-room check and right
before the wake, the delivery asks for a creation slot; when the pause
holds it, the continuation stays pending. The held backlog
(`continueHeldExecutions`) is not gated: recording a pending
continuation creates no run and costs nothing, and its delivery passes
this gate once.
- Assigned-work sweep `act()`: after the process rate limit and before
it claims a continuation, it asks for a creation slot; when held, it
returns `deferred_rate_limit` with the next try time, so nothing is
written to the per-task attempt ledger. New assignments with no earlier
run are not deferred.
- A reserved slot is given back (compare-and-set on its own timestamp)
when no run is created: the wake is refused or throws, the task is
capped, the claim changed, or the task went back to review. The
give-back is conditional in SQL on no run for that task having been
created since the reservation, so a wake that commits its run and then
throws keeps the slot.
- The sweep's overdue saved-work recovery path returns before the slot
is asked for: it starts a recovery attempt, not a run, and is never
held.
- Person work is never deferred. The gate is called only at the delivery
(`agentHasRoom`) and the sweep (`act()`), and each call passes the task
and the run it continues. While the pause is open, the gate does not
hold the record when any of these is true; it holds only when none is.
With no task or no source run it does not hold (fails open).

| Shape | How it is recognised |
| --- | --- |
| A Retry or "move forward" waiting for the task
(`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an
unstarted wake request from a user for the task |
| A Retry folded (`coalesced`) into the continuation, or a Retry while
the delivery is pending | a wake request from a user for the task,
requested after the source run finished (any status) |
| A person's message or a new assignment by a person after the last run
| the same: a wake request from a user for the task after the source run
finished |
| A held task selected because a person pressed Retry (the refused Retry
run) | a run for the task created after the source run finished whose
wake is a person reason (`task_retry`, chat message), or that has
`taskRetryRequestId` or `explicitUserContinuation` |
| A continuation of a person's Retry (any depth) | the source run's
chain reaches a person's run (`listPersonOriginSourceRunIds`, the
dispatch rule) |
| A Retry during pending delivery that becomes the continuation itself
(`deliverContinuationAsRun`) | not gated: that run is created by the
person's own wake, never by these passes |
| The held backlog's saved-input path | not gated: recording or
releasing a held continuation creates no run here |
- To prevent a deadlock, a pass may still create one automatic run per
minute per company when no automatic run of the company started in the
last 2 minutes, however many runs wait (compare-and-set, shared by
servers).
- `GET /api/companies/:id/assigned-work-invariant` now returns
`automaticWork` with the state, the time it paused, the window numbers
and a plain sentence, for example "Automatic work is paused for this
company since T because 47 of 50 recent automatic runs failed before
starting."
- `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is
now exported, unchanged, so the gate uses the same person-origin rule as
dispatch.
- The outcome rules (person-owned codes, restart codes, transient setup
codes, native pre-dispatch record) come from their owning modules
through one memoised dynamic import. A static import of
`heartbeat/retries.ts` made a module cycle at server start.

### What the pause consumes, and how waste is bounded

| Token | Taken | Given back or bounded |
| --- | --- | --- |
| Creation slot (one per minute per company while open) | right before
the run is created, after every other check | released only when no run
for the task was committed after the reservation |
| Probe slot (start gate) | when the start gate lets one automatic run
through | another run may take it when the reserved run has not started
within 10 seconds (claim declined, agent paused or full) or was
cancelled, deleted or finished |
| Gradual-release start (start gate) | one per run the start gate lets
through | a declined claim spends one start of that window; the window
ends within 60 seconds and the next one doubles |
| Probe success count | once per probe run, by compare-and-set on its id
| a probe run counts once; cancelled or deleted probes count as neither
success nor failure |

### When the pause opens, stays open and closes

| State | Leaves when | Bound |
| --- | --- | --- |
| closed | opens when, in the last 10 minutes since the last change, 12
or more counted automatic runs finished and 80% or more failed before
starting, or the newest 5 counted runs were sandbox refusals | none
needed |
| open | moves to half_open when it has been open 2 minutes or more and
the newest 2 counted runs are healthy, or the newest one succeeded |
evidence arrives at least once a minute: when no automatic run started
in the last 2 minutes, a pass may create one per minute |
| open | closes when no automatic run started or finished for 15 minutes
(queued runs do not count) | 15 minutes after the last activity |
| open | stays open only while new automatic runs keep failing before
starting | each new try is spaced by the existing passes and by one
creation per minute when nothing starts |
| half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6
minutes |
| half_open | reopens when the signal trips again, counted only since
half_open began | none needed |

Restart-class endings (server stop, restart, lease loss, process loss,
interrupted native session) are neutral in every state, so a release
swap does not open the pause.

## Verification

- `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests,
pass): classification; opens on 20 failures before start; 50 queued
automatic runs give exactly one probe and no failure or attempt; a
person's Retry, a new assignment, a run a person joined and the
continuation of a person's Retry are not held; mixed and person-owned
outcomes stay closed; old data is ignored; 40 working runs interrupted
by a restart stay closed, then the same continuations failing at sandbox
acquisition open it; 5 refusals in a row open it; failed probe waits for
the interval, two probes past setup move to half_open, the release gives
exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen
from half_open; probe slot frees after cancel, delete or no start; quiet
close after 15 minutes; two instances on one database agree and start
one probe; one creation per minute while open across servers; 250 held
runs over 100 passes use at most 5 queries (at most 255 with the
person-origin fallback, one per run, once); caches stay at 256 companies
and 1024 runs.
- `npx vitest run
src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one
test per person shape above at the delivery and the sweep, with the
pause open and no creation slot, plus a control that still holds a
continuation no person asked for): includes "no pause record: every pass
behaves as before", "the waiting run cannot start: a pass continues a
task on another agent, then the pause closes", "one held task spends one
creation slot", and "a person's Retry or message is never paused at the
delivery, the sweep or the held backlog".: sweep, delivery and held
backlog create nothing while open, write no attempts, and continue by
themselves after recovery; the sweep still continues one task a minute
when nothing automatic waits. With the three deferral lines removed, all
4 fail (red), and they pass with them (green).
- The pause reads as closed and logs once when its run rules cannot load
(test in `automatic-run-brake.test.ts`).
- `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the
module loads after the heartbeat service, in the server's import order.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify`: see the comment below for the result. CI supplies the
full suite.

## Risks

- This pull request does not hold starts. Already queued runs and
`transient_failure_retry` runs still start while the pause is open; they
act as the natural tries. The start gate goes in as a follow-up on top
of #21433, at a point before that pass counts a claim attempt and reads
the full run row, so a held run costs no attempt and no read.
- The sweep reuses the `deferred_rate_limit` outcome for a deferral, to
keep its outcome type unchanged while #21394 edits that file.
- The automatic-run predicate is built from master exports
(`RECOVERY_WAKE_REASONS`, retry markers). It will switch to
`isAutomaticContinuationRun` when #21394 lands.
- Migration `0480` adds one small table.
- Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`)
runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement:
it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in
`packages/db/src/client.ts`, like 0470 and 0476, and the runner drops
and rebuilds an invalid index after an interrupted build. The index is
declared in the Drizzle schema, so `pnpm db:generate` reports no
changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On
a large `heartbeat_runs` table the build is online (no write lock) but
takes time proportional to the table. After it exists, it is one more
index on a hot table: each run that finishes writes one more index
entry.
- No user-visible text changes outside the operator report.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell,
file edits, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 21:20:09 -06:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 6b631dd477 fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages

## Linked Issues or Issue Description

No public issue exists. This describes the bug in the format of the bug
report template.

**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.

**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.

**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.

**Paperclip version or commit**
`master` at 0a96f972e1.

**Database mode**
Hosted PostgreSQL. Reproduced on embedded PostgreSQL 18.

**Additional context**
The 138 second session was not a transaction held open across remote
work. The call path is `registerOwnershipReconciliationWork` →
`runOwnershipReconciliationPass(db)` → `kind.listInProgress(db, 200,
offset)` on the pool, with no transaction. Only the `leaseKind` and
`operationKind` listings send this exact unqualified column list. Of
these, only this kind's filter reads `heartbeat_runs`.

## What Changed

- `releasableRetainedOrphanCondition` and `nextRetainedOrphanDueAt`
first apply `retainedOrphanCandidateCondition`. This is a filter with no
subqueries and no NULL results: cleanup status, run, provider, lease
status, lease policy, and the metadata keys of the three retained orphan
shapes. Expired, active and pending-cleanup leases fail it at once.
- The held checks (always held and softly held) now run inside `CASE
WHEN <retained orphan condition> THEN ... ELSE false END`. PostgreSQL
evaluates a CASE in order, so the correlated subqueries run only for
candidates. The planner can no longer split the `NOT (a OR b OR ...)`
into separate top-level subplans and run them before the candidate
predicates.
- The two `heartbeat_runs` checks of the always-held part are now one
scalar subquery with a primary key lookup. It reads
`runner_profile_json->'nativeWorkspaceSync'` once through a fenced
lateral, instead of about 12 `#>>` path reads. A scalar subquery cannot
become a hashed subplan, so the planner cannot replace it with a full
scan of `heartbeat_runs`.
- The listed rows, the order and the paging do not change.
`softlyHeldSql` is now exported so that the test can rebuild the
previous condition.
- The paging stays OFFSET-based. The `listInProgress(db, limit, offset)`
contract is shared by all ownership kinds and by the I2 rule, which
compares consecutive passes. Keyset paging is a change to that shared
contract, so this pull request does not make it.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-retained-orphan-listing-cost.test.ts
src/__tests__/environment-retained-orphan-sandbox.test.ts
src/__tests__/abandoned-workspace-files-hold.test.ts`: 3 files, 55 tests
pass.
- New test `environment-retained-orphan-listing-cost.test.ts`:
- Equivalence: four seeded random fixtures of 600 leases each, across
two companies. The fixtures cover every lease status and policy, runs of
each status and runtime mode, valid and invalid native workspace
descriptors, every always-held metadata key, sibling leases on the same
sandbox, open and resolved recovery actions, open runs on the issue, and
past, future and absent expiry. Every page (limit 7 and 200) equals the
previous SQL, which is kept only in the test. `nextRetainedOrphanDueAt`
also equals the previous query.
- Red/green: over 2,000 history leases with the hosted status mix (71%
expired, 25% released, 4% failed), the test counts the `heartbeat_runs`
rows that the listing reads, from `EXPLAIN (ANALYZE, FORMAT JSON)`. On
`master` it reads 1,760 rows for 1 candidate, so the test fails. With
this change the count is within the bound of 10 per candidate.
- Incident regression: while listings run in a loop over 5,000 history
leases, `ALTER TABLE heartbeat_runs ADD COLUMN ...` and `DROP COLUMN`
with `lock_timeout = '5s'` complete in less than 5 seconds.
- Local measurement (embedded PostgreSQL 18, Apple M-series, warm
cache), not in CI:
- 18,640 leases with the hosted status mix and a 19 KB runner profile
per run. `master`: 422 ms, 296,771 buffers (12,357 read). This change:
26 ms, 36,004 buffers. The `heartbeat_runs` subplans run 13 times (once
for each candidate), not once for each lease.
- 300,000 history leases. `master`: one listing page takes 2,822 ms, and
the ALTER under load waits 2,448 ms. This change: 212 ms, and the ALTER
takes 96 ms.
- `pnpm check:module-boundaries` passes.

## Risks

- Low risk. The listing and the due-time query return the same rows, and
the equivalence test compares them. The settle is unchanged and still
re-checks every hold under the row lock.
- The candidate filter must stay a superset of
`retainedOrphanSandboxCondition`. If a later change adds a retained
orphan shape and does not update the candidate filter, the listing skips
that shape. The equivalence test catches this only for the shapes in the
fixture.
- The listing still reads each lease row once with cheap predicates. A
partial index on the candidate filter could make it scale only with
candidates. This pull request does not add a migration.
- The ownership pass still lists all kinds every 30 seconds when its
fingerprint changes. This pull request does not change that cadence.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edit, embedded PostgreSQL runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:21 -06:00
DottaandClaude Opus 5.5 1dee321844 perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every native event that announces work to another part of the server
(chat progress, a final response, an evidence outcome, recovery)
registers a wake in its transaction before it writes the event
> - Such an event takes six statements: BEGIN, the run lock read, a
replay probe, a read of the transaction id for the wake, the append, and
COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock,
the transaction id, the write, COMMIT
> - In the agent scaling load model these two paths are now the largest
per-agent cost after #21511 and #21520, and other events fall behind
first at 600 agents
> - This pull request returns the transaction id from the lock read that
each path already sends, and moves the replay probe into the run lock
read when it is safe
> - The benefit is four statements for such an event and four for a
checkpoint save, with the same locks, the same wake order and the same
replay rule

## Linked Issues or Issue Description

Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent
scaling plan and load model).

**What existing behavior does this improve?**
The statement count of native events that register a wake
(`appendNativeSourceEvent`) and of controller checkpoint saves
(`nativeRunnerCheckpointWriter`).

**Subsystem affected**
`packages/db/src/work-signals.ts` (one new export),
`server/src/services/heartbeat-run-events.ts`,
`server/src/services/native-runtime/native-runner-checkpoint.ts`.

**Current behavior**
`signalDatabaseWork` sends `select pg_current_xact_id()` for the first
wake of each transaction. A native event with a wake reads the run lock,
then probes for its replay in a separate statement, because a statement
that waits for the lock keeps its earlier snapshot.

**Proposed behavior**
- The run lock read and the checkpoint owner lock read also return
`pg_current_xact_id()::text`. Both already lock a row, so the
transaction has an id at that point. `primeDatabaseWorkTransactionId`
gives it to the transaction's wake state, and the wake sends no
statement for it.
- For an event that may register a progress, response, reassessment or
retention wake, the run lock read also reads `next_event_seq` from its
own snapshot and checks for a replay in that snapshot. Every event
append advances `next_event_seq` in its transaction. So when the locked
row has the same `next_event_seq` as the snapshot, no append committed
in between, and the probe result is current. Otherwise, or when the
probe finds a replay, the separate probe runs as before.

**Reason and benefit**
Fewer round trips while a pool connection is held in a transaction. In
the load model these paths hold the main pool and the three-connection
checkpoint pool longest per event.

**Breaking changes**
None. No schema change.

## What Changed

- `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`,
exported from `@paperclipai/db`): inside a transaction whose id is not
yet known, it records the id. Outside a transaction it does nothing. An
uncertain commit still resolves with `pg_xact_status` on the same id.
- `appendNativeSourceEvent`: the lock read returns the transaction id
when the event has any wake. For a probing wake it is one statement: a
CTE that reads `next_event_seq` from the snapshot, a CTE that locks the
run row (`for no key update`, unchanged), and `exists (...)` for the
replay. The wake is still registered after the replay check and before
the append.
- `lockNativeRunnerCheckpointOwner` returns the transaction id with its
owner lock and primes the wake state. Its return value and every caller
are unchanged.

## Verification

- Red/green on embedded PostgreSQL; each new assertion failed against
`origin/master` first:
- `heartbeat-run-event-atomic-append.test.ts` "registers a native
event's wakes before its write and only when the event is new": four
statements (before: six), the wake's intent and settlement are
published, and a replay publishes no wake.
- New: "finds a replay that another writer committed while the lock read
waited, and registers no wake for it". Another connection holds the run
row (the test waits until PostgreSQL reports the append waiting for that
lock), stores the same event and moves `next_event_seq`; the waiting
append returns `duplicate`, runs the separate probe, and publishes no
wake. With the `next_event_seq` comparison removed, this test fails.
- New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the
owner lock with the transaction id, the write, COMMIT (before: five
statements), and its recovery wake is published; a save after the lease
expired writes nothing and fails with
`native_runner_checkpoint_owner_lost`.
- `work-signals.test.ts`: a primed transaction sends no id statement for
its wakes; a primed transaction whose connection is killed after its
write stays pending and then resolves through `reconcileDatabaseWork`,
with no row stored.
- Added after merging `origin/master` (with #21511, #21520 and #21436)
into this branch:
- "gives the wake the transaction id of the transaction that took the
run lock": inside a caller's transaction, an event with a progress wake
sends no separate `pg_current_xact_id()` statement; the transaction's
connection is then killed, and `reconcileDatabaseWork` checks
`pg_xact_status` with exactly the id that `pg_current_xact_id()` returns
on that same transaction. Priming a wrong id makes it fail.
- "primes nothing when the lock read fails, so the transaction is not
left uncertain": a lock read that fails on `lock_timeout` leaves no
pending work; after the same failure inside a savepoint, the next signal
on that transaction still sends its own `pg_current_xact_id()`. Priming
before the lock read makes it fail.
- Both call sites pass only the value returned by the statement they
just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)`
then `primeDatabaseWorkTransactionId(tx, run.xid)`) and
`lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then
`primeDatabaseWorkTransactionId(tx, xid)`).
- Retest on the combined tree, one file at a time: atomic-append 28
(including the equivalence test and both race tests), the port suite
with conformance 105, checkpoint-save 2, `live-recovery-work` 51,
`work-signals` 25, runner `native-session-runtime` 297; server typecheck
passes. No code change was needed for the combination.
- Other focused suites pass, one file at a time: the port suite with its
conformance suite (100), `chat-native-progress-work` (35),
`native-observer-replay` (1), `heartbeat-run-event-sequencing` (5),
`native-event-boundary-golden` (1), `native-recovered-owner-followup`
(97), `native-restart-recovery-isolation` (1),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51),
`work-signals` (25).
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). Stacked: this pull request merged locally with #21511, and the
model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model
applies the port's batching rules itself; that mode is not in any pull
request). Default pool of 10 connections; embedded PostgreSQL on the
same, shared machine; runs one after the other with a pause between
them. No run had a failed operation, a failed checkpoint save or a lost
lease. "Before" rows are `origin/master` from the same session.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 /
12,000 | 1 / 70 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 /
24,000 | 2 / 1,274 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 /
6,335 | 61,158 / 72,000 | 11,678 / 21,212 |
| 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 /
72,000 | 1,136 / 4,980 |
| 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 |
71,958 / 72,000 | 1,319 / 5,712 |
| 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 /
48,000 | 1,974 / 3,835 |

With the three changes, one process finishes 99.9% of the offered work
at 600 agents, but its event-loop utilization is 0.90–0.93, above the
0.75 target, and p99 lateness is about 5 s. It saturates between 600 and
800 agents.
- Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5
where commit-time notifications are on).

## Risks

- The transaction id is read in a statement that already assigns one (a
row lock), so no transaction gets an id it did not have before.
- The probe in the lock read is used only when `next_event_seq` did not
change between the snapshot and the lock. Every writer of
`heartbeat_run_events` moves `next_event_seq` in the same transaction
(native and ordinary appends, the run coordinator store, the retry
queue). A future writer that inserts events without it would make this
probe miss a concurrent replay; the append statement then still finds
the replay and stores nothing, but the wake would already be registered
(a spurious wake, not a lost or duplicate event).
- This pull request and #21511 both edit `heartbeat-run-events.ts`, in
different functions.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:48:17 -06:00
Devin FoleyandPaperclip dc018f9bc0 Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization.

Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation.

Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 18:45:20 -08:00
DottaandClaude Opus 5.5 833689a3fc fix(queue): say a long-queued turn is waiting to start, and name the routine execution that holds it (#21447)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21405 made a task that waits for its agent's concurrency limit show
"Waiting for <agent> · N slots max · P of Q in line · Configure"
> - On hosted staging, after that deploy, a routine task stayed queued
for 8 minutes while its agent had 19 free slots. Its chat said "Working
for 7m · Thinking", and the server label was a plain "Queued"
> - The real cause was the routine's one-execution-at-a-time rule: the
previous execution of the same routine was still open with its run. That
is correct scheduling, but the UI did not say it
> - A queued turn also kept the startup feedback ("Working") for as long
as it stayed queued, with no limit
> - This pull request names the routine hold and limits the startup
feedback to 30 seconds
> - The benefit is that a queued task always says that it waits, and why
when the server knows

## Linked Issues or Issue Description

Refs #21405

**What happened?**

On staging, one routine task's continuation run was queued at 17:39:03Z
and claimed at 17:47:10Z. The agent ran 1 of 20 runs. The chat showed
"Working for 7m · Thinking". The run's projection label was "Queued"
with no cause.

**Why did it happen?**

1. The routine allows one open execution with a run at a time (unique
index `issues_open_routine_execution_uq`). The previous execution of the
same routine was still open and held its run until 17:47:10.172Z. The
waiting run was claimed 345 ms after that task closed. The claim was
refused with "Another execution for this routine is already in progress"
until then. This is the routine's intended policy, not a scheduling bug.
2. The execution projection did not name that hold, so the server label
was "Queued".
3. The task chat keeps the startup feedback ("Working" and "Thinking")
for a queued turn with no named hold (a DESIGN.md rule). That rule had
no time limit.

**Expected behavior**

A routine task that waits for the routine's open execution says "Waiting
for <task> to finish". A queued turn that has no named hold says
"Waiting to start" after a short grace period.

## What Changed

- Server: `executionProjectionsForRuns` reads, in one query per batch,
whether another open task of the same routine holds the routine's
execution. For a never-started queued run that waits on it, the
projection sets the cause `routine_execution_in_progress`, the label
"Waiting for <task identifier> to finish", and a next action. This check
comes before the agent-limit check, because a free slot would not start
the run.
- UI: a new hook, `useWaitingToStart`, keeps the startup feedback for 30
seconds. After that, a queued turn with no named hold shows "Waiting to
start": an hourglass, no spinner, no shimmer. The task chat's runner
header, its current-activity tail and the live run pill use it. The hook
uses one timer at the threshold and no per-second ticks.
- DESIGN.md: the startup-feedback rule now has the 30-second limit and
the routine hold.
- Storybook: `WaitingToStart` and `WaitingForRoutine` stories, with play
checks.

**Why 30 seconds:** DESIGN.md keeps "Working" from message submission
through admission, so that a normal start does not flicker. On staging,
runs that were not held started 3 to 8 seconds after they were queued
(from the company's run list between 17:30 and 17:48Z). 30 seconds is
several times that, and shorter than any real wait the user should be
told about.

## Verification

- Red/green server test (embedded Postgres),
`execution-projection-agent-queue.test.ts`: a routine task waits while
another open task of the same routine holds a run. The test checks the
label "Waiting for Q-3093 to finish" and the cause. It checks that the
holder is not marked as waiting on itself, and that the label returns to
plain "Queued" when the holder closes. It fails without the server
change.
- Red/green UI tests: `TaskChatRunnerTurn.test.tsx` (startup feedback
until the threshold, then "Waiting to start" in the header and the tail;
a running turn is never called waiting) and
`TaskChatLiveRunPill.test.tsx` (the label changes, with no spinner or
shimmer). Both fail without the UI change.
- `vitest run` on the four `execution-projection-*` server suites (36
tests) and all `task-chat` and `TaskChatThread` UI suites (1,414 tests).
UI and server typecheck.
- The new stories render in headless Chromium.
- On staging, before this change, the #21405 waiting line was confirmed
with a temporary 1-slot agent: "Waiting for Queue Check (temp) · 1 slot
max · 2 of 2 in line · Configure". The temporary tasks and agent were
cancelled and terminated.
- After deploy: confirm the new labels on staging.

## Risks

- Low risk. There is no schema change.
- One extra query per projection batch that includes a never-started
queued run on a routine task. It is a join on the company's
routine-execution tasks through the existing `issues_company_origin_idx`
index.
- A queued turn that is admitted after 30 seconds, for a reason the
server does not name, now says "Waiting to start" instead of "Working".
This is the intended change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, Chrome for staging checks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:22:07 -06:00
DottaandClaude Opus 5.5 a82e235828 fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs when a slot is
free. #21234, #21277, #21349 and #21391 order them: person work first, a
recovery share, aging, and a share for the longest-waiting run
> - On hosted staging, one agent with 20 slots had 295 queued runs. 139
of them were system recovery wakes that entered the queue within one
hour. Runs that people asked for waited 7 hours
> - Two rules let this happen. Recovery took every fourth start even
when a person's run waited, and an aged recovery run moved into the
agent-work tier ahead of todo tasks. The continuation producer also
admitted any continuation held past 5 minutes, however deep the queue
was
> - This pull request keeps recovery behind people's work, gives a
person's long-waiting run a bounded start, re-checks a recovery run
before it starts, and admits recovery to an agent's queue a few runs at
a time
> - The benefit is that a task a person asked for starts within a
bounded number of dispatches, and a recovery burst cannot flood an
agent's queue

## Linked Issues or Issue Description

Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502

**What happened?**

On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and
all 20 slots held hour-long runs. 295 runs were queued. 139 had wake
reason `issue_recovery_action_restored` (source `execution.reconciled`),
enqueued between about 10:46Z and 11:47Z. Two tasks that people created
were stalled by a restart at 07:25Z. Their next runs were queued at
07:29Z and were still queued 7 hours later. Both were system wakes: one
`issue_monitor_due`, one `issue_continuation_needed` from the
assigned-work sweep. Recent starts had waited from 1 to 290 minutes.

**Why did it happen?**

1. The recovery share gave every fourth start to system recovery, also
while a person's run waited.
2. A recovery run that waited one hour aged into the agent-work tier.
Recovery tasks are usually in progress, so they sorted ahead of todo
tasks in that tier.
3. The oldest-run share served the oldest aged run of any class. Older
agent work and older recovery came before a person's run.
4. Only a wake that a person requested counted as person work. A system
wake on a task that a person created and waits on (a monitor check, a
continuation after a restart) competed as ordinary work or recovery.
5. `deliverReconciledExecutions` admitted a continuation held past
`RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue
depth. At one per agent per 15-second pass, one agent got about 139
continuations in an hour.

**Expected behavior**

A run that a person created or asked for never waits behind system
recovery. Waiting never fails or cancels it. A long wait moves it
forward, so it starts within a bounded number of dispatches. Recovery
enters the queue a few runs at a time. A recovery run that is no longer
needed is dropped quietly.

**Steps to reproduce**

1. Give an agent 20 slots, all busy.
2. Queue 60 agent assignments, then two system wakes on tasks people
created (a monitor check and a continuation), then 30 later person runs
on in-progress tasks, then a burst of 139 recovery wakes.
3. Free one slot at a time while people and agents keep adding work.
4. See that the monitor check starts only at dispatch 244.

## What Changed

- `server/src/services/heartbeat/queued-run-dispatch-order.ts`:
- System recovery has its own tiers below agent work (fresh, then
backlog). Aging moves old backlog only into the fresh recovery tier,
never ahead of agent work.
- The recovery share applies only when no ready person run waits. It
comes out of agent work, never out of a person's turn. Within the share,
the longest-waiting aged recovery run goes first.
- The oldest-run share serves the longest-waiting aged run that a person
asked for or whose task a person created (`issues.created_by_user_id`),
first. This includes a system continuation of such a task. Then it
serves the longest-waiting aged agent-work run. It never serves other
recovery.
- New dispatch fact `personTask`, read with the queued tasks' status and
priority (same query).
- `queuedRunDispatchLine` (the O(n log n) place-in-line order from
#21405) follows the same rules. `queued-run-line.test.ts` still proves
it equals the dispatcher's order over 400 random queues, now with
`personTask` in them.
- `server/src/services/heartbeat/queued-run-line.ts`: the start-history
counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so
dispatch, the place-in-line view and the tests share one rule.
- New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and
`dispatchCountersFromStarts`. The queue and the tests use the same
counter rule (which includes the #21391 rule that a person's recovery
restart does not use up the recovery share).
- `server/src/services/heartbeat/queue.ts`:
- At claim, a system recovery run (`issue_recovery_action_restored`,
`issue_continuation_needed`, not requested by a person) is re-checked.
If the task is reassigned, or another run continues it (started after it
was queued, running, a queued non-recovery run, or a newer queued
recovery run), the run is cancelled before it starts with
`recovery_no_longer_needed` and its wake is skipped. This is the same
quiet path as the existing closed-task check: no failure, no provider
work. A person's undelivered message on the run still keeps it.
- `server/src/services/recovery-queue-admission.ts` (new):
`QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check.
- `server/src/services/execution-recovery-resolution.ts`: a continuation
is admitted only while its agent has fewer than 3 queued recovery runs,
fresh or backlog. The 5-minute hold bound still orders held
continuations, but no longer overrides the cap. The rest stays pending
in its recovery action. The existing continuation pass (every 15 s while
any continuation is pending) admits the next one after a queued recovery
run starts or is dropped. No new polling.
- `server/src/services/recovery/legacy-workspace-rehome.ts`: the
pending-wake delivery for rehomed and released-writer tasks uses the
same cap. A held wake waits 60 s for its next turn and does not count as
a refusal.
- The oldest-run share counts as served only by a start that people wait
on while its candidate is one, so older agent work never uses up a
person's turn (`servesOldestShare`, used by the dispatcher, the
place-in-line order and the start history, which now records
`personWaiting`).
- `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or
released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts
as the task's pending path while its assignee has queued recovery runs.
- `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's
stalled-queued-run replacement): a start of work people wait on does not
count as passing over a queued run that people do not wait on. Waiting
behind people's work is intended, so the sweep does not cancel such a
run and replace it with a recovery continuation. Test:
`assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run
that waits behind work on tasks people created". It fails without the
rule.
- `server/src/services/recovery/assigned-work-sweep.ts`: a pending
continuation still counts as the task's wake path past
`PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery
runs, so the sweep does not treat a held task as stranded.

**Bounds**

- A run a person asked for, or any run on a task a person created: once
it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within
`OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1)
dispatches. Before that, runs a person requested already go first. There
is no time limit, and waiting never cancels it.
- Recovery run, proposed bounded behavior: at most 3 queued per agent.
When no person run waits, an aged recovery run starts within
`RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1)
dispatches. At start it is re-validated against the current task, and it
is dropped quietly if stale. While people keep the agent busy, recovery
waits. It is held durably in its recovery action, not lost and not
failed.

## Verification

Regression test that reproduces today's shape
(`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`,
"deep per-agent queue: a recovery burst behind 20 busy slots"). It uses
20 busy slots, one dispatch per freed slot, 60 older agent assignments,
the two 07:29 system wakes on tasks people created (monitor check and
continuation), 30 later person runs, a 139-run recovery burst, and new
person and assignment arrivals on every step. The same model, run
against master's dispatcher and this one over 600 dispatches:

| | Before (master) | After |
| --- | --- | --- |
| Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20
starts/h) | 4 |
| Dispatch at which the 07:29 continuation starts | 5 | 8 |
| Recovery starts while a person-requested run waited | 89 | 0 |

With person-requested runs in the 07:29 slots instead (the first version
of the test), master starts them at dispatch 244 and 248, and this
change at 4 and 8.

- Red on master: the new dispatch tests fail (`expected -1 to be greater
than or equal to 0`, and recovery started ahead of agent assignments).
Green with this change.
- Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`,
"queues at most a few recovery runs per agent..."): with the cap
removed, 12 of 12 continuations enter one agent's queue in 12 passes.
With the cap, 3 enter, the rest stay pending, and all 12 are delivered
as queued runs leave. The queue never holds more than 3 recovery runs.
- Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older,
higher-priority assignments and no person arrivals, an aged
`issue_monitor_due` run on a person-created task starts within
`OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With
the old reset rule it does not start in 8 dispatches.
- Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the
agent's recovery queue full, the wake stays pending with no failure
counted, the sweep reads the task as `pending_intent`, and the wake is
delivered after a queued recovery run leaves.
- Start-time re-check, embedded Postgres, real heartbeat queue
(`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose
task was reassigned, or that a newer run already continued, is cancelled
with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter
is never called. A recovery run that the task still needs starts.
- Updated tests that encoded the old rules: recovery's share and aging
now come out of agent work, not out of a person's turn. Sweep liveness
while held is covered.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes.

## Risks

- Behavior change: a system run on a task a person created now takes the
oldest-run share ahead of older agent work. On an agent whose queue is
mostly such tasks, agent-made subtasks get that share less often. They
still start in normal order.
- Behavior change: while people keep an agent fully busy, system
recovery for that agent waits. This is intended. Recovery is held
durably and starts once no person run waits.
- Behavior change: a queued system recovery run can be cancelled at
start as no longer needed. It is cancelled before any provider work,
with the same quiet outcome as a continuation whose task closed.
- A continuation held behind the cap stays pending longer than 7
minutes. The sweep now treats it as live while its agent has queued
recovery runs. If such queued runs can never start (for example
dependency-blocked), the next continuations wait behind them.
- One extra indexed read per recovery claim, on the existing
`(company_id, context_snapshot->>'issueId', created_at)` index. No
schema change. Mixed versions are safe: an old server keeps its old
order until it is replaced.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:21:35 -06:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
DottaandClaude Opus 5.5 1f388ed385 Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm sandbox keeps an agent's Runner alive between runs, so the
next run on the task starts fast
> - When a run ends, the server closes the idle warm Runner in-process.
A separate SQL worker stops the sandbox when its warm window ends
(#21410, merged today)
> - Both steps start at the same moment. The sandbox stop wins, so the
idle close fails. The failed owner stays registered, and the next run
throws `native_instruction_materialization_handoff_unavailable` before
any provider work
> - Recovery must just work: a follow-up run must never fail because a
predecessor's warm copy is gone
> - This pull request lets the idle close finish before the sandbox
stops, prepares instructions fresh when the warm copy cannot be handed
off, and releases an owner whose sandbox the host already gave up
> - The benefit is that automatic follow-up runs start normally after a
warm window ends

## Linked Issues or Issue Description

Refs #21472 (base of this stacked PR), #21410 (introduced the
warm-window stop).

**What happened?**
In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three
automatic follow-up runs failed before any provider work with
`native_instruction_materialization_handoff_unavailable`: run
`5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a
release swap. In each case, the predecessor's idle close had failed
first with `provider_state_unreadable` / "Daytona sandbox lease … is no
longer active". The cause: the warm-window stop from #21410 stopped the
sandbox at the same moment that the idle close started. For the first
run's predecessor, the close was requested at 20:30:34 and the sandbox
got SIGTERM at 20:30:37.8.

**What did you expect?**
The follow-up run prepares its instructions fresh and continues. The
warm-window stop does not interrupt the host's own idle close.

**Steps to reproduce**
1. Run a warm Daytona task until it ends, so its warm window starts.
2. Let the warm window end while the idle timer closes the Runner.
3. Start a follow-up run on the same task. It fails with
`native_instruction_materialization_handoff_unavailable`.

Two of the three tasks then ended stranded. That downstream part (the
claim-gate cancellation with `execution_reconciliation_required` and the
stranded notice) is owned and fixed by another session. It is **not** in
this PR.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`
`claimWarmNativeInstructionCopy`: after it retires a stale warm owner
whose copy cannot be adopted, it returns `null`. The caller then
prepares instructions fresh. Before, it threw
`native_instruction_materialization_handoff_unavailable`.
- `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now
waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window.
The in-process idle close (close grace plus the 30 s proof read)
finishes first.
- `native-session-executor.ts` idle timer: when an idle close fails, it
reads the lease rows for that sandbox. If every row is no longer
`active` or `retained` (the host gave the sandbox up), it releases the
in-process owner. The next run then prepares fresh.
- `doc/execution-semantics.md`: one paragraph about the stop margin and
fresh preparation.

## Verification

Red → green (each test fails on the base `35ba76d0ef` and passes on this
branch):

- `native-session-executor.test.ts` › "hands off collection before
probing or retiring a warm owner: unavailable" — red: throws
`handoff_unavailable`; green: returns `null` (fresh preparation).
- `native-session-executor.test.ts` › "forgets a warm owner whose idle
close failed only once the host gave up its sandbox: pending_cleanup /
released / retained" — red on all three; green: the owner is released
for `pending_cleanup` and `released`, and kept (retired by the
successor) for `retained`.
- `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the
warm runner time to finish before stopping its sandbox" — red: the
sandbox stops 3 s after the window; green: it stops only after the
margin. Two existing tests ("keeps a sandbox still inside its warm
window, and wakes when the window ends", "reconciles a warm sandbox left
running across a restart") now expect the deadline with the margin.

Local runs on this branch:

- `server`: `tsc --noEmit` clean.
- `vitest run src/__tests__/warm-sandbox-expiry.test.ts
src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed.
- `vitest run
src/services/native-runtime/native-session-executor.test.ts -t "hands
off collection before probing or retiring a warm owner|forgets a warm
owner whose idle close failed"`: 10/10 passed.

Not run: a Daytona wave with this change.

## Risks

- Warm sandboxes now stop up to 2 minutes later than the configured
window. This is a small cost increase per warm sandbox. I chose a fixed
margin over an in-process "is a close running?" check because the stop
is pure SQL. The margin works when two servers run at the same time (the
server that runs the stop may not be the server that owns the Runner),
and it needs no busy-loop or cross-server signal.
- The margin does not cover a re-armed idle timer or a close that takes
longer than 2 minutes. Fresh preparation (fix 1) and the owner release
(fix 3) cover those cases.
- Not run on Daytona. The evidence is from the wave above and the unit
tests.
- In the wave, a `restore_failed` copy-back onto the just-restarted
sandbox was also seen. I did not find its root cause. It is not changed
here.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test runs) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:47:42 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
DottaandClaude Opus 5.5 4879a146fb fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paid model calls (company decisions, fast responses, browser use,
prompt-injection checks) pass a budget admission before they start
> - Admission runs in a REPEATABLE READ transaction and share-locks the
credential's secret row, so a key rotation cannot slip between admission
and the call
> - Every credential read also wrote "Last used" (`last_resolved_at`,
`updated_at`) to that same row
> - When two calls ran at once, one call's credential read aborted the
other's admission with a serialization failure (40001). The retry then
saw the first call's new budget hold and denied the work as
`budget_blocked`
> - This is the flaky `decision-models.test.ts` failure that held Cloud
readiness, and so releases, on 2026-11-13
> - This pull request stops credential reads from writing the secret
row. "Last used" now comes from the access event that every read already
records
> - The benefit is that concurrent paid calls are no longer wrongly
denied or retried, and the release gate no longer fails on this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (4/10))` failed on
master commits 6917d0568d and 5ede64f247. The test "admits concurrent
work from the same budget observation and records both obligations"
reported `expected "vi.fn()" to be called 2 times, but got 1 times`.
#21387 replaced the 1-second poll with an unbounded wait. That does not
fix it: one decision really is denied, so the test now waits until the
test timeout.

**Root cause**
I logged the accounting retry loop under load. Each batch of 50 runs had
6 to 10 retries with `could not serialize access due to concurrent
update` on `select … from company_secrets … for share`. The other
decision's credential read had updated the row. In 4 of 5 batches, the
retried admission saw the other decision's committed reservation and
returned `budget_blocked`.

**Expected behavior**
A credential read does not change admission. Two decisions admitted from
the same budget observation both run and both record their obligation.

**Steps to reproduce**
Run the updated test on the old `secrets.ts`. The test makes the second
credential read land after the first admission's snapshot. It then
commits an unrelated hold after both observations. The old code fails
every run with `first settled before its provider call:
{"status":"unavailable","reason":"budget_blocked"}`.

## What Changed

- `secretService` no longer writes `last_resolved_at` / `updated_at`
when it resolves a value. It still records the `secret_access_events`
row, as before.
- `secretService.list` reports `lastResolvedAt` as the later of the
stored column (legacy values) and the newest successful access event. It
reads one row per secret, backwards on the existing `(secret_id,
created_at)` index.
- A read without a consumer context (for example a GitHub API credential
read) now records a `system` access event, so it still counts as "Last
used".
- The removed `UPDATE` also made a credential read wait for a concurrent
rotation to commit. `credential()` in `ai-connections.ts` depends on
that wait to re-read the rotated version instead of handing a run a
consumed single-use token. Its existing version re-read now takes `FOR
SHARE`, so it keeps that wait. `prepareManagedAiRuntime` has its own
before/after freshness read around `runtimeCredential()`; that read
takes `FOR SHARE` too. A share lock creates no new row version and
cannot abort an admission. It adds no statement.
- The first version of this PR throttled the write to once a minute.
Review found that a read crossing the minute could still abort an
admission, so the write is removed instead.
- The concurrent-admission test is now deterministic. It starts decision
A and waits until A is blocked, then starts decision B and waits again.
It commits an unrelated 1-cent hold after both observations, then checks
that both decisions are admitted. A decision that settles before its
provider call fails the test immediately with its result, instead of
hanging.

## Verification

- Red: the new test against the old `secrets.ts` fails on every run with
the `budget_blocked` result above.
- Green: the same test with `repeats: 49` (50 runs) passes 50/50 locally
on a machine at load average 20 to 60.
- Before the change, 3 parallel batches of 50 runs of the old test each
produced a false denial.
- `npx tsc --noEmit -p server`: no errors.
- New `secrets-service.test.ts` test: a resolution leaves the secret row
unchanged, and `list()` still shows "Last used".
- `ai-connections.test.ts` quota test now checks that a read leaves
`updated_at` unchanged and records a success access event.

## Risks

- `updated_at` on a secret no longer moves when the secret is read. It
still moves on every real change.
- "Last used" on the Secrets page costs one indexed lookup per listed
secret.
- Only `list()` derives "Last used". Other endpoints that return a
secret row show the stored column, which is now set only by older
releases.
- No migration. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:50 -06:00
DottaandClaude Opus 5.5 ce5f10d534 fix(wake-queue): start a person's saved message after a legacy run is continued from saved work (#21518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails and cannot resume exactly, Paperclip
continues the task in a fresh run from its last saved work
> - If a person sent a message while the task was blocked, the
fresh-continuation fallback lets that saved message start the next run
(`delivery: saved_input`) and does not queue its own run
> - The release step that promotes the saved message skips the "needs
reconciliation" hold for a native run that the fallback superseded, but
not for a legacy run
> - So for a legacy run the task moves to To do, the person's message
stays saved, and no run starts until a watchdog finds the task idle
> - This pull request applies the same superseded-run rule to legacy
runs in the release step
> - The benefit is that a person's message after a failed run starts the
next run at once, for legacy runs as for native runs

## Linked Issues or Issue Description

**What happened?**
A person sent a message on a task whose last legacy run failed.
Paperclip continued the task from saved work and moved it to To do. The
message stayed saved. No run started until the assigned-work watchdog
queued one later.

**Steps to reproduce**
1. Let a legacy run fail with an active recovery hold, so the task is
Blocked.
2. Post a comment as a person. The wake is saved as
`deferred_issue_execution`.
3. Let the fresh-continuation fallback continue the task. It records
`delivery: saved_input`.
4. Run the deferred receipt and stranded-queue passes.

**Expected behavior**
One run starts and carries the person's message.

**Actual behavior**
No run starts. The release step returns `released` because
`legacyExecutionNeedsReconciliation` is true for the superseded legacy
run.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: the release
pre-drain check does not ask for legacy reconciliation when the run is
superseded by a fresh continuation. This matches the existing native
rule in the same function.
- `server/src/services/fresh-continuation-fallback.test.ts`: a red/green
test for native and legacy runs. A person's comment goes through the
real wake path while the task is blocked, the task continues from saved
work, and exactly one run must start with the comment.

## Verification

- Red: before the fix, the legacy case fails. The saved wake stays
`deferred_issue_execution` and no run starts. The native case passes.
- Green: after the fix, both cases pass.
- `pnpm verify` for the affected selection.

## Risks

- Low risk. The change applies only when the fresh-continuation fallback
has already marked the run superseded. That decision already records
that the old run is not replayed, and the successor runs as a fresh
turn. Other callers of `legacyExecutionNeedsReconciliationWithEvidence`
(retry scheduling, recovery) are not changed, so a superseded run still
cannot be retried automatically.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:40:42 -06:00
DottaandClaude Opus 5.5 f610e6a880 fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The assigned-work sweep is the backstop that makes sure every open
agent task has a live run, a pending path, or a named wait on a person
> - It counts a task with a queued run as healthy. A queued run that the
dispatcher declines on every pass is never started, so the task waits
forever and the sweep never looks again
> - On a hosted instance one agent had more than 30 queued continuations
that the claim declined on every pass. The sweep saw each task as
healthy
> - This pull request makes the sweep recognize a queued run that the
dispatcher keeps passing over, report it, and replace it once with a new
continuation when it is automatic work
> - The benefit is that an automatic task cannot stay stuck behind a run
that never starts, while a person's work is never cancelled

## Linked Issues or Issue Description

Refs #21391, #21349, #21433, #21394, #21443, #21450

**What happened?**

An agent had more than 30 queued automatic continuations
(`issue_continuation_needed`) that never started. The claim declined
each one on every dispatch pass. #21391 fixed that specific cause in the
chat-control gate. But the sweep still trusted any queued run as a
pending path. A run that the claim declines for any persistent reason
keeps the task "healthy" forever.

**Expected behavior**

A queued run that the dispatcher keeps passing over is not a healthy
pending path. The sweep reports it. For automatic work it cancels the
run through the run lifecycle and queues one new continuation in its
place. For a person's work it reports the run and leaves it.

**Steps to reproduce**

1. Queue an automatic run for a task and make the claim decline it
persistently (for example, a chat-control admission marker that does not
verify).
2. Let the run wait for more than one hour while its agent starts other
runs.
3. Run the assigned-work sweep for the task. Before this change, it
reports the task as `live_run` and does nothing.

## What Changed

- New violation `queued_run_stalled` in
`recovery/assigned-work-invariant.ts`. A queued run is stalled when all
of these are true (read from the database, on the database clock):
- it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes,
from `queued-run-dispatch-order.ts`);
  - it is the oldest queued run of its agent;
- its agent started more than `OLDEST_DISPATCH_SHARE` other runs after
the run became its oldest queued run: after the later of the run's aging
time and the moment the last older run of the agent started or finished.
- A stalled run is not a live run, and it does not let the task's
interruption projection ("working") count as a pending path. Its own
wake is not counted as a pending wake. A stalled run on a blocked task
still counts as live, so the block decides first.
- `listSuspects` lets a task through when its only active run is a
stalled queued run.
- `act()` handles the violation after all existing early returns:
- person work (the #21349 source-chain rule through
`listPersonOriginSourceRunIds`, which this change exports, or an
explicit user continuation): reported, never cancelled;
- a replacement that stalls again (`replacesStalledRunId` in its
context): reported, not cancelled again;
- both log one warning per run, with the run, agent and how long it
waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour);
- automatic work: the existing claim under the task row lock re-checks
the facts and the shared hourly cap and spacing. Inside that claim,
before the person check, it reads the run's row version (`xmin`) and the
number of wakes attached to it. Then the run is cancelled through the
run lifecycle (`setRunStatusFromLive` from `queued`, with that version
and wake count as its condition; wake cancelled, run event,
`releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A
person's wake that joins the run after the check, or a claim that starts
it, changes one of them, so the cancel does nothing. Then one
continuation wake is queued with `previousRunId` and
`replacesStalledRunId` set to the cancelled run. If the cancel does
nothing, the unpublished claim entry is removed and nothing is queued.
- New `heartbeat/stalled-queued-run.ts` builds that cancel from the run
lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to
the sweep.
- Plain activity text: "Continued automatically: this task had waited
over an hour to start while its agent started other work, so Paperclip
queued it again."

### Why this signal

- #21433 keeps refused claims and claim back-off in memory only
(`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`).
Another server cannot read them. Run rows can be read from any server,
on the database clock.
- After #21349, an aged run ranks no lower than ordinary work, and the
oldest-run share starts the oldest ready queued run at least once in
every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's
oldest queued run since after the aging bound, and its agent started
more runs in that time than the share allows, the dispatcher offered the
run a slot and the claim declined it. Starts that happened while older
runs were still waiting do not count: those older runs were getting
their turns.
- A run behind a full agent sees no such starts and is left alone. Under
a slow backlog only the oldest queued run can match. So the younger
queued runs keep their place in line.
- Starts are counted only after the run reached 60 minutes of age, not
after the current server's boot. 60 minutes is longer than a release
swap overlap. So a run that only waits for a new server to reattach to
its Runners does not match. #21450 will add a reattach-phase gate before
this path.

### Why this does not set `retryOfRunId`

An earlier plan added `retryOfRunId` to the sweep's continuation so the
chat-control gate could follow it. An audit of its readers found three
changes in behavior: the admission-time chat-control check would start
to refuse sweep wakes; the claim would wait on the source run's cleanup
(a possible wait forever); and the agent prompt would call the run "a
Retry" that "re-delivers the wake". #21391 already makes these runs
claimable. So this change leaves `retryOfRunId` unset, as today. The
replacement links its source with `previousRunId`, as the sweep already
does.

### Questions from review, answered from the code

- **How often is the invariant pass triggered, and can two passes
overlap?** The `assigned-work-invariant` delivery queue
(`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work
notifications, on each task's own deadline (`retryAt`), and on a 30 s
reconstruction of the suspect list. Passes are at least 1 s apart
(`minimumPassIntervalMs` default). In one process, passes of the queue
never overlap (`running` guard in `delivery-work-coordinator.ts`). Two
servers can sweep at the same time. Each repair is serialized by the
claim under the task row lock (`FOR UPDATE` in `claimContinuation`),
which re-reads the facts and the spacing. This change does not add a new
loop.
- **Can two runs or no run exist for one task?** Two runs: no. The
claim's spacing makes a second server back off, and the cancel happens
before the new wake (a queued run on the task would otherwise absorb the
wake). No run: there is a short gap between the cancel and the new wake,
because wake admission locks the task row in its own transaction. If the
server stops in that gap, the task has nothing pending, and the next
pass repairs it as `no_pending_path` under the same cap.

## Verification

- `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new,
embedded Postgres, real `createHeartbeatQueue` admission and
`claimQueuedRun`):
- an automatic run that the real claim declines persistently, whose
agent started runs after it aged: reported, cancelled, exactly one
replacement, and the real claim starts the replacement;
  - a run behind a full agent: left alone, not a suspect;
  - a person's declined Retry: reported, not cancelled, one warning;
- a replacement that stalls again: not cancelled on two checks, one
warning, next look one hour later;
- two sweep instances on the same task at the same time: one
replacement;
- three aged runs queued in order, the first two get their turns slowly:
the third is not stalled when it becomes oldest, and is stalled only
after more than four further starts pass it over;
- a person's message joins the run (real wake admission) between the
claim and the cancel: the run is not cancelled, there is no replacement,
and the person's wake stays attached to the run;
- the run starts between the claim and the cancel: no cancel, no
replacement.
- First round, red before the fix: 4 failed, 1 passed (the full-agent
case passes before and after). Green: 5 passed.
- Review round: the three new tests on the previous head: 2 failed (the
backlog case and the person-join case), 1 passed (the started-run case,
which the `queued` check already covered). Green: 8 passed.
- Existing files run one at a time: `assigned-work-invariant.test.ts` 31
passed, `assigned-work-sweep-round-trips.test.ts` 2 passed,
`failed-continuation-continues-again.test.ts` 28 passed,
`recovery-wake-herd.test.ts` 35 passed,
`saved-work-recovery-overdue.test.ts` 4 passed.
- `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean.
- `pnpm verify`: not completed locally (the machine is heavily loaded
and the run was stopped when the code changed). Required CI supplies the
broad coverage.

## Risks

- A task's oldest queued automatic run is cancelled and queued again at
most once per run. Its replacement starts at the back of the line. Only
a run that the dispatcher already passed over can match.
- A replacement that stalls again is only reported, so a persistent
claim refusal cannot cause repeated cancellations.
- The new subqueries run only for queued runs older than one hour. They
use the existing `(company_id, agent_id, started_at)` index. Healthy
runs and the path before `turn.start` do not change.
- #21443 (automatic-run brake) and #21450 (reattach first) touch
`assigned-work-sweep.ts`. The new path comes after every early return
that `act()` has today, so their gates can go before it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). A subagent on the same model
audited the `retryOfRunId` readers.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:40:38 -06:00
Devin FoleyandPaperclip 0a96f972e1 Connect Tailscale networks through device sign-in (#21526)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - SSH environments can reach machines through a company Tailscale
connection.
> - The initial connector requires a managed OAuth app that Tailscale
limits to its own tailnet.
> - Customers need to connect their own tailnets without registering an
app or supplying a key.
> - This pull request uses Tailscale's normal device sign-in for new
connections.
> - Customers approve the Paperclip device and then select their
machines in SSH environments.

## Linked Issues or Issue Description

**Problem or motivation**

The current Tailscale enrollment path cannot serve independent customer
tailnets through the documented self-service OAuth app model. It also
requires Cloud broker configuration and pilot admission before a
customer can start.

**Proposed solution**

Start a private Tailscale device in the tenant and open its normal
sign-in link. Save its encrypted device state before reporting a
successful connection. Retain the existing SSH machine selection and
agent account behavior.

**Alternatives considered**

Auth-key entry would require customers to copy a credential. A public
OAuth integration still needs a confirmed registration path. Device
sign-in uses a documented Tailscale flow without either requirement. See
[device
authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication)
and [OAuth app
limitations](https://tailscale.com/docs/features/oauth-apps).

**Related PR search**

Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for
`tailscale`. This follows the merged network connector in
[#20617](https://github.com/paperclipai/paperclip/pull/20617) and
preserves the SSH agent-account support in
[#21248](https://github.com/paperclipai/paperclip/pull/21248). The open
[#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes
storing an OAuth client and checking its health; this change instead
enrolls a device through interactive sign-in so customers do not need an
OAuth app.

## What Changed

- New connections use device sign-in. Existing connected devices and
pending legacy OAuth callbacks remain supported.
- Pending sign-ins are bound to their company and initiating user. They
have a fixed ten-minute deadline. Device state uses server-owned vault
secrets with trusted version and digest checks.
- The active Tailscale CLI waits for consent and device approval.
Cancellation, expiry, secret deletion, and stale completion cannot
restore a cancelled connection.
- The connector stays open during sign-in. It provides a link when
popups are blocked and shows device approval separately. Progress checks
stop on completion, failure, expiry, or unmount.
- Shared response types, OpenAPI contracts, SDK source pins, and the
API/setup documentation describe the new flow. No database migration or
Cloud provider configuration is required.

## Verification

- 109 focused tests cover network processes, the real database and
encrypted vault, routes, UI, and legacy connection behavior.
- Server and UI typechecks pass.
- Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure
locally. All 70 OpenAPI and Tailscale route tests and server typecheck
pass after synchronizing the API contract.
- Follow-up `79de05f38b9`: the transient-recovery regression failed
before the fix; all 20 connector UI tests and UI typecheck pass after
it. The query-error policy check reports no findings.
- The public SDK input registry check passes.
- [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2)
passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57
successful checks, no failures or pending checks. The interrupted
canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile
is 5/5 on this head with no unresolved review threads.
- `pnpm verify --explain` exits 2 because the documentation change
selects broad CI coverage. Focused local checks passed; broad CI passed
on `29cf296a597e470d432b84767050bc539caa280e`.
- Live local qualification on
`29cf296a597e470d432b84767050bc539caa280e`: native device sign-in
completed with interactive user approval. An isolated Ubuntu 24.04 ARM64
EC2 machine appeared in discovery and passed SSH pairing and workspace
probes. The assigned Claude Code agent used a Paperclip-managed
subscription, created `hello_world.txt` containing exactly `Hello,
world!` plus a newline, posted a completion comment, and marked the task
done. An independent read through the private connector verified the
bytes, remote user, and hostname. The remote machine had no stored
Claude login and no leftover temporary credential directories.
- Restarted the local Paperclip server and verified the same Tailscale
device identity, online peer discovery, successful SSH probe, and
unchanged file without new consent. Temporary public SSH ingress was
removed.
- Local qualification covers one user tailnet and the final Core
revision. Browser UI acceptance, two unrelated tailnets, a hosted
restart, and an agent task on the final staging revision are not yet
verified.

## Risks

- The enrollment method changes. It must pass the live qualification
above before broad rollout claims.
- Existing encrypted device state and the vault master key must survive
a restart or redeploy. Device expiry or removal still requires
reconnecting.
- The existing one-serving-process-per-instance constraint remains.
Concurrent replicas need durable network ownership.
- Pending setup uses bounded progress requests and an event-driven CLI
with a deadline. It adds no idle database scan.

## Model Used

OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed by the runtime. No
subagents were used.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 17:38:55 -08:00
DottaandClaude Opus 5.5 cefef8f11d perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).

**Current behavior**

Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.

**Proposed behavior**

The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:

- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.

A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.

A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.

**Reason and benefit**

Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.

## What Changed

- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.

## Verification

- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.

The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):

| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |

Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.

## Risks

- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:35 -06:00
DottaandClaude Opus 5.5 3cd671575d perf(attention): scope native finalization retention hints to the run's task graph (#21521)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server archives old attention items (reviews, blocked tasks,
decisions, failed runs) in a background queue, `attention-retention`.
> - A hint for a whole company makes this queue reload every attention
candidate of the company, 100 per page. Each page also reads review
attention, blocker graphs and source evidence.
> - Each native run finalization sent such a company hint, although a
run changes only the attention items of its own task graph.
> - On a hosted instance with about 1,400 open tasks and live native
runs, the decision retention read ran 26.7 times per second. Native run
finalization is one of the producers that sent a company hint on every
run.
> - The same finalization already sends the issue-scoped hint through
`notifyNativeStatusReassessment` when the run has a task or a run id.
This pull request sends the company hint only when the run has neither.
> - The benefit is that a finished run reloads only the candidates of
its task, its ancestors and the tasks it blocks, not every candidate of
the company.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The attention retention hint that native run finalization sends
(`notifyNativeFinalizationWork` in
`server/src/services/native-finalization-notifications.ts`).

**Current behavior**

`notifyNativeFinalizationWork` calls `notifyAttentionRetentionWork(tx, {
companyId })`. The `attention-retention` queue has no reconstruction
interval, so each finalization reloads all eight candidate kinds of the
company, page by page, with the batch eligibility reads for each page.

**Proposed behavior**

`notifyNativeFinalizationWork` sends the company hint only when the
scope has no `issueId` and no `runId`. Otherwise
`notifyNativeStatusReassessment`, which the same function already calls,
sends `notifyIssueAttentionRetentionWork(tx, { companyId, issueId, runId
})`. That helper resolves the task of the run, adds its parents and the
tasks it blocks (at most 100), and sends a hint with these task ids.
When the run has no task, or the graph has more than 100 tasks, the
helper sends the company hint itself.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The
attention items that a run can change are the failed run, the review,
the recovery actions, the decisions and the blocker items of its task
graph. The issue-scoped hint covers all of them, and the ordinary task
write path uses the same helper.

## What Changed

- `server/src/services/native-finalization-notifications.ts`: send the
company attention retention hint only for a finalization without a task
or run.
- `server/src/__tests__/attention-retention-work.test.ts`: new test
`scopes native run finalization to the task graph of the run`.

## Verification

- The new test sends the finalization hint once with a run id and once
with a task id. Before the change both hints are `{ companyId }`. After
the change both hints name the task and its parent.
- `npx vitest run src/__tests__/attention-retention-work.test.ts` and
the native work tests (`native-result-commit-work`,
`native-environment-settlement-work`, `native-run-projection-work`,
`native-recovery-notice-work`, `retained-native-session-cleanup-work`,
`native-status-reassessment-work`, `muse-terminal-settlement`,
`ownership-reconciliation-report`) pass (in `server/`).
- `pnpm check:module-boundaries` passes.

## Risks

- No new reads. The issue-scoped hint was already computed in the same
transaction by `notifyNativeStatusReassessment`.
- An attention item outside the task graph of the run is not reloaded by
this hint. Agent status, budget and company changes keep their own
hints.
- Low risk. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:04:30 -06:00
DottaandClaude Opus 5.5 a82fa96540 perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane port commits streamed text deltas of a run in
batches (#21460): the first text at once, later deltas after at most 250
ms
> - Each batch is cut when its timer fires, even when the previous batch
of the same run has not committed yet
> - When the database is slow, every run then queues one small batch per
250 ms behind its slow commit. The number of statements stays the same
exactly when the database has the least room for them, and the queue
grows
> - This pull request cuts a batch only when the previous batch of the
run has committed. Deltas that arrive while a batch commits all join the
next batch
> - The benefit is fewer, larger statements under load, with no change
when the database keeps up, and no longer wait for any delta

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21511 (one statement per delta
batch). Refs #21418 (agent scaling plan and load model).

**What existing behavior does this improve?**
How `PaperclipControlPlanePort` groups queued text deltas into batches
when a batch commit is slower than the batch time.

**Subsystem affected**
Server run event ingest:
`server/src/services/native-runtime/paperclip-control-plane-port.ts`.

**Current behavior**
`#flushStreamingEvents` takes the queued deltas when it runs (on the 250
ms timer, at 100 deltas, or before any other event) and chains their
commit behind the previous batch. If one commit takes two seconds, the
run queues about eight batches, each one statement, behind it.

**Proposed behavior**
`#flushStreamingEvents` chains one cut behind the previous batch and
takes the queued deltas when that cut runs. While a cut waits, later
flush requests join it. A run has at most one batch committing and one
cut waiting.

**Reason and benefit**
Under load, the number of delta statements per run falls to about one
per commit time instead of one per 250 ms. When commits are faster than
250 ms, nothing changes.

**Breaking changes**
None. Rows, receipts, order and the first-text rule are the same.

## What Changed

- `#flushStreamingEvents` queues one cut on the run's commit chain
(`#streamingFlushQueued`). The cut takes every delta queued at that
time, sets the first-text flag, commits the batch, and then releases the
pending count of that batch.
- A flush while a cut waits returns the same chain, so a caller that
waits for it (every other event, `checkpointSession`, `completeRun`,
`replayEvents`, `provesNoRunProviderWork`) still waits for every delta
queued before it.
- A batch can now hold more than 100 deltas when commits are slow. The
100-delta rule still asks for a flush; the batch is one statement with a
fixed number of bind parameters (#21460, #21511).

## Verification

- Red/green: new port test "collects the deltas that arrive while a
batch commits into one next batch, before the next other event"
(`paperclip-control-plane-port.test.ts`). A transaction holds the run
row, so the first batch waits. Six more deltas arrive 15 ms apart with a
5 ms batch time, then a non-streaming event. Before: 8 inserts (one per
timer). After: 3 inserts (the waiting batch, one batch of six, the other
event). Rows are stored 1–9 in order and every receipt resolves.
- The whole port suite with its conformance suite passes: 101 tests.
Other suites, one file at a time: `native-observer-replay` (1),
`paperclip-semantic-conformance` (1), `native-interaction-bridge` (7),
`chat-native-progress-work` (35). I ran the four streaming port tests
two more times; they pass each time.
- `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in
#21418). The model applies the port's batching rules itself, so for
these runs I changed my local copy only (not in this pull request): with
`DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append
chain and takes the queued deltas when the cut runs, as the port now
does. All rows run with #21511. Default pool of 10 connections; embedded
PostgreSQL on the same, shared machine; runs one after the other with a
pause between them. No run had a failed operation, a failed checkpoint
save or a lost lease.

| Agents | Batching | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 /
12,000 | 3 / 3,994 |
| 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 /
6,040 | 18,719 / 24,000 | 5,721 / 15,238 |
| 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993
/ 24,000 | 1,896 / 4,193 |
| 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 /
9,483 | 18,771 / 36,000 | 2,280 / 30,985 |
| 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 |
61,158 / 72,000 | 11,678 / 21,212 |
| 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 /
6,057 | 58,984 / 72,000 | 11,465 / 25,361 |
| 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 |
24,877 / 48,000 | 24,544 / 31,331 |

The process does more work per second with this change (more other
events finish), so statements per second rise; delta lateness falls. 600
agents are still not carried reliably: other events fall behind. #21528
removes two statements from each such event and one from each checkpoint
save; its description has the stacked results.

## Risks

- A delta never waits longer than before: it already waited for the
previous batch of its run. When the database keeps up, batches are the
same as before.
- Under load, one batch can be large. The live view then gets one larger
step instead of several small steps behind the same wait.
- A failed batch now rejects more deltas at once (every delta that
joined it). The runner resends them, as before; each resend is
idempotent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:00:08 -06:00
DottaandClaude Opus 5.5 0bba04675c perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - After #21460, text deltas commit in batches, but each batch still
takes five statements: BEGIN, the run lock, a replay probe, one insert,
and COMMIT. A single event without a wake takes four
> - In the agent scaling load model, the process issues about 36
statements per agent per second and saturates its one thread between 200
and 400 agents
> - This pull request writes a batch of in-order native events, or one
such event, with one statement that commits on its own
> - The benefit is about half the statements per agent, less CPU per
agent on the main thread, and less time on a pool connection, with the
same rows, receipts, cursor and locks

## Linked Issues or Issue Description

Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418
(agent scaling plan and load model).

**What existing behavior does this improve?**
The append of native source events that continue their source's stored
cursor: batched text deltas, the first delta after another event, and
every other native event that registers no wake.

**Subsystem affected**
Server run event ingest: `server/src/services/heartbeat-run-events.ts`,
and the port comment in `paperclip-control-plane-port.ts`.

**Current behavior**
A delta batch takes BEGIN, a run lock read (`for no key update`), a
replay probe, one insert, and COMMIT: five statements. One delta that
commits at once takes four. In the load model at 200 agents the process
sends about 6,700 statements per second at event-loop utilization 0.72.

**Proposed behavior**
The same append is one statement. Its update of the run row takes the
same row lock. It requires the stored cursor to end just before the
batch, requires the `next_event_seq` that its own snapshot read, and
finds no replayed event. Then it moves the sequence and the cursor and
inserts every row. If any check fails, it writes nothing and the append
takes the existing locked transaction.

**Reason and benefit**
Fewer statements free the main thread and the shared pool for other
agents. In the load model at 200 agents, statements per second fall from
about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to
0.56.

**Breaking changes**
None. No schema change. Rows, receipts and the cursor are the same.
Servers from before this change keep working on the same runs.

## What Changed

- `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one
statement: a CTE reads `next_event_seq` from the statement snapshot, an
`update heartbeat_runs ... returning` locks the run row and moves
`next_event_seq` and the source cursor, and an `insert ... select from
jsonb_array_elements($9)` writes every row. The text is fixed, with nine
bind parameters for any batch size.
- **Why the `next_event_seq` check.** A statement that waits for the run
row lock keeps the snapshot it took before the wait. Its replay probe
cannot see events that the lock holder committed. Every event append
(native, ordinary, the run coordinator store, the retry queue) moves
`next_event_seq` in its own transaction. So the update requires the
value that its snapshot read. PostgreSQL re-checks that condition on the
newest row version after the wait, and any append committed meanwhile
makes the statement write nothing. The unique indexes on `(run_id,
source_event_id)` and `(run_id, source_instance_id, source_seq)` are a
second guard: a unique violation from this statement also falls back.
- A source without a cursor starts at 0. The probe still finds any
stored event of that source, so a run that an older server wrote falls
back and recounts.
- `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the
one statement first for native events without a wake, outside a caller's
transaction. Every other case (a replay, a gap, an event with a wake, a
cursor that lags, a run changed meanwhile, a binding mismatch, a run
deleted meanwhile, a caller's transaction) takes the existing
transaction, unchanged.
- Inside a caller's transaction the append keeps its SAVEPOINT path. A
failed statement there would abort the caller's transaction, and the
fallback could not run.
- `assertWritable` runs before the statement is sent and after it
returns (see Risks).

## Verification

- Red/green on embedded PostgreSQL. I ran the new tests against the
previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests
in the file pass with the change
(`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`):
- One statement for an in-order event and for a batch (before: 4 and 5).
- A writer revoked before the send sends nothing; a statement that fails
on its third row stores no row and no sequence; the resend commits.
- A writer revoked while the statement runs: the rows stay, the receipt
is rejected, and the resend is a replay with the same rows.
- A batch waits for Stop on the run row and appends after it; a batch
and a single event that wait for a run deletion fail with
`heartbeat_run_event_binding_mismatch` and store nothing.
- An older server stores events 2 and 3 while it holds the run row,
without the cursor: the waiting batch 2–4 falls back, returns duplicate,
duplicate, committed, and the cursor ends at 4.
- Two connections append the same 200-event stream in batches of 10:
each event is stored once, in source order, sequences 1–200 without a
gap, one `committed` and one `duplicate` receipt per event.
- A 7,000-event batch (more than any bind parameter limit at 10
parameters per row) is one statement. Before: it fell back to one
statement per event and timed out at 15 s.
- A run without a cursor starts at its first event; a run with events
from an older server replays and then continues.
- Equivalence: 12 random streams of 30 steps (in-order batches, gaps,
replays of earlier ranges, fills of gaps, ordinary events, conflicting
replays). Each step goes to two runs, one through the new path and one
forced through the one-by-one locked path. Receipts, errors, rows,
`next_event_seq` and cursors are equal at every step.
- Mutation check: with the `next_event_seq` check and the
unique-violation fallback removed, the older-server race test fails.
- The port suite (`paperclip-control-plane-port.test.ts`, with the
package conformance suite) passes: 125 tests. Two tests changed: the
streamed-event test now expects one statement and the new revocation
behavior, and the batch test checks that the five deltas are one
statement sent before the boundary event.
- Other focused suites pass, one file at a time:
`heartbeat-run-event-sequencing` (5), `native-event-boundary-golden`
(1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1),
`native-interaction-bridge` (7), `chat-native-progress-work` (19),
`heartbeat-native-restart-suspension` (115), `live-recovery-work` (51).
In 2 of 3 runs with this change, all 51 tests passed but the file's
`afterAll` (closing the side pools and the embedded database) hit its 30
s timeout on this busy machine; the third run passed cleanly, and one
run on `origin/master` passed. The native events in that file all
register a wake, so they do not take the new path.
- `pnpm --filter @paperclipai/server run typecheck` and `pnpm
check:module-boundaries` pass.
- Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in
#21418 (per-source counts stop when the window closes). I did not change
its batching rules for this pull request. Model rates: 20 deltas/s, 1
other event/s, 0.3 API requests/s per agent; default pool of 10
connections; embedded PostgreSQL on the same, shared machine. I ran
"before" (`origin/master`) and "after" one after the other, with a pause
between runs, and I give the load average at the start of each run. The
machine was busy with other work (load 10–15 on 18 cores), so compare
rows that ran close together. No run had a failed operation, a failed
checkpoint save or a lost lease.

| Agents | Code | Window | Load avg | Event-loop utilization |
Statements/s | Delta lateness p50 / p99 (ms) | Other events done /
offered | Other event lateness p50 / p99 (ms) |
  | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 /
12,000 | 278 / 3,895 |
| 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 /
24,000 | 1,904 / 60,154 |
| 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 /
24,000 | 5,721 / 15,238 |
| 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 /
36,000 | 10,822 / 52,171 |
| 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 /
36,000 | 2,280 / 30,985 |
| 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135
/ 52,222 | 36,000 / 36,000 | 698 / 3,422 |

At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization
and 7,051 → 3,210 statements/s. One process with this change alone still
does not carry 600 agents: at 400 and above the 10-connection pool and
the shared database are the limit (event-loop utilization stays below
0.75 while other events fall behind). #21520 and a third pull request
for the remaining statements of other events and checkpoint saves
continue from here; their description has the stacked results.
- Statements: a delta batch 5 → 1; one delta or one native event without
a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save
(5).

## Risks

- **`assertWritable` window (read first).** Before, the writer check ran
inside the transaction, before the first statement and before COMMIT. A
revocation while the statements ran rolled the append back. Now a
revocation while the one statement runs does not roll it back: the rows
commit, and the caller gets the error instead of the receipt, so no
observer runs and the runner gets no ack. This check guards a controller
that hands its run to a successor at restart
(`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the
earlier check was not atomic with COMMIT either, so a revocation during
the COMMIT round trip already committed; the window grows from one
COMMIT round trip to one statement round trip plus its wait for a pool
connection; (2) the append is idempotent by source event id and payload
hash, so the successor sees the committed row as a replay of the event
that the runner resends; (3) the old controller never acts on the
rejected receipt. A new test covers exactly this.
- A crash loses nothing new: the statement commits or not as a whole,
and a resend after an unknown outcome is a replay.
- The check on `next_event_seq` makes the statement write nothing
whenever any other event of the run committed after its snapshot. Under
heavy contention on one run, more appends take the fallback. The
fallback is the previous code path.
- The statement text is fixed SQL sent through the driver
(`$client.unsafe`), like the rest of the event stream statements.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:59:01 -06:00
DottaandClaude Opus 5.5 2178c3df88 fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a recovery read to a remote Runner goes unanswered, the next
attempt of the same run carries `replaceUnansweredRunner`
> - That attempt verified the Runner was alive and then refused to adopt
it anyway. On a resumed sandbox lease the reuse check then failed with
`resumed_not_suspended` and left the Runner running. Otherwise a second
Runner was launched, and its launch fence killed the busy Runner, which
lost the provider turn.
> - After a late reconnect a Runner is often slow, not dead: it is still
replaying the events it saved with no controller attached
> - This pull request makes the attempt check progress first. It adopts
a live Runner that is making progress and replaces only a Runner that
made no progress.
> - The benefit is that a slow but working Runner keeps its turn, and
the retry no longer causes the failure it was meant to fix.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13. The retry after a
first-attempt timeout failed with these codes (counts from the traced
sample of 136 failures):

- `runner_ingress_unavailable: runnerd process failed:
runner_harness_state_mismatch: resumed_not_suspended` (part of 40
`ingress` failures)
- `runner_remote_process_identity_unavailable` (25): the Runner was gone
by the time the retry checked, because the launch fence or a previous
retry had killed it
- "restored the conversation after process loss, but…" /
`provider_turn_lost_on_restore` (10): the replacement Runner restored
from the old state and the provider turn was gone

Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9
failed. Over 420 s: 14 of 15 failed. The process ID of the Runner
changed inside the same sandbox in the failed runs, which shows the
replacement.

Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt
with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit`
verifies the Runner is alive and then returns `undefined` because of
that flag, so `adoptedProcess` is `undefined`.

**Expected behavior**

A live Runner that is making progress is adopted. A Runner that made no
progress is stopped with proof and then restored from its own journal.

**Steps to reproduce**

1. Reattach a remote run whose previous attempt left a recovery read
unanswered (`replaceUnansweredRunner: true`), with the Runner alive and
its saved event sequence past the controller's last acknowledgement.
2. On master the attempt launches a second Runner, or fails with
`resumed_not_suspended` on a resumed sandbox lease.
3. The new tests in `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` do this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): with
`replaceUnansweredRunner`, the attempt adopts the live Runner, launching
nothing and signalling nothing, when:
- the Runner's saved event sequence is past the controller's last
acknowledgement, or
- its saved state (event and command sequences, generation, lifecycle)
changes within a bounded wait of about 10 s.
- The baseline is the first successful state read; every successful read
is also compared against the controller's acknowledgement. A Runner
whose state cannot be read at all during the wait is adopted, not
replaced: missing evidence is not proof of a wedge. A Runner whose
identity changes during the wait fails closed.
- A settled exit during the wait keeps the attempt allowance that
admitted the replacement, so it also restores on the extra fourth
attempt.
- Only a Runner that made no progress is replaced. The relaunch's
existing launch fence stops it and its process lineage with proof, and
the new Runner restores its own journal. A resumed sandbox lease no
longer refuses that relaunch with `resumed_not_suspended`.
- A Runner that exits during the wait follows the existing exited-Runner
rules: a settled suspension restores, and a mid-turn exit stays
fail-closed.
- The wait runs only on this recovery path, never on a normal run start.
- `doc/execution-semantics.md`: describes adopt versus replace.
- The claim flag `replaceUnansweredRunner` is set in claim logic that
another session owns. This PR does not change it; it changes only what
the executor does with it.

## Verification

Red on master, green on this branch:

- `native-session-executor.test.ts`
- `adopts a live Runner that emitted events past the controller's last
acknowledgement`
- `adopts a live Runner whose saved state advances within the bounded
wait`
- `hands a live Runner that made no progress to the relaunch, whose
launch fence stops it, without signalling it here`
- `restores from the journal of a Runner that suspended and exited
during the wait`
- `stays fail-closed for a Runner that exited mid-turn during the wait`
- `native-runner-restart-recovery.integration.test.ts`: `reattaches a
live %s Runner whose recovery read went unanswered on a resumed sandbox
lease` (progressing, wedged)
- Review fixes, `native-runner-restart-recovery.integration.test.ts`
(the first four were red before the fix):
- `adopts a live Runner whose first state read failed and whose later
reads advance`
- `adopts a live Runner whose first readable state is already past the
controller's acknowledgement`
- `does not replace a live Runner whose state could not be read at all
during the wait`
- `restores from the journal of a Runner that suspended and exited
during the wait on the admitted fourth attempt`
- `fails closed when the Runner's identity changes during the wait`
(pins existing behavior)
- `native-remote-retry-admission.test.ts`: unchanged, still passes.

Local runs on this branch: `tsc --noEmit` is clean for `server` on this
branch (master c6b497d831, which contains #21442).
`native-runner-restart-recovery.integration.test.ts`, the
adopt-or-replace block: 12 of 12 pass. The full
`native-session-executor.test.ts` and
`native-remote-retry-admission.test.ts` runs did not finish on this
shared machine (load 20 to 96); CI ran them on this head and they
passed.

Conflict resolved: master added a relaunch branch for a Runner that
exited for a fresh bootstrap (#21374) at the same place. Both branches
are kept; the cheaper check for an unanswered Runner that made no
progress comes first.

Daytona results that exist. They ran on a combined branch with this PR,
#21450, #21442 (merged) and #21362 (merged), so they cannot be credited
to this PR alone:

- G5 baseline (master d1d71a7755; 480 s delay injected on the first
identity write per run): one Claude run reproduced the staging chain:
authentication 419 ms, identity +423.5 s, attempt 1 `native session
provider recovery timed out after 900000ms`, then the retry failed in 6
s with `runner_harness_state_mismatch: resumed_not_suspended`.
- G5c (same cell, combined branch 588c6ba267): 6 of 6 tasks finished
hands-off. The chain did not recur. All runs that crossed swap 2 were
adopted on the same Runner process ID.
- Caveat: the adopt-or-replace retry path that this PR changes was never
exercised in a cell. In G5c no first attempt failed, because all
identity writes started together 4 to 5 s after the claim. G5c also ran
before the phase timing fix in #21450.

## Risks

- Not proven on Daytona: the adopt-or-replace path itself (see the
caveat above).
- Without #21442 (merged) an adopted Runner can still time out on its
first command while it replays a backlog. This PR stops the replacement;
#21442 (merged) makes the adopted Runner answer.
- A Runner whose state cannot be read during the whole wait is adopted.
If it is in fact wedged, the adopted attempt times out again and the
attempt limit bounds it.
- A wedged Runner that still emits a few events counts as progressing
and is adopted. The existing attempt limit bounds this.
- The progress check adds a wait of up to about 10 s on this recovery
path only. A normal run start has no added wait.
- The claim construction (`native-restart-recovery.ts`) is owned by
another session and is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:56:11 -06:00
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
Devin FoleyandPaperclip 089748fbf9 fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.

## Linked Issues or Issue Description

**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.

**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.

**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.

**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
  guidance: "Reconcile the existing operation before repeating it. ..." }
```

No related public PR exists. I searched open and closed PRs for this
fence.

## What Changed

- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
  - A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.

## Verification

- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.

## Risks

- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:51 -08:00
DottaandClaude Opus 5.5 e82c4f0a31 fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**

This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it

## Linked Issues or Issue Description

**What happened?**

Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.

- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.

**Expected behavior**

A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.

**Steps to reproduce**

1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.

**Paperclip version or commit**

`master` at c6b497d831 (contains #21442).

## What Changed

- `runnerd-codex-transport.ts` (controller-side TypeScript):
- Each liveness probe in the close is bounded. A probe that does not
answer counts as possibly alive, never as gone, and the suspend goes
over the live connection. A connected Runner's probe gets a 1 s slice,
so drain and suspend keep their budget.
- Once the suspend command's completion is observed, what the
durable-state read shows decides how long the proof may take:

    | Read after the completion | Proof window | Maximum close time |
    | --- | --- | --- |
    | Suspended, right Runner | proven at once | unchanged |
| Another run's or Runner's state | ends at the close deadline, as on
master | close grace (10 s; 45 s for Pi) |
| Right Runner, not yet `suspended` | settle window of 3 s from the
first such read (`RUNNER_SUSPENSION_PROOF_SETTLE_MS`, test option
`closeProofSettleMs`) | close grace + 3 s |
| Unreadable or slow read | full proof window of 30 s from the
completion (`RUNNER_SUSPENSION_PROOF_READ_MS`, `closeProofReadMs`) |
close grace + 30 s |

All rows add the existing bounded exit-probe slices. runner-core saves
the `suspended` lifecycle before it delivers the suspend result
(`persist_lifecycle_before_command_delivery` in `durable/runner.rs`), so
a readable non-suspended state after the completion can only be a stale
remote read; the settle window covers a few read attempts and is not a
write-ordering wait.
- The settlement diagnostic names the branch that ended the window
(`proofWindowEnd`: `proven`, `wrong_identity`,
`not_suspended_after_settle`, `unreadable_after_window`,
`suspend_not_completed`).
- A close that failed only because readable proof was missing is retried
by the next close: it reads the durable state again with a fresh bound
and finishes settled. A close whose proof was refuted stays memoized, as
on master. This extends the failed-close retry from #21442.
  - The close failure carries per-step timings.
- `native-runtime/predecessor-retirement.ts`,
`heartbeat/run-completion.ts`: the predecessor-retirement fallback
records the task's status when the run failed.
- `modules/run-dispatch/domain/policy.ts`,
`modules/run-dispatch/adapters/postgres.ts`: a continuation of a run
that did no provider work, on a task that was already blocked then, is
not treated as blocked after recovery. Its dependency check follows the
wake it continues. The continuation keeps a person's comment wake.
- `task-interruption-presentation.ts`: while that run's failure handling
queues the continuation (up to 30 s), the task shows as working with no
Retry. The window ends when the continuation is scheduled, when the run
records `continuationScheduling: "refused"` (written by
`run-completion.ts` when the fallback does not schedule it), or after 30
s. The fallback record alone no longer ends it.

## Verification

Red on master, green on this branch:

- `runnerd-codex-transport-close-proof.test.ts`
- `proves a completed suspend on its own read window when the durable
state read outlasts the close grace`
- `suspends five adopted idle Runners whose liveness probes stall
together, instead of spending the close on the probe`
- `heartbeat-native-predispatch-retry.test.ts`
- `continues a follow-up on a blocked task whose predecessor could not
be retired`
- `does not continue a task that was blocked only after its run failed`
- `task-interruption-presentation.test.ts`
- `shows the task as working while its fresh-sandbox continuation is
being queued`
- `keeps the task working after the fallback is recorded while its
continuation is still being scheduled` (review fix)
  - `offers Retry when the continuation was refused or never queued`
- `runnerd-codex-transport.test.ts` (review fixes)
- `gives a suspend that completes near the close deadline its whole
proof window from the completion`
- `bounds the proof window from the observed completion, so a proof read
that never answers still ends`
- `proves a suspension whose suspended state appears 1.5 s after the
command completed`
- `fails a suspension whose state stays ready after the settle window,
not after the whole proof window`
- `fails at the close deadline when the durable state belongs to another
run`
- The three existing tests in
`runnerd-codex-transport-suspension.test.ts` that expect a conclusive
negative to fail within the close grace pass unchanged.

Local runs on this branch: `tsc --noEmit` is clean for `server` and
`packages/paperclip-runner`. Passing on this branch:
`runnerd-codex-transport-suspension`, `-close-proof`, `-adoption`,
`-remote-retention`, `runnerd-detach-durability` (all files), the
proof-window cases of `runnerd-codex-transport.test.ts`,
`task-interruption-presentation.test.ts` (72), `run-dispatch` policy and
Postgres adapter tests and `heartbeat-native-predispatch-retry.test.ts`
(166 together).

Conflict resolved: `task-interruption-presentation.ts` gained
`WAITING_CONTINUATION_MOVE_FORWARD_MS` on master (#21385). Both
constants are kept.

Runner content ID: the Runner change is controller-side TypeScript under
`packages/paperclip-runner/src` only. There is no Rust, `protocol/` or
`generated/` change. Existing Runner images work with this controller.

Daytona: this fix has not run on Daytona at all. The two banners above
are the evidence of the class (G5c and G6c ran on 588c6ba267, without
this fix).

## Risks

- Not proven on Daytona.
- A close can now take up to 30 s longer when the proof read is
unreadable or slow, and up to 3 s longer when the read is stale. This is
on the close path only, not before `turn.start`.
- A probe that does not answer now counts as possibly alive, so the
close sends a suspend instead of treating the Runner as gone.
- Ownership: this PR changes
`server/src/modules/run-dispatch/domain/policy.ts`,
`server/src/modules/run-dispatch/adapters/postgres.ts` and
`server/src/services/heartbeat/run-completion.ts`. CODEOWNERS does not
list these files. Their recent changes come from the restart-recovery
and Retry work (#21348, #21256, #20863, #21302). That area should review
the dispatch-policy change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:32:22 -06:00
Nicky LeachandClaude Opus 5.5 6d8ebc4a85 fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can bind a company secret to an environment variable for an
agent, an environment, a project, or a routine
> - Core resolves these values for each run and gives them to the agent
process or to the Runner
> - Core stores the run output: the run log, the run excerpts, adapter
events, and native (Runner protocol) transcript events
> - The run-output redactor did not know the resolved secret values.
Only failure diagnostics and the `meta.env` keys of `adapter.invoke`
were masked
> - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the
secret in plaintext, and every person who can view the run could read it
> - This pull request gives the resolved values of secret-bound keys to
the run-output redactor and to the native control-plane port
> - The benefit is that an accidental print of a secret does not put the
secret into stored run output, for local and native runs

## Linked Issues or Issue Description

**What happened?**
An agent printed a secret-bound environment variable during a run (for
example with `env` or `echo $STRIPE_KEY`). Core stored the value in
plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in
the transcript events. Anyone who can view the run can read the value.

**Expected behavior**
Core masks the value of each secret-bound environment variable before it
stores run output. Core does not mask plain (non-secret) configured
values.

**Steps to reproduce**
1. Create a company secret and bind it to an agent environment variable
with `secret_ref`.
2. Configure the agent so that its command prints the variable.
3. Run the agent and open the run log. The value shows in plaintext.

**Related**
Refs #21330. That PR keeps configured secrets out of OpenCode's own
storage and model requests inside the Runner. This PR is the Core side:
it masks secrets in the run output that Core stores, for every run type.
The two changes are independent.

## What Changed

- `secretEnvironmentValues(env, secretKeys)` in
`agent-identity-redaction.ts` selects the raw values of keys in
`secretKeys` (the keys that came from `secret_ref` or `user_secret_ref`
bindings). `secretOutputForms` derives the forms to mask: the full
value, the value without outer whitespace, the JSON-escaped form, and
each line of a multi-line value (PEM armor lines are not added). It
skips a form that has fewer than 8 characters after trimming. This is
the same 8-character limit that diagnostic redaction uses for unknown
values.
- The output redactor keeps these secrets in a separate set, beside the
existing agent private-key values. The private-key values are masked
everywhere, as before. Environment secrets are masked only in output
text. They are not masked in protocol fields: `schema`, `type`, `kind`,
`status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids,
refs, codes, revisions, versions, digests, timestamps, and names that
end in these words (for example `eventType`, `turnTerminalState`,
`reportedWorkDisposition`). So a secret that equals a protocol word (for
example `completed`) cannot change an event type or a terminal state.
- The streaming match now scans left to right. At each position it tries
the longest value first. If a longer value could still start at that
position, it holds the text until the next chunk, also when a shorter
value already matches there. So the tail of a longer secret cannot leak
when a shorter secret is its prefix.
- `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to
the run's output redactor immediately after it resolves them. This one
redactor already masks the local adapter log stream, the excerpts, live
log events, adapter events, `adapter.invoke` metadata, the adapter
result, and the canonical native log export.
- For a native run, `executeHeartbeatRuntime` stores the delivered
secrets (only values that have a form to mask) encrypted under a new
context key, `paperclipOutputSecretRedactions`, in one transaction. It
then adds all stored values of the run to the redactor. A run that
reconnects to a live Runner after a `"latest"` secret rotation then
masks the value that the Runner holds and the new value. This store
reuses the encryption and fingerprint logic of the run redaction
registry, but it is separate: API read redaction does not apply its
values, so a secret cannot change ids, event names or terminal states in
API responses. API reads hide the new key like the existing registry
key.
- `executeHeartbeatRuntime` gives the native executor the run's secrets
and the observability credentials (`observabilitySecretValues`).
- The native executor gives the secrets to its failure-diagnostic
redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks
them in each PRP event before it persists the event, with the existing
delta-aware handling for values that are split across `item.delta`
events. This covers native runs on the local Runner and on plugin-hosted
agent computers.
- After a controller restart, a new control-plane port cannot know the
text that the old port held back, because that text is never stored. On
the first delta of each stream it has not seen, the port masks leading
text that matches the tail of a secret (4 or more characters). It skips
text that can start a secret, and it keeps checking the stream while a
delta falls entirely inside such a tail.
- The native executor classifies failures from the original error. It
masks only the agent private key in place, as before. It masks
environment secrets in a separate redactor that it uses only for text
that it writes or logs (`failureDetail`, failure messages, stderr
diagnostics). Core masks the rethrown error before it stores it.
- `coalesceIntoActiveExecutionRun` (wake queue) writes the merged
context with `withRunRedactionHistory`, which takes
`paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from
the locked current row. Before, a coalesced wake built from an older
context could delete either key. This also protects the existing
registry. This is a change outside the secrets module, in the one place
that replaces a running run's context concurrently.
- When an adapter or native run fails, the heartbeat keeps the original
error for retry decisions and rethrows it. The copies that it saves in
`workspace_finalize` metadata (`errorMessage`) and in a failed session
goal action go through the run's output redactor first.
- `docs/deploy/secrets.md` describes the mask and its limits.

## Verification

- `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new,
embedded Postgres): a real `process` adapter run with a `secret_ref`
bound to `"latest"`. The agent prints the secret split across two
writes, in JSON, and on stderr. The stored log, run row, and run events
do not contain the value. The plain value and a 5-character secret are
not masked. The test then rotates the secret and runs again. The second
run masks the rotated value, and a SHA-256 digest in the output proves
the run received the rotated value. Red-green: it fails without the new
line in `execution-configuration.ts`.
-
`server/src/services/native-runtime/paperclip-control-plane-port.test.ts`:
the port masks a secret in a persisted tool event and in a value split
across two `item.delta` events; plain and short values stay (red-green:
fails without the port change). A new test binds secrets equal to
`interrupted`, `completed`, `succeeded` and `satisfied`, then appends a
`turn.interrupted` event, checkpoints, and completes the run. The event
type, payload status, terminal states and criterion status stay
unchanged and the result is saved (red-green: fails when protocol fields
are masked).
- `server/src/services/heartbeat/runtime-execution.test.ts`: a value
stored by an earlier attempt, the current value, and an observability
header value all reach native dispatch and the heartbeat redactor. Short
values are not stored. The shared run redaction registry stays empty,
and an API read hides the new key (red-green: fails without the store
read).
- `server/src/services/native-runtime/native-session-executor.test.ts`:
a native failure that contains the secret is masked in failure writes,
logs and the rethrown error, and the executor gives the secrets to the
control-plane port.
- `server/src/services/agent-identity-redaction.test.ts` (new): value
selection, the minimum length, multi-line and JSON-escaped forms, nested
values, every split point of a streamed value, overlapping secrets
(`abcdefgh` and `abcdefgh1234567890`) split at the shorter value,
protocol fields in events, terminal states and deltas, and secrets added
by a later attempt.
- Restart, retry and coalescing (each fails without its fix): a port
test restarts between two deltas of a secret and stores `echo ` and
`***REDACTED*** done`; an executor test binds `timed out` and keeps
`provider_session_open_timeout`; a wake-queue adapter test keeps both
redaction keys when a coalesced wake replaces the context.
- `server/src/services/heartbeat/run-output.test.ts`: the local output
path masks the log, excerpt, live event, adapter event, and adapter
metadata.
- `pnpm --filter @paperclipai/server typecheck` passes. The changed test
files and the other users of the redactor (`agent-identity`, `run-log`,
`heartbeat-case-handoff`, `native-run-log-export.integration`) pass in
full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test
files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify`
exits 2 (broader verification required) because `docs/deploy/secrets.md`
is a global CI input, so required CI supplies broad coverage.

## Risks

- This is a guard against accidental leaks. It is not a security
boundary. An agent can change a value before it prints it (for example,
encode it in base64 or print one character at a time), and Core cannot
detect that.
- Core does not mask a secret value or line that has fewer than 8
characters. This prevents a short value from erasing ordinary output. A
secret that short is not masked in run output.
- Core does not mask environment secrets in protocol fields (see What
Changed). If a secret appears in such a field, it stays. These fields
hold protocol words and generated identifiers, not agent output.
- If a value is common text of 8 or more characters, Core masks that
text wherever it occurs in that run's output text. This only affects
runs that bind that secret.
- If a key is secret-bound in one source and a later source sets the
same key to a plain value, `secretKeys` still includes the key, so Core
masks the plain value. This is the existing meaning of `secretKeys`.
- For native runs, Core now stores the delivered secrets encrypted in
`contextSnapshot` under `paperclipOutputSecretRedactions`, with the same
local encryption as the existing run redaction registry. It adds one
transaction per native run that has secrets. API reads do not apply
these values and hide the key.
- After a controller restart in the middle of a secret, a tail shorter
than 4 characters can still be stored. A new stream that starts with the
tail of a secret (4 or more characters) is masked at its start even
without a restart. This is a small amount of extra masking.
- Core masks values at write time. Run output that Core stored before
this change is not changed.
- No schema change and no migration. The new fields are optional and
in-process only.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 16:14:41 -08:00
DottaandClaude Opus 5.5 de80299d2c perf(recovery): check only blocked tasks whose blockers are done (#21517)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. One queue,
`resolved-dependencies`, wakes a blocked task when all of its blockers
are done.
> - The queue reloads every blocked task of a company when the company
gets a recovery hint. Every run dispatch and run state change sends that
hint.
> - Each reloaded task gets a dependency check of about five statements,
also when its blockers are still open and nothing can happen.
> - On a hosted instance with 577 blocked tasks and 5 live runs, this
loop ran all the time: five statements at 7.3 per second each.
> - This pull request loads only the blocked tasks whose every blocker
is done. These are the only tasks the check can wake.
> - The benefit is that the cost of this queue no longer grows with the
number of blocked tasks that wait on open work.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The background check that wakes blocked tasks when their blockers are
done (`resolved-dependencies` in
`server/src/services/recovery/work.ts`).

**Current behavior**

A company recovery hint reloads every blocked, agent-assigned task of
the company. Run dispatch and every run state change send this hint, so
with live runs it comes all the time. Each task then gets
`reconcileResolvedDependencyWakeBackstop` with its issue id. That reads
the task, reads the dependency readiness, and then reads the stall wakes
of the task (children, blockers, recovery actions). For a task with open
blockers this finds nothing to do. The queue reloads at most every 30
seconds per hint and works 25 tasks per pass with 1 second between
passes, so with hundreds of blocked tasks it never becomes idle.

**Proposed behavior**

The queue loads only blocked tasks that have at least one blocker and no
blocker that is not done. The per-task check is unchanged, so the
workspace-finalize barrier and the existing-wake dedup still apply. A
blocked task enters the queue when its last blocker becomes done,
because that change sends the company hint and the next reload now finds
it.

Stall wakes for tasks with open blockers come from the company pass of
the `active-recovery-actions` queue (`reconcileActiveRecoveryActions`).
That pass reads every stall of the company with the same predicate, and
it gets the same company hint.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. With 300
blocked tasks behind one open blocker, a company hint now checks 0 of
them instead of 300.

## What Changed

- `server/src/services/recovery/work.ts`: the shared `register` helper
takes an optional extra SQL condition. The `resolved-dependencies` queue
gives a condition that keeps only blocked tasks with at least one
`blocks` relation and only `done` blockers.
- `server/src/__tests__/issue-recovery-deadlines.test.ts`: a new test
with 300 blocked tasks behind an open blocker and 1 blocked task behind
a done blocker. It runs the real coordinator, notification and SQL.

## Verification

- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t
"blockers are done"` (in `server/`). Before the change it fails: the
first pass checks all 301 tasks instead of the 1 ready task. After the
change it passes. The test also shows that the 300 tasks are checked as
soon as their blocker becomes done.
- `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts
src/__tests__/issue-recovery-work-lifecycle.test.ts
src/__tests__/issue-blocker-stall-wakeups.test.ts` pass. The stall test
file covers the stall wakes through the company pass.
- `pnpm check:module-boundaries` passes.
- Local benchmark: the real server against PostgreSQL with statement
logging, one company, 60 agents, about 1,440 open tasks (780 in review,
580 blocked with blocker relations, 50 in progress, 30 to do), with run
churn from the recovery queues. Results are in the table below.

The benchmark table follows in an update of this description.

## Risks

- Low risk. The check itself is unchanged. The change only removes tasks
from the queue that the check skipped as not ready.
- A blocked task whose only blocker is cancelled stays out of the queue.
The check also treated it as not ready, because only a done blocker
resolves a dependent.
- Stall wakes for blocked tasks with open blockers now come only from
the company recovery-actions pass. That pass uses a superset of the
stall predicate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 17:52:03 -06:00
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
4c5e7bf867 fix(heartbeat): retry a legacy Stop that loses the task-row lock race (#21492)
Legacy Stop retries its terminal write up to 3 times when another writer
holds the task row past the 1s lock_timeout, and returns a retryable
409 run_stop_busy instead of a 500 when contention persists. Board Stop
also bounds its earlier applySupersession task-row lock to 1s so it can
no longer wait without limit.

Co-Authored-By: Paperclip <paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:29:44 -08:00
defed64af9 fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction
under a 1 s lock_timeout. When another transaction briefly held a row that
the write needs (for example the agent row that the new recovery action
references), the insert failed with 55P03 and the whole write rolled back.
The run kept its dead lease, and the task needed reconciliation instead of
the recorded automatic resume.

Retry the whole failure write on transient lock contention with the
existing bounded helper. A rolled-back attempt has no effect, so a fresh
attempt is safe. The regression test holds the agent row until one attempt
is cancelled by its lock timeout, so only a retry can complete the write.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:02:04 -08:00
DottaandPaperclip 0ead74b4f3 feat(slack): show useful progress without extra messages (#20827)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - Chat connectors show agent work in the source conversation.
> - Slack progress uses a small set of general status messages.
> - Native runtimes already mark some completed assistant notes as
public progress.
> - The server can select those notes and control their size and rate.
> - This pull request keeps one progress message and sends the final
answer as a new reply.
> - Users can follow useful changes without reading each tool event.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Automatic progress updates in Slack task threads.

**Current behavior**

Users see general messages such as “Maya is making progress”. The first
answer can replace that message. A source update that arrives during the
progress delay can be lost if no later event arrives.

**Proposed behavior**

Show the latest short public note in one message. Wait 10 seconds before
the first update and at least 30 seconds between updates. Recover
pending work after a restart. Send the final answer as a fresh reply.

**Reason and benefit**

Users can see what the agent checks and what it found. The thread stays
short. The model needs no new tool call or summary model.

Related change: [resolved-card chat
bindings](https://github.com/paperclipai/paperclip/pull/20825). This
branch includes that master revision. The progress change uses the
existing source authority checks.

## What Changed

- Select completed `agentMessage` items from the normalized PRP
`progress` channel. Keep unknown channels, reasoning, tool output, and
raw provider fields out of progress text.
- Sanitize each note. Keep at most two sentences and 400 characters. Add
a task link when a public board URL is available.
- Prefer public notes over general tool status. Remove repeated text and
combine bursts into the latest pending note.
- Use run events and the publication outbox as saved state. Use the
shared deadline worker for the grace period and update delay. A delivery
receipt wakes the worker when an earlier update is still in flight.
Settled history does not poll.
- Apply the run's registered-secret filter before text selection or
storage. Reuse the latest usable note within a sweep. Invalidate the
cache when the source event or secret registry changes.
- Reuse the exact source-bound Slack fast-response receipt when it
replaced the working placeholder. Check its current outbound message
link before editing it.
- Send Slack answers as new replies. After delivery, queue a separate
cleanup edit for a generic working message. Keep its current outbound
link as the edit authority. Leave useful public notes in place. Cancel
pending public progress when an answer, question, or terminal run takes
over. Keep the existing delivery policy for other connectors.
- Add one short Slack prompt instruction. Update product documentation
and regression tests.

The Slack module owns public-note selection, cadence, lane eligibility,
and cleanup. The shared progress service holds its existing issue and
run locks and calls a narrow provider policy. The secret registry
supplies redaction and a version token for the per-sweep note cache. The
existing outbox owns delivery, retries, and authority checks. The Slack
module owns receipt reuse. Shared delivery changes supply the receipt
wake and select the Slack reply policy. No schema or configuration
change is required.

## Verification

- The initial content-selection regression failed before the
implementation and passed after it.
- The follow-up reproduced two review defects before the fixes. A
registered synthetic secret appeared in saved progress. A generic
working message stayed active after the answer. Both regressions now
pass.
- The final content, database-worker, query, protocol-boundary, and
redaction selection passes 78 tests across four files.
- The earlier affected transport selection passes all 65 cases. It
covers public progress, native cadence, separate answers, Slack and
Telegram retry isolation, FIFO restart, and cleanup ownership. Its first
run passed 64 cases and found one old assertion that rejected the
intended cleanup edit. The corrected assertion checks the exact
successor message ID.
- Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm
--filter @paperclipai/server typecheck`. An earlier attempt found two
transaction type errors. The helper contracts now accept their actual
database operations. The final run passes all source and test layers.
- The broad Slack selection stopped after 189 passing cases when the
sandbox killed its worker with SIGKILL. That attempt is not a pass. The
focused transport run above supplies affected local coverage. Required
CI must supply broad coverage on the final head.
- `pnpm verify --explain` returns exit 2 because the product
documentation change requires broad CI coverage. No full local suite was
started.
- Preflight installed dependencies and built the SDK. Cargo is absent,
so local Rust checks could not run. No Rust source changed.
- [CI on
`02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242)
passed build, typecheck, policy, all browser shards, Runner/Rust checks,
and the other test lanes. One chat shard failed in three Slack cases.
Each expected one edit after an answer was already sent. The intended
cleanup adds a “Response sent.” edit before the later failure notice.
All three failures reproduced locally. The test assertions now check
both exact edit texts and the original run's message ID. They also check
that another sweep adds no edits.
- The latest local run passes all 85 selected transport cases. It
includes all 18 Slack/Telegram answer and failure orders, plus progress,
deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm
exec vitest run server/src/__tests__/chat-channels.integration.test.ts
-t 'preserves selected answer and failure lanes|public progress|native
progress|keeps intermediate.*agent comments|keeps
a.*retry.*queue|FIFO'`.
- [Final
CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545)
passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed
jobs passed. This includes build, typecheck, general tests, serialized
tests, all eight browser shards, Runner/Rust checks, and the required
aggregates. The affected chat shard now passes. The test-file selection
job was skipped because this run used full coverage.
- [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124)
completed on the same head with 5/5 and no new findings. All four inline
review threads are resolved. The review job and Socket checks passed. Of
61 check runs, 57 succeeded, three were skipped, and one was neutral. No
checks remain pending or failed. The PR is mergeable with a clean state.
- [Superagent Security
Scan](https://github.com/paperclipai/paperclip/runs/132142345827)
returned a neutral result with “PR scan workflow failed”. It supplied no
security verdict. Do not count it as a successful scan.
- Live Slack notification and unread behavior has not been tested.
Service tests use real PostgreSQL and a fake provider runtime.

## Risks

- Only runtimes that map commentary to the explicit public `progress`
channel supply contextual prose. Other runtimes retain general status
messages.
- Slack final answers now create a fresh reply. This is an intended
delivery change.
- Public commentary must suit the current conversation audience. Text
sanitization adds protection; it does not authorize sharing unrelated
private material.
- Source events and outbox rows supply recovery state. No new timer
table or recurring idle scan is added.
- The existing large integration suite retains its provider fixtures and
lifecycle assertions. No source file crosses the 1,000-line threshold in
this change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact
deployment ID was not supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:48:05 -06:00
Devin FoleyandClaude Opus 5.5 06a4a07781 fix(native): settle a Stop seen at budget admission as that cancellation (#21483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a sandbox, then reserve budget, then start
provider work
> - An operator can press Stop while a native run still prepares
> - Budget admission reads the recorded Stop on the locked run row and
throws a generic 409 "Run cancellation already requested"
> - Heartbeat treats that throw as a run failure, so an expected Stop
becomes a failed run and a `run_failure` error event
> - This pull request makes admission report the Stop through the
existing native startup Stop contract
> - The benefit is that a Stop during preparation ends as a
cancellation, with no false failure, no retry, and no stranded Stop
intent

## Linked Issues or Issue Description

**What happened?**
An operator stopped a native run while it prepared. `reserveRunBudget`
found `resultJson.cancellation` / `startupCancellation` on the locked
run row and threw `conflict("Run cancellation already requested")`.
Heartbeat sent this error to `failRun`. The run ended `failed` with
`adapter_failed` or `native_continuation_requires_reconciliation`, and
Sentry got a `run_failure` event with `cancellationExpected: true`. The
pending native Stop intent stayed `pending`. Restart recovery then
logged `native_stop_requires_existing_remote_session` for a session that
never existed. Two staging Sentry groups show this: 20 events since
2026-11-10.

**Expected behavior**
A Stop that the server records before provider work starts ends the run
as `cancelled`. It does not report a failure, it does not schedule a
retry, and it acknowledges the Stop intent.

**Steps to reproduce**
1. Start a native run on a task.
2. Press Stop while the run prepares, before budget admission.
3. See that the run ends `failed` with the error "Run cancellation
already requested".

No public issue or pull request covers this. A GitHub search for
"cancellation already requested" found no related fix.

## What Changed

- `reserveRunBudget` now throws the existing
`NativeCancellationPendingRecoveryError` for a native run when it finds
a recorded Stop. Heartbeat already settles this error through
`cancelRunInternal("Recovered durable native run cancellation")`. This
is the same path as a Stop that native startup sees when the session
opens.
- Legacy runs keep the 409 conflict. The incident evidence is for native
runs only.
- The final dispatch gate does not change. It already treats a recorded
Stop as a lost race, not as a failure.
- New regression file
`server/src/__tests__/heartbeat-native-stop-before-admission.test.ts`.

## Verification

- `npx vitest run
src/__tests__/heartbeat-native-stop-before-admission.test.ts` (in
`server/`, embedded Postgres):
- Before the fix: the test "settles an operator Stop recorded before
budget admission" fails. The run ends `status: "failed"`, `error: "Run
cancellation already requested"`. This is the staging result.
- After the fix: 2/2 pass. The run ends `cancelled`, the provider
session never opens, no retry run exists, any task hold records
`originalFailureCode: "cancelled"`, and the native Stop intent becomes
`acknowledged`.
- The second test is a safety case. A real `heartbeat.cancelRun` that
settles during admission still ends `cancelled`, and the Stop request
resolves.
- Focused suites pass, 13 files and 304 tests: native pre-dispatch
retry, admission and cleanup, budget enforcement scoping, closed native
receipt accounting, native recovery lease reacquire, native cleanup
admission, native startup cleanup recovery, native cancellation request,
native Stop pending, run-dispatch Postgres adapter, companies service,
execution preparation timing.
- `pnpm typecheck` in `server/` passes.
- `pnpm verify` selects a broad set because the change is under
heartbeat. CI supplies that coverage.
- Not verified: a live staging Stop. Staging gets this fix after merge
through the normal automatic deploy.

## Risks

- Low risk. The change applies only when admission finds a recorded Stop
on a native run. Before this change that run always failed.
- `budget-reservations.ts` now imports an error class from
`native-session-executor.ts`. The executor does not import
`budget-reservations`, so this adds no import cycle.
- A user Stop with `dispatched: false` can still put an
`active_run_watchdog` hold on the task. That is existing behavior for
Stops that reach no provider (#18820 keeps the strict dispatched
receipt). This PR does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:47:52 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
DottaandClaude Opus 5.5 3b03ac10e5 fix(retry): answer a refused Retry with its refusal instead of accepted (#21497)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person presses Retry on a stopped task, and the Retry API answers
before dispatch runs
> - The route answered 202 "accepted" when it saved the request.
Admission ran afterwards
> - When admission refused the request for a reason another click cannot
change, the person saw "accepted", no run started, and the task showed
no Retry
> - A Retry that is shown must work, or the answer must say clearly why
it did not start
> - This pull request admits a plain Retry before the route answers
(bounded wait) and answers a refusal as a refusal
> - The benefit is that the API never reports "accepted" for a Retry
that admission refused, and the UI shows the reason

## Linked Issues or Issue Description

Refs #21481 (a fix for one cause of the refusal).

**What happened?**
On a hosted staging instance, two tasks got HTTP 202 `accepted: true`
from `POST /api/issues/:id/retry`. No run was created. The tasks then
showed a failed state with no Retry button and the wait reason
`previous_execution_unverified`.

**Expected behavior**
The Retry answer either reports a Retry that admission took, or it names
the refusal so that the UI can show it. It never reports "accepted" for
a request that admission refused.

**Steps to reproduce**
1. Stop a task's run so that its outcome is unknown (for example, a
recorded provider event in a stopped preparation).
2. Press Retry.
3. See HTTP 202 `accepted: true`, then see that the saved request waits
on `previous_execution_unverified` and no run starts.

**Paperclip version or commit**
`master` at `a48eaaef57`.

## What Changed

- `POST /api/issues/:id/retry` runs admission of a plain Retry (not a
saved-result finalization, not a connected chat retry) before it
answers. It waits at most 5 seconds. If admission takes longer,
admission continues after the answer, and the answer stays "accepted" as
before.
- When admission refused the request for a reason another click cannot
change (`heldRetryOffersRetry` is false), the answer is HTTP 200 with
`accepted: false` and `refused: { reason, message }`. The message is the
same stable text that the task view shows.
- The saved request is kept. If its gate clears later (for example,
saved-work recovery verifies the files), it still starts by itself.
- New `taskRetryRefusal` and `taskRetryWaitMessage` in
`server/src/services/task-interruption-presentation.ts`.
- UI: `taskRetryNotStartedReason` shows `refused.message`; the API type
includes `refused`.
- Tests that asserted 202 for a held Retry now assert the refusal
answer.

## Verification

- Red without the route change, green with it: `cd server && npx vitest
run src/services/legacy-preparation-retry-hold.test.ts` (new test: a
stopped preparation with a recorded provider event; the route answers
200 with `refused.reason: previous_execution_unverified`, and the saved
request stays deferred).
- Red without the hook change, green with it: `cd ui && npx vitest run
src/hooks/useTaskRetry.test.tsx`.
- Updated and passing:
`src/services/workspace-recovery-preparation-gates.test.ts` (48),
`src/services/workspace-recovery-preparation-proof.test.ts` (53),
`src/services/workspace-recovery-preparation-dispatch.test.ts` (25).
- Passing unchanged:
`src/__tests__/issue-scheduled-retry-routes.test.ts` (34),
`src/services/unrecoverable-saved-work-retry.test.ts` (5),
`src/services/workspace-recovery-warm-handoff.test.ts` (44),
`src/__tests__/heartbeat-native-restart-suspension.test.ts` (115),
`src/__tests__/issue-agent-mutation-ownership-routes.test.ts` (229),
`src/__tests__/agent-cross-tenant-authz-routes.test.ts` (18).
- Not run locally: `src/__tests__/chat-channels.integration.test.ts`
(did not finish on the loaded machine). CI covers it.

## Risks

- The Retry answer can take up to 5 seconds longer when admission is
slow (for example, when the agent's start lock is held). Before, the
route answered at once and admitted in the background.
- A refused request can still start later by itself when its gate
clears. The answer then said "not started", but the task view updates
when the run starts.
- API change: a refused Retry is now HTTP 200 with `accepted: false` and
`refused`. The board UI handles it. Other clients that only check
`accepted` see a refusal, which is correct.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:45:03 -06:00
4cf04d64cb fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts a Runner in a remote sandbox for each
agent run. A bounded same-run retry (at most 3 startup attempts) handles
transient startup failures.
> - A server restart can stop a run during its first Runner bootstrap.
Restart recovery then claims `bootstrap_incomplete`. It stops and
quarantines the unfinished bootstrap and launches a fresh bootstrap on
the same run.
> - The executor builds its "unclaimed bootstrap" proof only when
`restartRecovery` is not set. Thus the fresh bootstrap of a
`bootstrap_incomplete` restart can never prove that a transient failure
is safe to retry.
> - A transient HTTP 502 from the Runner endpoint request in that fresh
bootstrap was then classified as `ambiguous_state`. The run stopped and
needed manual board recovery.
> - This pull request gives the unclaimed-bootstrap proof to a
`bootstrap_incomplete` restart as well as to a first attempt.
> - The benefit is that a transient endpoint failure after a restart
uses the bounded automatic retry and does not need a person to recover
the run.

## Linked Issues or Issue Description

Refs #20414 (related: it fixes executor admission of a
`bootstrap_incomplete` restart at state-root migration. It does not
change the retry evidence that this PR changes.)

**What happened?**
A server restart interrupted the first Runner bootstrap of a remote
native run. Restart recovery claimed `bootstrap_incomplete` and launched
a fresh bootstrap on the same run. The fresh bootstrap registered its
controller checkpoint. Then `runner.ingress.acquire` failed with HTTP
502, before any Runner connection or provider event. The run ended with
`nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3,
and the next action "resolve the ambiguous session state". The error was
`Runner connection endpoint could not be acquired (HTTP 502).`
(`runner_ingress_acquisition_failed`).

**What did you expect to happen?**
The transient 502 occurred before the Runner connected and before any
provider work. It should use the bounded same-run startup retry, as it
does on a first attempt.

**Steps to reproduce**
1. Start a native run on a remote sandbox target.
2. Restart the server during the first Runner bootstrap, so that restart
recovery claims `bootstrap_incomplete`.
3. Make the next `runner.ingress.acquire` call return HTTP 502.
4. See that the run stops as ambiguous state and does not schedule a
startup retry.

## What Changed

- Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`.
It returns true for a first attempt and for a `bootstrap_incomplete`
restart. It returns false for `reattach_existing_runner`,
`reattach_remote_runner` and `resume_dead_runner`, because these
continue an existing Runner or provider session.
- In `native-session-executor.ts`, use this helper in place of
`!input.restartRecovery` when the executor builds the
unclaimed-bootstrap evidence for a startup failure.
- Add `native-bootstrap-proof.test.ts` for the helper.
- Add a case to `native-startup-retry.integration.test.ts`. It runs the
real executor under a `bootstrap_incomplete` restart claim, and the
fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before
the Runner connects.

## Verification

- `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart
of an incomplete bootstrap" failed before the fix and passes after the
fix (5/5 tests pass, vitest 5.1.0).
- `native-startup-retry.integration.test.ts`, case "transient 502 after
a bootstrap_incomplete restart": the run must settle as
`retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1
and a set `nextAttemptAt`. Without the executor change, the executor
builds no bootstrap evidence and the run settles as `ambiguous_state`.
CI runs this suite; I could not run it locally.
- The other guards for a retry did not change. The checkpoint must be
unclaimed, there must be no provider events and no session checkpoint,
and the 3-attempt startup budget applies. If the earlier attempt opened
a provider session, provider events exist and the run still fails closed
as ambiguous.
- I could not run the full typecheck and the real-Runner integration
suites locally (no Rust toolchain in the sandbox). CI must run them.

## Risks

- Low risk. The change only affects a `bootstrap_incomplete` restart
whose fresh bootstrap fails during startup. Such a failure now gets the
same retry decision as a first attempt. All the other evidence checks
still apply, so a run with provider activity still stops for manual
recovery.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edit, and code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:37:24 -08:00
DottaandClaude Opus 5.5 532e502791 fix(heartbeat): drain native resume dispatches whose timer already fired (#21479)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs agent work and resumes interrupted native
sessions on timers
> - `drainActiveRunExecutions()` lets a caller that tears down shared
state (a test cleanup) wait until no background run work is in flight
> - The drain cleared resume timers that had not fired yet, but it could
not see a timer that had already fired and was still reading and
claiming run state
> - That untracked work raced the `fresh-continuation-fallback.test.ts`
cleanup TRUNCATE and deadlocked it, which failed Cloud readiness and
held releases on 2026-11-13
> - This pull request makes the drain wait for fired resume dispatches
too, and removes the deadlock retry from that suite's cleanup
> - The benefit is a deterministic cleanup: nothing still writes run
tables when the drain returns, so the release gate no longer fails on
this test

## Linked Issues or Issue Description

No public issue exists. The bug is described here.

**What happened?**
Cloud readiness `verify / General tests (server (5/10))` failed on
master commits 18fb6ee571 and 5ede64f247. The `afterEach` TRUNCATE in
`fresh-continuation-fallback.test.ts` failed with `deadlock detected`
(40P01). Another backend held a lock on one run table and was waiting on
a table that the TRUNCATE had already locked. #21375 added a drain and
wrapped the TRUNCATE in `truncateTablesWithDeadlockRetry`. The retry
hides the race but does not remove it.

**Root cause**
`scheduleNativeSessionResumeDispatch` arms a timer. When it fires, its
async body calls `getSchedulingSuppression`, then
`recoverNativeRunsAfterRestart` or `dispatchNativeSessionResumptions`.
All of these read and claim run rows before any execution is registered.
Recovery arms an immediate timer (`new Date()`).
`drainActiveRunExecutions()` only cleared timers that had not fired. It
did not track the fired body, so it returned while that work still ran.
A failed dispatch could also re-arm a 30-second timer after the drain
had cleared the timers.

**Expected behavior**
When `drainActiveRunExecutions()` resolves, no heartbeat background work
writes to the database.

**Steps to reproduce**
The new test `heartbeat-drain-native-resume-dispatch.test.ts` holds an
ACCESS EXCLUSIVE lock on `instance_settings` and `heartbeat_runs`. It
arms an immediate resume dispatch and waits until a backend is blocked
behind that lock, which proves the timer fired. Then it starts the
drain. On the old code the drain resolves at once. The leaked query then
also outlives the test and fails at teardown with `CONNECTION_ENDED`.

## What Changed

- `heartbeat.ts`: the fired resume-dispatch body is tracked in a
module-level set, `activeNativeResumeDispatches`, shared by all service
instances like the other drain sets. `drainActiveRunExecutions()` awaits
that set. It still clears only the timers armed before it started. A
timer armed during the drain that is already due (for example the
immediate dispatch a run arms when its executor detaches) counts as
pending work: the drain yields so it fires, then drains what it starts.
Timers due later keep their deadline. A first version cleared timers on
every pass, and CI showed that this cancelled the self-detach takeover
in `heartbeat-native-restart-suspension.test.ts`.
- `heartbeatService` exposes `scheduleNativeSessionResumeDispatch` so
the regression test can drive it. Recovery already uses it.
- `fresh-continuation-fallback.test.ts` uses a plain TRUNCATE again. A
future leak fails the test instead of being retried away.
- New test `heartbeat-drain-native-resume-dispatch.test.ts`, described
above.

## Verification

- Red: the new drain test fails on the old `heartbeat.ts` (`expected
true to be false`: the drain returned while the dispatch was blocked).
- Green: the new drain test passes with the fix.
- Repeated runs with the fix and the plain TRUNCATE (no retry): 50/50
for "runs a person's Retry pressed before delivery…", 50/50 for "…old
lease is pending_cleanup", and 80 runs of "…old lease is active". No
deadlock occurred.
- In the loaded local runs, 11 of the 80 "active" repeats hit the 30 s
hook timeout. I logged `pg_stat_activity` every 5 s during cleanup. In
each case the drain had finished within 5.6 s. The TRUNCATE was `active`
with no lock wait and no other open backend: it was slow cascade I/O on
a host at load average 30 to 60. These were not leaks.
- `heartbeat-native-restart-suspension.test.ts`, the drain test and
`fresh-continuation-fallback.test.ts` together: 188/188 locally.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- The drain now also waits for fired resume dispatches. These dispatches
are short. A slow one makes a test cleanup wait longer instead of racing
it.
- Production behavior does not change. Only callers of
`drainActiveRunExecutions()` see the difference.
- Follow-up, not in this PR: four other heartbeat suites still use
`truncateTablesWithDeadlockRetry`. They may no longer need it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:36:03 -06:00
DottaandClaude Opus 5.5 342fb308bd perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process stores every event of every running native agent
in `heartbeat_run_events`
> - Text deltas (`item.delta`) are about 85% of that process's SQL
statements: each delta is one transaction of four statements on the
shared 10-connection pool
> - At 50 agents and 20 deltas per second, the process falls behind and
unrelated agent requests wait for the pool
> - This pull request commits text deltas in batches: one transaction
for all deltas that arrive within 250 ms, with one multi-row insert
> - The benefit is about 7 times fewer transactions and 5–6 times fewer
statements per delta, with no delay for the first text of an item

## Linked Issues or Issue Description

Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a
of the agent scaling plan in #21418.

**What existing behavior does this improve?**
The persistence of native text deltas. `native-session-runtime` awaited
`PaperclipControlPlanePort.appendEvent` for each event, and the port ran
one transaction per event.

**Subsystem affected**
Server run event ingest (`paperclip-control-plane-port.ts`,
`heartbeat-run-events.ts`) and the runner package's turn consumer
(`native-session-runtime.ts`).

**Current behavior**
Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In
the load model with 50 agents (deltas only), that is about 4.0
statements and 1.0 transaction per delta.

**Proposed behavior**
The turn consumer starts the next append without waiting for a delta's
receipt. The port collects deltas and commits them in one transaction
with one insert. In the same load model, that is about 0.7 statements
and 0.14 transactions per delta.

**Reason and benefit**
Fewer transactions free the shared pool and the main thread for other
agents' requests. This is the largest per-agent cost in the scaling
plan.

**Breaking changes**
None. The row shape does not change. A port without the new capability
keeps the earlier serial behavior.

## What Changed

- `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends
native source events of one run and one source in one transaction. When
the events continue the stored source cursor (from #21435), it sends
four statements for the whole batch: the run lock, one replay probe, one
`insert ... select from jsonb_array_elements(...)` of every row, and the
commit. Any other batch (a replay, a gap, an event with a wake, or a run
without a cursor) appends its events one by one in the same transaction,
with the single-append checks.
- **Storage choice: one row per delta, written by one statement per
batch.** I did not store one row per batch. The transcript, live events,
run log export, chat progress, replay (`replayEvents`), recovery cursors
and the UI all read one row per delta. One statement per batch keeps
every reader unchanged, and rows written before this change read the
same.
- `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An
`item.delta` appended with the new `streaming` option goes into a queue:
- It commits at once until a delta with text commits after the latest
other event. So the first text of each item, and the time to first
token, never wait for the timer.
- Later deltas commit after at most 250 ms, or at once when the batch
has 100 deltas.
- Every other event, `checkpointSession`, `completeRun`, `replayEvents`
and `provesNoRunProviderWork` first commit the queue. A non-streaming
event is never committed before the deltas that preceded it.
- A delta appended without `streaming` commits at once, so callers that
wait for each receipt are not slowed.
- The receipt of a delta settles only after its batch commits and its
observers (live event, executor callback) run, in order.
- If a batch fails, every delta in it rejects and nothing from it is
stored.
- `native-session-runtime` `consumeTurn` appends deltas with `streaming`
and does not wait for their receipts (at most 256 pending). Before any
other event, any result, and the governed-wait settlement, it calls
`flushStreamingEvents()` and then waits for all pending receipts, so a
boundary never waits for the timer. A failed delta stops the turn before
a later event is appended. The consumer joins every pending delta append
before it settles, also when the provider stream fails.
- If one observer of a committed batch fails, only that delta's receipt
rejects. The other rows are still published and observed.
- `ControlPlanePort` contract: optional `pipelinesStreamingEvents`,
`flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`.

## Verification

- Red/green. The new tests failed before the change and pass after it:
- Runner: "starts text delta appends without waiting and flushes them
before any other event" (`native-session-runtime.test.ts`). The port in
the test has no timer, so the turn finishes only if the boundary
flushes. Before: the appends were serial, and later the boundary waited
for the timer.
- Runner: "joins pending delta appends before a failed turn settles".
Before: the turn settled while a delta commit still ran.
- Port: "observes every committed delta of a batch when one observer
fails".
- Port: "commits the first text delta at once and later deltas in one
batch before the next other event". It checks one transaction and one
insert for five deltas, flush before a non-streaming event, receipts
only after commit, and immediate commit of the first delta with text.
Before: each delta committed alone.
- Port: "flushes queued deltas on the timer and before a replay read,
and rejects every delta of a failed batch". It checks the
crash-before-commit case: a revoked writer fails the batch, nothing is
stored, and the resent deltas commit idempotently (the stored one is a
duplicate).
- Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`:
one lock, one probe and one insert per batch; replay idempotency and
payload conflict; a batch that fails before commit stores nothing and
its resend commits; the one-by-one path for no cursor and for gaps.
- Focused suites pass: `heartbeat-run-event-atomic-append`,
`heartbeat-run-event-sequencing`, `paperclip-control-plane-port`
(including the package conformance suite),
`paperclip-semantic-conformance`, `native-observer-replay`,
`native-event-boundary-golden`, `live-recovery-work`,
`native-interaction-bridge`, `chat-native-progress-work`, and
`native-session-runtime.test.ts` in the runner package.
- `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript`
and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm
check:module-boundaries` passes.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418,
`AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls
the store directly, so for this run I added a local `BATCH_MS` mode that
applies the port's batching rules (first delta at once, 250 ms timer,
100 cap, flush before other events) and does not wait for delta
receipts. That mode is not in this pull request. The machine was very
busy (load average 40–60), so absolute figures are noisy; per-delta
ratios are the stable result:

| Deltas only | Statements/s | Transactions/s | Deltas done of offered |
Statements per delta | Transactions per delta | Lateness p50 / p99 |
  | --- | --- | --- | --- | --- | --- | --- |
| One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 |
1.00 | 3,074 / 5,384 ms |
| Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 /
2,858 ms |
| One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 |
3.99 | 1.00 | 6,558 / 11,464 ms |
| Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17
| 187 / 556 ms |

With all sources on, the machine was saturated (event-loop utilization
0.96–0.98) in both modes and the results were not usable (764 and 12,800
of 20,000 deltas done).

## Risks

- **Ack and crash (read first).** In the live runnerd path the runner's
ack does not wait for the `heartbeat_run_events` row today.
`DurablePrpControlPlane.#event` acks after the in-memory journal commit.
Streaming events are not in the controller checkpoint. So a delta that
is acked but not yet appended is lost if the server crashes, and the
runner does not resend it. This is the earlier behavior. This pull
request makes that window up to 250 ms longer. The final text still
survives through `item.completed`, which commits immediately and in
order. This pull request does not change `durable-prp-control-plane.ts`.
- Live text in the UI arrives in steps of up to 250 ms after the first
text of an item.
- A delta failure now surfaces at the next non-delta event or at the end
of the turn, not at the delta. The turn stops before it appends a later
event.
- The governed-wait drain and the control-event replay still append one
event at a time. They do not carry deltas.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:34:01 -06:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
Devin FoleyandClaude Opus 5.5 2756be2f32 fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent setup lists the models that an OpenRouter key can select. The
server gets this list from the public OpenRouter catalog.
> - The catalog request has its own 10 second deadline. When the
deadline expired, fetch rejected with a raw `TimeoutError` (a
`DOMException`).
> - The error handler answered 500 "Internal server error" and sent a
crash report with no first-party stack frames. The user did not see the
existing retry guidance.
> - A slow or unavailable third-party catalog is not a fault in the
Paperclip server. It must not create a crash report.
> - This pull request maps catalog outages to a 503 response with the
existing retry message and a stable code. Only those errors skip crash
reporting.
> - The benefit: users see "Retry or enter a model ID manually", and
Sentry does not get noise from a provider outage.

## Linked Issues or Issue Description

**What happened?**

A staging server answered `GET
/api/companies/:companyId/adapters/opencode_local/models` with `500 —
The operation was aborted due to timeout`. At the same time, other
outbound requests from the host were slow. Sentry recorded
`TimeoutError: The operation was aborted due to timeout`. The event had
only Node internal frames (`AbortSignal.timeout`).

**Expected behavior**

The route returns a clear "catalog unavailable" response. The UI shows
the retry guidance. A third-party outage does not create a crash report.

**Steps to reproduce**

1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more
than 10 seconds), or make it unreachable.
2. Request the OpenCode model list with `provider=openrouter`.
3. See a 500 response and a Sentry `TimeoutError` event.

**Paperclip version or commit**

`e23549cd9c631e4595a3213e03d0c960360452c6`

I searched for duplicate PRs. #17083, #17961 and #10290 change the
OpenRouter catalog contents or model validation. They do not change how
the server handles a catalog timeout.

## What Changed

- `server/src/services/openrouter-models.ts`: The catalog now maps three
failure types to `HttpError(503)` with code
`openrouter_catalog_unavailable` and the existing retry message: its own
deadline, a classified transport failure (uses
`classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A
module-local `WeakSet` records only the errors that it creates for these
outages. `isOpenRouterCatalogUnavailable()` reads that record.
- Other non-OK statuses become a 502 with code
`openrouter_catalog_error`. These are still reported. Malformed JSON and
an invalid catalog shape are also still reported.
- `server/src/middleware/error-handler.ts`: The handler does not send a
crash report for an error from `isOpenRouterCatalogUnavailable()`. It
still answers 503 with the message and code. The new check is next to
the existing MCP connection-failure exclusion.
- Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for
the failures that stay reportable. A route-level test sends a stalled
catalog request through the real error handler. It checks for a 503
response and no crash report. It also checks that a different 503 is
still reported.

## Verification

- `vitest run server/src/services/openrouter-models.test.ts`: 7 tests
pass. I ran this file alone because the full workspace install did not
fit in the agent sandbox disk.
- Red check: with the original `openrouter-models.ts`, 5 of the 7 tests
fail. The deadline test receives `DOMException TimeoutError`, which is
the same error as the production event.
- `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs
the full workspace dependencies. CI runs it.

## Risks

- Low risk. Successful responses do not change. The cache and in-flight
deduplication do not change.
- The failure status changes from 500 to 503 (or 502) for the three
routes that read this catalog: adapter models, fast-response models and
prompt-injection-prevention models. The UI already shows
`error.message`, so users now see the retry guidance in place of
"Internal server error".
- The crash-report exclusion is narrow. It applies only to error objects
that this module created after an actual outage. A copied message or
code does not match.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry, Railway and
GitHub connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:00:24 -08:00
DottaandClaude Fable 5.2 1a96247480 fix(recovery): continue a failed continuation again instead of escalating it to the board (#21394)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic task recovery continues a task when its run ends for a
reason that is not a person's to fix
> - A continuation or recovery run can itself fail before it does any
work: a sandbox create timeout, a full sandbox disk, a lease mismatch, a
provider pack check, or a database deadlock
> - Today the stranded reconciler treats such a failure as non-retryable
and blocks the task for the board. The assigned-work sweep then leaves
the board hold alone, because its list of temporary causes is a short
allow-list. The held backlog pass uses a similar allow-list. Nothing
continues the task, and it shows "failed, Retry"
> - This pull request makes the cause test "not owned by a person" on
these paths, and routes a failed continuation run back through the
fresh-continuation fallback
> - The benefit is that such a task continues from its saved work, under
the existing hourly cap, and nobody needs to press Retry

## Linked Issues or Issue Description

**What happened?**
A task was continued automatically after an interruption. The
continuation run failed during setup, before any provider turn (for
example `setup_failed` with a sandbox HTTP timeout). The stranded
reconciler then set the task to `blocked` with a board-owned recovery
hold. The assigned-work sweep left the task to that hold, and the
fresh-continuation fallback never owns that hold kind. The task stayed
blocked and showed a Retry button. Settled holds whose run was a
recovery run that failed with `adapter_failed`, `setup_failed` or a
provider pack error stayed in the same state, because the held backlog
pass accepted only restart-class codes.

**Expected behavior**
A run that ends for a reason a person does not own is continued again
from saved work. The continuation goes through the fresh-continuation
fallback, so the old workspace epoch is fenced or a new isolated
workspace is used. The per-task limit of 6 automatic continuations per
hour still applies. At the limit the task waits for the next window and
is not blocked.

**Steps to reproduce**
1. Record a fresh continuation for an interrupted run of an
agent-assigned task.
2. Let the continuation run fail with `setup_failed` before provider
work.
3. Run the stranded reconciler. The task becomes `blocked` with a board
hold.
4. Run the assigned-work sweep. It returns `left_to_hold_owner`. No
continuation follows.

**Paperclip version or commit**
master at 6917d0568d

**Deployment mode**
Hosted (two servers can run at once during a deploy)

## What Changed

- New `continuation-cause.ts` holds the cause classification:
`PERSON_OWNED_FAILURE_CODES` (moved, unchanged), a new short list
`PERMANENT_FAILURE_CODES` (low-trust boundary codes,
`adapter_engine_unavailable`, `agent_not_invokable`, `agent_not_found`,
`agent_interaction_delivery_denied`), `isSystemOwnedFailureCode`, and
`isAutomaticContinuationRun` (a retry, scheduled retry, fresh
continuation, sweep or recovery run; never a person's Retry).
`fresh-continuation-fallback.ts` re-exports what callers use.
- `stranded-hold-continuation.ts`: the sweep continues a recent (24
hours) board hold when the failure cause is system-owned, not only for a
short allow-list. A reusable sandbox resume failure still needs a
transient cause.
- `recovery/service.ts`: the stranded reconciler does not escalate an
agent-assigned `todo` or `in_progress` task to the board when its latest
run was an automatic continuation or recovery run that ended within 24
hours with a system-owned cause. The assigned-work sweep owns that task.
A person's Retry, a person-owned cause, a permanent cause, or an older
failure escalates as before.
- `assigned-work-sweep.ts`: for a failed automatic continuation run, the
sweep asks the fresh-continuation fallback first. A recorded
continuation is the repair (fenced epoch or new isolated workspace). On
the fallback's hourly backoff the sweep records "not started" with the
retry time and keeps the task unblocked. While exact recovery still
waits, it records the retry time. A person-owned reason from the
fallback is written on a refused attempt.
- `task-automatic-attempts.ts`: the shared 6-per-hour ledger also counts
`server_restart` and `interrupted_preparation` scheduled retries, so the
sweep cannot add attempts on top of those lanes in the same hour.
- `fresh-continuation-fallback.ts`: the held backlog pass also accepts a
source run that was an automatic continuation or recovery run with a
system-owned code (24 hours, 5 per pass, 10 per minute unchanged). With
no hold to carry a pending check, a wait is measured from the source
run's end, so it is bounded.
- `fresh-continuation-fallback.ts`: rechecks the cause with the shared
classifier and applies the shared hourly cap inside the task lock. A
continuation keeps the old writer's workspace unless a wait was actually
cut short (bound measured from a recorded check, or a wait skipped).
- New test file `failed-continuation-continues-again.test.ts` (15
tests).

### Changed test expectations

- `interrupted-preparation-hold-settle.test.ts`, "bounds repeated lost
leases before provider work" (both cases): the lane cap still holds (2
and 12 successors). The end state is no longer `blocked`: the task keeps
its status, no further run exists, and the sweep returns a deferral with
a future retry time (`deferred_cap` at 12 ledger attempts,
`deferred_rate_limit` spacing at 2). The fixture now settles the
scheduled wakes and retry times the loop leaves behind, so the sweep
reads the real state. A person's Retry is still accepted.
- Same file, "blocks the task when a replaced preparation's lane is
spent" is renamed "leaves the task to the assigned-work sweep, not
blocked, when a replaced preparation's lane is spent". It still expects
no successor run, and now expects `todo` with no active hold.
- `heartbeat-process-recovery.test.ts` and `task-retry.test.ts`: no
change. Their runs either ended months before (outside the 24-hour
window) or were a person's Retry, so they escalate as before. The bound,
no-duplicate, and redaction tests keep their assertions.

## The one classifier

`continuation-cause.ts` `continuesAutomatically(run)` (and its SQL twin
`continuesAutomaticallySql`) decides whether a failed run continues on
its own:

- the cause is system-owned: not in `PERSON_OWNED_FAILURE_CODES`
(unchanged), not in the new `PERMANENT_FAILURE_CODES`, not an external
execution whose stop cannot be confirmed (Dot and Muse, the same rule
the retry scheduler uses to decline them permanently), and a reusable
sandbox resume failure only with a transient cause; and
- the run was an automatic continuation or recovery run (a scheduled
retry, a recovery wake, a fresh continuation; never a person's Retry),
or a first run that ended with a restart-class code or a transient setup
failure before provider work (the pre-existing allow-list for the
backlog and the board-hold takeover).

A person's failed Retry keeps master's outcome at every entry point: the
classifier never treats it as an automatic run. Whether it should
continue on its own is a separate product decision.

## Every entry point and the cap it uses

Shared cap = `sharedContinuationAttempts` in
`recovery/continuation-ledger.ts`: 6 per task per rolling hour, counting
fresh continuation records, settled sweep wakes, and transient, restart
and preparation retries. Only attempts that started or created a run
count.

- Disposition pass (`settleUnrecoverableExecutions` to
`continueFreshInTransaction`): cause test of the classifier and the
shared cap, inside the fallback under the task lock. Run kind is not
required on this path: it handles fresh failures of any run.
- Held backlog (`continueHeldExecutions`): `continuesAutomaticallySql`
in the listing; then the fallback's cause test and shared cap. Pacing 5
per pass, 10 per minute, 24 hours.
- System continuation wake in the queue (`heartbeat/queue.ts`,
unchanged) to the fallback: cause test and shared cap.
- Dispatch refusal (`run-dispatch` Postgres adapter, unchanged) to the
fallback: cause test and shared cap.
- Sweep board-hold takeover (`strandedHoldContinuesAutomatically`):
`continuesAutomatically`, 24 hours.
- Sweep route to the fallback (trigger `assigned_work`): recovery run
with a system-owned cause, then the fallback's cause test and shared
cap; the sweep claim also applies its ledger.
- Old reconciler deferral (`assignedWorkSweepOwnsFailedRun`): automatic
run with a system-owned cause (`canContinueRecoveryRunAutomatically`),
24 hours. First runs keep their existing escalation and budgets.
- Sweep plain wake (unchanged path, for first runs): the sweep claim
ledger (`taskAutomaticAttempts`: started sweep attempts, scheduler
retries, and fresh records from other paths), spacing from every
attempt.

## Self-review: places where the rule still differs, on purpose

- The fallback's fresh path continues a first run with any system-owned
cause, while the backlog limits first runs to restart-class and
transient setup codes. The backlog limit is the existing herd rule for
records older releases left behind.
- The fallback treats a pending question as a person's wait; the sweep
treats a question asked before a person's Retry as superseded. In that
one shape the sweep sends its ordinary wake, as before.
- The scheduler lanes (transient, restart, preparation) keep their own
bounds and do not read the shared cap before scheduling; their runs
count in it, so every other path stops once they fill the hour. Changing
the scheduler is outside these files.
- The sweep's plain wake path does not test the cause; it relies on the
invariant's waiting reasons, as before.

## Waits and their bounds

- Every exact-recovery wait in the fallback (settling execution, native
finalizer, live controller lease, preparation lane, saved-work recovery,
workspace restore owner, retained-source check, previous local process):
it starts at the first recorded pending check on the run's hold. When no
check is recorded (no hold, a hold nobody checks, or a replaced record),
it starts at the source run's end (`finished_at`, a database row that
every server reads the same). Bound: 10 minutes idle, 15 minutes while
the owner shows progress, 30 minutes while saved-work recovery advances.
Then the fallback continues from saved work.
- A source run that ended more than 24 hours ago with no recorded check
does not wait: the fallback answers `retry_needed` (the existing named
state that leaves an old hold for a person's Retry), so nothing waits
silently and the 24-hour rule holds.
- The sweep on a fallback backoff: waits until the oldest continuation
leaves the hour (6 per task per hour), task stays unblocked.
- The sweep while exact recovery waits: rechecks at the fallback's retry
time (at most 4 minutes), until the bound above.

## Decisions on permanent causes

- `PERSON_OWNED_FAILURE_CODES` is not changed. A separate
`PERMANENT_FAILURE_CODES` list keeps causes that cannot heal by
themselves out of the new automatic paths, so they still block and tell
a person. `repository_catalog_unavailable` was transient in the old
allow-list too; in the failing Retry test it was a person's Retry, which
the reconciler still escalates.

## Verification

- `failed-continuation-continues-again.test.ts`: 28 tests, including a
12-case table (person's Retry, automatic continuation, first assignment;
a person's Retry gets master's outcome everywhere) x (setup failure,
restart code, permanent code, person-owned code) through the backlog,
the sweep takeover and route, and the reconciler, and a test that
waiting checks spend no attempt (red before the fix for the first-run
case and for the cap); earlier: 15 tests (the latest three added
red-first: permanent codes and a person's Retry stay blocked on the
backlog; no entry point continues a person-owned or permanent code; six
mixed attempts stop the backlog until the next window; plus a pin that
the ledger uses the restart lane's window, cap and reason). Earlier:
Before the fix, the 10 behavior tests failed for the expected reasons
(board escalation, `left_to_hold_owner`, no backlog continuation,
endless wait with no hold, endless wait on an unchecked hold at 20
minutes and 30 hours, checked alternately by two sweep instances). The
person-owned guard passed before and after. After: 11 of 11 pass. After
the last change, alone: `fresh-continuation-fallback.test.ts` 72/72,
`fresh-continuation-move-forward-offer.test.ts` 14/14,
`assigned-work-invariant.test.ts` 31/31,
`interrupted-preparation-hold-settle.test.ts` 29/29.
- Locally after the change, one file at a time:
`assigned-work-invariant.test.ts` 31/31,
`recovery/assigned-work-invariant.test.ts` 50/50,
`interrupted-preparation-hold-settle.test.ts` 29/29,
`task-retry.test.ts` 117/117, the 10 previously failing
`heartbeat-process-recovery.test.ts` tests 12/12 (filtered run),
`fresh-continuation-fallback.test.ts` 66/72 in a loaded run with 6
timeouts at 30 to 90 s that pass alone (8/8 filtered).
- `tsc --noEmit` for `server`: clean.
- CI supplies the full suite.

## Risks

- Behavior change: a recent failure with a cause outside the
person-owned list is no longer escalated to the board by the stranded
reconciler. The sweep continues it instead, at most 6 times per hour per
task, then once per window. Some codes outside that list may be
configuration a person must change (for example `low_trust_*`,
`adapter_engine_unavailable`, `agent_not_found`). Such tasks now retry
at the capped pace instead of blocking. The person-owned list is not
changed in this pull request.
- Upgrade: tasks blocked by an older release with a board hold for such
a cause, within 24 hours, are continued by the sweep on its first
passes. The sweep's 10 repairs per minute per process, the delivery
pacing, and the hourly cap bound the burst.
- Two servers: the sweep's claim and the fallback each run under the
task row lock, and the fallback is idempotent per source run.

## Model Used

- Claude Fable 5.2 (Anthropic), via Claude Code with tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:49:16 -06:00
DottaandClaude Opus 5.5 b5adf35119 test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner (`paperclip-runnerd`) that the
server launches in a sandbox.
> - A deploy can stop the server after the launch command reached the
sandbox but before the server read back the Runner's process identity (a
lost acknowledgement).
> - The database then has no PID for the run, so the successor server
launches the Runner again on the same state folder.
> - Two Runners must never serve the same run's state. #20852 proposed a
large launch-intent design for this window. Since then, #21079 added a
fence that stops a leftover Runner on the same state folder before each
launch.
> - This pull request adds a Linux regression test that drives the real
launcher through the lost-acknowledgement window.
> - The benefit is that a future change to the launch path cannot bring
back a second Runner without failing this test.

## Linked Issues or Issue Description

Refs #20852 (superseded: current master already prevents the duplicate
Runner through #21079). Refs #21079.

**What happened?**
#20852 claimed that a relaunch after a lost launch acknowledgement could
start a second Runner for the same run. Nothing proved whether current
master still has this problem.

**Expected behavior**
After a lost acknowledgement, the successor's relaunch stops the
orphaned Runner. Exactly one Runner serves the run's state folder, and
the process marker names it.

**Steps to reproduce**
1. Launch a remote Runner, and lose the launch RPC's acknowledgement
before the identity read (the old server stops).
2. Relaunch the same run on the same sandbox from a new launcher (the
successor).
3. Count the live `paperclip-runnerd` processes that serve the run's
`--state-dir`.

## What Changed

- New
`server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`.
It is a test only and changes no product code.
- It runs only where `/proc` exists (Linux CI), like the existing #21079
fence tests.
- It drives the exported `createRemoteRunnerProcessLauncher` with a
local command runner. The old launcher's runner runs the launch command,
then throws `remote_launch_ack_lost`. After that, every command fails,
because the old server is gone.
- A small bash stand-in for `paperclip-runnerd` keeps the Runner's
argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty
lineage.
- The test checks these results: the orphan was alive and had no PID on
the handle; the successor's Runner has a new PID; the orphan stops;
exactly one Runner serves the state folder; the marker names the
successor's PID.

## Verification

- Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact
outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I
extracted from this head:
- master scripts: `runners after successor relaunch [61], orphan alive
false`. Result: one Runner (PASS).
- the same scripts without the #21079 stale-state fence
(counterfactual): `runners after successor relaunch [24, 56], orphan
alive true`. Result: duplicate Runner (FAIL).
- macOS: `npx vitest run
src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`
reports the file as skipped (1 skipped), as intended, because there is
no `/proc`.
- The test runs for real only on Linux CI. CI on this head is the first
real run of the vitest file.

## Risks

- Low risk. The change is a test only.
- The test starts real local processes. Its teardown kills only the PIDs
it found on its own temporary state folder, and it never signals PID 0.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, Docker, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:24 -06:00
DottaandClaude Opus 5.5 c1ea9304f7 fix(recovery): continue Runner preparations whose remote workspace was handed on unchanged (#21481)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.

**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.

**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.

**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.

**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.

## What Changed

- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.

## Verification

- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.

## Risks

- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:46:05 -06:00
DottaandClaude Opus 5.5 a48eaaef57 perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent keeps its own files (AGENT_HOME). A warm native session
saves these files at the end of each turn (an agent-file checkpoint)
> - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on
the small file-storage pool, also when the run changed no file
> - A checkpoint that changed files also uploaded the new bytes to
object storage while it held that lock. Instruction edits and other runs
of the same agent waited for the upload
> - With many agents, these long transactions use the 2-connection
file-storage pool and slow every agent
> - This pull request removes the lock from an unchanged checkpoint and
moves the uploads of a changed checkpoint before the lock
> - The benefit is that an unchanged checkpoint takes no row lock, and
the row lock is no longer held across object-storage I/O

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm agent-file checkpoint: `checkpoint()` in
`server/src/services/agent-directory-working-copies.ts`,
`agentFileStore.locked` and `apply` in `agent-file-store.ts`, and
`persistDurableAgentFiles` in `agent-file-durability.ts`.

**Current behavior**
- When the run changed no file, the checkpoint still ran
`store.locked(..., write = true)`. That transaction locks the agent row
(`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the
local projection, walks the canonical tree, and compares the durable
head. Its only result is the authorization check.
- When the run changed files, `persistDurableAgentFiles` uploaded each
new file to object storage inside the transaction that holds the agent
row lock.

**Proposed behavior**
- An unchanged checkpoint (`saveUnchanged`) first reads, without a lock,
whether the locked file-tree step would change anything
(`agentFileProjectionCurrent`): the durable head exists and the local
projection is marked at its generation, or durable files are off, no
head exists and the local adoption is recorded. If so, it runs only the
write authorization (`authorizeSave`): the agent interaction policy and
`authorizeInstructionCommit`, in a short transaction without row locks
that always rolls back, so it writes nothing. If not, it runs the locked
save as on `origin/master`, which adopts, restores or heals the durable
head. A de-authorized agent still gets the same error (the existing test
"rechecks authorization even for an unchanged warm checkpoint" still
passes).
- A changed checkpoint runs the same read-only authorization (it
consumes no accepted change confirmation), then uploads the run's
changed files before it takes the lock (`uploadAgentFilesBeforeCommit`).
A run that lost its save permission uploads nothing. The locked commit
runs the authorization again, and only that commit consumes a
confirmation. The locked commit uses an uploaded object only for a file
with the same SHA-256 and size. It uploads any other file itself, as
before.

**Reason and benefit**
The target is 1,000 agents in one controller process. Most warm turns
change no agent file. These turns now take no row lock and send fewer
statements. A changed turn holds the row lock for less time, because
object-storage latency is outside the lock.

## What Changed

- `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)`
and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a
checkpoint, then uploads the run's changed files before `locked`, and
passes the uploaded objects to `persistDurableAgentFiles`. `locked`
takes an optional map of uploaded objects.
- `agent-file-durability.ts`: move the verified upload stream into
`uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the
durable head without a lock, skips files the head already has at the
same path and digest, and uploads the others. If an upload fails, it
stops and the locked commit uploads the rest, so errors are the same as
before. `persistDurableAgentFiles` uses an uploaded object when the
canonical file has the same digest and size.
- `agent-directory-working-copies.ts`: an unchanged checkpoint calls
`saveUnchanged` instead of `store.locked(..., true)`.
- `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head
read and two file checks, no lock).
- The commit is still inside the agent row lock. The head cannot change
between the read and the write of the commit, so it needs no separate
compare-and-set. The upload before the lock does not need one either: an
object is immutable and verified by digest, and the commit records it
only for a file with that digest.

## Verification

- New `server/src/__tests__/agent-file-checkpoint-locking.test.ts`
(embedded PostgreSQL, object-storage fixture). All tests fail on the
code before the change and pass with it:
- An unchanged checkpoint completes while another transaction holds the
agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR
SHARE` statement and only one unlocked read of `agent_file_roots`.
- Durable files on, the agent has local files and no head: an unchanged
checkpoint creates the head, as on `origin/master` (added after review;
failed without the fallback).
- The local projection is not marked at the head: an unchanged
checkpoint takes the locked path and marks it again (added after review;
failed without the fallback).
- A changed checkpoint uploads its file while the agent row is not
locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from
another connection during each upload). The head then records that
object.
- A run loses its save permission before a changed checkpoint. The
checkpoint is refused and uploads nothing (added after review; I saw it
fail without the authorization before the upload).
- An agent with `agents:suggest-changes` and one accepted change
confirmation saves a changed checkpoint. An unchanged checkpoint first
leaves the confirmation unconsumed. The changed save succeeds once and
consumes the confirmation once; the next change is refused before it
uploads (added after review; I saw it fail when the preview committed).
- Two warm runs of the same agent save at the same time. Both files are
in the canonical tree and in the head, and every head object has the
correct digest.
- An instruction edit commits while a checkpoint waits in its upload,
without a wait for the checkpoint. Then the checkpoint commits. Both
changes are in the head.
- A save stops between upload and commit (the instruction entry changes
while the bytes upload, so the commit refuses). The head does not change
and stays restorable. The next save commits the same file. The unused
object is never authoritative; the existing object collector removes it
after its grace period.
- Existing `agent-directory-working-copies.test.ts`,
`agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts`
pass.
- Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`,
test "measures checkpoint statements and agent row lock time"): one warm
copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that
each change one file, with 250 ms object-store latency. Wall times are
noisy (load average about 40). Statement counts are exact.

| | Before (`origin/master`) | After (final head) |
| --- | --- | --- |
| Unchanged checkpoint: statements | 26 | 25 (one unlocked head read,
and the read-only authorization) |
| Unchanged checkpoint: row-lock statements | 2 | 0 |
| Unchanged checkpoint: median time | 214 ms | 26–126 ms |
| Changed checkpoint: statements | 29 | 47 (the read-only authorization
before the upload and one head read) |
| Changed checkpoint: median time the agent row is locked | 924 ms |
51–227 ms |
| Changed checkpoint: uploads while the agent row is locked | 1 | 0 |

The "after" ranges come from two runs at different machine load (load
average between about 18 and 40).

## Risks

- Master side effects of the locked unchanged save, and what this change
does with each:
- Kept: the write authorization (`assertAgentInteraction`,
`authorizeInstructionCommit`), so a run that lost its permission still
gets the same error.
- Kept through the fallback: one-time adoption (no head yet), restore of
a projection that is not marked at the head, re-mark after a failed
persist (a write path removes the marker before it changes files), and
heal of a head that lags the root after such a failure.
`agentFileProjectionCurrent` returns false in each of these cases, so
the locked save runs.
- Dropped when the projection is current: the re-mark at the same
generation (no change), and the compare of the local tree with the head.
With the marker at the head generation, every writer that changed the
tree went through the locked save and moved the marker, so the compare
finds no difference. An edit made outside Paperclip directly in the
canonical directory is not picked up by an unchanged checkpoint any
more; the next save that goes through the locked path still records it.
- Dropped when the projection is current: the storage-limit check of the
canonical tree in `persistDurableAgentFiles`. An over-quota agent whose
run changed nothing no longer gets an error from that check; any save
that changes files still enforces the limits.
- Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction`
takes on the actor and target agents (`lock: "update"`). Those locks
hold permission changes off a write until it commits. The read-only
authorization writes nothing, so there is nothing to hold them off. The
locked commit of a changed save still takes them. An unchanged save has
no mention texts, so master locked no other agents for it.
- Effects of `authorizeSave` outside the rolled-back transaction: none.
I read `assertAgentInteraction` (`agentInteractionDenial`: reads only),
`authorizeInstructionCommit`, `resolveInstructionActor` (reads only),
`authorizationService.decide` (reads only; its memo maps live on the
bound actor and on the service instance, both created for this call and
dropped after it), and `changeConsentGateService.assertConsented` (one
`UPDATE` of the confirmation row, rolled back). None of them publishes
live events, queues notifications or wakes, writes activity through
another connection, makes external calls, or changes an in-process cache
that outlives the call.
- Behavior change: on `origin/master`, an unchanged checkpoint of a
consent-gated agent (`agents:suggest-changes`) consumed an accepted
change confirmation, because the locked transaction committed. Now the
fast path does not; only the commit of a changed save consumes it. When
the fast path cannot be used (adoption, restore or heal), the locked
save runs as on master and can still consume a confirmation.

- Not in this change: the manifest is still stored twice in the receipt
(`baseline` and `checkpointManifest`). Servers of the previous version
read `receipt.baseline` during a deploy overlap, and the next capture
needs the stamps in `checkpointManifest`, so neither can be removed
safely now.
- Not in this change: on the volume-free server, `serial()` still opens
a dedicated connection with an advisory-lock transaction for each
checkpoint. That lock serializes controllers and is not a row lock;
changing it is a separate design decision.
- A changed checkpoint can now leave an unused object in storage when
its commit is refused after the upload (for example, the instruction
entry changes during the upload) or the process stops before the commit.
This was already possible for a failed commit. Such an object is never
in the durable head, so no reader can reach it, and the existing
`collectAgentFileObjects` maintenance removes it after
`AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the
save stops the save before any upload.
- Old and new servers can run at the same time: the head format and the
receipt format do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:41:46 -06:00
DottaandPaperclip 8f105e743a fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 14:37:13 -06:00
DottaandClaude Opus 5.5 11f4a76743 perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every native agent session. Each session
gets the app tools (MCP tools) that are assigned to its agent.
> - Each live or warm session kept two private copies of its assigned
tool catalog: the gateway descriptors and a canonical copy of every
input schema.
> - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap
for each session. With 800 tools, it was about 15 MB. Warm sessions stay
for 5 minutes after the last turn, so idle agents also hold this memory.
> - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the
server cannot get there.
> - This pull request keeps one deeply frozen catalog for all sessions
of the same company that get the same admitted tools and schemas.
> - The benefit is that each additional session on the same catalog adds
about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is
faster.

## Linked Issues or Issue Description

Refs #21415 (releases a finished session's tool authority). Refs #21440
(adds the catalog memory benchmark that this change extends).

**What existing behavior does this improve?**
`createAssignedMcpTools` in
`server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call
kept every gateway descriptor (with its provider metadata) and a
separate canonical clone of each input schema. Each `definitions()` call
also made a new `structuredClone` of every schema.

**Subsystem affected**
Native runtime: assigned app tools (MCP) for native agent sessions.

**Current behavior**
Retained heap grows by one full catalog for each session. Measured with
the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB
per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100
sessions. Each `definitions()` call takes 14 to 41 ms because it clones
every schema.

**Proposed behavior**
Sessions of the same company that get identical admitted tools share one
frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB
at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms.

**Reason and benefit**
The server must hold 1,000 concurrent agents in one process. The catalog
was the largest per-session heap cost. Revocation and authorization do
not change: every call is still authorized again by the gateway, and
every new session still reads a fresh listing.

**Breaking changes**
None in the API or data. The direct definitions that `definitions()`
returns are now frozen. A consumer that writes to a returned definition
or schema gets a `TypeError` and does not change other sessions. No
production consumer writes to them (see Risks).

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`:
- A session keeps only a small frozen record for each tool (gateway
name, display name, description, risk), plus the frozen schema and the
frozen definition. It does not keep the gateway descriptors.
- New per-process index of shared catalogs. The key is the SHA-256 of an
identity string: `[companyId, [[assignedName, gatewayName, displayName,
description, risk, sha256(JSON of the input schema)] for each listed
tool, in listing order]]`. On a hit, the code also compares the full
identity string, so a hash collision cannot give a wrong catalog.
- The identity is computed from the fresh listing that the gateway
returns for this session. The gateway applies all per-agent, per-run,
per-profile, and policy filtering, and the approval description suffix,
before it returns the listing. Thus two sessions share only if they were
given exactly the same tools, names, descriptions, risks, and schemas in
the same company. Without `companyId`, nothing is shared.
- Schemas, definitions, tool records, and the definition arrays are
deeply frozen. Search results still return cloned copies.
- Sessions that start at the same time with the same catalog wait for
one build and share it.
- The index holds each catalog through a `WeakRef`, so it never keeps a
catalog alive. The index has at most 512 entries and 256 MiB (accounted
schema and description bytes). The oldest entries are removed first. A
catalog that is larger than the byte limit is not indexed.
- New `close()` on the assigned tools. When the last session that holds
a catalog closes, the index entry is removed. A `FinalizationRegistry`
also removes entries whose catalog was collected.
- `definitions()` returns the shared frozen definitions for the
session's work mode. It no longer clones every schema on each call.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
`close()` also closes the assigned tools. The session executor already
closes the authority when it revokes a session's tool authority.
- `server/src/services/native-runtime/native-session-executor.ts`: one
line. It passes the run binding's `companyId` to
`createAssignedMcpTools`.
- `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes
`companyId`, adds `--companies`, measures session start and
`definitions()` time, and returns fresh copies of fixed rows (like
database reads).
- Tests in `assigned-mcp-tools.test.ts` and
`paperclip-runner-tool-authority.test.ts`.

## Verification

Red/green tests in
`server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared
assigned tool catalogs"):

- Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema
catalog. The heap for the other 19 sessions must be less than two
catalogs. Red on master: `expected 14025976 to be less than 3788112`.
Green with this change.
- Shared deeply frozen schemas, also for sessions that start together.
Red on master (`Object.is` equality fails).
- Mutation isolation: writes to a returned definition, schema property,
and `required` array, and a delete, throw `TypeError`. Changes to the
gateway's source rows and to a search result do not change any session.
Red on master (no throw).
- Release and bounds: the entry stays while one session holds it and
goes when the last session closes. `close()` is idempotent. A new
session after release builds a new catalog. 517 companies keep the index
at 512 entries or fewer. Red on master (no `close` or stats).
- Guard tests: two companies with identical catalogs do not share.
Sessions without a company do not share. Fewer tools, an edited schema,
an edited description, or a different risk do not share. A revoked or
edited tool is served from fresh discovery. On master these pass,
because master never shares. To prove they test the key, I removed
`companyId` and the schema digest from the identity: three guard tests
failed, and they pass with the full key.
- An existing test changed: "does not project gateway secrets..." now
expects a write to a returned schema to throw `TypeError` (before, the
write changed only a private clone). The isolation check after the write
is the same.
- `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes
the assigned tools.

Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one
company). Command: `cd server && node --expose-gc --import tsx
scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N>
--schema-bytes 5700 --sessions <S> --turns 5`.

| Tools | Sessions | Total before | Per session before | Total after |
Per session after |
| --- | --- | --- | --- | --- | --- |
| 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB |
| 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB |
| 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB |
| 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB |
| 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB |
| 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB |

The first session also uses about half the heap of before, because the
session no longer keeps the gateway descriptors. When all sessions are
dropped without `close()`, the catalog is collected (0 bytes held after
release).

Session start, interleaved A/B in one process (old and new
`createAssignedMcpTools` alternate, 40 starts each, median of the last
35, includes a fresh copy of the listing; the machine had a load average
of 25 to 65):

| Tools | Listing copy only | Before | After |
| --- | --- | --- | --- |
| 400 | 22.6 ms | 68.0 ms | 37.6 ms |
| 800 | 30.5 ms | 72.5 ms | 41.2 ms |

A start that reuses a catalog does no schema clone and no admission. It
still serializes and hashes each schema to compute the key. The first
start of a new catalog does the same admission work as before (the cold
first start was about 600 to 750 ms in both versions on this busy
machine).

`definitions()` median: 14 to 41 ms before, 0.01 ms after.

Other checks:

- `cd server && npx vitest run
src/services/native-runtime/assigned-mcp-tools.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-tool-results-prp.integration.test.ts
src/__tests__/deployment-connections.test.ts`: 204 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
"assigned|tool authority|released|MCP"`: 31 passed.
- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10
passed.
- `cd server && npx tsc --noEmit`: pass.
- `pnpm check:module-boundaries`: pass.
- `pnpm verify` selects a broad plan (the benchmark script is a broad
input), so CI supplies the full suite.

## Risks

- Frozen definitions: the direct definitions now go to the runner
drivers as frozen objects. I checked the consumers.
`PaperclipRunnerToolAuthority.definitions()` filters and spreads them.
The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them,
spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP
bridge), or serialize them. Their types are `readonly
Readonly<Record<string, unknown>>[]`. The first-party semantic tool
definitions are already shared module constants that go through the same
path without clones. If a consumer that I did not find writes to a
definition, it gets a `TypeError` instead of a silent change. Catalogs
over 256 tools or over the listing budget use the two on-demand tools
and are not affected.
- Authorization: the shared data is only the result of a listing that
the gateway already authorized for that session, keyed on the full
content plus the company. Execution still goes through the gateway with
the session's own token on every call. Search still intersects with a
fresh listing. A new session still reads a fresh listing, so a revoked
or edited tool takes effect for new sessions and for existing sessions
at the same points as before.
- Several server processes: the index is a per-process memo of the
listing content. Each process builds its own catalog from its own
listing. Nothing is stored or shared outside the process.
- The index is bounded (512 entries, 256 MiB) and holds catalogs only
weakly. Index eviction only stops future sharing. It never frees memory
that a session uses.
- No schema change. No migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap and timing benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:30 -06:00
DottaandClaude Opus 5.5 533d02ff1a perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Live-run recovery uses the shared scoped deadline worker
(`scoped-deadline-work.ts`). Each queue keeps its entries (one for each
running run) and the scopes that writes marked dirty (for example one
company and one run)
> - To schedule and to pick due work, the worker tested every entry
against every dirty scope. It also decoded each scope hint with
`JSON.parse` again inside those loops, and each reconstructed scope
walked all entries to drop vanished ones
> - With many runs and many dirty run scopes, this work grows with
entries × scopes on the event loop of the one server process that
controls all runs
> - This pull request lets an owner declare a partition key for its
scopes. Live-run recovery uses the company. The worker then tests an
entry only against the scopes of its own company, and decodes each hint
once
> - The benefit is that matching cost grows with entries + scopes, with
the same results

## Linked Issues or Issue Description

Refs #21444 (the lease renewal change from the same load model).

**What existing behavior does this improve?**
The scoped deadline worker in
`server/src/services/scoped-deadline-work.ts`, which live-run recovery
(`server/src/services/heartbeat/live-recovery-work.ts`) uses for its
seven queues.

**Subsystem affected**
Server: background work scheduling for live-run and resource recovery.

**Current behavior**
For 400 entries in 40 companies and 40 dirty run scopes, one schedule
computation and one pass call `matches` 27,830 times and decode each
hint many times.

**Proposed behavior**
The same case calls `matches` fewer than 3,200 times and decodes each
hint at most once. Owners that do not declare a key behave exactly as
before.

**Reason and benefit**
The plan for 1,000 concurrent agents needs per-run background work that
does not grow with the product of runs and changes. The work runs on the
main thread of the controlling process.

**Breaking changes**
None. `scopeKey` is optional.

## What Changed

- `scoped-deadline-work.ts`: new optional `scopeKey` option
(`entry(entry)` and `scope(scope)`). The contract: a scope with a
defined key matches only entries with the same key. A scope without a
key (for example `*`) is tested against every entry, as before.
- The worker indexes the dirty scopes by key, so `next()` and the due
filter test each entry only against the scopes of its key and the
unkeyed scopes.
- When a reconstructed keyed scope drops vanished entries, it visits
only the entries of its key (an index built once for each pass).
- Scope hints are decoded once and the decoded value is reused while the
hint is dirty or scanning.
- `live-recovery-work.ts`: declares the company as the key. Its
`matches` already requires the same company for every defined scope.

## Verification

- New test in `scoped-deadline-work.test.ts`: 400 entries in 40
companies, then 40 run-scoped invalidations. It requires fewer than
3,200 `matches` calls and at most 40 decodes, and the next deadline is
unchanged.
- Red first: with `origin/master` `scoped-deadline-work.ts`, the test
failed with 27,830 `matches` calls.
- `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22
passed.
- The live-run recovery matrix (`live-recovery-work.test.ts`) is covered
by CI.

## Risks

- If an owner declares a key that its `matches` does not respect, the
worker can miss a match. Only live-run recovery declares a key, and its
`matches` returns false for any defined scope of another company.
- Low risk otherwise: owners without a key keep the old code path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:21 -06:00
DottaandClaude Opus 5.5 754b1f36d3 perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every running native agent. Each run
holds a native execution lease that the process must renew before it
expires (60 seconds for a remote run, 20 minutes for a local run)
> - Each run renewed its lease on its own timer, in its own transaction,
on a private pool of two connections. At 1,000 remote runs that is about
50 transactions each second through two connections
> - When renewals queue behind each other, a renewal can miss its 60
second deadline, and the run is lost although the server is healthy
> - Each renewal also woke six live-run recovery queues for its run. The
recovery owner then re-checked a healthy run about every 20 seconds,
although a renewal only moves the deadline later
> - This pull request renews all leases of one process with one shared
timer. All leases that are due go into one transaction of fixed size,
the same model that the legacy controller lease already uses. A renewal
no longer wakes recovery, as #20924 did for the legacy lease
> - The benefit is that the renewal cost no longer grows with the number
of runs, and all safety properties of a single renewal stay the same

## Linked Issues or Issue Description

Refs #20924 (the same change for the legacy controller lease), #21242.

**What existing behavior does this improve?**
Native execution lease renewal in
`server/src/services/native-runtime/native-execution-lease.ts`. Each run
had one `setInterval` and one transaction per renewal: statement
timeout, row lock, recovery notification, update, commit.

**Subsystem affected**
Server: native runtime, controller lease renewal on the liveness
connection pool.

**Current behavior**
The renewal cost grows with the number of runs. A local load model with
1,000 remote runs sends 35.7 lease transactions and 212.8 lease
statements each second, and the event-loop utilization is 0.34. All of
them use a pool of two connections.

**Proposed behavior**
One timer for each database client renews all due leases in one
transaction for each 200 runs. With 1,000 remote runs the same model
sends 0.29 lease transactions and 1.7 lease statements each second, and
the event-loop utilization is 0.07.

**Reason and benefit**
The plan for 1,000 concurrent agents needs renewals that do not scale
with the run count (item 1d). A renewal that misses its deadline loses
the run.

**Breaking changes**
None. The `NativeExecutionLeaseRenewal` contract (`stop()`,
`remainingMs()`) and `renewNativeSessionExecutionLease` do not change.

## What Changed

- `native-execution-lease.ts`: a renewal (batched or single) no longer
calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline
later. Recovery already holds a deadline at the earlier expiry. At that
deadline it re-reads the row and moves to the new expiry. A lapsed lease
is still found at its expiry. This change does not touch the signals of
the writers that release, revoke or take over a lease, or that change
the run status. This also removes six notifications for each renewal,
which was 1,200 notifications for a batch of 200 runs during a deploy
overlap.
- `native-execution-lease.ts`: one shared schedule for each database
client replaces the timer of each run. When the first lease is due, all
leases that are due within a quarter of their own interval renew with
it. Thus the leases of a process converge on one batch for each
interval.
- A batch is one transaction on the liveness pool for each 200 runs:
`statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the
coordinator rows, and one `UPDATE … FROM (VALUES …)`.
- The batch keeps the guarantees of a single renewal:
- It locks each coordinator row before it reads the expiry, as before.
Thus an owner whose lease expired while another writer held the row is
never revived.
- It extends a row only for the exact owner: run, company, task, lease
owner, attempt, controller boot id, pid, process start, and a live
lease.
- The confirmed deadline is sampled after the locks and before the
update, as before.
- A run whose row was locked and not extended fails with
`native_session_lease_lost`. The other runs in the batch succeed.
- These rows do not renew in the batch. They renew alone through the
unchanged single-run path:
- A row that another writer holds (`SKIP LOCKED`). Thus one busy row
cannot hold the batch until the statement timeout.
- A row that the lock did not find (another owner, or no row). The
single path gives the same answer as before.
- A local lease that lapsed (host sleep). Only the single path may take
it back, under the run row lock.
- A batch that is cancelled (statement timeout `57014` or deadlock
`40P01`) extends nothing. Its runs retry after 1 second, and each run
keeps its own deadline timer. Before this change, a statement timeout
waited for the next 20 second tick.
- Any other batch failure, also an error thrown while the statement is
built, may come from one bad row. Each run of that batch then renews
alone at once through the single path, so a bad row can cost only its
own run. A run that fails alone while others in its batch renew keeps
renewing alone (out of later batches) until a single renewal succeeds.
During a database outage the single renewals also fail; those runs
return to the batch at their next interval, not after a 1 second retry,
so an outage causes at most one queued single transaction for each run
and interval.
- Runs that become due while a batch is in flight wait for that batch. A
batch that runs for more than 15 seconds no longer holds back the next
batch.
- A tick that passed while a renewal was in flight is skipped, as a
fixed interval skips it. Thus a slow database gets no extra load.
- The shared timer replaces the interval timer of each run. It exists
only while this process holds a lease, and it fires at the next renewal
deadline. With no held lease there is no timer and no SQL.
- New export `renewNativeExecutionLeasesNow(db)` renews all leases of
the process at once. Tests use it.

## Verification

- New PostgreSQL tests in `native-execution-lease-batch.test.ts`:
- 3 runs and 60 runs each renew in 1 transaction with the same number of
statements.
  - 450 runs renew in 3 transactions.
- In one batch, the runs whose lease was taken over, whose controller
boot changed, or whose lease expired fail with
`native_session_lease_lost`. Their rows do not change. The healthy runs
renew. A newer attempt under the same owner is not lost, as before.
- Interleaving: another transaction holds a coordinator row from before
its lease expires until after it expired. The healthy run in the same
batch renews at once. After the lock is released, the expired owner is
lost and its row stays expired.
- A takeover that commits before the batch: the batch extends nothing
and the run is lost.
- A local and a remote lease renew in one transaction, each for its own
lease length.
- A poison binding (an attempt outside the integer range) makes the
shared UPDATE fail with `22003`. The other 4 runs still renew in the
same tick, only the poison run is lost, and the next tick renews the 4
runs in 1 transaction again.
- An invalid date throws while the statement is built. The other runs
renew, the broken run is lost alone, and no lease stays busy (the next
call renews again in 1 transaction).
- New PostgreSQL test in `live-recovery-work.test.ts` with the real
registered recovery owners: another server's native run renews its
lease. No recovery pass runs after the renewal. At the old expiry
recovery re-reads the row, keeps the run running and moves its deadline
to the new expiry. When renewals stop, recovery acts on the run within
10 seconds after the end of the last lease.
- Red first: with a shim on `origin/master` that renews each lease
alone, the transaction test failed (`expected 3 to be 1`) and the
450-run test failed.
- The unit tests in `native-execution-lease.test.ts` now drive the
batch. New tests show that a non-cancelled batch failure renews the run
alone at once, and that an outage (batch and single renewal both fail)
does not retry every second and loses no run.
- `native-execution-lease-reacquire.test.ts`: the host-sleep tests now
go through the batch, which hands the lapsed local lease to the single
path.
- Load model (`startNativeSessionExecutionLeaseRenewal` for N remote
runs that start at random times over 20 seconds, 70 second window, local
PostgreSQL, machine load average about 50):

| Runs | Lease transactions/s before → after | Lease statements/s before
→ after | Event-loop utilization before → after | Smallest margin to the
60 s deadline | Lost |
| --- | --- | --- | --- | --- | --- |
| 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0
s | 0 → 0 |
| 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s
| 0 → 0 |

The window includes the 20 second ramp, so the steady rate before is
higher (one transaction each 20 seconds for each run, 50/s at 1,000
runs). After the change, one batch carries up to 200 runs.

## Risks

- **Behavior change: batch-wide error handling.** One shared transaction
now carries up to 200 runs, so one database error reaches all of them. A
cancelled batch only skips the extension and retries after 1 second. Any
other batch error sends each run to the single path at once, so one bad
row (a bad value, a cast error, an error while the statement is built)
costs only its own run. In a database outage, up to one single
transaction for each run waits on the two-connection pool at each
interval. Each run's own deadline is unchanged.
- **Behavior change: no recovery signal on renewal.** See below.
- During a deploy overlap the old server renews its runs one by one and
the new server renews in batches. Both use the same row lock and the
same conditions, so they do not conflict. Rows written by the old
version have the same columns.
- `SKIP LOCKED` sends a busy row to the single path, which waits for the
lock as before.
- **No recovery signal on renewal:** recovery of a healthy run of
another server now runs once for each lease period at the old expiry,
not after each renewal. A dead run is found at the same time as before:
at the expiry of its last lease.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:36:16 -06:00