26 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 47a2c82d35 fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server stops and the incoming
server takes over the live runs on remote Runners
> - The incoming server reached those runs late: it waited up to 60 s
for the outgoing server's controller lease to lapse, re-verified each
run's whole log from object storage, and deferred a reattach when the
agent directory was busy. Queued starts competed with the reattaches for
the same resources.
> - On hosted staging, the later a live run was reached, the more likely
it was to fail. Runs reached in under 150 s never failed; runs reached
after 420 s almost always failed.
> - This pull request hands live runs over at the start of shutdown,
makes a reattach skip work that the same run already did, and holds
queued starts while handed-over runs are being reached, with a 180 s
cap.
> - The benefit is that live runs are reached within seconds of a swap
instead of minutes, so they continue instead of failing.

## Linked Issues or Issue Description

**What happened?**

Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each
run that crossed a swap we measured the gap from the last Runner output
to the moment the new server recorded the Runner's process identity:

- gap under 150 s: 0 of 9 runs failed
- gap over 420 s: 14 of 15 runs failed
- median gap: 180 s for runs that resumed, 470 s for runs that failed

In one traced run the incoming server waited on
`remote_controller_lease_live` three times (09:14:21, 09:15:18,
09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of
that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the
run's whole log from object storage. A control run spent 23 s in
`task.prepare`. Queued starts used the same preparation path, the same 8
object-I/O slots and the same database pool, and nothing gave a reattach
priority. The first-attempt failures were `PRP command session.snapshot
timed out` and `codex_history_read_failed`. The retry then failed with
`runner_harness_state_mismatch: resumed_not_suspended`,
`runner_remote_process_identity_unavailable` or
`provider_turn_lost_on_restore`.

**Expected behavior**

Every live run is reached within about 30 s of the swap. Queued work
starts after that, or after a bounded wait.

**Steps to reproduce**

1. Run a two-server release swap with live remote runs and queued runs.
2. Observe `native.recovery.transition` "waiting" events with
`remote_controller_lease_live`, then the claim, then a long
`heartbeat.prepare.run_output` span.
3. The new tests in `reattach-phase.test.ts` and
`heartbeat-native-early-hand-off.test.ts` model this.

**Paperclip version or commit**

`master` at 426e8a35dc.

## What Changed

- Early hand-off at shutdown (`heartbeat/recovery.ts`,
`server/src/index.ts`): in deploy-overlap mode, the outgoing server
releases the remote controller lease of its live runs at SIGTERM
(`recoveryState` `awaiting_runner_reattach`). A run another server
already took over is left alone. The incoming server no longer waits for
the lease to lapse.
- Reattach fast path:
- `heartbeat/run-output.ts`, `native-runtime/native-run-log-export.ts`:
a reattach starts without waiting for the existing log to be
re-verified.
- `heartbeat/agent-directory-busy.ts`, `heartbeat.ts`: a reattach that
meets a busy agent directory waits in process (at most 30 s) instead of
being deferred to a full re-preparation. An ordinary start is still
deferred at once, with no added wait.
- Reattach phase lifecycle (`heartbeat/reattach-phase.ts`): the watch is
a worker on the shared delivery coordinator, queue `reattach-phase`.
- Trigger: one startup listing of a previous server's live runs; then
the hand-over notification (`notifyReattachHandOver`, sent in the
hand-over transaction); the earliest lease deadline it watches (a killed
owner); and a 2 s re-check only while handed-over runs are still
pending, which ends with the phase (cap 180 s). The bounded re-check
exists because a run can also end on another path that sends no
notification to this queue.
- Source of truth: `native_run_finalizations` and `heartbeat_runs` rows.
- Quiet condition: nothing pending and no watched lease, so no deadline
and no SQL. It stops with the coordinator at shutdown.
- Reached hook for every driver (`native-session-executor.ts`): a
reattached run counts as reached when the native session runtime hands
the executor its recovered or opened session (`onSession`). Every driver
reaches that point only after the Runner answered this server's attach
and recovery commands, including an adopted `RunnerdExternalDriver`
Runner, which records no new process identity. The identity hook stays
as an earlier signal.
- Claim wake: when the reattach-phase worker finds runs that just became
claimable (hand-over notification or a killed owner's lease deadline),
it starts this server's restart-recovery claim for them at once. A
validation cell showed the claim otherwise waited about 21.5 s after the
hand-over.
- `recovery/assigned-work-sweep.ts`: the `deferred_reattach_phase`
return now sits before every action path of `act()`, except starting an
overdue saved-work recovery (it resumes an existing run).
- The in-transaction admission re-check of a queued claim
(`confirmRunAdmissionStillOpen`) also honours the reattach phase. A
phase that begins during a claim returns the run to the queue through
the existing rollback.
- A reattach whose log re-verification runs in the background awaits it
before the log is finalized (`settleDeferredLogRecovery`, both
finalization sites in `run-completion.ts`).
- Reattach phase (`heartbeat/reattach-phase.ts`,
`heartbeat/run-admission.ts`): `isReattachPhaseActive()` is true while
runs that a previous controller handed over (lease released or observed
expired, or claimed by this server's restart recovery) are claimable but
not yet reached. A run is reached when its process identity is recorded.
Authentication is logged as a mark. The cap (180 s,
`PAPERCLIP_REATTACH_PHASE_CAP_MS`) counts from the moment the runs
became claimable. Later batches join or re-arm the phase. Runs reached
after 150 s are counted and logged.
- Gates on the phase:
- `claimQueuedRun`, start-next, resume-queued and dispatch in
`heartbeat/queue.ts`, `run-lifecycle.ts` and
`native-runner-execution-driver.ts` leave queued runs queued.
- `execution-recovery-resolution.ts`: one line, the first line of
`deliverReconciledExecutions`.
- `recovery/assigned-work-sweep.ts`: one line, outcome
`deferred_reattach_phase`.
- `heartbeat/recovery.ts`: a one-line claim hook
(`noteReattachRunsClaimed`) after restart recovery claims runs.
  - The owner of those three files agreed to these one-line gates.
- `tests/restart-chaos/daytona-image.ts`:
`PAPERCLIP_CHAOS_TEST_ACCEPT_IMAGE_CONTENT_ID` lets a local wave run a
controller-only branch against an existing Daytona image. It accepts a
content-ID mismatch only for the exact content ID of the named image,
and it prints a TEST OVERRIDE warning. This is harness-only.

## Verification

Red on master, green on this branch:

- `heartbeat/reattach-phase.test.ts` (27 tests). Five of the six
two-server scenarios were red on the earlier phase timing:
- `does not hold queued starts while the previous owner is alive and not
handing over`
- `closes the gate when the previous owner hands its runs over, and
counts reach latency from the hand-over`
- `runs the cap from the hand-over, so a wedged run ends the phase
without the overlap eating it`
- `begins when a killed owner's lease is observed expired, without any
hand-over`
- `re-arms for a later batch, including runs the previous owner started
after discovery`
- `closes the gate at once when this server claims a run between two
lease reads, dated from the hand-over`
- plus `claimQueuedRun leaves a queued run queued and unstarted, in the
admission gate's shape`, `is the same gate the assigned-work sweep and
reconciled continuation delivery read`, `ends at the cap, counted from
when the runs became claimable, so one wedged Runner cannot hold run
starts`
- `heartbeat-native-early-hand-off.test.ts`: `releases the remote
controller lease of %i live runs at once` (1, 20, 40 runs; 40 runs in 56
ms), `leaves a run another server already took over alone`, `holds the
incoming server's queued starts only from the hand-over, not while the
outgoing server still drives its runs`
- `heartbeat/run-output.test.ts`: `lets a reattach start before its
existing log is re-verified`
- `heartbeat/agent-directory-busy.test.ts`: `defers an ordinary start at
once, with no wait added to the run-start path`, `lets a reattach wait
its turn in process instead of a deferred full re-preparation`, `still
defers a reattach once its bounded wait runs out`
- `tests/restart-chaos/daytona-image.test.ts`: `accepts a mismatch only
for the exact image content ID named by the test override`
- Review fixes (red before the fix, green after):
`reattach-phase.test.ts` `starts the claim of handed-over runs within a
second of the hand-over notification`; `assigned-work-invariant.test.ts`
`takes no action while handed-over live runs are being reached, and acts
once the phase ends`; `heartbeat-native-early-hand-off.test.ts` `never
lets the outgoing server claim back a run it handed off, while the
incoming server may take it over`; `reattach-phase.test.ts` `retries a
failed first listing instead of going quiet, and still finds the
previous owner's runs`, `keeps a hand-over notification whose listing
failed until a listing succeeds`, `moves the cap earlier when a claimed
run turns out to have been handed over before its claim`;
`queue.test.ts` `returns the claim to the queue when a reattach phase
begins while the claim is being checked`; `run-completion.test.ts`
`finalizes the run log in completeRun|failRun only after a reattach's
deferred log recovery settles`. `native-session-executor.test.ts` `ends
the reattach phase once a reattached %s session is driven by this server
again, without the cap or manual settling` over codex_app_server,
opencode_server, acpx_runtime, muse_external, grok_bot_mcp and
openai_dot_mcp (6 of 6 red without the hook; the test drives the
executor with a mocked session runtime, not each real driver). Lifecycle
tests for the new worker: `does no database work and keeps no deadline
on an instance with nothing to reattach`, `waits for the hand-over
notification or the lease deadline instead of polling while the previous
owner drives its runs`, `begins at a killed owner's lease deadline,
without any hand-over or notification`, `stops with the delivery
coordinator at shutdown and ignores a later hand-over`.

Local runs on this branch: `tsc --noEmit` is clean for `server`. Passing
on this branch: `reattach-phase.test.ts`,
`heartbeat-native-early-hand-off.test.ts`, `run-output.test.ts` (53
tests together), `agent-directory-busy.test.ts`,
`heartbeat-native-restart-suspension.test.ts` (115 of 115, alone), the
new cases in `queue.test.ts` and `run-completion.test.ts`. Under load 30
to 96 on this shared machine, the full `queue.test.ts` and
`recovery-wake-herd.test.ts` runs stalled in their embedded-Postgres
setup; CI supplies them.

Daytona results that exist (local servers with Daytona sandboxes,
release-swap cells). They ran on a combined branch with this PR, #21462,
#21442 and #21362 (merged), so they cannot be credited to this PR alone:

- G5 baseline (master d1d71a7755): SIGTERM to PRP authentication 21.5 to
38 s at swap 1.
- G5c (combined branch 588c6ba267): SIGTERM to authentication 6.2 to 7.7
s at swap 1. All runs that crossed swap 2 were adopted on the same
Runner process. The staging chain (`provider recovery timed out after
900000ms`, then `resumed_not_suspended`) did not recur.
- Caveat: G5c ran before the phase timing fix in this PR. In G5c the
phase opened at process start, its 180 s cap ran out during the overlap,
and swap 2 ended `cap_reached` with 0 runs reached, 1 s before the lease
claims. The timing fix (the phase begins when the runs become claimable)
has not run on Daytona.
- G6c (12 live runs and 20 queued, two graceful swaps, no injection,
combined branch 588c6ba267): 0 failed runs at either swap; 11 of 11
survivors at both.
- Swap 1: hand-off of 11 runs in 291 ms; claims 0.3 to 2.9 s after
hand-off; authentication 7.6 to 12.5 s and identity 8.3 to 13.3 s after
SIGTERM. The 30 s target was met.
- Swap 2: hand-off took 8,562 ms and `hot_restart_prepare` timed out at
19 s, so the outgoing server drained gracefully. The incoming server did
not claim on the hand-over; it claimed only when the outgoing server
exited, 33 s after SIGTERM. Authentication was 46 to 54 s and identity
47 to 57 s after SIGTERM. The target was missed. Why the claims waited
for the exit is not explained.
- The host load was 49 to 70 from other work. 7 of 32 tasks timed out in
the queue or budget after the swaps; we suspect the overload.

## Risks

- Product trade-off: while handed-over live runs are being reached,
queued starts wait, up to the 180 s cap. On an instance with no
handed-over runs, nothing waits.
- Not proven: two-server behavior with the phase timing fix and the
notification-driven watch (no cell has run them); warm standby becoming
active; runs that the outgoing server starts after discovery.
- Early hand-off at SIGTERM orphans a server-side exec tool that is in
flight. With #21442 the next controller settles it as interrupted with
an unknown outcome; it is never run again.
- Swap 2 in G6c: the incoming server claimed only at the outgoing
server's exit. That is unexplained.
- Cross-session ownership: the gates in
`execution-recovery-resolution.ts` and
`recovery/assigned-work-sweep.ts`, and the claim hook in
`heartbeat/recovery.ts`, are in files that the restart-recovery work
owns. They are one-line gates agreed with that owner. That owner said
the gate did not need to be inside the claim transaction; that was
before the phase could begin during a claim. After review, the
in-transaction re-check in `heartbeat/queue.ts` also reads the phase,
through the existing rollback. #21427 (open) also edits
`execution-recovery-resolution.ts`, `heartbeat/queue.ts` and
`recovery/assigned-work-sweep.ts`; whichever merges second needs a
rebase. `native-runtime/native-restart-recovery.ts` (claim logic, owned
by the restart-recovery work): a 3-line guard in
`claimNativeRestartRecoveries` returns a run this process handed off at
shutdown as `blocked` / `handed_off_to_next_controller` before takeover
is evaluated, so the outgoing server never claims back a run it handed
off. #21478 (merged) added a sweep path that starts an overdue
saved-work recovery; it is deliberately not gated by the reattach phase,
because it resumes an existing run (reattach-like work), which the phase
makes room for rather than competes with.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 18:55:31 -06:00
DottaandClaude Opus 5.5 6917d0568d fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server starts queued agent runs. A legacy-runtime run gets a 60
s controller lease when it is claimed. Only the run's own execution
renews that lease.
> - The queued-run dispatch claimed all the runs of one agent in a loop.
It started their executions only after the last claim finished.
> - When the database is slow (for example, while a flood of recovery
runs fills the queue), one claim can take tens of seconds. The earlier
claims' leases then expire before their executions start. The run fails
with `controller_lease_lost` and `renewalCount: 0`.
> - Also, the lease watch waited for its first 10 s tick before it
renewed, even when the claim-time lease was almost spent.
> - This pull request starts each run as soon as it is claimed. It also
makes the lease watch renew immediately when the lease is older than one
renewal interval.
> - The benefit is that a slow claim batch no longer makes runs fail
before they do any work.

## Linked Issues or Issue Description

Refs #21127, #21042, #21222

**What happened?**
On staging, legacy-runtime runs failed with `controller_lease_lost`.
Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`,
and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease
watch started on a lease that had already expired, and no renewal was
ever sent. In one run, the claim was at 23:20:02 and the loss was at
23:23:11. Losses came in groups on the same agent at the same moment.
For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all
lost at 23:23:11.6.

**Why did it happen?**
`startNextQueuedRunForAgent` claimed every run of a batch before it
called `executeRun` for any of them. Each claim commits a 60 s lease,
but nothing renews it until the run's execution starts its lease watch.
A queue flood (fixed separately in #21222) made each claim slow, so a
batch took minutes. This bug predates today's changes. The queue flood
exposed it. The run-start hand-over (#21127) did not cause it: no
renewal was queued on the controller-lease connection.

**Expected behavior**
A claimed run starts its execution, and renews its lease, without
waiting for other claims. A lease watch that starts late renews at once.
A lease that lapsed and was revoked, reaped or taken over is still lost
and is never revived.

## What Changed

- `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent`
starts each run's execution right after its own claim, not after the
whole batch. The concurrency cap, the agent start lock and the
run-admission check before each claim are unchanged. If a later claim
throws, the runs claimed before it are already executing, not stranded.
After merging master, the loop walks master's queued-run dispatcher
(#21234: person-initiated runs first, paced recovery delivery): each
successful claim is reported to the dispatcher (`dispatcher.claimed`)
and then started at once, so master's order and pacing are unchanged.
Master's long-queued-run bookkeeping (#21349) is kept too: a claim that
returns nothing is still recorded as declined (`noteDeclinedAgedClaim`),
and a successful claim still clears that record before the run starts.
- `server/src/services/legacy-controller-lease.ts`:
`watchLegacyControllerLease` renews immediately when the claim-time
lease has less than 50 s left (older than one renewal interval). It uses
the same conditional renewal statement. That statement extends only a
lease that this boot still holds on a running row. The deadline is
unchanged.
- Tests: red/green tests for both changes. A test also proves that a
late watch on a revoked, reaped or lapsed lease still loses it and does
not change the row.

## Verification

- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass.
- Red on master (source reverted, new tests kept): "starts each claimed
run before claiming the next" fails (`executeRun` not called while the
second claim is in progress). "renews at once when the watch starts on a
lease older than one renewal interval" fails (aborted at the claim-time
deadline). The three "starts late ... still loses it" cases fail on
master because master loses them by deadline and does not check
ownership.
- After deploy: `controller_lease_lost` with `renewalCount: 0` should
drop to about zero on staging, also with a full queue. A late watch now
shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a
silent expired start.

## Risks

- Low. The executions of a batch now start while later claims of the
same batch are still in progress. Execution does not claim runs and is
not awaited by the loop, so the cap and the lock are not affected.
- A late lease watch sends one extra renewal at start. A normal claim
starts its watch within milliseconds with about 60 s left, so it sends
no extra statement and adds no wait to a turn start.
- No schema change. Mixed versions are safe: the renewal statement is
unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:08:38 -06:00
DottaandClaude Opus 5.5 566843820c fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.

## Linked Issues or Issue Description

Refs #21250, #21257 (merged), #21258, #21234

**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.

**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.

**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.

## What Changed

- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.

### Old → new wording

| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
&lt;remedy&gt;, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "&lt;what
happened&gt;, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |

Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.

### Deliberately changed test expectations

Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.

Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.

Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.

Local runs, on master after #21257 (e063587edd) plus these commits
(server `tsc --noEmit` also passes):
- `stranded-notice.test.ts`, `recovery/assigned-work-invariant.test.ts`,
`task-retry.test.ts` and `__tests__/assigned-work-invariant.test.ts`: 4
files plus `heartbeat/queue.test.ts`, 329 tests passed.
- `heartbeat-process-recovery.test.ts` and `recovery-wake-herd.test.ts`:
475 tests passed.

Code comments that the original commits added were removed to follow
CONTRIBUTING; the explanations are in this description.

## Risks

- The sweep now resolves board-owned stranded holds for recent temporary
failures. It is bounded to failures within the last 24 hours with a
restart-class or temporary-setup cause, one continuation per task under
the usual cap and spacing. Lasting causes stay with their owner.
- Lifting a block on explicit continuation is limited to blocks the
replaced interruption set. A declared unblock action or an open blocker
keeps the block.
- Wording only changes server text. The UI-recognized "No live execution
path" notice is unchanged.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 04:04:20 -06:00
DottaandClaude Opus 5.5 99d9e608f4 fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person reassigns a task, the old owner's native run is
cancelled and the new owner's run is queued
> - The old run's executor keeps closing its provider session for
several seconds after the run is marked cancelled
> - #21031 holds a queued run while the task's previous session close is
pending. That hold reads the runner's cleanup gate, which exists only
after the close begins
> - The stale-lock sweep cleared the task lock as soon as the old run
was terminal. The new owner's run was then claimed before the close
began, met the close as it escalated, and failed with
`native_session_cleanup_quarantined`
> - This pull request keeps the new owner's run queued while the old
owner's stopped run still settles in this process, and wakes it when
that executor finishes
> - The benefit is that a reassignment never fails the new owner's run
and needs no Retry

## Linked Issues or Issue Description

Refs #21031

**What happened?**
In the paid chat eval `tasks.edit-properties`, the old owner's run
`6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process
stopped only at 04:03:57. The stale-lock sweep cleared the task lock at
04:03:47. The new owner's run was claimed at 04:03:48.626. The runner
admission waited for the old close, which escalated to operator-required
at 04:03:56, and the new run failed with
`native_session_cleanup_quarantined`.

Why #21031 did not apply: its hold reads the runner's in-memory cleanup
gate. At claim time the old executor had not started its session close,
so no gate existed. The existing `predecessor_settling_in_process` hold
covers only the same agent, and a reassignment changes the agent. #21260
does not touch this path: its new checkpoint runs only before a turn in
a recovered session, and both sessions here were new.

**Steps to reproduce**
1. Run a native task for agent A.
2. Reassign it to agent B. A's run is cancelled, but its executor is
still settling.
3. The stale-lock sweep clears the task lock.
4. B's queued run is claimed before A's session close begins.

**Expected behavior**
B's run stays queued until A's executor has finished. It starts by
itself after that, then #21031's gate covers any close that is still
settling.

## What Changed

- `server/src/services/heartbeat/queue.ts`: the
`predecessor_settling_in_process` hold in `claimQueuedRun` also holds
for another agent's native run on the same task when that run is
terminal but its executor is still active in this process. A live run of
another agent (for example a reviewer) does not hold it.
- `server/src/services/heartbeat/queue.ts`:
`notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch
signal when a native executor settles and another agent has a queued run
on the same task. It does nothing when no run waits.
- `server/src/services/heartbeat.ts`: the executor calls it after its
cleanup, next to the existing same-agent `startNextQueuedRunForAgent`.
- `server/src/services/heartbeat/queue.test.ts`: the exact ordering
(cancelled old-owner run still in the executor set, lock cleared, new
owner's run queued), a control for a live run of another agent, and the
dispatch wake.

## Verification

- `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t
"reassigned owner|another agent's live run|next owner once"` — the hold
and the wake tests fail on master and pass here.
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts
src/services/queued-run-dispatch-liveness.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts` (all passed).
- `cd server && npx tsc --noEmit` passes.
- Paid E2E: see the combined-branch run in the comments.

## Risks

- A queued run on a task waits while another agent's terminal native run
on that task is still settling in this process. That wait ends when the
executor finishes, and the new wake re-drives dispatch. The queued-run
liveness redrive still covers a lost signal.
- The wake is event-driven: one indexed query when a native executor
settles, and one notification only when another agent waits on that
task. It adds no polling.
- Previous-release saved state: no persisted shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 23:53:49 -06:00
DottaandClaude Opus 5.5 7154030ef6 fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - During a release swap, the outgoing server drains. It must not start
new runs, because its own exit would cut them off.
> - The drain gate (run admission closed at SIGTERM) and the hand-over
gate (#21127, yield to a replacement that already dispatches) were
checked only by the queued-run dispatch passes.
> - Other start paths (executeRun of a queued run, delivered
continuations, immediate wake starts) went straight to `claimQueuedRun`,
so an outgoing server could still start runs after traffic moved.
> - Also, a system continuation queued while its task was open started
after another run had finished the task, and took an extra turn on a
done task.
> - This pull request makes `claimQueuedRun`, which every start path
uses, honour the drain and hand-over gate, and cancel a system
continuation whose task closed while it was queued.
> - The benefit is fewer runs cut off by a swap, and no extra turns on
finished tasks.

## Linked Issues or Issue Description

Refs #21127, #21135

**What happened?**
1. An outgoing server kept starting runs minutes after traffic moved to
its replacement. Each such run was cut off when the old server exited.
The run start went through `executeRun` of a queued run, which called
`claimQueuedRun` without the drain or hand-over check.
2. In release-swap waves, a continuation (execution reconciliation
delivery, invariant or recovery continuation, blocker-stall wake) was
queued while its task was open. Another run then finished the task. The
continuation still started and took an extra turn on a done task.

**Expected behavior**
A draining or yielding server leaves a queued run queued for the serving
server, on every start path. A system continuation does not start on a
task that was closed while it was queued. A person's message on a done
task still runs.

**Steps to reproduce**
1. Close run admission (drain) or set the hand-over yield on a server.
2. Start a queued run through `executeRun` (for example a delivered
continuation).
3. See the run claimed and started on the draining server.

## What Changed

- `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes
and checks the run-admission gate first. When admission is closed
(draining or yielding), it leaves the run queued and returns. The
dispatch passes already did this check; now every start path does.
- `claimQueuedRun` re-reads the task for a system continuation
(`issue_recovery_action_restored`, `issue_continuation_needed`,
`issue_assignment_recovery`, blocker-stall wake). If the task is `done`
or `cancelled`, it cancels the run as `issue_closed_before_start`, skips
its wake, and releases the task. Other wake reasons, including a
person's message, are not affected.
- That cancel never consumes a person's message. Admission can attach
deferred comments to a queued system continuation without changing its
wake reason. The cancel locks the run's wake and the run (the order
admission's coalescing uses), reads the run's current comment ids, and
if any is a person's comment that no run of this agent has received yet
(read-only use of `deliveredQueuedCommentIds`, which includes the
history-read rule), the run is not cancelled: it is claimed as that
person's follow-up, which may reopen the task through the normal path. A
comment attached concurrently is either seen under the lock or, after
the cancel commits, saved behind the cancelled run by admission.
- Admission is checked again inside each claim transaction (ordinary
claim, queued-comment claim, native review claim), right after the write
that moves the run to `running`. If admission closed during the awaited
checks, the transaction rolls back: the run stays queued with no
`startedAt`, its wake and the task's execution lock are untouched, and
nothing is charged.
- Every cancellation `claimQueuedRun` makes from the caller's copy of
the run is now a compare-and-set from `queued`
(`setRunStatusFromLive(..., ["queued"])` through a new
`cancelRunIfStillQueued` helper): the closed-task cancel, the
refused-interaction cancel, the missing-@mention cancel, the daily-cap
cancel and the blocked-dependency cancel. A run another dispatcher
already started or finished keeps its status and result, and no wake or
release follows. The queue module no longer receives the unconditional
`setRunStatus`.
- The closed-task cancel takes its decision under the wake and run
locks, then cancels through `setRunStatusFromLive` fenced on the row
version (`xmin`) the decision read. That saves the same never-started
proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted:
false }`) as every other queued-run cancel, so task release promotes a
person's message saved behind it. A message attached after the decision
changes the run, the cancel matches nothing and the decision is taken
again. `HeartbeatRecoveryDependencies.setRunStatusFromLive`
(`recovery.ts`) now types its existing `cancellationCondition` parameter
as `SQL`.
- The dispatch ranking in `startNextQueuedRunForAgent` is not changed.

### What I checked in `claimQueuedRun` for writes from the caller's copy

- Cancellations through `setRunStatus`: all five now conditional on
`queued` (above).
- The final claims (ordinary, queued-comment, native review) already
lock and re-read the run and update `where status = 'queued'`.
- The case-handoff cancel and the chat-control stop already lock the run
and update `where status = 'queued'`.
- The budget-stamp update already writes `where status = run.status`
(`queued`).
- `cancelRunInternal` (agent removed, agent not invokable, board-owned
budget stop, pause hold, budget cancel) re-reads the run and only
cancels a non-terminal status, so it cannot overwrite a finished result.
It can still stop a run another server has just started; that is a
policy stop (agent removed, budget, hold) that would apply to that run
anyway, and it is unchanged here.
- `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`.

## Verification

Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`:
- "claims no queued run once run admission is closed, whatever path asks
(draining or yielding server)": red before the gate was added to
`claimQueuedRun`, green after.
- "does not start a %s run whose task another run already finished":
`issue_continuation_needed` and `issue_recovery_action_restored` are
cancelled as `issue_closed_before_start`; `issue_commented` (a person's
message, carrying its `wakeCommentId`) is claimed and the stored run is
`running`.
- "answers a person's message attached to a system continuation on a
closed task once": still undelivered, the continuation is claimed and is
the only unfinished run (red when the cancel ignores comment ids, green
after); already received by the closing run, the continuation is
cancelled as `issue_closed_before_start` and no other run exists.
- "keeps the result of a continuation another dispatcher already started
and finished": the stale queued copy is passed after the run succeeded;
the stored run stays `succeeded` with its result, and no wake or release
follows. Red before the compare-and-set (the run was overwritten to
`cancelled`), green after.
- "returns the claim to the queue when run admission closes while the
claim is being checked": admission closes inside an awaited check; the
run stays `queued` with no `startedAt`, the task's execution lock and
the wake are unchanged. Red before the in-transaction re-check, green
after.

Red-to-green test through the real release path, in
`server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`:
- "delivers a person's message saved behind a system continuation
cancelled at claim because its task closed": a never-started legacy
continuation on a done task, with a person's deferred message behind it.
Red before the fix: the cancelled run had `resultJson: null`, with no
never-started proof. Green after: the proof is present, the wake is
skipped, and the message is promoted and runs once.

Test expectations changed because they asserted the old behaviour:
- `server/src/services/agent-interaction-delivery.test.ts` "cancels
refused queued runs through the public queue claim without dispatch":
now expects the cancel through `setRunStatusFromLive(runId, "cancelled",
["queued"], …)` instead of `setRunStatus`.
- `server/src/services/heartbeat/queue.test.ts` (refused-assignment held
case): asserts `setRunStatusFromLive` is not called, instead of
`setRunStatus`.
- The `issue_commented` case of "does not start a %s run…" now asserts
the run is claimed and `running` (it previously asserted only that the
error code was not `issue_closed_before_start`).

Local command, on this branch based on current master (includes #21234;
the dispatch ranking it changed is untouched here):
- `cd server && pnpm exec vitest run
src/services/heartbeat/queue.test.ts
src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the
proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue
test passed. (The file's `afterAll` database cleanup hook times out on
this loaded machine on master too; it is not related to this change.)

Why the local release-swap wave does not show the yield fix:
- The "yield to the replacement" signal exists only when the run-start
handover is created (`server/src/index.ts`, about line 1320). That
happens only when a direct deployment listen URL exists, that is, with
`PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL`
(the volume-free configuration).
- The local short form of the wave leaves `--ephemeral-server` out, so
the handover is disabled and `isRunAdmissionClosed()` becomes true only
at SIGTERM. The wave therefore cannot exercise the yield.
- The evidence for this fix is the unit test above and the reasoning
that every start path now reaches the same gate in `claimQueuedRun`.
Hosted confirmation must come from a volume-free instance.
- The closed-task fix was found in release-swap waves (an extra turn on
a done task in two tasks).

## Risks

- `claimQueuedRun` can now await an admission refresh. The refresh
returns at once when the cached state is fresh, so the normal path adds
no wait before `turn.start`.
- A system continuation for a closed task is cancelled. This matches
admission, which already refuses such wakes for a closed task.
- No migration. No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 21:36:05 -06:00
DottaandClaude Opus 5.5 dfd2a6efca fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs on a remote sandbox keep a warm Runner between turns. A
follow-up run takes over its settled predecessor's provider session and
Runner.
> - When a deploy stops the server, a follow-up that is still preparing
fails with `local_runtime_api_listener_unavailable`. The executor saves
this failure as a same-run bootstrap retry (#20786). The saved failure
proves that the run did no provider work of its own.
> - After the restart, restart recovery saw a checkpoint and a provider
session, and chose `reattach_remote_runner`. But the checkpoint and the
Runner identity belong to the predecessor. The Runner identity check
then failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The run ended terminal
and the task needed Retry.
> - This pull request recognizes that durable shape in restart recovery
and restarts the run as a fresh bootstrap on its warm session. It does
not try to reattach to a Runner that belongs to a different run.
> - The pull request also stops the queue from claiming a queued run
after shutdown closed run admission during the queue scan.
> - The benefit is that a deploy during a follow-up's preparation no
longer ends the run terminal with a Retry.

## Linked Issues or Issue Description

Refs #20786, #20928, #20852, #20904

**What happened?**
In three Codex runs across two Daytona restart-chaos waves (warm
sandboxes, CR1 and K15 with overlap), a warm follow-up run
(`automation`, `retryOfRunId` = its predecessor) started 0–2 s before
its server got SIGTERM. Its bootstrap failed with
`local_runtime_api_listener_unavailable` and saved a `bootstrap_retry`
with `providerSessionEstablished: false`, `providerEventsExist: false`.
After the restart, `native.recovery.transition` chose
`reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw
`identity_run_id`, because the warm Runner's identity still named the
predecessor. The run ended with
`native_continuation_requires_reconciliation`. In every sandbox capture
the Runner was cleanly suspended, with no stderr.

**Expected behavior**
The run did no provider work and never wrote its own Runner identity.
The next server must restart it without a Retry.

**Steps to reproduce**
1. Finish a remote warm native run, so that its provider session and
Runner identity are saved.
2. Start a follow-up run. Admission carries the predecessor's provider
session and Runner process identity into the follow-up.
3. Send SIGTERM while the follow-up prepares, so that it fails with
`local_runtime_api_listener_unavailable`.
4. Restart the server. Restart recovery chooses
`reattach_remote_runner`, and the run fails with `identity_run_id`.

## What Changed

- `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the
`remoteFollowUpNeverAttached` proof. It is true only when all of these
are true:
- the run is a failed, settled `bootstrap_retry` whose original failure
is `local_runtime_api_listener_unavailable`, with
`providerSessionEstablished: false`, `providerEventsExist: false`, no
successor and no lease owner;
- the run is on a prepared remote sandbox workspace with the current
preparation fencing version, it is not a plugin task, and it has no
Runner controller checkpoint and no staged attachment of its own;
- its session checkpoint is only the carried admission checkpoint (no
cursor, turn, result, terminal or pending request), and a settled
earlier run of the same session saved that exact provider session;
- its Runner process identity is absent or is exactly the identity of a
settled predecessor;
- it has no provider event, no Runner-originated event, and no
`native.process_identity_recorded` event of its own.
With that proof, the run gets a `bootstrap_incomplete` claim with the
reason `remote_follow_up_never_attached`, and it passes the same
current-admission check as the other settled bootstrap retries.
- `verifyRemoteRunnerReattachment` is unchanged. It still refuses a
Runner that names a different run.
- `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run
admission again before each claim. A run that a scan reaches after
shutdown closed admission stays queued for the next server.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: eight cases for
the staging shape. Three restart (inherited Runner identity; inherited
identity plus a Runner request; a Runner request with no identity). Five
stay held (own recorded Runner identity; a Runner-originated event; a
carried active turn; a provider session that no settled run saved; a
saved failure that established a provider session).
- `queue.test.ts`: a queued run stays queued and is not executed when
admission closes during the scan.

### How this relates to #20852 and #20904

- #20852 hands a preparing run to the next controller and persists
launch intents, so that preparation itself survives a deploy. It does
not change restart recovery classification for a settled
`bootstrap_retry`, and it does not touch the run-admission gate. This PR
covers runs that already failed and saved a `bootstrap_retry` (including
state that older releases saved), so the two are complementary.
- #20904 changes startup readiness for native-controller preservation
obligations. It does not touch either code path in this PR.

### Finding about the "draining server" admission

In both waves the follow-up was claimed before the process observed
shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and
started at 21:54:04.854Z; the server logged "shutdown signal received"
at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load
average 10). In the K15 overlap (seed 9513) the runs started at
23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at
23:31:13Z. During the overlap the old server is fully live and has no
signal that a newer deploy serves traffic. So the existing admission
gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the
gate was the check-then-claim window inside
`startNextQueuedRunForAgent`, which this PR closes. Holding work on a
superseded but not yet signaled server needs a new "superseded deploy"
signal; this PR does not add one.

## Verification

- Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs
packages/db/src/test-postgres-command.ts -- node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "closed the runtime listener before it attached"`: the three restart
cases failed with `"kind": "reattach_remote_runner"` (the staging
disposition); the five held cases passed.
- Red on master: `queue.test.ts -t "closes admission during the queue
scan"` failed (`expected [ { …(81) } ] to deeply equal []`).
- Green with the fix: both filters pass.
- `native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-startup-retry` and `native-controller-recovery` integration
files: 267 passed, 14 skipped.
- `queue.test.ts`: 89 passed.
- `cd server && pnpm typecheck`: passed.
- Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1)
and 9513 (K15 overlap).

## Risks

- The new proof applies only to the closed-listener `bootstrap_retry`
that #20786 writes, on a remote sandbox, with a carried session that a
settled predecessor saved. Every other shape keeps its earlier
classification.
- A `bootstrap_incomplete` claim sends the run through the existing
bootstrap path, which verifies the prior owner of the session root
before it starts. The s9205 follow-ups already use this path with an
inherited Runner identity.
- Upgrade safety: no persisted shape changes. The proof reads only
fields that the current and recent releases write. A row saved without
these facts keeps the earlier verdict.
- No wait or file work is added before `turn.start`. The new queries run
only in restart recovery.
- The queue change only stops claiming after admission closed. Runs stay
queued for the next server, the same as the existing entry check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:45 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5 c4ba3974a7 fix(heartbeat): close run admission before a graceful shutdown closes its listener (t09) (#20102)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A graceful restart (K2) drains the control plane: it closes HTTP
admission, suspends running runs, and the next boot resumes them
> - Run admission stayed open while the HTTP listener closed.
Heartbeat's shutdown flag was set only later, inside hot-restart
preparation
> - A run that finished in that window released its issue and promoted
the queued comment wake. That started a new run that had no callback
listener
> - A remote warm continuation then failed with
`local_runtime_api_listener_unavailable`, and the task needed a Retry
(restart chaos t09)
> - This pull request closes run admission before the listener closes,
so the promoted wake stays queued for the next boot
> - The benefit is that a comment sent during a deploy continues on its
own after the restart

## Linked Issues or Issue Description

Refs #19596

**What happened?**
In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, Codex C1,
K2), a K2 SIGTERM started a graceful drain at 19:42:52 UTC, and the HTTP
listener closed first. At 19:43:03 the running turn finished and
released its issue. That promoted the queued comment wake, and
`startNextQueuedRunForAgent` started a new run about 11 s into the
drain. Backend setup resolved the local runtime API URL, found the
listener closed, and threw `local_runtime_api_listener_unavailable`. The
remote target is excluded from the closed-listener handoff, so the run
failed, and the task waited for a manual Retry.

**Expected behavior**
No queued run starts once a graceful shutdown has begun. A wake promoted
during the drain stays queued, and the next boot starts it with no
Retry.

**Steps to reproduce**
1. Run a remote warm task and post a comment while its turn runs, so the
comment wake queues behind it.
2. Send SIGTERM to the control plane just before the turn finishes.
3. Before this change, a new run starts during the drain and fails with
`local_runtime_api_listener_unavailable`.

**Paperclip version or commit**
master at d007afbf86 (the wave ran on c5a172a183).

## What Changed

- `heartbeat/run-admission.ts` (new): a process-wide run admission gate
(`closeProcessRunAdmission`, `isProcessRunAdmissionClosed`). HTTP routes
build their own heartbeat instances, so the gate cannot live in one
instance.
- `heartbeat/queue.ts`: a new optional dependency,
`isRunAdmissionClosed`, which defaults to the process-wide gate.
`startNextQueuedRunForAgent`, `resumeQueuedRuns` and
`resumeQueuedRunsAfterDrain` start nothing (and read nothing) once it
returns true. Wakes are still recorded and promoted; only the start is
held.
- `heartbeat.ts`: also honors its own `shutdownInProgress` flag.
- `shutdown.ts`: `closeHttpListenerForShutdown` takes
`closeRunAdmission` and calls it first, before it checks or closes the
listener. `index.ts` passes `closeProcessRunAdmission`.

## Verification

- Red/green, new tests:
- `queue.test.ts`: "starts no run once shutdown closed run admission"
(for `startNextQueuedRunForAgent` and `resumeQueuedRuns`), and "honors
the process-wide run admission gate in every heartbeat instance".
- `shutdown.test.ts`: "closes run admission before it stops accepting
requests".
- `pnpm exec vitest run src/services/heartbeat/queue.test.ts
src/services/heartbeat/recovery.test.ts
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/services/native-runtime/native-startup-retry.integration.test.ts
src/shutdown.test.ts` (server): 5 files, 277 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (server): passed.

## Risks

- Low risk. Only queued-run starts are held during a shutdown that has
already begun; wakes, drains, suspension and finalizers are unchanged.
The next boot's queued-run dispatch starts held runs.
- Known remaining window: a run admitted just before shutdown can still
reach backend setup after the listener closed. A local runner hands off
to restart recovery there. A remote warm continuation still fails,
because the remote target is excluded from that handoff (#19596) and the
bootstrap retry does not cover warm continuations. This change removes
the t09 sequence, where the run started after shutdown had begun; it
does not remove that narrower window.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution, a read-only research subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 14:59:55 -06:00
DottaandPaperclip 384fa2b7be Allow context compaction while Agent Chat waits (#19801)
## Thinking Path

> - Paperclip manages agents and their work.
> - Agent Chat can wait for a user between useful turns.
> - The task Properties panel already lets that user request context
compaction.
> - The queue treated that maintenance request as an ordinary wake and
refused it while the chat waited.
> - Compaction must maintain that session without starting a new
conversation turn.
> - This change admits the authenticated request, preserves the waiting
state during execution, and prevents competing requests from saving
extra maintenance work.
> - Ordinary and unauthorized wakes remain blocked.

## Linked Issues or Issue Description

Refs #19378, which introduced the existing Compact now action. Refs
#19377, whose review identified this independent queue admission gap.
Searches for `context_compaction` with `waiting` and `Compact now` with
`waiting` found no existing fix for this admission bug.

**What happened?**

An owner clicks Compact now for a waiting Agent Chat conversation. The
real compaction service reaches the heartbeat queue, receives no run,
and reports that compaction did not start.

**Expected behavior**

The authenticated owner can request one separate compaction run while
the chat waits. The request must keep its existing actor, task, agent,
budget, and execution checks. Ordinary waiting-chat wakes must stay
blocked.

**Steps to reproduce**

1. Use an open Agent Chat conversation whose agent supports compaction.
2. Wait until the conversation is waiting for its owner.
3. Request Compact now from the task Properties panel.
4. Observe the refusal before this fix. The new database regression
calls the real compaction service and queue to reproduce it.

## What Changed

- Admit only the existing complete manual compaction envelope. Require a
user actor, no coalescing, the compaction intent, and matching nonempty
request IDs.
- Preserve the waiting conversation state, session generation, reply
boundary and unanswered questions in the existing conversation
preparation owner.
- Recheck active and deferred maintenance under the existing issue
admission lock. Competing clicks cannot save a second maintenance
request.
- Add execution coverage with and without a pending question. Add
concurrent admission and deferred-request controls, including admission
after the pending receipt is cancelled. Keep the existing authorization
and ordinary conversation controls.

## Verification

- The original service-to-queue request failed before the admission fix.
- The execution regression then caught the review finding: compaction
succeeded, but both waiting-state controls failed. The same complete
controls passed after the conversation preparation fix.
- A real concurrent request test recorded two durable maintenance
receipts before the atomic admission fix. It counts deferred authority
as well as admitted runs.
- Current integrated candidate: normal current Runner core and surface
TypeScript exports passed. All seven normal App typecheck programs
passed. The whole queue/execution suites passed 86/86, the whole
context-compaction suite passed 5/5, and 11 selected ordinary
conversation controls passed. There were 102 actual passes; the 29
unselected conversation cases are not counted as passes.
- Database controls use fresh dedicated PostgreSQL 17 databases, Node
24, one test worker and pools capped at four. Private fixture adapters
use the existing dedicated database and normal migrations. They keep
test bodies, assertions and time limits intact.
- Earlier fixture and setup failures remain recorded separately from
product failures. These include an omitted external URL, an invalid
closed-task fixture, a missing crypto import, an incorrect
result-envelope matcher, and an unavailable test-library helper. Each
was corrected at its owning boundary. A test-output text matcher also
missed Vitest's abbreviated failure; the raw assertion stack and failure
were retained.
- Local session tests exercise the actual admission, native Runner
lifecycle, accepted result and persisted conversation state through the
supported test backend. They do not prove live provider compaction, UI
rendering or deployed suspension.
- Exact-head remote CI passed: 59 successful checks, three intentional
skips and one neutral check. Greptile and Ponytail both reviewed this
head at 5/5 with no actionable findings. Ponytail explicitly marked it
Ship.
- Master advanced during review. A local merge evaluation preserves
every unrelated master change and all four PR files. The current App
programs and all three execution controls passed again on that projected
squash tree to check the workspace-finalization seam.

## Risks

The change affects admission and preparation for one maintenance
request. A broad exception could start ordinary work or reopen a waiting
chat. The request therefore keeps the complete existing manual envelope
and authorization checks. Conversation preparation still checks the
session generation. The existing issue lock owns duplicate admission.
There is no new timer, database migration or provider protocol.

## Model Used

OpenAI Codex based on GPT-6 managed the change and reviewed the shared
queue fix. OpenAI `gpt-6.1-sol` authored the focused regression
coverage. Both used reasoning, repository tools and local code
execution. The runtime did not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing Compact now documentation remains accurate; no new
configuration is added)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 05:11:44 -06:00
DottaandPaperclip 6364fa7327 fix(heartbeat): wait for retry predecessor cleanup (#18989)
## Thinking Path

> - Paperclip manages agent work and preserves its task ownership.
> - Heartbeat admission controls when a queued run can start.
> - A failed run can still own workspace saving and environment cleanup.
> - Automatic recovery can transfer the task execution lock to its
queued retry before that cleanup finishes.
> - The old queue check then sees the retry itself and misses the
predecessor's remaining work.
> - This pull request checks the exact retry predecessor before claiming
the queued run.
> - Existing cleanup notifications can then resume that same run through
normal admission.

## Linked Issues or Issue Description

Refs #18898 and #18691. The first repair covers canonical connected-task
retries with a native source. The second covers live remote workspace
writers. This change covers ordinary automatic continuation when the
predecessor failed before native runtime selection. The search found no
open repair for that path.

**What happened?**

A preparation failure saved a terminal legacy run row while its remote
workspace save and environment cleanup were still active. Generic
continuation created one queued successor and moved the task execution
lock to it. The queue checked that current lock but did not check the
successor's predecessor. The new run started and failed at the existing
unfinished-writer guard.

**Expected behavior**

Keep that same successor queued until its exact predecessor finishes
execution and cleanup. Check company, agent and task identity before
using retry provenance as a wait condition. Preserve Stop, reassignment,
failed cleanup and independent recovery gates.

**Steps to reproduce**

1. Save a failed legacy preparation run with an active environment
lease.
2. Queue its ordinary automatic continuation and bind the task execution
lock to the successor.
3. Claim the queued run before the predecessor finishes cleanup.
4. The old code starts the successor. The fixed code retains it in the
queue.
5. Commit successful cleanup through the existing cleanup owner. The
ordinary dispatcher starts that same run once.

**Paperclip version or commit**

The causal regression fails on master
`86814c6550f52e99ede150a883bdc773cbb540e3`.

**Deployment mode**

Source checkout on macOS arm64 with disposable PostgreSQL, the assembled
heartbeat service and the production work coordinator. The adapter
boundary is controlled. No hosted provider replay or deployment is
claimed.

## What Changed

- Check the current task execution owner and the retry's exact
predecessor under the existing task claim lock.
- Use the saved native task binding, ordinary task binding and
historical context fallback in that order. Ignore unrelated retry
provenance.
- Reuse the existing live-executor and pending-cleanup checks. Reuse
committed cleanup notifications and the existing dispatcher.
- Add real PostgreSQL claim controls and assembled cleanup-event
acceptance. Keep the original source failure, saved files and current
task ownership intact.
- Document the queued predecessor-cleanup rule.

## Verification

- RED: the unchanged queue claimed the successor and wrote a new
controller lease while its legacy predecessor retained an active remote
environment lease. The regression expected the successor to remain
queued.
- GREEN: the same regression passes after the change. The complete queue
suite passes all 71 tests.
- The assembled heartbeat suite passes all six tests. The added cases
cover one continuation after committed cleanup, saved-file reuse, final
task lock release, board Stop, reassignment and failed cleanup.
- Scope controls cover a deleted source after scan, another company,
another task, another agent, canonical native task binding and the
existing current-owner hold. A failed cleanup and a still-live local
executor retain the wait.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass.
- Initial fixture attempts failed on database constraints and observer
timing. The final fixtures use valid immutable bindings, the board Stop
attribution and a completed real dispatcher call after reading the exact
committed lease state. Production ownership guards were retained
throughout.
- Full low-memory repository typecheck passed at `5cddb49322`. The sync
to master `04a24f4e06` adds only the independent retained-allocation
test fixture from #18984; production code is unchanged.
- Full `pnpm build` passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`.
- The post-sync run executes all 80 behavior assertions successfully
across the queue, assembled cleanup and updated upstream retention
suites. Its queue `afterAll` then exceeds the existing 30-second
database shutdown hook. That failed suite result is retained. The
controlled post-build queue rerun passes all 71 tests with normal
teardown. The earlier shutdown cause remains undetermined. Its owned
queue-test PostgreSQL process has exited. The separate local default
`pnpm test:run` was intentionally stopped after full unfiltered CI
passed on the exact head. Four local chat partitions completed before
that stop; this is not a local full-suite pass. The verified owned
process group and its PostgreSQL fixture have exited. Logs and retained
fixture data remain available. The timeout has not been increased.
- Full unfiltered CI passes on current head
`c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all
six internal chat partitions, all nine serialized server shards, the
workspace groups, native Runner lanes, typecheck, build and browser E2E.
Both required [ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613)
and [ci /
e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146)
pass. The file selector is skipped, so this is the full CI test matrix.
The head has 55 successful checks and three expected skipped checks. The
ready-for-review contributor trust and security scan also pass.
- Current-head Greptile is 5/5, its check is successful, and no inline
review threads remain. The ordinary review check also passes. No merge,
deployment or original task retry was performed.

## Risks

- A predecessor with unfinished cleanup keeps its successor queued. Its
existing cleanup owner remains responsible for settlement.
- A task claim reads at most one additional predecessor row. A matching
source also uses the existing ownership and cleanup queries. The change
adds no timer, schema, credential, provider or model policy.
- A committed cleanup signal only requests normal admission. Current
ownership and independent execution gates still apply.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and
PostgreSQL verification. The session does not expose a more specific
serving model ID or context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused suites pass; the
separate local full-suite attempt is explicitly incomplete above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:16:54 -06:00
Devin FoleyandPaperclip 1a39dea9aa fix(heartbeat): keep a routine claim queued while its slot is held (#18933)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A routine creates one task per tick. Only one open task of a routine
can hold an execution lock (`issues_open_routine_execution_uq`).
> - Since #18879, a failed native run that waits for a scheduled resume
keeps its task's lock. Routine dispatch counts only queued, running or
scheduled runs as live, so it can queue a newer task of the same routine
meanwhile.
> - When that newer run is claimed, stamping its execution lock violates
the unique index. The raw database error is not a deferrable claim
rejection, so it escapes the queue loop. The queue code notes that this
can stop the agent's queue and startup recovery.
> - This pull request turns that exact violation into a 409 claim
rejection, so the run stays queued until the slot is free.
> - The benefit is that a busy routine slot delays one run instead of
stopping the queue or startup.

## Linked Issues or Issue Description

Refs #18879 (keeps a failed run's lock while its resume is pending),
#18916 (defers the same unique-index conflict in restart recovery's lock
reclaim) and #18931 (open; keeps one run's restart-recovery failure from
stopping startup recovery). This is the same index conflict on a
different path: the heartbeat queue claim, not restart recovery. It does
not change #18916 or #18931. Its doc paragraph is separate from the one
#18931 edits.

**What happened?**

While auditing the lock paths from #18879 for this index, I found that a
queued run of a newer routine task can be claimed while an older task of
the same routine still holds the slot through a failed run with a
pending resume. The claim's lock stamp then raises `duplicate key value
violates unique constraint "issues_open_routine_execution_uq"`.
`isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error
escapes `startNextQueuedRunForAgent`.

**Expected behavior**

The claim is rejected as a conflict and rolls back. The run stays queued
and is claimed after the older task releases the slot. Other errors
behave as before.

**Steps to reproduce**

1. Create two open tasks of the same routine (same origin and
fingerprint).
2. Give the older task an execution lock held by a failed run.
3. Queue a run for the newer task and claim it. On `master` the claim
throws the unique violation.

The new queue test runs these steps against PostgreSQL.

**Paperclip version or commit**

`master` at `0dedbf23c35`.

**Deployment mode**

Any server with routines whose tasks run on native runners.

## What Changed

- `bindClaimedIssueExecution` in
`server/src/services/heartbeat/queue.ts`: if stamping the execution lock
violates `issues_open_routine_execution_uq`, it throws
`conflict("Another execution for this routine is already in progress")`.
The claim transaction rolls back, and the queue loop keeps the run
queued as it does for other 4xx claim rejections. Every other error is
rethrown unchanged. The issue checkout route already maps this violation
to the same 409.
- `doc/execution-semantics.md`: one paragraph on the behavior.

## Verification

- Failing first: `queue.test.ts` "keeps a claim queued while another
task of its routine holds the execution slot". On `master` the claim
raised the `issues_open_routine_execution_uq` violation. Now it rejects
with 409, the run stays `queued`, neither task's lock changes, and
nothing executes.
- Failing first, review round 1:
`heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run
queued while another task holds the routine slot, then runs it once the
slot is free" goes through `resumeQueuedRuns`. On `master` it rejected
with the unique violation. Now the routine run stays queued with no
error code while another queued run for the same agent executes. After
the slot is released, the routine run executes exactly once (executions
are counted per run, because the finished task can queue its own
follow-up run).
- `cd server && npx vitest run src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed.
- `npx vitest run src/services/heartbeat/recovery.test.ts
src/__tests__/routines*.test.ts`: 4 files, 205 passed.
- `pnpm --filter @paperclipai/server typecheck`: pass.
- Not verified: the queue loop end to end on a real instance. The loop's
handling of a 4xx claim rejection is existing behavior.

## Risks

- While the older task holds the slot, the newer run stays queued and
the loop logs "queued heartbeat run claim was rejected" on each pass. It
is claimed once the older task's resume finishes or its lock is
released.
- Only this one constraint is mapped. Other database errors still escape
as before.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, test runs). The runtime does not expose the context
window or reasoning configuration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 21:33:00 -08:00
DottaandPaperclip f211d44c00 fix(heartbeat): preserve queued coalesced comments at claim (#18913)
## Thinking Path

> - Paperclip manages agents and their work.
> - The wake queue records task comments before an agent starts.
> - A queued run can accept another comment through coalescing.
> - Claim rebuilds that run's input from its primary wake.
> - Coalescing updated the run and its receipt but left the primary
batch unchanged.
> - This change saves the accepted batch atomically in the owning
writer.
> - The benefit is complete input at claim without repeating old work.

## Linked Issues or Issue Description

Related: #18887 preserves offline queued turns. #13948 addresses
equivalent automatic wake deduplication. This change repairs the
accepted comment batch at the existing claim boundary.

**What happened?**

A queued run started with comment A. Admission accepted comment B, saved
`[A, B]` in the run context, and finished B's coalesced receipt. The
primary wake still held `[A]`. Actual claim rebuilt input from that wake
and dropped B.

**Expected behavior**

Claim must retain every accepted live comment in order. Stop, changed
source authority, deleted comments, and company boundaries must still
apply. Already claimed wakes must retain their existing recovery source.

**Steps to reproduce**

1. Queue a task run whose primary wake has an authoritative comment
batch `[A]`.
2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the
real Postgres admission writer.
3. Confirm that admission accepts `[A, B]` and records B's coalesced
receipt.
4. Claim the original queued run. Before this fix, the claimed input
contains only A.

**Paperclip version or commit**

The untouched producer regression fails on
`d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native
Runner/Codex adapter selection. Provider execution is disabled in the
fixture.

## What Changed

- Save the accepted comment batch in the unclaimed primary wake, run
context, and coalesced receipt in one transaction.
- Lock the primary wake before the run. Recheck the expected run status
and same-company, same-agent and run bindings; apply the existing actor
equality fence only to durable receipts before mutation.
- Preserve incoming input as normal deferred work when the locked source
changed. Preserve receipt identity and actor fields.
- Preserve the original primary actor and trigger comment while updating
its accepted batch. Each incoming receipt retains its actual actor. Keep
claimed source payloads and legacy envelopes on their existing recovery
paths.
- Safely defer a same-named different agent without rewriting the
original agent's run, wake or receipt. Missing and foreign-company
targets remain rejected. Do not rebuild input from historical receipts.
- Add actual process/native producer, real database rollback, Stop,
deletion, foreign-source, and fallback controls. Update three direct
writer fixtures for the required expected-status contract.

## Verification

- The untouched process and native producer red on
`d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at
actual claim. The initial incomplete mock is retained separately and
excluded from behavioral proof.
- Current process/native matrix: 22 passed, including mixed authors,
system primary, same-named different agents, changed status, claimed
sources, Stop, deletion and foreign-source boundaries. Queue, Postgres
admission and use-case suites passed 234 cases before the
source-preserving serializer follow-up; its 22 actual producer cases
passed again afterward.
- CI on the previous head found a real ordinary mixed-author
workspace-copyback regression. The earlier 13-case selection omitted
this test; that coverage gap is preserved. The first correction exposed
a second real source-provenance/copyback failure. After preserving the
primary trigger and actor while updating only its batch, the complete
workspace recovery file passed all 188 cases, including ordinary
mixed-author input and native copyback.
- Real HTTP queued-comment consumption, discard/reorder and Stop
boundaries: 101 passed before these review corrections. Existing chat
and Slack consumers each passed their selected real writer case.
- Previous-head CI also timed out in the CLI child-server startup
fixture and a ZIP transfer case. The full untouched transfer file passed
17 cases locally. Startup remains unclassified; its intentionally
reserved port is part of the fixture and is not diagnosed as the cause.
No CI retry was issued.
- An earlier queue run passed its 231 assertions but timed out during
teardown; its preserved replay passed all 60 queue cases. This is
distinct from the behavioral reds.
- Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers
and production server/Runner build passed. The unchanged-default
serializer file passed all 11 cases. Current-head CI and both
substantive reviews remain required; previous grades and checks do not
qualify this head.
- No hosted task was retried or changed. Local tests do not establish
any hosted deferred event's cause, provider execution, native
acknowledgement, physical cleanup, or serving revision.

## Risks

- Admission now performs bounded row reads and locks for each coalesced
wake. It adds no idle scan or remote request. The existing task
transaction owns the operation; lock order follows queued claim.
- Claimed-wake recovery uses source identity. Its payload remains
unchanged, and the existing actual native copyback tests cover that
boundary.
- A source that changes before the lock can defer the new input. This
preserves work instead of consuming it into a stale run. No migration or
historical backfill is required.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell
execution, and test tools. The runtime does not report the
context-window size. A separate OpenAI coordinator reviewed the source.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 22:09:58 -06:00
DottaandPaperclip f187de99aa Preserve domain deadlines across owned idle sleep (#18656)
Reviewed and verified at 9b4acc456d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 15:40:56 -06:00
DottaandPaperclip 27d35314be feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Native Runner owns execution and task authority.
> - Operators need an explicit native Cursor Cloud choice.
> - Tools/files need company, task, session and run scope.
> - This connects setup and lifecycle to Runner Core.
> - Legacy agents retain their selected runtime/history.

## Linked Issues or Issue Description

Depends on #18068. Managed-only API placement and API-owned Stop
containment are in #18310. Refs #5664, #11803, #17299, and #17471. No
duplicate native integration was found.

## What Changed

- Add explicit native setup, Test, saved settings and dispatch with
no-prompt account/catalog/callback checks.
- Scope MCP to company, agent, task, session and run through
authenticated HTTPS. Register verified artifacts/work products; preserve
unpriced accounting, completion and budgets.
- Connect Question/Plan continuation, cold handoff and admitted fresh
sessions; recheck waiting admission under the task lock.
- Present audited Stop intent while preserving physical containment
proof; bound typed Stop metadata. Preserve title edits through
preference/navigation readiness.
- Select one live/history representation per run to prevent duplicate
settled cards. Integrate upstream task/read-state/retry/schema contracts
and repair two test fixtures.

## Verification

Core #18068 merged normally as `3910352c923a`. Product head
`5a60f541c982` has passing repository CI/build/browser checks, including
required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile
5/5 review with zero unresolved threads. Two non-required external app
checks remain pending: Paperclip Review is queued waiting for its
assigned agent; Superagent Security Scan is reviewing the diff. These
are not claimed completed. The normal merge endpoint enforces the
repository rules; no policy or status override is used. The dependent
Top PR is addressing its fresh deferred-Stop review finding.

Product `5a60f541c982` passed 270 UI and 20 selected
Cloud/native/disk-full tests, UI/server typechecks, tokens and maps.
Top’s clean normal merge passed tokens/maps/diff checks; its matrix is
pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native
retirement and 19 settlement controls, normal Runner build, and runtime
9af build/recursive typecheck. Reproduced loading/shutdown CI fixture
failures passed 7/7 browser and 121/121 unit tests, including a
broken-unsubscribe negative control. No timeout/oracle was relaxed.

The actual-parent restack adopts 75 mainline paths, including
schema/permission and generic runtime changes, beyond the prior two
fixture repairs and client projection fix. The full difference from
hosted `9af1a5de42a2` is therefore larger than four files; backend
source also differs. The merged config-read checks match Core391, and
the Cursor placement/runtime/relay/selection modules match frozen9af;
selected affected regressions pass. Paid proof keeps its actual 9af
source/image identity. The separately tested live/history card fix is
not claimed deployed on that host.

On 9af, original saved settings survived reopening and fresh no-prompt
Test authenticated catalog/model/binding; no new Save is claimed. Fresh
Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded
exact 27 bytes and matched 121 canonical identities in complete
S3-served history. First Stop’s strict quiet FAILED on a 10.373-second
gap despite independent cancellation/no late artifact. One targeted
unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet /
maximum gap 4.584 seconds, 67 ordered canonical identities, and separate
CANCELLED/artifact-absent status after the original 180-second deadline.
The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for
the three new provider agents is awaiting authorization, so full grades
and the fresh Artifact’s post-cleanup download remain blocked. A
historical deleted-provider artifact was newly downloaded on 9af;
original F226 Question/Plan/reconnect and cleanup proof keep their
identities.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier
title/startup/custody/Stop/deletion failures with unknown causes/bounded
recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair
ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite
attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun
passed 128. Retry alone establishes no cause or full local-suite pass.
All paid attempts remain recorded; vendor dollar cost is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and checked through behavior; unpriced provider dollar
cost stays UNKNOWN.

Callbacks need authenticated reachable HTTPS. Scope, revocation,
budgets, completion and cleanup remain authoritative. Operator Stop
intent is separate from physical containment. Saved legacy agents retain
their selected runtime. Prior shared title-save/deletion failures remain
disclosed.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 09:01:39 -06:00
DottaandPaperclip 61bd4ac691 feat(agents): set up Antigravity with saved Google subscriptions (#18189)
## Thinking Path

> - Paperclip manages agents and their work.
> - Users choose providers and managed accounts while adding agents.
> - Antigravity was missing from the ordinary additional-agent picker
and its saved Google path did not complete native readiness.
> - A user needs a successful Test, an enabled task action, and a stable
account binding.
> - This pull request connects the ordinary setup flow to the native
candidate.
> - It also preserves retired credential refreshes across database
failures and server restarts.

## Linked Issues or Issue Description

Built on merged #18188 and #18187. Refs: #12503, #14938 for related
legacy CLI proposals.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission remains required; the local candidate result does not
enable general release.

## What Changed

- Show Google Antigravity in the ordinary adapter grid and preserve the
draft choice when setup resumes.
- Reuse the same user’s saved personal default Google subscription and
discover the exact low model through the native account.
- Run a prompt-free native Test and wait for authoritative readiness
before enabling task assignment.
- Carry the managed account through native runs and recover retired
refreshes through the existing encrypted, fsynced journal with exact
generation checks.
- Retain the latest rotated credential through verification, storage,
and partial home-removal failures. Retry the exact account’s retained
cleanup before another provider use.
- Reconcile cleanup before taking the credential mutation lock, fence
owners arriving during lock acquisition, and support nested lock holders
without a join cycle.
- Bound sign-in retirement outside database transactions and preserve
claims for the existing cleanup owner; one pending sign-in does not
block unrelated accounts.
- Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model
editing usable, and keep adapter discovery available after candidate
expiry.
- Skip taskless native timer work before run or credential admission.
Preserve current master monitor and deadline scheduling.

## Verification

- The registered local saved-Google-subscription journey passed all 16
matchers in 196,520 ms using eval source `12aba` against actual serving
source `4a37ee7c`. Both turns stayed in one conversation, produced exact
downloaded files of 61 and 119 bytes after reload, and used distinct
native primary processes. All observed owners retired, leases released,
and credential fences settled. Independent readback confirmed Done work,
no active owned runs, a connected saved account, and the retained agent
paused for inspection.
- The measured runtime is production-equivalent to pre-restack
`65ef31dc` except its test-only transport fixture correction.
Current-master head `ff7c9249` preserves all 99 product paths: 98 exact
blobs and only eight inherited plugin-panel lines in `agents.ts`. Its
native binary was rebuilt for inherited Codex-only diagnostic changes;
the changed Rust behavior is outside Antigravity's ACPX execution path.
Shared native/UI changes have focused regression coverage. The recorded
live result is not represented as a measured replay of `ff7c9249`.
- Full workspace build and typecheck passed on `ff7c9249`. All 376
focused tests in 14 files passed, covering Google credential/retirement
controls and the inherited native-tool, redaction, route, and UI
recovery changes. Token gates, maps, and diff checks passed. The
required-close fixture was separately demonstrated red, then passed all
317 selected provider-wiring cases.
- [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972)
is green: 55 successful checks, two skipped workflow checks, and an
optional neutral review check. Required `ci / verify` and `ci / e2e`
passed. Greptile and Paperclip Review both report 5/5 on exact
`ff7c9249`, with zero unresolved review threads.
- This qualifies the selected operator-authorized local
saved-subscription candidate. The original exploratory journey remains
historical evidence; the repeatable fixed-driver eval is in the next PR.

## Risks

- Google refresh recovery must preserve the exact grant, account, and
credential generation. An uncertain native owner or failed encrypted
handoff retains the private home and blocks cleanup.
- A successful local saved-subscription candidate does not qualify fresh
OAuth, API keys, Cloud, Linux, other models, or general release.
- Timer filtering changes only taskless native Runner admissions.
Existing task-bound monitors and legacy adapters retain their owners.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 16:51:50 -05:00
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00
Devin FoleyandPaperclip 8a85d9d647 perf(heartbeat): coalesce repeated native stranded releases (#17547)
Skip repeated successful native release checks for the same unchanged source within one recovery pass. Preserve every saved request, failure retry, cursor update, source change, legacy path, and current admission gate.

Verified with 201 focused tests, 32 full-control queue cases, build and typecheck. Exact-head CI is green, Greptile is 5/5 with Low impact, and no review threads remain unresolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-24 21:04:36 -07:00
Devin FoleyandPaperclip 14e1f59fbf fix(heartbeat): avoid large run reads during deferred-message recovery (#17460)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - Deferred messages must survive restarts and execution cleanup.
> - Periodic recovery checks the latest run before it attempts release.
> - That check and two release checks load large execution records.
> - These checks use only admission evidence and completion identity.
> - This change selects those fields while keeping the release policy
and fresh reads.
> - Full execution records remain available for execution and detail
reads.

## Linked Issues or Issue Description

**What happened?**

Deferred-message recovery repeatedly transfers stored context, Runner
profiles,
and output excerpts to make small admission and release decisions.

**Expected behavior**

Select the fields each decision consumes. Preserve queued messages,
current
admission checks, cleanup ownership, and fresh completion reads.

**Steps to reproduce**

1. Save a terminal run with large context, profile, and output fields.
2. Leave an accepted message deferred after the issue lock is released.
3. Call `resumeQueuedRuns` and follow execution release.
4. Observe full-row reads before the admission decision.

**Paperclip version or commit**

Baseline: `410e5c6eedb`. Related read optimization: Refs #17405.
Searched open and closed PRs for run projections and stranded-queue
reads.
Searched open issues for database transfer. No matching fix was found.

**Deployment mode**

External PostgreSQL. The regression tests use disposable real
PostgreSQL.

## What Changed

- Select seven admission fields for the stranded queue's latest-run
lookup.
- Select authentication classification evidence before execution
release.
- Read completion status and typed issue identifiers after post-commit
effects.
- Keep company scoping on both release reads.
- Add large-payload and fresh-state regressions. Document the read
contract.

## Verification

- Focused PostgreSQL tests: 68 passed across queue, run control, and the
new release-read suite.
- Release reads return less than 2 KiB with three large synthetic
execution fields.
- Queue recovery passes less than 512 bytes to release, retaining the
full stored message.
- Coverage includes legacy/native owners, paused recipients, live
successors, authentication repair, and fresh completion identity after
post-commit effects.
- Root recursive typecheck and full build pass. The full local stable
suite remains in progress; no local full-suite pass is claimed.
- All current-head CI checks pass on
`c819fbb6b46bd5480a197407f06e02f245667792`. Greptile: **5/5, medium
impact**, zero comments and zero unresolved threads on that head.
- The first focused attempt found a syntax error in the new test; it was
corrected before the 68-test passing run. A redundant standalone
compiler was stopped while the required recursive check continued.
- Hosted impact is not measured yet. Staging verification follows merge.

## Risks

The projection must retain all admission evidence and preserve JSON
types.
The release policy still owns transactions, locks, cleanup, and
promotion.
There is no schema migration, polling change, message deletion, or API
change.
Configuration repair evidence is read directly, including when unrelated
result output is large.

## Model Used

OpenAI GPT-6 through Codex, with source inspection, code execution, and
GitHub tools.
The exact served model ID and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 12:07:45 -07:00
DottaandPaperclip dff63551fe fix: resume unfinished work after repository preparation (#17047)
## Thinking Path

> - Paperclip manages agents that perform work for a company.
> - Task repositories are prepared at normal run admission.
> - A running native task can stage a repository while its work remains
unfinished.
> - The controller did not offer that saved setup as a continuation.
> - The agent could add a timer or wait for a user restart.
> - This change uses the existing same-agent continuation and status
ledger.
> - The accepted explanation and proven conversation remain available
after setup.

## Linked Issues or Issue Description

Refs #17045. That prerequisite has merged. This PR targets master.

**What happened?**

A projectless native task can request a repository for the next
admission. An unfinished task has no server-selected repository
continuation. An explicit same-agent continuation also loses chat
provenance.

**Expected behavior**

An unfinished standard task can yield and resume through the existing
admission queue. Its explanation and final result retain the authorized
conversation. A task that only stages a repository can complete without
another run.

**Steps to reproduce**

1. Start a native standard task in a task directory.
2. Request an authorized repository through prepare_repository.
3. Yield with unfinished work and no explicit continuation.
4. Observe that the former driver rejects the yield, although the
pending repository is saved.

## What Changed

- Offer a saved same-agent continuation for the current task's pending
repository.
- Accept that current server continuation in the shared provider driver.
- Resolve chat provenance through applied status decisions, delivered
effects, and admitted wakes.
- Preserve accepted setup explanations without repeating an owned
question card.
- Document optional reproduction details and configured repository
discovery.
- Extend database regressions and workspace access maps.
- Preserve Copilot profile v52 and bind the shared continuation source
in v53. Include the active identity in the QA image fingerprint.

## Verification

- Five focused server suites: 100 checks pass.
- Shared continuation driver: 12 checks pass.
- Server typecheck passes.
- Map contract and whitespace checks pass.
- Existing API and driver assertions fail on the former implementation.
- Real dispatcher and queue cohort: 43 checks pass. The original
reply-loss regression fails before the dispatch receipt fix.
- Continuation driver and installation identity cohort: 62 checks pass.
- Regenerated capability inventory and contract checks pass.
- Final response precedence cohort: 109 checks pass. The stale-comment
case fails on the former implementation.
- All required CI checks, build, and both source reviews pass on
b9c8677635.
- Live provider journeys and deployed acceptance remain pending. No live
qualification is claimed.

## Risks

A pending repository now offers a next admission when unfinished
standard work yields. Incorrect eligibility could create unwanted work.
The lookup requires the current workspace and task. Explicit completion
does not use it. Existing Stop, admission, approval, and transport
authorization remain authoritative. No schema or public task-status
change is included. The shared continuation change requires a new
Copilot source identity. This PR preserves v52 history and does not
claim fresh Copilot live qualification.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose its exact deployment ID or context-window
size. Those values are not inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:17:50 -05:00
DottaandPaperclip 505e1a5582 feat(cases): add persistent workflows with same-task agent handoffs (#16959)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent workflow needs persistent facts and a safe way to transfer
work.
> - Creating a task for each phase splits the objective, discussion,
workspace, and review policy.
> - A case can retain the facts while one primary task moves between
agents.
> - Transfers must survive process failure and preserve private access,
budgets, and completion gates.
> - This pull request adds durable case handoffs, native case tools, and
read-only case pages.
> - Prompts describe the workflow. Existing task controls decide when
work can run and finish.

## Linked Issues or Issue Description

**Problem or motivation**

Prompt-driven agent teams need a persistent record that can move through
triage, implementation, verification, and correction. Ordinary
reassignment does not provide a durable instruction, safe release
barrier, or recoverable delivery receipt for an agent handing off its
own running task.

**Proposed solution**

Extend standalone cases with one primary task, a descriptive phase,
retained access sources, and a handoff ledger. Apply assignments after
source execution and workspace settlement. Deliver each transfer through
the existing durable wake queue. Provide `create_case`, `read_case`,
`update_case`, and `handoff_case` in the native runner. Creation
requires kickoff and explicit workflow text. An explicit handoff to
`done` closes the case when normal task completion is accepted.

**Alternatives considered**

Do not add a workflow engine, filesystem queue, configured state graph,
or one task per phase. The workflow document combines transition prose
with an optional Mermaid `stateDiagram-v2` map. The server does not
execute the diagram. The Pipelines experiment removed on master stays
removed.

**Roadmap alignment**

This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run
Recovery, and Activity Log attribution. Existing approval, review,
budget, privacy, workspace, and responsible-user rules remain
authoritative.

Related work: [ordinary handoff wake
recovery](https://github.com/paperclipai/paperclip/pull/15019). This
feature adds a separate persisted case-handoff receipt and release
barrier.

## What Changed

- Add case routing fields, retained access sources, durable handoff
history, delivery leases, and replay-safe migrations.
- Keep the same primary task across agent phases. Preserve each
returning agent's own session and the task's workspace.
- Fence outgoing mutations, wait for provider and workspace release, and
recover delivery without duplicate task comments or runs. Recheck
current assignment permissions as the original caller before applying a
delayed transfer. Scope a run Stop to that run’s own handoff intents, so
repeated old Stop requests preserve newer work. Cancelling terminal
intent without a source run also requires fenced current task ownership,
including native retries and preparation with an old checkout. For Cloud
callers, revalidate current stack membership with an uncached Cloud RPC
and derive owner elevation from the current managed setting.
- Keep case reads, documents, assets, Audit, and live events inside
retained task/project permissions.
- Add native case creation/read/update/handoff tools. Store required
workflow text and initial documents on the first task. Page oversized
case facts and referenced document bodies through the dedicated tool
with bounded UTF-8 JSON, current routing versions, snapshot conflicts,
and current source authorization on every page.
- Add read-only case lists and detail pages, task-style list controls,
compact dotted field tables, linked handoff tasks, and case filters in
Audit.
- Add native/legacy integration tests, migration/restart tests, browser
coverage, and bounded live receipt fixtures. Extend the trusted sandbox
condition to cover the new subscription fixture profile.
- Hide obsolete paused-agent warnings on completed or cancelled task
threads.
- Synchronize the company-switch settings test with React effects after
a CI failure. Preserve its security assertions and production remount
behavior.

## Verification

The branch directly includes master `9ef434fe79`. The required `ci /
verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614)
passes typecheck, build, tests, and all eight browser shards. Security
checks pass. [Greptile gives this exact head
5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549),
with no unresolved review threads. The optional Paperclip Review check
remains queued because its assigned agent is paused. It is not a
required branch check. No check result was overridden.

Local workspace typecheck and build pass after the master merge, along
with database migration checks, token gates, map checks, Storybook
catalog checks, and the Copilot profile identity check. Standard builds
refresh Runner vendor output. Focused checks pass 2,008
API/paging/context tests and 7 strict case protocol tests. The final
commit changes one test fixture only; its ten tests, UI typecheck, and
token gates pass. No production source changed in that commit.

On current head `765fccd67c`, all **194 critical workflow tests**
executed and passed with no skipped tests in those files:

- 38 Stop/ownership tests, including board completion intent, native
retries, stale Stop, and preparation with an old checkout.
- 2 actual legacy process/HTTP handoff tests, including scratch failure,
a closed-stdio child that continues writing, and recovery through a
fresh worker after the process group stops.
- 5 migration tests, including SQL column metadata, preserved null
authority context, and all six prior-numbering receipts.
- 3 process-settlement and 5 scratch tests.
- 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool
integration tests.
- 7 bounded case/document-read projection tests and 7 strict
protocol-contract tests.

The current CI checkout was `fed65d49`, merging `765fccd67c` into master
`9ef434fe79`. Its new document regressions reconstruct exact
Unicode/escaped bodies and revisions through real native authority and
authenticated HTTP with generic API tools disabled. They cover
case-owned, primary-task, and linked-task documents; routing on every
page; stale document snapshots; access loss and unlinking;
unrelated/foreign/private sources; system documents; and quarantined
content. Active-task document tools retain their existing behavior.
These are current-head test results from the first CI attempt; the
browser-only retry did not rerun these suites.

The full `64fdd3403f` CI failed one separate settings test: after a
company switch, it observed an unsettled API-key draft. The production
component and test were unchanged from prior green source. Production
switches remount both keyed settings layers; the test directly reused an
unkeyed view and used a timer tick instead of waiting for React effects.
The final commit replaces that fixture synchronization with React `act`.
The exact secret-clearing assertions and all timeouts remain unchanged.
The original assertion failure remains recorded; no unchanged retry was
used. The narrow unchanged baseline and fixed fixture each pass 10
tests.

The upstream result-paging merge preserves case capture namespaces and
current asset privacy. Its API integration suite executed 39 of 41 tests
successfully, with two skipped; the optional assigned-result PRP
integration test was skipped. Neither skipped suite is counted in the
194 critical tests or as live provider proof.

All 361 upstream migration entries and SQL bodies remain unchanged. Six
case migrations follow master as 0363–0368 with identical SQL hashes.
Official Drizzle generators produced the combined metadata and five
retained snapshots. Generation reports no schema changes, and the
snapshot drift test passes. The earlier `ce361ae5d0` CI passed on
attempt 1, including its then-current 179 critical tests. Fresh review
later identified unusable document references with generic API tools
disabled; the bounded document reads and ten additional authenticated
HTTP tests address that finding.

The full local `pnpm test:run` attempt was cancelled after this Mac
exhausted PostgreSQL semaphore capacity; it is **not** counted as a
pass. Later database attempts also stopped before assertions or skipped,
including one sandbox socket denial. Two original stale-Stop regressions
did reproduce the bug against the old source. A REST serialization
assertion passed after two preserved database-startup failures. No test
timeout was increased. All 194 assertions listed above executed and
passed in current-head CI. The complete 385-file, 3,963-test UI shard
also passed after the fixture repair.

A configured-provider receipt workflow passed 13 backend checks across
triage → implementation → verification → correction → done, using one
task and four work runs. The saved receipt is revision 2 with total 6
USD; case and task both closed. The original canonical attempt remains
failed because its browser assertion confused the terminal destination
with completed-task tooltips. The corrected full browser verifier passed
on the same retained database and original runs, with no additional
provider calls. The last clean-build browser readback is on
`6a86c54a53`; later changes affect recovery, authorization, bounded
reads, and settlement. Accounted work cost is a $1.010574 rate-card
estimate; provider billing and setup-probe usage are unavailable.

Copilot profile 49 passed the unchanged seven-fixture private protocol
pack (46/46 checks, no retries) and the product file-edit canary after
master's completion-replay change. All 60 bound implementation hashes
remain unchanged on the current source. Historical profile-47/48
identities and results remain intact. See the [profile-49 compatibility
record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json).
This is bounded compatibility evidence, not a fresh full matrix,
installed-package product test, or Daytona rollout.

Preserved CI failures and retries:

- On `20662930ac`, Slack managed setup failed a five-second success
assertion after reload. Its test/application path was unchanged from a
prior green head. One unchanged-source retry passed all 25 tests in that
shard; no source or timeout changed.
- On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19
UTC. Neither log had an assertion failure before interruption. One
unchanged-source retry of each passed all Runner static checks and all
32 tests in browser shard 6; both aggregates passed. The next head,
`7b4374edaf`, passed all CI without retries.
- On `295f289b5a`, serialized shard 9 timed out during its first
device-login test, then the next test found an active login lease.
Direct login/test paths were unchanged from a prior green head. Cold
import plus a timed-out async body using the next test's mutable
actor/store is the supported fixture-race explanation; no request trace
proves that sequence. The original failure remains recorded. One
unchanged-source retry passed all 294 tests in 19 files, including all
41 device-login tests. No login source or timeout changed. The
subsequent `ce361ae5d0` CI passed without retries.
- On `64fdd3403f`, one company-switch settings assertion observed an
unsettled draft. The final test-only commit uses React `act` without
changing assertions, timeouts, or production code. On `765fccd67c`, all
ten settings tests and the complete 385-file, 3,963-test UI shard
passed.
- On `765fccd67c`, browser shard 3 completed the split-handoff
assertions, then cleanup received HTTP 500 from run cancellation after a
one-second PostgreSQL row-lock timeout. The extra chat run was reporting
completed work while teardown began. Cleanup stopped before restoring a
global experiment flag, causing a later assertion to fail. Relevant
production and test paths are unchanged from an earlier green commit.
Stop's lock order is consistent, and no nested self-lock was found; the
exact PostgreSQL blocker was not recorded. One unchanged-shard retry
passed. The original failure, trace, and downstream flag failure remain
recorded. No Stop implementation, cleanup assertion, or timeout changed.
The retry supports intermittent cleanup contention; it does not identify
the original blocking transaction.


## Risks

- Local process transfers require host-owned PID/group absence. Missing
or uncertain termination evidence keeps delivery pending, independently
of scratch setup.
- Handoff admission and task cancellation share execution ownership. New
regression tests cover temporary holds, late source writes, and
uncertain provider termination.
- Retained private sources use an intersection of existing permissions.
This intentionally fails closed when a source is missing or an existing
task cannot preserve that boundary.
- Migrations add tables and indexes. They preserve master history and
can replay over prior feature installs. Migration 0368 adds nullable
internal authorization provenance with no default or authority backfill.
Historical transfers still awaiting release without this context fail
closed; an operator must cancel them and request a new handoff.
- Delayed Cloud handoffs depend on the current Cloud membership RPC. An
unavailable, ambiguous, expired, or mismatched response holds the
handoff for retry; it does not reuse cached authority or grant access.
- Bounded reads reload and reauthorize the selected projection per page.
Underlying work grows with the data size and number of pages. Changed
data or routing requires restarting from offset zero without the old
snapshot.
- Phase names and workflow diagrams remain advisory. They do not satisfy
review or approval gates.
- This is a large, cross-cutting PR. The maintainer requested one PR for
the whole feature; the commits separate data, runtime, UI, and
qualification work.

## Model Used

OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated
code review. The exact deployment model ID and context-window size were
not exposed in the session. Live fixture agents used the explicitly
configured provider model recorded in their test evidence.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 12:46:14 -05:00
Devin FoleyandPaperclip d30648dee8 Preserve accepted messages for paused recipients (#16679)
Keep accepted deferred requests pending while their recipient is paused and deliver through normal gates after resume. Preserve company, task hold, and cleanup checks.

Verification: 247 focused checks, recursive typecheck, the full CI test and build matrix, and Apex 5/5. The local Git fixture initial-branch limitation is documented in the PR.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 18:38:17 -07:00
Devin FoleyandPaperclip 391dd15ff2 fix: recover deferred messages and completed unblock waits (#16673)
Recover accepted deferred messages through the latest finished issue run and
recognize exact completed agent unblock notifications as passive idle waits.

Preserve assignment handoffs, pause and Stop, failed outcomes, and cleanup
ownership. Honor local and SSH release contracts and atomically settle cleanup
claims. Add PostgreSQL recovery, concurrency, and interruption regressions.

Validation: recursive typecheck and build pass; 678 focused tests pass; full CI
passes; Apex review is 5/5 with no unresolved findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 16:53:29 -07:00
DottaandPaperclip 60e03c002d fix(grok): hold new runs while connection needs recovery (#16445)
## Thinking Path

> - Paperclip manages agents and their work.
> - Grok Bot assignments use a managed Runner and a verified connection.
> - Stop can leave the connection held for remote-work inspection.
> - The queue started new runs before it checked that connection.
> - Those runs allocated Runners, failed, and caused more preparation
attempts.
> - This pull request checks readiness at queue admission.
> - Pending input waits for recovery without using managed compute.

## Linked Issues or Issue Description

Refs #16427 and #16422. The open Grok changes cover the coding adapter
and do not duplicate this fix.

**What happened?**

A real Stop test held the Grok connection for inspection. A second task
was pending. Four new preparation attempts acquired managed Runners,
then failed because the connection was held. No new external assignment
or webhook was sent. Pause stopped the loop, and all leases completed
cleanup.

**Expected behavior**

Keep fresh input in the durable queue until its connection is ready. Do
not allocate a Runner to discover a known connection hold. Keep existing
session reconciliation under its recorded authority.

**Steps to reproduce**

1. Connect Grok Bot and start an assignment.
2. Queue another task for that agent.
3. Stop the first assignment and leave the inspection hold in place.
4. Run queue recovery. The pending task repeatedly starts preparation
and fails.

**Paperclip version or commit**

Observed on packaged Core 5a9e6b6315 in
Cloud staging. This branch includes current master
afe5b3418d.

## What Changed

- Reuse the broker's current connection and authority checks before
fresh Grok run claims.
- Let valid connection-pool routing select a native provider before
applying any unused Grok connection hold.
- Leave pending input queued while readiness is denied. Recheck at the
claim boundary if Stop races the queue scan.
- Preserve already-started reconciliation and all later
runtime-selection and dispatch checks.
- Propagate database failures. Treat only explicit connection-authority
errors as a hold.
- Add queue, OAuth, broker and assembled heartbeat regressions.
- Update the Grok guide and existing release evidence plan.

## Verification

- Two queue admission assertions fail before the correction.
- Six focused suites pass all 813 checks, including current master's
native ownership coverage and a real healthy Codex pool member with a
held Grok connection. The routing test reaches native runtime selection
and then injects a controlled stop before provider start.
- The assembled test uses real OAuth and broker state, repeats public
queue recovery, and checks zero leases, assignments and extra webhooks.
- Repository typecheck and build pass on final head
83664a91ce. Its full local suite is still
running. The earlier full attempt was stopped as superseded after the
valid router review.
- The full suite uses a clean tool PATH. On the earlier baseline, mocked
Gemini tests reached the installed host CLI and exceeded their
five-second deadline. Both tests pass with that host CLI absent. Test
source and deadlines are unchanged.
- The existing native cleanup and queue claim-isolation tests pass. The
process-agent Stop browser journey passes locally. Its first CI attempt
queried the queue 67 ms after comment creation, before asynchronous
publication. The test now polls for the same durable entry with its
existing deadline.
- One server CI worker shut down after 1,047 tests passed. The failed
attempt remains recorded. No routing flags or gates were changed. No UI
component or schema changes.
- The final head passes 55 CI checks, with two optional skips. Greptile
reviewed that exact head at 5/5, with the earlier thread resolved and no
new actionable finding.
- Actual packaged recovery admission still needs a live repeat. The
provider currently rejects routine starts with an external rate-limit
error. This patch does not remove that provider limit.

## Risks

Readiness can change after preflight. Runtime selection and dispatch
retain their current authority checks. Already-started runs bypass this
fresh-work check so they can reconcile their original session. Other
providers keep their current queue behavior. Expected connection denials
hold input; unexpected database errors still propagate. No migration is
needed.

## Model Used

OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools
and computer use. The exact serving model ID, snapshot and context
window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 11:32:46 -05:00
Devin FoleyandPaperclip 307c27e3d8 fix(native): reclaim task execution before same-run recovery
Restore the original task execution binding atomically during native same-run recovery. Preserve current owners, stops, pauses, assignments, and dependency waits.

Isolate candidate failures, bound deferral writes, prioritize undated retries, and allow recovery ahead of queued waiters without premature queue promotion. Add PostgreSQL attachment, ownership, contention, fairness, and queue regressions.

Verification: 99 focused tests, workspace typecheck, build, full current-head CI, and Apex 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 01:53:26 -07:00
DottaandPaperclip 3fc64c5156 feat(slack): add Require at-mention setting (#15612)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Slack messages can create tasks and continue existing tasks.
> - A reply in a subscribed thread can start agent work without a new
mention.
> - Some users want to control each reply with an explicit bot mention.
> - This pull request adds a Require at-mention setting to Slack app
settings.
> - The setting checks new messages and queued messages before they
start work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Slack app settings and the rules that admit inbound messages.

**Subsystem affected**

Server, UI, database, and shared API contracts.

**Current behavior**

After the bot joins a thread, an ordinary reply can wake the agent.
Direct messages do not need a bot mention.

**Proposed behavior**

Add Require at-mention below Allow direct messages. When enabled, each
channel message, thread reply, and direct message must mention the bot.
Direct messages must also be allowed.

**Reason and benefit**

Users can prevent ordinary conversation from starting agent work.

**Breaking changes**

New Slack connections require a bot mention by default. Existing
connections keep their saved setting. Work that has already started is
not cancelled.

**Additional context**

A search for existing at-mention PR titles found no duplicate.

## What Changed

- Add the endpoint field, schema validation, API response, and UI
toggle. New Slack connections enable the setting.
- Add an idempotent database migration and generated metadata. Renumber
it to 0321 after current master.
- Check the current setting at inbound admission, task mutation, and the
first heartbeat queue claim.
- Keep Slack tool access and recovery for already-running work when the
setting changes.
- Cancel queued batches that contain an unmentioned input, including
coalesced messages.
- Dispatch promoted follow-ups after the scheduler releases its agent
start lock. Preserve these checks in the new queue and run-control
modules.
- Filter unmentioned messages without retaining their content.
- Keep provider-message deduplication and existing access checks.
- Add thread, DM, duplicate-delivery, queued-message, and new-connection
default regression tests.
- Document the setting and its scope.

## Verification

- Previous head `16791bb7`: typecheck, build, all server shards, Runner
checks, and E2E passed. Two chat shards failed in three fixtures that
used unmentioned Slack messages.
- Update setup follow-ups to mention the Slack bot. Disable the mention
rule explicitly in the callback ordering test, which tests ordinary
subscribed replies.
- Previous local checks passed: six Slack integration cases, all 46
durable chat scheduler tests, all 24 queue/run-control tests, migration
safety, and UI token gates.
- Greptile reviewed `16791bb7` with 5/5. All review threads are
resolved.
- Head `53163c20f`: all five focused integration cases passed. These
cover all three prior CI failures and both default-on channel/DM cases.
Fresh CI and Greptile are running.
- Full local recursive typecheck, build, and repository tests were not
repeated for this fixture-only change. CI supplies these checks.
- Manual check: create a Slack connection. Require at-mention must be
on. An ordinary thread reply or DM must not wake the agent. A bot
mention must wake it once. Disable the setting and confirm ordinary
replies work again.

## Risks

- The migration adds a non-null boolean with a false default. Apply it
before the new server runs.
- The setting also applies to DMs. The UI help text states this.
- Enabling it blocks queued unmentioned work but does not cancel a run
that has already started.
- Full local verification is limited by the environment constraints
above; latest-head CI must pass before merge.

## Model Used

OpenAI GPT-6 (Codex agent), with reasoning, tool use, and code
execution. The runtime does not expose a more specific model version or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 14:03:02 -05:00
DottaandPaperclip f6d2a6b79e refactor(heartbeat): extract queue admission and wakeup dispatch (#15626)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The heartbeat service admits and dispatches agent runs.
> - Earlier extractions separated workspaces, preparation, state,
retries, and restart recovery.
> - Queue admission and wakeup dispatch still occupy more than 4,200
lines in the main service.
> - Their transaction, budget, ownership, and concurrency checks must
stay together.
> - This pull request moves those operations into one queue module and
adds boundary tests.
> - Reviewers can inspect queue policy separately from adapter
execution.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The structure and testability of heartbeat queue admission and wakeup
dispatch.

**Subsystem affected**

server/ — orchestration services.

**Current behavior**

Wake coalescing, queued claims, dispatch, daily limits, and native wake
intents live inside `heartbeat.ts`.

**Proposed behavior**

Keep their behavior in `server/src/services/heartbeat/queue.ts`. Bind
lifecycle callbacks and shared ownership with `createHeartbeatQueue`.

**Reason and benefit**

This removes 4,243 lines from `heartbeat.ts`, leaving 14,201 lines. The
queue module has 4,576 lines and keeps related policy together in one
file.

**Breaking changes**

None. The service methods, public exports, database transactions, locks,
and execution gates keep their existing behavior.

**Additional context**

Continues #15622 and the earlier merged heartbeat extractions. A search
of open and closed PRs found no duplicate queue extraction. Related
#15600 adds a new experimental scheduler; this change preserves the
current scheduler policy. This does not add a roadmap feature.

## What Changed

- Move wake coalescing and batching, queue claims, daily caps,
concurrency and priority checks, timer admission, queue resumption, and
native status wake intents into `heartbeat/queue.ts`.
- Pass lifecycle callbacks and the existing process-wide execution and
wakeup sets into the factory. Preserve construction order with three
forwarding callbacks.
- Keep adapter execution, cancellation, and status writes in the
service.
- Add nine tests for inert construction, scheduling suppression, shared
wake tracking, policy wiring, claim release, newer ownership, and budget
gates.
- Document the queue module boundary in `doc/DEVELOPING.md`.

## Verification

- Before extraction: 133 tests passed across nine existing suites.
- After extraction: **150 tests passed across 12 files**, including all
nine new module tests and real PostgreSQL coverage.
  ```sh
pnpm exec vitest run server/src/services/heartbeat/queue.test.ts
server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts
server/src/__tests__/heartbeat-start-lock.test.ts
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts
server/src/__tests__/heartbeat-scheduling-suppression.test.ts
server/src/__tests__/heartbeat-comment-wake-batching.test.ts
server/src/__tests__/heartbeat-running-followup.test.ts
server/src/__tests__/heartbeat-dependency-scheduling.test.ts
server/src/__tests__/heartbeat-auto-checkout.test.ts
server/src/__tests__/heartbeat-archived-company-guard.test.ts
server/src/__tests__/heartbeat-task-drain-admission-release.test.ts
server/src/__tests__/heartbeat-worktree-suppression.test.ts
  ```
- Additional control and recovery coverage: **390 tests passed across
five files**. Combined focused coverage is **540 passing tests across 17
files**, with no skips.
  ```sh
pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/heartbeat-native-cleanup-admission.test.ts
server/src/__tests__/heartbeat-native-status-context.test.ts
server/src/__tests__/heartbeat-task-drain.test.ts
server/src/services/chat-control-admission-retry.test.ts
  ```
- Full `pnpm -r typecheck` and `pnpm build` passed.
- Full local `pnpm test:run` was started and then stopped after the full
CI suite passed. It did not finish and is not counted as a full local
pass.
- All **54 current-head checks passed**; two optional Storybook checks
were skipped. This includes the full general and serialized test matrix,
typechecks, build checks, Runner verification, and canary dry run. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/37847175452).
- Greptile completed on head `cc0001961a052a716d81c136929f89312e0b5dee`
with **5/5 and no actionable findings**. There are no review threads or
merge conflicts.
- Structural comparison confirms 31 moved function bodies and eight
declarations match the originals. Remaining service function bodies and
all 140 public exports are preserved.
- An initial local run skipped database tests because the host had
exhausted its PostgreSQL shared-memory slots. After verified stale
segments were cleared, all 150 focused tests ran and passed with no
skips.

## Risks

Moving closures can break shared ownership or callback construction
order. The factory receives the original sets and explicit callbacks. It
starts no work during construction. Transaction boundaries,
compare-and-set predicates, budget gates, cleanup holds, and lock
ordering stay intact. This PR has no schema or API changes.

## Model Used

OpenAI GPT-6 via Codex. The exact serving model ID and context window
were not exposed in this session. Used reasoning, repository inspection,
shell tools, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 07:59:19 -05:00