Commit Graph
415 Commits
Author SHA1 Message Date
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
DottaandClaude Opus 5.5 dfc510b887 fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud replaces the server process on each release. The old
server keeps serving until the new one passes health checks and takes
the traffic.
> - During that overlap the old server keeps starting new runs right up
to SIGTERM, although the new server can already start them.
> - SIGTERM then cuts off exactly those runs while they still prepare.
In one audited swap the old server started 30 runs after the new server
booted, and 23 of them ended `process_lost`.
> - Admission must never pause, so the fix must hand over who starts new
runs, not stop admission.
> - This pull request makes the new server record when it starts runs.
The old server then leaves new run starts to it, and starts them again
if the new server stops or dies.
> - The benefit is that a deploy no longer starts runs on a server that
is about to stop, and the company always has one server that starts
runs.

## Linked Issues or Issue Description

Refs #20852 (preserves preparing and launching turns that a deploy
interrupts; this PR reduces how many such turns the old server starts).
Related: #21046, #21107 (shared controller-instance owner proof), #20986
(queued-run dispatch signal).

**What existing behavior does this improve?**

The start of new runs while two servers overlap during a Cloud deploy.

**Subsystem affected**

Server run admission (`server/src/services/heartbeat/queue.ts`), the
controller instance table, and startup and shutdown in
`server/src/index.ts`.

**Current behavior**

The old server starts queued runs until it receives SIGTERM. Runs that
it starts in the last minutes of the overlap are interrupted while they
prepare. Nothing tells the old server that its replacement already
starts runs: `controller_instances.epoch` is written at process start,
and that happens minutes before the new server can start runs.

**Proposed behavior**

The new server records `dispatching_at` and a short
`dispatch_lease_expires_at` on its `controller_instances` row right
after its startup queued-run pass. The old server reads this through an
in-memory flag at run admission and leaves new run starts to the new
server. Runs it already started continue and are handed over at SIGTERM
as today. If the new server stops, drains or dies, the old server starts
runs again within seconds.

**Reason and benefit**

Fewer runs are cut off by each swap. There is never a moment without a
server that starts runs.

**Breaking changes**

None. The migration adds two nullable columns. Releases without this
code ignore the columns, and their rows never carry the marker, so a
mixed-version overlap behaves as today.

## What Changed

- `controller_instances`: new nullable columns `dispatching_at` and
`dispatch_lease_expires_at` (migration
`0471_controller_instances_dispatch_marker`, two `ADD COLUMN`
statements, no lock beyond the catalog update).
- New `server/src/services/run-start-handover.ts`. It is one worker on
the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`).
Its renewal and recheck are coordinator deadlines, and a drain change or
expiry wakes it.
- Replacement side: `begin()` records the intent and writes the marker
(it creates the boot's row if the row is missing). If the server drains
at that moment, the worker writes the marker when the drain ends. The
worker then publishes a `run-start-handover` hint and renews the 20 s
lease every 5 s. It renews only while an older boot that is not stopped
or fenced may still serve, and for at most 30 min. All marker writes run
in one serialized chain. `release()` at shutdown lands after any renewal
already in flight. A clear that fails is retried, and shutdown waits for
the release. While the server cannot dispatch (drain, quiesce, admission
closed), the worker clears the lease.
- Old-server side: `isYielding()` is an in-memory read. The hint
refreshes it, and so does one read that a dispatch pass waits for (at
most 1 s) when the last read is more than 2 s old. A read that gets no
answer within 1 s never keeps the server yielding. The server admits,
and the yield also ends when the replacement's last seen lease runs out
on the old server's own clock. Reads are a single flight on the
dedicated connection. While it yields, the server rechecks every 3 s,
calls `onResume` when the marker is gone, and sends a coalesced
queued-run-dispatch hint for skipped starts.
- It yields only to a newer epoch with a live marker. It never yields to
an older boot, to itself, or to a row without the marker, and it does
not yield without a row of its own.
- **Database invariant:** every statement of the hand-over runs in a
small transaction on the dedicated one-connection controller-lease pool
(`controllerLeaseDb`) with a 1 s statement timeout. That covers reads,
marker writes, the `run-start-handover` notification (commit-bound, in
the marker write's own transaction) and the skipped-start hint. None of
it touches the shared pool or registers tracked database work. The pool
is shared only with legacy controller-lease renewals, and never with the
liveness pool that native lease renewals use. A hand-over statement
waits at most one legacy batch (10 s), inside the 15 s between a marker
renewal and the marker's expiry. It delays a legacy renewal (60 s lease,
renewed every 10 s) by at most 1 s. I did not piggyback on the
controller-instance lease renewal, because that renewal runs only while
the server holds work and only every 30 s.
- **Import audit of `run-start-handover.ts`:**
  - `drizzle-orm` `sql`: builds statements, no I/O.
- `controllerLeaseDb`: the dedicated connection, used by every statement
here.
- `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs
`pg_notify` only on the transaction it is given and registers no intent,
so the coordinator's `reconcileDatabaseWork` has nothing of ours to
check.
  - `logger`: no I/O to the database.
  - `controller-process-identity`: in-memory boot id and host hash.
- `delivery-work-coordinator`: type only. The worker is registered for
lifecycle and deadlines; its `run` awaits only the serialized marker
chain, and reads are not awaited.
  - `DELIVERY_QUEUES`: constants.
  - `task-admission` drain subscriptions: in-memory events.
- `heartbeat.ts`: run admission is also closed while the server yields.
A `runStartHandover` service option (default: the process's registered
hand-over) lets one process host two assembled services in tests.
`queue.ts`: `startNextQueuedRunForAgent` and
`resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That
call returns `null` (nothing to await) unless the hand-over read is
stale.
- `index.ts`: creates the hand-over only when the instance lease is on,
the deployment notification channel exists, and the scheduler is
enabled. It calls `begin()` after the RunDispatch startup pass and wakes
that queue. It releases the hand-over when run admission closes at
shutdown, and stops it before the lease stops.
- `deployment-notifications.ts`: keeps the LISTEN connection open while
a hand-over involves this server, so skipped starts and lease changes
reach the other server at once.
- `doc/execution-semantics.md`: new section "Run starts during a deploy
overlap". It covers the marker, who yields, failure handling, what
counts as starting a new run, and what a post-deploy audit should show.

**Marker contract for other components (for example the deployer):** a
replacement starts runs when its `controller_instances` row has an
`epoch` greater than the old boot's, and `dispatching_at is not null and
dispatch_lease_expires_at > now() and stopped_at is null and fenced_at
is null`. `dispatching_at` stays set after the lease lapses, as the time
the boot began starting runs.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test
with two `heartbeatService`s, each with its own pool, listener,
coordinator and boot. A wake that the old server accepts stays queued
there. The replacement's real queued-run dispatch starts it, and claims
are observed per pool. After a replacement release, the old server
starts the run itself. Red: with the yield removed from
`isRunAdmissionClosed`, the old server claims the run.
- `cd server && npx vitest run
src/__tests__/run-start-handover.test.ts`: 19 tests pass against
embedded PostgreSQL. Each test uses two servers, each with its own pool,
boot id, LISTEN connection and delivery coordinator. The tests cover:
- the old server is ready and the new server starts; the new server sets
the marker; the old server yields; the queued run is claimed only by the
new server, and a skipped start wakes the new server's dispatch
- the old server learns from its own dispatch path without a
notification: a read older than 2 s is refreshed before the start
- the new server dies after it sets the marker: the old server resumes
and runs its queued-run pass
  - the new server shuts down: the old server resumes at once
- the new server drains: it releases the marker and takes it back
afterwards
- a replacement that starts up during a drain takes over when the drain
ends
  - a shutdown release during an in-flight renewal stays released
- a clear that fails at shutdown (statement timeout behind a row lock)
is retried by a later release
- an older server that outlives the lease TTL keeps yielding while the
replacement renews
- a yield that the old server cannot confirm within 1 s (its only
connection is busy) does not stop starts (red/green)
- a yield ends when the replacement's lease runs out, even while every
read waits for a connection (red/green)
- renewals blocked past the TTL hand starts back once, then the
replacement takes them again (bounded flapping)
- a clear that fails when the replacement starts draining is retried
well inside the lease (red/green)
- with both servers' shared pools blocked past the TTL, the marker is
written, renewed and released, and the old server yields and resumes,
all before the block ends (red when hand-over statements use the shared
pool)
  - an older-release row without the marker changes nothing
- a server never yields to an older boot or to itself, and does not
yield without a row of its own
  - a lone server writes once, never renews, and admits from memory
  - without a hand-over (self-hosted) there is nothing to await
- Red: on `origin/master` the test file cannot run, because the module
and the columns do not exist. The old server has no signal and keeps
starting the handed-over run.
- `cd server && npx vitest run
src/__tests__/deployment-notifications.test.ts
src/services/heartbeat/dispatch-work.test.ts
src/__tests__/delivery-work-coordinator.test.ts
src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`:
all pass.
- `cd packages/db && pnpm check:migrations`: safety and compatibility
checks pass. `cd server && npx tsc --noEmit -p .`: clean.
- `pnpm verify` asks for broad verification because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Run starts move from the old server to the new server during the
overlap. If the new server cannot dispatch, the old server resumes on
the lease clear (about 3 s) or the lease lapse (up to about 23 s after a
crash). If a hand-over read fails, the server keeps admitting.
- If the old server was not listening for notifications, it can start
runs for up to about 2 s after the marker (the age of its last read). On
a dispatching server, the pre-dispatch read adds one small bounded query
at most every 2 s.
- An older boot that has been idle for more than 30 min counts as gone.
The new server then does not renew, and the old server admits again as
today.
- The migration adds nullable columns only. The deployer can apply it
while the previous release serves.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 11:09:53 -06:00
DottaandClaude Fable 5.2 2174f2dfa5 perf(server): stop re-reading and re-decrypting unchanged checkpoint headers on every upgrade readiness probe (#21130)
Cloud repeats the upgrade readiness probe. Every probe loaded, decrypted and parsed the checkpoint header of each active native run and of each retained task session's last run, even when nothing had changed.

The statement that reads a run now also reads its header's row revision (xmin, ctid, updated_at, stored size) and returns the material only for a revision the process has not inspected. A bounded memo in the checkpoint owner keeps the four identifiers the answer compares. It reuses an answer only for the same binding, row revision and secrets key, loads the key again for each answer it serves, never remembers a header that failed to decrypt, and forgets a header the latest probe did not inspect.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 10:45:44 -06:00
DottaandClaude Opus 5.5 0a2f618cae fix(native): leave a run another live server is preparing alone, and re-adopt a run a shutdown handed off (#21107)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for a few minutes, then the old server gets SIGTERM and hands
its native runs to the next controller.
> - #21046 taught the legacy reaper to leave a run with its owner while
the owner's server process is proven alive. Native runs have a similar
gap.
> - Runtime selection writes a native run's coordinator with no
controller and no lease, and stops the run lease. The executor claims
the coordinator about 20 seconds later. In that window the other
server's sweep sees no owner and marks the run "ownership could not be
verified".
> - When the old server then hands the run off before any Runner
checkpoint exists, no sweep looks at it again. Only the next server boot
does, 15 to 40 minutes later. The task and its queued runs wait.
> - This pull request leaves such a run with its live preparing server,
and lets the serving server re-adopt a handed-off run through the
ordinary restart claim.
> - The benefit is that a run started during an upgrade continues within
one sweep, with no false "blocked" mark and no wait for another upgrade.

## Linked Issues or Issue Description

Refs #21046

**What happened?**
On a hosted staging server, an upgrade started a new server at
12:19:57Z. The old server started a native run and selected its runtime
at 12:20:32Z. At 12:20:35Z the run logged "Native execution ownership
could not be verified; automatic recovery is blocked"
(`observed_owner_unverified`). The old server's executor claimed the run
at 12:20:51Z. The old server got SIGTERM and suspended the run at
12:20:54Z (`awaiting_runner_reattach`). After that the run showed
"Confirming execution" for 29 minutes, and two queued runs waited behind
it. At 12:49:45Z the next upgrade's startup recovery restarted its
bootstrap, and the run succeeded at 12:58Z.

**Expected behavior**
- A server does not mark a run unverified, or recover it, while another
live server is preparing it.
- A run that a graceful shutdown hands to the next controller is
re-adopted by the serving server at once, not at the next boot.

**Steps to reproduce**
1. Seed a native run whose coordinator is `observed` with no controller
and no lease, and whose `controllerProcess` names another server with a
live controller instance lease.
2. Run the recovery sweep on this server. See that the run gets
`native_execution_ownership_unverified`.
3. Claim the coordinator for the other server, then write what its
graceful shutdown writes (`restart_suspended`,
`awaiting_runner_reattach`, lapsed lease) and stop its instance lease.
Give the run a predecessor's Runner PID.
4. Run the sweep again. See that nothing claims the run.

**Root cause**
- `recoverActiveExecutions`
(`server/src/services/heartbeat/recovery.ts`) treats an `observed`
coordinator without a live controller lease as an owner it cannot
verify. Between runtime selection and the claim, the only owner on
record is the run's `controllerProcess`, and the sweep did not check it.
- `listExpiredNativeRunnerAuthorities` (`native-runner-checkpoint.ts`)
selects runs for the restart claim only when they have a controller
checkpoint, or no process identity at all. A run handed off just after
its claim has no checkpoint, and a follow-up carries its predecessor's
Runner PID. The sweep then skips `awaiting_runner_reattach` runs, so
only a server boot looked at them again.

**Paperclip version or commit**
Staging Core `814fe5d20f` (old server) and `3eca7a6112` (new server).
The behavior is the same on master `849f109f62`.

## What Changed

- `recovery.ts`: before it lists expired native runs, the sweep checks
runs in the runtime-selection window (coordinator `observed`, no
controller, no lease, not tracked here). When the run's
`controllerProcess` names another server process that the shared owner
prover proves alive, the sweep leaves the run with that server: it does
not recover it and does not mark it unverified. It re-checks within 30
seconds. After 5 minutes without a claim
(`LEGACY_LIVE_OWNER_SETTLE_GRACE_MS`), or once that server is gone, the
existing rules apply.
- `native-owner-stamps.ts`: add `preparingControllerOwnerStamp` for that
proof.
- `native-runner-checkpoint.ts`: `listExpiredNativeRunnerAuthorities`
also selects a run that a graceful shutdown handed off and whose lease
lapsed: recovery state `awaiting_runner_reattach` or
`awaiting_evidence`, with a `restart_suspended` history entry that no
restart claim followed. It stays selected through ownership waits (the
departing server can still be alive at the first look). `recovery.ts`
schedules a 30-second re-check for an ownership wait that has no retry
of its own. The ordinary restart claim then proves the takeover and
picks the disposition, exactly as the startup pass of a new server does.
A row that an older release already marked unverified is re-adopted too:
the claim clears the mark.
- `doc/execution-semantics.md`: document both rules.
- No schema change. Nothing changes on the path that starts a run or a
turn: the new proof runs only in the recovery sweep, and only for runs
in the window.

## Verification

- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
block "a run the old server started during an upgrade overlap". Red on
master, green on this branch:
- "leaves the run with its live preparing server, then re-adopts it
after that server hands it off": two concurrent sweeps leave the run
`running` with no error and do not touch the coordinator. After the old
server claims, hands the run off and stops, one sweep of the new server
re-adopts it and reaches provider dispatch. Master: the run is marked
`native_execution_ownership_unverified`, and after the handoff nothing
claims it.
- "keeps a handed-off run eligible while the departing server is still
alive, and re-adopts it once it exits": the first sweep waits
(`awaiting_evidence`), the sweep after the departing process exits
re-adopts the run. Red with a filter that matches only the last history
entry.
- "re-adopts a handed-off run that an older release already marked
ownership-unverified": the stuck staging shape. Master: nothing claims
it.
- Related suites pass: `heartbeat-native-restart-suspension` (all 109),
`services/heartbeat/recovery.test.ts` (all 103).
- `tsc --noEmit` for `server`.

## Risks

- A run that another live server prepares, but never claims, now stays
`running` for up to 5 minutes before the existing rules apply. That is
the same bound #21046 uses.
- The guard trusts the shared owner prover. It reads "alive" only from a
live controller instance lease or a live PID in the same PID space.
- More handed-off runs reach the restart claim earlier. The claim keeps
all its fencing and evidence checks; this change only makes the serving
server ask the same question a new boot asks.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only API inspection) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 08:06:57 -06:00
DottaandClaude Opus 5.5 67403baaa4 fix(native): an agent termination Stop with no turn result stays a cancellation (#21093)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person terminates an agent, Paperclip stops the agent's
running native turn
> - Two paths can record that Stop: the terminate route ("Cancelled
because the agent was terminated ...") and the lifecycle worker ("Agent
lifecycle stop requested"). Both are system-initiated and expected
> - When the stopped turn ends without a semantic result, the native
executor keeps the run cancelled only for a person's Stop, a
reassignment, or a governance pause. A termination Stop fell through, so
the run ended `failed` with `native_session_interrupted`
> - Whether the run ended cancelled or failed depended on which path won
the race
> - This pull request treats an acknowledged termination Stop like a
person's Stop in the executor and in the lease-release terminalization
> - The benefit is that a terminated agent's interrupted run ends
cancelled, whichever path records the Stop first

## Linked Issues or Issue Description

Refs #20186 (the same treatment for a governance pause Stop)

**What happened?**
A person chose Terminate for an agent while its native turn ran. The
lifecycle Stop recorded an acknowledged native cancellation (initiator
system). The turn ended without a semantic result. The executor then
wrote the run as `failed` with `native_finalization_missing: session
returned no semantic result`.

**Expected behavior**
The run ends `cancelled`. A termination is an intentional Stop, not a
failure to retry.

**Steps to reproduce**
Terminate an agent while its native turn runs, so that the Stop
interrupts the turn before it records a result. Before this change, the
run became `failed` when the lifecycle worker recorded the Stop first.

## What Changed

- `governance-pause-stop.ts`: adds `AGENT_TERMINATED_STOP_REASON`,
`COMPANY_ARCHIVED_STOP_REASON` and `isAgentLifecycleStopRecord`. The
predicate accepts a system-initiated, expected control-plane
cancellation whose reason is a lifecycle, termination or company-archive
Stop.
- `acknowledged-native-stop.ts`: adds
`hasAcknowledgedNativeLifecycleStopIntent`, which requires that record
and an acknowledged, dispatched native Stop receipt.
- `native-session-executor.ts`: the no-semantic-result branch accepts
that intent, so it settles the coordinator as `native_retry_cancelled`
and raises `NativeCancellationPendingRecoveryError`, as for a person's
Stop.
- `heartbeat/run-lifecycle.ts`: lease-release terminalization keeps such
a run `cancelled`.
- The terminate route and the lifecycle driver use the shared reason
constants. The reason text does not change.

## Verification

- `native-session-executor.test.ts`: a new case for each reason
(lifecycle Stop, terminate route, company archive). Each was red before
the change and is green now. The earlier control case now uses a system
Stop that is neither a pause nor a termination, and it still takes the
retry path. Full file: 1414 passed.
- `heartbeat-run-lease-release-terminalization.test.ts`: a new case for
a lifecycle Stop when teardown wins the finalizer race. It was red
before the change (`interrupted`) and is green now.
- Product evidence: a paid chat-eval run of `agents.retire-agent` on a
branch that combines this change with three related fixes. The result is
in a PR comment.

## Risks

- A termination or company-archive Stop that interrupts a turn no longer
schedules an exact resume of that turn. The agent is leaving service, so
no resume can run.
- Other system Stops keep their current behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:21 -06:00
DottaandClaude Opus 5.5 c4927e1ec2 fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade stops the old server gracefully. A legacy run that is
still preparing ends `interrupted` with `server_shutdown_interrupted`.
The new server must continue it automatically (#21019, #21046).
> - A run that a person started with Retry (or a message) carries
`explicitUserContinuation`. Its automatic retries need their own receipt
from `admitExplicitContinuationRetry` (#14987, #19384).
> - That admission refused every shutdown-interrupted Retry run. The
task then stayed in progress with no run, and the stranded-task sweep
refused it again about every 2.5 minutes, forever. A person had to press
Retry.
> - An upgrade is routine. It must never leave a task that needs a
manual Retry.
> - This pull request lets the person's original authorization cover a
continuation after a shutdown that stopped the run before provider work
started. All other checks of the gate stay.
> - The benefit is that Retry-started tasks continue through an upgrade,
and tasks that earlier releases left waiting continue on their own after
deploy.

## Linked Issues or Issue Description

Refs #21019, #21046, #14987, #19384

**What happened?**
On a hosted staging server, each automatic upgrade stopped legacy runs
that were still preparing (`executionRecovery {kind: bootstrap,
providerWorkStarted: false}`). When the run came from a person's Retry
(wake reason `task_retry`), the continuation was refused with "The
automatic retry could not revalidate the original user continuation."
(`continuation_user_authorization_missing`). The task stayed
`in_progress` with no run and no next retry. Runs from older upgrades
were refused again by each sweep. A run from the same upgrade that did
not come from a Retry got its continuation in the
`interrupted_preparation` lane.

**Expected behavior**
A Retry-started run that a graceful shutdown stopped before provider
work continues automatically, once, under the person's original Retry.
Tasks that an earlier release already refused continue after deploy with
no manual step.

**Root cause**
Three conditions refused it (`server/src/services/heartbeat/retries.ts`,
`server/src/services/explicit-native-continuation.ts`):
1. The scheduler admits an explicit continuation only for the
`transient_failure`, `server_restart`, preparation-wait and
agent-directory lanes. A shutdown-interrupted preparation with unspent
failure budget uses `interrupted_preparation`, so it was refused before
admission ran.
2. `admitExplicitContinuationRetry` accepted only a `failed` or
`timed_out` parent. These parents are `interrupted`.
3. It required `task.executionRunId` to be the parent. The refusal
released the claim (`releaseIssueExecutionAndPromote`), so every later
sweep was refused by this check too.

**Paperclip version or commit**
master `f30005440b`.

## What Changed

- `retries.ts`: an explicit continuation may also use the
`interrupted_preparation` lane. Lane choice is unchanged (see below).
- `explicit-native-continuation.ts`: `admitExplicitContinuationRetry`
also accepts a parent that is `interrupted` by a graceful shutdown
(`isServerReplacementInterruption`, so `server_shutdown_interrupted`)
with bootstrap evidence `providerWorkStarted: false`
(`isShutdownInterruptedPreparation`).
- For such a parent only, a released claim (`executionRunId` null) is
accepted. Admission restores the claim to the parent under the task
lock, as the shutdown path leaves it, so the scheduler passes the claim
to the successor and dispatch accepts it. If the parent validation then
refuses, admission releases the claim again in the same transaction. A
claim that another run holds still refuses.
- `doc/execution-semantics.md`: describe the rule and the conditions
that still refuse. Also apply the sentence that #19384 left pending (a
plain task Retry is renewable).
- No schema change, no backfill. Nothing changes on the path that starts
a turn.

**Which lane a graceful-shutdown legacy preparation uses (after
#21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation`
for an `interrupted` run with the bootstrap evidence.
`usesServerRestartLane(run, true)` returns false while the failure
budget is unspent, so the run uses `interrupted_preparation`: it does
not spend the failure budget, the first continuation is due at once, and
it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a
spent failure budget it uses `server_restart`. This is the intended
design (#21046 comment on `usesServerRestartLane`), so no lane change.
The `server_restart` lane is for `process_lost` replaced preparations
and for runs past dispatch.

**Conditions that still refuse** (each has a test or an existing test):
- The task has another assignee, or is `done` or `cancelled`.
- Another run holds the task's claim.
- Newer work on the task (a run created after the parent, a retry child,
or a live run).
- The Retry request changed: other actor, not admitted, other
generation, bound to another run, a chat Retry, a fresh-workspace Retry.
A message continuation with an edited or deleted message.
- A missing or revoked receipt (dispatch validation of the parent
fails).
- An execution blocker on the task, or a live adapter controller for the
parent.
- An `interrupted` parent with another cause (for example an operator
interrupt), or with no bootstrap evidence, or with `providerWorkStarted:
true`. A parent that failed after provider work keeps the existing rules
(`failed`/`timed_out` only).
- Native runtime parents (unchanged, see below).

**Native runtime.** A native `paperclip_runner` run is not interrupted
by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and
the next server reattaches it (`restart_suspended`). It does not reach
this gate. Native replacement paths (`native-safe-replacement`,
`native-provider-capacity-retry`) remove `explicitUserContinuation` from
the successor and use their own contract. So the native path does not
have this refusal, and this pull request does not change it.

## Verification

- `server/src/services/task-retry.test.ts`, new tests. Red on
`origin/master` (no successor), green on this branch:
- "continues a Retry-admitted run that a graceful shutdown stopped while
it was preparing": production shape. A legacy run admitted by a task
Retry, `preparing`, owned by this server, with bootstrap evidence.
`drainRunningRunsForShutdown("SIGTERM")` interrupts it with
`server_shutdown_interrupted` and schedules one
`interrupted_preparation` successor (`failureRetries: 0`,
`interruptedPreparations: 1`) with its own receipt. The claim passes to
the successor and dispatch accepts it. A concurrent stranded sweep and a
second schedule call reuse the same successor.
- "continues a task an earlier release left waiting after refusing its
shutdown continuation" (a direct Retry run, and an automatic retry of a
Retry run): the row and task as an older release left them (claim
released). Two concurrent stranded sweeps schedule exactly one
successor; the claim passes to it; dispatch accepts it.
- "does not continue a shutdown-interrupted Retry run without its
original authority" for: provider work started, no bootstrap proof,
another interruption cause, claim held by another run, reassigned,
superseded, other Retry actor, cancelled task. No successor, no receipt,
and the claim is left as it was.
- I confirmed that each source change is needed: without the lane change
both main tests fail; without the released-claim rule the older-release
test fails.
- Suites: `task-retry`, `explicit-native-continuation`,
`legacy-explicit-continuation-recovery`, `quarantined-task-retry`
(587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling`
(520/520).
- `tsc --noEmit` for `server`.
- `pnpm verify` selects a broad run (exit 2) because `retries.ts` is
central; required CI covers that.

**After deploy, the next audit should show:** a Retry-started run that a
shutdown stops while preparing gets a `scheduled_retry` successor with
`scheduledRetryReason: interrupted_preparation` (or `server_restart`
with a spent failure budget) and no
`continuation_user_authorization_missing` event. The tasks that earlier
upgrades left waiting get one such successor on the first stranded sweep
of the new server, with the task claim on the successor.

## Risks

- Behavior change: a shutdown-interrupted preparation of a Retry or
message turn now continues automatically, within the existing uncharged
and bounded lanes. The run did no provider work, so there is nothing to
reconcile.
- Rows saved by older releases: a refused row whose claim was released
is continued once. If another run took the claim, or newer work exists,
it stays refused.
- Two servers at once: admission runs in the scheduler transaction under
the task and run row locks, and the scheduler reuses an existing
successor (`retryOfRunId`). The tests run two sweeps concurrently and
get one successor. An old server that still refuses cannot clear the
claim after the new server moved it to the successor (its release
matches only the stopped run).
- The claim restore writes `executionRunId` for a terminal run inside
the scheduling transaction. If admission refuses after the restore, it
releases the claim again before the transaction commits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, Vitest on embedded PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 05:02:56 -06:00
DottaandClaude Opus 5.5 f30005440b fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.

## Linked Issues or Issue Description

**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.

**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.

**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.

**Paperclip version or commit**
master at 4228bcf89c

**Additional context**
Root cause: `withRuntimeSkillEntries` holds the company-wide skill cache
lock (a Postgres advisory lock) while it materializes each saved skill
version. A new server has an empty skill cache, so each reconnect
downloads its skills again, one run at a time. The Runner already has
these files, and the reconnect path does not read them.

## What Changed

- `company-skills.ts`: new `admittedVersionsOnly` option. It still
checks that each saved version exists and that the agent can read the
skill. It does not materialize the files or take the company skill cache
lock.
- `heartbeat/live-runner-adoption.ts`: `adoptsLiveRemoteRunner` is true
only for a `reattach_remote_runner` claim for this run, with the claim's
own lease owner, on a native run.
- `heartbeat/execution-configuration.ts`: uses the option only when all
of these are true: the reconnect gate is true, the run has its saved
native input with a runtime context, and the run has its saved skill
versions. In that case it times the phase as
`configuration_skills_adopted` and adds one lifecycle run event
(`skillPreparation: "reused_admitted"`).
- `heartbeat.ts`: passes the gate to execution configuration.
- `native-runtime/runtime-context.ts` and `native-session-executor.ts`:
on the same reconnect to a remote Runner filesystem, the Runner
controller describes the Runner's skills by their admitted digests. It
rehydrates the pinned skill files (which takes the same lock) only if it
must launch a Runner process and stage them. All other starts rehydrate
as before.
- `doc/execution-semantics.md`: describes the new behavior.

Not in this PR (by design): a separate "dispatch-only" reconnect path,
and a new startup scheduling lane. The native executor needs about 40
values from preparation (runner environment, MCP token, managed AI
credentials, the reattached execution target, run output). Connector and
instruction preparation also feed the session fingerprint. That cut is
not mechanical, so this PR removes the largest measured single cost
only.

Other `heartbeat.prepare.*` costs that stay on the reconnect path (use
post-upgrade timings to choose the next cut):
- `configuration_secrets`, `configuration_connector_assignments`,
`configuration_connection_instructions`,
`configuration_connector_skills`/`_delivery`,
`configuration_agent_identity`, `configuration_ai_runtime`
- `environment_selection`, `workspace` (retained lease resume and
realization), `session`, `task_context`, `run_identity`
- `github_environment`, `github_launchers`, `run_output`,
`runtime_services`, `instructions`, `runtime_selection` (including
`runtime_selection.skill_assets` and `runtime_selection.workspace_sync`)
- The wait from detach to "run started" (queue and controller-preparing
waits) is outside `heartbeat.prepare.*`.

## Verification

- New test `reconnects to a live remote Runner with its admitted skills
without re-downloading them or waiting on the cache`
(`server/src/__tests__/heartbeat-execution-configuration.test.ts`):
- It admits a run, deletes its skill cache (like a new server), and
holds the company cache lock.
- The reconnect preparation finishes while the lock is held. It returns
the same key, runtime name and version. The session fingerprint is
identical. The cache is not created again.
- Normal preparation of the same saved selection still waits for the
lock and materializes the files ("normal start unchanged").
- Red on master source: `A live Runner reattach must not queue behind
skill downloads: expected null not to be null`. Green with this change.
- New test in `managed-skill-content.test.ts`: with the cache lock held
and an empty cache, `prepareRuntimeContextSource` for a reconnect
returns at once with the admitted context. Its staging step and a normal
start still wait for the lock and then rehydrate the same digests.
- New `server/src/services/heartbeat/live-runner-adoption.test.ts`: the
gate is false for a normal start, a scheduled resume, a dead-Runner
resume, an incomplete bootstrap, another run's claim, a different lease
owner, and a legacy run.
- `npx vitest run
src/__tests__/heartbeat-execution-configuration.test.ts
src/__tests__/skill-version-cache.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/heartbeat/live-runner-adoption.test.ts`: 289 passed, 14
skipped. After the controller change: restart-recovery, diskless
restart, runtime skill preservation, managed skill content, session
executor and execution configuration suites: 1686 passed, 14 skipped.
- `tsc --noEmit -p server/tsconfig.json`: pass.
- `pnpm verify` asks for broader coverage (exit 2). Required CI supplies
it.

After deploy, a reconnect run shows
`heartbeat.prepare.configuration_skills_adopted` and the
`reused_admitted` event, and no `skills.cache.acquire` or
`skills.version.materialize` phases.

## Risks

- Low. The option applies only to a same-run reconnect of a retained
remote Runner that has a saved runtime context and saved skill versions.
Every other start keeps the current path.
- Access revocation and deleted versions still fail the reconnect, as
before. A failure to download the files on this server no longer fails a
live reconnect, because the Runner already holds them.
- Nothing new is saved, so rows from older releases, two live servers,
and a deploy during a reconnect behave as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 03:58:01 -06:00
DottaandClaude Opus 5.5 424e2c6488 fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An upgrade replaces the server. The old and the new server serve
together for several minutes. A legacy run proves that its owner is
alive by renewing a 60-second lease on its `heartbeat_runs` row.
> - #21019 added an uncharged `server_restart` retry lane for runs that
a restart or replacement ended. On the first upgrade that shipped it, no
run used that lane. The new server also failed many runs as
`process_lost` while the old server that owned them was still serving.
> - Each of these runs then spent one failure retry. A run that was
already the last automatic retry was held for manual reconciliation, so
the task needed a person.
> - An upgrade is routine. It must not cost a task its work or a retry,
and it must not leave a task waiting for a manual Retry.
> - This pull request sends a run that a replacement stopped during
preparation to the `server_restart` lane. It also stops the reaper from
failing a run while the server process that owns it is proven alive.
> - The benefit is that a task continues by itself through an upgrade,
also when the database is slow during the swap.

## Linked Issues or Issue Description

Refs #21019, #21042

**What happened?**
A hosted staging server was replaced by an automatic upgrade. The new
server started at 06:18:46Z, became ready at 06:26:17Z, and the old
server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new
server failed 44 runs as `process_lost` ("Process lost -- server may
have restarted; retrying once",
`processLossDiagnostic.controllerReplaced: true`). The old server still
owned these runs and was still serving. Every reaped run was a Runner
run in `executionStage: preparing` with no process recorded. Each
successor was scheduled with `scheduledRetryReason: transient_failure`
and `executionRetryAccounting.failureRetries: 1`. No run on the server
has `retryReason: server_restart` or
`executionRetryAccounting.serverRestarts`.

The run evidence also shows why the leases lapsed. The old server's
lease renewals waited 19 to 50 seconds on the database
(`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time
(`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This
pull request fixes how the reaper and the retry scheduler treat these
runs.

**Expected behavior**
- A server never fails a run as lost while the server process that owns
it is alive and serving.
- A run that a replacement really ended continues once, automatically,
without spending the failure budget.

**Steps to reproduce**
1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`,
owned by another server boot, with a lapsed run lease.
2. Run `reapOrphanedRuns()` on the new server.
3. See that the successor is `transient_failure` with `failureRetries:
1`. When the chain already spent its failure budget, see that no
successor is created and the run is held for reconciliation.
4. Give the owner a live controller instance lease (the old server is
still serving) and run `reapOrphanedRuns()` again: the run is failed as
`process_lost` anyway.

**Root cause**
- `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`)
left every run with the bootstrap evidence `providerWorkStarted: false`
to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun`
gives that lane only to an `interrupted` run (or a native preparation).
The reaper writes the same evidence on a `failed`/`process_lost` run
that it stops during preparation. That run fell through to the charged
`transient_failure` lane. The #21019 tests used runs in `dispatching`,
so they did not catch it.
- `legacyExecutionNeedsReconciliation` held every run whose failure
budget was spent before it checked the bootstrap evidence. A run that a
replacement stopped before dispatch did no provider work, but it was
held for reconciliation, and the task waited for a person.
- The orphan reaper (`recoverActiveExecutions`,
`server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease
as proof that the owner was gone. It checked the shared owner prover
only for "gone", never for "alive". During the swap the old server's
renewals were slow, so its leases lapsed while it was alive.

**Paperclip version or commit**
Staging Core `51d77648bc` (first build with #21019). The behavior is the
same on master `148d2b629c`.

## What Changed

- `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)`
keeps the `interrupted_preparation` exception only when that lane really
takes the run. `scheduleBoundedRetryForRun` passes its own verdict.
Without one, only an `interrupted` run qualifies. A `process_lost` run
that a replacement stopped during preparation now continues in the
`server_restart` lane.
- `legacy-execution-recovery.ts`: a replaced preparation
(`isReplacedPreparation`) does not need reconciliation, also when its
failure budget is spent. The `server_restart` bound limits it.
- `recovery.ts` (orphan reaper):
- When a run's lease lapsed but the shared owner prover proves its owner
process alive (for example, the owner's controller instance lease is
live), the reaper leaves the run with that owner. The owner stops and
settles the run itself (`controller_lease_lost`, uncharged
`interrupted_preparation` lane). The reaper recovers the run once the
owner is gone or stopped. It also recovers the run when the owner has
not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes)
after the lease lapsed.
- A setup that a replacement stopped before dispatch uses the
`server_restart` bound for its retry decision, whatever the adapter.
- `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which
returns the shared prover's verdict. `legacyControllerProvenGone` uses
it, and the reaper computes the proof once per run.
- `server-replacement-interruption.ts`: add
`isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted:
false`, a recorded controller process stamp, and a server-replacement
interruption. A row saved by an older release without the stamp keeps
the charged lane (the existing documented rule).
- `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a
reconciliation hold that an older release saved on a replaced
preparation is settled once its owner is proven gone, even when the
failure budget is spent. The run continues in the `server_restart` lane.
- `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the
`legacy_interrupted_preparation_hold` kind judges a finished run's hold
by its recorded controller process
(`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper
swaps in a random boot id, so before this change a reaped preparation's
hold had owner "none" and was never settled.
- `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports
exhaustion from the scheduler's own `retry_exhausted` result, so a spent
`interrupted_preparation` lane blocks the task even when the synchronous
budget check reads an unspent restart bound.
- `doc/execution-semantics.md`: document these rules.
- No schema change. Nothing changes on the path that starts a run or a
turn (no new waits and no file work).

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- "continues a first attempt / a run that was already the last automatic
retry that a replaced server left preparing, in the restart lane". This
is the production shape: a `paperclip_runner` run in `preparing`, two
servers sweep at the same time. Master: `transient_failure` with
`failureRetries: 1`, and no successor for the spent chain. Now: one
`server_restart` continuation, due at once, with the failure count
unchanged.
- "leaves a preparing/dispatching run whose lease lapsed with its owner,
then recovers it once the owner stops". The owner's controller instance
lease is live and two servers sweep. Master: `process_lost` at once.
Now: the run stays `running` with its owner. After the owner stops, it
becomes `process_lost` and a `server_restart` continuation is scheduled.
- "recovers the run when its live owner has not settled it within the
grace".
- `interrupted-preparation-hold-settle.test.ts`: "releases an older
release's hold on a preparation a server replacement stopped with its
failure budget spent" for an unrevoked and a reaper-revoked lease (red
without the settlement and owner-stamp changes) and "blocks the task
when a replaced preparation's lane is spent even though the restart
bound is not" (red without the sweep change).
- Related suites: heartbeat process recovery, retry scheduling, orphaned
active lease sweep, recovery deadline (bootstrap and ordinary
bootstrap), legacy continuation authority, undispatched execution wait,
agent status recovery, legacy workspace restore recovery, legacy
execution recovery, legacy controller lease, server replacement
interruption, heartbeat recovery, provider failure classification,
reconciliation run read, and wake-queue policy.
- `tsc --noEmit` for `server`.

## Risks

- A run whose owner is alive but no longer drives it now stays `running`
for up to 5 minutes after its lease lapsed, and then the reaper recovers
it. The owner's own sweep still recovers a run that the owner itself
lost the handle for (this boot is never deferred).
- The guard trusts the shared owner prover. That prover reads "alive"
only from a live or grace controller instance lease, or from a live PID
in the same PID space. A server whose instance lease cannot renew for 60
seconds stops itself, so a stuck owner cannot hold runs forever.
- More runs continue in the uncharged lanes. The `server_restart` bound
(6 in 60 minutes) still stops a loop, and the stop is shown on the task.
- Rows saved by older releases: a `process_lost` row with the bootstrap
evidence and an old-boot owner now continues in the `server_restart`
lane instead of being held. These runs did no provider work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution, read-only log and API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 02:18:39 -06:00
DottaandClaude Fable 5.2 fd5ee7e385 fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes (Daytona) use a detached Runner.
When the server is replaced during an upgrade, the new server reattaches
to that Runner
> - On staging, upgrades every 15–40 minutes lose the in-flight Runner.
The same-run reattachment then finds `process_verification` /
`process_not_running`, the run fails with
`runner_remote_process_identity_unavailable`, and the task is blocked
> - #21039 made these runs continue once in a fresh session (the turn is
recorded as having an unknown outcome and is not replayed). It covers
user wakes and automatic retries (`retryOfRunId`) only
> - A routine's system assignment (`issue_assigned`, requested by the
system) and an external chat message wake have no `retryOfRunId`. They
got `not_user_request` and stayed blocked for a person
> - This pull request treats those two wakes as owed task work, so they
continue the same way once the Runner is proven gone
> - The benefit is that an upgrade no longer leaves these tasks waiting
for a Retry click

## Linked Issues or Issue Description

Refs #21039, #21023, #20923

**What happened?**
Staging replaced its server at about 07:10Z. 17 remote native runs were
in flight. Each new controller reached its Runner 114–152 s after
detach. Recovery failed with "PRP command session.snapshot timed out".
The same-run reattachment then found the Runner process gone, and each
run failed with `native_continuation_requires_reconciliation`. #21039
continued 14 of the 17 automatically. The other 3 were a routine's
assignment (two runs) and an external chat message. They have no
`retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned
`not_user_request`. Those tasks are still blocked with no retry.

**What did you expect to happen?**
The Runner is proven gone, so nothing can still run the turn. The run
should continue once in a fresh session, the same as an automatic retry
of owed work.

**Steps to reproduce**
1. Start a remote native run from a routine assignment, or from an
external chat message.
2. Replace the server while the provider turn is active, so that the
reattachment finds `process_not_running`.
3. The run fails and the task stays blocked. No automatic Retry is
requested.

## What Changed

- `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the
Runner is proven gone, a system wake also continues when it is a system
assignment (`issue_assigned`), or when it is a comment or external chat
message wake whose message is still on the task (not deleted, not
written by a run). It runs under the run's responsible user, at most
once per chain of runs, like an automatic retry already does. A linked
person's external chat message continues under that person when the
message is still on the task. Monitors and other system wakes are
unchanged.
- The same file and `server/src/app.ts`: the hook now passes the chat
service's Retry callback (`prepareFailedChatRunRetry`, registered at app
startup next to the budget-resume callback). A task in a connected
conversation whose endpoint restricts external execution then continues
through the chat Retry path. Before, `requestTaskRetry` refused it
(`retry_refused`) and the task stayed blocked.
- `server/src/services/runner-lost-mid-turn-continuation.test.ts`:
production-shape cases for a system assignment, an unlinked external
chat sender, and a linked person's chat message. Negative cases: a
deleted message, a message written by a run, a deleted linked message,
and a system assignment whose Runner was not proven gone.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- Red on `origin/master` with only the new tests: `cd server && npx
vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails
4 of 16. The three new continuation cases return `{ requested: false }`.
The deleted linked-message case returns `not_user_request` instead of
`message_unavailable`.
- Green with the fix: 16 of 16. Two more cases bind the task to a real
restricted Slack endpoint and conversation. With the chat callback
registered, the run continues and the Retry carries the chat action (18
of 18). Without the callback, the result is `retry_refused`, which was
the behavior before for every connected conversation.
- Related suites pass: `runner-lost-mid-turn-continuation.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts` (116 tests).
- `cd server && npx tsc --noEmit` passes.
- Production evidence was read from staging run events through the
read-only API: 3 of 17 losses at the 07:10Z swap took this path. After
this change the next post-upgrade audit should show an automatic
`task_retry` for every `native_continuation_requires_reconciliation` run
whose refusal is `process_not_running`, including routine assignments
and chat messages.

## Risks

- Low. The change applies only after the Runner process is proven gone
(`process_verification` / `process_not_running`), so no Runner can still
run the turn. The turn is not replayed. The fresh session continues over
the saved work, as in #21039.
- The task can contain partial tool work from the lost turn. This is the
same accepted risk as #21039.
- Previous-release story: no persisted shape changes. The rule reads the
wake's reason and comment id, which earlier releases already write. A
run an older release already settled keeps its visible Retry.
- This does not stop the Runner loss itself. That is a separate fix.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic) via Claude Code, with tool use (shell, file edits,
test runs, read-only staging API reads). Model ID as attributed in the
commit trailer: Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-12 02:11:28 -06:00
DottaandClaude Opus 5.5 aa94939988 fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click

## Linked Issues or Issue Description

Refs #20853, #20923, #21023, #20278, #20627, #20828

**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.

**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.

**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.

## What Changed

- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.

## Verification

- Red on `origin/master` (51d77648bc), with only the new tests: `cd
server && npx vitest run
src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 6.
The three continuation cases return `{ requested: false, reason:
"provider_turn_unsettled" }`, and the monitor case returns that reason
instead of `not_user_request`.
- Green with the fix: 6 of 6. The cases cover a user's assignment
through the restart claim, the staging shape (system
`transient_failure_retry` with `retryOfRunId`), and a same-run retry.
Each one checks one recorded unknown-outcome event, an idempotent second
call, saved-work recovery first, one queued successor with
`forceFreshSession: true`, and one Retry intent. Negative cases: a
refusal that did not prove the process gone, no refusal, and a monitor
wake. Each keeps the visible Retry and records nothing.
- `cd packages/adapter-utils && npx vitest run
src/startup-reuse.test.ts`: the new receipt case fails before (`fresh /
created`) and passes after.
- Focused suites pass: `workspace-recovery-retry-copyback.test.ts`,
`governed-wait-runner-loss.test.ts`,
`heartbeat-native-remote-failure-retention.test.ts`,
`native-runner-restart-recovery.integration.test.ts` (302 passed, 14
skipped), and `native-session-executor.test.ts -t "startup
reuse|restart|reattach"` (149 passed).
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/adapter-utils typecheck` pass.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input. CI supplies that coverage.
- Not run: a live Daytona restart. The change is in the server's
recovery decision; the tests use the fixture's sandbox confinement.

## Risks

- Medium. A run whose Runner died in the middle of a turn now continues
by itself. The turn's outcome is unknown, so tool work from that turn
can be partly done. The successor is a fresh session over the saved
workspace; it does not replay the turn. Saved-work recovery stops and
keeps the sandbox before the successor starts, so the old provider
cannot still run.
- Without positive proof that the Runner process is gone, an in-flight
turn keeps the visible Retry, as before.
- An automatic retry of owed work now continues after a lost Runner.
Monitor and other system wakes without `retryOfRunId` do not. The bound
is still one automatic Retry per chain.
- The environment receipt changes only for the reattachment path. No
execution decision reads it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test runs, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 00:13:25 -06:00
DottaandClaude Opus 5.5 940d681974 fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner runs keep a provider session per task. When a run
stops, its session close continues in the server process
> - The runner refuses every new session of the task until that close
settles. A close that cannot prove the old runner suspended needs a
stopped-process proof before it is released
> - A reassignment stops the old assignee's run mid-turn. The new
assignee's run was dispatched one second later into that gate, and it
failed with `native_session_cleanup_quarantined`
> - Only a Retry continued the task. Recovery must work without a
person, and the needed-retry rate must be 0%
> - This pull request keeps such a run queued until the close settles or
its old owner is proven stopped
> - The benefit is that the new run starts by itself after cleanup,
without a failed run or a Retry

## Linked Issues or Issue Description

**What happened?**
In the paid chat eval `tasks.edit-properties` (and
`tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a
task while the old assignee's native run was mid-turn, with a Paperclip
tool call in flight. The reassignment cancelled that run. Its session
close then failed with `native_session_close_unrecoverable: runner did
not durably suspend before checkpoint`, and the runner kept an
operator-required cleanup gate for the task. The new assignee's run had
started 1 second after the cancel. The runner's admission waited for the
in-flight close. When the close escalated, admission refused the run
with `native_session_cleanup_quarantined`. No run was queued again.

**Steps to reproduce**
1. Run a native task and stop its run while its session close cannot
settle at once.
2. Queue a run of another agent for the same task.
3. The run is dispatched into the cleanup gate and fails before it
launches.

**Expected behavior**
The run waits for the cleanup, and starts once the close settles or the
old owner is proven stopped. It never fails, and no Retry is needed.

## What Changed

- `server/src/services/native-runtime/native-cleanup-admission-hold.ts`:
`nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an
operator-required gate it first tries
`retireStoppedSessionCleanupGates`, which retires the gate when the old
owner's runner and provider processes are proven stopped. This is the
same proof that Retry uses.
- `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an
unstarted `paperclip_runner` run queued while its task has a settling or
unproven close. This matches the existing
`predecessor_settling_in_process` hold, which covers only the same
agent. A reassignment changes the agent.
- `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a
released close now also wakes queued-run dispatch. A close that
escalates to operator-required also wakes it, so admission tries the
stopped-process proof at once. The queued-run liveness redrive covers
any later proof.
-
`server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`:
an assembled heartbeat test. It is red on master (the run is claimed
into a pending close and into an operator-required close). It is green
here (the run stays queued, then is admitted after the close settles). A
control case shows that a run for another task is not held.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3
passed; 2 red on master)
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts
src/services/heartbeat/queue.test.ts
src/__tests__/heartbeat-workspace-busy.test.ts
src/services/queued-run-dispatch-liveness.test.ts` (180 passed)
- `cd server && npx tsc --noEmit` passes.
- Paid E2E run on a combined branch:
https://github.com/paperclipai/paperclip/actions/runs/43161593494
(results in a comment below).

## Risks

- A run whose task has an operator-required gate that is never proven
stopped now stays queued instead of failing. Before, that run failed,
and Retry was blocked by the same proof. The liveness redrive re-checks
it with back-off.
- The hold applies only to unstarted `paperclip_runner` runs of the
gated task. Other tasks and agents are not affected.
- Previous-release saved state: no persisted shape changes. The gate
lives in process memory, and runs saved by the previous release are
claimed by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 23:53:54 -06:00
DottaandClaude Opus 5.5 156453fe7c fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A deploy replaces the server. The old server stops its running
legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server
finds runs whose owner died without that drain (`process_lost`, "server
may have restarted").
> - Each of these runs got its follow-up from the bounded transient
retry. That retry spends the failure budget (2 attempts) and waits 30 to
60 seconds.
> - So the first deploy spent one failure retry. A run that was already
the last automatic retry got no follow-up at all. The task then rested
on a manual Retry.
> - A deploy is routine. An interruption by the server is not a failed
attempt, and it must never leave a task waiting for a person.
> - This pull request moves these runs to their own `server_restart`
retry lane. The lane never spends the failure budget, starts the first
continuation at once, and has its own bound.
> - The benefit is that a task continues by itself through several
deploys a day, and a real loop still stops and shows why.

## Linked Issues or Issue Description

**What happened?**
A zero-downtime deploy replaced a hosted server. In the 11 minutes
around the swap, 11 legacy runs ended with `server_shutdown_interrupted`
and 7 ended with `process_lost`. At least 7 tasks were left with no
follow-up run and no scheduled retry, so each needed a manual Retry. Two
of the stranded runs were already automatic retries.

**Expected behavior**
Each interrupted run continues once, automatically, on the surviving
server, with its original wake.

**Paperclip version or commit**
Core commit 7d9973405c. The behavior is
the same on master bf18b348c5.

**Steps to reproduce**
1. Start a legacy run with a conversation adapter (for example
`codex_local`) as the second automatic retry of its chain (failure
budget spent).
2. Send SIGTERM to the server (`drainRunningRunsForShutdown`), or let a
new server boot find the run without its owner (`reapOrphanedRuns`).
3. See that no successor run is created. The scheduler logs "Bounded
retry exhausted".

**Root cause**
- `enqueueProcessLossRetry`
(`server/src/services/heartbeat/recovery.ts`) calls
`scheduleBoundedRetryForRun`. That function put these runs in the
`transient_failure` lane, with `maxAttempts = 2` counted from
`executionRetryAccounting.failureRetries`
(`server/src/services/heartbeat/retries.ts`). A first attempt got a
charged retry 30 to 60 seconds later. A run with 2 spent failure retries
got `retry_exhausted`.
- The orphan reaper (`recoverActiveExecutions`) skipped the retry when
`failedRunRetryBudgetSpent(run)` was true. It also skipped a
conversation run that had no local child process recorded (for example a
run that was dispatching), because `shouldRetry` needed a local child, a
preparing stage, or a monitor wake.
- The stranded-task sweeper then saw the spent budget and moved the task
to `blocked`.

## What Changed

- `execution-recovery-attempt.ts`: add the `server_restart` retry lane.
It is an uncharged lane. Its counter (`serverRestarts`,
`serverRestartWindowStartedAt`) is saved in `executionRetryAccounting`,
so it is visible on each run. The bound is 6 continuations in one
60-minute window. A later interruption after the window starts a new
window.
- New `server-replacement-interruption.ts`:
`isServerReplacementInterruption(run)` names a legacy
`server_shutdown_interrupted` run, or a `process_lost` run whose owner
was another server boot. It reads the new
`processLossDiagnostic.controllerReplaced` verdict. For rows saved by an
older release, it reads the owner boot id or `runPredatesObserver`.
- `retries.ts`: `scheduleBoundedRetryForRun` uses the `server_restart`
lane for these runs (`usesServerRestartLane`). The first continuation is
due at once; later ones in the window back off like the
interrupted-preparation lane. `failedRunRetryBudgetSpent` reports the
restart bound for these runs, so the sweeper escalates only when that
bound trips. An interrupted preparation keeps its own lane while its
chain has failure budget left.
- `recovery.ts` (orphan reaper): record `controllerReplaced`. A
conversation run that a restart ended is eligible for continuation even
without a local child process, and it checks the restart bound, not the
failure budget.
- `wake-queue/adapters/postgres.ts`: carry the restart counter to
successors that the wake queue creates.
- No schema change. No change to how runs start (no new waits before a
turn).

Behavior that did not change:
- At most one successor per run: every path still uses the predecessor
row lock and the `retry_of_run_id` check in
`scheduleBoundedRetryForRun`. Two servers alive at once reuse the same
successor.
- A non-conversation adapter that reached dispatch stays held for
reconciliation, because a replay can repeat its actions.
- Native runtime runs keep their own recovery.
- A server that began shutdown already starts no queued run
(`closeProcessRunAdmission` in `server/src/index.ts`, checked in
`heartbeat/queue.ts`). Its queued and newly scheduled runs stay in the
database for the new server.

## Verification

- Red on `origin/master`, green on this branch
(`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs
ended by a server restart or replacement"):
- Graceful shutdown, first attempt: master scheduled a charged
`transient_failure` retry. Now one immediate `server_restart`
continuation.
- Graceful shutdown, already the last retry: master scheduled nothing.
Now one continuation; the failure count stays 2.
- Owner lost at boot (`process_lost`), first attempt and
already-retried: same results, with two servers running the boot sweep
at the same time. Only one successor.
- Old server exited after it committed the interruption and before it
scheduled the continuation: the new server's sweep continues the run.
- Run saved by an older release (no `controllerReplaced`, spent failure
budget): the sweep continues it instead of moving the task to `blocked`.
- The restart bound trips after 6 in one window, records `retryReason:
server_restart` on the run, and the sweeper escalates. The same count in
a closed window continues.
- A run that failed by itself (`adapter_failed`) with a spent failure
budget still gets no retry.
- Each continuation's execution continuation reports the original wake
(`trigger.retriedReason = issue_monitor_due`).
- Unit tests:
`server/src/services/server-replacement-interruption.test.ts`.
- Six earlier tests asserted escalation after "three deploy restarts"
spent the failure budget. They now spend the restart bound instead, so
they still prove escalation when the bound trips.
- Existing test `server/src/services/heartbeat/queue.test.ts` covers
"starts no run once shutdown closed run admission".
- `pnpm verify`: see the PR checks.

## Risks

- A run that a restart ends now continues more often (up to 6 times per
hour per chain, not 2 in total). A run that itself crashes the server is
still stopped by the bound, and the stop is shown on the task.
- Tasks that an older release already moved to `blocked` stay blocked.
Tasks that are still `in_progress` or `todo` with a stranded run
continue on the next sweep.
- A `process_lost` row from an older release whose owner was the same
boot that later restarted is read as a restart. That case is a lost
in-memory handle, which is also not a failure of the agent's work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code
search, shell, test execution) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:56:49 -06:00
DottaandClaude Opus 5.5 5a7378af56 fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued runs. The run-dispatch worker
re-runs the queue only when a write sends a "queued run dispatch"
signal.
> - A follow-up run can queue while its predecessor still holds its
sandbox lease. The claim then declines the run without a log line, and
the run stays `queued`.
> - After a controller restart, nothing sent a new signal once that
lease was settled. The run stayed `queued` forever. The issue had no
Retry, and the reconciler counted the queued run as a valid next action.
> - This pull request sends a dispatch signal from the native
finalization writes that can clear a predecessor's cleanup gate, logs
why a claim was declined, and bounds the wait. The dispatch worker
re-drives an undispatched queued run after 60 s as a backstop, and the
ownership reconciler reports and re-drives one still queued after 5
minutes.
> - The benefit is that a follow-up starts within seconds of its
predecessor settling, a lost signal delays it by about a minute instead
of forever, and a run that still cannot start is visible.

## Linked Issues or Issue Description

Refs #20909, #20935, #20953, #20970

**What happened?**

Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` +
#20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold,
`--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the
outage with an accepted result. The next controller finalized it at
22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z
(`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source
`native_status_decision`). The run stayed `queued` with `claimed_at =
null` until the wave ended 19 minutes later. The agent stayed `running`
with no running run. The task had no Retry. The grade was
`R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`.

**Root cause**

`claimQueuedRun` (`server/src/services/heartbeat/queue.ts`,
`lockIssueExecutionClaim`) declines a queued run while
`hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is
true for its predecessor. After the restart, the predecessor's warm
`reuse_by_environment` lease was still `active` when the continuation
queued, so the first claim was declined. The decline is correct, but
nothing owned the retry. `registerRunDispatchWork`
(`server/src/services/heartbeat/dispatch-work.ts`) has `next: async ()
=> null`: the queue is only re-run by a later dispatch signal. At wave
end the lease was `retained` with `cleanupStatus: success`, which does
not block a claim. A dispatch pass in that state starts the run
(verified locally). So no pass ran after the gate cleared. The ownership
reconciler listed the run as `queued_run` with owner `run_dispatch`
(always "alive"), and I7 treats any queued run as a next action, so
nothing reported it.

This is not the #20935 throttle. The wave base predates #20935, and a
throttled intent keeps its wake intent `queued` with a `nextAllowedAt`
deadline. Here the run row existed and its claim was never retried. On
current master #20935 parks this specific `response_wake` self-wake, but
any continuation, comment wake or Retry that queues behind a settling
predecessor after a restart has the same gap. The red test uses a
`same_agent` continuation, which master still enqueues.

**Steps to reproduce**

`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`,
"starts a continuation that queued behind its predecessor's sandbox
release after a graceful restart".

## What Changed

- Event-driven dispatch. A lease release already signals queued-run
dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`,
stop-and-retain settlement). I enumerated every `environment_leases`
status writer: each one that moves a lease out of a blocking state
already signals. The writers that do not signal only acquire a lease or
set `pendingCleanupInFlight`, so they add a block and do not clear one.
The native coordinator side had no signal, so:
- `notifyNativeFinalizationWork` (the shared notifier of coordinator
commits and terminal transitions) now also signals queued-run dispatch
for the company.
- The coordinator writes that clear `leaseOwner` or end the phase
outside that notifier now signal in the same transaction:
`releaseLapsedCoordinatorOwner`, native startup cleanup,
observed-workspace preservation, `record_finalization_error` in the
interaction bridge, the wake-queue terminal recovery, and the
reconciler's ambiguous-state terminal failure.
- Not changed: three non-transactional pre-dispatch closes in
`run-completion.ts` and `agent-execution-stop.ts`. Their callers queue
or settle the successor after the write, so the successor's own enqueue
starts the queue.
- `claimQueuedRun` now logs the reason when it declines a queued run at
the issue execution claim (`predecessor_cleanup_pending`,
`predecessor_still_executing`, `issue_execution_owned_by_another_run`,
case handoff, task chat cleanup), when a predecessor is still settling
in this process, and when a temporary budget hold keeps it queued.
- New `server/src/services/queued-run-dispatch-liveness.ts`:
`listUndispatchedQueuedRuns` reads queued runs in active companies whose
ready, non-paused agent has no running run. It reads every page with a
keyset cursor until the set is exhausted, so older held runs cannot hide
a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each
such run 60 s after it was created, then every 2, 4, 8 minutes, capped
at 15 minutes. Run ids that leave the set are dropped.
- `server/src/services/heartbeat/dispatch-work.ts`: the queued-run
dispatch worker's `next` now plans those re-drives. It re-reads only the
companies the pass just dispatched, keeps the deadlines of every other
company, and runs no query when the pass dispatched nothing. Only an
instance-wide pass (startup or explicit wake) reads every company. It
adds each due company to the dispatch scopes and returns the next due
time.
- `packages/db`: new partial index `heartbeat_runs_queued_created_idx
(company_id, created_at, id) WHERE status = 'queued'` (migration
`0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX
CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in
`client.ts`). It returns `null` (no deadline, no SQL until a signal)
when no run is undispatched or scheduling is suppressed. Every re-drive
goes through the normal `startNextQueuedRunForAgent` and
`claimQueuedRun` gates, so it never bypasses a hold, budget, pause or
cleanup gate.
- `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns`
into the worker (two lines).
- `server/src/services/recovery/ownership-invariant-report.ts`: new
check. A queued run still undispatched 5 minutes after creation is an I7
violation (`kind: queued_run`, `reason: queued_run_not_dispatched`),
written to the activity log once. A younger one sets the report's next
due time.
-
`server/src/services/native-runtime/native-finalization-reconciler.ts`:
`applyOwnershipReportBackstops` sends a queued-run dispatch signal once,
for each company with a newly found violation. Repeat passes do not
signal again, so a held run does not wake the report every 30 s.
- `tests/restart-chaos/README.md`: records the wave and the fix in the
CR results table. The existing CR1 cell (#20909) and the enforced U1
check already cover this class in the harness.

## Verification

- RED on master (`23909077ad`):
- `heartbeat-native-restart-suspension.test.ts` "starts a continuation
that queued behind its predecessor's sandbox release after a graceful
restart": the continuation is `queued` behind the predecessor's `active`
warm lease. The run-dispatch worker is registered, the lease then
settles to `retained`/`success` with no dispatch signal, and the run
stays queued: `AssertionError: expected null not to be null` after 75 s.
- `ownership-reconciliation-report.test.ts` "reports and re-drives a
queued run that no dispatch started while its agent had nothing
running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no
violation for a run queued 19 minutes).
- `ownership-reconciliation-report.test.ts` "signals queued-run dispatch
when finalization work settles or a finished coordinator's lapsed owner
is released": `expected [] to include 'settled'`.
- GREEN on this branch: all pass. The restart test has two cases. When
the lease is released through `releaseLease`, the continuation starts
within 5 s. When the lease is settled by a write with no signal, the
backstop starts it within 75 s. A new report test puts three older held
runs ahead of a stranded run with a page size of 2, finds the stranded
run, and checks that the backstop signals once.
- `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap,
and going quiet.
- Focused suites pass (22 files, 495 tests):
recovery-deadline-acceptance, live-recovery-work,
heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work,
deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the
ownership kinds suites, and others.
`heartbeat-native-restart-suspension` and
`ownership-reconciliation-report` pass in full. `tsc --noEmit` for the
server passes.

## Risks

- A queued run that is held on purpose (for example a temporary budget
hold or a case handoff waiting for release) is now re-checked at most
every 15 minutes after the first minute. Each re-check is one indexed
query and the normal claim gates. Such a run is also reported once as an
I7 violation.
- An instance with an undispatched queued run keeps a dispatch deadline,
so it does not idle while that run waits. An instance with no such run
is unchanged.
- Schema: one additive partial index, built concurrently. No data
change. State saved by an older version is read as is. Nothing runs
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:04:58 -06:00
DottaandClaude Opus 5.5 601babe322 fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages

## Linked Issues or Issue Description

**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.

**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.

**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.

**Paperclip version or commit**
master at 9dfdeaad86.

## What Changed

- Added `GET /api/companies/{companyId}/observability/status`. Agents of
the company and all members can read it. It returns `configured`,
`enabled`, `provider`, `credentialsConfigured`, `updatedAt` and the
latest test status. It never returns the destination or a credential.
- Added the shared `AgentObservabilityStatus` type and
`agentObservabilityStatus()` mapper, and registered the route in the
OpenAPI catalog so that `search_api` finds it.
- The chat task markdown now fills the conversation's own company prefix
(from its identifier, for example `COM-1` gives `COM`) into the chat
directives in place of `/<prefix>/`.
- `search_api` results now get the same substitution for the caller's
company. The shared catalog is not changed.
- Pinned the changed shared source in
`tests/runner-e2e/source-sdk-public-inputs.json`.

## Verification

- `cd server && npx vitest run src/__tests__/agent-observability.test.ts
src/__tests__/agent-observability-status-guidance.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`
- Red/green: with the product change removed, 10 of the new or updated
tests fail (the agent status read gets 403, search does not find the
route, and the chat markdown still contains `/<prefix>/`). With the
change, all pass.
- `npx tsc -p server/tsconfig.json --noEmit` passes.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
reports no change.

## Risks

- Low risk. The new route is read-only and returns no destination and no
credential. The existing settings route keeps its board-only rule.
- The chat prompt text changes for chats with an identifier. The text is
constant for each company, so prompt caching is not affected inside a
conversation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:55:21 -06:00
DottaandClaude Opus 5.5 dfd2a6efca fix(native): restart a follow-up the closing server cut before it attached, instead of reattaching to its predecessor's Runner (#20979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs on a remote sandbox keep a warm Runner between turns. A
follow-up run takes over its settled predecessor's provider session and
Runner.
> - When a deploy stops the server, a follow-up that is still preparing
fails with `local_runtime_api_listener_unavailable`. The executor saves
this failure as a same-run bootstrap retry (#20786). The saved failure
proves that the run did no provider work of its own.
> - After the restart, restart recovery saw a checkpoint and a provider
session, and chose `reattach_remote_runner`. But the checkpoint and the
Runner identity belong to the predecessor. The Runner identity check
then failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The run ended terminal
and the task needed Retry.
> - This pull request recognizes that durable shape in restart recovery
and restarts the run as a fresh bootstrap on its warm session. It does
not try to reattach to a Runner that belongs to a different run.
> - The pull request also stops the queue from claiming a queued run
after shutdown closed run admission during the queue scan.
> - The benefit is that a deploy during a follow-up's preparation no
longer ends the run terminal with a Retry.

## Linked Issues or Issue Description

Refs #20786, #20928, #20852, #20904

**What happened?**
In three Codex runs across two Daytona restart-chaos waves (warm
sandboxes, CR1 and K15 with overlap), a warm follow-up run
(`automation`, `retryOfRunId` = its predecessor) started 0–2 s before
its server got SIGTERM. Its bootstrap failed with
`local_runtime_api_listener_unavailable` and saved a `bootstrap_retry`
with `providerSessionEstablished: false`, `providerEventsExist: false`.
After the restart, `native.recovery.transition` chose
`reattach_remote_runner`. `verifyRemoteRunnerReattachment` then threw
`identity_run_id`, because the warm Runner's identity still named the
predecessor. The run ended with
`native_continuation_requires_reconciliation`. In every sandbox capture
the Runner was cleanly suspended, with no stderr.

**Expected behavior**
The run did no provider work and never wrote its own Runner identity.
The next server must restart it without a Retry.

**Steps to reproduce**
1. Finish a remote warm native run, so that its provider session and
Runner identity are saved.
2. Start a follow-up run. Admission carries the predecessor's provider
session and Runner process identity into the follow-up.
3. Send SIGTERM while the follow-up prepares, so that it fails with
`local_runtime_api_listener_unavailable`.
4. Restart the server. Restart recovery chooses
`reattach_remote_runner`, and the run fails with `identity_run_id`.

## What Changed

- `native-restart-recovery.ts` (`claimNativeRestartRecoveries`): add the
`remoteFollowUpNeverAttached` proof. It is true only when all of these
are true:
- the run is a failed, settled `bootstrap_retry` whose original failure
is `local_runtime_api_listener_unavailable`, with
`providerSessionEstablished: false`, `providerEventsExist: false`, no
successor and no lease owner;
- the run is on a prepared remote sandbox workspace with the current
preparation fencing version, it is not a plugin task, and it has no
Runner controller checkpoint and no staged attachment of its own;
- its session checkpoint is only the carried admission checkpoint (no
cursor, turn, result, terminal or pending request), and a settled
earlier run of the same session saved that exact provider session;
- its Runner process identity is absent or is exactly the identity of a
settled predecessor;
- it has no provider event, no Runner-originated event, and no
`native.process_identity_recorded` event of its own.
With that proof, the run gets a `bootstrap_incomplete` claim with the
reason `remote_follow_up_never_attached`, and it passes the same
current-admission check as the other settled bootstrap retries.
- `verifyRemoteRunnerReattachment` is unchanged. It still refuses a
Runner that names a different run.
- `heartbeat/queue.ts` (`startNextQueuedRunForAgent`): check run
admission again before each claim. A run that a scan reaches after
shutdown closed admission stays queued for the next server.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: eight cases for
the staging shape. Three restart (inherited Runner identity; inherited
identity plus a Runner request; a Runner request with no identity). Five
stay held (own recorded Runner identity; a Runner-originated event; a
carried active turn; a provider session that no settled run saved; a
saved failure that established a provider session).
- `queue.test.ts`: a queued run stays queued and is not executed when
admission closes during the scan.

### How this relates to #20852 and #20904

- #20852 hands a preparing run to the next controller and persists
launch intents, so that preparation itself survives a deploy. It does
not change restart recovery classification for a settled
`bootstrap_retry`, and it does not touch the run-admission gate. This PR
covers runs that already failed and saved a `bootstrap_retry` (including
state that older releases saved), so the two are complementary.
- #20904 changes startup readiness for native-controller preservation
obligations. It does not touch either code path in this PR.

### Finding about the "draining server" admission

In both waves the follow-up was claimed before the process observed
shutdown. In CR1 (seed 9511) the run was created at 21:54:04.695Z and
started at 21:54:04.854Z; the server logged "shutdown signal received"
at 21:54:05 (the harness sent SIGTERM at 21:54:03.256Z, under load
average 10). In the K15 overlap (seed 9513) the runs started at
23:31:10.9Z and 23:31:11.5Z, and the old server got SIGTERM at
23:31:13Z. During the overlap the old server is fully live and has no
signal that a newer deploy serves traffic. So the existing admission
gate (`closeProcessRunAdmission`) was not bypassed. The only gap in the
gate was the check-then-claim window inside
`startNextQueuedRunForAgent`, which this PR closes. Holding work on a
superseded but not yet signaled server needs a new "superseded deploy"
signal; this PR does not add one.

## Verification

- Red on master (`23909077ad`): `node cli/node_modules/tsx/dist/cli.mjs
packages/db/src/test-postgres-command.ts -- node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "closed the runtime listener before it attached"`: the three restart
cases failed with `"kind": "reattach_remote_runner"` (the staging
disposition); the five held cases passed.
- Red on master: `queue.test.ts -t "closes admission during the queue
scan"` failed (`expected [ { …(81) } ] to deeply equal []`).
- Green with the fix: both filters pass.
- `native-runner-restart-recovery`, `native-runner-restart-preparation`,
`native-startup-retry` and `native-controller-recovery` integration
files: 267 passed, 14 skipped.
- `queue.test.ts`: 89 passed.
- `cd server && pnpm typecheck`: passed.
- Chaos waves: see the PR comments for the reruns of seeds 9511 (CR1)
and 9513 (K15 overlap).

## Risks

- The new proof applies only to the closed-listener `bootstrap_retry`
that #20786 writes, on a remote sandbox, with a carried session that a
settled predecessor saved. Every other shape keeps its earlier
classification.
- A `bootstrap_incomplete` claim sends the run through the existing
bootstrap path, which verifies the prior owner of the session root
before it starts. The s9205 follow-ups already use this path with an
inherited Runner identity.
- Upgrade safety: no persisted shape changes. The proof reads only
fields that the current and recent releases write. A row saved without
these facts keeps the earlier verdict.
- No wait or file work is added before `turn.start`. The new queries run
only in restart recovery.
- The queue change only stops claiming after admission closed. Runs stay
queued for the next server, the same as the existing entry check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 18:36:45 -06:00
DottaandClaude Opus 5.5 827bd21017 fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy

## Linked Issues or Issue Description

Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).

**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.

**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.

**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.

**Paperclip version or commit**
master at c1edb9ec02.

**Agent adapter(s) involved**
Not adapter-specific (core bug). Seen with `paperclip_runner` (Daytona
sandbox).

## What Changed

- `server/src/services/cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` accepts a failed
`controller_lease_lost` run with the bootstrap certificate
(`providerWorkStarted: false`), the same as a certified `process_lost`
run. The error label alone still grants nothing. This lets the existing
`legacy_interrupted_preparation_hold` settlement, terminal-time proof,
status receipt, and `interrupted_preparation` retry lane apply to these
runs.
- `server/src/modules/wake-queue/*`: the release snapshot carries
`retryableBeforeProviderWork` (bootstrap certificate plus a
`transient_infra` continuation classification). When an automatic
continuation retry failed for such a cause, release returns `released`
instead of blocking. The stranded-task sweep then retries with its
existing backoff and 3-attempt cap, and blocks only after that budget is
spent. Non-retryable causes, uninvokable agents, and failures after
provider work still block at once.
- `doc/execution-semantics.md`: documents both rules.
- No migration. Holds saved by the running release are settled by the
existing ownership reconciler after deploy.

## Verification

- Red then green, in the staging shape (failed `controller_lease_lost`,
`renewal_deadline` lease-loss evidence, `restore_failed` with
`workspaceRestoreRecovery`, exhausted `verify_execution` with
`source_unavailable` / `use_alternative_strategy`, retired prepared
epoch):
- `src/__tests__/interrupted-preparation-hold-settle.test.ts`: before
the fix, both new cases fail (`settled: 0`). After the fix, the hold
settles as `false_positive`. With the status receipt, the task returns
to `in_progress` and one `interrupted_preparation` successor is queued.
For a hold saved by an older release without a receipt, the execution
gate clears and one Retry starts a run. A lost lease without the
no-provider-work certificate stays held.
- `src/modules/wake-queue/domain/policy.test.ts` and
`application/use-cases.test.ts`: before the policy change, the two new
release cases fail (the task is blocked). After it, a pre-provider
lease-loss retry failure is released to the sweep, and other failures
still block.
-
`src/modules/wake-queue/adapters/retryable-before-provider-work.test.ts`:
classification of the release snapshot fact.
- `pnpm verify` and server `tsc --noEmit` pass locally.

## Risks

- Lost-lease preparations now use the `interrupted_preparation` retry
lane. That lane is uncharged but bounded at 12 retries, and it does not
revive a chain whose failure budget is spent.
- A released continuation waits for the stranded-task sweep (30 s
reconstruction, plus backoff) instead of a direct requeue. The sweep's
budget bounds the chain.
- Settlement still needs every existing proof: the controller is gone or
settled, leases are released with stop receipts, there are no execution
events or native coordinator, and the prepared epoch is retired or the
files are repaired.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, in Claude Code with tool
use (shell, file edits, browser reads of staging with GET requests
only).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 17:30:32 -06:00
DottaandClaude Opus 5.5 edd00526e7 fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.

## Linked Issues or Issue Description

**What happened?**

Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.

Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.

**Expected behavior**

A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.

**Steps to reproduce**

Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.

## What Changed

- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.

## Verification

- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.

## Risks

- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:40:44 -06:00
DottaandClaude Opus 5.5 34f71f7d57 fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed

## Linked Issues or Issue Description

**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.

**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".

**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.

## What Changed

- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.

Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.

### How the directory-wait retry keeps the original wake

The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.

| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |

## Verification

- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).

- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.

- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.

- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.

## Risks

- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:45 -06:00
DottaandClaude Fable 5.2 a56b648602 fix(upgrade-readiness): report restartable runs and a handoff verdict that blocks only on lost work (#20910)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deployer can replace the server while agent runs continue. Before
the swap, it reads the upgrade-readiness report of the old server
> - The version 1 report is ready only when every active run is
adoptable and no wake admission is in flight
> - On a busy instance, new runs arrive each minute. Some runs are
always in preparation, so the report is almost never ready and the
instance does not upgrade
> - A run in preparation has done no agent work. Restart recovery starts
it again. A swap loses nothing
> - This pull request adds a `handoff` object to the report. It
separates recoverable, restartable and blocking runs
> - The benefit is that a deployer can wait only for work that a swap
would lose

## Linked Issues or Issue Description

Refs #20852 (reference only; this change does not depend on it and does
not use its version 2 shape).

**What happened?**

A busy instance had 16 running runs and 1 queued run. Fourteen had a
live process. Two or three native runs had not launched a process yet.
One was a legacy run. `upgradeReadiness.ready` needs `blockingRuns === 0
&& pendingWakes === 0` in one probe. That state almost never occurs, so
the deployer never swaps the server.

**Expected behavior**

The report tells the deployer which runs a swap would lose. Runs that
only wait for a launch, and wake admissions in flight, do not block a
graceful replacement.

**Steps to reproduce**

1. Keep a steady flow of new runs on an instance.
2. Read `GET /api/instance/task-drain?upgradeReadiness=1` repeatedly.
3. See `ready: false` each time, because one or more runs are in
preparation or a wake admission is in flight.

## What Changed

- `summarizeUpgradeReadiness` returns an additive `handoff` object:
`version`, `ready`, `recoverableRuns`, `restartableRuns`,
`blockingRuns`, `pendingWakes`, `unsavedWakes`.
- `handoff.ready` is true when the instance is not draining, the census
is complete, and `handoff.blockingRuns` and `handoff.unsavedWakes` are
0. Saved pending wakes do not change it.
- The wake queue tracks each public wake admission until its wake
request and queued run commit. `unsavedWakes` is the count of admissions
before that commit. Without a valid count, every pending wake is
unsaved.
- The top-level version 1 fields keep their values and meaning. `ready`
and `blockingRuns` are unchanged, so a current consumer sees no looser
verdict.
- The census query marks a run `restartable` only with all of these:
`paperclip_runner` adapter; no process identity; no provider session
before or after; no checkpoint header; no dispatched runner task; no
execution-start event. It must also be a native run in the `observed`
phase with preparation fencing and an unsettled finalization record, or
a legacy run in the `preparing` stage under a recorded controller.
- All other runs without adoption evidence stay blocking. This includes
local adapters, legacy runs with a child process, follow-up runs that
carry a provider session, and runs the census could not observe.
- The OpenAPI description and `doc/testing-native-remote-continuity.md`
describe the new fields and their limits.

Why these classes are correct on master:

- Restartable native runs: restart recovery claims a run with no
checkpoint and no provider evidence as `bootstrap_incomplete` and
executes the same run again (`native-restart-recovery.ts`
`classifyNativeRunnerRecoveryEvidence`; `heartbeat/recovery.ts` executes
each claim). A graceful shutdown suspends the run and notifies recovery.
Existing tests: "recovers an unclaimed remote startup once without
requiring a provider checkpoint" and "recovers an unclaimed remote
startup that a graceful shutdown suspended during preparation".
- Restartable legacy runs: process-loss recovery retries a `preparing`
run with `providerWorkStarted: false`. Existing test: "continues a chain
whose preparations were killed twice and then interrupted again, without
a Retry".
- Pending wakes: the count is wake admissions in flight in the process,
not queue rows. A wake commits its wake request and its queued run in
one transaction. After the commit, the next server dispatches the queued
run at startup, so a saved wake does not block. Before the commit,
nothing is saved. Shutdown waits at most 30 seconds and then exits, so
an unsaved wake can be lost and blocks.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/upgrade-readiness.test.ts
src/services/heartbeat/upgrade-readiness-post-result.test.ts`: 28
passed.
- Red/green: before the implementation, the 7 new tests failed (6
summary tests, 1 PostgreSQL census test). After it, all pass.
- Red/green for unsaved wakes: a test holds the agent row lock while a
wake admission waits. Without the fix, `handoff.ready` was true with no
saved wake. With the fix, it is false with `unsavedWakes: 1`, and the
wake saves after the lock releases.
- New cases: only recoverable runs; recoverable plus unlaunched runs
plus pending wakes (`handoff.ready` true, top-level `ready` false and
`blockingRuns` unchanged); a started run without recovery evidence; an
unobserved execution; draining; truncated census.
- The PostgreSQL test changes one fact at a time (process identity,
provider session, phase, fencing, checkpoint, adapter, each
execution-start event) and expects the run to become blocking.
- `cd server && npx vitest run src/__tests__/openapi-routes.test.ts
src/__tests__/instance-settings-routes.test.ts`: 157 passed (187 with
the two readiness files).
- `cd server && npx tsc --noEmit`: passed.
- `pnpm verify` exited 2 (a documentation path is a global CI input).
Required CI supplies the broad coverage.
- Not run: a live server replacement. The classification relies on the
existing recovery tests named above.

## Risks

- Low risk for current consumers. The change is additive and the version
1 fields are unchanged.
- `handoff.ready` is an observation at one instant. A wake admission
that starts after the probe is unsaved until it commits.
- Restart recovery applies its own proofs and a bounded attempt budget
before it starts a restartable run again. The report is an observation,
not a guarantee for each run.
- The probe adds one indexed `not exists` on run events per running row,
inside the existing two-second statement timeout.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`), with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 13:55:47 -06:00
DottaandClaude Opus 5.5 7e8901aa60 fix(recovery): let Retry and the person's answer continue a held task with a pending decision (#20872)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process.
Recovery then saves the sandbox files and puts an execution hold on the
task
> - The hold offers Retry. A person can also continue the task with a
new message
> - Many held tasks also have a pending decision: the question that a
parked turn asked, or an older connection request
> - Continuation admission refuses every continuation while a decision
is pending ("decision_pending"). The answer to that decision is not a
message, so admission defers it behind the hold ("execution_recovery")
> - So Retry waits for the answer and the answer waits for the hold. The
task shows a Retry that does nothing, and nothing continues it
> - This pull request lets a person's Retry pass an open question, and
lets the person's own answer continue the task through the hold
> - The benefit is that a task with a lost Runner and a pending decision
always has a control that works

## Linked Issues or Issue Description

**What happened?**
On staging, the same-run retry of six remote native runs found the
Runner process gone (`process_not_running`, lifecycle `ready`). Each run
failed with `runner_remote_process_identity_unavailable`. Saved-work
recovery then recovered the sandbox files, and the task showed "Saved
work recovered. Retry or send a message to continue." with `canRetry:
true`. Four of these tasks also had a pending decision: the question
that the parked turn asked, or an older pending connection request. For
those tasks, Retry is deferred with `decision_pending`, and an answer to
the decision is deferred with `execution_recovery`. No control continues
the task.

**What did you expect to happen?**
Retry starts one new run. The person's answer to the pending decision
also continues the task.

**Steps to reproduce**
1. A remote native run asks the user a question and parks the turn.
2. The Runner process exits. The same-run retry fails with
`runner_remote_process_identity_unavailable`.
3. Saved-work recovery recovers the files. The task offers Retry.
4. Click Retry: the Retry waits with "A pending approval or question
must be resolved before this message can start."
5. Answer the question: the answer waits with "Waiting for execution
recovery. Your message is saved."

Refs #20838, #20853

## What Changed

- `admitExplicitNativeContinuation`
(`server/src/services/explicit-native-continuation.ts`): a task Retry is
no longer held by a plain pending question (`ask_user_questions`) or
connection request (`connection_intent`). Any interaction with approval
semantics still holds Retry: a plan review (`issue_document` target), a
governed tool action, a secret proposal, a connection authorization, and
other confirmations. A pending approval still holds every continuation.
A new message supersedes a plain question, as on master, and waits for a
decision with approval semantics.
- `admitExplicitNativeContinuation`: the person's own resolution of an
interaction (`mutation: "interaction"`, answered/accepted/rejected,
resolved by the same user) is explicit user input. It continues the held
task like a new message: a fresh session, the hold resolved as
`explicitUserContinuation` with the `interactionId`, and
`issue.execution_recovery_settled` activity. It never replays the failed
run. Governed tool-action confirmations are excluded. The existing
`readQueuedInteractionResponse` reads the answer.
- `heartbeat/queue.ts`: both admission calls pass the wake payload to
the admission.
- `buildExecutionContinuation`
(`server/src/services/execution-continuation.ts`): dispatch rechecks the
user authorization. It accepted an interaction receipt only through a
saved interrupt queue, so the answered successor failed with
`continuation_user_authorization_missing`. It now also accepts the
person's own answer: the interaction named by the durable authorization
was resolved by that user and not by an agent, and the authorization
names this successor run. The check does not depend on the run's current
trigger, so a later answer that replaces the trigger cannot invalidate
it.
- New API-level test
`server/src/services/governed-wait-runner-loss.test.ts` with the two
staging shapes (a parked question from the failed run; an older
connection request on a monitor wake). It checks that Retry starts
exactly one run and leaves the decision pending, that the person's
answer starts exactly one run and settles the hold, and that an answer
resolved by a different user keeps the hold.
- `doc/execution-semantics.md`: describes the rule.

## Verification

- `cd server && pnpm exec vitest run
src/services/governed-wait-runner-loss.test.ts`: 5 passed. The answer
tests now also build the successor's continuation as dispatch does and
check that the agent receives the answer.
- Red for dispatch (first head, ac17aebe): both answer tests fail with
`continuation_user_authorization_missing`.
- Ponytail follow-ups: a two-answer test (the second answer waits behind
the successor; dispatch also holds with a replaced trigger) and 'keeps
Retry on hold while %s is pending' for four approval-bearing kinds. On
head 60beb773, 5 of these fail. Two more tests show that a new message
supersedes a plain question and waits for a plan review (both unchanged
from master). All 12 tests in the file pass now. The PR adds no code
comments; the rationale is in `doc/execution-semantics.md` and here.
- Continuation suites (execution-continuation,
execution-continuation-provenance, legacy-continuation-authority,
explicit-native-continuation, task-retry, governed-wait-runner-loss):
634 passed.
- Red on master (product hunks removed, test kept): 4 of 5 fail. Retry
tests: `successors: 0`, deferred `decision_pending`. Answer tests:
`successors: 0`, deferred `execution_recovery`. Every case asserts
`runError: "runner_remote_process_identity_unavailable"`.
- Related suites: explicit-native-continuation, task-retry,
workspace-recovery-admission, retry-workspace-continuity,
workspace-recovery-retry-copyback, workspace-recovery-fallback,
native-workspace-reconstruction, native-recovery-preparation-failure,
issue-thread-interactions-service. 957 of 959 passed. The 2 failures are
30 s `afterEach` hook timeouts in native-workspace-reconstruction; all
51 of its tests pass alone, and the same file on unchanged master shows
5 hook timeouts on this machine.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Medium. A Retry can now start while a plain question or connection
request is pending; decisions with approval semantics still hold it. The
new run sees the open decision in its task context, and the decision's
answer still wakes the agent later. Approvals keep holding Retry, so
approval gates are unchanged.
- An answer now continues a held task in a fresh session. All other
admission gates still run: stop proof, saved-work recovery, workspace
repair, active execution, pause, budget and ownership. Only the user who
resolved the interaction can continue this way.
- This does not make recovery automatic. A task whose Runner was lost
still rests on the hold until a person clicks Retry or answers.
Automatic settlement of a parked governed wait is a separate change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:42:36 -06:00
DottaandClaude Opus 5.5 c02de922af fix(chat): deliver the person's time zone to chat agents (#20898)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to agents in chat, and agents create routines and other
scheduled work from that chat
> - Paperclip does not record the person's time zone, so a chat agent
cannot read "every weekday at nine" or "today" in the person's zone
> - The agent then saves schedules in UTC, and the person gets the work
at the wrong hour
> - This pull request saves the browser time zone with the person's
personal preferences and gives it to chat agents in the task prompt
> - The benefit is that schedules and dates from chat match the person's
local time

## Linked Issues or Issue Description

**What happened**
A person in America/Chicago wrote in chat: "Every weekday at nine in the
morning, summarize Launch for me." The agent saved the routine schedule
`0 9 * * 1-5` with the UTC time zone. It replied that it used UTC
because the person's time zone is not recorded.

**Expected behavior**
The agent knows the person's time zone. It saves one enabled `0 9 * *
1-5` schedule with the time zone America/Chicago. Its reply names the
zone.

**Steps to reproduce**
1. Open the board in a browser with the time zone America/Chicago.
2. Open a chat with an agent.
3. Send "Every weekday at nine in the morning, summarize Launch for me."
4. Look at the routine trigger that the agent creates. Its time zone is
UTC.

## What Changed

- Database: add the nullable `time_zone` column to
`user_workspace_preferences` (migration `0468`). This is an
instance-wide personal preference by user, as AGENTS.md section 5
permits.
- Shared: `WorkspacePreference` has `timeZone`.
`updateWorkspacePreferenceSchema` accepts an optional IANA `timeZone`
(max 64 characters, checked with `Intl.DateTimeFormat`).
`coordinatorWorkspaceEnabled` is now optional. The request must contain
at least one field.
- Server: the preference service returns `timeZone` and updates only the
fields in the request. The activity log entry stays the same. The
OpenAPI entry uses the shared schema.
- UI: a new `useBrowserTimeZoneSync` hook in `Layout` saves the browser
time zone one time when it is different from the saved zone. It is
silent and best-effort. It uses the selected company as the audit
context.
- Prompt: for a conversation (any adapter), the task context reads the
chat owner's saved zone. Recovery keeps the zone that the first turn
admitted. The chat block adds one line only when the zone is known:
`The person's time zone is America/Chicago (from their browser). Read
times and dates they give, such as "nine in the morning" or "today", in
that zone, and save schedules such as routine triggers with that
timezone; say the zone when you report a schedule.`

## Verification

- `cd server && npx vitest run
src/__tests__/workspace-preferences.test.ts
src/__tests__/workspace-preferences-routes.test.ts
src/services/heartbeat/task-markdown.test.ts
src/__tests__/heartbeat-task-context.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts`
- `cd ui && npx vitest run src/hooks/useBrowserTimeZoneSync.test.tsx
src/hooks/useCoordinatorWorkspaceEnabled.test.tsx
src/components/Layout.test.tsx`
- Red/green: without the task-context and task-markdown changes, the two
new prompt tests fail. With the changes, they pass.
- `pnpm check:token-gates` is clean. `tsc --noEmit` passes for
`packages/shared` and `packages/db`.
- Manual: open the board in a browser with a non-UTC zone. Then ask a
chat agent to create a weekday routine at nine in the morning. The
trigger uses the browser zone.

## Risks

- Low risk. The migration adds one nullable column.
- The PUT body is now more flexible: `coordinatorWorkspaceEnabled` is
optional. An empty body still gets a 400 response.
- The prompt grows by one line only for conversations with a known zone.
- The saved zone changes when the person uses a browser in a different
zone. The latest browser wins.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:11:57 -06:00
DottaandClaude Opus 5.5 8eabb13ebd fix(recovery): retry a system-woken run once when it lost its runner before any turn (#20873)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can lose its Runner process before
its same-run retry reattaches. The run then fails with
`runner_remote_process_identity_unavailable`
> - After saved-work recovery, the server requests one Retry by itself,
but only for user wakes. System wakes (monitors, automatic assignments)
rest on the visible Retry
> - On staging, a monitor wake's run lost its Runner before it started
any provider turn. Nothing it did could be repeated, but the task stayed
blocked until a person clicked Retry
> - This pull request lets that one case continue automatically, once,
under the run's responsible user
> - The benefit is that a system-woken task whose run never reached a
provider turn continues without a person

## Linked Issues or Issue Description

**What happened?**
On staging, a monitor wake (`issue_monitor_due`) started a remote native
run. Its `session.snapshot` command timed out before any turn. The
same-run retry found the Runner process gone (`process_not_running`,
lifecycle `ready`) and the run failed with
`runner_remote_process_identity_unavailable`. The run had no turn event
and no tool event. Saved-work recovery recovered the files, but
`requestInterruptedNativeTurnRetry` refused with `not_user_request`, so
the task stayed blocked on a manual Retry.

**What did you expect to happen?**
When the run proves that it did no provider turn, the server requests
the Retry once by itself, as it does for user wakes.

**Steps to reproduce**
1. A system wake starts a remote native run.
2. The run fails before any turn event, and the same-run retry finds the
Runner gone.
3. Saved-work recovery completes. No Retry is requested; the task waits
for a person.

Refs #20872, #20853

## What Changed

The rule that changed, exactly: in `requestInterruptedNativeTurnRetry`
(`server/src/services/heartbeat/restart-interrupted-turn.ts`), a run
whose wake was not requested by a user used to stop with
`not_user_request` in all cases. Now it requests the task Retry only
when all of these are true:

- the run failed with one of the existing reattachment-loss codes
(unchanged gate);
- a positive durable record proves that no turn was ever issued: the
run's last `native.remote_runner_recovery_failed` event carries
`remoteRunnerRecovery` with `stage: process_verification`, `reason:
process_not_running` and `providerTurnIssued: false`;
- the run also recorded no turn event and no tool event;
- the run has a `responsibleUserId`; the Retry runs under that user;
- the run is not itself a `task_retry` run;
- the task is still open and assigned to the same agent (unchanged
gate).

An absence of run events is never enough: a turn can reach the provider
before its events are stored (Greptile, first review).

- New `withRemoteRunnerTurnProof` (`native-session-executor.ts`): when a
same-run reattachment is refused because the sandbox proved the Runner
process gone (exit 45, not an unreachable sandbox), the executor reads
the run's controller journal. The controller records each turn command
there before it sends it. Only when
`runnerdStateProvesNoRemoteProviderTurn` holds (just
`run.prepare`/`session.open` and maintenance commands such as
`session.snapshot`, and only session-open events) does it add
`providerTurnIssued: false` to the diagnostic. This is the same proof
that restart recovery already uses for `bootstrap_incomplete`.
`RemoteRunnerRecoveryDiagnostic` keeps the field only as `false`.
- Which staging shapes this covers: a run whose journal holds no turn
command. Whether c14bab19 (the `session.snapshot` timeout) qualifies
depends on its controller journal, which staging does not expose; if its
journal held a `turn.start`, it stays on the manual Retry floor.
32376b3a (`turn.submitted`, then a `turn.start` timeout) always stays on
the floor.
- Everything else about system wakes is unchanged: without the journal
proof, with any turn or tool event, or without a responsible user, a
system wake still rests on the visible Retry.
- Bounded: the Retry uses the existing idempotency key
`restart-interrupted-turn:<runId>`, so it is requested at most once per
run. Its successor is a user-attributed `task_retry` run, so if that run
also loses its Runner, the existing `automatic_retry_used` rule leaves
it on the visible Retry. No loop is possible.
- Attribution: `requestTaskRetry` records `issue.retry_requested` with
actor `restart-recovery` on behalf of the run's responsible user.
Admission keeps every gate (saved-work recovery, stop proof, approvals,
pause, budget, ownership).
- Tests in
`server/src/services/workspace-recovery-retry-copyback.test.ts`: a
positive case (monitor wake, Runner proven gone, journal proof),
including the second-loss `automatic_retry_used` check. Negative cases:
no run events but no journal proof (the Greptile shape), a settled turn,
a tool event, no responsible user.
- New
`server/src/services/native-runtime/remote-runner-turn-proof.test.ts`:
the proof is recorded only for a journal of session commands; not for a
`turn.start` command, a committed turn event, an unreadable journal, or
an unreachable sandbox (`probe_transport_failed`).
- `doc/execution-semantics.md`: describes the rule.

## Verification

- `cd server && pnpm exec vitest run
src/services/workspace-recovery-retry-copyback.test.ts`: 83 passed.
- Red on master (product hunk removed): the positive case fails with
`expected { requested: false, … } to match object { requested: true }`.
- Red on the first head (a244ad3b): the "no run events but no journal
proof" case is requested automatically; CI's `run-completion.test.ts`
("replays saved user intent … system_wake") failed for the same reason.
Both pass now.
- `pnpm exec vitest run
src/services/native-runtime/remote-runner-turn-proof.test.ts`: 4 passed.
`native-session-executor.test.ts`, `run-completion.test.ts`,
`runtime-execution.test.ts`: 1571 tests passed (one local `afterAll`
hook timeout under load).
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.

## Risks

- Low to medium. A system-woken task can now start one new run without a
person. This happens only when the controller journal proves that no
turn command was ever issued, so no provider work or tool side effect
from that run can be repeated. The Retry starts a fresh provider
session, like a manual Retry.
- The Retry runs with the run's responsible user's authority. That user
already authorized the failed run. If the user lost access,
`requestTaskRetry` refuses and the visible Retry remains.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:22:51 -06:00
DottaandClaude Opus 5.5 566a97b882 fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a reused sandbox keep the previous run's warm Runner,
and a new run must retire that owner before it starts
> - #20844 retires the owner when Paperclip can prove it stopped. Some
owners can never be proven stopped: the controller journal drain
outlives the close grace, the Runner never confirms suspension, or a
parked question leaves a tool input unanswered
> - Then the new run failed before any provider work, and the task
stayed blocked with no automatic continuation
> - The general answer is to leave the old Runner on its sandbox and
start the next attempt on a fresh sandbox restored from the saved
workspace
> - This pull request adds that fallback for every path that retires a
predecessor, and settles a parked question's tool input once its Runner
is gone
> - The benefit is that an unprovable predecessor no longer fails a task
that did no work

## Linked Issues or Issue Description

**What happened?**
An automatic wake started a new run 30 to 60 seconds after a long
successful run on the same task, on the same reused sandbox. The new run
had to retire the previous run's warm session first. The retirement
failed and the run failed before any provider work, with one of:
- `Controller journal drain timed out; preserve the session before
replacing its writer.` (during instruction preparation, before runtime
selection, or in the executor);
- `provider_transport_failed: runner did not durably suspend before
checkpoint` (during instruction preparation);
- `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered` (the previous run had
parked on a question).
The tasks rested `blocked` with a reconciliation hold or with nothing
scheduled. One bounded retry reused the same sandbox and ended
`execution_finalization_deadline_exceeded`, then its recovery refused
with `allocation_ownership_ambiguous`.

**Expected behavior**
A run that did no provider work does not fail the task because its
predecessor cannot be retired. The next attempt drops the reused sandbox
and starts on a fresh one restored from the saved workspace. Unsaved
work is never dropped: when the workspace or the session backup is not
durably saved, the sandbox is kept and today's behavior applies.

**Steps to reproduce**
1. Complete a warm native run on a Daytona sandbox.
2. Wake the task again so that the next run must retire the warm owner
(configuration change, new session, changed agent files).
3. Make the owner's close fail (journal drain timeout, unconfirmed
suspension).
4. The new run fails before provider work and the task is not continued.

Refs #20844, #18373.

## What Changed

- `native-runtime/predecessor-retirement.ts`: a typed pre-provider error
`NativePredecessorRetirementUnprovenError`
(`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse`
lease marker.
- Native executor: every path that retires a predecessor for a new run
(instruction-copy claim, instruction-directory reservation, session
reset, configuration change) throws the typed error when the close
fails. The owner stays registered for its own sandbox, so no run on that
sandbox launches over it. A run on a different sandbox forgets that
owner instead of replaying its cached rejected close. The executor does
not settle this error as a provider failure.
- A prior-owner fence that refuses a terminal owner on a resumed sandbox
(`prior_owner_terminal_state_indeterminate`) also throws the typed
error.
- Retirement of a finished remote owner accepts an unanswered tool input
when the owner's committed result is a governed-wait park
(`response_wake` on a task interaction) and its Runner's sandbox lease
ended elsewhere.
- Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the
typed error with no execution-start evidence and no result, mark this
run's reusable lease on that sandbox `replaceBeforeReuse` and log
`environment.lease_replacement_requested`. After the run ends, `failRun`
hands the prepared workspace on only through the shared unchanged-files
proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any
runner command refuses it). With the proof it records
`predecessorRetirementFallback`, closes the run's coordinator without a
result and re-queues the run in the uncharged `interrupted_preparation`
lane (native and legacy runtime modes); a run that answered a task
interaction keeps its own wake in that uncharged lane, so the retry
still carries the answer. Without the proof the workspace stays dirty,
heartbeat runs today's failed-workspace preservation and no retry is
scheduled.
- Environment runtime: a marked lease is replaced instead of resumed,
under the same rule as a failed resume
(`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a
committed save, verified harness backup). The old lease goes to
`pending_cleanup` for the sweep. Otherwise the sandbox is resumed as
before.
- Chaos: session fault SR1U, the SR1 variant where the retained owner
cannot be retired. The server gets a test-only fault
(`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when
`NODE_ENV=test`). The oracle allows only the fallback failure and
requires the succeeding follow-up on a different sandbox.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README.

## Verification

- RED on master before this change (with #20844), then GREEN:
- `native-session-executor.test.ts -t "predecessor cannot be retired"`:
base fails with `Error: Controller journal drain timed out…` instead of
the typed error.
- `native-session-executor.test.ts -t "unretired predecessor"`
(instruction preparation): same base failure.
- `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base
fails with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered`.
- New contract tests: `heartbeat-native-predispatch-retry.test.ts`
(claimed native run: lease marked, activity logged, uncharged
`interrupted_preparation` retry, no recovery action, task not blocked,
retry succeeds) and
`environment-runtime-reusable-resume-replacement.test.ts` (a marked
lease is replaced without a resume; with unsaved work or no verified
backup it resumes as before),
`remote-workspace-unchanged-release.test.ts` (an untouched workspace is
handed on; a setup-written one stays dirty) and `retries.test.ts` (an
answered interaction re-queued in the uncharged lane stays a
resolved-interaction continuation with no failure retry spent).
- Existing suites pass: the native executor, recovered-owner follow-up,
instruction working copy, run completion, run failure cause and resume
replacement test files. `pnpm exec vitest run --config
tests/restart-chaos/vitest.config.ts question-handoff scheduler config`
passes. Server and chaos harness `tsc --noEmit` pass.
- Not run: a Daytona SR1U wave. It needs a runner image that matches
this checkout.

## Risks

- A sandbox marked for replacement is resumed when it cannot be replaced
safely. That run can meet the same unretired owner again, fail again in
the bounded lane, and then rest with Retry, as today.
- The replacement session restores the latest verified harness backup.
When the old Runner's final checkpoint was never saved, a same-session
continuation resumes from that backup.
- An unanswered tool input is accepted only for a committed
governed-wait park whose Runner's sandbox lease ended on another
sandbox.
- Previous-release story: the marker and the run record are additive
JSON. An older server ignores the marker and resumes the sandbox as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:10:24 -06:00
DottaandClaude Opus 5.5 e0e6462a73 fix(heartbeat): keep an agent's explicit resume comment that a stale continuation dropped (#20856)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents coordinate on tasks through comments. A comment posted with
`resume: true` is an explicit request for more work; on a finished task
it reopens the task, for a person or for another agent.
> - A comment that arrives while the assignee's run is still settling
queues its own run. If the settling run then closes the task, that
queued run is cancelled before dispatch as a stale continuation
(`continuation_task_ownership_changed`).
> - For a person's comment, the outbox saves the unread message first
and the release drain reopens the task (chaos s9401/t72). For another
agent's explicit `resume: true` comment, nothing was saved. The request
was dropped and the task stayed `done`.
> - This pull request saves that agent comment too, with its explicit
resume intent. The existing release-drain rule for an agent's explicit
resume reopens the task and delivers the comment once.
> - The benefit is that a parent agent's "do this next turn" request is
not lost to a timing race. An agent comment without `resume: true` stays
communicative, as before.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, a parent agent posted a comment with
`resume: true` on its child's task: "a changed requirement for your NEXT
turn". The child's provider turn had finished 90 s earlier, but its run
was still settling (workspace copy-back). The comment queued a new run.
40 s later the settling run applied its `done` decision. The queued run
was then cancelled with `continuation_task_ownership_changed`. The task
stayed `done` and no run ever read the comment. The cancelled run itself
is not shown as a failure (the task is terminal, so task chat hides it,
and Sentry skips expected cancellations), so nothing told anyone the
request was lost.

**Steps to reproduce**
1. Run an agent on a task. While its run is still settling, post a
comment from another agent with `resume: true`.
2. Let the run close the task (`done`).
3. The comment's queued run is cancelled as a stale continuation. The
task is not reopened and the comment is never delivered.

**Expected behavior**
The task reopens for the explicit resume request, and the comment is
delivered exactly once, as for a person's comment.

## What Changed

- `deferUndeliveredCommentsOfStaleRun` finds, per owed comment, the
comment's own `issue.comment_added` activity record: written by its
author (another agent, not the assignee) with `resumeIntent: true`.
Merged runs, whose wake is rewritten as comments join, therefore keep
each comment's own intent.
- `saveUndeliveredUserComments` also admits those comments
(`explicitResumeCommentIds`). When only such comments remain, the
deferred receipt records the agent as requester and carries
`resumeIntent`/`followUpRequested` in its payload and wake context. The
release drain's existing rule for an agent's explicit resume on a `done`
task then reopens and delivers it. Person-comment behaviour is
unchanged.
- `doc/run-log-events.md` describes the rule.

## Verification

- RED on `origin/master` (`1d67e309e0`): `pnpm --filter
@paperclipai/server exec vitest run
src/__tests__/heartbeat-held-comment-wake.test.ts -t "explicit resume
comment"` fails: `expected [] to have a length of 1 but got +0` (the
comment is never delivered after the stale cancellation).
- GREEN: the whole file passes (29/29), including two-requester
merged-run tests in both orders, the new negative test (`does not reopen
a finished task for another agent's comment without resume intent`) and
the existing person-comment and exactly-once tests.
- Related suites `ownership-reconciliation-report`,
`execution-continuation`, `chat-wake-conversation-batch` plus this file:
118/118 tests pass locally. One `afterAll` cleanup hook in
`ownership-reconciliation-report` timed out under local machine load; CI
runs the suite fresh.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- Not done: no restart-chaos cell. The harness posts follow-ups as the
board user, which s9401/t72 already covers; an agent-authored `resume:
true` comment needs a two-agent workload the harness does not have.

## Risks

- Low risk. Only a comment whose own wake was an agent's explicit
`resume: true` qualifies, only on a task an agent run closed, and only
when no run read the comment. The assignee's own comments never qualify.
- The ownership reconciler's durable backstop
(`checkStaleDroppedComments`) still covers person comments only. If the
fast path fails, an agent's resume comment is not recovered later.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:57:47 -06:00
DottaandClaude Opus 5.5 57f1afdf19 fix(daytona): retry a sandbox create that timed out before any agent work (#20828)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On a Daytona sandbox environment, every new task run first creates a
fresh sandbox
> - When one Daytona create did not start, the plugin waited the full
300 s acquisition budget
> - The run then failed `setup_failed` with `lease_acquire_failed`. No
agent work had started, but nothing retried it
> - On staging this hit two agents in 15 minutes. Each attempt used
about 5 minutes, and the task stopped until a person pressed Retry. The
manual Retry then got a sandbox in 28 s
> - This pull request ends a stuck create early, adopts a create that
started late, and routes the timeout to the bounded bootstrap retry that
already exists for transient provider refusals
> - The benefit is that a stuck create costs about 90 s and recovers by
itself. The task shows Retry only after the bounded attempts are used up

## Linked Issues or Issue Description

Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait
out a Daytona state change), Refs #20609 (force a stuck stop).

**What happened?**
A run on a Daytona sandbox environment failed after 307 s with `Failed
to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona
lease acquisition timed out; allocation cleanup is pending`. The run
recorded `errorCode: setup_failed`, `failureCause:
lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap",
providerWorkStarted: false }`. The task became blocked with Retry. No
automatic retry was scheduled.

**Steps to reproduce**
1. Use a Daytona sandbox environment with the default `timeoutMs`
(300,000).
2. Start a run while Daytona keeps the new sandbox in `creating`.
3. See that the run waits 300 s, fails `setup_failed`, and is not
retried.

**Expected behavior**
A create that has not started before provider work must not cost 5
minutes. The run must retry by itself within bounds. A late sandbox that
has started must be used. A sandbox that was abandoned must be removed.

**Root cause**
- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`
(`onEnvironmentAcquireLease`): the only bound on the create step was the
whole acquisition budget (`timeoutMs`, default 300 s).
- `server/src/services/environment-provider-unavailable-error.ts`
(`readTransientEnvironmentProviderFailure`): only an HTTP status or
errno on the create call counted as transient. The timeout envelope
carries neither, so the run fell through to generic `setup_failed`,
which `isTransientWorkspacePreparationCode` does not retry.

## What Changed

- Daytona: one fresh acquisition is now capped at 180 s. A create that
has not returned after 90 s is looked up once by its creation name. A
sandbox in `started` is adopted, and setup continues. A sandbox in
`pending_build`, `building_snapshot`, or `pulling_snapshot` keeps
waiting inside the cap. Any other state ends the attempt with the
existing cleanup envelope and an acquisition diagnostic whose elapsed
time reached the expired budget.
- Daytona: a late SDK failure no longer runs the compensating delete for
a sandbox that was adopted.
- Host: `readTransientEnvironmentProviderFailure` reads the validated
acquisition diagnostic. An acquisition whose first failure was its own
deadline is transient. The run fails with
`environment_provider_unavailable`, category `timeout`, and uses the
bounded bootstrap retry (30 s, then 60 s). An earlier setup failure
stays `setup_failed`. Message text is never parsed.
- Host: the task reason for this case is plain: "The sandbox for ... did
not finish starting in time. No agent work had started. Paperclip
retries automatically ...".
- Docs: `doc/execution-semantics.md` and
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`.

## Verification

Red before the fix, green after:
- `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh
sandbox create that timed out through bounded bootstrap recovery" uses
the staging error shape through the orchestrator. Red: `errorCode` was
`setup_failed`. Green: `environment_provider_unavailable`, category
`timeout`, bounded retry scheduled. The `started` and `early_failure`
cases stay `setup_failed` with no retry.
- `server/src/services/environment-provider-unavailable-error.test.ts`:
deadline diagnostic is transient (red, then green). An early failure is
not. The task message is plain.
- `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends
a create still pending at 90 seconds" (red: still pending at 90 s;
green) and "adopts a create that started by 90 seconds" (red; green, and
a late SDK rejection does not delete it). "keeps waiting for a create
that is still preparing its snapshot" is a safety case.
- Existing tests that encoded the 300 s acquisition budget now use the
180 s cap.

Commands run locally:
- `npx vitest run src/plugin.test.ts` (Daytona): 414 passed.
- `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts
src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The
run-completion file reports an `afterAll` embedded-Postgres teardown
timeout. The same timeout occurs on unchanged `origin/master`, so it is
a local platform fault.
- Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass.
- `pnpm verify`: asked for broader CI coverage, because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

Upgrade safety: nothing new is persisted. A task already blocked by this
failure keeps `canRetry: true`. Its Retry starts a new run with a new
creation record, and that run uses the new path. On staging, the manual
Retry of the affected task got a fresh lease in 28 s. The creation
record of the earlier attempt stays with the creation reconciler.

Not verified: a live Daytona create that stays in `creating`. That
cannot be forced on demand.

## Risks

- A legitimately slow create that stays in `creating` or `starting` for
more than 90 s now ends the attempt and retries. Snapshot preparation
states still wait inside the 180 s cap. Staging creates normally take
about 20 to 30 s.
- An abandoned create is removed by the creation reconciler. It can stay
alive until that reconciler settles the record (the existing settle
window). This PR does not change that window.
- Other sandbox providers get the host classification only when they
send the same acquisition diagnostic.

## Model Used

Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, browser for read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 10:04:17 -06:00
DottaandClaude 889b33da72 fix(native): apply a saved accepted result on Retry instead of a new turn (#20819)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run on a remote sandbox can finish with an accepted result,
and then fail to save its workspace. Saved-work recovery saves the files
again from the retained sandbox, with no provider turn.
> - #20758 made a save that finds the sandbox disk full stop at once as
`sandbox_disk_full` and hand the task to the board with a Retry.
> - When the user frees space and presses Retry, the save now succeeds.
But the Retry then started a new agent turn (`strategy: continue`). That
turn repeats work that already finished and spends provider budget
again. The accepted result was not applied (`replacementDenied:
explicit_user_continuation`).
> - The decision for this case: when a durable record proves that the
run's result was already accepted, Retry applies that saved result and
starts no new turn.
> - This pull request makes Retry admission wait for the run's
saved-work holds and then applies the accepted result through the
existing preserved-result finalization.
> - The benefit is that a Retry after a failed save finishes the task
from the work that was already done, with no repeated provider work or
spend.

## Linked Issues or Issue Description

Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry),
Refs #20810 (chaos fault DF1 for a full disk on the final workspace
save)

**What happened?**
A native run had an accepted, succeeded result. Its workspace save
failed because the sandbox disk was full, and the task rested at
`sandbox_disk_full` with a Retry. The user freed space and pressed
Retry. The save succeeded, and then Retry admission started a new run
with `strategy: continue`. The accepted result stayed unapplied (the
finalization coordinator stayed in `terminal_failure`).

Two paths caused this:
1. Retry admission let the Retry through as soon as the saved files were
verified. That can happen before preserved-result finalization runs: the
recovery worker dispatches the Retry right after one hold settles, while
the hold that Retry added for the same run is still scheduled. On master
this is deterministic in that order, and a race otherwise.
2. Preserved-result finalization reads the newest saved-work hold of the
run. While that hold was still scheduled, it declined, and nothing held
the Retry back.

**Expected behavior**
After the save succeeds, and a durable record proves that the run's
result was already accepted, Retry applies that result. The run and task
finish from it. No new `turn.start` and no new provider spend. If the
save fails again with a full disk, the task keeps the
`sandbox_disk_full` rest and a working Retry. If there is no proven
accepted result, Retry keeps the previous behaviour and starts one run.

**Steps to reproduce**
1. Give a native run an accepted, succeeded result whose copy-back
failed (`native_workspace_sync_out_retry_exhausted`).
2. Make the saved-work recovery save fail with
`remote_workspace_sandbox_disk_full`. The hold goes to the board.
3. Free space. Press Retry. Let recovery save the files, then dispatch
the Retry before preserved-result finalization runs.
4. On master, a new run starts with `previousRunId` set to the source
run, and the accepted result is not applied.

## What Changed

- `preserved-result-finalization.ts`: the eligibility checks move into
one read (`readPreservedResult`). They are unchanged: an accepted,
succeeded result of this run, a completed saved-work recovery with a
verified receipt, no Stop, no newer run, no current execution, and no
applied generation. A new read-only
`canResumePreservedResultFinalization` uses the same read without locks.
`resumePreservedResultFinalization` keeps the finalizer's lock order
(coordinator first) and takes the issue and run rows with NOWAIT, so it
cannot deadlock with the finalizer, owner-stop confirmation or Retry
dispatch; a busy row is retried briefly, and the last attempt waits for
its locks. It also writes an activity entry,
`issue.saved_result_finalization_resumed`, when it applies a result.
- `explicit-native-continuation.ts`: Retry admission holds an accepted
Retry (`evaluate`) of a run whose saved files are verified and whose
accepted result is still held, with the wait `finalizing_saved_result`.
It holds while a saved-work hold of that run is still `scheduled`,
`running` or `awaiting_evidence` and the result is otherwise applicable
(no Stop, newer run, other owner or current execution), or once
`canResumePreservedResultFinalization` is true. In every other case
admission is unchanged.
- `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if
admission held the Retry with `finalizing_saved_result`, it calls
`resumePreservedResultFinalization`. That coalesces the Retry into the
source run with `strategy: finalize_result` and re-arms finalization.
This is one key read after admission; the resumption runs only for a
held Retry. The recovery worker already calls the same function after
each saved hold, so either path finishes the work.
- Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry
behaviour (red on master), a dispatch that lands while another hold of
the run is still saving (red on master), a Retry while the disk is still
full, and three negative cases that keep one new run.
`workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and
`explicit-native-continuation.test.ts` (copy-back + `task_retry`)
encoded the old "new run" result for the same situation. They now expect
the saved result to be applied. A fixture helper `recoverRunHolds` runs
the recovery steps for every hold of a run, as the worker does.
- `ui/src/components/WorkspaceExportRecovery.tsx`: at the
`sandbox_disk_full` rest, the retained-file panel no longer shows its
own "Recover saved files" button and no longer hides the task's Retry.
Before, the API said `canRetry: true` and the note said "Retry saves the
files again", but the board showed only "Recover saved files" (seen in
the DF1 chaos wave). The panel now says to free space and use Retry. A
new UI test covers it. `pnpm check:token-gates` is clean.
- `doc/execution-semantics.md`: documents the Retry behaviour and its
conditions.

## Verification

Red/green (`cd server && npx vitest run
src/services/workspace-recovery-disk-full.test.ts`):

- On master source with the new tests: 3 failed, 3 passed (the 3
negative cases pass on master, as they should). Each failure shows a
second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply
equal [ Array(1) ]`.
- On this branch: 7 passed. A fourth negative case (an operator Stop
while a hold is still settling keeps one new run) was added after
review; it also passes on master.
- Guard removal 1: with the `canResumePreservedResultFinalization` check
replaced by `true`, the negative test "an operator Stop recorded on the
run" fails (`expected [] to have a length of 1 but got +0`: the Retry
waits forever instead of starting its one run).
- Guard removal 2: with the "hold still saving" check removed, the test
"holds a Retry dispatched while another saved-work hold of the run is
still settling" fails (a second run starts).

Related suites pass on this branch: `workspace-recovery-disk-full`,
`workspace-recovery-preparation-gates`, `workspace-recovery-admission`,
`native-exhausted-finalization-retry`, `explicit-native-continuation`
(641 tests).

On the first revision, `task-retry`,
`workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`,
`preserved-native-response-wait.integration`, `deferred-execution-wait`
and `deferred-execution-receipt-work` passed (317 tests). Other retry
and recovery suites were not run locally; required CI `general_tests`
covers them. After the review fixes, the five related suites above
passed again (642 tests).

`npx tsc --noEmit -p server/tsconfig.json` is clean.

`pnpm verify --explain` selects a broad plan (exit 2) because
`doc/execution-semantics.md` is a global CI input. Required CI supplies
that coverage.

## Risks

- Behaviour change: an accepted Retry of a terminal run with an
accepted, succeeded result and verified saved files now applies that
result and starts no run. Before, it started one new run. This is the
intended change. It also covers the s12 and RC-50b recovery-handoff
cases, which used to start a new run.
- A held Retry waits while any saved-work hold of the run is still
saving. The recovery worker drives those holds to `complete` or
`exhausted`. When a hold ends `exhausted` or the records do not prove
the result, the wait ends and the Retry starts one run, as before. A
hold that stays in `awaiting_evidence` keeps the Retry waiting with
"Paperclip is finishing the saved result."
- Upgrade-safe: no persisted shape changes. The checks read existing
records (`native_run_results`, `native_run_finalizations`, the recovery
action evidence, workspace receipts). A record written by an older
server that lacks any required fact keeps the old behaviour. An older
server that reads the new state sees a coalesced Retry and a re-armed
finalization, which it already handles.
- No new work before `turn.start` on the normal path. Admission adds the
eligibility read only when it has already proven an accepted result and
verified saved files for a Retry of that run. `resumeTaskRetry` adds one
key read after admission.
- Lock order: `resumePreservedResultFinalization` keeps the finalizer's
order (coordinator, then issue, then run). Because it can now run beside
Retry dispatch (which locks the issue first), it takes the issue and run
rows with NOWAIT. A busy row ends the attempt, and it retries up to 9
times at 100 ms. The 10th attempt waits for its locks, as this
resumption always did, so a recovered result is never left unapplied.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:03:53 -06:00
Nicky LeachandClaude Opus 5.5 369e238295 fix(projects): make workspace storage the project's only placement choice (#20835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A repository project gets its working files from core staging or
from a workspace storage plugin
> - #20459 reported that an installed storage plugin attached every new
repository project, so sandbox agents could not use those projects
> - #20587 fixed creation with an **Available in** choice that selects
storage
> - #20651 then saved an environment on every new project and used it
before the agent default to place runs. That solved a problem that did
not exist: the project needs storage, not a run environment
> - Creation saves the first environment of the chosen group, so a
project silently sends every agent's runs to that environment. Project
Configuration also shows two controls (Environment and Workspace
storage) for one decision
> - This pull request removes the project run environment and makes
storage the only placement choice a project owns. Storage is fixed at
creation
> - The benefit is that runs follow the agent again, and project
settings show one clear, read-only fact about storage

## Linked Issues or Issue Description

Refs #20459. Follow-up to #20587 and #20651.

Refs #11644. That issue asks for a project-level run environment
override. This pull request removes the override that #20651 added for
it, because the override is not necessary to fix #20459. #11644 stays a
separate feature request.

Related: #18176 (open) repairs placement through the legacy
workspace-policy field. This pull request does not restore project
placement, so it conflicts with the direction of #18176.

## What Changed

- **Remove the project run environment.** Migration `0466` drops
`projects.environment_id`. Run placement no longer reads the project.
Runs use the agent default, then the instance default, then local
execution. The low-trust task sandbox and the enforced sandbox policies
keep their precedence.
- **Keep the #20651 recovery and binding guards.** A recovered run still
uses its admitted environment lease (source `run`). Company-binding
checks before admission and during lease creation stay.
- **Keep the creation choice.** **Available in** in New Project still
selects storage. It still shows only when the environments need more
than one storage owner. That can only occur when a storage plugin is
installed. `POST /projects` with `environmentId` selects the storage of
that environment and does not save the environment.
- **Fix storage at creation.** `PATCH /projects/:id` rejects any storage
change with `409 project_storage_locked`, from core to a plugin and from
a plugin to core. Before, a core project could move to a plugin.
- **Drop legacy writes.** Older clients can still send
`executionWorkspacePolicy.environmentId`. The service removes it on
create and update.
- **Simplify Project Configuration.** Remove the Environment and
Workspace storage selectors. Show an advisory **Available in** list of
the environments that can use the project's storage. Show it when a
storage plugin is installed or when the project uses one. When the
project's plugin is not installed, show **Storage plugin unavailable**.
Without a storage plugin, show nothing.
- **Issue workspace card.** Remove the "project default" environment
label.
- **Shared contract.** Remove `Project.environmentId`, the
`environmentId` project field, and `expectedStorageProviderPluginId`
from the update schema. The create schema keeps `environmentId` and
`expectedStorageProviderPluginId` from #20587.
- **Docs and maps.** Update `doc/project-environment-compatibility.md`,
`docs/api/goals-and-projects.md`, and the project feature and
agent-access maps.
- **SDK source pins.** Refresh
`tests/runner-e2e/source-sdk-public-inputs.json` for the two shared
files. The shared file blobs return to their values before #20651. The
refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`:
`master` pins an older lockfile blob than the one it contains.

## Verification

- Targeted server suites pass (227 tests): `project-environments`,
`project-workspace-provider.integration`,
`heartbeat/environment-selection`, `execution-workspace-policy`,
`environment-selection-route-guards`, `environment-run-orchestrator`,
`openapi-routes`.
- Related server suites pass (349 tests): `environment-service`,
`environment-routes`, `plugin-workspaces.integration`, `issues-service`.
- Targeted UI suites pass: `ProjectProperties.concurrency`,
`IssueWorkspaceCard`, `NewProjectDialog`,
`NewProjectDialog.managed-sandbox`, `ProjectDetail`.
- New tests:
- Creation saves storage, not an environment. Core and plugin projects
both reject a storage change with `project_storage_locked`.
  - A legacy policy `environmentId` is dropped on create and update.
- A new run follows the agent default for any project. A recovered run
keeps its admitted lease after the agent default changes.
- Project Configuration has no storage or environment selector. The note
is hidden without a plugin. It lists the core or plugin environments for
the project's storage. It shows **Storage plugin unavailable** for a
removed plugin.
- `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared
typecheck` pass.
- `pnpm --filter @paperclipai/db check:migrations`, `pnpm
check:token-gates`, `node scripts/check-maps.mjs`, and `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass.
- Not run locally: the full `pnpm test:run`, `pnpm build`, and browser
suites. CI supplies that coverage.
- Manual check: open Project Configuration on an instance with a storage
plugin. Confirm that there is no Environment or Workspace storage
selector, and that **Available in** lists the environments.

## Risks

- **Migration.** `0466` drops `projects.environment_id` and its foreign
key. The migration deletes the saved project environments. Apply it
before you serve the new revision. An old server revision that reads
`projects` fails until it stops. This is the same pattern as earlier
drop-column migrations (`0229`, `0289`).
- **Behavior change for existing projects.** Projects created after
#20651 shipped have a saved environment that overrides the agent. After
this change, their runs follow the agent default again. This is the
intended fix. Before #20651, a project environment selection was not
applied to runs (#11644), so no older behavior is lost.
- **No path from core to a plugin.** A core project, including one
created before a plugin was installed, cannot move to the plugin. To use
plugin storage, create a new project.
- **API.** A `PATCH` that sends `environmentId` or
`expectedStorageProviderPluginId` no longer has an effect. The schema
strips these fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with
tool use (shell, file edits, test runs). Extended thinking was
available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 08:00:21 -08:00
DottaandClaude f25fee3532 fix(native): hash runner artifacts and run archive tar off the event loop (#20722)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process prepares native runs, renews every controller
lease on timers, and serves every request
> - On a run path, the server hashed the 31 MB runner binary with
`readFileSync` (more than once per run), hashed provider packs and
harness backups synchronously, and ran `tar` with `execFileSync`
(timeouts up to 60 s, one with no timeout)
> - On a slow disk, each of these calls freezes the whole process. A 103
s stall on a hosted instance lapsed every lease. The last event before
that stall was immediately before a synchronous read and hash of the
runner binary (see #20694 and #20709)
> - This pull request moves those reads, hashes and archive commands off
the event loop. It computes the runner binary digest once per boot
> - The benefit is that a slow disk delays only the run that waits for
it. Turns also start faster, because the runner binary is no longer
re-read for every run

## Linked Issues or Issue Description

Refs #20694, #20709

**What happened?**

On a hosted instance, the server process stopped for 103 s without a
restart. Every controller lease lapsed, and two tasks needed a manual
Retry. The last run event before the gap was immediately before
`matchesControllerRunnerArtifact`, which does
`createHash("sha256").update(readFileSync(controllerRunnerBinary))`.
Four workspace restores were running at the same time, and file I/O was
very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is
not proven. But each synchronous call below blocks the event loop for as
long as the disk is slow.

**Expected behavior**

File hashing and archive commands on a run path must not block the event
loop. A slow disk must delay only the run that waits for it.

**Steps to reproduce**

1. Make a remote runner binary slow to read (the test uses a FIFO whose
writer waits), or make `tar` slow (the test puts a `tar` wrapper that
sleeps first on `PATH`).
2. Call `resolveRemotePaperclipRunnerBinary`,
`stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a
10 ms timer runs.
3. Before this change, the timer stops for the whole slow step (814 ms,
1,764 ms and 3,379 ms). After this change, it keeps firing.

## What Changed

- `server/src/services/native-runtime/off-loop-file-work.ts` (new):
  - `sha256FileStreamed`
- `cachedFileSha256`: a streamed digest, reused while the file's device,
inode, size, mtime and ctime do not change
- `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file
only when the header cannot answer, so the result equals
`runnerBinaryTarget(readFileSync(path))`
  - `sha256DirectoryTreeStreamed`
- `execFileBuffered`: `execFile` with stdin input, `maxBuffer`,
`timeout` and `env`. It rejects on a non-zero exit, a timeout or a
`maxBuffer` overflow, like `execFileSync`.
- `native-session-executor.ts`:
- `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot
reads the binary once, not once or twice for every run.
  - The remote Claude daemon target is read from the binary header.
- The bundled remote runner binary is checked by
`verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a
cached digest.
- Provider-pack upload verification uses the new
`verifyRemoteProviderPackManifest`. It runs the same checks, in the same
order, with the same errors. One generator drives both the sync and the
streamed readers.
- Harness backup build and verify on run paths use streamed digests
(`buildNativeHarnessBackupManifestStreamed`,
`verifyNativeHarnessBackupStreamed`). The digest format does not change.
- `stageRemoteRunnerDirectory` (3 calls),
`assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract
use `execFileBuffered`, with the same arguments, timeouts and buffer
limits. `stageRemoteRunnerFile` uses `readFile`.
- `packages/paperclip-runner/src/live/runner-binary.ts`: the default
daemon lookup (`resolvePackagedRunnerBinary`, used by the local and
remote resolvers) classifies the daemon with `runnerBinaryFileTarget`.
It reads the 64 KiB header, not the whole file.
- `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`:
the release-manifest checks now live in one helper,
`bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and
the server's streamed check both use it.
- `native-codex-runner.ts`:
  - `resolveRemotePaperclipRunnerBinary` reads only the header.
  - The local `runnerDigest` uses `cachedFileSha256`.
  - New `verifiedBundledRemoteRunnerBinary`.

**Ordering and atomicity.** I checked each converted call for code that
relied on the sync call being atomic with respect to other runs:
- **Checkpoint copy-back.** It extracts into a private `mkdtemp`
directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`)
stays synchronous, so no other run can see a half-installed checkpoint.
This function already awaited a remote command before the extract, so
this change adds no new interleaving with the target.
- **Harness backup verify then restore.** The verify was sync, but the
restore after it was already async. The window between them existed
before. One owner serializes the turns of a session, so no other turn
publishes a backup for that session while a turn verifies it.
- **Staging and provider-pack hashing.** These read build-owned,
immutable trees, or the private staging copies of a run.

## Verification

- Red/green: `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/off-loop-file-work.test.ts`
- With `native-session-executor.ts` and `native-codex-runner.ts` from
`origin/master`: 3 failed. The longest timer gaps were 814 ms (runner
binary target read), 1,764 ms (directory staging tar) and 3,379 ms
(checkpoint list, size check and extract).
- With this change: 6 passed. The gaps stay under 400 ms. The same file
also checks the digest cache, the header fallback, and the
`execFileSync`-like errors (exit code, timeout kill,
`ERR_CHILD_PROCESS_STDIO_MAXBUFFER`).
- `native-session-executor.test.ts`:
- The streamed provider-pack verifier returns the same manifest as the
sync reader, and rejects a changed asset tree with the same error.
  - The streamed harness backup manifest equals the sync one.
- The streamed verify agrees with the sync verify, for a valid backup
and for a corrupt one.
- Run together: `native-session-executor`, `off-loop-file-work`,
`native-codex-runner`, `setup-readiness`, `antigravity-connection-probe`
(1,653 passed). `run-completion.test.ts` passed all 153 tests. Its
`afterAll` embedded-Postgres cleanup timed out locally, which is not
related to this change; CI runs it.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.

## Risks

- **Turn start.** This change adds no wait before `turn.start`; it
removes work. The first run of a boot streams the runner binary once.
Later runs reuse the digest after one `stat`.
- **Cache staleness.** The cache keys on device, inode, size, mtime and
ctime. An in-place rewrite changes mtime and ctime, and a replaced file
changes its inode, so a changed binary is hashed again.
- **stderr.** `execFileSync` wrote a child's stderr to the server's
stderr. `execFileBuffered` captures it. Tar warnings no longer appear in
server logs; errors and exit codes are unchanged.
- **Persisted state.** No persisted shape changes. Old saved state and
new code (and the reverse) read the same manifests and digests.
- **Not converted in this PR:** small metadata syscalls (`existsSync`,
`lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the
native state machines. Some of these rely on running synchronously. Also
not converted: the sync harness backup verify inside the sync
restart-authority checks, and the sync stamp verify in
`native-harness-backup-storage.ts`. Both are listed for a follow-up.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use, shell, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (code
comments carry the contract)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 00:27:12 -06:00
DottaandClaude 16024ffa2a fix(native): a same-boot stall never leaves a run waiting on itself (#20721)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep an execution lease. Restart recovery takes over a
run only when it can prove that the recorded controller stopped.
> - A stall (SIGSTOP) longer than the lease, with no restart, makes the
executor lose its own lease. The executor then detaches for "the next
controller" and exits.
> - The next controller is this same process. Recovery asks the PID
probe if the recorded controller is alive. The recorded controller is
this process, so the answer is always "alive".
> - The task then waits forever: `in_progress`, no Retry, an active run,
and an active environment lease.
> - A second gap: after back-to-back stalls mid-tool, the executor
returns and nothing in this process ever looks at the run again. No
terminal state, no recovery and no Retry.
> - This pull request lets recovery recognise its own process. When this
process no longer drives the run and its own lease lapsed, the prior
owner is gone. The existing evidence checks then decide, as for any
stopped controller.
> - It also makes heartbeat schedule that recovery as soon as such an
executor exits.
> - The benefit is that a same-boot stall never needs a manual Retry.
When provider work may have started, the run fails with a working Retry
instead of a wait with no end.

## Linked Issues or Issue Description

Refs #20694 (the same class of fix for legacy `preparing` runs), #20529,
#20691.

**What happened?**
Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault
TM1). The server process was frozen with SIGSTOP for about 84 s while
run `89a3922b` prepared. After SIGCONT, `native.process_start_requested`
was recorded and `native.session.execute` failed after 118 ms: the lease
renewal found the lease lapsed, and the executor detached and exited.
From then on, recovery recorded `native.recovery.transition` "waiting
for safe ownership evidence" with reason `controller_still_alive` every
30 s for more than 20 minutes. The run stayed `running`, the issue
stayed `in_progress`, no Retry was shown, and the environment lease
stayed `active`.

Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode,
T1, TM1). Three stalls of about 90 s came back to back, with less than 1
s between them. Each run was in a tool call. After the second thaw,
`native.session.execute` failed for all three runs (lease lost, executor
detached). After that, no run had any event: no terminal state, no
`native.recovery.transition` and no Retry, for 20 minutes. The
in-sandbox turns kept going. A single 90 s stall (wave s9340) stays
green, because a later recovery signal happens to find the run. With
stacked stalls, that signal can land while the executor is still tearing
down, so it skips the run and sets no deadline.

**Expected behavior**
When this process no longer drives a run that it recorded as its own,
recovery takes the run back. No turn reached the runner here, so the
bootstrap restarts on the same run. When a turn or runner work may have
started, the run fails and shows a working Retry.

**Steps to reproduce**
1. Start a native remote (Daytona) run.
2. Send SIGSTOP to the server during preparation, and hold it longer
than the execution lease.
3. Send SIGCONT. The executor loses its lease, detaches and exits.
4. See that every recovery scan waits with `controller_still_alive`, and
the run never settles.

The new tests in `native-runner-restart-recovery.integration.test.ts`
("same-boot controller whose own lease lapsed") reproduce this at the
recovery layer.

## What Changed

- `claimNativeRestartRecoveries`: when the recorded controller is this
exact process (same boot ID and PID):
- If this process still drives the run (heartbeat execution or native
executor registered), recovery leaves the run to it
(`current_controller_preparing`).
- If neither drives it and the self lease lapsed, the takeover is
allowed (`current_controller_released`). The existing evidence checks
then choose the outcome: bootstrap again on the same run when a durable
record proves no command reached the runner, or fail with a Retry
(`blocked`) when runner or provider work may have started.
- An unexpired self lease still waits. A claim that has not reached its
executor holds it, and it lapses by itself.
- A run that this process handed to the next server (graceful drain
`restart_suspended` record, or the in-process hot-restart handoff
marker) stays with the next server.
- `heartbeat.ts`: when the executor ends with
`NativeControllerDetachedForRestartError`, there was no shutdown or
hot-restart handoff, and this process's own lease lapsed
(`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery
for the run when the executor exits. Recovery then reattaches the remote
runner whose turn is still running, restarts an unstarted bootstrap, or
fails the run with a working Retry.
- `claimNativeRestartRecoveries`: a self-owned lease that is still live,
with no driver here, now returns a `retryAt` at its expiry, so the wait
is bounded.
- `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`.
Heartbeat recovery passes it to the claim.
- `suspendedByCleanDrain` now uses the shared
`handedToNextControllerSinceClaim` helper. Behavior is the same.
- Heartbeat test: an executor detaches after its own lease lapsed, with
no restart. The same process must take the run back. Red before (timed
out, the run stayed `running`), green after.
- Tests: five cases for the same-boot lapsed lease: bootstrap again with
no runner command; fail with a working Retry when a runner identity was
recorded; leave the run to a live in-process executor; leave a
handed-off run to the next server; still wait for a different live
controller.

## Verification

- Red before the fix: the three same-boot cases returned
`awaiting_evidence` / `controller_still_alive`.
- Green after the fix:
- `pnpm --dir server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Daytona wave `w-20261111T050504-s9331` from this branch (same seed and
shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about
1 s between them). All 3 tasks reached `done` with no Retry. Run
`632f0564` (claude) shows the exact s9331 sequence:
`native.session.execute` failed after 216 s, then
`reattach_remote_runner` 0.4 s later (before the third stall), then a
second reattach after it, and the turn completed. Two tasks are graded
FAIL only on the T1 crossing check: their own fault landed after their
tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their
status is `done`, with no Retry.
- `tsc --noEmit -p server` is clean.

## Risks

- A same-process recovery scan can now claim a run that this process
recorded as its own. The guard needs all of these: the self lease
lapsed, no in-process execution of the run, and no handoff to the next
server. A claim that is still on its way to its executor holds a live
lease, so it is not taken twice.
- Upgrade safety: the decision uses only the coordinator row and
in-process state. Old saved rows are judged by the same rules. A new
process never matches an old boot ID, so deploys keep the existing proof
path.
- No new waits or file operations run before `turn.start`.
- There is still no ceiling on `awaiting_evidence` for a genuinely
different live controller (case c). This PR does not change that policy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use
(shell, file edit) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 23:55:10 -06:00
DottaandClaude a46b9ac724 fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one

## Linked Issues or Issue Description

**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.

**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.

**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.

Refs #20666 (other recovery work in the same staging census; not changed
here)

## What Changed

- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.

Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.

## Verification

- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.

## Risks

- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:57:03 -06:00
DottaandClaude 8c0dbffa70 fix(heartbeat): retry a legacy run whose lease this process lost before dispatch (#20694)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat orphan reaper settles legacy (non-native) runs that
lost their controller lease
> - #16355 taught the reaper to retry a legacy run that a restart
stopped while setup was still `preparing`, because revocation fences the
old controller and `preparing` proves no adapter dispatch
> - That proof was applied only when the lapsed lease belonged to a
different boot
> - A stall can make the current process lose its own lease while setup
is still `preparing`, and its executor can end without settling the row
> - The reaper then wrote "retrying once" but recorded no bootstrap
evidence, so the retry was refused and the task was blocked for
reconciliation
> - This pull request applies the same revocation proof to a lease from
the current boot
> - The benefit is that these tasks recover by themselves instead of
waiting for a person to press Retry

## Linked Issues or Issue Description

Refs #16355

**What happened?**

On a hosted instance, one server process stalled for about 100 seconds
(lease renewals were 42–46 s late). Several legacy `paperclip_runner`
runs on a remote sandbox were still in setup (`executionStage:
"preparing"`, no process id). Their controller leases expired. Some
executors settled their own runs as `controller_lease_lost`. Two runs
had no executor left, so the orphan reaper of the same process settled
them 4 minutes later. Each run got `error: "Process lost -- server may
have restarted; retrying once"` and
`processLossDiagnostic.retryEligible: true`, but no `executionRecovery`
evidence. `enqueueProcessLossRetry` then saw a run that needs
reconciliation and scheduled nothing. The next sweep recorded
`legacy_execution_requires_reconciliation` with
`preserve_without_replay_v1`, and both tasks stayed `blocked` until a
person pressed Retry.

**Expected behavior**

Revocation commits the stage. Renewal requires the old boot id, so no
controller can move a revoked run to `dispatching`. A revoked run that
is still `preparing` with no process identity did no provider work,
whichever boot owned the lease. The reaper should record the bootstrap
evidence and use the bounded retry.

**Steps to reproduce**

1. Claim a legacy run in this process: `controllerBootId` is the current
boot, `executionStage: "preparing"`, no process id.
2. Let its controller lease expire with no local executor (for example
after an event-loop or database stall).
3. Run the orphan reaper.
4. Before this change: `process_lost; retrying once`, no
`executionRecovery`, no retry, and a reconciliation hold. After this
change: bootstrap evidence and a `scheduled_retry` successor.

## What Changed

- `server/src/services/heartbeat/recovery.ts`:
`restartInterruptedBeforeDispatch` no longer requires the lease to come
from another boot. The other clauses stay: legacy runtime, a stamped
controller boot id, and `preparing` with no process id or group in the
state committed by the revocation. Locally tracked runs and runs with a
live lease are still skipped before this point.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: two cases
in the hosted shape (assignment wake and due-monitor wake,
`paperclip_runner`, prepared remote workspace, same-boot expired lease,
`preparing`). They expect bootstrap evidence, a scheduled retry, no
recovery hold after a second sweep, and no adapter call. A control case
keeps a same-boot `dispatching` run held for reconciliation.

## Verification

- Red on `origin/master` (source change reverted): both new cases fail.
The monitor case shows the exact hosted shape: `retryEligible: true`
with no `executionRecovery`.
- Green with the change: `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts -t "lease this
process lost|this process dispatched held|restart stopped before adapter
dispatch|reached adapter dispatch held|lost monitor|older build without
a controller stamp|failure budget"` (10 passed).
- Related files, run together: `heartbeat-process-recovery`,
`live-recovery-work`, `interrupted-preparation-hold-settle`,
`resource-wait-stop`, `legacy-controller-lease`,
`execution-recovery-dead-controller`, `legacy-process-settlement`,
`run-failure-report` (876 passed; 2 timing failures under load in
unrelated setup-failure tests, both pass when run alone).
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.

## Risks

- Low risk. The proof is the one #16355 already relies on across boots:
the revocation compare-and-set replaces the boot id, so the in-memory
controller of this process also fails its next renewal and cannot
dispatch.
- A same-boot run reaches this code only when it is not in
`activeRunExecutions` or `runningProcesses` and its lease expired. If an
executor is stalled before it registers, it aborts at its first renewal.
Setup work can run again in the retry; provider work cannot.
- Upgrade safety: no persisted shape changes. The reaper reads the same
fields (`executionStage`, `controllerBootId`, process ids) that older
releases write. Runs that older code already parked keep their hold and
their working Retry.
- Runs that reached `dispatching`, recorded a process, or have no
controller stamp keep the current behavior.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use, shell, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
code comment carries the contract)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:54:53 -06:00
DottaandClaude b93df4ca38 fix(chat): consume a saved chat wake whose message a completed run answered (#20689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered

## Linked Issues or Issue Description

**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.

**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.

**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.

**Paperclip version or commit**
`origin/master` at 6e524d8a69.

**Additional context**
This was seen on a staging deployment on five runs across three tasks in
one day. It is a message timing race, not a deploy or authorization
problem.

## What Changed

- `server/src/modules/wake-queue/adapters/postgres.ts`: a saved wake
whose reason is `External chat message received` and whose payload
mutation is `chat_message_received` is treated as message-only. If a
completed run already received all of its messages, the release consumes
it, the same as a comment wake. If any message is still unread, it is
promoted as before.
- `server/src/services/chat-inbound-wakeup-publications.ts`: a chat
receipt consumed for that reason does not publish "This follow-up was
not started". Other declines keep that notice.
- `server/src/services/queued-comment-delivery.ts`: constants for the
chat message wake reason and the delivered-message reason.
- New test
`server/src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`.

## Verification

- `cd server && npx vitest run
src/modules/wake-queue/adapters/chat-wake-delivered.test.ts`
- RED on `origin/master`: the release promotes the consumed chat wake
("Unexpected promotion of a consumed chat wake"), and the receipt
resolves to a `not_started` notice.
- GREEN with this change: the wake is cancelled with the delivered
reason, no run starts, no "not started" notice, and a chat wake with an
unread message is still promoted.
- Related suites: `src/modules/wake-queue`,
`heartbeat-reviewed-chat-binding.integration.test.ts`,
`durable-chat-wakeup.test.ts`,
`heartbeat-native-status-context.test.ts`,
`external-chat-wait.integration.test.ts`,
`heartbeat-held-comment-wake.test.ts`.
- `npx tsc --noEmit -p server` passes.

## Risks

- A chat wake that merged with another kind of wake takes the merged
wake's reason and mutation. Only a wake whose current reason and
mutation are both the chat message values is treated as message-only, so
other merged intents keep promotion.
- A visible "Your follow-up is queued." notice is not followed by a "not
started" notice when the message was already answered. That matches the
existing behavior when the wake is promoted.
- Not changed: a separate case where messages from two different chat
conversations are merged into one saved wake. That case fails with
`binding_denied` and needs its own change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edit, read-only API access). Extended
reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:15:13 -06:00
DottaandClaude Opus 5.5 4a61a2501e fix(storage): never promise a storage retry no storage wait owns (#20676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that hits a file-storage outage gets a bounded storage wait: a
retry that starts again after storage is back
> - The run error says "Paperclip will retry automatically when storage
is back"
> - On staging, a server restart reattached a remote native runner while
its turn was still running. A storage outage then failed the
reattachment
> - Restart recovery kept the workspace under a no-replay hold, so the
run took the code `native_continuation_requires_reconciliation`. Only a
`storage_unavailable` run gets a storage wait, so no retry was
scheduled. The run still showed the storage retry promise, and the task
stayed blocked for about 15 hours
> - This pull request makes the storage text follow the run's final code
and the real storage wait outcome, and reads old rows truthfully
> - The benefit is that every storage message is true: it promises a
retry only when one is scheduled, and it offers a working Retry when
nothing retries the run

## Linked Issues or Issue Description

**What happened?**
A native run on a remote sandbox was reattached after a server restart.
Its provider turn was still in flight. File storage failed during the
reattachment. Restart recovery settled the run as
`native_continuation_requires_reconciliation`, kept its files, and did
not replay the unfinished turn (this is the existing policy for a turn
with an unknown outcome). The run error said "Paperclip will retry
automatically when storage is back". No storage wait was scheduled, so
nothing retried. The task stayed blocked until an operator clicked
Retry.

**Root cause**
`server/src/services/heartbeat/run-completion.ts` chose the message and
the error code separately. The message used
`isStorageUnavailableError(err)`. The code used a chain where
`nativeTerminalFailureCode` wins over `storage_unavailable`. The storage
wait is scheduled only when the code is `storage_unavailable` (and no
native terminal failure is recorded). So the message promised a wait
that the code made impossible. The same false promise stayed when the
storage wait was refused (`not_scheduled`) or its write failed.

**Expected behavior**
A storage failure promises an automatic retry only when the storage wait
is scheduled. Otherwise it offers Retry. Rows saved by earlier releases
read the same way after deploy, with no migration.

**Steps to reproduce**
Fail a native run with a storage-unavailable error after its native
coordinator is settled as `terminal_failure` /
`native_continuation_requires_reconciliation`. See the new regression in
`run-completion.test.ts`.

## What Changed

- `storage-unavailable.ts`: add the held messages (outage and rejected
credentials). They say Paperclip is not retrying automatically and offer
Retry or a message. Add `storageUnavailableHeldMessage()` and
`presentStorageFailureMessage()`.
- `run-completion.ts` (`failRun`): compute the error code first. Use the
storage retry promise only when the code is `storage_unavailable`. Under
any other code (for example a native recovery hold), use the held
message. No storage wait is created for such a run, because it could
only be refused.
- `run-completion.ts` (`failRun` and `failRunSetup`): replace
`markStorageWaitExhausted` with `settleStorageWaitMessage`. It keeps the
promise only when a retry is scheduled. It writes the exhausted text
when the budget is spent, and the held text when the wait was refused,
its write failed, or a case handoff took over.
- `run-completion.ts`: when the promise is withdrawn, the failure's
`error` event (written before the retry decision) gets the same final
text, so the transcript does not keep the promise after reload.
- `run-completion.ts` (`failRunSetup`): a lost controller lease keeps
its own code and text, not the storage promise.
- `run-lifecycle.ts`: `decorateHeartbeatRunRuntimeStatus` presents a
saved storage retry promise under a code other than
`storage_unavailable` with the held text. This covers run detail and
issue run presence. `listRuns` in `run-state.ts` applies the same
projection, so the company run list and the Inbox failure read agree.
Rows from earlier releases read truthfully with no migration.
- `doc/run-log-events.md`: document which runs get a storage wait and
what each message means.

Previous-release story (upgrade-safe run state): this change adds no
persisted shape. Rows saved by the previous release carry forward. A row
with the storage promise under another code reads with the held text. A
`storage_unavailable` row keeps its text. The native hold policy is
unchanged: an unfinished turn with an unknown outcome is not replayed
automatically, and the existing Retry continues it.

## Verification

- Red before the fix (assertion failures, not setup errors): the new
`run-completion.test.ts` cases received "Paperclip will retry
automatically when storage is back" under
`native_continuation_requires_reconciliation`, and the
`run-lifecycle.test.ts` cases read the saved promise unchanged.
- Green after the fix:
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts -t storage` (13 passed)
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/storage/storage-unavailable.test.ts` (213 passed)
  - `cd server && npx tsc --noEmit -p tsconfig.json` (clean)
- Existing storage cases still pass: a scheduled storage wait keeps its
promise and suppresses rapid recovery; an exhausted wait writes the
exhausted text.
- `pnpm verify` exited 2 (a doc is a global CI input), so required CI
supplies the broad coverage.
- The full `run-completion.test.ts` file passed every test locally. Its
`afterAll` hook timed out under machine load (load average about 17),
also with the new tests skipped.

## Risks

- Low risk. The change affects only the text of failed runs and one read
projection. It does not change retry scheduling, recovery ownership, or
replay policy.
- The read projection matches this repository's own two promise
constants under a code other than `storage_unavailable`. A future change
to those constants must keep the projection in sync.
- Error events already written by earlier releases keep their original
text in the run log. Live events already published before a withdrawn
promise are not resent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads for GET-only diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:22:26 -06:00
Devin FoleyandPaperclip e9343d8501 fix(heartbeat): release task Retries held by a drain as soon as it ends (#20640)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person clicks Retry on a stopped task, Paperclip saves the
request as a held wake and dispatches it right away, unless admission is
held. Admission is held during a task drain, for example while a
deployment restarts the server
> - A Retry held by a drain is released later by the deferred-receipt
worker. That worker shares one queue with every other held wake: comment
waits, execution waits, and interrupted queues
> - On a busy instance that queue holds hundreds of passive waits that
come due every 30 seconds. A Retry is checked in due-time order behind
all of them. Ending a drain does not wake the worker either
> - In a real case, a Retry clicked during a deployment drain was still
held 40 minutes later, after the drain expired and the server restarted.
It ran only when the person clicked Retry again
> - This pull request checks operator Retries ahead of passive waits,
and wakes the worker as soon as a drain stops or expires
> - The benefit is that a Retry held by a drain starts within one worker
pass after the drain ends

## Linked Issues or Issue Description

**What happened?**

A person clicked Retry on a blocked task 14 seconds after a deployment's
task drain started. The Retry was saved as `deferred_issue_execution`,
and the immediate dispatch returned early because of the drain. The
drain had a 20-minute TTL, and the deployment restarted the server about
6 minutes in. The held Retry still did not start, and its `updated_at`
never changed. Forty minutes later the person clicked Retry again, and
the new request started a run and merged with the held one.

That instance had 390 held wakes: 271 execution waits and 116 others,
against 6 task Retries. The execution waits come due every 30 seconds.
Measured on the instance, the deferred-receipt worker took about 7
minutes to get through that backlog once.

**Expected behavior**

A Retry that a drain held starts soon after the drain stops or expires,
or after the server restarts, whatever the size of the passive backlog.

**Steps to reproduce**

1. Start a non-idle task drain (`POST /instance/task-drain`).
2. Click Retry on a stopped task. The receipt is
`deferred_issue_execution`.
3. Stop the drain. On `master`, the deferred-receipt worker only notices
on its next 30-second re-check. It then checks the Retry in due-time
order behind every passive wait that came due before it.

**Paperclip version or commit**

`master` at 8148fc0b1e.

## What Changed

- `server/src/services/scoped-deadline-work.ts`:
- New optional `priority(entry)`. Due entries are sorted by priority
first, then by due time and id as before.
- New `refresh(hint)`. It clears a scope's reconstruction delay and
invalidates it. Failure backoff is kept.
- New optional `onStop` hook, called when the coordinator stops the
worker.
- `server/src/services/heartbeat/deferred-receipt-work.ts`:
- A `task_retry` receipt that a person requested is an operator Retry,
with priority 1. Passive receipts keep priority 0, so their order does
not change.
- Each deferred scan loads operator Retries first, with their own query,
then pages by id as before. A Retry is therefore in memory in the first
pass even beyond the first 100-row page. A receipt loaded twice merges
by id.
- New optional `owner.subscribeRefresh`. The work calls `refresh()` on
all three of its queues when it fires, and releases the subscription
from `onStop`, so the coordinator's own stop removes it.
- `server/src/services/heartbeat.ts`: passes `subscribeRefresh` that
listens to task-drain changes and expiry. Before this change, a deploy
drain left the worker waiting for its 30-second re-check, behind the
scan's reconstruction delay.
- `server/src/__tests__/deferred-execution-receipt-work.test.ts`, four
new tests:
- A real Retry, requested during a deploy drain, starts a run (to
`succeeded`) within 5 seconds after the drain stops.
- An operator Retry whose id sorts after 120 passive execution waits
(more than one load page) is resumed first, in the first worker pass.
- A held Retry inserted with no notification right after a full scan is
resumed as soon as the drain refresh fires.
  - Stopping only the coordinator removes the drain listener.

## Verification

- `npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts
-t "held by a deploy drain|ahead of a backlog|ordinary committed task
Retry"`: 3 pass. The third is the existing ordinary Retry dispatch test.
- Red checks, removing one piece at a time:
- Without the drain subscription, the drain test fails: the Retry is
still `deferred_issue_execution` with no run after 5 seconds.
- Without the priority, or without the prioritized load, the backlog
test fails: the first resumed request is the passive `00000000-…` wake.
- With `invalidate()` instead of `refresh()`, the post-scan test fails
after 5 seconds.
  - Without the `onStop` release, the listener test fails.
- Full `deferred-execution-receipt-work.test.ts`: 50 pass.
- `chat-channels.integration.test.ts -t "still batches several replies
in one finding thread"`: it timed out once on CI and passes 2 of 2 times
locally with this change.
- `scoped-deadline-work`, `heartbeat-task-drain`,
`heartbeat-task-drain-admission-release`, `cloud-control-task-drain`,
`task-admission-deadline`, `onboarding-first-task-admission` and
`delivery-work-coordinator` suites: 164 pass.
- `pnpm run typecheck` in `server` (layered): clean.

## Risks

- Operator Retries are a small set; there were 6 on the instance above.
They now take the first slots of a worker pass, so passive waits can
move by at most that many places. Passive waits keep their own order.
- A Retry that keeps failing its check still gets the existing failure
backoff, so it cannot take every pass.
- A drain start also wakes the worker. That pass ends at once, because
the worker's run gate still reads the drain.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, vitest runs against embedded PostgreSQL, read-only
production log and database analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 15:00:50 -08:00
Nicky LeachandPaperclip f30ac226d6 Keep project environment mappings independent of shared defaults (#20651)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A project supplies work context and repository storage to task runs.
> - Creation selected a storage owner but did not retain the selected
environment on the project.
> - Configuration stored an environment inside checkout policy, but
policy reads and run selection dropped it.
> - Projects therefore continued to follow shared agent and instance
defaults.
> - This pull request saves placement on each project and uses it for
new runs.
> - The benefit is an independent Project to Environment association
with the existing storage, trust, and recovery safeguards.

## Linked Issues or Issue Description

Refs #20459 and #11644. Follow-up to #20587.

The reported journey is Project > Configuration: changing one project's
environment must not change another project's mapping or shared
defaults.

Related open PR #18176 repairs placement through the legacy
workspace-policy field. This follow-up introduces a separate project
association, saves creation defaults, and migrates saved legacy
references. It overlaps that PR's placement, permission, and recovery
safeguards. The changes should be reviewed together. #9588 concerns
environment-variable response privacy and does not address placement.
ROADMAP.md has no separate project-association work planned.

## What Changed

- Add nullable `projects.environment_id` and expose
`Project.environmentId`. Save the effective creation default once on the
project. Keep environment placement separate from checkout/isolation
policy.
- Read and write Environment directly in Project Configuration. Keep the
control available with isolated workspaces disabled. Each form reads its
project's value, even when environment choices share a query cache.
- Offer environments compatible with the current storage owner. Change
the target and storage together when the existing storage selector
attaches a core project to a plugin. Preserve retained-data restrictions
and validate the observed storage owner again on save.
- Resolve new task runs from their organizational project's target
before agent and instance defaults. File sources from another project do
not change placement. Explicit low-trust sandbox choices and enforced
sandbox policies retain precedence. External provider placement remains
separate.
- Restore admitted native runs from their owned environment lease. Later
project edits apply to new admissions. Require workspace-write
permission for environment changes and clears. Check company bindings
before runtime admission and during lease creation.
- Migrate valid saved policy references into the project column.
Normalize legacy API writes. Count and clear matching project references
when an environment is deleted. Update API and behavior documentation.
- Refresh the maintained SDK source registry for the two changed shared
files. Runtime pins are unchanged.

## Verification

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146)
is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful
checks**, three skipped gates, no failures, and `ci / verify` passed.
GitHub reports no merge conflicts. Standard Greptile scored the same
commit **5/5** with both review threads resolved.
- Source checkout on macOS arm64, Node 26.12.0, with fresh isolated
PostgreSQL. No production data or live provider calls.
- The two-project component test changes the first form with isolation
disabled and a shared query cache. The second form keeps its target.
Storage changes update only the selected project. A saved target remains
selected while the list loads, and an explicit clear remains possible.
- API integration creates two projects from the same default, patches
one, reads both, and checks that agent and instance defaults remain
unchanged. A later instance-default change does not rewrite either saved
project target. Tests cover unavailable/incompatible choices, anonymous
and cross-company denials, and storage preconditions.
- The migration test runs the actual SQL against an existing project row
and verifies its environment and disabled checkout policy survive.
Deletion accounting and clearing affect only matching project
references.
- The final guard set passes **53 tests across four files**:
`project-environments.test.ts`,
`environment-selection-route-guards.test.ts`,
`heartbeat/environment-selection.test.ts`, and
`ProjectProperties.environment.test.tsx`. All **64
environment-run-orchestrator tests** pass, including direct lease
acquisition without an extra caller argument. The broader affected set
passed **109 tests across eight files** before the additional admission
guards. These sets overlap.
- UI typecheck, migration numbering/safety, token gates, module
boundaries, query-error rendering, and diff checks pass. `pnpm verify
--explain` returns exit 2 and requires broad CI because shared
schema/runtime paths and behavior documentation changed. Broad coverage
is delegated to current-head CI.
- Early iterations failed due to a legacy parser that drops the
environment field, a fixture without real company membership, and
incorrect expectations for privacy-preserving 404 denials. The service
now reads legacy references before normalization, the API fixture has
real membership, and denial assertions match the existing privacy
contract. The orchestrator mock also needed the reused company-binding
guard in its service contract. The focused suites above pass after those
corrections.
- Local server typecheck remains blocked by installed ACPX, Slack, AI
SDK, and Runner type mismatches in unchanged dependency-facing code. It
reports no errors in the changed production paths. This is not a full
local typecheck or suite pass. Hosted CI supplies full current-head
coverage.
- SDK source-proof and privacy qualification passes **56 tests across
three files**, including the real built SDK. The maintained source
registry matches the committed revision; runtime pins are unchanged.
- Review reproduced two failures: an incoming policy without a storage
owner could be validated before the existing owner was restored, and
uppercase legacy UUIDs missed migration matching. Preserve the owner
before validation and compare normalized UUID text. Both regressions now
pass. The post-fix project/storage/orchestrator batch passes **107
tests**: `node node_modules/vitest/vitest.mjs run
server/src/__tests__/project-environments.test.ts
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/environment-run-orchestrator.test.ts --silent`.
- The first hosted run failed build, typecheck, and canary at the same
`run-completion.test.ts` call because lease acquisition required a new
argument. The lease method now enforces binding internally. Its existing
caller compiles without a new field, and the orchestrator tests prove
the binding flag reaches the runtime. The full server check no longer
reports that error; the local dependency errors above remain.

## Risks

- Apply the normal database migration before serving the new revision.
The migration adds a nullable foreign key and copies valid legacy
project selections. It does not move repository data or resize provider
storage. The five newest schema snapshots are retained.
- New projects retain their creation target when shared defaults change.
Existing projects without a saved selection continue to inherit
defaults. A null target explicitly opts into inheritance.
- Existing saved project selections now affect placement. Missing,
archived, incompatible, and foreign-company targets fail validation;
they do not silently choose another storage owner. A running admitted
process keeps its original owned lease.
- Local dependency limitations are listed above. No live-provider,
deployed-browser, or production rollout qualification is claimed.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving snapshot and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 14:31:48 -08:00
DottaandClaude Opus 5.5 1b7ec13acb perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn

## Linked Issues or Issue Description

**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.

**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.

**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).

## What Changed

- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
  1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.

## Verification

- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).

  | | Before (per-turn sandbox) | After (this change) |
  |---|---|---|
  | Follow-up, Codex | 30.4 s | 7.6 s |
  | Follow-up, Claude | about 30 s | 11.0 s |
  | First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
  | First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |

The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
b7f6243d60 with image `paperclip-daytona-runner@sha256:76a33275…6fe29`
(built from this branch by the `image_only` runner workflow; runner
content id `ed503454…a479` matches every later commit here, which change
only server, adapter-utils types and tests). `--daytona-inherit
--lifecycle default --adapters codex,claude --workloads A1,C1 --triggers
time,comment --faults K1,K2,K4 --tasks 6 --concurrency 3`.
- **6/6 tasks pass** (R1–R6 and T), all 6 crossed a restart; cells:
Codex A1 K2, Codex A1 K4, Claude A1 K4, Claude C1 K1, Claude C1 K4,
Codex C1 K2 at the mid-run comment. **No task needed Retry** (R3 n/a on
every task; one Claude task recovered automatically from
`process_lost`). No open signatures, 0 speed breaches.
- The default applied: the R6 sweep found 6 retained reusable leases
(one per task, Daytona keeps them) and no orphan or leaked sandbox; the
environment was deleted cleanly.
- Timings (Daytona, first turns, n=3 each): TTFT p50 Codex 21.9 s,
Claude 31.0 s. Restart recovery p50/p95: Codex 10.5 s / 29.6 s (n=3),
Claude 8.8 s / 23.9 s (n=2). Warm follow-up TTFT has no clean sample in
this wave (both follow-ups were crossed by a fault), so the follow-up
comparison above (30.4 s → 7.6 s Codex, ~30 s → 11.0 s Claude) remains
the speed evidence; no separate per-turn chaos wave was run.
- An earlier attempt, `w-20261110T170357-s20048`, never started tasks:
the first sandbox from the new image stayed `creating` past the
harness's 300 s Codex qualification bound (Daytona building the new
snapshot). It left one sandbox that the harness reported after deleting
the environment; the rerun on the cached snapshot is the wave above.
- The idle-window fix (9a74116082) landed after the wave; it changes
only runs whose inheriting environment sets `runnerIdleTimeoutMs`, which
this wave's environment does not.

## Previous-release story

No persisted state changes shape. A session admitted per-turn before
this change keeps its saved policy (`persistedPolicy` wins) until a new
session starts. A run in flight at the cutover keeps its admitted
lifecycle for recovery (`admittedLifecycleMode` wins in the environment
view).

## Cost and opt-out

- **Cost:** after a turn, the sandbox and its warm runner stay up for
the warm idle window (the environment's `runnerIdleTimeoutMs`, else the
agent's `idleTimeoutMs`, else 5 minutes). Then the controller suspends
the runner, and the provider auto-stop follows the same window. One idle
window per active task.
- **Opt-out:** set the agent's lifecycle to "Turn by turn"
(`lifecycleMode: per_turn`), or set the environment's
`runnerLifecycleMode` to `per_turn`. Either wins over this default.

## Reuse scope, restart and saved work

- **One lineage per sandbox.** The Daytona plugin keys a reusable lease
by the agent plus the execution workspace (or issue). Another task never
gets a sandbox holding a different task's files, and subtasks still get
fresh workspaces from the project base, so they get their own sandbox.
The workspace is adopted only after its stamp matches the host's.
- **After a restart,** a warm sandbox is reattached or replaced through
the existing warm recovery and owner fencing. A sandbox with a live
owner is never claimed twice. The Daytona chaos wave above checks this.
- **No lost work.** Each turn's workspace copyback
(`workspace_finalize`) runs at the end of the turn, before any idle
suspend, as for per-turn runs.

## Risks

- **Cost:** an idle sandbox stays up for the warm idle window (5
minutes) after a turn, then suspends. Only on providers with reusable
leases.
- **Probe cost:** one plugin-registry lookup before acquisition, only
for runs the default can apply to. No wait for a worker: a provider
whose worker is not running yet reads as unsupported, so that run is
per-turn.
- **Known gap (pre-existing, not widened):** with an explicit agent warm
choice on an inheriting environment that sets `runnerIdleTimeoutMs`, the
provider gets the environment's window while the host uses the agent's,
as on master.
- **Restart behavior on warm remote sandboxes** uses the existing warm
recovery paths. The Daytona chaos wave above checks K1, K2 and K4.
- Sessions admitted per-turn before this change stay per-turn until a
new session starts.
## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution, Daytona measurements).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:49:14 -06:00
Nicky LeachandPaperclip 98ce93dd4e Choose project storage from compatible environments (#20587)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Projects supply repository files to agent runs through core staging
or a storage plugin.
> - A storage plugin's default claim silently attached new projects even
when their agents used a different environment.
> - An environment name or driver type does not establish shared storage
access. The existing binding and task-driver contracts do.
> - This pull request groups creation choices by that contract and
selects storage from the creator's effective environment.
> - The benefit is a visible, validated choice and a clear error before
an incompatible managed workspace operation.

## Linked Issues or Issue Description

Fixes #20459.

Related: #20156 covers a separate provider-failure diagnostic. The PR
search found no other fix for #20459. ROADMAP.md has no duplicate
project-creation change.

## What Changed

- Show **Available in** when configured environments require different
project storage. Each option names the environments that can use the
same storage owner. Hide the selector when all choices are compatible.
- Derive compatibility from repository staging and managed binding
ownership. Local, SSH, and sandbox environments use core staging.
Managed storage requires a registered task driver from the same
installed storage plugin. Separate storage plugins remain incompatible.
Unknown plugin contracts and the fake sandbox are excluded.
- Follow the creating agent's environment, or the person's primary
agent, then the instance default and local fallback. Apply managed-only
and forced Kubernetes policy. The product has no separate user or
company environment-default field today.
- Add a company-authorized discovery endpoint and an optional creation
`environmentId`. Validate the choice again on save. Store the storage
owner without overriding agent run environments. Preserve explicit API
storage choices and existing project ownership.
- Keep the hidden repository-free private-task project independent of
environment availability. Submit the observed storage owner as
`expectedStorageProviderPluginId` so a changed contract fails at save
without setting a workspace policy. Instance task-isolation defaults
stay inherited.
- Reject an incompatible managed workspace before a provider operation.
Name the storage and selected environment in the error. Add regression
tests, API documentation, creation-story fixtures, and minimal
corrections to obsolete map text.

## Verification

- **102 tests passed across 7 files:** project environment integration,
installed storage policy, repository persistence, project environment
routes, project creation telemetry routes, OpenAPI, and
NewProjectDialog. Run with `node node_modules/vitest/vitest.mjs run
server/src/__tests__/project-environments.test.ts
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/project-repositories-persistence.test.ts
server/src/__tests__/project-routes-env.test.ts
server/src/__tests__/project-goal-telemetry-routes.test.ts
server/src/__tests__/openapi-routes.test.ts
ui/src/components/NewProjectDialog.test.tsx --silent`.
- The review fixes cover both archived-default private-task cases and
changed-storage rejection. With both instance isolation flags enabled, a
new core repository project keeps a null policy and resolves to an
isolated Git worktree. The complete affected set passes all 102 tests.
- The CI query-error policy failure was fixed by using the shared query
view. The form and shared query-view suites pass **24 tests**, including
cached choices during a transient refetch failure. `node
scripts/check-query-error-rendering.mjs --enforce --summary` reports
zero findings.
- CI exposed stale SDK source hashes and heartbeat fixtures that paired
plugin storage with Local. Refreshed exactly four shared-source hashes
with the maintained generator (runtime pins unchanged). All **53 SDK
privacy/source-proof tests**, including real built SDK output, pass. All
**7 corrected managed-storage heartbeat scenarios** pass; the fixtures
declare the owning plugin's task environment and substitute transport at
the existing execution seam.
- Shared and UI TypeScript checks pass. The UI production build passes.
`pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node
scripts/check-module-boundaries.mjs`, and diff checks pass.
- Tested UI states: compatible choices, incompatible groups, default
within a group, manual override, create failure, loading, discovery
failure and retry, and an unavailable default. Tests use the production
form and existing design tokens.
- The integration tests use real isolated PostgreSQL on macOS arm64 with
Node 26.12.0. They cover user/agent defaults, storage persistence,
policy restrictions, company and authentication denials, and rejection
before a provider call. Discovery uses saved configuration and makes no
worker or provider requests.
- Local verification limits: `pnpm -r typecheck` and `pnpm build` stop
in unchanged adapter-utils code because the installed ACPX types lack
`inheritProcessEnv`. A direct server check also finds an unchanged AI
SDK type-version mismatch. The existing heartbeat test `inherits the
instance default environment across companies while preserving explicit
agent overrides` times out at line 518 on both this change and
unmodified master with the same dependencies. The full `pnpm test:run`
attempt passed all six chat shards (1,948 tests), then the Git workspace
expiration suite reported 14 failures. I stopped that broad run. Two
sampled failures reproduce on unchanged master with the same
`unknown_worktree_git_files` refusal; the other expiration failures were
not independently compared. A later broad workspace-preparation run also
hit workspace validation failures and was stopped; the seven corrected
fixture scenarios pass independently. These limits do not establish a
full local suite pass. Hosted CI remains required.
- Initial missing-package and loopback-denial failures were resolved by
repairing isolated dependency links, building the SDK, and running
database tests with loopback access. No deployed or live-provider
execution is claimed.

## Risks

- Creation defaults now follow the effective environment. A plugin's
`defaultForProjects` claim alone no longer selects storage. This is the
intended behavior change.
- Compatibility covers repository project storage. It does not guarantee
arbitrary host paths, retained-workspace moves, adapter/image support,
credentials, or worker readiness. Matching provider configs cannot prove
a cross-plugin storage contract.
- A person's primary agent supplies the personal default because there
is no separate personal environment setting. An unavailable configured
default requires a valid explicit choice or a configuration repair.
- No migration runs. Existing projects keep their storage owner. This
change does not detach existing managed data or provide a storage
migration.
- Full local verification has the limits listed above. CI and standard
Greptile review are pending.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving snapshot and context-window size were not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 17:38:31 +00:00
DottaandClaude 7368a0b15c fix(recovery): re-queue native runs that fail on a transient database fault before provider work (#20561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.

## Linked Issues or Issue Description

**What happened?**

In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.

**Expected behavior**

A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.

**Steps to reproduce**

1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.

**Paperclip version or commit**

`master` at 902f376796.

## What Changed

- `run-failure-cause.ts`: new durable record
`resultJson.nativePreDispatchFailure = { version: 1,
providerWorkStarted: false }`, and
`isNativePreDispatchTransientFailure`. The predicate needs a native
failed run, a generic code (`adapter_failed`/`setup_failed`), the
record, and a transient database `failureCause` (`40001`, `40P01`,
`55P03`, `57P01`-`57P03`, `53300`, `08xxx`, postgres.js connection
codes).
- `run-completion.ts`: `failRun` and `failRunSetup` write the record for
a native run only when all of these are true:
- the caller reports `nativeDispatchMayHaveStarted: false` (an omitted
value means "may have started");
- the run was not stopped during the failure, and has no terminal native
coordinator failure;
- the run's `native_run_finalizations` coordinator, if any, is still the
unclaimed row that runtime selection creates (`observed`, attempt 0, no
lease owner, no controller, no result). Any executor claim, in any
generation, disqualifies;
- the run has no recorded process (`processPid`, `processGroupId`,
`processStartedAt`), no `sessionIdAfter`, and no `native_run_results`
row. Process fields copied from a warm predecessor (marked by
`runnerProfileJson.inheritedProcessIdentityRunId`) are not this run's
own process and do not disqualify; the run's own start events are still
checked;
- the run's whole event history has no execution-start evidence. This
uses the canonical `hasNativeExecutionStartEvidence` helper from restart
recovery (`native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`,
`tool.execution.started`, or any runner event with a source event ID).
The helper is unchanged; the earlier inline copy matched it exactly and
was removed.
- `run-completion.ts`: a recorded transient failure closes the unclaimed
coordinator as `terminal_failure` (`native_pre_dispatch_failure`),
because `hasPendingRunCleanup` otherwise holds the successor on it. It
then calls the existing `scheduleBoundedRetryForRun` with `retryReason:
"interrupted_preparation"`. An answered-interaction continuation keeps
its own retry lane. The storage-outage lane is unchanged.
- `heartbeat.ts`: passes `nativeDispatchMayHaveStarted`. It is true once
native dispatch started, once the dispatch gate entered the executor
(also when the gate then fails), while native ownership is held, or for
a restart-recovery or native-lease resume.
- No change to the native hold in
`modules/wake-queue/adapters/postgres.ts`. The release already returns
early when a queued/scheduled successor exists, so a scheduled retry
never opens the hold. See Risks for why the hold stays for the exhausted
and no-successor cases.

## Verification

- `heartbeat-native-predispatch-retry.test.ts` (assembled heartbeat,
mock native backend): budget admission fails once with `40001`. The run
records the proof, one uncharged `interrupted_preparation` successor is
scheduled, no recovery action opens, and the promoted retry succeeds
once. A failure after the dispatch gate entered the executor gets no
record and no such successor.
- `run-completion.test.ts` (embedded Postgres):
- native pre-dispatch `40P01` in `failRun` writes the record and
schedules `interrupted_preparation`; the release suppresses immediate
recovery;
  - native `setup_failed` on `40001` does the same;
- a run with `native.process_start_requested` in its history gets no
record and no retry;
- `nativeDispatchMayHaveStarted: true` and an omitted value (older
caller) get no record and no retry;
- a claimed native coordinator gets no record and no retry; an unclaimed
one is closed and the run re-queues;
- a run with a recorded process or provider session gets no record and
no retry;
  - a non-transient cause gets the record but no retry.
- `heartbeat-retry-scheduling.test.ts`: the real scheduler chains two
`interrupted_preparation` retries for a recorded native failure. The
first is due at once. `failureRetries` stays 0. A spent lane returns
`retry_exhausted`, queues nothing, and the run stays `failed`.
- `wake-queue/adapters/postgres.test.ts`: with a scheduled successor,
the release opens no recovery action and the task stays `in_progress`.
Without one, today's hold opens (`blocked`,
`native_continuation_requires_reconciliation`, Retry available).
- `run-failure-cause.test.ts`: the predicate is false for a row without
the record (older server), for `providerWorkStarted: true`, for a
non-transient cause, a non-generic code, a legacy run, and a cancelled
run.
- Command: `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts
src/services/heartbeat/run-failure-cause.test.ts
src/services/heartbeat/run-completion.test.ts
src/modules/wake-queue/adapters/postgres.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts` (all new cases pass).
`tsc --noEmit` for `server` passes. `pnpm verify` selected 2451 files
because `heartbeat.ts` is a hub; I stopped it and rely on CI for the
broad suite.
- Live path unproven: I did not run a restart-chaos wave for this
change.

## Risks

- A wrong "no provider work" record could cause duplicate work. The
record needs several independent facts (no gate entry, no resume claim,
no claimed coordinator, no process, session or result, no
execution-start evidence in the whole history). Any read error keeps the
hold.
- Rows saved by an older server have no record. They keep today's hold
and manual Retry. Nothing is migrated. A run resumed after a deploy is a
restart-recovery or lease resume, so it gets no record.
- The normal path gains no I/O. The evidence reads run only on the
failure path, and only when dispatch was never entered.
- The native hold is not skipped when no successor exists. Skipping it
would send the run to the generic immediate-recovery tail, whose new run
does not carry this lane's counter. That could loop without a bound. So
a spent lane, and a crash between the failure write and the retry
scheduling, still end in today's hold with a working Retry.
- The lane is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`
(12) with backoff after the first attempt, and it never revives a chain
whose failure budget is already spent.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (file edits, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20484, #20385.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:24:37 -06:00
Nicky LeachandPaperclip 2e3cbe57a4 fix(native): preserve parked Runners across controller restarts (#20586)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution parks a released sandbox Runner for a later turn.
> - The warm idle timer detaches Core and leaves Runner retention to the
host.
> - Controller shutdown instead stops recently parked Runners through
checkpoint cleanup.
> - This makes reuse depend on whether the idle timer fired before
shutdown.
> - This pull request uses the same detach operation for idle expiry and
shutdown.
> - The benefit is consistent Runner retention without a new retention
or recovery mechanism.

## Linked Issues or Issue Description

Fixes: #20012.
Related: #20005 unified cached and adopted process reuse. Closed #19975
included this shutdown change within a broader retention proposal.

## What Changed

- Detach confirmed released entries during controller shutdown. Keep
busy, unreleased, and close-marked entries on the existing cleanup path.
- Share the detach operation with idle expiry and cached-owner handoff.
- Record that a local handle has started detaching. Later cleanup cannot
fall back to stopping its Runner after a partial detach failure.
- Preserve explicit environment deletion, release proof, admission, and
process-reuse checks.
- Add Codex and Claude restart, detach-failure, and environment-deletion
regressions. Add real Runner adoption cases after the restart sweep,
including managed GitHub rebinding.
- Document the lifetime rule and make the shutdown warning cover both
detach and checkpoint failures.

## Verification

- Final source revision: `e4bb6ecbd7`. Local environment: macOS arm64,
Node 26.12.0.
- New controller-restart and detach-failure tests fail against unchanged
master: 4 expected failures. Old shutdown calls close or reports success
instead of reporting detach failure.
- Focused executor tests: 22 passed.
- Full native session executor suite: 1,328 passed.
- `cd server && node scripts/typecheck-layers.mjs`: passed all source
and test layers.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`:
passed.
- Manual diff review and added-line credential/PII scan: passed.
- `cd server && node ../node_modules/vitest/vitest.mjs run
src/services/native-runtime/plugin-runner-release-resume.integration.test.ts
--bail=1 --reporter=verbose`: 14 passed on this revision. This includes
new Codex and Claude restart cases and managed GitHub rebinding.
- Failure history: an earlier concurrent run had 4 passes and 10
failures. Its first failure was the existing Claude cached case
exceeding 60 seconds. Later cases encountered a leftover prototype spy,
recursive calls, and startup timeouts. No source changed before the
passing full rerun. A clean worktree of unchanged master (`e01fe7f84a`)
passed all 11 original cases. The first timeout's exact cause remains
unknown; the later pass does not establish that cause.
- Standard Greptile review: 5/5 on
`e4bb6ecbd7bd965b5f56507d2c42c580e8a1d547`, with no actionable findings.
- `node scripts/local-verify.mjs --explain` returned exit 2 because the
execution-semantics document requires broad CI. Required CI will supply
broad typecheck, test, and build coverage.
- A hosted deployment and an actual controller process replacement have
not been run. The added integration cases exercise the shutdown sweep
and a new adopting transport against a real Runner.

## Risks

- SIGINT and SIGTERM preserve confirmed parked Runners, including
ordinary service stops. The host still owns eviction and retention.
- Detach failure retains a fenced local owner and reports failure. Core
does not infer permission to stop the Runner from that failure.
- Reuse remains conditional on current admission, process configuration,
and host availability.
- No persisted schema, checkpoint format, launch identity, or migration
changes. State from the previous release uses the existing adoption or
verified restore paths.

## Model Used

- OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use,
and local test execution. The exact deployment identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 16:48:30 +00:00
DottaandPaperclip ce1ee9e89f fix: stop prompts from encouraging subtasks (#20362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents read runtime prompts, tool help, skills, and role
instructions.
> - Several of these texts tell agents to create child tasks as a normal
next step.
> - One accepted-plan prompt even forbids work on the current task.
> - Task parentage is useful, but it does not need a prompt that
encourages task creation.
> - This pull request removes those prompts and keeps the task APIs
available.

## Linked Issues or Issue Description

Refs #10642. This change adjusts prompt wording; it does not add
delegation-cycle guards.

**Current behavior**
Prompts tell agents to create children after plan approval, use children
for long work, and create a task just to pass context. Onboarding
requires a child even for a single request.

**Proposed behavior**
Continue work on the current task. Keep explicit assignments, optional
task creation, parentage, dependencies, and approval gates. Use the
separate-task recipes when the user requests that structure or when
continuing existing work.

**Reason and benefit**
Reduce needless task trees without removing subtask support. Keep the
patch limited to wording, existing test expectations, and generated
catalog data.

## What Changed

- Align accepted-plan prompts and remove direct child-creation advice
from runtime and error text.
- Make task tool descriptions neutral. Keep their fields, defaults, and
handlers.
- Update the main skill, onboarding, CEO instructions, company
templates, specialist workflows, and matching guides.
- Keep historical release snapshots intact. Refresh the explicitly
seeded schema fixture's catalog digest; the server still verifies
historical release snapshot hashes.
- Refresh the affected generated catalogs, capability inventory,
semantic contracts, and fixture manifest.

## Verification

- Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`.
- All 66 check runs are complete: 60 pass, one is neutral, and five are
skipped by CI conditions. Snyk also passes. No checks fail or remain
pending.
- [PR
CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554)
passes typecheck, build, server/general tests, all eight E2E shards,
Runner support, Runner Rust, and the aggregate verify gate.
[Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546)
and [Docker
Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108)
also pass.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835)
and [Paperclip
review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010)
give this exact head 5/5. All six inline threads are resolved. Neither
current summary has findings.
- Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved
three generated-file conflicts with the canonical generators.
- Local verification passes: 473 focused prompt, continuation, UI,
catalog, runtime and native-execution tests; Runner TypeScript build;
generated metadata; UI token gates; and diff checks.
- The prior head `c22018cb0f` failed three SDK evidence tests because
two source hashes were stale. Reproduced all three failures locally. The
existing generator refreshed only the hashes for the denial-copy file
and its test. All 53 affected evidence/source-pin tests then passed,
including private-value rejection. No scanner logic, assertions, or
runtime dependency pins changed. Current CI confirms the support suite
passes.
- A prior commit failed the Rust process reaping test. Its cause remains
unproven. No Rust code or assertions changed. Later Rust runs pass; this
does not establish the cause of the earlier failure.
- Local preflight built shared/plugin SDK packages, then stopped because
`cargo` is absent. Current-head CI supplies Rust and full build
coverage. `pnpm verify --explain` selects broad required CI; full local
suites were not repeated.

## Risks

- Wording changes guide the model; they do not enforce a ban on
subtasks.
- Existing installed skill versions and saved custom instructions need
their normal refresh process. This PR does not migrate company
configuration.
- Existing approved task assignments and approval gates remain in
effect.

## Model Used

- OpenAI GPT-6, through Codex, with reasoning, tool use, and code
execution. The exact routed model ID and context-window size were not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 09:58:54 -06:00
DottaandClaude Opus 5.5 902f376796 fix(recovery): keep a remote restart claim's lease through slow preparation (#20540)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, a remote run resumes on the same heartbeat run
under a restart claim: a lease on its native coordinator row, 60 s for a
remote claim (`REMOTE_CONTROLLER_LEASE_MS`)
> - Preparation then runs before the native executor takes its own lease
renewal. It includes the run-log begin and the workspace restore, which
can each wait out a storage outage for minutes (#20299)
> - Preparation renewed the claim only for `reattach_remote_runner`. A
`bootstrap_incomplete` or resume claim kept its 60 s lease with nothing
renewing it
> - So a preparation wait longer than a minute let the lease lapse. The
run failed at its next writer-authority check
(`run_log_writer_authority_lost`) the moment storage came back
> - This PR renews every remote restart claim during preparation. When
the same owner's executor claims at the next attempt, it takes over that
renewal
> - The benefit is that a resumed remote run survives a long storage
outage during preparation instead of failing as soon as storage returns

## Linked Issues or Issue Description

**What happened?**
In restart-chaos waves on Daytona R1 with `--storage s3 --storage-faults
ST1` and K2 at +110 s, a resumed `bootstrap_incomplete` run's
`heartbeat.prepare.run_output` ended `failed` right after the outage
ended:
- `w-20261110T101517-s11936` t01, run `9d39550a`. The claim was at
10:19:55 and the outage ended at 10:22:18.15. `run_output` failed at
10:22:18.17, after 141 s.
- `w-20261110T081647-s11936` t02, run `9da10424`. The claim was at
08:21:29 and the outage ended at 08:23:30.

During the outage the journal claim fails on its head `GET` before it
checks authority. The first attempt after the outage reaches
`claimFrom`'s `assertAuthority` (`run-log-authority.ts`), which requires
a live coordinator lease. That lease expired 60 s after the claim.

**Why not just widen the condition?** For a `bootstrap_incomplete`
claim, the executor's own claim moves the coordinator attempt to N+1
(`nextNativeProviderAttempt`). `startNativeSessionExecutionLeaseRenewal`
shares one renewal per lease owner and threw
`native_session_renewal_binding_changed` when the attempt differed. That
is presumably why preparation renewal was limited to reattach claims.

**Related:** #20460 (live status while a run waits out a storage
outage). #20514 settles a failure that this lapse used to leave silent.

## What Changed

- `heartbeat/recovery.ts`: start the preparation lease renewal for every
remote restart claim (`scope.remote`), not only
`reattach_remote_runner`.
- `native-runtime/native-execution-lease.ts`:
- When the same owner's executor starts its renewal at a higher attempt,
it takes over the shared renewal, rebinding the attempt and `onLost`. A
lower attempt or any other binding change still throws.
- A renewal that failed under the old binding is not counted as a loss.
- A failed renewal is not a loss when the coordinator shows the same
owner already at a higher attempt (the executor's claim committed while
the renewal was in flight). The renewal deadline still ends ownership if
no executor takes over.
- Tests: one recovery case and three lease-renewal cases (below).

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/heartbeat/recovery.test.ts
src/services/native-runtime/native-execution-lease-reacquire.test.ts
src/services/native-runtime/native-runner-restart-lease-renewal.integration.test.ts
src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`:
143 pass. `npx tsc --noEmit`: pass.
- Red/green, recovery test "keeps a remote bootstrap restart claim
through preparation longer than its first lease, then hands it to the
next attempt": the claim's first lease ends 4 s in. Preparation runs
past it and the run-log writer authority still passes. Then the executor
claims attempt 2 and its renewal keeps extending the lease with no
detach.
  - Red with master's `recovery.ts`: the lease is never extended.
- Red with only the `recovery.ts` change:
`native_session_renewal_binding_changed`.
  - Green with both.
- Red/green, lease unit tests: handover to the next attempt (and refusal
of a lower one); a failure under the old binding is not a loss; a
failure while the coordinator shows the same owner at a higher attempt
is not a loss. All three fail on master.
- Live wave: results in a comment below.

## Risks

- Medium-low.
- The preparation renewal now runs for remote `bootstrap_incomplete` and
`resume_dead_runner` claims too. It renews the same coordinator row
under the same CAS conditions reattach claims already use.
- If preparation truly loses the lease (another owner takes it), its
`onLost` still runs (`detachNativeSessionsForRestart`) and authority
checks still fence writes.
- The new non-loss case applies only when the coordinator names this
exact owner at a higher attempt. The renewal deadline still bounds it.
- Previous-release state: this persists nothing new. A claim saved by
the previous release resumes unchanged; its preparation is simply
renewed now.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 08:07:16 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00
DottaandClaude Opus 5.5 c9226de30b fix(heartbeat): leave a preparing run to the next controller after a graceful drain (#20514)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs each agent task. A graceful shutdown
(SIGTERM) drains running native runs and hands them to the next server.
> - The drain detaches the native runner session and releases the run's
lease with the recovery state `awaiting_runner_reattach`.
> - An in-process execution can still be in run preparation at that
moment, for example while the run-log journal claim waits on a slow or
unavailable object store.
> - When that execution then lost writer authority
(`run_log_writer_authority_lost`), the generic failure path settled the
run as `adapter_failed`. The next server then saw
`native_continuation_requires_reconciliation`, and replay stayed
blocked.
> - This pull request makes a preparing execution that a restart
detached stop quietly, the same way a detaching native session already
does.
> - The benefit is that a slow preparation step plus a graceful restart
no longer turns into a failed run that needs a manual retry.

## Linked Issues or Issue Description

**What happened?**
In restart-chaos wave `w-20261110T092411-s701` (local, A1/C1, K1/K2, ST7
read-after-write lag), three of four tasks failed with R3
`retry_needed`. For run `218a0425`, the K2 SIGTERM drain wrote
`native.recovery.transition` with `awaiting_runner_reattach` and
released the lease. About 3 s later, the old server's in-process
execution, which was still in `runOutput.begin`, got
`run_log_writer_authority_lost`. `runCompletion.failRun` then settled
the run `adapter_failed` with a durable settlement. The next server
found `native_continuation_requires_reconciliation` and did not replay
the run.

**Expected behavior**
The drain hands the run to the next server. The old execution must not
settle it. A real authority loss outside a drain must still fail the
run.

**Steps to reproduce**
1. Start a native run recovery, and hold it in preparation (slow store,
ST1 outage wait, or ST7 lag).
2. Send SIGTERM so `drainRunningRunsForShutdown` suspends the run.
3. Let preparation continue and check writer authority. The run settles
`adapter_failed`.

**Paperclip version or commit**
`master` at 914e3ae14f.

## What Changed

- `native-session-executor.ts`: add `watchNativeRestartDetach(runId)`.
It records the run's restart-detach marker when the execution starts. It
reports `true` only when a later restart detach for that run happens. A
marker left by an earlier detach does not count.
- `heartbeat.ts`: the execution creates the watcher at start. In the
inner failure path (before `runCompletion.failRun`) and the setup
failure path (before `runCompletion.failRunSetup`), if a restart
detached the run and no provider dispatch started, the execution logs
the handoff and stops without settling. It uses the same teardown flag
(`nativeSessionResumeScheduled`) as
`NativeControllerDetachedForRestartError`.
- `heartbeat-native-restart-suspension.test.ts`: add a
red-before/green-after regression. A recovery execution is drained
during preparation and then checks the real run-log writer authority.
With the drain, the run stays `running` with `awaiting_runner_reattach`,
and the next controller resumes it. Without the drain (the lease only
expired), the run still fails `adapter_failed`.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 98 passed.
Before the fix, the new drained case failed with `status: failed,
errorCode: adapter_failed`.
- `cd server && pnpm typecheck`: passed.
- Live restart-chaos wave: see the comment below.

## Risks

- Low risk. The new branch applies only when this process's own restart
detach for this run happened after the execution started, and before any
provider dispatch. Typed native outcomes (startup cleanup, ownership
unverified, cancellation pending, resume scheduled) keep their earlier
handling because they are checked first.
- Previous-release story (upgrade-safe run state): no persisted shape,
launch identity or pack identity changes. The marker is process-local.
State saved by the previous release carries forward unchanged. The
deploy-lands-mid-run case is the regression test itself: a drain lands
during recovery preparation and the next controller resumes the same
run.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 08:00:09 -06:00
DottaandClaude Opus 5.5 15a734d068 fix(heartbeat): show a storage outage wait as live status, then record it once (#20460)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In cloud mode a run reads and writes file storage on its critical
path: it starts its run log (`s3_segments`), restores its saved
workspace, and saves it again
> - Since #20299 a run waits out a storage outage there for up to 5
minutes instead of failing
> - That wait was silent. The board showed "running" with no sign of
life, so a storage wait looked like a dead run (restart-chaos R5
`silent_dead_run`)
> - The first version of this PR wrote a run event on every wait. That
writes the run row (`next_event_seq`, `updated_at`) while the run-log
claim is still in progress, and it covered only the run-log begin
> - This PR shows the wait as live run status instead, writes nothing
during the wait, and records one lifecycle event when the wait ends. It
covers every critical-path wait in the run's storage scope
> - The benefit is that a storage outage shows on the board as waiting,
the run's history keeps one honest line about it, and the claim path is
left alone

## Linked Issues or Issue Description

Refs #20299 (wait out a store outage on a run's critical path).

**What happened?**
- `w-20261110T043533-s11936` (Daytona R1, ST1 194 s, K2): run `d49f58d8`
waited 186 s in `heartbeat.prepare.run_output` with no event. The grader
failed it as `R5:silent_dead_run`.
- `w-20261110T105903-s11936`: the restore download
(`storage.snapshot.download`) waited 182 s with no sign of life.

**Expected:** the board shows the run waiting for file storage, and
nothing about the notice can disturb run-log ownership.

**On the failed `run_output` seen on builds of the first version**
(`w-20261110T101517-s11936` t01 run `9d39550a`,
`w-20261110T081647-s11936` t02 run `9da10424`): the evidence points to
an expired restart lease, not to the notice's row writes. Neither the
writer-authority check (`run-log-authority.ts`) nor the journal claim
compares `next_event_seq` or `updated_at`. A remote
`bootstrap_incomplete` restart claim gets a 60 s lease
(`REMOTE_CONTROLLER_LEASE_MS`), and preparation renews it only for
`reattach_remote_runner` claims. In t01 the claim was at 10:19:55, the
outage ended at 10:22:18.15, and `run_output` failed at 10:22:18.17, on
the first authority check after the store came back. That fix ships
separately as its own PR.

## What Changed

- `storage/storage-unavailable.ts`:
- The run's storage scope can carry a wait observer
(`reportStorageOutageWaitsTo`).
- `retryThroughStorageOutage` tells it about each backoff (`waiting`).
The operation that owns the wait reports one `ended` (`recovered` true
or false), so nested waits do not double-report.
  - `storageOutageWaitFor(purpose, work)` names the step.
- An observer that throws never changes the wait.
`endStorageOutageWaitOn` keeps the observer.
- `heartbeat/storage-wait-notice.ts` (new):
- Each wait sets live runtime status (phase `waiting_for_storage`,
"Waiting for file storage before starting the run log (45s so far)") and
publishes `heartbeat.run.progress`. Nothing goes to the database.
- When the wait ends, it writes one `lifecycle` warn event: "Waited Ns
for file storage before …", with payload `reason:
"storage_outage_wait"`, `purpose`, `waitedMs` and `recovered`.
- `heartbeat.ts`: installs the notice next to `endStorageOutageWaitOn`
(one line).
- Purposes:
  - run-log begin: "starting the run log";
  - `loadWorkspaceSnapshotArchive`: "restoring the workspace";
  - `uploadWorkspaceSnapshot`: "saving the workspace".
- `packages/shared`: adds `waiting_for_storage` to
`HeartbeatRunStatusPhase`.
- `tests/restart-chaos/live-status-evidence.ts` (new): R5 also counts
the board's live run status (`/api/issues/:id/live-runs`
`currentStatusUpdatedAt` / `lastEventAt`) as evidence that a running run
is alive. Wiring it into `observer.ts` takes one import and one changed
line.
- `doc/run-log-events.md`: documents the live status and the one event.
- Removed: the first version's per-wait run event.

## Verification

- `cd server && npx vitest run src/services/heartbeat/run-output.test.ts
src/__tests__/storage-outage-wait.test.ts
src/services/remote-workspace-snapshot-storage.test.ts
src/services/object-byte-journal.test.ts
src/__tests__/storage-s3-credential-rotation.test.ts`: 103 pass.
- Red/green: with the observer disabled (master behaviour), these new
tests fail. With this change they pass:
- run-output "shows the wait as live status and writes nothing to the
run while its log claim waits": no run-row write and no event during the
claim's waits; the live status is `waiting_for_storage`; the claim's
writer-authority check still passes; one lifecycle event afterwards;
- the helper's observer tests (one end per wait; gave-up reporting; a
throwing observer; the Stop signal keeps the observer);
- snapshot storage "tells the run it is waiting for storage while a save
or a restore waits out an outage".
- `PAPERCLIP_IN_WORKTREE=false npx vitest run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/live-status-evidence.test.ts
tests/restart-chaos/observer.test.ts
tests/restart-chaos/triggers.test.ts`: 65 pass. `node
scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. `cd
server && npx tsc --noEmit`: pass.
- Live restart-chaos waves: results in a comment below.

## Risks

- Low.
  - The retry policy, its budget and Stop handling are unchanged.
- During a wait, the only new effect is an in-memory status and a live
event.
  - Afterwards, one event per wait, written without blocking the run.
- Runtime status is per server process. A board on another instance sees
the waiting line only after the closing event.
- Shared journal work (`runSharedStorageWork`) keeps its own scope
without the observer. Its waits surface through the enclosing run-scoped
wait, as before.
- Previous-release state: this persists no new run or session state. A
run saved by the previous release resumes unchanged.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 07:42:43 -06:00
DottaandClaude Opus 5.5 fea4530a30 fix(recovery): retry a preparation killed by a restart in the uncharged lane (#20534)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the control plane restarts, an interrupted run must continue on
its own. A user must not have to press Retry because a deploy or a crash
happened
> - A preparation that a restart stops before the provider starts spent
no provider work. It is retried in the uncharged
`interrupted_preparation` lane (12 retries), not the charged
`transient_failure` lane (2 retries)
> - #20496 made that work for a graceful drain (SIGTERM). A hard kill
(SIGKILL) during workspace preparation still landed in the charged lane
> - Two killed preparations in one chain then spent the failure budget,
and the next interruption could not continue: the task needed a manual
Retry
> - This pull request lets restart recovery hand a killed preparation to
the uncharged lane once the killed controller is proven gone
> - The benefit is that a task survives any number of restarts during
workspace preparation (up to the crash-loop cap) without a Retry click

## Linked Issues or Issue Description

**Subsystem affected**
Heartbeat restart recovery: the process-loss reaper in
`server/src/services/heartbeat/recovery.ts`.

**Problem or motivation**
The restart-chaos harness on Daytona with `--storage s3
--storage-trigger snapshot-download` (seed 603, task t02, a codex C1
follow-up) kills the server while a workspace snapshot downloads. On
builds that contain #20496, two replays still ended in
`R3:retry_needed`. The run chain in replay
`w-20261110T061107-s603-replay-t02-1794309789008`:

1. K1 SIGKILL during preparation: `process_lost`, then "Scheduled
bounded retry 1/2 … transient_failure".
2. K2 drain during preparation: `interrupted_preparation` 1 (the #20496
path, correct).
3. K4 SIGKILL during preparation: `process_lost`, then "Scheduled
bounded retry 2/2 … transient_failure".
4. K2 drain during preparation: the failure budget was spent
(`failureRetries` 2), so the hold could not continue. The harness had to
click Retry.

**Root cause**
The reaper does record the no-provider-work certificate
(`executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`)
for a preparing run from another boot. But before it fails the run,
`revokeExpiredLegacyControllerWithState` swaps in a random boot id with
a 60 s lease, a tombstone that no controller holds.
`scheduleBoundedRetryForRun` then asks `isInterruptedNativePreparation`,
whose `preparerMayStillUnwind` check sees that live lease and says the
old preparer may still be unwinding. The lane falls back to
`transient_failure` and charges the failure budget. Every killed
preparation is charged, whether or not its controller is provably gone.

**Proposed solution**
When the reaper has already proven the old controller's recorded process
gone (`legacyControllerGone`, the shared owner prover that #20496 also
uses) and the revoked state shows the run was still preparing with no
process recorded (`restartInterruptedBeforeDispatch`), expire that
tombstone lease in the same `process_lost` write. The lane decision then
sees an expired lease and applies its existing checks (no coordinator,
result or execution events; leases released with stop receipts;
workspace repair when needed). Any evidence of provider work, or a
controller that cannot be proven gone, keeps today's charged behaviour.

**Alternatives considered**
- Revoking with an already expired lease: the 60 s revocation lease
fences other servers while this pass fails the run, so the expiry
belongs in the terminal write.
- Teaching `preparerMayStillUnwind` to ignore unstamped leases: that
would also skip the wait for a lease of a live controller saved by an
older release.

## What Changed

- `server/src/services/heartbeat/recovery.ts`: the `process_lost` write
sets `controllerLeaseExpiresAt` to now when
`restartInterruptedBeforeDispatch && legacyControllerGone`.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`:
- New: a chain whose preparation is killed twice (controllers proven
gone) and then drained. Each kill schedules `interrupted_preparation`
(attempts 1 and 2, `failureRetries` 0), and after the drain the restart
settlement schedules `interrupted_preparation` 3 without a Retry.
- New old-saved/new-resumed case: a preparing run saved by an older
release without a controller process stamp keeps the charged
`transient_failure` lane.
- `doc/execution-semantics.md`: one additive sentence saying the lane
also covers a hard kill once the killed controller is proven gone.

## Verification

- Red/green: `npx vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "killed twice"`
fails on master (the first kill schedules `transient_failure`) and
passes with the fix.
- Related suites pass: `heartbeat-process-recovery`,
`interrupted-preparation-hold-settle`,
`execution-recovery-dead-controller`, `legacy-controller-lease` and
`ownership-reconciliation-report` (5 files, 521 tests).
- Live (Daytona, `--storage s3 --storage-trigger snapshot-download`,
replay of `w-20261110T061107-s603/t02` with K1/K2/K4/K2, image
`paperclip-daytona-runner@sha256:05a4a092…` whose runner content matches
this branch): running; the result will be posted as a comment.

## Risks

- Low. The change applies only when the shared owner prover returns
`gone` for the exact recorded controller and the run was still preparing
with no process recorded. The lease's boot id stays the random
revocation id, so the old controller can still never renew it. Old rows
without a controller stamp keep today's behaviour.
- Previous-release story (upgrade-safe-run-state): no persisted shape
changes. A run saved mid-preparation by an older release has no
`controllerProcess` stamp, cannot be proven gone, and keeps the charged
lane (covered by the new old-saved test). Retries an older release
already scheduled in the charged lane keep their counters.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 06:48:43 -06:00
DottaandClaude dc9971bdb8 fix(native): keep another server's orphan sweep off a lease its owner took back (#20508)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an executor lease on its
`native_run_finalizations` row. #20474 lets the same owner take back its
own lease when the lease lapsed only because the host slept.
> - That take-back is safe only if every other actor that acts on a
lapsed lease records its action under the coordinator row lock first.
The plan `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`
lists the actors that do not. Gap 2 is another server's orphan sweep.
> - The sweep reads the lease as lapsed. Later it writes the
ownership-unverified marker on the run, without the coordinator lock and
without a new check of the lease. If the owner wakes and takes its lease
back between the read and the write, a healthy run gets the marker.
> - The board then shows "automatic recovery is blocked" on a run that
works. After the next host sleep, the owner cannot take its lease back,
so the turn is interrupted and the user must click Retry. The owner's
rule is that a task interrupted by sleep must resume by itself.
> - This pull request makes the sweep write its marker under the
coordinator row lock, and only while no executor lease on the run is
live.
> - The benefit is that exactly one side wins: the sweep marks a run
whose owner did not come back, and it leaves alone a run whose owner
took its lease back.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
Two servers share one database. Server A's host sleeps, so the executor
lease of its native run lapses. Server B's orphan sweep reads the lapsed
lease. Server A wakes and takes its lease back (#20474). Server B then
writes `native_execution_ownership_unverified` on the run that A still
owns.

**What did you expect?**
Server B writes the marker only if the lease is still lapsed when it
writes. One side wins.

**Steps to reproduce**
`server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`
holds the owner's take-back open, runs the real
`heartbeatService(db).reapOrphanedRuns()` until its write waits on a
lock, then commits the take-back. Before this change the run ends with
`errorCode: native_execution_ownership_unverified` and the owner's next
take-back fails with `native_session_lease_lost`.

## What Changed

- `setRunStatusFromLive`
(`server/src/services/heartbeat/run-lifecycle.ts`) has a new option
`onlyWhileNativeLeaseLapsed`. It locks the coordinator row first. This
is the same order that `lockNativeExecutionLeaseForReacquire` uses. The
UPDATE then requires that no executor lease on the run is live.
- `markNativeOwnershipUnverified`
(`server/src/services/heartbeat/recovery.ts`) takes `leaseLapsed`. The
orphan sweep in `recoverActiveExecutions` sets it when it read the lease
as lapsed or unset. Other callers do not change. They keep the same call
shape.
- A new regression test, red before the fix:
`server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`.
It also checks that the sweep still marks a lapsed run whose owner did
not come back, and that the late owner stays fenced out.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: gap 2 is
marked closed without the epoch design.

## Verification

- `cd server && pnpm exec vitest run
src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` — red on
`origin/master` (the run keeps the marker), green with this change.
- `cd server && pnpm exec vitest run
src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts
src/services/heartbeat/recovery.test.ts
src/services/native-runtime/native-execution-lease-reacquire.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts` — 129
passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json` — clean.
- Restart-chaos time-fault wave on this branch: see the wave comment
below.

## Risks

- Low risk. Only the orphan sweep's marker write changes, and only for a
run whose lease it read as lapsed. The sweep now takes the coordinator
row lock before the run row. The owner's take-back uses the same order,
so the two cannot deadlock.
- If the owner took the lease back, the sweep now writes nothing and
appends no event. The next sweep sees a live lease and defers, as
before.
- The other gaps in the plan (workspace finalization, shared-workspace
holder detection, board copyback confirmation) do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 03:57:20 -06:00
DottaandClaude Opus 5.5 0b20ea6fe9 fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Before a run starts provider work, Paperclip reserves its budget and
binds a usage recorder. Both write the run's `heartbeat_runs` row.
> - Run-log appends, output progress and lifecycle events update that
same row while the run prepares and streams.
> - #20385 fixed this race for the standalone usage-receipt write only.
Budget admission and the recorder handoff still take a repeatable-read
snapshot before they lock the run row.
> - Each concurrent row update makes that snapshot stale. Four retries
run out with SQLSTATE `40001`, and the run ends `adapter_failed` with
raw SQL as its error.
> - This pull request applies the #20385 pattern (READ COMMITTED under
the existing locks) to those writers and to the final dispatch gate. It
also gives exhausted retries a bounded cause.
> - The benefit is that busy runs start and account normally, and a
remaining failure names its cause.

## Linked Issues or Issue Description

Refs #20385. Builds on #20483 (bounded failure cause and
`withoutSqlStatementText`).

**What happened?**

The restart-chaos L1 workload with `--db-pressure 4` (harness writers
that update each running run row, one row per statement, like a fast
stream) failed on master. In one wave, 0 of 4 tasks passed. Every first
run failed in `reserveRunBudget` before spawn. In an earlier wave, a run
failed in `createRunUsageRecorder`. The stack in both cases is `could
not serialize access due to concurrent update` (`40001`) on the run-row
`FOR UPDATE`. The board `runError` showed the full `select ... from
heartbeat_runs ... for update` statement. The server log line `heartbeat
execution failed` dropped the database cause.

Realism: the harness writers are synthetic, but the product has the same
writers on the same row. `appendHeartbeatRunEvent` increments
`next_event_seq`, output progress updates `last_output_*`, and lifecycle
events land while a run prepares. On a hosted database the window
between snapshot and row lock is wider than locally, so a busy run
reaches the same race without the harness.

**Expected behavior**

Budget admission and the recorder handoff commit while other writers
update the run row. Budget invariants stay intact: one reservation per
run, a reached hard stop still refuses work, and committed holds count
against later admissions. Usage receipts are journaled once. If retries
still run out, the run shows a bounded cause without SQL, and the server
log records the SQLSTATE.

**Steps to reproduce**

Run `server/src/services/run-accounting-contention.test.ts` against the
parent commit. The deterministic cases commit a real
`appendHeartbeatRunEvent` after the writer's transaction checks the
company and before it locks the run. Six of the eight cases fail with
`40001`.

## What Changed

- `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow
the shared admission fences, the agent `FOR SHARE` lock and the run-row
`FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic).
The `budget_reservations_run_idx` unique index and the run lock still
prevent a second hold.
- The final dispatch gate's accounting phase
(`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks`
in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its
ownership phase already read the same staleness facts at READ COMMITTED
under the same issue and run locks. Retry promotion and stale-run
cancellation keep their repeatable-read snapshot. The first L1 proof
wave found this writer exhausting `40001` once the other three were
fixed.
- `createRunUsageRecorder` runs its receipt-source handoff and its
capture-failure fence at READ COMMITTED. Both lock the run row first,
the same as `persistUsageReceipt` after #20385.
- `withOperationAccountingTransaction` raises
`AccountingContentionError` when retries run out. It has a bounded
message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the
database error, and a warning logs both.
- `failRun` shows that bounded message. The code stays `adapter_failed`,
so chat and recovery retry paths still apply. #20483's `failureCause`
records `40001`. All other SQL text handling comes from #20483
(`withoutSqlStatementText`), unchanged here.
- `doc/DATABASE.md` documents the isolation choice and the failure
contract.

### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write
run-scoped rows

| Writer | Status | Notes |
| --- | --- | --- |
| `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ
COMMITTED under the run lock. |
| `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in
waves (`budget-reservations.ts:90`). |
| `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here |
Seen in waves (`usage-receipts.ts:572`). |
| `createRunUsageRecorder` capture-failure fence (`persistFailure`) |
Fixed here | Same shape: run lock first, runs mid-stream. |
| Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe
| A default READ COMMITTED transaction with `FOR UPDATE` on the
coordinator row. It cannot raise `40001`. The wave errors there were
`native_finalization_missing`, a symptom of the failed reservation (no
coordinator existed yet). |
| Final dispatch gate, accounting phase
(`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof
wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`,
`fea0ee5a`). Its ownership phase already used READ COMMITTED under the
same locks with the same staleness reads. |
| Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`,
`cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at
repeatable read: `loadGateFacts` combines wakes, comments, dependencies,
tree holds and repair state in one decision. They act on queued or
scheduled runs, not streaming ones. Not seen failing in waves. If
retries run out, they now fail with the bounded cause. |
| Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up
| Seen with `40001` in a wave log. Leaves the run pending;
`reconcileRunCosts` retries it durably. Multi-aggregate (coordinator,
receipts, images, closure). |
| Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) |
Safe | Insert `cost_events` only; the run row is referenced, not locked
or updated. |
| Budget enforcement and evaluation (`budgets.ts`), billing
reconciliation, accounting-integrity repair | Safe | Lock policy,
invoice or company rows, not a streaming run row. Run reads are plain
selects. |
| Browser use, decisions, prompt-injection, fast responses | Safe |
Write their own invocation/reservation rows, not `heartbeat_runs`. |
| Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool
gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or
no run-scoped writes. |

## Verification

- Red on the parent commit, green on this branch:
`server/src/services/run-accounting-contention.test.ts` (real embedded
PostgreSQL).
- `reserveRunBudget > reserves once when run events commit between its
company check and its run lock` (red: `40001`)
- `reserveRunBudget > reserves every run while back-to-back writers
update the run row` (red: `40001`)
- `reserveRunBudget > counts every committed hold, so successive
admissions never exceed the budget` (red)
- `reserveRunBudget > still refuses a run once the hard stop is reached`
(red: `40001` instead of `409`)
- `createRunUsageRecorder > binds its receipt source when run events
commit between its company check and its run lock` (red: `40001`; green
asserts the spooled receipt is journaled exactly once)
- `createRunUsageRecorder > binds while back-to-back writers update the
run row` (red: `40001`)
- `reserveRunBudget > never creates a second reservation for one run
when two dispatches race` (invariant; passes before and after)
- `final dispatch gate > admits and hands off a reserved run when run
events commit between its company check and its run lock` (red:
`AccountingContentionError` after 4 × `40001`)
- `createRunUsageRecorder > persists its capture-failure fence once when
run events commit between its company check and its run lock`
- `final dispatch gate > admits while back-to-back writers update the
run row` (red: `40001`)
- `exhausted retries > fail with a bounded, SQL-free cause that keeps
the SQLSTATE` (new contract)
- `run-completion.test.ts > saves a bounded cause and keeps the
retryable generic code when accounting retries are exhausted` (saved
code, message and `failureCause`, and log fields, through `failRun`)
- Related suites pass: `usage-receipts-read`,
`admission-independent-of-cleanup`, `budget-live-attempts`,
`idle-accounting-work`, `budget-enforcement-scoping`,
`closed-native-receipt-accounting`, `budget-evaluation-work`,
`run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126
run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown
exceeded 30 s on this loaded host, the known local flake).
- `pnpm --filter @paperclipai/server typecheck` passes.
- Latency of `reserveRunBudget` on the normal path (no contention,
embedded PostgreSQL, 350 measured reservations per run, 3 runs each):
before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50
14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the
statements are the same, only the isolation level changes. Under
contention it removes the retry backoff.
- Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and
claude, K1):
- Master before this change: `w-20261110T050818-s20386`, 0/4. Every
first run failed in `reserveRunBudget` with `40001`.
- Head `90530789b0` (first three writers fixed):
`w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude
failures (t02, t03) trace to the dispatch gate, now fixed. Those runs
already showed the bounded cause, with no SQL on the board. The failure
was logged with `sqlState` 40001. No `reserveRunBudget` or recorder
`40001` remained.
- Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`.
All 4 tasks reached done with no retry and R1–R6 passed. There were no
failed runs and zero `40001` in the server log, under 310k pressure
updates. Each task still failed the T-oracle `company_spend_settled`
with `pending_runs=1`. That pending run is t02's continuation run
`601ae80a`, cancelled before provider work as `issue_terminal_status`,
whose accounting debt was not yet settled when graded about 15 s later.
It is not a contention failure. Caveat: I edited the worktree during
this wave (`worktree_changed_mid_wave`), so a clean wave on the final
head follows.
- Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`.
All 4 tasks reached done with no retry, and R1–R6 passed for each. There
were no failed runs and zero serialization failures in the server log,
under about 523k pressure updates. Each task still fails only the
T-oracle `company_spend_settled` (`pending_runs=1`). It is the same
cancelled pre-provider continuation run (`issue_terminal_status`), whose
accounting debt was still unsettled at grading. That is a separate
settlement-timing follow-up, not database contention. Master was 0/4 on
the same seed.

## Risks

- READ COMMITTED gives each statement its own snapshot. Admission reads
policy, scope and exposure in separate statements. The shared
company/project fences and the agent share lock exclude concurrent pause
and policy edits, and exposure stays one statement, so each read is at
least as fresh as the old snapshot.
- Concurrent admissions of different runs stay best-effort, as
`doc/SPEC-implementation.md` already states ("concurrent starts can
exceed the same observed balance"). This change does not widen that
window: each statement snapshot is newer than the old transaction
snapshot. It does not add a company mutex.
- A run that fails after exhausted retries now records
`accounting_contention` instead of `adapter_failed`. Retry rules that
key on `adapter_failed` spawn-like messages did not match this failure
before, so no automatic retry changes.
- No schema, migration or persisted-format change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with shell, file-edit and code-execution tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 03:47:48 -06:00
DottaandClaude a14ae9b5c9 fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts, Paperclip prepares the task workspace and
runs the project's configured setup commands (for example the workspace
provision command)
> - When that command fails, the run fails as `setup_failed`, and the
task stops with a Retry
> - The task page shows only "Failed" and a generic recovery notice. The
real error (the command, its exit status and output) is visible only on
the run page
> - The user cannot see from the task which setting is broken or what to
change
> - This pull request records a bounded, named cause for a failed setup
command and shows it on the task page
> - The benefit is that the task names the broken setting and what to
change, and classification, retry and recovery stay the same

## Linked Issues or Issue Description

**What happened?**
The restart-chaos config fault wave CF1 sets a project
`workspaceStrategy.provisionCommand` to `bash
./scripts/provision-worktree.sh`, a script that does not exist. The run
fails as `setup_failed` with `Execution workspace provision command
"bash ./scripts/provision-worktree.sh" failed: bash:
./scripts/provision-worktree.sh: No such file or directory`. The task
page shows "Failed" and "Paperclip automatically retried continuation …
still has no live execution path". The grader fails the task as
`CF1/provision_command:message_only_in_run_detail`.

**Expected behavior**
The task page names the failed setup step, the configured command, its
exit status and a short tail of its output, and says what to change.
Retry behavior stays the same.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks
2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`.

**Paperclip version or commit**
`master` at 5ab98b0 (wave `w-20261110T043929-s204-cf`, tasks t01 and
t03).

## What Changed

- `WorkspaceCommandExitError` now carries the setup step
(`workspace_provision`, `workspace_seed`,
`workspace_runtime_provision`), the configured command and the command
output.
- New `server/src/services/workspace-setup-command-failure.ts`: it
builds a bounded, redacted `setupCommandFailure` record (step, command
up to 200 characters, exit code, last 300 characters of output). It also
recognizes the older message format, so runs recorded before this change
still get a name.
- `run-completion.ts` adds `setupCommandFailure` to the setup failure
`resultJson`. The error code (`setup_failed`) and all retry and recovery
fields stay the same.
- `TaskInterruptionPresentation.setupFailure` (shared type) is set for
`setup_failed` runs. It goes through the same per-run secret redaction
and user-name redaction as `workspaceFailure`.
- `ExecutionBlockerNotice` shows "Couldn’t set up the task workspace.
<command> failed with exit code N: <output>. Fix the command in the
project’s workspace settings, then retry." A scheduled transient retry
keeps its countdown and only adds the name.
- The restart-chaos config wave reads `interruption.setupFailure` as a
task-level text for its actionable-message check.

## Verification

- Red before, green after (the tests fail with the source changes
reverted):
- `server/src/__tests__/workspace-runtime.test.ts` — "names a failed
project provision command with its exit status and output (CF1)" runs a
real git worktree with a missing provision script (exit 127).
- `server/src/services/task-interruption-presentation.test.ts` — "setup
command failure presentation (CF1)": names the command, keeps
`state`/`canRetry`/`userAction` identical, recognizes the older message,
keeps a scheduled retry, bounds output and strips credentials from the
command.
- `ui/src/components/ExecutionBlockerNotice.test.tsx` and
`tests/restart-chaos/config-faults.test.ts` cover the notice text and
the harness check.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` pass.
- Live proof: config wave `w-20261110T083227-s204-cf` (CF1, codex, 2
tasks, K1 and K2, seed 204) on commit ae9a4b7: **2/2 pass**. Both tasks
show `The project's workspace provision command "bash
./scripts/provision-worktree.sh" failed with exit code 127: bash:
./scripts/provision-worktree.sh: No such file or directory` on the task,
rest again after the restart, and finish after the repair and Retry. On
`master` the same wave failed both tasks as `message_only_in_run_detail`
(`w-20261110T043929-s204-cf`).

## Risks

- Low risk. The change adds a presentation field. It does not change the
error code, retry, recovery or blocking behavior.
- The command and output are user-configured text. They are bounded,
passed through `redactSensitiveText`, the run's secret registry and
user-name redaction. If redaction fails, the field is omitted, like
`workspaceFailure`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:35:18 -06:00