Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,049 Commits 1,880 Branches 2,440 Tags
b32ef9f111f90838cb63761936c9e31f5a93d468
Commit Graph
283 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 b32ef9f111 perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077) 
## Thinking Path

- The native speed budgets (plan invariant 8) judge how fast an agent
responds. They judged `task.time_to_first_token`: time to the first
deliverable **text**.
- In the clean local restart storm (w-20261108T153612-s9401), Claude
breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The
per-run spans showed this was not a control-plane wait: in all 49 cold
runs (and 76% of warm), Claude started a tool call before writing any
text. Its first visible activity came at 7.4 s p50 from run creation;
first text came at 19.6 s. Codex never calls a tool first.
- So first text charges model tool work as latency for tool-first
adapters. What the user experiences as "the agent responded" is the
first visible activity: a text delta, a tool-call start, or a thinking
indicator.
- The coordinator decided: budgets judge time to first visible agent
activity from run creation, with the budget numbers unchanged; first
text stays a reported metric with no budget for tool-first adapters.

## Linked Issues or Issue Description

**What happened?**
Local storm s9401: the Claude TTFT budget breached on every aggregate,
while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner
startup ~4 s, first provider event 1.3–1.8 s) were well within budget.
The rest of the first-text wait was Claude streaming tool input (about
5.5 s per call) and running tools before it wrote text.

**Steps to reproduce**
1. Run a Claude task whose first action is a tool call (most real
tasks).
2. Read the run log: `provider.time_to_first_agent_event` ends at the
tool call; `task.time_to_first_token` ends many seconds later at the
first text.
3. The speed budget judges the later one and reports a breach.

**Expected behavior**
The budget judges the first visible agent activity from run creation
(`task.time_to_first_agent_activity`). Time to first text is still
recorded and reported.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: when
the first meaningful agent event commits (the existing
`firstMeaningfulAgentEventKind`: text delta, tool-call start, or
reasoning), record `task.time_to_first_agent_activity`. It uses the same
detached anchor resolution as TTFT (accepted comment, answered question,
durable wake request, or run creation), so it adds nothing to the turn
path.
- `server/src/services/native-runtime/native-speed-budgets.ts`: the
metric is now named `time_to_first_activity` (was `ttft`), with
identical budget numbers. The type comment documents the definition and
why first text is unbudgeted.
- `server/src/services/native-runtime/native-run-trace.ts`: the budget
check judges `task.time_to_first_agent_activity` from run creation,
keeping the existing input-wait handling. `task.time_to_first_token` is
no longer budgeted.
- `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`:
report first activity from run creation (the budgeted value) next to
TTFT.
- Docs: `doc/observability.md` (new span) and
`doc/plans/2026-11-07-ttft-baseline.md` (metric table).
- Tests:
- A tool-first turn records first activity at the tool call (5 s) while
TTFT records first text (16 s).
- A tool-first trace with slow first text but fast first activity is
within budget.
  - The existing budget tests now judge the activity span.

Observability path (AGENTS.md section 5.7): the new run-log span uses
only the existing `eventKind` and `anchor` attributes. Its name survives
run-log redaction intact; checked with `redactEventPayload`. The
operator OTLP gate is unchanged. It is not Telemetry.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "first
visible activity|first assistant text once|first-token timing"`: 7/7
pass.
- `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on
`tests/restart-chaos` is clean.

## Risks

- **A metric rename.** Consumers of `run.performance.budget_exceeded`
see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo
there are none besides the tests; Phase 3's chaos speed grade switches
to the new span.
- **A looser budget for slow first text.** Intentional: first text is
reported, not judged. A regression in control-plane latency still shows
up in first activity.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 12:32:42 -06:00
DottaandClaude Opus 5.5 c910651115 fix(native): start each rebound run with a fresh restart-continuation marker (#20053) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the control plane restarts during an agent turn, the runner
restores the same conversation. A Codex, Claude (ACPX) or OpenCode turn
lost in the restart is marked `provider_turn_lost_on_restore`, and the
run continues it once in the same conversation
> - That one continuation is guarded by a disposition-only recovery
marker in the session checkpoint. It is run state: it names a turn of
the run that used it
> - A follow-up run (a user comment on the same task) reuses the
provider session by re-binding the previous run's checkpoint to the new
run. Re-binding clears the run-scoped turn, result and request state,
but it kept the marker
> - The chaos K13 storm (wave w-20261108T134237-s9401) hit this three
times (t24 and t45 on Claude, t77 on Codex). The task's first run had
already used its restart continuation. The follow-up run inherited a
spent marker naming a turn it never had, so when a restart lost its own
turn, the driver kept the session closed. The run failed, and the user
had to click Retry
> - This pull request resets the marker at both re-bind sites, as for
every other run-scoped field
> - The benefit is that a restart during a follow-up turn continues on
its own instead of putting a Retry click on the user

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan, restart continuation). Found by the
restart chaos campaign. In all three failed follow-up runs:

- After `resume_dead_runner`, the run emitted `turn.failed {code:
provider_turn_lost_on_restore, recoverable: true}` for its own active
turn.
- The checkpoint still carried `dispositionOnlyRecoveryConsumed: true`,
with `dispositionOnlyRecoveryTurnId` set to the previous run's
continuation turn.

## What Changed

- `native-session-resume.ts` (`rebindNativeSessionCheckpoint`) and
`native-session-executor.ts` (warm checkpoint rebind): a rebound run
starts with `dispositionOnlyRecoveryConsumed: false` and
`dispositionOnlyRecoveryTurnId: null`.
- Nothing changes inside a run: the driver's reopen rule
(`mapTerminalTurn`) and `nativeRestartInterruptedTurnId` are untouched.
The first commit changed the helper; review found the driver keeps its
own copy of the rule, and the second commit reverts the helper in favour
of this fix.

## Verification

- `native-session-resume.test.ts` "retains provider identity but clears
prior turn and event state" now seeds a spent marker on the previous
run. It fails on `master` and passes here.
- `native-session-executor.test.ts` "starts a new run's usage from the
prior run's total when a warm checkpoint resumes" (3 cases) seeds the
first run's checkpoint with a spent marker and asserts that the second
run starts with it reset. It fails on `master` and passes here.
- The other `native-session-resume.test.ts` tests pass. One test,
"archives a damaged prior epoch before completing guarded same-task
replacement with real runnerd", fails locally identically with and
without this change (ENOENT under `authority-epochs`, from the
real-runnerd environment). It's left to CI.
- Next: a targeted chaos rerun (Codex and Claude C1 under repeated K2)
on this branch. I'll add the result here.

## Risks

- **Previous-release story (upgrade-safe run state):** no persisted
shape changes. A run re-bound by this build gets a fresh marker. A run
re-bound before the deploy, still in flight with an inherited spent
marker, behaves as before: a lost turn falls back to Retry, once.
- Inside a run, nothing changes: the marker still bounds same-run
continuations, together with the provider-attempt restart budget.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 08:49:38 -06:00
Nicky Leach e7e870cfc4 fix(native): decide warm and adopted Runner process reuse from one key (#20005) 2026-11-08 01:02:50 -08:00
DottaandClaude Opus 5.5 a4dbde6181 refactor(recovery): judge native owner processes with the shared prover (#19997) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs record the server and adapter processes that own them,
and recovery must prove an owner gone before it takes over or releases
its work
> - Phase 2 of the durable ownership plan gave the repository one
prover, `proveOwnerGone`. The native runtime still had four private
proofs of the same fact
> - The private proofs disagreed with the shared rules in edge cases:
some judged a PID from another PID space, and one accepted an
inconsistent start time as proof of reuse
> - This pull request moves those four proofs onto `proveOwnerGone` and
deletes the private liveness code
> - The benefit is one rule for "the owner is gone" across native
recovery, with the stricter shared behaviour where they differed

## Linked Issues or Issue Description

Refs #19933, #19950, #19953 (the shared ownership path and earlier Phase
2 kinds).

**What is the problem?**
Native recovery proved dead owners with four private checks instead of
the shared prover. The checks used different rules for PID spaces,
reused PIDs and unreadable start times.

**What should happen?**
Every native owner proof uses `proveOwnerGone`. Only a `gone` verdict
releases or takes over. The remote fence and the one-release legacy
fallback stay as they are.

**Why does it matter?**
Recovery decisions about the same process must agree. A dead PID in
another PID space must never be used as proof.

## What Changed

- `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: the
local-process branch now calls `proveOwnerGone`. The remote fenced
branch (#19830), the `operator_attested` sentinel and the null-host
fallback (`processSpaceUnrecorded` with `legacyHostFallback`, recorded
by callers as `legacy_host_unknown_dead_pid_proof`) are kept. Reasons
are unchanged, plus a new `controller_pid_start_inconsistent`. Owners
built from receipts with no PID space are judged in this PID space:
their callers check the recorded host first.
- `native-failed-run-owner-release.ts`: `recordedProcessMayBeLive` is
deleted. `runProcessMayBeLive` and `controllerMayBeLive` release only on
a `gone` verdict. Each use of the legacy fallback is recorded as
`legacy_host_unknown_dead_pid_proof` when the release commits.
- `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` is
replaced by `recoveryProcessMayBeAlive`, which is false only when the
prover proves the runner gone. Signalling a recovered runner requires a
proven same process.
- `native-local-process-stop.ts` `recordedLocalExecutionStopped`: the
bound identity (launch event, host, PID, group, start time) is judged by
the prover. It uses `process_host_hash` when the row has one.
- New `native-runtime/native-owner-stamps.ts`: controller and
adapter-process stamp builders, shared with the ownership registry.

Not changed: `proveLocalWorkspaceOwnerStopped` (workspaces),
`evaluateNativeProviderProcesses`, the stop observation in
`recordNativeLocalProcessStop`, and `runProcessMayBeLive` in
`native-workspace-sync-retention.ts`.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-restart-recovery.test.ts
src/services/controller-process-identity.test.ts`: 51 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-failed-run-owner-release.test.ts`: 14
passed.
- `cd server && npx vitest run
src/services/native-runtime/native-controller-host-recovery.test.ts`: 2
passed.
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts -t "keeps quarantine"`: 18
passed. `explicit-native-continuation.test.ts`: 414 passed.
`ownership-reconciliation-report.test.ts`: 23 passed.
- Red with the master versions of the two modules: 3 new tests fail
("accepts a reused PID only when the live process started definitely
later", "never releases on a dead PID recorded in another PID space",
"keeps the one-release fallback for a run process saved before its PID
space, and records it"). The tests were written after the code.
- Two fixtures changed to match the shared rules:
- `controller-process-identity.test.ts`: the "recycled" case recorded a
start after the live process, which is an inconsistent stamp. It now
records an earlier start (a real reuse), and the inconsistent case is
asserted as alive.
- `quarantined-task-retry.test.ts` `worker_alive`: the live worker now
records this process's real start time. The old fixture was a reused
PID, which the prover correctly proves gone.
- `cd server && npx vitest run
src/services/native-local-process-stop.test.ts`: 5 passed (review round
1). With the master module, 2 fail: the other-PID-space case and the
reused-PID case.
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

Behaviour changes, all from the shared rules:
- A live PID observed to start before the recorded controller is no
longer accepted as recycled (`controller_pid_start_inconsistent`; the
run waits for evidence).
- A failed-run owner whose recorded adapter process or controller is in
another PID space, or is a remote sandbox PID, is no longer released by
a local probe. It keeps the lease path.
- An unreadable start time is no longer exit evidence for a recovered
local runner.
- A local execution whose PID was reused by a later process now counts
as stopped. Before, any live PID blocked the proof.
- A local execution whose row records another PID space is no longer
counted as stopped by a dead local PID.
- A PID space that cannot be read here makes a hostless receipt owner
unprovable.

No persisted state changes. Rows saved by the previous release keep the
one-release null-host fallback, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 02:35:15 -06:00
DottaandClaude Opus 5.5 f755361ffe fix(recovery): deliver a steered message the restart lost before the model read it (#19962) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can steer a running Codex or Pi turn: a comment goes into the
live turn instead of waiting for the next run
> - The provider acknowledges the steer at once, but the model reads it
only at its next step, after the in-flight tool call
> - A control-plane restart in that window kills the provider with the
message in memory. Restart recovery starts a continuation turn that does
not carry the message. The message is lost and no error shows
> - The restart chaos harness showed this in 6 of 6 steered runs (Codex,
K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no
reply to the steered message
> - This pull request makes restart recovery deliver every acknowledged
steer that the model did not read, exactly once
> - The benefit is that manual Steer no longer loses messages on
restart, and the planned automatic steering can ship on top of it

## Linked Issues or Issue Description

**What happened?**
A comment steered into a running Codex turn got no reply when a
control-plane restart (SIGKILL, SIGTERM or crash loop) happened during
the agent's tool call. The run succeeded, and the steered comment had no
effect.

**Steps to reproduce**
1. Start a Codex task that runs a 40-second command.
2. During the command, post a comment and click Steer. The provider
acknowledges it.
3. Restart the control plane before the command ends.
4. Recovery resumes the turn with a continuation. The agent finishes the
task and ignores the comment.

**Expected behavior**
The agent reads and handles the steered comment once, after recovery.

**Root cause**
The provider keeps acknowledged steering input in memory until its next
model step. The restart continuation
(`paperclip.native-continuation.v1`) only told the model to continue the
current request, so the acknowledged input was gone.

## What Changed

- `unreadSteeredMessageIds` (runner package): derives
acknowledged-but-unread steers from run-log events in order. A
`steering_acknowledgement` item adds the message id (once per id). A
`userMessage` item in the same turn reads the oldest unread steer. Codex
records each steered input as one `userMessage` item when it enters the
conversation. A `userMessage` item in a later turn reads all earlier
steers, because the continuation carries them. `turn.completed` reads
all steers.
- `readUnreadSteeredMessages` (server): reads those facts from
`heartbeat_run_events` (only transport-validated events with a
`sourceEventId`). It also reads the comment or interaction text for the
same company and issue. The run log is the durable record of both
"acknowledged" and "read", so no new column is needed.
- The restart continuation now includes `steeredMessages` and one
instruction to handle them, oldest first. The runtime reads them through
a lazy `getUnreadSteeredMessages` option, only when a restart
continuation starts. Native recovery wires it for every run.
- `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a
steered interaction response with the same text the steer sent.
- Fake Codex gets `--consume-steer`, which emits the `userMessage` item
for an accepted steer, like real Codex.

## Verification

- Red/green live test with real runnerd and fake Codex
(`runnerd-restart-continuation.test.ts`, "delivers a steered message the
restart lost exactly once"): start a held turn, steer it, SIGKILL runner
and provider, restore and recover.
- "unread": the continuation carries the message exactly once. Without
the runtime change this test fails.
- "read" (`--consume-steer`): the continuation does not carry it again.
- Both: after the continuation completes, a later restart has nothing to
deliver.
- `steering-delivery.test.ts`: ordering, dedupe, initial-input,
later-turn and completed-turn rules.
- `unread-steered-messages.test.ts` (embedded Postgres): ignores events
without a source id and messages from other issues. It reads interaction
text, ignores model-output items as read evidence, and returns nothing
after the continuation input.
- `pnpm vitest run src/steering-delivery.test.ts
src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest
run src/services/native-runtime/unread-steered-messages.test.ts`
(server): pass on the rebased revision.
- `issue-queued-comments-routes.test.ts` and
`native-session-runtime.test.ts -t restart`: pass.
`native-session-resume.test.ts` "archives a damaged prior epoch…" fails
locally on plain `origin/master` too (local runnerd binary); CI runs it
with a fresh build.
- Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in
the harness slot. Before: 0 of 6 steered messages answered.

## Risks

- Duplicate delivery: a provider that records steered input without a
`userMessage` item, or a `userMessage` event lost in the crash window
before persistence, makes recovery send the message again. The model
then sees it twice. That is safer than losing it, and the instruction
asks the model to handle each one once.
- Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi
steer that the model already read before a restart may arrive again in
the continuation.
- Only the restart continuation carries the messages. Other recovery
paths start after a completed turn, which already read every steer.
- No schema or migration change. A session saved by an earlier release
resumes normally; its run log already holds the acknowledgement events.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 22:22:52 -06:00
DottaandClaude Opus 5.5 bf3a067076 feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep their ownership in `native_run_finalizations`, and
the shared ownership reconciler now settles proven-gone owners (Phase 2
of the durable ownership plan)
> - The finalizer lease (`native-finalizer:*`, 5 min) and the
retained-session cleanup lease (`native-cleanup:*`, 60 s) recorded only
a random token. They named no holder process
> - A server killed while it finalized or cleaned up kept the run
waiting for the whole lease. The lease was released by age alone. A
graceful shutdown also keeps an in-progress finalizer lease for the next
boot to wait out
> - This pull request records the holder process with each claim and
moves both kinds onto the shared prover and a compare-and-swap settle
> - The benefit is that a dead holder's run continues in the first pass
after boot, and a live or unprovable holder is never released

## Linked Issues or Issue Description

Refs #19933 (the settle hook and the first moved kind, merged).

**What happened?**
A finalizer or cleanup lease whose server process died stayed held until
the lease expired (5 min or 60 s). The run stayed `running` and its
follow-up work waited.

**What did you expect?**
The next boot proves the dead holder gone and frees the lease at once. A
live holder, or one that cannot be proven gone, keeps its lease.

**Steps to reproduce**
1. Accept a native result and let the finalizer claim it.
2. Kill the server (SIGKILL) before the finalizer commits.
3. Start the server. The run stays `running` for up to 5 minutes.

## What Changed

- `native_run_finalizations.lease_holder` (jsonb, migration 0446):
`{token, bootId, pid, hostHash, processStartedAt}` of the process that
took a token-only lease. It names the holder only while `token` equals
`lease_owner`, so renewals keep it and a later lease never inherits it.
The executor's `controller_*` columns are not changed.
- The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim
(`native-session-executor.ts`) write it in the same UPDATE as the lease,
from `currentControllerProcessStamp()` (no I/O).
- New `native-token-lease-settlement.ts`: compare-and-swap settle that
expires exactly the proven-gone holder's outstanding lease, after it
notifies the finalization or cleanup work that the expiry frees.
- Both claims also wake the ownership worker queue in their transaction,
so a quiet worker watches the new holder (Greptile round 1).
- `ownership-registry.ts`: `native_finalizer_lease` and
`native_cleanup_lease` register `settle`. Their owner is the holder
process; rows without a holder keep the token fence. Listings cover
outstanding leases only.
- `heartbeat/recovery.ts`: restart recovery settles both kinds before it
finalizes and schedules cleanup. A failure is logged and the reconciler
pass retries it.
- Two sparse partial indexes on `lease_expires_at` for the two listings,
so the settle watch re-check stays indexed.

Bespoke code deleted: none. These kinds had no liveness check; their
only release was lease age (the takeover in the claim), which stays as
the fence for unprovable holders.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts -t
"finalizer|cleanup|holder|exact lease"`: 12 passed (rerun after the
rebase on master). The crash-point test also asserts the ownership wake
for its run; it fails without the signal.
- `cd server && npx vitest run
src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed.
- Red on the base branch (server code at `bea3bfb20b`, tests and schema
from this branch): 7 new tests fail, 5 pass (the 5 are existing tests).
Red: the crash-point restart test, the two reconciler settle tests, the
three live/unprovable counter-cases (owner was a token fence), and the
exact-lease CAS test. The tests were written after the code.
- New tests:
- crash point: a finalizer claim was written and its holder killed
before commit; the next restart finalizes the run, and this boot's claim
stamps itself as holder;
- a dead finalizer or cleanup holder is settled by a recording pass,
never by a read-only pass (`record: false`); the accepted result and
phase are untouched;
- live counter-cases: a live earlier-boot PID (`pid_alive`), the current
boot (`current_process`), a dead PID in another PID space
(`different_process_space`, unprovable) keep the lease through a pass
and a restart;
- a holder written for another token, and a holderless row of the
previous release, stay token fences, also with
`PAPERCLIP_EPHEMERAL_SERVER=true`;
- settle refuses a lease taken over after the proof
(`owner_changed_since_proof`).
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved
by the previous release have no `lease_holder`. They stay token fences
and the existing claim takes them over at expiry. Neither lease outlives
5 min, so this covers at most 5 min after the upgrade. A rollback
ignores the new column.
- Unprovable holders (other host or PID space, unknown space): no
operator action. The lease is short and self-expiring; at expiry the
next claim takes it over and the old holder's next token
compare-and-swap fails. That fence is the handoff. A different host is
never proof.
- The settle only expires a lease. Finalization and cleanup still run
under their own checks (cleanup keeps its maintenance-history proofs).
Settling ownership does not settle the accepted result, workspace
preservation, or accounting.
- Migration adds a nullable column and two partial indexes on a sparse
predicate. Low risk.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 22:16:34 -06:00
DottaandClaude Opus 5.5 a4a3e342ec fix(recovery): charge a restart attempt only when the provider accepted a turn and made no progress (#19948) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run has an execution recovery budget: three provider attempts for
native runs, and two failure retries for legacy runs
> - In the restart chaos campaign, control-plane restarts used up that
budget for runs that were healthy (RC-22, 22b, 22c, 40 and 60)
> - #19720 and #19747 fixed two variants with watermark heuristics, and
legacy interrupted preparation still spent the failure budget
> - The durable-ownership plan (section F, invariant 5) asks for one
rule: charge an attempt only when a recorded outcome proves that
provider work started and no progress followed
> - This pull request adds that rule and moves the native restart charge
and the legacy preparation retry onto it
> - The benefit is that a restart never spends budget for work the
provider never started, while real failure loops and crash loops stay
bounded

## Linked Issues or Issue Description

Refs #19720, #19747, #19856

**What happened?**
A control-plane restart charges a provider attempt by heuristics. The
first crash restart always charges. A later restart charges unless the
previous resume had no start evidence (#19747) or completed work
(#19720). A runner that spawned but whose provider never accepted a turn
counted as started, so a K6 crash loop could still spend the budget.
Legacy runs whose preparation a restart interrupted
(`providerWorkStarted: false`) spent `failureRetries`, the same counter
as real failures (`execution-recovery-attempt.ts`).

**Expected behavior**
An attempt is charged only when the provider accepted a turn
(`turn.accepted` or `turn.started`) and no accepted turn then completed
work. A startup before spawn, a runner that spawned but was never
admitted, and a `turn.start` the provider did not accept are not
charged. The generation-12 cap still bounds uncharged restarts.

**Steps to reproduce**
Start a native run at its last attempt. Kill the controller after the
runner spawns but before the provider accepts a turn. Before this
change, the next restart claim blocks the run with
`execution_recovery_budget_exhausted`.

## What Changed

- New `server/src/services/native-runtime/provider-attempt-outcome.ts`:
the one rule. `readProviderAttemptOutcome()` returns
`provider_not_started`, `provider_started_without_progress` or
`provider_progressed` from run events. `providerAttemptChargesBudget()`
charges only the second. `providerAttemptWindowStart()` starts the
attempt window at the latest `native.process_start_requested`, so
earlier work never counts for a later attempt.
`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT` (12) is the shared cap.
- `restartResumeKeepsAttempt()` (`native-restart-recovery.ts`) now
applies that rule to the attempt the restart interrupted, including the
first restart. It keeps the unstarted-resume carry from #19747, the
reattachment rule (an adopted provider counts as started), the drain
exemption from #19625, and the generation-12 cap. A history entry from
an older release without a recorded position still charges.
- Legacy runs: `interrupted_preparation` is now an uncharged retry lane
(`execution-recovery-attempt.ts`). It keeps its own counter, bounded at
the same cap (12), and never changes `failureRetries`. `retries.ts` uses
the longest transient delay after the second lane attempt.
- `doc/execution-semantics.md` states the rule.
- Accounting has its own "provider work started" readers (inventory
Summary finding 5). Phase 3 owns those; this PR exposes the rule for
them to adopt and does not change them.

## Verification

- `native-runner-restart-recovery.integration.test.ts` (embedded
Postgres, real `claimNativeRestartRecoveries`): 129 passed. New cases,
red on master:
- not charged at the last attempt for a claimed remote bootstrap, a
remote bootstrap retry, and a claimed unlaunched remote run;
- not charged when the runner spawned but was never admitted, and when
the provider did not accept `turn.start`;
- not charged on the first restart when the original attempt already
made progress;
- a normal retry that spawned but was never admitted is not charged, and
earlier work still does not count for it.
- Counter-cases, green on master and with the change: an accepted
`turn.accepted` or `turn.started` with no progress is charged; the
generation cap blocks at the last attempt; crash loops with and without
progress stay bounded; the #19720 and #19747 cases still pass.
- `execution-recovery-attempt.test.ts`: new lane tests (red on master).
- `workspace-recovery-preparation-dispatch.test.ts` (25 passed),
`heartbeat-process-recovery.test.ts` (interrupted-run cases),
`run-dispatch/adapters/postgres.test.ts` (interrupted cases).
- `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory mode):
no errors.
- Not run: chaos waves (the coordinator schedules them on this host),
and the full server suite (CI covers it).

## Risks

- Behavior change: the first crash restart no longer charges when no
accepted turn is recorded. If a crash lands before the controller
projects the provider's `turn.accepted`, that attempt is not charged.
This fails toward more retries, and the generation-12 cap bounds it.
- A legacy preparation crash loop now gets up to 12 retries instead of
2. Each retry still passes the normal issue gates and keeps the
transient delay.
- The new queries use the `(run_id, seq)` index, once per restart claim.
Nothing new runs before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 21:20:05 -06:00
DottaandClaude Opus 5.5 18644a3263 feat(recovery): instrument 20 durable write boundaries with crash points (#19938) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A control-plane restart between two durable writes leaves an
in-progress record owned by a dead process
> - #19937 added `crashPoint(name)` and a registry of 20 named
boundaries from the Phase 0 inventory
> - A name without a call site cannot fire, so the crash-point suite
cannot test it
> - This pull request puts one call at each registered boundary and adds
a test that keeps the registry and the call sites in step
> - The benefit is that the on-demand suite (next pull request) can kill
the control plane at every known write-ahead gap

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead
coverage"). Stacked on #19937 (helper and registry). Refs #19861
(ownable kind ids).

**Problem**
The registered crash points have no call sites.

**Expected behavior**
Each registered point is called at its gap: after the first durable
write or outside effect returns, before the next one starts. Disarmed
(always in production) each call is one comparison.

**Scope**
Call sites and a coverage test only. The suite runner and CI workflow
follow.

## What Changed

- Server call sites (`@paperclipai/shared/crash-points`):
- `sandbox-creation-operations.ts`: around the provider `create` call
(intent → provider → lease).
- `native-session-executor.ts`: after `native.process_start_requested`,
before spawn.
- `paperclip-control-plane-port.ts`: after the result-accept
transaction, before finalization starts (only when a result was
inserted).
- `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before
the run projection.
- `workspace-operations.ts`: `workspace_finalize` only — after the
running row, and after `settle()` before the done row.
- `environment-runtime.ts`: provider release → lease row; teardown claim
→ destroy; reusable destroy → lease release; stop-and-retain claim →
stop.
- `heartbeat.ts`: lease release → runtime services; budget reservation
hold → dispatch.
- `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`.
- `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit →
spool unlink; receipt spool → DB row → unlink.
- `packages/paperclip-runner/src/crash-point.ts`: a small copy of the
helper for controller code in this package, which does not depend on
`@paperclipai/shared` (adding that dependency would change the runner
content ID). It arms only `runner.*` names. runnerd never sees the
variable: its environment is an allowlist.
- Runner call sites: after the runnerd spawn, before its identity is
published (both spawn paths); `turn.start` result received → controller
journal saved → next send.
- `server/src/__tests__/crash-point-call-sites.test.ts`: every
registered point has a call site, every call names a registered point,
and every `kinds` entry is an ownable kind id of the Phase 1 registry.

## Verification

- `cd server && npx vitest run
src/__tests__/crash-point-call-sites.test.ts` → 3 passed.
- `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter
@paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean.
- Behavior with nothing armed is unchanged: each call returns at its
first comparison (measured in #19937: about 3.7 ns).

## Risks

- Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT`
names it and `NODE_ENV` is not `production`.
- Points on the path before `turn.start` (process start, runner spawn)
cost nanoseconds, within plan invariant 8.
- `usage-receipts.ts`: the persist-then-unlink one-liner became a block
with the same statements in the same order.
- The runner package change gives the runner a new content ID, as any
runner source change does. No persisted run state changes
(rules/upgrade-safe-run-state.mdc).

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits) and a read-only exploration subagent that
mapped the boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 20:56:33 -06:00
Nicky Leach 94718ce168 fix(native): continue on a replacement when a retained Runner retires before claim (#19897) 2026-11-07 14:51:32 -08:00
DottaandClaude Opus 5.5 99529f9d71 feat(observability): local speed budgets and a run-log budget check (#19863) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start as fast as
possible, and restart-safety work must not slow it
> - The durable ownership plan makes this invariant 8 ("fast turns") and
asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per
adapter
> - Without budgets, a regression in time to first token, agent boot or
restart recovery shows up only when someone reruns the baseline by hand
> - This pull request records the budgets and adds a per-run check that
writes a run-log row when a run exceeds its p95 budget
> - The benefit is that every later phase can show "no regression
against the budgets" from the run log, without new waits on the turn
path

## Linked Issues or Issue Description

Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8).
Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on
#19861.

**Problem**
The plan requires p50/p95 budgets per adapter for TTFT, agent boot and
restart recovery latency, and a check against them. Neither exists.

**Expected behavior**
Budgets are recorded in the plan and in code. A run that exceeds a p95
budget gets one run-log row that names the metric, the measurement and
the budget.

**Impact**
Phase 2–5 exit criteria can cite budget breaches from chaos waves and
staging. The check adds no wait before `turn.start`.

## What Changed

- New `server/src/services/native-runtime/native-speed-budgets.ts`: the
local budget table (codex, claude, opencode) for TTFT and agent boot
(cold and warm) and restart recovery latency. Each budget is 1.2× the
baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run
against p95. An unknown start kind uses the looser budget. Remote runs
and adapters without a baseline are not judged.
- `native-run-trace.ts`: the trace takes an optional `budget` subject.
It checks `task.time_to_first_token`, agent boot (end of
`heartbeat.queue` to the end of the first `runner.turn.submit` or
`runner.session.resume`) and restart recovery once each. It writes
`run.performance.budget_exceeded` through `defer`, so the write is
detached from the event path. `finish` now drains diagnostics that defer
more work.
- `native-session-executor.ts`: passes the adapter and environment to
the trace, labels cold or warm at the existing `provider_session` reuse
receipts, and checks restart recovery at a recovered run's first
provider event, measured from the startup recovery `ready` time.
- Docs: budget tables in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed
budgets (Phase 1)"), and the new event in `doc/run-log-events.md`.

## Verification

- `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34
tests pass. They cover:
  - every budget has p50 ≤ p95;
  - the provider-to-adapter mapping;
  - p95 judgement, and the looser budget when the start kind is unknown;
- one breach for a slow boot, and none for a second submit or a fast
TTFT;
  - a deferred TTFT breach is written before the trace closes;
  - restart recovery is checked once;
  - remote runs and runs with no budget subject write nothing.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- Low risk. This is a run-log change only (AGENTS.md §5.7, no extra
review). The OTLP and Telemetry paths are unchanged.
- The baseline samples are small (2–12 per cell) on a shared host.
Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is
tight. Expect some breach rows on a loaded machine. They are diagnostics
and change no behavior.
- Restart-recovery budgets use run 4 of the baseline, as merged in
#19846.
- Restart recovery is measured from the startup recovery `ready` time to
the first provider event, which matches the baseline definition.
- Previous-release story: no persisted run state changes. The new event
is additive.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 15:27:42 -06:00
DottaandClaude Opus 5.5 76994c5027 fix(native): bound failover backup restores by inactivity, not total time (#19906) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native per-turn session that continues an earlier conversation
first restores its failover backup from durable storage. This happens
before the coordinator claim, so no provider has started.
> - That restore ran under a fixed 300 s total deadline
(`AbortSignal.timeout(300_000)`). The deadline covers the download and
the local decrypt, digest, extraction and provider-artifact steps.
> - On staging, restore time for resumed sessions grew during the day.
It reached 300 s, and then every resumed run of one review agent failed
with the generic "The operation was aborted". Fresh sessions were not
affected.
> - A failure at this point also enters the saved-work reconciliation
path, so the user request stopped.
> - This pull request bounds each durable read by inactivity instead of
total time. A slow transfer continues while bytes arrive. A read that
stops sending bytes fails with a named code.
> - The benefit is that resumed sessions no longer fail only because
their backup is large or the store is slow.

## Linked Issues or Issue Description

Refs #19894 (adds the same idle-deadline rule for workspace snapshot
reads; this PR covers the separate native failover-backup reads).

**What happened?**

On staging, a PR-review agent had 17 runs fail in about 80 minutes. Each
failure had the message "The operation was aborted" and the code
`native_continuation_requires_reconciliation`. Every failing run resumed
an existing session (`sessionIdBefore` set, per-turn lifecycle). Every
fresh-session run in the same window succeeded. In each failing run, the
error came 304–309 s after the `environment.startup` span, and no
`native.coordinator.claim` span was recorded. The run result had
`executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`.
Earlier resumed runs of the same agent show the time from
`environment.startup` to `native.coordinator.claim` growing from about
30 s to about 200 s before the failures started. The only fixed 300 s
deadline on that path is the failover-backup restore in
`restoreDurableHarnessForExecution`.

**Expected behavior**

A resumed session restores its backup when the store keeps sending
bytes, even when the total time is more than five minutes. A read that
stalls fails with a named, diagnosable code. A failed restore publishes
no partial backup and keeps the saved backup.

**Steps to reproduce**

1. Persist a native failover backup with provider artifacts.
2. Restore it through a storage reader that sends bytes slowly, so the
full restore takes longer than the deadline.
3. Before this change, the restore fails with "The operation was
aborted". A reader that stops sending bytes, with no caller deadline,
waits indefinitely.

## What Changed

- `native-harness-backup-storage.ts`: add `downloadIdleTimeoutMs` to
`restoreDurableNativeHarnessBackup` and
`verifyDurableNativeHarnessBackup`. When set, each durable `getObject`
(the backup object and the provider artifacts) runs under an idle timer.
Each received chunk renews the timer. The timer is active only while a
read is in flight, so local decrypt, digest, extraction and copy are
never timed. On expiry the whole restore aborts and throws
`native_harness_storage_download_stalled`. Caller cancellation keeps its
own error.
- `verifyRecoverableNativeHarnessBackup` and the executor's
`restoreDurableHarnessForExecution` use the five-minute idle limit
(`NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS`) instead of
`AbortSignal.timeout(300_000)`.
- The whole restore or verification keeps a 30-minute outer limit. It
fails with `native_harness_storage_restore_deadline_exceeded`, so a hung
local step stays bounded and named.
- The executor passes the startup Stop signal
(`providerPackSourceAbort`) into the restore, so Stop (also a Stop that
came before the restore began) ends it. Only a startup that already
completed does not reuse its signal. The restore checks the signal after
the local copy and before it publishes `current`.
- The wrapped storage is a `Proxy` that replaces only `getObject` and
forwards every other member to the real service.
- `doc/DEVELOPING.md`: document the idle limit and the named code.

Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Backups, artifact rows, receipts and
checkpoints saved by the previous release restore unchanged. A deploy
that lands during a restore stops the process. The new process starts
the restore again from the saved durable backup under the new limit. The
existing hash, size, encryption and atomic publication checks are
unchanged.

## Verification

Run from `server/`:

- `pnpm exec vitest run
src/services/native-runtime/native-harness-backup-download-deadline.integration.test.ts`
— 4 tests passed (embedded PostgreSQL, local-disk storage, an OpenCode
backup with provider artifacts). The tests are in a focused suite, so
the existing 990-line storage suite does not grow.
- `pnpm exec vitest run
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`
— 68 existing tests passed.
- New: "keeps restoring a backup whose download outlasts the idle window
while bytes keep arriving". With the former total-deadline policy, the
same slow transfer fails with "The operation was aborted" and publishes
nothing (the staging message). With the idle limit, the restore
completes after more than twice the window, restores the session and the
OpenCode launch artifacts, and verification also completes.
- New: "names a download that stops sending bytes and publishes no
partial backup". It checks restore and verify, checks that no `current`
backup is published, and then restores the same saved backup
successfully.
- New: "stops at the outer bound during local copy and publishes
nothing".
- New: "lets Stop end a progressing download with its own reason,
including a Stop before it starts".
- RED: with the production files reverted to `origin/master`, the stall
test does not finish and times out after 10 s, because no stall is
detected. The other two tests pass on the old code, because the total
deadline was the executor's caller policy. The first test reproduces
that policy directly.
- `pnpm exec vitest run` on `native-harness-online-preservation`,
`native-harness-filesystem-independence` and
`native-controller-migration` integration suites passed.
`native-warm-checkpoint-ownership.integration.test.ts` fails locally on
master as well, because the local Rust `paperclip-runnerd` binary is not
built.
- `node scripts/typecheck-layers.mjs` passed for all layers.

Not verified: a live resumed run on staging with this build. The staging
evidence shows where the time goes (before the coordinator claim, on
resumed sessions only) but does not show which restore step was slowest.
The fix removes the total deadline for every step on that path.

## Risks

- A restore that progresses very slowly can now take longer than five
minutes. Before, it failed. The run still stops on caller cancellation.
- A local step that hangs (for example a stalled disk) is now cut off by
the 30-minute outer limit instead of 5 minutes. A stalled local writer
also pauses the read through backpressure, so it can be reported as
`native_harness_storage_download_stalled`.
- A failure before the coordinator claim still enters the existing
saved-work reconciliation path. This PR does not change that
classification. It removes the cause of these failures.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with
tool use (shell, file edits, read-only browser for API inspection) in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 14:34:57 -06:00
DottaandClaude Opus 5.5 f439150cb3 fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #19868 and #19871 made restart recovery judge a recorded pid only in
the PID space that recorded it
> - Rows saved before those host columns existed have no recorded host.
They kept the previous local dead-pid proof, so in-flight native runs
would not be stranded at upgrade
> - On an ephemeral (multi-host) server that fallback is unsafe: the row
may have been written on another host
> - This pull request limits the fallback to single-host servers,
records every use, and gives ephemeral servers a working board action
> - The benefit is that single-host upgrades still recover on their own,
multi-host servers never accept a cross-host proof, and the next release
can remove the fallback with evidence

## Linked Issues or Issue Description

**Bug description**
After #19868 and #19871, a `native_run_finalizations` row with
`controller_host_hash = null`, or a run with `process_host_hash = null`,
still accepted "the pid is dead here" as proof on every server. That
includes an ephemeral server, where the row may come from another host.
Nothing recorded how often the fallback was used, so it had no safe
removal path.

**Expected behavior**
- Single-host server: the fallback applies for one release, and each use
writes a `legacy_host_unknown_dead_pid_proof` activity-log entry.
- Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is
unprovable, and the board has an action to confirm the previous server
stopped.
- A mismatched host is always unprovable.

Refs #19868, #19871, #19846

## What Changed

- `native-restart-recovery.ts` `evaluateNativeControllerTakeover`:
- A coordinator row with `controllerHostHash: null` returns
`controller_host_unrecorded` when `ephemeralServerEnabled()` is true.
- Otherwise a pid proof from that row carries `legacyHostFallback:
true`.
- `operator_attested` is accepted as proof
(`operator_attested_controller_stopped`).
  - Receipt-built owners, which have no host key, are unchanged.
- New `recordLegacyHostFallbackProof` writes the activity-log entry. It
is not a run-log event: run events are progress evidence for restart
recovery's crash-loop credit, and the record must not change any
recovery decision. It is called from restart recovery, the
dead-controller result-lease release, native retirement, and remote
warm-session recovery.
- New `native-controller-host-recovery.ts`:
- `requestNativeControllerStopConfirmation` (board-owned
`native_controller_host_unverified` recovery action, idempotent, never
displaces another active action).
- `confirmNativeControllerStopped`. It verifies the exact recorded
controller and stamps `operator_attested` with compare-and-set. It then
resolves the action, wakes live-run recovery, and logs
`issue.native_controller_stop_confirmed`.
- Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`.
It is board-only and requires `runtime:confirm_owner_stop`. Generic
`restored`/`false_positive` resolution of this cause is refused. A
shared `confirmNativeControllerStoppedSchema` validates the body.
- UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server
stopped" control for this cause, and the generic recovery card hides it.
Adds the `issuesApi.confirmControllerStopped` client call.
- `recovery/service.ts` orphan reaper: a null `process_host_hash` is
judged only when not ephemeral. A successful use records the entry
best-effort, after the terminal write. On an ephemeral server the run is
left to its existing Stop control.
- `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts`
(re-exported by `ephemeral-server-contract.ts`). Importing the contract
module from recovery code created an import cycle in the runner child
process.
- The new route is registered in OpenAPI.
- `doc/execution-semantics.md`: documents the host rule and the
one-release sunset.

## Verification

- `native-restart-recovery.test.ts`: null host on a single-host server
is released with the flag. Null host on an ephemeral server is
unprovable. A mismatched host is unprovable in both modes. An attested
controller is accepted. Receipt owners are unchanged.
- New `native-controller-host-recovery.test.ts` (embedded Postgres):
  - The fallback writes the activity-log entry and no run event.
  - Ephemeral mode creates exactly one board action.
  - A wrong run id is refused.
- Confirmation stamps the attestation, after which takeover is allowed.
The action resolves and the activity is logged. A second confirmation is
refused.
- `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a
null host is terminalized and records one event. On an ephemeral server
it is left alone with no event. The existing audit-failure case still
clears the lock.
- `WorkspaceExportRecovery.control.test.tsx`: the controller
confirmation posts to `confirmControllerStopped` and never to the
copyback endpoint.
- Locally: these suites plus `remote-warm-session-recovery`,
`native-controller-recovery.integration` and
`native-startup-cleanup-recovery` (136 passed), and after the CI fixes
`native-runner-restart-preparation` and `native-runner-restart-recovery`
integration, `openapi-routes` and the suites above (256 passed); the UI
recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no
errors in the touched files; `pnpm check:token-gates` clean.

## Risks

- Previous-release story: no schema change. Null-host rows saved by
earlier releases recover as before on single-host servers, now with an
event, and wait for the board on ephemeral servers. The next release
removes the fallback; the event count shows whether any rows remain.
- On an ephemeral server, a null-host orphaned run process is not reaped
automatically. The operator uses Stop. That is the intended "unprovable"
outcome.
- A board confirmation is an attestation. It applies only to the exact
recorded controller (boot id + pid) and only while the host is still
unrecorded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 13:25:47 -06:00
DottaandClaude Opus 5.5 0189fedffc fix(native): bound an idle warm local runner whose controller died (#19879) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd alive between
runs so a follow-up turn starts fast. The follow-up default flip (next
PR) makes warm the default for local Codex and OpenCode agents.
> - The controller owns a warm runner's idle window. Runnerd ignores the
idle timeout itself.
> - A graceful controller shutdown suspends idle warm sessions. A
controller that dies (SIGKILL, crash) does not. Local runners launch
with no process reconnect grace, so the idle runner kept running until
its 1-hour connection lease ran out, owned by nobody.
> - This pull request gives warm local runners the 7-minute reconnect
grace that remote runners already use.
> - The benefit: idle runners are reaped by proof within a bounded time,
never left orphaned, before warm becomes the default.

## Linked Issues or Issue Description

Refs #19874 (warm completion contract), #19846 (TTFT baseline), #19852.

**What existing behavior does this improve?**
Ownership of an idle warm local runner after its controller dies.

**Current behavior**
`runnerProcessReconnectGraceMs` is `undefined` for local targets, so
runnerd launches without `--reconnect-grace-ms`. An idle warm runner
whose controller was killed stays alive until its connection lease (one
hour) expires. The restart-chaos harness cannot show this: its K1 kills
the controller's whole process tree, runner included.

**Proposed behavior**
`localRunnerProcessReconnectGraceMs(policy)` returns the remote grace (7
minutes) for a warm policy and `undefined` for per-turn. After the
grace, runnerd records `transport_reconnect_grace_exceeded`, stops its
provider, keeps durable state and exits. The next run resumes cold.

**Reason and benefit**
The restart plan requires idle runners to be reaped by proof, never
orphaned. Seven minutes is far longer than a controller restart (13–20 s
locally) and the long-outage fault holds (3–5 minutes), so a live turn
still reattaches.

## What Changed

- `native-runner-recovery-contract.ts`:
`localRunnerProcessReconnectGraceMs`.
- `native-session-executor.ts`: the local runnerd transport passes it as
`runnerProcessReconnectGraceMs`. Remote targets keep `null`
(lease-owned).
- `runnerd-warm-idle-orphan.test.ts` (new): real runnerd and fake Codex.
An idle warm runner, after its turn completed, whose controller
detaches.
- `doc/DEVELOPING.md`: the grace and its bound.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-warm-idle-orphan.test.ts`: with a process grace, the
idle runner exits with `transport_reconnect_grace_exceeded` and the next
transport resumes the same provider session cold. With no grace (`null`,
today's local launch), the runner is still alive after the controller
left.
- `cd server && npx vitest run
src/services/native-runtime/native-runner-recovery-contract.test.ts`:
warm gets the remote grace; per-turn keeps none; the grace outlasts a
controller restart and a long outage.
- `tsc --noEmit` passes for `server` and `packages/paperclip-runner`.
- Local restart measurement with warm runners (K1/K2, codex, claude,
opencode, n = 3 each): every interrupted run recovered, with recovery
latency equal or lower than per-turn. The chaos waves with this change
are in the default-flip PR.

## Risks

- A warm local runner whose controller stays down for more than 7
minutes exits. A turn in progress then resumes through
`resume_dead_runner`, the same path as a controller crash that kills the
runner. Remote runners already behave this way.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted state changes. A warm runner launched by the previous release
keeps the launch arguments it started with (no grace) until it exits.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 13:11:18 -06:00
DottaandClaude Opus 5.5 68db7ed9e3 fix(recovery): never accept a dead local pid as proof for an owner on another host (#19868) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, recovery decides that an in-progress record's owner
is gone and takes it over
> - Two of those proofs use `kill(pid, 0)` on the local host. A pid that
is dead here says nothing about a process on another host or in another
PID namespace
> - So a controller or runtime-control owner that is still alive
elsewhere can be taken over while it still runs
> - This pull request records the owner's PID space (the existing
`controller-process-identity` host hash) and judges a pid only in that
same space
> - The benefit is that a different host is never proof. Those records
follow their existing lease or TTL instead

## Linked Issues or Issue Description

**Bug description**
The Phase 0 ownership inventory (finding 2) found three proofs that
accept "the pid is dead here" as proof for an owner on another host:
`evaluateNativeControllerTakeover` (`native_run_finalizations` has no
host column), the runtime-control sweep (`workspace-operations.ts`), and
the orphan reaper `terminalizeOrphanedRunningRun`. Only
`controllerProcessExited` checked the host.

**Steps to reproduce**
1. Run two control-plane processes on different hosts, or in different
PID namespaces, against one database.
2. Let host A own a native finalization lease or a runtime-control
operation.
3. On host B, run restart recovery. If A's pid number is not alive on B,
B takes over the live owner.

**Expected behavior**
An owner whose PID space differs from this process's, or cannot be read,
is unprovable. Its record waits for its lease, fence, or TTL.

Refs #19846

## What Changed

- Migration `0441`: add nullable
`native_run_finalizations.controller_host_hash`.
- `controller-process-identity.ts`: export `currentProcessSpaceHash()`
and `sameProcessSpace()`. An unreadable space is stored as `"unknown"`,
which never matches.
- `native-restart-recovery.ts`: `currentNativeControllerIdentity`
carries `hostHash`, which both coordinator claim writes
(`native-restart-recovery.ts`, `native-session-executor.ts`) store.
`evaluateNativeControllerTakeover` returns `controller_host_unprovable`
before any pid probe when the recorded space is not this one.
`native-dead-controller-result-lease.ts` selects the column.
- `workspace-operations.ts`: runtime-control owner stamps record
`hostHash`. `ownerProcessProvenGone` requires the same PID space (EPERM
= alive), otherwise the existing staleness TTL applies.
- Not in this PR: the orphan reaper `terminalizeOrphanedRunningRun`.
`heartbeat_runs` has no host for `process_pid`, and its writers are
spread across legacy and native paths. It follows as a separate PR with
its own column.

## Verification

- `npx vitest run
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/workspace-runtime-control-recovery.test.ts`: 55 passed.
New cases cover another host and an unreadable space (dead and recycled
pid) for takeover, plus another host, unreadable space, and legacy stamp
for the sweep. The sweep cases also confirm that the TTL still reclaims
after the heartbeat lapses.
- Red before the fix: the 5 new cases fail on the old code. Green after.
- `native-controller-recovery.integration.test.ts`,
`native-failed-run-owner-release.test.ts`,
`native-startup-cleanup-recovery.test.ts`: 71 passed.
- `pnpm db:generate` produced `0441_lucky_chameleon.sql` (one `ADD
COLUMN`) and pruned to five snapshots. `pnpm --filter @paperclipai/db
check:migrations` passed. Server `tsc --noEmit` shows no errors in the
changed files.

## Risks

- Previous-release story: `native_run_finalizations` rows saved before
this column have `controller_host_hash = null`. They keep the previous
local dead-pid proof for this release. Making them unprovable would
leave every native run in flight at the upgrade in `awaiting_evidence`
with no TTL, because local takeovers have no lease fallback. Rows
written by this release onward are strict. Runtime-control stamps
without `hostHash` are strict now, because that path has a staleness
TTL.
- Owners built from other receipts (workspace finalization owner,
workspace stop proof) pass no host and keep the previous behavior. Track
2's shared `OwnerStamp` should absorb them.
- A new row on a host that really moved (redeploy onto a new machine)
waits for its lease or an operator instead of being taken over by pid.
That is the intended "different host is never proof" rule.
- No new I/O before `turn.start`. The hash is computed once at module
load.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 11:14:48 -06:00
DottaandClaude Opus 5.5 a9ac2e2039 feat(native): split local runner startup into stage, launch and session-load spans (#19852) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a person messages an agent, the agent must start its turn fast.
Time to first token (TTFT) and agent boot are first-class metrics.
> - The Phase 0 baseline (#19846) found that local runner startup costs
1.6–17 s at p50 and up to 50 s at p95. It is the largest part of TTFT.
> - The run log records that startup as one span
(`runner.session.bootstrap` or `runner.session.resume`). We cannot see
if the time goes to staging, to the `paperclip-runnerd` process launch,
or to the provider session load.
> - This pull request splits local runner startup into three sub-spans
in the run log.
> - The benefit: the follow-up work to cut follow-up-turn TTFT can
measure each step before and after.

## Linked Issues or Issue Description

Refs #19846 (TTFT baseline, finding 1 and gap "No sub-spans exist inside
local runner startup"). Refs #19838.

**What existing behavior does this improve?**
The run-log performance spans for local native runner startup.

**Current behavior**
The local path records one `runner.session.bootstrap` or
`runner.session.resume` span that covers runtime staging, the runner
process launch and the provider handshake together. The remote path
already records `runner.runtime.stage` and `runner.process.launch`; the
local path does not.

**Proposed behavior**
The local path records three sequential child spans of
`runner.session.startup`: `runner.runtime.stage`,
`runner.process.launch` and `provider.session.load`.

**Reason and benefit**
The TTFT work needs to know which step dominates before it changes the
runner lifecycle. Without these spans, a change to warm runner reuse
cannot show which cost it removes.

## What Changed

- `RunnerdCodexTransport` has a new optional `onStartupStep` callback.
`#start` and `#resume` time three milestones and call it synchronously.
A failing callback never changes startup. A nested startup (exact
adoption from `thread/start`) keeps the outer clock.
- `runner.runtime.stage`: launch preparation before the runner process
exists.
- `runner.process.launch`: spawn until runnerd accepts the run
(`run.prepare`, or the recovery attachment on resume). An adopted live
runner records no launch. This keeps the existing meaning in
`summarizeStartupReuse`: a launch span means a fresh runner.
- `provider.session.load`: `session.open` (or the recovery probe) until
the provider identity is confirmed.
- `recordNativeRunnerStartupStep` (`native-run-trace.ts`) records each
step as a `run.performance.span` row with `mode` and `target: "local"`.
It uses `trace.defer`, so the run-log write never delays `turn.start`.
- The native executor passes the callback only for a local target.
Remote launchers keep their own spans, so there are no duplicates.
- `redaction.ts`: allowlist `runner.process.launch` and
`provider.session.load` as native span names.
- `doc/run-log-events.md`: new section "Local runner startup sub-steps".

Data path (AGENTS.md §5.7): run log only, plus Observability spans when
an operator sets an OTLP endpoint. The attributes (`mode`, `target`,
`outcome`) are already in the closed span-attribute allowlist. No
Telemetry change.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/live/runnerd-codex-transport-startup-steps.test.ts`: a real runnerd
and the fake Codex fixture. Bootstrap and a cold resume each report
stage, launch and session load in order, with `ok` outcomes and no
overlap. A throwing sink does not change startup.
- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/__tests__/redaction.test.ts`: the rows have the correct parent,
offsets, durations and failed level. A stalled run-log sink does not
block the caller. The rows settle before
`task.provider_session.measured`. The names survive redaction.
- The existing `runnerd-codex-transport*` suites pass (412 tests). Under
host load (load average 16–27), the first full run had 7 timing failures
in suites that do not set the callback. Each failing file passed when I
ran it again alone. `runnerd-codex-transport-settlement` passed 3/3 for
the failing case alone.
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.

## Risks

- Low risk. Without `onStartupStep`, the transport behaves as before:
the clock is not created and each milestone call is an optional-chain
no-op.
- More run-log rows: three per local runner startup.
- The step boundaries are controller-side observations.
`runner.process.launch` includes the runnerd authentication and the
`run.prepare` acknowledgement, not only the OS spawn.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 10:57:31 -06:00
DottaandClaude Opus 5.5 824b25654c fix(ui): keep a Retry for retained-file holds whose source cannot be verified (#19840) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a native run finishes but its files cannot be saved, the task
stays blocked behind a retained-file recovery hold.
> - The server records that hold as blocked and retryable, and the task
notice tells the user to "Retry to try again".
> - The task page hides its own Retry for these holds and relies on the
retained-file recovery panel instead.
> - That panel only binds the recovery record when a read-only source
inspection returns `verified`.
> - Inspection can return `blocked` for normal reasons: a sandbox that
predates ownership receipts (state saved by an older release), or a
workspace that moved on after its files were recovered.
> - Then neither surface offers an action, and "Check file recovery
status" always answers "File recovery changed. Refresh the task", also
after a reload.
> - This pull request binds the record by its own identity, uses
inspection only to gate the physical file-recovery request, and keeps
the task Retry visible whenever the panel has no control of its own.

## Linked Issues or Issue Description

Refs #18848 (the change that added the retained-file recovery panel and
hid the generic Retry for these holds).

**What happened?**

A blocked task with an active
`native_workspace_sync_out_retry_exhausted` hold showed "The saved files
need recovery before work can continue", "The recorded source is
unavailable or has changed", and "Saving the agent's files failed after
automatic retries. Retry to try again". It showed no Retry and no
recover button. "Check file recovery status" always answered "File
recovery changed. Refresh the task to inspect the current recovery", and
a reload did not change it. The hold was saved by an earlier release:
`systemRecovery` exhausted with `allocation_ownership_ambiguous` /
`service_required`, and `workspaceRecovery` blocked with `retryable:
true`. The source inspection endpoint returned `status: "blocked"` for
it. A second task whose files were already recovered showed the same
"need recovery" and "has changed" text, because its inspection reported
a reference mismatch after recovery.

**Expected behavior**

A retryable hold always shows a working Retry or recover control. A
notice that says "Retry" has a Retry button. Status checks never loop on
"changed, refresh". Recovered files show as recovered. When the source
cannot be verified, the page names the reason and who must inspect it.

**Steps to reproduce**

1. Use an active retained-export hold whose `workspaceRecovery` is
blocked and retryable, and whose `systemRecovery` is exhausted.
2. Make the source inspection (`GET
/api/issues/:id/recovery-actions?inspectSourceRunId=…&inspectSourceActionId=…`)
return `status: "blocked"`.
3. Open the task as a board operator.
4. Before this change, the page shows no Retry and no recover button.
"Check file recovery status" reports "File recovery changed. Refresh the
task".

**Paperclip version or commit**

Observed on build `8812c19`. The regressions fail on master `0f5ce5050e`
before the production edits.

## What Changed

- `WorkspaceExportRecovery`: bind the recovery record by action, run and
schema identity, not by inspection status. Inspection status now gates
only the physical "Recover saved files" request (and the request
re-checks it before POST).
- `WorkspaceExportRecovery`: report through `onRecoveryControlChange`
whether the panel offers its own control. It reports `true` only while
the source is loading or verified and the files are not yet recovered,
or while it shows the previous-server stop confirmation.
- `IssueDetail`: hide the task Retry for retained-export holds only
while the panel reports that it owns the control. Otherwise the existing
task Retry (`POST /issues/:id/retry`) shows. The server renews a bounded
recovery generation and re-verifies the source.
- Show recovered files as recovered even when inspection no longer
matches after recovery.
- Name an unverifiable source in plain words (with the code for other
reasons). State that Paperclip does not apply files it can't verify.
- Recovery text never asks a person to inspect a sandbox or environment,
and never names an owner or administrator as the one who must act. This
covers the retained-file panel, the run source inspector, the copyback
wait, and the native cleanup quarantine message. The execution blocker
and the recovery action read model describe a spent recovery budget from
its current state, so text saved by older releases ("If Retry keeps
failing, inspect the run and its environment") is not shown.
- Task Retry uses the existing `createUuid()` helper, so it works on
plain-HTTP previews without `crypto.randomUUID`. An accepted Retry also
refreshes the inspected file-recovery state. After a failed accepted
request, the unverified-source reason stays visible; it hides only while
recovery runs.
- When the recovery record was replaced, "Check file recovery status"
reloads the task detail instead of telling the user to refresh.
- Tests: component regressions for an old-release hold, a recovered hold
with a mismatched source, verified ownership, and a replaced record. A
page regression proves Retry is shown and calls `retryTask` for
unverified and stale sources. A server safety test proves that Retry for
the old-release hold shape is accepted, opens one new bounded
generation, queues file recovery again, keeps the hold active, and is
idempotent.

## Verification

- RED (master `0f5ce5050e` production files, new tests): 5 component
tests fail. The failure output shows the exact staging text: "Check file
recovery status", "File recovery changed. Refresh the task to inspect
the current recovery.", "The recorded source is unavailable or has
changed." 2 page tests (unverified and stale) fail with `expected [
'Check file recovery status', … ] to include 'Retry'`.
- GREEN: `cd ui && npx vitest run
src/components/WorkspaceExportRecovery.control.test.tsx
src/components/WorkspaceExportRecovery.test.tsx
src/components/RunWorkspaceRecoverySurface.test.tsx
src/components/ExecutionBlockerNotice.test.tsx
src/pages/IssueDetail.test.tsx` passes 320 tests.
- Server: `cd server && npx vitest run src/services/task-retry.test.ts`
passes 81 tests, including the new old-release safety case. This is
safety evidence, not a red: the server path was already correct, and the
defect was in the UI.
- Full UI suite: 11564 pass. 4 tests in `IssuesList.test.tsx` and
`CompanyEnvironments.test.tsx` failed under full-suite load and pass in
isolation (106/106). They are unrelated to this change.
- `pnpm check:token-gates` is clean. `node scripts/run-tsc.mjs -p
ui/tsconfig.json --noEmit` reports no errors in the touched files. The
remaining errors are local artifacts that were not built (plugin SDK UI
entry, generated Storybook catalog).
- Staging was read only (GET requests). The inspect request did not loop
in a fresh tab: two calls in 30 seconds. Most of the tab's request
volume is periodic company issue-list refetches, which this change does
not touch.

## Risks

- Low risk. This is a UI-only behavior change. Server authorization,
source verification, stop receipts, and the bounded recovery policy are
unchanged and remain authoritative.
- The task Retry is now visible for retained-export holds whose source
does not verify. Retry renews one recovery generation. The worker
re-runs the same source-identity checks, so Retry cannot bypass an
ownership refusal. If the refusal is permanent (for example an ambiguous
sandbox allocation), the hold returns to exhausted and the page names
the reason. An automatic fallback for such sources is follow-up work.
- Wording: a parametrized UI test renders every recovery state and
asserts that no administrator, owner, or "inspect the environment" copy
appears. A server test proves that a stored old-release message is
replaced by state-derived text. Both failed before the change.
- Previous-release story: the old-release hold shape (board-owned,
`systemRecovery` revision 5 exhausted with `service_required`,
`workspaceRecovery` blocked and retryable) is covered by component,
page, and server tests. No persisted state shape changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (repository search, file edits, shell, test
execution, and a read-only browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 10:24:03 -06:00
DottaandClaude Opus 5.5 630411af53 fix(recovery): require a 1 s start-time gap and treat EPERM as alive in owner proofs (#19848) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, recovery code decides whether the process that owns
an in-progress run is gone
> - Two proofs compare process start times to the millisecond, but `ps
-o lstart` on macOS and BSD reports whole seconds
> - So a live owner can look like a reused PID, and recovery can release
it while it still runs
> - Signal-0 handling also disagrees: one check treats EPERM as dead,
while all others treat it as alive
> - This pull request puts one start-time rule and one EPERM rule in
`controller-process-identity.ts` and uses them at every site the Phase 0
ownership inventory named
> - The benefit is that no proof releases a live owner because of clock
precision or a permission error

## Linked Issues or Issue Description

**Bug description**
The Phase 0 ownership inventory (finding 1) found inconsistent "owner
gone" proofs. `native-failed-run-owner-release.ts`
(`recordedProcessMayBeLive`, `runProcessMayBeLive`) and
`native-session-executor.ts` (`verifiedRecoveryProcessIsAlive`) treat
any start-time difference of 1 ms or more as a reused PID.
`native-restart-recovery.ts` and `controllerProcessExited` require a gap
of at least 1 s. `local-service-supervisor.ts` (`isPidAlive`,
`isProcessGroupAlive`) treats EPERM as dead.

**Steps to reproduce**
1. On macOS, record a live process start time with millisecond
precision.
2. Run the abandoned-owner release pass, or the recovered-runner
liveness check, against that live PID.
3. The pass releases the owner, or the check reports the runner as gone.

**Expected behavior**
A sub-second start-time gap is ambiguous and keeps the owner. EPERM
means the process exists.

Refs #19846

## What Changed

- `controller-process-identity.ts`: export `processExists` (EPERM =
alive) and `processStartsDefinitelyDiffer` (gap >= 1 s; a missing time
proves nothing). `controllerProcessExited` uses the new helper.
- `native-failed-run-owner-release.ts`: the controller and run-process
checks use the shared helpers instead of exact millisecond equality.
- `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` accepts
a start time within 1 s as the same runner. Its signal path uses the
same check.
- `native-restart-recovery.ts`: `definitivelyDifferentProcessStart`
delegates to the shared helper. No behavior change.
- `local-service-supervisor.ts`: `isPidAlive` and `isProcessGroupAlive`
treat EPERM as alive.

## Verification

- New `controller-process-identity.test.ts`: the 1 s boundary, EPERM and
ESRCH for every signal-0 check, and a live spawned runner whose recorded
start is 500 ms off.
- `native-failed-run-owner-release.test.ts`: two new cases (controller
and run process) with a live child and a recorded start 500 ms off. The
owner stays `observed`.
- Red before the fix: with the old call sites, all 4 new cases fail
(`releaseAbandonedNativeOwners` returns 1 and releases the live owner;
EPERM returns false; the live runner reads as gone). Green after the
fix.
- `npx vitest run src/services/controller-process-identity.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/local-service-supervisor.test.ts`: 59 passed.

## Risks

- Low risk. The change only makes proofs stricter: a sub-second gap or
EPERM now keeps an owner that the old code released. Such a row waits
for the existing lease, TTL, or real process exit.
- A recycled PID whose new process started within 1 s of the recorded
one now stays held until it exits. That is the same rule the takeover
path already used.
- Previous-release story: no persisted shape changes. Rows saved by the
previous release are evaluated with the stricter rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 10:06:35 -06:00
DottaandClaude Opus 5.5 e2d490c61b feat(observability): anchor native time to first token at the accepted wake (#19838) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start fast. The
durable ownership plan (#19831) makes time to first token (TTFT) a hard
invariant.
> - Phase 0 of that plan records a TTFT baseline per adapter and
environment, measured from the moment the message or wake is accepted.
> - Today `task.time_to_first_token` starts at the wake comment, the
answered question, or the run row's creation. A wake request that was
accepted earlier (deferred, coalesced or retried) is not part of the
measurement, and the span does not say which start it used.
> - This pull request moves the start back to the earliest accepted
input and names it on the span.
> - The benefit is that the baseline and later budgets measure what the
user waits for, with no new work before `turn.start`.

## Linked Issues or Issue Description

Refs #19831

**What is the problem?**
`task.time_to_first_token` does not include the time between an accepted
wake request and the creation of its run row. It also does not record
which input it started from.

**What is the expected outcome?**
The span starts at the earliest accepted input and carries an `anchor`
attribute: `comment`, `question_response`, `wake_request` or
`run_created`.

**Why does it matter?**
Phase 0 of the durable ownership plan sets TTFT budgets from this
baseline. A start that skips queue time hides the latency that deferred
wakes add.

## What Changed

- `resolveNativeTimeToFirstTokenStart` (pure) in `native-run-trace.ts`
picks the start and the anchor. `anchor` joins the closed span-attribute
allowlist.
- `readNativeRunWakeRequestedAtMs` reads
`agent_wakeup_requests.requested_at` for the run. It never throws.
- `native-session-executor.ts` calls it only after the first deliverable
text, so it adds no wait before `turn.start`. The span now ends at the
text commit time, not after the trace write.
- Updated `doc/run-log-events.md` and `doc/observability.md`.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-run-trace.test.ts
src/services/native-runtime/native-run-wake-anchor.test.ts` (26 pass).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t first`
(41 pass). The existing TTFT test now asserts the anchor for each
ingress.

## Risks

- Low risk. This changes only the run-log and OTLP span. It is not
Telemetry (AGENTS.md §5.7), and it does not change runtime behavior.
- One indexed primary-key read per native dispatch, after the first
text. A read failure drops the anchor and keeps the old start.
- TTFT values can grow for deferred wakes. That is the intended
correction. `startOffsetMs` can be negative when the wake request comes
before the trace start.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 09:57:30 -06:00
DottaandClaude Opus 5.5 84908d176b fix(recovery): name the failed check in native_remote_recovery_lease_mismatch (#19839) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote native runs execute in a sandbox, and recovery can reattach a
run to its original sandbox runner
> - Reattachment checks the saved lease, workspace, and runner identity
in five places. Each failed check throws the same bare
`native_remote_recovery_lease_mismatch`
> - On staging, 34 OpenCode runs on 31 tasks failed with this bare error
in one evening. The run records could not show which check failed, so
the cause could not be confirmed from the run alone
> - This pull request makes every producer name the failed step and
field with closed values
> - The benefit is that an operator and the next investigation can see
which reattachment check failed, without new logging or identifiers in
the error

## Linked Issues or Issue Description

Refs #19399, Refs #19733, Refs #19755, Refs #19779

**What happened?**
Remote same-run retries failed with
`native_remote_recovery_lease_mismatch` and the task was blocked for
reconciliation. The error is thrown by environment acquire, environment
resume, workspace preparation, workspace sync, and remote runner
verification. All of them used the same bare string. The run's `error`
field and its run events did not show which check failed.

**Steps to reproduce**
1. Start reattachment for a remote native run whose saved lease,
workspace reference, or runner identity no longer matches.
2. Read the run's `error`.
3. It shows only `native_remote_recovery_lease_mismatch`.

**Expected behavior**
The error names the step and the field that differed, with no
identifiers, and existing consumers of the code keep working.

## What Changed

- New
`server/src/services/native-runtime/remote-recovery-lease-mismatch.ts`:
the code constant, a closed `stage` and `field` vocabulary, and
`remoteRecoveryLeaseMismatch(stage, field)`. It builds
`native_remote_recovery_lease_mismatch: stage=<stage> field=<field>`.
- `environment-run-orchestrator.ts` (stages `environment_acquire` and
`environment_resume`), `heartbeat/workspace-preparation.ts`
(`workspace_preparation`), `native-workspace-sync.ts`
(`workspace_sync`), and `verifyRemoteRunnerReattachment` in
`native-session-executor.ts` (`runner_verification`) now throw the
detailed form. Each check keeps the same condition and order. Only the
message changes.
- `restart-interrupted-turn.ts` uses the shared constant. It already
matches the code exactly or as the `code: ` prefix, so automatic
continuation after a reattachment loss is unchanged. No other consumer
matches the code.
- `doc/execution-semantics.md` documents the detail format.

## Previous-release state

This change adds no persisted state and changes no saved identity. Runs
that a previous release failed keep the bare
`native_remote_recovery_lease_mismatch` in `error`. Runs failed by this
release store the detailed form. The only consumer (`isReattachmentLoss`
in `restart-interrupted-turn.ts`) accepts both forms, and
`workspace-recovery-retry-copyback.test.ts` covers both.

## Verification

- Exact-detail assertions in the producer tests:
- `server/src/__tests__/environment-run-orchestrator.test.ts`: every
scope, status, expiry, provider, lease-ID, cwd, plugin-task, and resume
case asserts its stage and field. Against the previous producer, 21 of
these cases fail (the message was the bare code). With this change, all
57 tests pass.
- `native-session-executor.test.ts`: a replaced sandbox reports
`runner_verification`/`provider_lease_id`. A missing durable runner
identity reports `identity_run_id` before any sandbox read.
- `native-workspace-sync.test.ts`,
`heartbeat/workspace-preparation.test.ts`,
`native-controller-recovery.integration.test.ts` assert their exact
detail.
- End to end on embedded PostgreSQL:
`heartbeat-native-remote-failure-retention.test.ts` records
`native_remote_recovery_lease_mismatch: stage=environment_resume
field=lease` on the run when the provider no longer resumes the lease,
and it still requests the user's Retry once.
`workspace-recovery-retry-copyback.test.ts` adds a detailed-form row
that still continues once.
- The affected server test files pass locally. `pnpm --filter
@paperclipai/server typecheck` passes.

## Risks

- Low risk. Only the error text changes. Conditions, ordering, and
fail-closed behavior are unchanged.
- Logs and run errors show a longer message. The detail has closed
values only, so it adds no identifiers or secrets.
- A first version of this PR also changed remote retry selection. Review
showed that a plain resume in a new sandbox hits the remote workspace's
unsaved-work hold, and that a `destroy` disposition does not prove the
old sandbox is gone. That change is removed. It needs a separate design.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, run in Claude
Code with tool use (shell, file editing, read-only browser access to
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 09:27:14 -06:00
DottaandClaude Opus 5.5 a94881d1a7 fix(heartbeat): save a comment refused by a temporary hold and route post-quarantine input to its own gate (#19829) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user's follow-up comment or Retry must be delivered once,
eventually, even when a temporary company admission hold (accounting
still settling) is active
> - #19741 kept claimed runs queued through such a hold and woke held
work when it cleared
> - A restart-chaos wave still lost a comment, because its wake was
refused before admission and written as a terminal skip
> - The same wave left a Retry waiting forever for a stop record of a
process that never existed
> - This pull request saves the refused comment as a deferred receipt,
and routes input after a pre-launch quarantine failure to the
quarantine's own gate
> - The benefit is that the comment is delivered after the hold, and the
task asks for the action that actually unblocks it instead of offering a
Retry that can never start

## Linked Issues or Issue Description

Refs #19741 and #19702.

**What happened?**
In a restart-chaos wave, a failed run's accounting held the company for
600 s.
1. A user commented while the task's run was live. The enqueue budget
gate refused the comment's wake before it reached the deferral behind
the live run. Because it was a user request with no saved receipt, it
was written as `skipped` (`budget.blocked`). The live run finished
without it and the task ended `done`, so the comment was lost.
2. A Retry created a run that the native cleanup-quarantine fence
refused before it requested any runner launch
(`native_session_cleanup_quarantined`, no process events). A second
Retry then waited with `process_identity_missing` ("The previous run has
no verified stop record") for a process that never existed. The task
kept offering Retry, and that Retry could never start.

**Expected behavior**
A comment refused by a temporary hold stays saved and is delivered once
later. Input after a run that was refused before launch is handled by
the quarantine gate: Retry is not offered (send a new message), and a
new message gets the verified-stop check of the previous provider
session.

## What Changed

- `heartbeat/queue.ts`: when the enqueue budget gate refuses a user's
comment wake and the block is temporary (not a reached hard stop, per
`isBoardOwnedBudgetBlock`), the wake is saved as a
`deferred_issue_execution` receipt. The receipt carries the comment ids
and an `executionWait` of `budget.blocked`, and the gate notifies the
deferred-receipt worker and throws `PersistedExecutionWaitRefusal`. It
is not written as `skipped`. Normal promotion (when the live run ends)
and #19741's hold-release wakeups deliver it.
- `native-local-process-stop.ts`: new `nativeRunRefusedBeforeLaunch`. It
returns true for a failed native run with
`native_session_cleanup_quarantined`, no recorded process, and no
launch-request or identity event.
- `explicit-native-continuation.ts`: such a run passes the
process-identity check, so the existing quarantine branch decides. A
Retry gets `cleanup_quarantined` (shown with `canRetry: false` and "send
a new message"). A new message gets
`verifyStoppedNativeSessionForContinuation`. Any launch request or
identity record keeps the original stop proofs.

- `native-session-executor.ts`
(`verifyStoppedNativeSessionForContinuation`): when the source has no
process record and no launch event, the verifier relies on the previous
owner's stop evidence. Before this change it required the source to
share the owner's process identity, so a new message stayed held even
after the owner stopped. A source that carries a process identity must
still match the owner exactly.

## Verification

Each new test fails on `master` and passes with this change, on embedded
Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: a comment
whose wake the hold refuses at enqueue is saved with a `budget.blocked`
wait and is not skipped. It is delivered exactly once after accounting
settles.
- `server/src/services/explicit-native-continuation.test.ts`:
- A run refused by the quarantine before launch routes a new message to
the quarantine's stop verification and a Retry to `cleanup_quarantined`.
These failed on `master` with `process_identity_missing`.
- Guards (green on both): a quarantined run that requested a launch, and
another failure without a process record, still require
`process_identity_missing`.
- `server/src/services/quarantined-task-retry.test.ts`: a never-launched
quarantined source is certified once the previous owner has stopped (red
on `master`). It stays held if the source requested a launch or the
owner is still running. All 31 tests pass, and the `stopped native
conversation physical cleanup` suite passes (46).
- Related suites pass locally (6 files, 620 tests): explicit native
continuation, held comment wake, queued comments routes, comment wake
batching, refused assignment wake, deferred receipt work. Two
`deferred-execution-receipt-work` tests failed once under parallel load
and passed when run alone.

## Risks

- The saved-comment path applies only to user comment wakes refused by a
temporary budget hold. Hard stops, system wakes, case handoffs, durable
chat requests, and existing saved receipts are unchanged.
- The pre-launch rule is limited to the quarantine failure code with no
launch evidence. Every other missing process identity still requires the
existing stop proof.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
kept restart-chaos instance inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 08:54:14 -06:00
DottaandClaude Opus 5.5 b2af45f0f1 fix(runner): let OpenAI API-key Codex agents generate images with their existing key (#19654) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.

## Linked Issues or Issue Description

Refs #19516

**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.

**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.

**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.

**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.

I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.

## What Changed

- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
  - The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
  - A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
  - Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.

## Verification

**Red on `master` (e482f0f632).** Behavioural red with the first version
of the new test:

```
 FAIL  src/services/native-runtime/generate-image-tool.test.ts > generate_image for OpenAI API-key Codex agents (N-12) > lets an API-key Codex agent make an image with its existing OpenAI key, saved in its workspace and billed
AssertionError: expected [ 'connections_search', …(51) ] to include 'generate_image'
 ❯ src/services/native-runtime/generate-image-tool.test.ts:78:60
      Tests  1 failed | 3 passed (4)
```

The final test file on master's source also fails. The reason is that
the module does not exist: `Cannot find module
'./generate-image-tool.js'`.

**Real pinned Codex binary.** The new test is
`packages/paperclip-runner/src/drivers/codex/codex-api-key-image-tools.test.ts`.
It uses the runner's isolated launch, the API-key `auth.json` and a
local Responses endpoint. It shows that the model gets no `image_gen`,
`imagegen` or `image_generation` tool, and that a server-advertised
`generate_image` tool does reach the model:

```
 Test Files  1 passed (1)
      Tests  1 passed (1)
```

**Green after the fix:**
`server/src/services/native-runtime/generate-image-tool.test.ts` uses
embedded PostgreSQL and a stubbed OpenAI fetch.

```
 Test Files  1 passed (1)
      Tests  11 passed (11)
```

The 11 tests cover these behaviours:

- **Main path.** The tool is advertised. The request goes to
`api.openai.com/v1/images/generations` with the existing key. The PNG is
written to the workspace. The key does not appear in the result or in
activity. One `cost_events` row is written: 50 text tokens and 1000
output tokens, which is $0.03025, estimated, from the rate card. A retry
and `recover()` replay the receipt with no second fetch and no second
cost row.
- **No binding:** the tool is not offered and `execute` refuses.
- **Budget hard-stop.** The stop pauses the agent, so the call is
refused before any fetch and no file is written.
- **Bad grant:** the call is refused before any fetch.
- **Project budget stop.** The task's project is paused by its budget,
so the call is refused before any fetch.
- **Charge settlement.** A crash left a `charged` receipt. The retry and
`recover()` record the cost once, report the image path and do not buy
another image.
- **Oversized reply.** The read stops before the body is parsed.
- **Save failure.** The spend is still recorded. A same-call retry does
not pay again and reports the failure.
- **Binding gate.** Only Codex on a direct OpenAI API key gets the
binding. A subscription, ACPX, OpenRouter or a routed connection does
not.
- **Rate card math.**
- **Provider errors.** The error text does not contain the key, and
non-PNG data is rejected.

**Neighbouring suites:**

- `src/drivers/codex` in the runner package: 24 files, 350 tests passed.
- Server: `paperclip-runner-tool-authority.test.ts`,
`native-session-resume.test.ts`, `fast-responses.test.ts`,
`ai-connection-session.test.ts` and the new suite. 157 of 158 passed.
- The one failure is `native-session-resume.test.ts › archives a damaged
prior epoch … with real runnerd` (ENOENT on `authority-epochs`). It
fails the same way on unmodified `master` in this local environment, so
this change did not cause it.
- `tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` exits 2 (broad). Required CI supplies the broad
coverage.

## Risks

- **New security surface: this needs maintainer sign-off.** The server
now sends the agent's OpenAI API key to `api.openai.com` on the agent's
request. The tool is limited in these ways:
  - The key stays in the server.
- Each call checks run, task, budget, mutation and connection authority
again.
- The destination name comes from the server and must match
`^[a-z0-9-]+-[0-9a-f]{8}\.png$`. The write is create-only and confined
to the workspace root. On macOS it uses `O_NOFOLLOW_ANY`. On Linux it
uses a `/proc` descriptor walk.
- **Spend.** Each call costs money. The tool description tells the agent
to make only what the task needs. The cost is an estimate from a rate
card (`costStatus: "estimated"`), not an invoice. The rate card applies
the image-output rate to all output tokens, so the estimate errs on the
high side.
- **Checkpoint rotation.** The tool-contract bump from v18 to v19 makes
existing native provider threads start fresh once at their next turn.
This is what the contract requires when the tool surface changes.
- **Pricing drift.** If OpenAI changes `gpt-image-2` prices, the rate
card must be updated.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, sub-agents) and extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 07:48:39 -06:00
DottaandPaperclip 909f4d1a33 fix(native): recover exact historical provider pack code (#19779) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions retain the provider pack that admitted their saved
state.
> - An image update can replace that pack's executable, proxy, or
contract.
> - Older sessions can predate the durable three-file launch artifact
receipt.
> - The current image's archive cannot satisfy a different pinned
manifest.
> - This pull request obtains the exact public code from its recorded
release image.
> - Full source, archive, and installed-file checks still control
provider admission.

## Linked Issues or Issue Description

Refs #19558, #19622, #19755.

A saved OpenCode session can fail with
`runner_remote_provider_archive_invalid` after an image update. Its
original provider state and checkpoint remain valid, but its pinned pack
differs from the controller's pack. The old manifest can name a release
that still has the exact code in GHCR. Matching the OpenCode version
does not prove matching proxy or contract bytes.

## What Changed

- Resolve a pinned historical pack from the recorded source-SHA image in
the public Paperclip GHCR repository. Verify the platform manifest, raw
config, source identity, dedicated code layer, selected pack manifest,
archive contents, paths, and hashes.
- Keep a private temporary cache of verified public code. Rebuild it on
an empty host. Bound each source to 4 GiB and three minutes, with two
active imports or cache verifications, 32 distinct queued sources, and
disk-headroom checks. Share warm-cache verification by manifest while
each caller keeps its own cancellation.
- Select this verified archive through the existing uploader. Preserve
installed-pack and preinstalled-pack checks. Refuse unavailable
historical transport before clearing the target. Keep the current-source
identity guard from #19755. Refuse an oversized layer argument before
any target effect.
- Attach the source wait to the actual per-turn startup. Stop, lease
loss, and startup completion cancel it. Keep existing Runner admission
and turn limits.
- Document the registry retention requirement and the distinction
between public code recovery and encrypted provider-state backup.

## Verification

- Genuine baseline RED: the real uploader rejects the selected old pack
before any remote effect. Treatment installs the exact old code through
that uploader.
- Current head: all 19 source/integrity/cache/deadline controls, all 45
archive controls, and 29 actual-caller and manifest/startup controls
pass (93 total). Four review regressions fail against the prior head and
pass with the fixes. The server and colocated test TypeScript graph
passes. Module-boundary and whitespace checks pass.
- Actual public GHCR cold reconstruction: 1,054,587,008 compressed code
bytes, 115.4 s, about 248 MiB maximum owner RSS. All 114 file layers and
the overlay passed source validation.
- Linux Node 24.22.2 fixture: the real historical pack passes the normal
uploader, full installed-file verifier, and pinned OpenCode version
check. It used one transfer, no application startup, no network, and no
provider calls. Revision12b0d2ba passed in 220.2 s with about 282 MiB
owner RSS. The earlier prototype passed in 108.5 s; both measurements
are retained. The three-minute source limit and the fixture's
four-minute limit were unchanged. The later review fixes add shared
cache verification, hash cancellation, and an early size guard. They do
not change extraction or the accepted Linux fixture input.
- Earlier fixture failures are retained: unsafe hardlink classification
was corrected with confined-link controls; a cached image lacked the
required zstd CLI; Docker Desktop bind storage caused GNU tar
directory-identity errors. The passing fixture uses the official Debian
zstd binary and an owned Linux tmpfs target. Production extraction code
is unchanged.
- Fresh Greptile review is 5/5 on
7b1a249478 with no open findings. All
required CI gates passed on the reviewed head. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42353011485),
[fresh
review](https://github.com/paperclipai/paperclip/pull/19779#issuecomment-6704065994).
No hosted session was retried by this change.

## Risks

- Cold historical recovery reads about 1 GiB of public code for the
measured release. Registry latency or missing images can block it. The
source deadline remains three minutes.
- GHCR administrators can delete release images. Source-SHA tags and
this temporary cache are not a permanent code-retention guarantee. A
provider-state S3 backup does not replace legacy executable bytes.
- Targets must support the existing qualified archive transport and
selected platform. Failed checks preserve the saved state and prevent
provider dispatch.
- The Linux fixture proves code installation and verification. Hosted
conversation continuation still needs acceptance after normal delivery.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and local
code execution. The host does not expose the exact deployment ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-07 02:06:02 -06:00
Nicky LeachandPaperclip beb3e55072 Continue a released Runner after confirmed graceful eviction (#19760) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core can release a completed Runner while its host retains the
session.
> - A host can later retire that Runner under its retention policy.
> - Graceful retirement saves a suspended session before physical
shutdown.
> - A cached Core owner still expects the old physical task and rejects
a new allocation.
> - This pull request retires that dead local controller after verified
stop.
> - The benefit is ordinary session continuation for Codex and qualified
Claude.

## Linked Issues or Issue Description

Refs #19611, #19683, #19196, #19733 and #19157.

After a host gracefully evicts a released task, the next run can receive
a new allocation. A cached controller rejects it with
`native_session_released_owner_scope_mismatch`. The detached-controller
path can already recover the suspended session.

## What Changed

- Require the exact source, old task and current account/workspace scope
before releasing a cached controller.
- Use the existing physical-stop receipt checks. Detach only the local
controller, then use the same suspended-state admission as the detached
path.
- Preserve the current run's instruction reservation. Retain a failed
detach for cleanup.
- Add actual Core, PostgreSQL and native Runner tests for Codex and
qualified Claude through ACPX. Route them once through the native CI
lane.

## Verification

- The original real-entry Codex case failed at the cached-task scope
check after native retirement, confirmed process exit and a recorded
stop receipt.
- The current head's native CI lane passed all 16 eviction cases for
Codex and qualified Claude. These check cached and detached owners,
current instruction preparation, reused-allocation inspection, missing
stop/state, foreign identity and non-suspended state. They verify the
same provider session, two ordinary turns, one accepted source result
and no replacement submit on refusal.
- Local checks passed the composition matrix, 12 targeted executor
controls, 43 CI lane-selection checks and server TypeScript. All eight
refusal cases now assert the expected error and whether the checkpoint
request or host read was reached. This addresses the first standard
review's test finding.
- Commands: `pnpm exec vitest run
src/services/native-runtime/plugin-runner-eviction.integration.test.ts`,
focused `native-session-executor.test.ts` controls, `node
../node_modules/typescript/bin/tsc --noEmit -p tsconfig.json`, and `node
--test scripts/__tests__/run-vitest-stable-shard.test.mjs`.
- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/42349579107)
passed on head `9a25709ef43365b1f1849413b935785838eca19b`. [Standard
review](https://github.com/paperclipai/paperclip/pull/19760#issuecomment-6703629255)
is 5/5 on this head, with no unresolved findings. Whitespace and
added-line secret/PII checks passed.

## Risks

- Controlled provider processes and an authenticated local host fixture
model the Launcher edge. This does not qualify live vendor transcript
storage, mounts or a deployment.
- This covers graceful eviction before allocation or before stopped-task
inspection during `describe`. Loss after that inspection or an ambiguous
claim remains held for reconciliation. No unavailable reply proves
physical stop.
- Missing or invalid suspended state still refuses continuation. The
deterministic refusal rule from #19733 remains unchanged. Abrupt death
without a suspended checkpoint stays with the separate #19157 work.
- No new capability, mode, schema, dependency, deployment or activation
is added. The existing draft #19157 and dependent branches remain
unchanged.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with
repository tools and code execution. The runtime does not expose a
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 23:21:45 -08:00
DottaandClaude Opus 5.5 3895195f4b fix(native): report a retained provider pack the controller does not ship as incompatible (#19755) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote OpenCode runs stage a controller-owned provider pack into the
sandbox before the provider starts
> - Since #19558, an ordinary OpenCode continuation retains the pack
that an earlier release admitted for its saved provider session
> - After a controller release changes the pack, the controller no
longer ships those bytes, and the continuation must stay blocked
> - The configured-pack upload path did not detect that case: it read
the build-time archive receipt for the old digest and failed with
`runner_remote_provider_archive_invalid`, a generic, retryable error
that suggests corrupt image files
> - This pull request compares the configured pack's identity with the
selected identity before any upload and fails as
`runner_remote_provider_artifact_incompatible`
> - It also puts a fixed reason in each real archive integrity error
> - The benefit is a correct, permanent classification that matches the
bundled-image path, no wasted retries, and a diagnosable message for
real archive corruption

## Linked Issues or Issue Description

Refs #19558, #19743

**What happened?**

On a hosted deployment, two remote OpenCode runs of a review agent
failed before the provider process started. The error was
`runner_remote_provider_archive_invalid` (classified as a generic
`native_session_interrupted`). Both runs continued an issue whose
provider session was admitted by an earlier release. In both runs, the
run log shows `provider_pack.compatibility`, `provider_pack.verify`, and
`provider_pack.verify_preinstalled` fail, then
`provider_pack.archive.verify` fails after 4–6 ms. Fresh runs of the
same agent on the same controller passed pack preparation with the
current pack.

The cause is not a cache race. The archives are created at image build
time (`Dockerfile` `runner-provider-pack` stage) and the server never
rewrites them. The retained authority selects the old pack digest.
`readProviderPackLayers` then compares the receipt's
`providerPackDigest` (current release) with that old digest and throws
`runner_remote_provider_archive_invalid`. Directory staging (no archive)
instead clears the staged remote pack, uploads the different pack, and
fails only at the final verification.

**Expected behavior**

A continuation pinned to a pack that the controller does not ship fails
as `runner_remote_provider_artifact_incompatible`, as the bundled-image
path already does ("the image provider pack did not match the bundled
release identity"). It does not clear or upload anything. Real archive
corruption keeps `runner_remote_provider_archive_invalid` with a fixed
reason that the run report shows.

**Steps to reproduce**

1. Build a provider pack and its archives for digest A.
2. Call `uploadVerifiedProviderPack` with that pack and a selected
(retained) digest B.
3. Before this change: the call fails with
`runner_remote_provider_archive_invalid` (v1 and v2 archives), or
resolves after it clears and re-uploads the remote pack (directory
source).

## What Changed

- `uploadVerifiedProviderPack` takes the configured pack's
`sourceDigest`. If it is not the selected `expectedDigest`, the call
fails with `runner_remote_provider_artifact_incompatible: selected
provider pack <12 hex> is not the controller's pack <12 hex>; ...`. The
check runs before the remote clear, the archive read, the directory hash
and any upload.
- The executor passes the configured pack's identity
(`readRemoteProviderPackIdentity(root, false)`, the same reader that
selects the current pack).
- Archive integrity errors keep the
`runner_remote_provider_archive_invalid` prefix and add a fixed reason:
`receipt_unreadable`, `receipt_identity_mismatch`, `receipt_invalid`,
`layer_invalid`, `layers_unreadable`, `archive_unreadable`,
`archive_not_file`, `size_mismatch` or `sha256_mismatch`. The run report
keeps only the outer message, so the reason must be in the message.
Filesystem causes stay on `cause` and do not put private paths in the
message.
- The v1 reader now uses the shared `verifyArchive` helper.
- `doc/execution-semantics.md` documents the rule.

## Verification

- Red before the fix (production files unchanged, new tests only): `npx
vitest run src/services/native-runtime/provider-pack-archive.test.ts`
gave 5 failed / 38 passed. The v1 and v2 cases failed with exactly
`runner_remote_provider_archive_invalid`, which is the hosted error. The
directory case resolved after it cleared and uploaded the remote pack.
The reason tests got a bare `runner_remote_provider_archive_invalid`.
- Green after the fix: the same file gave 43 passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: 1245
passed. This includes the #19558 continuation cases.
- `remote-provider-pack`, `remote-provider-pack-authority` and
`remote-provider-pack-verification` tests: 69 passed.
- `pnpm --filter @paperclipai/server typecheck`: all layers passed.
- Review follow-up (`c8156841a9`): an unreadable numbered layer now
reports `archive_unreadable`. The layered-corruption test deletes
`0.tar.zst` and checks for that reason before any transfer. The archive
file gave 43 passed.
- Hosted evidence came from read-only run, event and issue-run reads
only. No hosted retries, recovery or deploys.

## Risks

- This change does not unblock an affected session. It makes the failure
honest and permanent: `runner_remote_provider_artifact_incompatible` is
already a permanent, pre-session failure code, so the run is not retried
with the same result. To recover, the session needs its original
artifacts (the captured provider-artifact path, which #19743 repairs for
new admissions) or an authorized continuity break.
- Low risk for other paths. Fresh launches select the configured pack's
own identity, so the digests are equal. Restored provider artifacts and
compatible installed packs return before this upload path.
- Callers that match `runner_remote_provider_archive_invalid` with a
substring still match. No code compares the exact message.
- Not tested end to end against a hosted sandbox. The executor-level
wiring of `sourceDigest` has typecheck coverage but no dedicated
executor test.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic tool use with
shell, code editing, test execution and read-only authenticated browser
diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 01:10:29 -06:00
Nicky Leach 167e39c4ef fix(runner): keep warm run.attach events on ordinary accounting (#19707) 2026-11-06 23:09:57 -08:00
Devin FoleyClaude Opus 5.5PaperclipPaperclip
 		c3180398fa fix(native): resume a remote Runner after a settled suspension 
fix(native): resume a remote Runner after a settled suspension

A scheduled checkpoint retry can retain a remote reattach claim after
closing its failed predecessor cleanly suspends and exits the Runner.
Resume only that exact settled source after confirmed exit and a fresh
identity/state check. Preserve pending work and uncertain-exit fences.

Admit the last already-charged provider attempt and share one bounded
recovery deadline with the final state read, retaining closed diagnostic
labels on read failures. Cover fresh ACPX Runner relaunch on attempts
two and three through actual processes and the durable close barrier.

Six affected suites pass 1,579 tests, with 14 environment-gated skips;
all server typecheck layers pass on Node 25.6.1. The complete provider
turn continuation failure remains preserved as follow-up #19751.

Refs: #19366

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
 2026-11-06 22:39:40 -08:00
DottaandPaperclip 18b6b71d83 fix(native): refresh per-turn projections from the selected checkpoint (#19744) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Per-turn native sessions save controller checkpoints and suspended
Runner backups under separate owners.
> - A local controller projection can still name the first turn after
the next turn has saved its checkpoint and backup.
> - The old per-turn restore path selected that stale local reference
instead of the newer canonical source.
> - Admission then saw different turn identities and archived the valid
backup with the old projection.
> - This change uses the canonical selected checkpoint and the existing
finished-owner proof before it replaces a stale projection.
> - The benefit is that a third turn keeps the valid saved conversation
and backup.

## Linked Issues or Issue Description

**What happened?**

After two successful per-turn remote runs, a third turn could fail
before provider execution with unavailable provider resume state. The
selected database checkpoint and suspended backup belonged to the second
run. The local controller projection still belonged to the first run.

**Expected behavior**

The selected checkpoint must govern controller rehydration. Preserve the
older projection and the selected run's backup. Do not overwrite
current, newer, foreign, changed or unsettled ownership evidence.

**Steps to reproduce**

1. Save two real controller checkpoints for one per-turn conversation
and finish both owners.
2. Leave the first checkpoint's readable projection or external marker
beside the second run's suspended backup.
3. Admit the third run through the real canonical selector and backend
owner.
4. Before this change, admission archives the active backup with the
stale projection and leaves no active controller state.

Related: #18018 added per-turn controller restoration. #18718 refreshed
warm projections. #18846 added finished-owner retirement. This change
keeps the selected per-turn source from entering that retirement path
because of an older projection. The open warm-only change #18436
concerns a different restart branch.

## What Changed

- Resolve the canonical source before refreshing a prior per-turn
controller projection.
- Re-read it under the existing sorted coordinator locks. Replacing an
older owner reuses finished-owner checks for both runs and verifies
their full session scope. A same-source marker only rehydrates its exact
selected identity before the normal retained-sandbox verification.
- Recheck the exact initial local bytes after asynchronous reads and
lock waits. Preserve the old control-plane directory in the existing
archive, while keeping the newer backup and provider files active.
- Restore a current-run external marker only from its own checkpoint
before the existing bootstrap retry proof. A missing current checkpoint
cannot select a previous turn. Retain readable current-run, legacy
file-backed, backup integrity and provider resume checks.
- Stage the existing OpenCode proxy bytes into the controlled
fake-sandbox filesystem in the real ingress-retry fixture. Keep remote
artifact path and digest checks unchanged.
- Add real PostgreSQL/filesystem coverage for Codex and ACPX, readable
projections and external markers, missing or corrupt parts, held owners,
foreign scope, newer history and a current owner arriving during a real
coordinator-lock wait.

## Verification

- Baseline RED: the two new readable-projection cases failed against
unchanged production code because active control-plane state was absent
after admission. The real selector selected the second run. Setup
failures were separate and are not counted as RED.
- Exact final existing per-turn restore, active-owner refresh and
retained-owner wait controls: 19/19 passed in 5.91s.
- Exact final local source (`fd8d9d9021`, based on `131b566ca4`): 58/58
real PostgreSQL/filesystem and native Runner tests passed in 31.58s.
This includes both existing ingress bootstrap retries through the real
Runner and fake Codex/OpenCode providers, plus 56 focused owner cases.
Cargo validated and rebuilt current source from a private APFS clone of
an unqualified local cache; the original cache was untouched. A private
local configuration resolves workspace DB, Shared and Runner sources
without replacing behavior. The local CLI allowed a 90s setup hook; no
default timeout changed.
- Exact final full server and test-tree TypeScript check with Runner
source paths: passed. The stock fresh-worktree check first failed on
unbuilt Runner declaration entries; no product types were changed to
hide that setup gap.
- Exact final module-boundary and diff-whitespace checks: passed.
- Native-server CI RED on the previous head: both real ingress retry
cases failed because a current-run marker was returned without
hydration. The aggregate gate failed from that one underlying lane. The
correction preserves the own-checkpoint bootstrap path. The first local
OpenCode attempt hit a controller-host fixture path guard before its
intended failure; staging the same proxy bytes into the fake sandbox
made both native cases pass.
- Review regression RED: the real retained-sandbox fixture failed at
`sandbox_resumed` on the first PR head. GREEN: the corrected owner
hydrates the same-source marker and reaches the existing exact
retained-lease/suspended-runner read. A mismatched item identity still
refuses before any remote read.
- Local environment: macOS arm64, Node 26.4.0, PostgreSQL 18.1. Provider
execution is deliberately outside this admission fixture. The real
three-turn workflow still needs verification on the normally deployed
fixed source.

## Risks

- Refresh requires settled, committed, released and stopped or fenced
owners. Missing or changed evidence remains blocked.
- The refresh holds existing coordinator locks while it validates
bounded controller state. It performs no provider calls.
- The old control-plane bytes remain in the archive. A restore write
failure still fails admission; it does not reset provider state.
- Final-head full CI is green, including typecheck, build, Rust,
native-server, server and browser suites; Greptile is 5/5 with no
unresolved threads. Hosted deployment and continuity verification remain
separate acceptance steps. A successful replacement conversation is not
proof of provider continuity; hosted continuity acceptance is still
pending.

## Model Used

OpenAI Codex, GPT-6 family, with code execution and repository tools.
The host does not expose the exact backend model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-07 00:14:17 -06:00
DottaandClaude Opus 5.5 24cbea136f fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native recovery saves the qualified OpenCode launch files so a cold
allocation can resume the same session (#19622)
> - Core reads those files from the sandbox with the provider's bulk
`syncOut`
> - The launch root is under the sandbox account home, not under the
workspace remote dir
> - Daytona, Kubernetes, and CreateOS confine every export source to the
workspace remote dir, so the export always fails
> - Every remote OpenCode run on a durable-storage instance now fails
40-90 s after start with `native_session_interrupted`
> - This pull request exports a confined scratch copy of the three
launch files from inside the workspace runtime dir
> - The benefit is that remote OpenCode runs work again, and the
provider confinement guard stays unchanged

## Linked Issues or Issue Description

Refs #19622

**What happened?**

After #19622, every remote OpenCode run on Daytona fails at provider
admission. The run error is `Daytona sync source path escapes the
workspace remote dir:
/home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`.
The error code is `native_session_interrupted`. The last successful run
of the affected agent was immediately before #19622 reached the
deployment.

**Expected behavior**

Core saves the qualified launch files and the run continues. The
provider confinement guard still refuses any export source outside the
workspace remote dir.

**Steps to reproduce**

1. Configure an OpenCode agent on a Daytona environment with durable
harness storage.
2. Start a run.
3. The run fails when Core captures the provider artifacts, before the
first turn.

## What Changed

- `captureRemoteNativeProviderArtifacts` now takes the workspace
`remoteCwd`.
- Before the export, a sandbox command copies the three launch files
into a new scratch directory at
`<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`.
The command refuses a symlink or a non-regular file. It uses `cp -P`, so
it never follows a link.
- `syncOut` reads from that scratch copy. A `finally` step removes the
scratch copy after success or failure. A failed removal is logged and
does not replace the transfer result.
- The launch root does not move. OpenCode still binds to the same
absolute paths. Core still verifies each downloaded byte against the
manifest SHA-256 values.
- `.paperclip-runtime` is already excluded from workspace sync, so the
scratch copy never enters the user workspace.
- The Daytona `assertConfinedSandboxPath` guard is not changed.

## Verification

- New regression
`server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`.
It runs the real sandbox commands on a local directory tree that has a
home-rooted launch root and a separate workspace. Its `syncOut` applies
the same workspace confinement rule as the Daytona provider.
- Before the fix: it fails with `Daytona sync source path escapes the
workspace remote dir:
.../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`.
This is the staging error.
- After the fix: it passes. The published bytes match, every export
source is under the workspace runtime dir, and the scratch directory is
empty after the capture.
- Safety cases: a failed transfer removes the scratch copy and does not
publish. A launch file replaced by a symlink is refused before any
export, and nothing is published.
- `native-session-executor.test.ts`: the durable OpenCode admission case
now asserts that the capture receives `remoteCwd` and that every
`syncOut` source is under `/workspace/`.
- `pnpm vitest run` for the capture, confinement, remote OpenCode
artifact, and session root entries tests: 14 passed.
- `vitest run native-session-executor.test.ts -t "opencode|provider
artifact|durable"`: 62 passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Not run: a live Daytona run with this revision. The fix is not
deployed to staging.

## Risks

- Low risk. The capture now runs two short `sh` commands in the sandbox
and holds a second copy of the launch files during the transfer. The
files are bounded by the existing 512 MiB limit and are deleted
afterwards.
- The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`.
The Daytona image already uses these tools for the workspace sync.
- If the sandbox stops between the copy and the removal, a scratch
directory can remain under `.paperclip-runtime`. Workspace sync excludes
that directory.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, and read-only browser access to
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 00:04:21 -06:00
DottaandClaude Opus 5.5 9f0ebf6f70 fix(accounting): close a run whose turn was lost on restore, and stop succeeded runs holding all work (#19724) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budgets hold new work while the cost of completed runs is unknown
> - In a restart chaos wave, a native ACPX Claude run crossed a server
kill, resumed, and succeeded. Its usage receipt never became complete.
> - For the 10-minute closure bound, admission refused all work in the
company. This included tasks that never crossed a fault.
> - This pull request fixes the receipt for a turn that was lost on
restore. It also stops a succeeded run from holding all new work.
> - The benefit: one late usage report no longer freezes a company, and
hard stops still apply.

## Linked Issues or Issue Description

Refs #19700 (the accounting closure bound that this follows).

**What happened?**
A native ACPX Claude run crossed a server SIGKILL in the middle of a
turn. The Runner restored the session, and the run succeeded 54 seconds
after it started. All 17 usage receipts stayed incomplete. Accounting
settled the run only at the 10-minute closure bound. During those 10
minutes, admission refused all company work with "Company cannot start
work while completed runs await accounting".

**Root cause**
ACPX reports provisional usage (`runDeltaComplete: false`) until a turn
ends. The kill lost the turn that was active, so that turn never got a
complete report. The restored Runner failed that turn with
`provider_turn_lost_on_restore`. Native turn accounting
(`createNativeTurnAccounting`) required every terminal turn to have
usage, so this turn kept every later receipt incomplete. This was true
even after the continuation turn completed with full usage. The
admission gate then counted the succeeded run as a pending terminal run
for the whole company.

**Expected behavior**
The receipt closes when the run's turns end. A turn that was lost on
restore makes the usage a lower bound. A succeeded run with only usage
outstanding does not hold other work.

## What Changed

- `native-usage-accounting.ts`: a turn that failed with
`provider_turn_lost_on_restore` and has no complete usage no longer
holds the receipt open. The snapshot marks `accountingUsageLost`.
- `native-session-executor.ts` and `native-usage-delivery.ts`: such a
receipt carries the provenance note "a provider turn was lost on restore
and never reported its usage; observed usage is a lower bound".
- `anthropic-pricing.ts` and `codex-pricing.ts`: the rate-card pricers
keep a producer's provenance note.
- `native-usage-delivery.ts` and `usage-receipts.ts`: remote producer
delivery keeps a `lostOnRestore` marker on the receipt's producer event,
so replay also excuses the lost turn. Receipts saved without the marker
still replay.
- `budget-exposure.ts` and `budgets.ts`: admission waits only for
pending runs that ended without succeeding (`waitingRunCount`). A
succeeded run whose provider reported observed usage is bounded like a
running run: its observed cost counts toward spend and reservation
capacity, and its settled charge can still reach the hard stop.
- `budget-exposure.ts`: a closure that timed out because of a
control-plane restart keeps the unpriced-usage exemptions, whatever its
token count. Restart evidence is a replaced receipt source, a native
recovery transition that is not an ordinary same-run retry, or a turn
lost on restore. A second chaos run showed the need. It recovered and
succeeded, but its final usage report was lost with the restart, so it
settled as unpriced 0 cents. The blocking policy then opened a hard
incident and paused the company. A timed-out closure with no restart
evidence still counts.
- `run-cost-accounting.ts`: a lost receipt that carries a positive
provider-reported cost settles at that cost as an estimate marked as a
lower bound, not as unpriced.
- `paperclip-runner-tool-authority.ts` and `budgets.ts`: a running run's
own tool mutations are evaluated as admitted work. Hard stops and pauses
still refuse them. Accounting holds, recovering runs and budget inputs
that await evaluation do not. In a third chaos run, a live run's write
to its task document was refused during a temporary hold, so the run
proposed a blocked result.
- `doc/SPEC-implementation.md`: documents these rules.

Policy choice: a succeeded run's provider work is over, so it has less
unknown spend than a running run. Running runs never held admission.
Failed, cancelled, timed-out and interrupted runs still hold new work,
because their provider may not have stopped. A per-agent gate would
still freeze unrelated work of the same agent. A per-reservation gate
does not help when `reservationCents` is 0, which was the case in this
wave.

## Verification

- Live: a restart chaos wave ran master plus this PR (c2a48fb1e0) with
no masking. It passed 6/6, and every task crossed a restart. There were
0 accounting holds, 0 refused agent writes, 0 company pauses, and every
cost settled as estimated.

- Red then green: `native-usage-accounting.test.ts`, "closes a run whose
earlier turn was lost on restore, as a lower bound". It uses the event
sequence of the chaos run.
- Red then green: `closed-native-receipt-accounting.test.ts`, "does not
hold other work for a succeeded run with only usage outstanding". It
also checks that the hard stop still applies.
- I changed "neither pauses nor holds admission while a succeeded run's
charge commits" to the new policy. A failed run in the same state still
holds new work.
- Red then green: `closed-native-receipt-accounting.test.ts`, "never
hard-stops a blocking policy for a restart-derived timed-out receipt".
It uses the zero-token row shape of the chaos run and a case with
unpriced tokens. Without the fix, the company was paused.
- Red then green: `runner-api.integration.test.ts`, "lets a running run
write during a temporary accounting hold, but not after a hard stop".
- Red then green: `closed-native-receipt-accounting.test.ts`, "does not
hold other work for the s7504 succeeded row shape".
- Red then green: `closed-native-receipt-accounting.test.ts`, "settles a
timed-out restart receipt at its provider-reported cost, without holding
or pausing". It uses the OpenCode/OpenRouter row shape from a fourth
chaos run.
- `anthropic-pricing.test.ts`: the rate-card provenance keeps the
lower-bound note.
- Local: `closed-native-receipt-accounting`, the budget suites,
`native-usage-accounting`, `native-usage-delivery.integration`,
`native-runner-restart-recovery.integration` and `anthropic-pricing` all
pass (187 tests). Server `tsc --noEmit` is clean. `pnpm verify` asks for
CI coverage because the spec changed.

## Risks

- A succeeded run whose final charge is higher than its observed cost
can let one more task start before the charge commits. The hard stop
then pauses the scope as it does for a running run that overruns.
- A run that includes a lost turn settles with a lower-bound usage
figure. The provenance says so, and a later receipt still reconciles
through the late-receipt path.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 23:57:48 -06:00
DottaandClaude Opus 5.5 fc7e3a6503 fix(recovery): never replace a native session after a deterministic refusal (#19733) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action

## Linked Issues or Issue Description

Refs #19659, Refs #19516, Refs #19334

**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).

The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".

**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.

**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.

**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.

## What Changed

- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.

## Verification

Red before the fix, green after (each test first run against unchanged
source from `origin/master` 644ee66f5d):

- `native-session-runtime.test.ts`, test "never replaces a session after
a deterministic, non-destructive recovery refusal". Red: replacement was
attempted (`TypeError: Invalid value used as weak map key` from the stub
replacement). Green: typed error, and `openReplacementSession`,
`openSession` and `openRun` are not called.
- `harness-driver-backend.test.ts`, test "keeps the runner's closed
class of a failed recovery". Red: the typed error did not exist. Green:
both new classes are read from the command result and from the typed
error. The same message text without the type stays unclassified.
- `tests/codex_provider.rs` (real durable executor and fake Codex
app-server):
-
`attach_with_a_pending_tool_call_is_classified_as_a_pending_side_effect`.
Red: `left: None, right: Some("provider_side_effect_pending")`.
-
`saved_codex_thread_attaches_after_a_release_changes_its_feature_toggles`
(protected sandbox change). Red: `left: None, right:
Some("provider_attach_refused")`.
- `native-session-executor.test.ts`:
- Test "stops same-run retries for an operator while a saved
side-effecting call has no result". Red: `expected
'native_session_interrupted' to be
'native_provider_side_effect_pending'`.
- Test "retains the local controller checkpoint until remote continuity
archiving succeeds: failed/timeout". Red: the message had no detail.
Green: the message has the exit status and output, a token in the output
is redacted, and the local runner and control-plane files are intact.

Safety and regression checks:
- New test "still replaces a session after an unclassified resume
failure when the checkpoint allows it". It passes before and after: a
transport drop still reaches replacement.
- Existing test "types a runner-proven incompatible session when
same-session recovery is required" still passes.
- `native-session-resume.test.ts`: a source stopped with
`native_provider_side_effect_pending` is not rebound.
- `run-completion.test.ts` "replays saved user intent through normal
Retry …" passes with the detailed archive message.
- Runner Vitest: `native-session-runtime.test.ts` and
`harness-driver-backend.test.ts` (266 passed);
`runnerd-codex-transport-warm-attachment.test.ts` and
`codex-app-server-driver.recovery.test.ts` (74 passed).
- Rust: `--lib` (560 passed, plus one ACPX timing test that failed under
load and passed when rerun alone), `codex_provider` (118 passed),
`native_provider_backend` (45 passed), `durable_recovery` (3 passed).

Not run: a live staging recovery at this revision. The local server
typecheck could not resolve the unbuilt runner package, so required CI
provides typecheck.

## Risks

- A run that a replacement session would have continued after an attach
refusal now stops for an operator. The refusal repeats on every retry,
so the replacement was the only way past it. It also destroyed saved
state, which is what blocked the staging tasks. A board Retry still
continues the task in a fresh session in the same workspace. This is the
existing `native_provider_state_incompatible` behavior.
- A new failure code `native_provider_side_effect_pending` appears in
run records and recovery actions. It joins
`EXECUTION_RECONCILIATION_CAUSES`, so the UI shows the reconciliation
flow, not Retry.
- An attach refusal used to reach an automatic replacement; it now needs
one board Retry. This is intended: the replacement archived saved state
and failed part way on staging.
- The pending-side-effect class means "no result delivered to the
runner". The controller replays a recorded tool outcome only after
attach. A call whose outcome the server already recorded therefore also
stops for an operator. That is conservative; the reconciliation shows
the recorded outcome.
- Only a pending tool call classifies an attach blocker. Other blockers
(active turn, ambiguous turn start, active goal) keep their current
behavior.
- The archive error message is longer. The one exact-match consumer now
accepts the `code: detail` form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 23:25:54 -06:00
Nicky Leach b21884bdcf Release and resume retained plugin Runner sessions in Core (#19611) 2026-11-06 20:45:02 -08:00
DottaandClaude Opus 5.5 4612fa9fb4 fix(native): restart a drained Daytona run whose runner opened its session (#19711) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. A graceful shutdown (every rolling deploy) suspends live native
runs "without cancelling provider work". The next boot then decides how
each run continues.
> - #19583, #19616 and #19684 let restart recovery start a remote
bootstrap again when durable evidence proves that no provider work
exists.
> - A chaos replay found another window. The drain suspended a remote
run just after its sandbox runner opened the provider session, before
the controller sent a turn and before a session snapshot existed.
> - Recovery accepted only an unfinished `session.open` in the
controller checkpoint, so it blocked the task as ambiguous. A Retry
recovered it, but automatic continuation was lost.
> - A related replay showed that the same-run bootstrap reuses the
sandbox. The interrupted runner can still hold the runner ingress port
there (`runner_ingress_bind_conflict`).
> - This pull request proves the "session opened, no turn" case and
stops the interrupted runner before relaunch.
> - The benefit is that a deploy during Daytona startup continues
automatically, without a port conflict.

## Linked Issues or Issue Description

**What happened?**

A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run
was starting. The drain recorded `restart_suspended`. The dying
controller still launched the runner and finished `session.open`. On the
next boot, restart recovery returned
`ambiguous_remote_recovery_evidence` with disposition `blocked`.

The decrypted controller checkpoint of that run showed:
- `run.prepare` completed and `session.open` completed (provider session
opened).
- Committed events `session.started`, `session.capabilities.updated` and
`session.goal.snapshot` (`ackedSourceSeq: 3`).
- No turn command. No run-profile session snapshot and no projected
provider events.

On the next run in that replay, a same-run bootstrap relaunched the
runner in the same sandbox (one lease for all attempts). It failed with
`runner_ingress_bind_conflict: failed to bind listener: Address already
in use`, because the interrupted runner from 51 s earlier was still
listening there. The bounded same-run retry recovered it on its last
attempt.

**Expected behavior**

When the durable controller record proves that no turn was ever sent,
recovery starts the bootstrap again. It first stops the interrupted
runner in the reused sandbox.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Send SIGTERM to the control plane while the runner opens its provider
session.
3. Restart the control plane after the controller lease expires.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause: in
`server/src/services/native-runtime/native-restart-recovery.ts`, the
remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583)
required an unfinished `session.open` with no committed events, and no
provider events at all. A completed `session.open` with only
session-open events had no rule, so it fell through to
`ambiguous_remote_recovery_evidence`.

Refs #19583, #19616, #19684, #19524

## What Changed

- `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn`
proof. The state must hold only `run.prepare` and `session.open`
(pending or completed) and no warm, staged or adopted state. Every
committed event must be a session-open event, with contiguous sequence
numbers from 1, so no event is hidden. Only the controller issues turn
commands, and it writes each one to Postgres before sending it.
- `native-runner-checkpoint.ts`:
`nativeRunnerCheckpointRemoteBootstrapProof` replaces
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a
bounded number of `event:N` parts and returns which proof holds. It
still rebuilds the state in memory.
- `native-restart-recovery.ts`: a remote run can now claim
`bootstrap_incomplete` with reason `remote_session_opened_without_turn`.
Projected provider events may only be session-open events
(`harness.ready`, `session.started`, `session.resumed`,
`session.updated`), and only together with the stronger session-opened
proof. A turn or provider event still blocks.
- `native-session-executor.ts`:
- For a remote sandbox target, the executor accepts the same proof on
the restored record.
- Before the same-run bootstrap relaunches,
`stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in
the reused sandbox. It uses the verified identity marker and process
check from reattachment, which signals only a process whose identity
matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL.
- The stop is best effort. If no prior runner can be verified, nothing
is stopped, and the bounded same-run retry still handles any conflict.
- `doc/run-log-events.md`: document the reason and the stop.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "drained remote"`. Before the fix, the opened-session case (with a
projected `session.started`) returned `blocked` /
`ambiguous_remote_recovery_evidence`.
- These keep the hold: a `turn.start` command, a committed
`turn.started`, an acknowledged event that is missing from the record,
and a projected `turn.started`. The executor test restores the
checkpoint and checks that the local proof is false and the remote proof
is true.
- Regression: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460
passed, 14 skipped.
- Stale runner stop: controlled sandbox-runner tests
(`native-session-executor.test.ts`, "stale runner stop before a same-run
remote bootstrap") run the real reattachment identity and process
scripts against a fake sandbox runner:
  - a verified runner that exits on SIGTERM gets SIGTERM only;
  - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL;
  - a runner with another identity gets no signal;
  - no live prior runner means no signal.
- Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude,
`--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image
was built from current master
(`paperclip-daytona-runner@sha256:71f711ab…`), and its content id
matches this branch.
- All 4 tasks crossed a graceful restart, and all passed R1. Two of the
K2s fired on the `result` trigger, during a running remote turn.
- The K2 during a turn reattached to the live remote runner
(`reattach_remote_runner`, generation 2), as designed.
- Task t01 failed R6 with an orphan Daytona sandbox that no lease
references. The graceful restart came during sandbox creation. This is a
separate signature, outside this change.
- The timed faults did not land in the exact "session opened, no turn"
window. The integration tests cover that window with the evidence from
the saved replay database.

## Risks

- The proof relies on two facts: only the controller issues turn
commands, and the controller writes each command before sending it
(`#sendJson` waits for the checkpoint flush). A turn always leaves a
`turn.start` command in the record.
- The provider session opened in the old sandbox runner is not resumed.
The new bootstrap opens a fresh session. No turn ran in the old one, so
no work is lost.
- The runner stop sends signals inside the run's own sandbox. It signals
only after the existing reattachment identity and process checks pass.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 20:58:32 -06:00
DottaPaperclipCodex
 		18be7ac696 fix(server): retire idle native sessions before tenant sleep (#18988) 
## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - Native agents can retain a warm session between useful turns.
> - That owner must close and save its files before the tenant can
sleep.
> - A failed close must keep the owner visible until retirement
succeeds.
> - This PR prepares idle resources under the exact current admission
hold, then takes a fresh safety snapshot.
> - Busy turns, uncertain closure and changed holds still prevent sleep.
> - CI also exposed a chat authorization race. The fix repeats full
authorization after the exact delivery commits.

## Linked Issues or Issue Description

Refs #18639 for local work and incoming wake boundaries. Related #18829
covers suspension receipts after admission.

**What happened?**

Idle preflight did not retire warm Native owners. Three failed-close
paths could forget an owner without proving that its provider stopped.
CI also found a false chat rejection when a delivery changed from
processing to processed between two validation reads.

**Expected behavior**

Retire only idle owners under the exact bounded hold. Wait for closure
and file collection. Retain failed owners. Scan safety after
preparation. Keep diagnostic reads passive. A newly committed chat
delivery must pass the full current authorization checks before
attestation succeeds.

**Steps to reproduce**

Finish a warm Native turn. Start an idle drain and request safety with
its owner ID. Delay or reject provider closure, or replace the hold
during preparation or scanning. The report must keep the instance awake.
For the chat race, commit the exact delivery between the first binding
read and its fallback read.

**Paperclip version or commit**

Current head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` normally
integrates actual master `1aa0c3639714f4976cfa56f95aae6e2b7d1235ac`. The
merge preserves every unrelated master path. Fresh exact-head CI and
complete review remain required.

## What Changed

- Retain uncertain Native owners after failed restart, idle timeout and
failed-turn closure.
- Prepare Native retirement through the existing heartbeat facade. Reuse
provider closure and stopped-file collection.
- Track preparation under the exact idle owner and expiry. Reject new
activity, changed holds and expired holds before and after awaited work.
- Take the final durable and local safety snapshot after preparation.
Keep diagnostic inspection passive.
- Cover ephemeral workspace retirement and useful next use, supported
plugin drain and SDK persistence, busy owners, failed closure and hold
replacement.
- Keep dedicated recovery storage explicit. Use the same validated
options for support and storage. Probe-free helpers retain the original
implementation bodies and preserve default callers.
- Complete the Native fixture through real retirement and budget
evaluation. Keep its rejection, file, credential, accounting and strict
NONE assertions.
- Repeat the complete chat authorization once if the fallback sees the
exact batch now processed. Preserve owner locks, principal and binding
checks, denial handling and the original retry deadline.

## Verification

- The support-mode regression reproduces one failure with four passes
under the old policy, then passes all five controls. It retains prior
allocation and cleanup cases. The explicit mode does not probe embedded
PostgreSQL or use an admin alias.
- The deterministic chat race baseline reproduces two authorized
failures with all four current-authority denial controls passing. Both
Slack and Discord use real PostgreSQL and the real authorization
function. The negative setup commits before the attestation transaction;
it does not claim to revoke access through a row lock.
- The product fix passes normal own App declaration emission with no
public declaration changes, plus both current normal owning compiler
programs (399 and 611 roots). The same six controls pass after the fix.
The full original file plus these controls passes all 30 tests with no
skips.
- The original controlled Native continuation passes: one test, no
failures or skips, 1.718 seconds in its body and 23.138 seconds in its
launcher. The original 40-second case and 180-second outer bounds are
unchanged. It verifies held closure, strict NONE, an ordinary authorized
comment, saved session/files, useful output, encrypted credentials,
accepted results and settled accounting. All ten observed processes and
groups close; worker connections and listeners reach zero. A separate
post-owner audit finds zero sessions; the successful owned database is
normally removed.
- Earlier preparation, HTTP, lifecycle, workspace and plugin controls
remain source-specific evidence. They are not a claim that every
final-tree test ran locally.
- The original CI assertion log is retained. The controlled race proves
a possible cause; the original CI SQL schedule was not captured. Two
other jobs lost their self-hosted runners and provided no test verdict.
Earlier asset timeouts, readiness failures and unexplained later
recovery remain separately recorded. No timeout, failure predicate or
authority check was weakened.

Focused public commands:

```sh
pnpm exec vitest run server/src/__tests__/idle-sleep-preparation.test.ts server/src/__tests__/instance-settings-routes.test.ts --maxWorkers=1 --no-file-parallelism
pnpm exec vitest run server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts --maxWorkers=1 --no-file-parallelism
```

Local database qualification uses fresh owned test databases on existing
PostgreSQL, bounded pools and one worker. The local allocation adapter
changes only database allocation for this test module; it runs normal
migrations and client cleanup. It is not part of the published source.
No new cluster is created.

Local acceptance measured the `0a69efa6` integration with the same
committed authorization bytes. The later normal master merge adds
startup-timeout classification and recovery changes. Those paths are
outside the controlled healthy-backend witness; retirement and
authorization hunks are preserved. Exact-head remote CI at
`7f6f02b611dcf5a9a2b05173a44e9631eaf78058` has completed with 56
successful contexts and four skips. The complete current Greptile
summary is5/5 with no new actionable findings; all review threads are
resolved. Ponytail’s current turn is explicitly paused without a
current-head verdict; earlier actual findings are fixed and accounted
for. No whole final-tree local repository build/test run, live-provider
restore or deployed suspension is claimed.

Actual master subsequently advanced to
`d5c4f7dd02d408a261cedd9a1926b9900a4aedc4`. Its local normal-merge
projection with this reviewed head preserves all20 owned paths and every
unrelated master blob and mode. Current Runner TypeScript, App
declarations and their full owning compiler programs pass. Focused
Native cleanup and warm-continuation composition checks are being
completed before merge; the new upstream suspension-retry behavior is
not inferred from the older controlled-backend witness. The remote PR
head remains unchanged.

## Risks

- Privileged idle preparation can close idle warm Native sessions. It
requires the exact current hold.
- Failed closure retains its owner and can keep the tenant awake until
normal retry succeeds.
- A changed hold cannot authorize sleep. Started cleanup remains tracked
until it settles.
- The chat repair can perform one extra full authorization read. It
keeps the existing deadline and does not grant authority from the
fallback query.
- Unsupported resources and active work retain their existing blockers.
This PR does not certify every transport, plugin or deployed host.

## Model Used

OpenAI `gpt-6.1-sol` produced the delegated changes and verification.
OpenAI Codex, based on GPT-6, performed managing review and integration.
Both used repository tools and code execution. The coordinator's exact
backend snapshot and context-window limits were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green at exact head
`7f6f02b611dcf5a9a2b05173a44e9631eaf78058`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
at the exact current head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-11-06 20:49:30 -06:00
DottaandClaude Opus 5.5 30857db55e fix(native): recover a run killed between runner launch and its identity record (#19694) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd` process. The server
records the request to start it, then records its process identity after
it spawns.
> - If the server is killed between those two records, there is no
process ID to check after the restart
> - Restart recovery then restarted the bootstrap, but the executor
failed with `durable_identity_unreadable` because the runner never wrote
its state
> - The user's Retry then waited forever with "Retry is waiting for
Paperclip to verify that the previous execution stopped", because the
stop check needs a process ID
> - This pull request restarts that bootstrap and proves on the host
that no runner for the run exists
> - The benefit is that a restart in the runner launch window no longer
loses the task or leaves Retry permanently stuck

## Linked Issues or Issue Description

**What happened?**
In restart-chaos testing, an ACPX Claude run was created and the server
was SIGKILLed 1.3 s later, right after `native.process_start_requested`
and before the runner identity was recorded. On restart, recovery
claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`).
The executor then threw `runner_state_identity_mismatch:
durable_identity_unreadable`: the state root existed, but
`control-plane/control-plane-state.json` had never been written. The run
failed with `native_continuation_requires_reconciliation`. A Retry was
accepted (HTTP 200), but no run was ever created. The issue stayed
blocked with `retryWait.reason=process_identity_missing` for over 20
minutes and through another restart.

**Expected behavior**
When the runner never wrote state, no provider started, so the bootstrap
can start again. A Retry should be admitted once the host proves that no
runner for the run exists.

**Paperclip version or commit**
`master` at `fb3cf1a072`.

**Steps to reproduce**
Create a native local task and SIGKILL the server about 1 s after the
run starts, after the runner launch is requested and before its identity
is recorded. Restart the server, then click Retry.

## What Changed

- `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`):
for a `bootstrap_incomplete` restart on a local target, a root is
quarantined (kept for inspection) and the bootstrap starts again only
when both of these hold:
  - the root has no `control-plane-state.json`;
  - no live runner for the run exists on this host.

A state file that exists but cannot be read, or a runner that started
but has not connected yet, still fails closed with
`durable_identity_unreadable`.
- `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`)
proves that a runner is absent when both of these hold:
- no host process carries the run's `--run-id <runId>` argument
(`paperclip-runnerd` is always started with it, detached and
unreferenced);
- no saved control-plane state for this run exists, in the canonical
root or in quarantine.

A provider starts only after its runner connects and writes that state.
- `native.process_start_requested` now records the launching `hostname`.
- `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`)
applies the proof only when all of these hold:
  - the run is native;
  - the run has no process ID;
- its latest launch event is `native.process_start_requested`, from this
host;
  - all of its leases are local.

  A launch recorded without a host stays blocked.
- The process listing runs asynchronously with a 5 s deadline. A failed
or slow listing proves nothing.

It then writes `native.local_process_stopped` with null process IDs and
`basis: "unidentified_runner_absent"`. A later launch request
invalidates it as usual.
- The continuation and Retry gate (`explicit-native-continuation.ts`)
uses this proof before returning `process_identity_missing`. A dry run
checks the proof but records nothing.
- `doc/run-log-events.md` documents the new receipt basis.

## Verification

- `native-session-executor.test.ts`:
- The bootstrap restart with absent state was red before the fix
(`durable_identity_unreadable`) and is green after it (root quarantined,
backend admitted).
- Unreadable state, and a live runner that has not connected, still
reject.
- Proof cases: absent → true; a live process with `--run-id <runId>` →
false; this run's saved state → false; another run ID → false.
- `explicit-native-continuation.test.ts` (embedded Postgres, real
admission):
- The `absent` and `dry_run` cases were red before the fix (blocked) and
are green after it (admitted). Only the real admission records the
receipt.
- The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch`
cases stay blocked.
- Broader local run after the review fixes:
`explicit-native-continuation`, `native-session-executor`,
`native-recovered-owner-followup`,
`native-runner-restart-recovery.integration` and `task-retry` gave 1836
passed, 0 failed.
- A chaos replay of the original task (`--replay
w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after
the runner identity was recorded (`resume_dead_runner`), not in the
launch window, so the tests above are what reproduce that window.
- `tsc --noEmit` for `server` passes.

## Risks

- The proof depends on `ps -axww` listing every process on the launching
host. If `ps` fails or times out, the proof fails closed (still
blocked). Remote runners and other hosts are excluded.
- A runner that was spawned but has not yet written its state could be
mid-start when admission runs. It would still carry `--run-id` in its
arguments, so the proof refuses.
- Only the bootstrap path with no state file changes. Ownership,
checkpoint and reattach checks are unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 20:06:36 -06:00
DottaandClaude Opus 5.5 9a52a2dfeb fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs start an ACPX provider through Runner. `session.open`
opens the provider session before any prompt.
> - On a loaded host, a cold ACPX Claude `session.open` took longer than
its 30-second bound. The controller reported
`runner_local_connect_failed: PRP command session.open timed out`.
> - The server classified this as the generic
`native_session_interrupted` with `ambiguous_state`, so the task was
blocked at once. A person had to click Retry, sometimes several times.
In one restart-chaos storm, one task needed three Retry clicks, and on
another task both Retry runs timed out too.
> - The server already has a bounded same-run bootstrap retry for
failures that come before provider work (#18854). It did not recognize
this timeout. The 30-second bound also cut off normal slow startups.
> - This pull request gives cold ACPX Claude a 60-second `session.open`
bound, like Pi and OpenCode (#19487). It also retries a startup
`session.open` timeout automatically on the same run, from clean state.
> - The benefit is that a slow cold start on a busy host completes, or
retries itself, instead of waiting for a person.

## Linked Issues or Issue Description

Refs #18854 (bounded bootstrap retry for definite `session.open`
failures), #19487 (60-second cold OpenCode admission), #19665 (the
bootstrap proof that this change reuses).

**What happened?**

ACPX Claude runs failed with `runner_local_connect_failed: PRP command
session.open timed out` about 35 to 40 seconds after they started. The
coordinator recorded `native_session_interrupted` with `recoveryMode:
"ambiguous_state"` and `providerSessionEstablished: false`. The task was
blocked with a board-owned action. No provider event and no checkpoint
existed. The host 1-minute load was 17 to 44, sometimes with no restart
nearby.

**Expected behavior**

A normal slow cold start completes. A `session.open` that still times
out during runner startup, with no provider event, no checkpoint and no
turn, retries on the same run with backoff. At most two automatic
attempts follow the first. A run where provider work may have started is
never retried automatically.

**Steps to reproduce**

1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or
more.
2. Observe a run fail after about 35 to 40 seconds with `PRP command
session.open timed out`.
3. Observe the task blocked with `ambiguous_state` and a manual Retry.

**Paperclip version or commit**

`master` at `6c7cc2c5d6`.

## What Changed

The bound (evidence below):

- `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live
adoption keeps 30 seconds). The runner's Rust sidecar
`session_open_timeout` and the sidecar's provider handshake
(`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All
three bounds must agree, or the shortest one fails first. Claude's warm
run attachment uses the same cold budget, like Pi.

The automatic retry:

- `nativeSessionFailureSourceCode` returns
`provider_session_open_timeout` only for exactly `<startup failure
code>: PRP command session.open timed out`. The startup failure codes
are `runner_local_connect_failed`, `runner_direct_wss_failed` and
`runner_ingress_unavailable`. The transport uses them only before
startup completes. After startup the code is
`provider_transport_failed`, which is unchanged.
- `nativeProviderRecoveryEvidence` treats the cause as pre-session.
Durable evidence still decides: a bootstrap retry needs no provider
event (`session.started`, `turn.started` and the others) and no
checkpoint. The existing budget applies: three attempts with backoff of
about 30 seconds, then about 60 seconds. When the budget is used, the
task is blocked with the existing "Provider startup failed on 3 attempts
before any provider work began" action.
- A timeout is indeterminate: the runner can open a provider session
after the controller stops waiting. Live runs showed both leftovers, so
the retry now starts from clean state:
- `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure
settles. It quarantines a local runner root only when the
restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665)
shows that no turn began and every recorded process is gone. Any other
root, and any remote runner, stays `ambiguous_state` as before.
- `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh
open (no runner-recorded identity). If ACPX has a saved record for the
session key with no messages, no prompt and no events, it is marked
`reset_on_next_ensure`. Claude saves nothing before the first prompt, so
reusing such a record fails the first turn with
`session_resume_required`. A record that holds any conversation is
unchanged.
- A chat retry of an exhausted `provider_session_open_timeout` run is
accepted like the other pre-provider causes.
- `doc/execution-semantics.md` and the Runner README describe the cause
and the bound.

## Verification

**Measured startup distribution** (ACPX Claude
`runner.session.bootstrap` / `runner.session.startup` spans from
restart-chaos run events, 1-minute load 8 to 44):

| Source | Bound | Successful cold bootstraps (s) | Failures |
| --- | --- | --- | --- |
| Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4,
27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open`
bound plus spawn and `run.prepare`) |
| This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3,
22.8, 22.9, **37.3** | 0 `session.open` timeouts |
| This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max
25.7 | 0 `session.open` timeouts |
| This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4,
11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0
`session.open` timeouts |

Under load, successes clustered just under 30 s and failures sat just
above it, so the 30 s bound was cutting the distribution off. With 60 s,
cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would
have failed. 60 s is about 1.6 times the largest observed success, and
it matches Pi and OpenCode.

**Live restart-chaos waves** (local, ACPX Claude only, 4 in flight,
`--rc06-masked`, real provider, K1/K2 restarts):

- s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout.
The one failure is an R3 Retry signature after K1
(`native_restart_recovery_blocked`), not this cause.
- s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout.
Three Retry clicks were for `native_provider_terminal_failed`
(`provider_turn_lost_on_restore`), a different restart signature.
- Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault
injection gave every other cold startup a 1.5 s `session.open` deadline.
Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run,
and then succeeded through four K2 restarts with no person involved. Run
02799a85 hit the injection three times in a row; the budget stopped it
after attempt 3, and one Retry completed it.
- Before the two clean-state fixes, the same forced wave (s3607 and
s3608) showed the retry reusing leftover state: `turn.start` failed with
`session_resume_required`, and then "ACPX session is not open". Those
runs are the red evidence for the two clean-state fixes.
- Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8
PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for
`provider_turn_lost_on_restore`, a different restart signature.

**Tests:**

- Red before the fix: `native-startup-retry.integration.test.ts -t
timeout` (embedded PostgreSQL, real executor, real resume claim). The 3
retry cases failed with `native_session_interrupted` /
`ambiguous_state`.
- Green: `native-startup-retry.integration.test.ts` passes 36/36. New
cases:
- for each startup failure code, a timeout retries on the same run with
backoff, keeps the task `in_progress` with an agent-owned
`resume_native_run` wake, and blocks after attempt 3 with a board-owned
action;
- no automatic retry when `session.started` or `turn.started` is
durable, when the code is `provider_transport_failed`, or when another
command timed out;
- a real runner root with an opened session and no turn is quarantined;
a root with a turn command stays `ambiguous_state` and intact; a remote
runner is never prepared.
- `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed
handshake is admitted for Claude and rejected for Codex; a
never-prompted record is reset only on a fresh open; records with
messages, a prompt or events are kept.
- `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with
`createRuntimeStore` and a controlled ACP peer that, like Claude, cannot
load a never-prompted session): without the reset, a fresh open reuses
the record and the first turn fails with SESSION_RESUME_REQUIRED after
`session/load`. With the reset, the provider receives `session/new` and
the turn completes.
- `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935
tests pass.
- `native-session-executor.test.ts -t "session.open|startup|bootstrap"`:
144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`:
4 pass. `cargo test -p paperclip-runner-core --lib admission_budget`
passes.
- `npx tsc --noEmit -p server` and the runner `tsc -p
tsconfig.test.json` pass.

## Risks

- **This changes Runner Rust code. The Daytona runner image must be
rebuilt after merge** for remote sandboxes to get the 60-second sidecar
bound. Until then, a remote Claude `session.open` still fails at 30
seconds in the sidecar (as a definite failure, which already retries). A
remote timeout stays operator-owned, because a remote root cannot be
proven clean here.
- A cold ACPX Claude start that hangs now fails after about 60 seconds
instead of 30 seconds.
- A timed-out `session.open` may have created an empty provider session.
No prompt was sent. The retry quarantines that runner root and asks ACPX
for a new session.
- Same-run retries of a fresh-attempt Retry run can still fail with
`isolated_task_directory_binding_mismatch`. This is RC-35 (workspace
re-preparation), which has its own fix in progress.
- No schema or API change. The new error code appears on runs and in run
error codes.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with
tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 19:37:11 -06:00
DottaandClaude Opus 5.5 6c7cc2c5d6 fix(native): do not spend a recovery attempt resuming a cleanly drained run (#19625) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs have an execution recovery budget of three provider
attempts, so a run that keeps failing stops retrying
> - A graceful shutdown (deploy or SIGTERM) drains native runs: it
suspends them for reattachment by the next server
> - When the runner did not survive, the next boot resumed the turn and
charged one attempt
> - Restarts that the run did not cause used up its budget, so recovery
stopped and the task needed a manual Retry
> - This pull request does not charge an attempt for resuming a run that
a graceful drain suspended
> - The benefit is that deploys and graceful restarts no longer exhaust
a healthy run's recovery budget

## Linked Issues or Issue Description

**What happened?**
In restart-chaos testing, native runs reached
`execution_recovery_budget_exhausted` (3/3) only because of server
restarts. In one wave, two restarts crossed each of two runs. In a
second schedule (K1, then K2 when ready, then K2 again), each drain
correctly suspended the local ACPX Claude run, yet the fourth boot
blocked it with `native_restart_recovery_blocked`. Every resume after a
drain was classified `resume_dead_runner` and charged an attempt
(`nextNativeProviderAttempt` in `native-restart-recovery.ts`, applied by
the executor's execution claim).

**Expected behavior**
A restart that the run did not cause should not consume its recovery
budget. A graceful drain records `restart_suspended` and
`awaiting_runner_reattach`. The resume that follows should keep the
attempt count.

**Paperclip version or commit**
`master` at `4ef3fa4271`.

**Steps to reproduce**
Start a native local run. Send the server SIGTERM three times while the
run is active, with a full restart each time. The third resume is
blocked with `execution_recovery_budget_exhausted`.

## What Changed

- `suspendedByCleanDrain()` in `native-restart-recovery.ts` recognizes a
run whose most recent recovery claim is older than the drain's
`restart_suspended` entry. Ownership waits and deferred blocks between
the drain and the next claim do not erase it. The next claim consumes
it.
- A scan that waits for the old controller to exit (`awaiting_evidence`)
records one history entry per reason, so repeated scans cannot push the
drain record out of the bounded history.
- Restart claims carry `chargesProviderAttempt`. It is `false` for a
claim that follows a clean drain.
- `nextNativeProviderAttempt()` takes that flag. The executor passes it
when it takes the execution lease, so a drained resume keeps the current
attempt.
- The first execution is always counted: a run drained at attempt 0 (no
execution claimed it yet) still charges its first provider start.
- A run drained during its third attempt may continue that attempt. The
local resume and remote bootstrap gates skip the `>= 3` exhausted check
only for a non-charging resume, and the executor still refuses a fourth
attempt.
- A crash restart (no drain record) still charges one attempt.
Reattachment to a live runner still charges none.
- `doc/execution-semantics.md` states the rule next to the three-attempt
budget.

## Verification

- New test in
`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`
(embedded Postgres, real heartbeat drain and restart recovery):
- Red before the fix: the claim after the drain had no
`chargesProviderAttempt: false`.
- Green after the fix, for drained attempts 0, 1 and 3: the crash claim
charges (`true`). After the drain, a controlled backend lets the
executor take its execution lease and records the saved
`native_run_finalizations.attempt` at provider dispatch: 1, 1 and 3.
Attempt 3 is claimed instead of blocked.
- Red at the executor: when the executor ignores the flag, the drained
attempt-1 resume saves 2.
- Ownership-wait case: the drained controller (the test process) is
still alive, so two scans wait. The history keeps `restart_suspended`
followed by one `awaiting_evidence` entry. After the controller is gone,
the claim does not charge, and attempt 3 continues.
- Classifier cases (local resume and remote bootstrap): attempt 3 after
a crash is still `execution_recovery_budget_exhausted`. Both new cases
were red on the previous revision.
- `heartbeat-native-restart-suspension.test.ts` 12/12,
`native-runner-restart-recovery.integration.test.ts` and
`native-runner-ownership.test.ts` 94 passed.
- `tsc --noEmit` for `server` passes.

## Risks

- A run that itself makes the server shut down gracefully, again and
again, would no longer be stopped by this budget. Graceful shutdowns are
operator or deploy actions, so this is unlikely. A crash loop (SIGKILL,
OOM) still charges an attempt on every restart, and a resumed turn that
fails still goes through the existing failure and replacement budget.
- This changes only the attempt count. Ownership checks, checkpoint
verification and the exhausted gate are unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 18:12:12 -06:00
DottaandClaude Opus 5.5 e89be6ee83 fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs start a local `paperclip-runnerd` process. The control
plane keeps a durable state root for each session.
> - After a restart, recovery gives a run with no provider evidence the
`bootstrap_incomplete` disposition. The executor quarantines the old
root and starts the bootstrap again, but only when the root proves that
no provider work started.
> - #19524 made that proof accept a root killed after `run.prepare` and
before `session.open` (a hard kill).
> - A graceful restart (SIGTERM) leaves different shapes. The drain
sends `runner.drain` and `runner.suspend`, and the kill can land just
after `session.open` failed or just after it started a fresh session.
The proof refused both shapes. The run failed with
`runner_state_identity_mismatch: bootstrap_not_proven_incomplete`,
although no turn was ever sent.
> - This pull request extends the proof to both drain shapes and keeps
provider work fail-closed.
> - The benefit is that a graceful restart during native bootstrap
continues automatically.

## Linked Issues or Issue Description

Refs #19524

**What happened?**
Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after
the run started) failed with `adapter_failed` and
`runner_state_identity_mismatch: bootstrap_not_proven_incomplete`.
Startup recovery chose `bootstrap_incomplete`, and the executor proof
refused the state root. Two real roots were kept:

1. `session.open` failed (the ACPX sidecar timed out), then
`runner.drain` completed and `runner.suspend` completed. There was 1
connection, no committed or acknowledged event, an empty runner outbox,
and `nextSourceSeq: 1`.
2. `session.open` completed with a freshly started session (`status:
"started"`). The committed events were only `session.started`,
`session.capabilities.updated` and `session.goal.snapshot`, and the
runner outbox held one `harness.diagnostic` (the sidecar process). No
turn command existed, and the sidecar process had exited. The database
had not yet recorded the session events, so the classifier saw no
provider evidence.

**Steps to reproduce**
1. Start an ACPX Claude task on a local runner.
2. Send SIGTERM to the control plane about 10 s after the run starts,
before the first turn.
3. Let the control plane restart.

**Expected behavior**
Recovery starts the bootstrap again, and the task continues without a
user action.

**Actual behavior**
The run fails and the task stops.

## What Changed

- `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that
is pending or `failed` is now accepted, as are the runner lifecycle
commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any
state. The controller journal and the runner journal must both agree.
Any committed or acknowledged event, any turn command, a completed
`session.open`, and any queued runner event still fail closed.
- New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed
`session.open` only when all of these are true:
  - the session was freshly started (`status: "started"`, not resumed);
- the commands are only `run.prepare`, `session.open` and runner
maintenance commands;
- every committed event and every event in the runner outbox is a
session setup or runner lifecycle event;
  - `ackedSourceSeq` equals the count of committed events, with no gap;
  - the runner journal is present;
  - every recorded provider process ID has exited on this host.
- `native-session-executor.ts`: the executor applies the opened-session
proof only for a local execution target. Remote runners stay fail-closed
for that shape, because their process IDs belong to another host. The
remote checkpoint classifier still reads only commands 1 and 2.

## Verification

- Red before the fix: "allows a fresh bootstrap after a drain suspended
a runner whose session.open failed" and "allows a fresh bootstrap after
a drain that landed between session.open and the first turn" both failed
with `expected false to be true`. Green after the fix: the `native
incomplete-bootstrap evidence` block passes 32/32.
- Fail-closed counter-tests:
  - completed `session.open`;
  - committed event, acknowledged event, or queued runner event;
  - turn command, `turn.stop`, or a turn processed by the runner;
  - `turn.started` or `provider.event`;
  - resumed session;
  - compacted events;
  - no recorded provider process;
  - missing runner journal;
  - live provider process;
  - remote runner.
- I ran the new proof read-only against the three real kept roots: the
K1 root from #19524, the K2 `session.open` failed root and the K2
opened-session root. All three return `true` for a local runner. The
opened-session root returns `false` when the processes are on another
host.
- `native-session-executor.test.ts` and
`native-restart-recovery.test.ts` filtered to bootstrap, restart, remote
and drain: 333 passed.
`native-runner-restart-recovery.integration.test.ts` and
`native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit
-p server/tsconfig.json` passes.
- Live restart chaos replay: see the PR comment.

## Risks

- After a `session.open` failure or timeout, the provider sidecar might
have created an empty session. No turn command was sent, so no provider
work can repeat.
- An opened session before any turn is discarded, and a new one is
started. The proof requires a fresh `started` session, so no earlier
conversation is lost. Resumed sessions stay fail-closed.
- The opened-session proof is local only. It requires every recorded
provider process to be gone, so a live provider process blocks it.
- Out of scope: the Retry offer when the proof fails legitimately
(RC-05/06).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test execution). Default context
window. Extended reasoning was available.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 17:55:47 -06:00
DottaandPaperclip 9197abf673 fix(native): preserve controller state when remote archiving fails (#19663) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native continuity replacement must preserve the state needed to
recover saved work.
> - The current replacement moves local controller files before it knows
whether the remote archive succeeded.
> - A remote failure then leaves the local controller without its
recovery files.
> - This pull request confirms the remote archive before it moves those
files.
> - A user turn that never started can use the existing bounded Retry
path after verified preservation and cleanup.
> - The task retains its saved work and can continue once under current
user authority.

## Linked Issues or Issue Description

Refs #19638, #19547, #19399 and #19659.

The prerequisite #19638 merged as
`0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master.
A normal dependency integration at `febed950f4` keeps only the six-file
archive repair in the PR diff. Fresh master-based CI and review are
required before merge.

**What happened?**

A native continuity replacement moved its local controller journal and
provider checkpoint. The following remote archive failed. A later
recovery attempt could no longer read the local recovery files. The
first-attempt failure also had no normal Retry receipt for the saved
user turn.

**Expected behavior**

A failed or timed-out remote archive retains the local recovery files
and starts no replacement provider. A user turn that has not started can
request one normal Retry. The existing user, provider-turn, Stop,
preservation, cleanup, budget and approval gates remain in force.

**Steps to reproduce**

1. Keep a local controller journal and provider checkpoint for a
retained remote conversation.
2. Request continuity replacement and fail or time out the remote
archive command.
3. Check the local controller files before any later recovery.
4. Settle the native failure under a current user wake and recover the
saved workspace.

## What Changed

- Confirm remote continuity archiving before moving the local controller
state.
- Add the exact remote archive failure to the shared bounded user Retry
admission from #19638.
- Verify failed, timed-out and successful archive ordering with real
local files.
- Verify ordinary failure settlement and saved-file recovery produce one
fresh successor. Keep Stop, system wake and unsettled-provider holds.
- Document the file preservation and Retry contract.

## Verification

- RED: failed and timed-out remote archives removed the local controller
files. Reading those files failed with ENOENT. The saved-user Retry
request returned false instead of true.
- Initial archive unit verification: eight cases passed, including the
three archive outcomes and five existing Cloud archive safety cases.
- Initial stacked revision: `9104ee399b`, based on #19638 at
`d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`.
- Final master-based revision: `febed950f4`. All 13 affected cases
passed across three Vitest files after dependency integration. They
include the seven-mode real PostgreSQL completion case, saved-file
successor and negative-hold cases, and real-file archive outcomes. All
seven server typecheck layers passed again. Module boundary, whitespace
and added-code-comment checks passed. All required CI passed on this
exact head. The final snapshot has 59 successful checks, three intended
skips, one neutral result, and no pending or failed checks. Current-head
Greptile and the independent review are both 5/5, with no open findings.
- Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm
9.15.4 and embedded PostgreSQL. The three real-file archive outcome
cases passed. The real PostgreSQL completion test passed all seven
internal scenarios, including archive failure, prior provider activity,
system wake, Stop before and after terminal settlement, and stale
runtime promotion. Own Runner/shared/SDK builds and all seven server
typecheck layers passed. Nine saved-work continuation and negative-hold
cases passed. They verify exported file contents, one user-attributed
fresh successor, idempotent hooks, no second automatic chain, and
first-use manual Retry for held cases.
- `node scripts/check-module-boundaries.mjs`, whitespace checks and the
added-code-comment audit passed.
- Focused commands from `server/`: Vitest with `-t 'retains the local
controller checkpoint until remote continuity archiving succeeds'`; `-t
'replays saved user intent through normal Retry'`; and `-t 'continues a
user turn once|does not continue automatically'`. PostgreSQL cases used
a fresh local cluster and a 120-second setup hook budget. Application
timeouts were unchanged.
- The initial stacked head passed all test, build, typecheck and E2E
jobs. Greptile and the independent review both scored it 5/5 with no
findings. These results do not replace the fresh master-based
qualification now running.

## Risks

- A remote command can fail after it changed remote state. This change
retains local evidence in that case. It does not claim a distributed
atomic transaction.
- The exact archive error can request one normal Retry only under the
existing saved-user admission. The normal path must prove stop, saved
files and cleanup before a successor starts.
- The protected launch-argument rejection remains enforced. Its exact
hosted argument difference was not available through ordinary access.
#19659 separately addresses a saved-thread feature-policy argument
difference. This PR preserves local archive evidence and does not claim
direct hosted verification of that primary cause.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, tool use and code
execution. The model is inherited; no override was used. The exact
serving model ID and context-window size are not exposed by this
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 17:52:12 -06:00
DottaandPaperclip 0819f4ea78 fix(native): verify controller ownership before retrying saved turns (#19638) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task turns can keep a provider conversation from an earlier
run.
> - A copied conversation checkpoint names the new run before its
controller has attached.
> - A failed attachment can make recovery treat that copied identity as
current execution authority.
> - The current run needs its own durable controller checkpoint, or a
valid staged attachment, before exact resume.
> - This pull request checks that ownership and uses the normal Retry
path to continue a saved user turn after verified cleanup.
> - The task can continue once with its saved work and history.

## Linked Issues or Issue Description

Refs #19399 and #18860.

**What happened?**

A new user comment resumed a completed native conversation. The retained
runner process was gone before the new controller attached. The new run
still had a copied provider session ID. Recovery selected exact resume
for that new run and then failed its unchanged owner and lease checks.
Ordinary first-attempt failure settlement also omitted the saved-user
Retry handoff that restart recovery already uses.

**Expected behavior**

Exact resume requires current controller evidence. A saved user turn can
request one normal Retry after this infrastructure failure. The normal
path must verify provider stop, saved files, current user authority and
all execution gates before it starts a fresh session.

**Steps to reproduce**

1. Finish a native warm run and retain its provider checkpoint.
2. Admit a user comment as a new run and copy the provider checkpoint
through the real resume producer.
3. Stop the retained runner before attachment. The real remote process
check reports that the process is not running.
4. Observe that the copied provider ID selects exact current-run resume.
Observe that ordinary failure settlement saves no Retry request.

## What Changed

- Check a bounded encrypted controller header for fenced remote Runner
recovery. Keep local Runner recovery on its existing disk-backed path.
Match the run, conversation and runner identity. Preserve the existing
staged-attachment path.
- Share the existing saved-user Retry admission between restart recovery
and ordinary terminal native settlement. Keep its exact infrastructure
errors, user wake, comment, provider-turn and retry-chain checks.
- Add real PostgreSQL and stopped-process regressions. Verify malformed
and foreign checkpoint holds, current-owner resume, user authority and
Stop. Verify saved-file recovery creates one fresh-session successor.
- Document the ownership and Retry contract.

## Verification

- RED: the real checkpoint rebind producer and stopped-process check
selected exact current-run resume without a current controller
checkpoint.
- RED: ordinary settlement created zero Retry receipts for the real
native failure code; expected one.
- Review RED: a real current local controller file existed with no
database checkpoint rows. The initial gate returned ambiguous state. The
fix takes a required trusted execution-target kind and applies the
database requirement only to the remote persistence path.
- Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`,
macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL.
- Initial server Vitest: 42 selected cases passed across controller
recovery integration, remote restart adoption, ordinary run completion
and retained workspace recovery. The cases cover the stopped-process
producer, current/foreign/malformed ownership, automatic user
continuation, Stop, unsettled turns, system wakes and a manual Retry on
first use.
- `pnpm --filter @paperclipai/server typecheck`: passed with this
worktree's Runner, shared and SDK builds and all seven server layers.
- `node scripts/check-module-boundaries.mjs`, whitespace checks and the
added-code-comment audit: passed.
- Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The
real local/remote controller case passed against a fresh PostgreSQL
cluster. It verifies local disk recovery without database rows, remote
current-owner resume, and remote missing/foreign/malformed holds. Fresh
`node scripts/typecheck-layers.mjs` passed all seven server layers. The
existing offline chat reader now requests metadata only and retains its
full independent native-state safety check.
- A broader local recheck hit PostgreSQL setup/cleanup hook timeouts.
These are retained as failed setup attempts; the fresh-cluster
regression above passed. Required CI supplies broad final-head coverage.
- Settlement-order correction:
`dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL
regression first proved both failures: Stop after an already-failed row
created one Retry instead of zero; a DB-native promotion with an old
legacy caller snapshot created zero instead of one. All six settlement
scenarios now pass. The hook requires the local failed outcome and the
canonical settled native row. Fresh seven-layer server typecheck passed
again.
- Required shared fixture base integrated at
`d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills
fixture cases and all seven server typecheck layers passed. Required CI
passed for this head:
[workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592),
including all server layers, tests, build, Runner verification and E2E.
The final snapshot has 57 successful checks, three intended skips and
one neutral result. Current-head Greptile is 5/5; both concrete review
threads are resolved. The independent re-review ended without a final
response; its prior Stop finding has a real RED/GREEN repair.

The focused command was run from `server/`:

```sh
../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically'
```

## Risks

- A terminal failure of a fenced Runner adds one exact-run header read.
The read selects one encrypted header and limits inspection to 1 MiB.
Invalid or missing ownership evidence stays held.
- Saved user input can request one automatic Retry. It uses normal
preservation, current authorization, budget, approval, Stop and cleanup
gates. Unsettled provider turns and system wakes do not start an
automatic successor.
- This PR changes no provider process, credential, model or lease
authority checks.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, tool use and code
execution. The model is inherited; no override was used. The exact
serving model ID and context-window size are not exposed by this
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 17:31:23 -06:00
DottaandPaperclip 374a918d67 fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must keep admitted provider state and useful
workspace files.
> - OpenCode binds its saved session to exact executable paths and
hashes.
> - A replacement image can remove those files even when session data is
saved.
> - Retaining a manifest alone cannot restore the missing launch files.
> - This pull request saves the qualified files before provider work and
restores the same owned paths.
> - The benefit is cold continuity without relaxing the Runner guard or
replaying a prior turn.

## Linked Issues or Issue Description

Refs #19558. That fix retains the selected pack for ordinary
continuation. This change adds durable file retention for a cold
allocation. Related #19487 adjusts the admission budget; it does not
retain launch files. A search of open and closed OpenCode cold/durable
artifact PRs found no duplicate.

**What happened?**

A saved OpenCode session could outlive the image that supplied its Node
executable, proxy, and native executable. A new image supplied different
bytes. The Runner then correctly refused the changed launch profile. The
saved harness did not contain the old files.

**Expected behavior**

Core must save qualified launch files before the first turn. It must
restore those exact files under the same owned account and absolute
paths. Unknown ownership, missing files, and changed hashes must still
stop recovery.

**Steps to reproduce**

Start an admitted remote OpenCode session. Save its workspace, encrypted
checkpoint, and harness data. Destroy the original allocation and
controller files. Restore into a new allocation with an image whose
qualified proxy has different bytes. Resume the same session. The new
opt-in local qualification command automates this path with actual
OpenCode and a local synthetic model.

## What Changed

- Store company-bound, encrypted launch objects and canonical run/backup
references. Generate migration 0438 through the normal database
workflow.
- Stage qualified Node, proxy, and native OpenCode files under a private
content-addressed OS-account home. Record the account UID and exact
launch paths. Preserve the Rust profile guard.
- Gate first provider work on durable capture under the current native
claim. Bound Stop/deadline waits while the actual late transfer keeps
its worker and scratch ownership until settlement. Restore cold caches
with verified staging and atomic rename. Keep live reattachment and
historical image-rooted sessions on their existing paths.
- Reuse bounded canonical expiration owners. Atomically fence old
checkpoint authority with a strict company/run-scoped retirement marker
before external deletion. Keep its indexed cleanup anchor across failed
deletes and restart. Preserve earlier resume handles and reject later
header or source-consumer writes. Add no idle scan or network call
inside a transaction.
- Add storage, race, executor, filesystem, and actual Rust resume
regressions. Add the opt-in local Linux qualification command and
operating limits in `doc/opencode-cold-recovery.md`.

## Verification

- RED before repair: real PostgreSQL/AES cold restore lacked the launch
files; normal executor preparation chose the new image pack. Separate
regressions exposed lost upload receipts, missing cache repair,
last-owner retirement, an account-UID mismatch, and a resume-handle race
during external deletion. Each expected failure was captured before its
repair.
- Local qualification before the final master integration: 1,169
executor tests, 233 storage/archive/admission/expiration tests, all
seven server typecheck layers, migration safety, and the actual Rust
cold-resume test passed.
- The maintained Linux command passed with two cached immutable
Paperclip images whose proxy hashes differ. It removed the original
account/controller trees, restored the production published cwd under a
new lease, resumed the same actual OpenCode session, kept a useful file
and prior model context, and made one new synthetic model request. No
external model call or network was available in either container.
- The object-store fixture uses real encrypted StorageService streams
with an independent owned local directory. It is not hosted S3
acceptance or a complete task HTTP/actor workflow. Historical tasks
still require normal effects reconciliation and recovery after
deployment.
- Review regressions reproduced partial-cache publication,
resume-versus-deletion, fresh-writer and source-consumer resurrection,
and unbounded Stop/deadline waits before each repair. They now pass
through real filesystem, encrypted storage, PostgreSQL, and SDK
boundaries. File validation is split from database ownership (384/831
source lines).
- Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the
maintained actual OpenCode cold qualification passed with `dirty:false`.
It preserved the provider session and recorded launch paths, restored
useful workspace content and prior model context, and made exactly one
resumed synthetic model request. The 314,560,000-byte launch capture
took 23.118 seconds; the later harness was 833,536 bytes. Subsequent
repairs change backup-reference retirement and governed continuity-cache
classification; capture, restore, launch, and published-workspace inputs
are unchanged, so that runtime receipt remains scoped to those paths.
The continuity-break consumer is verified separately.
- Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68
affected real PostgreSQL/AES storage tests and 15/15 focused executor
authority/admission cases pass. All seven server typecheck layers pass
(source layers and unchanged test shards, with the final 590-file test
layer repeated after fixture edits). All review threads have
evidence-backed resolutions. Its subsequent CI caught the separate
lifecycle-classification gap described below. The refreshed Paperclip
review reported paused work, not a completed new review.
- Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused
root-entry/executor cases and all three source declaration layers pass.
[Greptile reviewed this exact head at
5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874).
Final-head [CI
verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339)
passes: 60 successful checks, five conditional skips, and the explicitly
neutral paused Paperclip review. No check is pending or failing.
- CI also caught the launch cache missing from the session-root
lifecycle contract. Both the writer scan and a real governed-replacement
consumer failed locally before repair: old launch bytes remained in the
active root. The cache now archives with the prior session state; the
consumer verifies exact bytes are preserved in the archive and absent
from the active root.
- The latest review found that backup retirement could delete an
independent run pin while a selected session retained the checkpoint. A
real PostgreSQL/AES regression failed before the repair. Backup-only
ownership now uses a NULL-safe exact backup predicate; the run pin
belongs to fenced checkpoint expiration. The regression verifies
checkpoint decryption and launch-file restoration after pending-backup
cleanup, followed by deletion only after removing the final resume
handle.
- An executor first-session case checks new-session durable capture
through the real capture helper and receipt propagation into the actual
harness manifest at the backup boundary. Real storage cases verify the
run-profile receipt and canonical backup/run pins.
- Sequential closure reuse and simultaneous first admission are
different scopes. A deterministic two-uploader test measures both
independent uploads, one verified ready winner, loser-only deletion, and
usable restoration for both runs. Each capture is bounded to 512 MiB and
five minutes; up to three pending receipts per run are retained. There
is no claim of global single-flight capture or a company-wide egress
bound.
- Earlier optional warm-ownership verification encountered missing local
binaries. After exact-source setup, all nine ownership assertions
passed; two other suites encountered local PostgreSQL setup/teardown
contention, so those interrupted attempts are not passes. Required
current-head CI supplies the broad final coverage. No deployment or
hosted historical recovery is claimed.

## Risks

- This adds two tables and a forward migration. The file object is
bounded to 512 MiB. Capture adds one initial transfer; later harness
archives exclude those large files. Company-scoped ready objects are
shared and retained until the last canonical owner retires. Concurrent
initial admissions can duplicate bounded transfers; each keeps
independent ownership and cancellation rather than waiting on another
controller. A continuity break retains the controller cache with its
prior state until normal session-root cleanup, adding up to one bounded
launch closure per retired state.
- Cold continuity requires the same qualified Linux platform, supported
runtime dependencies, owned account UID/home, and published absolute
cwd. A mismatch fails closed. No alias or Rust guard change is included.
- Existing image-rooted profiles have no retained file object. This
change cannot backfill missing historical binaries. A fresh governed
session must reconcile prior effects first.
- Live reattachment does not read artifact objects. Storage or
provenance failures before a new turn hold admission. They do not grant
new execution, model, tool, or replay authority.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and local browser tools. The exact served model ID and
context window are not exposed in this session. No claim is made for
unavailable model metadata.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 17:30:00 -06:00
DottaandClaude Opus 5.5 15ee6a25a6 fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.

## Linked Issues or Issue Description

**What happened?**

A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".

Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.

**Expected behavior**

When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".

Refs #19583, #16349

## What Changed

- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
c6eecac1e0, with runner inputs pinned to it, because the image-only
build sat in the Actions queue. Result: **3/3 PASS** (R1 pass; R2, R4,
R5, R6 and T pass). Each run's faults landed after the runner launched.
Recovery waited on the live remote controller lease, then continued
without a Retry. Run 3 continued through `reattach_remote_runner`. The
saved logs for runs 1 and 2 do not name their claim kind. The earlier
pre-launch window (the subject of this PR) is not hit reliably by replay
timing. The integration tests cover that window with the same evidence
shape as the saved failing database.

## Risks

- The proof depends on the executor writing
`native.process_start_requested` before a backend may spawn a runner
(`onSessionAdmission` in `executeNativeSession`). A future change that
spawns a runner earlier would weaken this proof. The fail-closed tests
pin the event as the gate.
- A fresh bootstrap acquires a new sandbox. The staged sandbox from the
dead attempt keeps its lease until run settlement releases all run
leases. It never ran a runner.
- The note change affects only settled holds with the cause
`native_restart_recovery_blocked`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 17:17:14 -06:00
DottaandClaude Opus 5.5 7941f6b051 fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model

## Linked Issues or Issue Description

**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".

**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.

**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".

## What Changed

- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.

## Verification

- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.

## Risks

- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 17:08:04 -06:00
Devin FoleyClaude Opus 5.5PaperclipPaperclip
 		ee6c101854 Reject failover backups behind the accepted event cursor (#19613) 
A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors.

Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on cd8c9cd07c are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. One e2e shard was retried after GitHub reported lost self-hosted runner communication, and the same-head rerun passed. No acknowledgement was lowered, provider replay authorized, or recovery files discarded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-11-06 13:55:43 -08:00
Nicky LeachandClaude Opus 5.5 a7d3d4b78c fix(runner): rotate local and external authority through one resumable path (#19547) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.

**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.

**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.

The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.

**Paperclip version or commit**
`master` at 35280516c0.

**Deployment mode**
Hosted, with a remote (plugin) Runner environment.

**Agent adapter(s) involved**
Codex (native Paperclip Runner). ACPX providers use the same rotation
code.

## What Changed

- **One rotation path.** `rotateAuthorityEpoch` replaces
`rotateLocalAuthorityEpoch` and `rotateExternalAuthorityEpoch`. The only
difference between Runner placements is a `RunnerStateArchive` (`read`,
`archive`):
- `localRunnerStateArchive(root)` for Runner state under `<root>/runner`
  - `#externalRunnerStateArchive()` for a plugin task or a sandbox host
- **One caller.** The transport chooses the rotation source (the live
control plane, or the latest archive), then makes one call. Locally
owned state is still quarantined on failure.
- **Committed rotation, normalized once.** Retry preparation restores
the prior run's controller checkpoint, but an earlier attempt may
already have archived that authority and its Runner state.
`takeCommittedRotation` runs right after the transport reads the control
plane. When the epoch archive for the active copy's exact identity
exists, it removes the restored copy and continues on the archive-only
path. Every later check (the warm-handoff evidence read, the
source-attachment settlement read, and the rotation) then sees the state
the rotation committed, instead of reading retired live Runner state. A
copy that still holds an unfinished warm transition, staged attachment,
or pending `run.attach` fails closed.
- **Restored controller state.** When the epoch archive exists, an
active control plane with the same schema and the exact archived
identity is a copy of authority that the rotation already retired. The
rotation removes it and resumes from the archived snapshot. This
compares identity, not bytes, because a checkpoint restore serializes
the state again and drops fields such as `ackedSourceSeq`. Any other
active state still fails closed.
- **Duplicate Runner state.** Local Runner state now follows the
computer Launcher's rules: move it, accept it already archived, drop a
byte-identical live copy beside the archive, or fail closed. The local
archive also checks that the live state is suspended before it moves it.
- **Sandbox archive script.** The script is now
`REMOTE_RUNNER_AUTHORITY_ARCHIVE_SCRIPT` and follows the same rules. It
uses `cmp` for the duplicate check and fails closed when `cmp` is not
available.
- **Sandbox host preparation after a resumed rotation.** A resumed
rotation archives the remote Runner state without preparing the host
first. When `archiveExternalRunnerState` succeeds, the server now
records the archived snapshot path. The launch's host preparation
(`inspectRemoteHarnessState`) validates that snapshot instead of the
missing live `runner-state.json`, with the same provider-state and
identity checks. Only these host preparation checks use the archived
snapshot; close checkpointing and backup-restore verification always
inspect the live `runner-state.json`. The SSH staging check treats it as
present, so retired state is not staged back.
- **Host preparation.** The external archive prepares its host only when
it reads live Runner state. Before, only one of the two caller branches
prepared the host. A resumed rotation finds the state already archived,
and preparing again could stage it a second time.

## Verification

- `npx vitest run src/live/runnerd-codex-transport.test.ts -t
"rotation"` in `packages/paperclip-runner`: 16 passed.
- External and local rotations with a byte-identical, a re-serialized,
and an other-authority restored control plane. The first two resume from
the archived snapshot. The third fails closed and keeps the copy.
- Local rotation with an identical duplicate Runner state (resumes) and
a different duplicate (fails closed and keeps it).
- The existing crash-before and crash-after remote archive retry cases.
- Before the source change, the byte-identical external case failed with
`native_runner_authority_archive_conflict`.
- `npx vitest run src/live/runnerd-codex-transport.test.ts -t "transport
retry after a committed rotation"` in `packages/paperclip-runner`: 4
passed. These go through the transport's real start path. A completed
warm handoff and a source attachment hook now resume from the archive;
before the normalization, both read the archived live Runner state and
failed. An unfinished warm transition and a pending attachment fail
closed; before, the pending attachment was silently dropped.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "remote
authority archiv"` in `server`: 6 passed. Five new cases run the real
archive script with `sh` against temporary files: a new archive, an
existing archive, an identical duplicate, a different duplicate, and no
state. The existing mocked archive test also passes.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "resumed
rotation archived"` in `server`: 2 passed. A resumed sandbox with a
retained backup goes through the real archive script and host
preparation. It failed with `runner_harness_state_mismatch:
resumed_unavailable` before the host preparation fix. A settled close
checkpoint with no live Runner state then fails closed with
`runner_remote_checkpoint_incomplete`, rather than checking the archived
snapshot. Without a completed rotation, a missing live state still fails
closed.
- `pnpm run typecheck:typescript` in `packages/paperclip-runner` and
`node scripts/typecheck-layers.mjs` in `server`: pass.
- The full `runnerd-codex-transport*.test.ts`,
`cursor-cloud-runner-launch.test.ts`, and adoption test files fail on
the same 110 tests with and without this change. These tests need the
built `paperclip-runnerd` binary, which needs `cargo`. `cargo` is not
installed locally, so CI covers them.

## Risks

- The change is only on the rotation path. A rotation that has no
archive marker behaves as before: it reads and checks live Runner state,
then archives the control plane.
- A rotation removes an active control plane only when an archive for
that exact identity exists and the active copy has the same schema and
identity. A new run's authority has a different identity, so it cannot
match.
- On a resume, the rotation now uses the archived controller snapshot,
not the caller's copy. The archive is the exact pre-rotation state. A
checkpoint restore can omit fields from it.
- An external Runner whose options define `readRunnerState` and
`archiveExternalRunnerState` but not `prepareExternalRunnerState` can
now rotate a live control plane. Before, that branch required the
prepare hook. The plugin and sandbox drivers define all three.
- The sandbox duplicate check needs `cmp`. A host without `cmp` fails
closed as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). It diagnosed the
failure from the run events, the server logs, and the on-disk controller
and Launcher state, then wrote the fix and the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 13:27:55 -08:00
DottaandClaude Opus 5.5 da9a4cb96c fix(native): retire a finished local Runner that stopped before suspending (#19597) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs each per-turn agent turn through a
runnerd process, and the next turn must prove that the previous Runner
is finished before it takes over the saved session root
> - Under load, a local turn can have its result accepted while its
`runner.suspend` command does not finish, so the Runner's own receipt
stays unsuspended
> - The retirement proof that handles the same case for sandbox Runners
refused every local Runner with `target_not_sandbox_runner`
> - So the next turn failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and its session root was
quarantined
> - This pull request extends the retirement to a local Runner when the
prior Runner process is proven stopped by exact PID and start time
> - The benefit is that a follow-up turn (for example, after a user
answers a question) continues instead of failing

## Linked Issues or Issue Description

Refs #19525 (keeps the accepted result in the same incident; this is the
next run).

**What happened?**

In the runner E2E cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
the first turn asked a question and its result was accepted. The Runner
then logged `runner did not prove durable suspension before checkpoint`
(`runner.suspend` pending, Runner lifecycle `ready`). After a server
restart and the answer, the next run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`, and its root was
quarantined.

**Expected behavior**

When the prior local Runner is provably stopped and every other
retirement proof holds, the next run retires that owner as a continuity
break and continues. If the stop cannot be proven, the run keeps the
named rejection.

**Steps to reproduce**

1. Run a local per-turn native turn whose `runner.suspend` does not
finish after its result is accepted (host load average ~40).
2. Start the next turn for the same session.
3. See `retirement_refused=target_not_sandbox_runner`.

## What Changed

- `server/src/services/native-runtime/native-restart-recovery.ts`: add
`proveNativeLocalRunnerStopped`. It uses the existing restart-recovery
checks: the recorded Runner PID is dead or now belongs to a process with
a different start time; its recorded process group is not alive; each
recorded provider process is dead or recycled. A missing PID, start time
or process group, a live recorded process, or a start time that cannot
be compared is not proof. When settlement already cleared the run's
process fields, the saved `native.local_process_stopped` receipt is the
Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus
run events) so restart recovery and this proof read provider PIDs the
same way.
- `server/src/services/native-runtime/native-session-executor.ts`
(`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a
local target is now eligible. For a local Runner, the projection accepts
the Runner's own unsuspended receipt for the same identity and
unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`) only. All other proofs are unchanged: committed
result, released lease, no recovery state, stopped controller, delivered
terminal, other commands complete, tool inputs answered, quarantine
copies. The process proof runs before retirement and again just before
the archive.
- `doc/execution-semantics.md`: describe the local case.
- Tests: seven integration cases in
`native-recovered-owner-followup.test.ts` and ten unit cases in
`native-restart-recovery.test.ts`.

## Verification

- Red before the fix: the four new integration cases failed with
`retirement_refused=target_not_sandbox_runner`, which is the incident
error.
- Green after the fix: `pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`
35/35. The new cases prove that a stopped Runner is retired (root
archived under `continuity-breaks`, no quarantine), and that a live
recorded Runner (`runner_process_alive`), a missing identity
(`runner_process_identity_missing`), and a pending non-lifecycle command
(`journal_commands_unsettled(turn.start.pending)`) keep the rejection.
All existing sandbox cases still pass, including "keeps an unsuspended
Runner receipt decisive" for sandbox targets.
- `native-restart-recovery.test.ts` unit cases: recycled PID, live
process, unknown start time, live group, live provider, missing PID or
start time.
- `native-session-executor.test.ts` + `native-restart-recovery.test.ts`:
1190 pass. `native-runner-restart-recovery.integration.test.ts` +
`native-restart-recovery-isolation.test.ts` (shared provider-PID
extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass.
- Live: `pnpm test:e2e:runner -- --max-parallel 1
--max-automatic-retries 0 --id
local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
3 runs at eec57dd5ed: 3/3 PASS. The host load average was ~15 (the
incident had ~40), and `runner.suspend` finished in all three runs, so
these runs show no regression but did not exercise the new retirement
path. The integration cases cover that path.

## Risks

- The new path lets a follow-up archive a local root whose Runner
receipt is unsuspended. This is safe only if the Runner is stopped, so
the proof uses exact PID and start-time identity and fails closed on any
uncertainty. `ps` start times have one-second precision; a sub-second
difference is treated as the same process (not proof).
- A zombie-only process group on macOS can report as alive. That keeps
the rejection, as before.
- No quarantine behavior changes. A refused retirement still quarantines
the root as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 15:11:00 -06:00
DottaandClaude Opus 5.5 586caa5f49 fix(native): hand a shutdown-interrupted startup to the next controller (#19596) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs an agent turn through a runnerd process
that the control plane owns
> - A restart of the control plane must keep the task going without a
click, if that is safe
> - A graceful shutdown closes the HTTP listener first and then drains
the heartbeat runs
> - A native run that was between provider dispatch and runnerd backend
setup in that gap failed with `local_runtime_api_listener_unavailable`,
so the drain did not see it as running and did not suspend it for the
next boot
> - This pull request makes that failure a restart detach: the run stays
running with its claim, the drain suspends it, and the next controller
adopts it
> - The benefit is that a SIGTERM in this window no longer spends a
failure and a same-run retry, and the task continues by itself

## Linked Issues or Issue Description

Refs #19530 (the same restart window, before provider dispatch).

**What happened?**

In a restart chaos replay (ACPX Claude, local runner), a
restart-recovered run got SIGTERM while
`createRunnerdBackendWithinSessionClaim` built the runnerd backend.
`resolveLocalRuntimeApiUrl()` threw
`local_runtime_api_listener_unavailable` because shutdown had already
closed the listener. The run settled `failed`. The drain then logged
`restartSuspendedRunIds: []`. The next boot used the one bootstrap
retry. A second SIGTERM hit the same window, and the task stopped at a
manual Retry.

**Expected behavior**

A run that this process cannot continue because it is shutting down is
handed to the next controller. It is not settled as failed.

**Steps to reproduce**

1. Start a native Runner task and restart the server (K1) so that
startup recovery resumes the run.
2. Send SIGTERM while the resumed run sets up its runnerd backend.
3. See the run fail with `local_runtime_api_listener_unavailable` and
the drain report no suspended runs.

## What Changed

- `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the
new `LocalRuntimeApiListenerClosedError` when this process had a
listener and closed it (the `listening` flag is false or the `close`
event fired). The error message stays
`local_runtime_api_listener_unavailable`, so the existing classifiers do
not change. A listener that never started still throws the plain error.
- `server/src/services/native-runtime/native-session-executor.ts`: for a
local Runner, the startup failure handler treats
`LocalRuntimeApiListenerClosedError` like a restart detach. It stops
lease renewal, keeps the claim and throws
`NativeControllerDetachedForRestartError`. Heartbeat already keeps such
a run `running`. A remote Runner keeps the same-run bootstrap retry from
#19462, because startup recovery has no process evidence to adopt a
claimed remote startup
(`native-runner-restart-preparation.integration.test.ts` still covers
that path).
- `doc/execution-semantics.md`: describe the new rule.
- Tests: a heartbeat regression that closes the real runtime API
listener before backend setup, then drains and restarts; a unit test for
the closed versus never-started listener; two `vi.mock` factories keep
the real module exports.

## Verification

- Red before the fix: `pnpm exec vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener`
failed with `restartSuspendedRunIds: []` and the run `failed`
(`local_runtime_api_listener_unavailable`, `bootstrap_retry`).
- Green after the fix: the same test passes. The drain suspends the run,
the run stays `running` with `awaiting_runner_reattach`, and the next
controller claims it and reaches the backend once.
- `pnpm exec vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts
src/http/server.test.ts
src/__tests__/server-startup-feedback-export.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-recovered-owner-followup.test.ts`:
1333 tests pass. One timing test in `native-session-executor.test.ts`
failed once under host load average ~30 and passed alone.
- `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass.
- Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`),
`--rc06-masked`, this branch at a35ec8b845 (before the remote-scope
commit 5dc63f24fb, which does not change the local path). The original
replay schedule no longer reached the window, because the task now ends
before the second fault. So I moved the faults earlier: K1 at 10 s, then
K2 as soon as the recovered server is ready.
- K1 + K2, 2 runs: both runs PASS with **R1 pass**. Each drain logged
`restartSuspendedRunIds: [<run>]` (before: `[]`). The next boot
classified the run `resume_dead_runner` and the same run completed. No
`local_runtime_api_listener_unavailable` failure.
- K1 + K2 + K2, 2 runs: each drain suspended the run. The fourth boot
blocked the run with `native_restart_recovery_blocked`, because three
restart claims used the three-attempt recovery budget. Retry then
completed the task (R3 pass). That budget count is a separate open item.
This pull request does not change it.

## Risks

- Low. The new path applies only after this process closed its own
listener, which happens only during shutdown. The run then waits for the
drain or the next boot, as for the other restart detaches.
- If shutdown does not reach the drain (for example SIGKILL after the
listener closes), the run keeps its live claim until the claim expires.
Then startup recovery handles it as a hard restart. This is the same as
any run that is running at a hard kill.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, GitHub CLI). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 15:10:44 -06:00
DottaandClaude Opus 5.5 d728be8fbf fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. The local control plane keeps the runner's controller state in
an encrypted Postgres checkpoint.
> - When the control plane restarts, restart recovery classifies each
live native run and decides to reattach, resume, bootstrap again, or
hold the task.
> - A restart chaos wave killed the control plane twice while a Daytona
runner was connecting. The run was claimed and the sandbox runner was
launched, but `session.open` never completed. No provider evidence
existed.
> - Recovery waited correctly for the old 60 s controller lease to
expire. Then the classifier had no rule for a claimed remote bootstrap.
It returned `ambiguous_remote_recovery_evidence` and blocked the task as
"Workspace repair required".
> - The controller writes each command to the Postgres checkpoint before
it sends it. After the lease expires, that checkpoint is the complete
record of what reached the sandbox runner.
> - This pull request lets recovery prove from that checkpoint that no
provider work exists, and then start the same-run bootstrap again.
> - The benefit is that a restart during Daytona runner startup
continues automatically instead of stopping the task.

## Linked Issues or Issue Description

**What happened?**

A restart during Daytona remote runner startup (native ACPX Claude, two
SIGKILLs about 20 s apart) left the task blocked. The run history showed
`remote_controller_lease_live` twice, then
`ambiguous_remote_recovery_evidence` with disposition `blocked`. The
task showed "Workspace repair required"
(`native_restart_recovery_blocked`) and did not continue.

**Expected behavior**

When the durable controller record proves that the runner received only
bootstrap commands, recovery continues the run with a fresh bootstrap.
Any evidence of provider work keeps the hold.

**Steps to reproduce**

1. Start a native run on a Daytona environment.
2. Kill the control plane after the sandbox runner connects and before
`session.open` completes.
3. Restart the control plane and wait for the old controller lease to
expire.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.

Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`
`classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only
an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a
run with a provider checkpoint. A claimed bootstrap with no provider
evidence fell through to `ambiguous_remote_recovery_evidence`.

The decrypted checkpoint of the failed run held `connectionCount: 1`,
`run.prepare` delivered once and pending, `session.open` pending and
never delivered, no committed events, and `ackedSourceSeq: 0`.

Refs #16349, #19524

## What Changed

- `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates
from the executor into a small pure module. Restart recovery and the
executor now use the same proof. No behavior change.
- `native-runner-checkpoint.ts`: add
`nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a
checkpoint with the bootstrap parts (header, cursor, `command:1`,
`command:2`), checks the run, runner instance, and session identity, and
applies the shared proof. A remote runner journal is not on this host,
so the proof uses the controller record alone.
- `native-restart-recovery.ts`: for a classic sandbox run whose owner
lease has expired, with a durable checkpoint, no session checkpoint, no
provider evidence, and no staged attachment, claim
`bootstrap_incomplete` with the reason
`remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At
three provider attempts, block with
`execution_recovery_budget_exhausted`.
- `native-session-executor.ts`: add
`restoreRemoteBootstrapCheckpointForRecovery`. For a
`bootstrap_incomplete` recovery on a sandbox target, it restores the
Postgres checkpoint over the local external-state marker. The existing
incomplete-bootstrap proof then checks the restored state before it
quarantines the root and bootstraps again.
- `doc/run-log-events.md`: document the new disposition reason.

## Verification

- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "remote bootstrap"`. Before the fix, the proven case and the budget
case returned `blocked` / `ambiguous_remote_recovery_evidence`.
- New cases: the proven claimed bootstrap gets `bootstrap_incomplete`
(attempt kept, generation +1, run still `running`, no recovery action).
The hold stays for a queued `turn.start`, an acknowledged event, adopted
state, no checkpoint, and a spent budget. The executor helper restores
the checkpoint only for a bootstrap recovery on a classic sandbox and
only over a marker for the same run.
- Regression set: `native-runner-restart-recovery.integration`,
`native-session-executor`, `native-restart-recovery`,
`native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed,
14 skipped.
- Live Daytona replay of the failing chaos task (double SIGKILL during
remote runner startup), 3 runs: REPLAY_RESULTS

## Risks

- A fresh bootstrap acquires a new sandbox. The old sandbox keeps its
lease until run settlement releases all run leases. The old runner gets
no turn command, because only the controller issues one and the new
controller never connects to it.
- If `session.open` reached the old runner, the old provider session can
exist without a turn. No provider work or tool call can happen in it.
- Any checkpoint shape other than the bootstrap-only shape keeps the
previous fail-closed hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 14:43:22 -06:00
DottaandPaperclip b55d8a9201 fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558) 
## Thinking Path

> - Paperclip manages AI agents and preserves their work across runs.
> - Native OpenCode turns can restore a saved provider session after the
prior daemon stops.
> - The Rust provider binds that saved state to exact qualified launch
paths and artifact bytes.
> - Ordinary continuation selected the current provider pack, although
live reattachment retained the admitted pack.
> - A new proxy or a diagnostics setting could therefore replace the
launch profile behind saved state.
> - This pull request retains the recorded pack for ordinary OpenCode
continuation with a verified durable binding.
> - The benefit is compatible continuation without weakening artifact
integrity or granting replay authority.

## Linked Issues or Issue Description

Refs: #17512, #18237. These added saved authority and compatible pack
reuse. This change covers ordinary OpenCode continuation.

**What happened?**

An ordinary remote OpenCode continuation could select a newer pack while
restoring its existing provider session. The Rust recovery guard then
rejected the changed launch profile. The same error can occur after a
controller release changes the bundled proxy.

**Expected behavior**

A continuation with a valid saved binding and pack receipt uses the same
admitted artifacts. If those artifacts are unavailable, it holds the
saved state instead of selecting another pack or creating a provider
session.

**Steps to reproduce**

1. Admit a remote OpenCode session with a qualified protected provider
pack.
2. Persist its pack receipt and suspended durable Runner state.
3. Change the controller bundle to a valid pack with a different proxy
digest.
4. Start an ordinary continuation with diagnostics enabled.

The new executor regression fails before the repair because selection
rejects the available original pack in favor of the new bundle. It
passes after the repair.

## What Changed

- Retain the saved provider pack for ordinary OpenCode continuation with
a verified durable session binding.
- Keep provider-pack retention independent of surviving-daemon identity:
a new daemon records its current binary, and an authorized continuity
break restores normal compatible pack selection.
- Test retained pack selection after a proxy upgrade and fail closed
when only a newer pack is available.
- Test the real Rust executor with changed, valid proxy file bytes. It
rejects drift without changing saved state and accepts the original
artifacts again.
- Document the selection rule and its limits.

## Verification

- Initial prevention focused Vitest: 220 passed across native session
execution, pack authority, pack verification and pack preparation. The
executor regression was observed failing before the production change.
- Rust native provider tests: all five OpenCode tests passed. These
include the existing drift guard, actual changed proxy bytes, original
artifact restoration and restart-lost turn behavior.
- Advisory review follow-up: reproduced stale daemon identity in the
actual executor and incompatible pack selection after a governed
continuity archive. Both fail before their corresponding repairs and
pass after them. Added an executor regression for a missing authority
receipt; it remains at the current release integrity gate without
creating a new receipt.
- Updated head `639c080856` integrates master `17901d74ba` by normal
merge. Affected Vitest: 202 passed across four files, including all new
review regressions; all seven server typecheck layers passed. Exact-head
CI and fresh Greptile/Paperclip review are pending. The prior head had
57 successful checks plus successful Snyk and Greptile 5/5; those
results do not qualify this revision.
- Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4
dependencies, controlled remote command boundary and real Rust provider
executor. No paid provider calls or hosted retries.
- This is a bounded prevention change. It does not deliver historical
provider artifacts to a replacement sandbox or replacement controller.
Missing saved receipts or unavailable old artifacts remain a separate
cold-recovery gap. No full hosted recovery is claimed.

## Risks

- A continuation now requires its exact recorded pack when it has a
durable binding. A newer installed pack cannot substitute for missing
historical bytes.
- Fresh launches, other providers and explicit continuity replacement
keep their existing paths. Current ownership, company, model, permission
and receipt binding checks remain in effect.
- The repair does not relax the Rust launch-profile guard, replay prior
tool effects, or authorize a new provider conversation. Existing
published review receipts remain history.

## Model Used

OpenAI Codex, GPT-6 family. The host does not expose the exact model ID
or context window. Capabilities used: reasoning, code editing, shell
execution, tests and authenticated read-only browser diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 14:21:17 -06:00
Nicky LeachandPaperclip 1391f64551 Add gated release/resume methods to plugin Runner owners (#19584) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core owns Runner tasks through a lease-bound plugin owner.
> - The SDK already defines release and resume in #19468.
> - The owner still expects an accepted result for both operations, so
it cannot use those SDK results.
> - This pull request adds both methods behind the existing capability,
warm-task and account checks.
> - It uses the SDK parser to verify the result and task ID, and
preserves the owner's cancellation checks.
> - This prepares the owner boundary for later lifecycle integration. It
does not start automatic release or resume.

## Linked Issues or Issue Description

**Problem or motivation**

The plugin Runner owner cannot forward the existing SDK release/resume
pair. Its local result mapping rejects their valid results. Later
lifecycle code needs both operations on the same checked task owner.

**Proposed solution**

Pass the existing description capability to the owner. Add explicit
release and resume methods and use the SDK's operation-result parser.
Require warm-task support and a selected account. Reject calls and late
receipts after owner cancellation or detachment.

**Alternatives considered**

Adding automatic release here would also require connection quieting,
settled-event drain, pending-ticket retry ownership and safe disposal.
Keep those changes together in the next integration. Do not add a new
flag, schema, timer or lifecycle mode here.

Related: #19468. I checked ROADMAP.md and found no duplicate planned
item. An open-PR title search for release/resume found no overlapping
pull request.

## What Changed

- Add the capability-gated owner facade for Codex and ACPX. Resume
returns only `resumed` or `unavailable`.
- Use the SDK parser for exact task-ID and operation-result validation.
- Pass `releaseResumeVersion` from the checked description and test that
it does not cause an automatic operation.
- Cover account-preserving owner rebind, missing prerequisites,
malformed results, ambiguous errors, cancellation and late replies.
- Document the caller's remaining settlement, connection, ticket and
attachment duties.

## Verification

Passed locally on Node 26.12.0 with the pinned workspace dependencies:

- In `server/`: `pnpm exec vitest run
src/services/native-runtime/plugin-runner-task.test.ts` — 74 tests
passed.
- In `server/`: `pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t 'uses a
task-only'` — 3 cases passed; 1,160 unrelated cases were not selected.
- Built the real workspace SDK dependencies and Runner TypeScript. A
focused strict TypeScript program for the owner and its tests passed.
- `git diff --check` and the added-line secret/PII scan passed.

The first root test command selected a nonexistent Vitest project name
and ran no tests. Running the test from the server package fixed the
command. The first focused TypeScript setup could not find Node types;
pointing it to the package's installed type definitions fixed that
setup. The full regression suite and release checks run in GitHub.

## Risks

The facade has no automatic caller. Existing warm timeout, bootstrap,
outbound connection and pool behavior stay in place. No plugin
capability is advertised or task control channel enabled by this pull
request.

Future integration must serialize release with new work and disposal. It
must quiet the owned connection, keep the exact pending ticket after an
ambiguous reply, authenticate the source again and finish ordinary run
attachment before new work. This change does not qualify provider
parking, cold recovery, retention policy or deployment.

## Model Used

OpenAI GPT-6.1 Sol, exact model ID `gpt-6.1-sol`, High reasoning. Used
code execution and targeted tests. The context-window size was not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-06 11:42:32 -08:00
DottaandClaude Opus 5.5 07ddf9a5c2 fix(native): keep accepted results when runner close cannot prove suspension (#19525) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs persist an accepted result first. The Runner then closes
and suspends its session.
> - Under heavy host load, the Runner close can fail to prove a durable
suspension after the result is accepted.
> - The executor then kept its finalization lease. The heartbeat
finalizer got `native_finalization_lease_busy`, and the run ended
`failed/adapter_failed`.
> - The provider work was complete. Only Runner bookkeeping failed, so
the run must not fail.
> - This pull request hands the exact lease to finalization and
completes the run from its accepted result.
> - The benefit is that an accepted result stays accepted, independent
of host load.

## Linked Issues or Issue Description

**What happened?**

A native ACPX Claude run asked a structured question. The control plane
accepted its result (`Waiting for…`) and recorded `run.terminal` with
`runTerminalState=succeeded`. The Runner close then took about 8.6 s.
Its `runner.suspend` command ended `failed`, but the durable Runner
state was `suspended` with a matching identity. The close threw
`NativeSessionCloseUnrecoverableError`. The executor emitted
`native.post_result_failure` (`category: runner_settlement_incomplete`,
`resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`.
Its finalization lease stayed held (its TTL is 20 minutes).
`finalizeNativeRun` got `native_finalization_lease_busy`, and the run
ended `failed/adapter_failed`. No restart was in progress. Host load
average was about 40.

**Expected behavior**

After the control plane accepts a result, finalization bookkeeping must
not fail the run. The run settles from its accepted result. Repair of
the Runner checkpoint stays with the existing next-run admission rules.

**Steps to reproduce**

Run `pnpm exec vitest run
src/services/heartbeat/runtime-execution.test.ts -t "accepted local
result"` in `server/`. The test uses the real executor, the real
control-plane result commit, and embedded PostgreSQL. The provider
boundary accepts a result and then throws
`NativeSessionCloseUnrecoverableError`. On the base revision, the
finalization row keeps the executor lease (`phase:
workspace_finalizing`) and the finalizer reports `lease_busy`.

**Paperclip version or commit**

Base `ff1d1dea5d`. The incident was on `819420a60a`.

## What Changed

- `native-session-executor.ts`: after the rejected failure write proves
an accepted result under the exact, unexpired lease, the executor
releases that lease to finalization. It uses the same exact-authority
predicates as the normal release. Before, the lease stayed held until
its TTL expired.
- `runtime-execution.ts`: for `runner_settlement_incomplete` on a local
workspace (no workspace sync and no remote workspace authority),
Paperclip records `workspace_finalize` as succeeded, finalizes the
accepted result, and returns it to the ordinary completion path.
- New `native-accepted-result-settlement.ts`: rebuilds the adapter
result from the durable accepted result row, its terminal, the run's
native session, and the accepted usage checkpoint. The result carries
the closed `nativePostResultFailure` diagnostic. It contains no provider
text.
- Other post-result categories (for example `io_not_found`) keep their
workspace-failure handling. They now also reach finalization at once,
instead of after the lease TTL.
- `doc/execution-semantics.md`: describes the settlement.

## Verification

- Red, then green: `settles an accepted local result when runner close
cannot prove suspension` and `keeps workspace failure handling for other
accepted-result failures on a local workspace` in
`server/src/services/heartbeat/runtime-execution.test.ts`. Both failed
on the base revision with the executor lease still held. Both pass with
this change.
- Safety case `settles an accepted local result ...
(workspace-check-failed)`: when the `workspace_finalize` check fails,
the run keeps the existing failure path. Case `terminal-unreadable`:
when the accepted result cannot be rebuilt, only a failed
`workspace_finalize` record is written.
- `pnpm exec vitest run
src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases
time out at 30 s only under host load above 40; they pass on a narrow
rerun). This includes the 14 existing post-result diagnostic and
authority cases (expired, replaced owner, generation, attempt, boot,
pid, process start, stopped, lock busy, and others).
- `pnpm --filter @paperclipai/server typecheck`: pass.
- Live cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`
(real ACPX Claude, local runner): 3 runs at host load average about 40.
Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run
emitted `native.post_result_failure` (`runner_settlement_incomplete`)
and now ended `run succeeded` (before this change it ended
`failed/adapter_failed`). The cell then failed later, in the resume run
after the answer: `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`. That is a separate
follow-up (see Risks).

## Risks

- A changed lease handoff. The executor releases the lease only after
the rejected failure write proves an accepted result under the exact
owner, attempt, controller generation, controller identity, and an
unexpired lease. A stale or replaced owner does not release anything. A
failed release is bounded and best effort. The old behavior then
applies.
- The run now completes as `succeeded` while the local Runner checkpoint
is not proven suspended. The next run on the same session then meets the
existing fence: the continuity-break retirement
(`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a
local Runner is refused with `target_not_sandbox_runner` and its state
root is quarantined. This PR does not change that fence. Extending the
retirement to local Runners needs its own proof that the local Runner
process stopped, and is a follow-up.
- Settlement happens only when the accepted result and its terminal are
rebuilt first, and the `workspace_finalize` record then succeeds. A
failed branch or plugin workspace check keeps the failure path.
- Synced and remote workspaces are not changed. They keep their
copy-back retry path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs). Extended reasoning
was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #19126 added the closed post-result diagnostic that this change
uses.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-06 12:36:34 -06:00