Commit Graph
56 Commits
Author SHA1 Message Date
DottaandClaude 289b402c26 fix(runner): settle a saved governed wait whose parked turn was lost on restore (#20519)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run can ask the board a question and park on that
saved (governed) wait
> - PR #20494 made a saved wait settle after a control-plane restart for
Codex, which reports the parked turn as interrupted
> - Claude runs through ACPX. After a kill -9 restart, Runnerd restores
the conversation and reports the parked turn as `turn.failed` with
`provider_turn_lost_on_restore`
> - The governed-wait drain and the saved-replay check rejected every
failed terminal as unproven, so the saved question failed and the task
needed a human Retry
> - Runnerd emits `provider_turn_lost_on_restore` only after it proves
that the old provider lifetime ended. That is the same proof as an
interrupted terminal
> - This pull request accepts that recoverable restart loss as proof for
a saved governed wait, and only that failure
> - The benefit is that a Claude question survives a control-plane kill
the same way a Codex question does

## Linked Issues or Issue Description

Refs #20494. Related: #20498 (Claude lost cancel reply; this change does
not touch that code).

**What happened?**
Restart chaos wave w-20261110T094930-s214 (Claude, Q1, PV1 and PV2 with
`--provider-fault-restart K1`) failed 2 of 2 Q1 tasks with
`R3:retry_needed`. The asking run ended `failed` with
`native_provider_terminal_failed`. Its events show the question saved,
the K1, recovery with `native_governed_wait_settlement_pending`, and
then `turn.failed` with `provider_turn_lost_on_restore` for the parked
turn. The thrown `providerCode` was
`governed_wait_provider_terminal_unproven`.

**What did you expect?**
The saved question settles after the restart. The asking run ends
`succeeded`. The board answer wakes the agent without a Retry.

**Steps to reproduce**
Run the harness with `--adapters claude --workloads Q1 --faults PV1,PV2
--provider-fault-restart K1 --provider-fault-delays 3
--provider-fault-restart-after-seconds 1 --concurrency 1 --tasks 4`. The
new unit-test variants `lost-on-restore-after-restart` and
`lost-on-restore-saved` failed with the same `providerCode` before this
change.

## What Changed

- `native-session-runtime.ts`: add `governedWaitTerminalProven(event)`.
A failed terminal proves the stop only when
`isNativeRestartInterruption(error)` is true (code
`provider_turn_lost_on_restore`, `recoverable: true`). Other terminals
keep the old rule (`providerTerminalObserved !== false`).
- The governed-wait accounting drain and the saved-replay check both use
that helper.
- Tests: two new variants of the #20494 test (a lost terminal after the
restart, and a lost terminal already saved before the next recovery).
Two new rejection cases: a non-restart failed terminal and a
non-recoverable restart loss still fail with
`governed_wait_provider_terminal_unproven`.

## Verification

- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime`: 265 tests pass. The two new settle variants
failed before the fix.
- `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .`
passes.
- A live restart chaos wave on this branch (Claude Q1, PV1/PV2 + K1) is
linked in a comment below when it finishes.

## Risks

- Low risk. The change accepts one more terminal as proof for a saved
governed wait. That terminal comes only from Runnerd after it proves the
old provider lifetime ended, and the code already trusts it to continue
a restart-lost turn (`nativeRestartInterruptedTurnId`).
- A failed terminal with any other code, or a non-recoverable restart
loss, is still rejected.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 05:13:21 -06:00
DottaandClaude ca472d543a fix(runner): settle a saved governed wait whose shutdown drain left sequence gaps (#20494)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run can ask the board a question and park on that
saved (governed) wait
> - After the question is saved, the runner interrupts the provider and
drains its usage and terminal for a bounded time
> - The drain keeps only usage, the terminal and cancellation facts of
the parked turn. It discards every other numbered provider event
> - If the control plane restarts before the drain settles, recovery
replays the saved journal. That replay required strictly contiguous
`sourceSeq`, so the discarded events made a valid saved wait look like
the wrong conversation
> - This pull request accepts a gap after the saved trigger only when
every event after the gap is a drain-retained fact of the same turn.
Identity checks do not change
> - The benefit is that the asking run settles its saved wait after a
restart instead of recording a failure

## Linked Issues or Issue Description

**What happened?**
Restart-chaos wave `w-20261110T063253-s208`, tasks t02 (Codex Q1, PV2:
provider terminal dropped) and t03 (Codex Q1, PV1: provider terminal
held), each with K1 (`kill -9` of the control plane about 3-4 s after
the fault). The question, the board answer and the final brief were all
correct. But the asking run was recorded `failed:
native_session_interrupted`, and the grader failed
`T:asking_run_settled`.

**Cause**
The run log of the asking run shows `failureDetail.message:
native_governed_wait_replay_mismatch`. It is thrown at
`packages/paperclip-runner/src/native-session-runtime.ts:2344` (master).
The persisted runner events of the asking run have `sourceSeq` 33..40
and then 43 (t02), and 41, 42 and then 45 (t03). The two missing numbers
are events that the accounting drain (`consumeTurn` →
`settleDurableResult`, around line 1206 on master) read and discarded,
because they are not usage, a terminal or a cancellation fact. Runs
without a restart have the same gap (for example
`w-20261110T032350-s201/t01`: 34, then 37). Those runs settle without
replay, so the gap was not seen before. With a held or dropped terminal
and a kill before the shutdown settlement receipt, recovery must replay.
Its strict contiguity check then fails on a gap that the runner made
itself.

**Expected behavior**
The question was saved before the provider was interrupted. Recovery
must settle that same saved wait, as #20336 does after verified
shutdown. Recovery must not start a new model turn.

**Related**
Refs #20336 (settle saved handoffs after verified shutdown). Not related
in cause or code: #20477, #20482, #20483.

## What Changed

- `isGovernedDrainRetainedEvent` (new, in `native-session-runtime.ts`)
names the facts that the governed-wait drain persists: usage, the turn
terminal, the interrupt acknowledgement, cancelled or expired runtime
requests, and a failed or cancelled `run.terminal`, all for the parked
turn. The drain now uses it in place of the same inline condition. Drain
behavior does not change.
- Governed-wait recovery replay accepts a `sourceSeq` gap only when:
- the trigger is the first replayed event (a gap before it still fails),
  - sequence numbers strictly increase,
- every event after the first gap is a drain-retained fact of the
trigger's turn.
- The run, normalized session and source instance identity checks do not
change. Any mismatch still fails with
`native_governed_wait_replay_mismatch`.
- `doc/execution-semantics.md`: the governed-wait text said that
shutdown keeps a contiguous journal. It now describes the gaps and the
replay rule.

Note for the sibling Claude/ACPX `turn.cancel` work: this change touches
`settleDurableResult` only in the drain filter line (same condition,
shared helper). It does not change cancel, the drain timeout or
`settleCancelledTurn`.

## Verification

- Red/green:
`packages/paperclip-runner/src/native-session-runtime.test.ts`
- `settles a saved governed wait whose drain left sequence gaps before a
restart (terminal-after-restart)`
- `settles a saved governed wait whose drain left sequence gaps before a
restart (late-terminal-saved)`

The first run uses the real drain. It discards two numbered events, so
the history is trigger, acknowledgement, then usage with a gap. The
stream then ends without a terminal, as with PV1/PV2 plus K1. On master,
recovery rejects with `native_governed_wait_replay_mismatch` (the live
error). With this change, recovery settles the saved wait as yielded. It
does not call `resolveGovernedWait` again. It starts no second turn. The
terminal is appended once. When the terminal arrived late and is already
saved, recovery does not cancel the provider again. A further recovery
of the settled checkpoint does not submit or settle again.
- Fail closed: `keeps rejecting an unproven governed-wait replay with a
sequence gap: …` covers a gap before the trigger, a foreign run, a
foreign session, a foreign source instance, usage of another turn, a
non-accounting event after a gap, and an accounting event followed by a
non-accounting event.
- `cd packages/paperclip-runner && npx vitest run
src/native-session-runtime.test.ts`: 261 passed. `npx tsc --noEmit -p
.`: clean.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t
governed`: 12 passed.
- `pnpm verify` returned exit 2 (test file outside resolved graphs).
Required CI supplies the broad coverage.
- Live restart-chaos wave: see the comment below with the wave id and
result.

## Risks

- Low. The rule loosens only the sequence check, and only after the
saved trigger, and only for facts that the drain itself retains and that
carry no work authority. Identity checks stay strict. A non-accounting
event after a gap still fails closed.
- No change to the normal start path or turn startup (fast-turn rule).
- No schema, protocol or env change. The pack `contractDigest` and
provider `commandDigest` do not change. No Daytona image rebuild is
necessary.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:19:21 -06:00
DottaandPaperclip 0e325e8d52 fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent can save a question or approval handoff before its provider
stops.
> - A missing provider terminal notification does not prove that the
saved work failed.
> - Runner already verifies provider exit, event drain, tool settlement
and suspension.
> - This change uses that proof to settle the same saved handoff.
> - Unproven shutdown holds continuation and explains that the work is
saved.
> - Recovery must never repeat the completed work or allow overlapping
execution.

## Linked Issues or Issue Description

Refs #20330. That PR extends the Codex terminal drain window. This
change handles the separate case where no terminal arrives but the owned
shutdown can prove stopped execution.

**What happened?**

A native run saved a human handoff, accepted cancellation, and later
stopped its provider. The missing terminal notification still made the
attempt fail and show “Couldn’t finish this attempt.”

**Expected behavior**

Keep the saved handoff. Accept its yielded result after either an
authoritative provider terminal or verified shutdown of the exact
execution. If shutdown remains uncertain, retain the handoff and explain
the hold without offering a duplicate Retry.

**Steps to reproduce**

Save a governed wait, acknowledge interruption, and omit the provider
terminal. The new regression uses the real Runner binary and a
controlled Codex process. The old runtime rejects the handoff instead of
asking the existing shutdown owner to settle it.

**Deployment mode**

Native Runner with Codex. Local verification used macOS arm64 and Node
26.4.0. Hosted sandbox recovery has not been qualified.

## What Changed

- Add an optional driver shutdown-settlement operation. Runner returns
proof only after its existing stop, drain, tool settlement and
suspension barrier passes.
- Bind the receipt to the run, normalized session and turn. Persist it
with the saved wait and validate it during recovery.
- Complete the same yielded handoff after verified shutdown. Retire the
stopped session from warm reuse. Do not invent a provider terminal or
complete usage.
- Retain a named shutdown-pending condition under the existing bounded
same-run recovery policy. Preserve actual provider failures and
integrity failures.
- Show saved-work shutdown guidance in task and inbox presentation. Keep
genuine approvals authoritative and disable duplicate Retry.
- Document the completion and restart contract. The checkpoint field is
additive; older checkpoints retain the terminal-or-shutdown path.

## Verification

- Red: the new runtime regression failed against the original
implementation because verified shutdown was never requested. Server
presentation and error-classification regressions also failed before the
fix.
- Red: both inbox shutdown states showed generic text before the inbox
change.
- Green: Runner runtime, shutdown and harness-backend suites pass 301
tests. Tests cover missing terminals, failed stop proof, wrong identity,
cancellation, late proof, genuine provider failure, and restart after
receipt persistence. Recovery completes once without another model turn.
- Green: the native executor and task-presentation suites pass 1,371
tests. Execution projection, inbox and blocker presentation pass 125
tests. Two test-authoring mistakes were corrected: an integrity-error
text expectation and an assertion for a nonexistent projection field.
Their initial failures remain in local logs.
- Runner TypeScript and full server layered type checks pass. Token
gates and diff whitespace checks pass.
- Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one
disposable local workspace, one tool call, and one deliberately dropped
terminal notification. The saved file, bound settlement receipt, one
accepted yielded result, one provider turn, confirmed provider exit and
suspended Runner all passed. The tool handler and persistence port were
controlled. This does not claim a full server or hosted journey.
- The full browser question-and-answer cell stopped before execution
because OPENAI_API_KEY was absent. No model work ran in that attempt.
Hosted sandbox recovery and the full browser continuation journey remain
unverified.
- `pnpm verify` returned exit 2 because the execution-spec change
requires broad CI. Required CI will supply that coverage; a full local
repository suite was not repeated.

- Review correction in `8553da244e`: a failed trailing-event write could
hide a provider failure. A new regression reproduced red, then passed
after all ordinary drain errors retained the named pending condition. No
settlement receipt or result can be accepted after that error. The live
missing-terminal check passed again on the corrected runtime.
- The real PostgreSQL control-plane regression and three existing
governed-wait controls pass. The new case persists one accepted yielded
result, a bound shutdown receipt, and the original pending human
question without a provider terminal.
- First CI revision `7c2cb9b1`: test-program typecheck found a missing
`PrpEvent` annotation in the new fixture. The annotation is fixed and
the Runner test-program typecheck passes. Runner support also timed out
in the unchanged `registered roots remain protected even when equal to
genuine public static bytes` case after 30 seconds. One unchanged local
diagnostic passed in 3.69 seconds overall. The hosted timeout cause
remains unknown; no timeout, fixture, or assertion was weakened.
[Original failed support
job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133).

- Rebased onto the merged #20330 and current recovery changes. The
combined branch passes 1,402 executor/presentation tests. Both prompt
and 18-second provider terminals still pass. A warm session with no
terminal passes after the full 30-second drain and verified
stop/suspension. Its initial overly broad assertion rejected a
nonterminal turn event; the corrected assertion rejects every terminal
type. The rebased CI gate was cancelled before any runner or step
started; its cause is unconfirmed and it supplies no validation credit.
Final revision `b49ec5a144` starts fresh CI and review.

- Final-head CI attempt 1 failed before creating jobs. GitHub reports an
internal server error (correlation
`2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty
runner-selection outputs for the aggregate jobs. One unchanged-commit
retry was requested; the initial admission failure remains recorded. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42673159868).

- **Current blocker:** final head
`b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile
review and zero unresolved threads, but required CI is not complete.
GitHub reports an active [Actions and Checks API
incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt
2 ran only part of the full selected graph. Build, canary, Rust, sandbox
providers, admitted browser shards and other completed test jobs passed.
The E2E aggregate correctly failed because `policy` and `e2e_shards` had
`abandoned` results. Server shard 5/12 received a runner shutdown signal
at 18:36:55 UTC and its test step was cancelled; its job still reports
in progress. The shutdown cause is not established. The separate review
workflow had the same pre-execution internal-error class and passed one
bounded retry. No further retry was submitted during the incident.
Resume with the missing/cancelled coverage after GitHub recovers; do not
count the absent typecheck, Runner or support jobs as passes. [Failed
coverage
gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236),
[interrupted server
shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174).

## Risks

- A missing terminal can use one additional bounded shutdown interval.
Unknown shutdown still blocks continuation.
- Provider terminal and usage accounting keep their separate meaning.
The receipt proves stopped execution and settled tool outcomes, not
model success.
- Recovery rejects a foreign receipt, a later turn, an active turn with
a settled receipt, and an actual failed provider terminal.
- There is no database migration. Older checkpoint shapes remain
readable. Drivers without this optional proof retain a named pending
state.
- This PR does not deploy the fix or repair an existing hosted task.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and local
code execution. The exact serving model ID and context window are not
exposed in this session. Live test inference used gpt-6.1-sol.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:46:16 -06:00
DottaandClaude Opus 5.5 42bc238aa5 fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records it
as `provider_turn_lost_on_restore` and continues in a same-conversation
continuation turn
> - #20118 taught the event consumer to skip a stale restart-lost
terminal of another turn once a continuation is running. It applied only
when the checkpoint already recorded the interruption
> - When the checkpoint instead names the lost turn as active (the drain
hit right after the first turn was submitted), the consumer starts the
continuation on the first lost terminal. If the same lost terminal is
then delivered again, it was taken as the run's terminal and failed the
run
> - This pull request applies the skip to any recovered run once its
continuation has started. For a checkpointed interruption it still waits
for the startup decision first
> - The benefit is that a turn lost on restore always continues, however
often its lost terminal is delivered

## Linked Issues or Issue Description

Refs #20118

**What happened?**
Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master
9b71d03215): 94 of 95 passed. In t90 (claude M1 parent), run `e5eab3cb`:
1. The run submitted its first turn at 06:28:25, and a graceful drain
followed at 06:28:28.
2. After restore at 06:28:41, the ACPX runner replayed the turn's accept
and start, then reported `turn_lab_1abbeca2…` as
`provider_turn_lost_on_restore`.
3. The controller started continuation `turn_lab_a3d009c6…`; the saved
checkpoint names it as the active turn.
4. The run failed at 06:28:45 with `native_provider_terminal_failed: The
runner restored the conversation after process loss, but the previous
turn is no longer active.` and needed a Retry.

**Expected behavior**
Once the continuation is running, a later delivery of the lost turn's
terminal is history, not the run's terminal. The run finishes on the
continuation.

**Steps to reproduce**
1. Recover a run whose checkpoint names the interrupted turn as active
(not yet recorded as terminal).
2. Deliver that turn's restart-lost terminal, which starts the
continuation, then deliver it again before the continuation completes.
3. Before this change, the second delivery ends the run as failed.

**Paperclip version or commit**
9b71d03215 (wave). Branch rebased on current master.

## What Changed

- `native-session-runtime.ts`: `supersededByRestartContinuation` applies
to every recovered run.
- Once a continuation has started, a restart-lost `turn.failed` for any
turn other than that continuation keeps the consumer going.
- With a checkpointed interruption, it still waits for the startup
decision first. Before this change it excluded the checkpointed
interrupted turn itself, which could also end a run if that turn's
replayed terminal arrived before the startup continuation began.
- `native-session-runtime.test.ts`:
  - a lost terminal delivered again after the continuation starts;
- the checkpointed interrupted turn's own terminal arriving before the
continuation starts.

## Verification

- Red/green: both new cases fail on master (the run ends failed or on
the stale terminal) and pass with the change. The #20118 cases still
pass.
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 389 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip still requires a recovered run, a recoverable
`provider_turn_lost_on_restore` failure, a continuation that actually
started, and a terminal for a different turn than that continuation. A
loss of the continuation itself, real provider failures, and runs that
started no continuation keep their current handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:08:19 -06:00
DottaandClaude Opus 5.5 b54317d3fb fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.

## Linked Issues or Issue Description

**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.

**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.

**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.

**Paperclip version or commit**
Staging revision b43cccbb0.

## What Changed

- `server/src/services/native-runtime/first-turn-queued-input.ts` (new):
`claimFirstTurnQueuedInput` seals a run's first-turn input in one
transaction. It locks the task, the deferred wakes, and the run in the
same order as the queue routes. It takes the responsible user's plain
queued comments in queue order (not interaction answers, interrupt
intents, independent continuations, conversations, or other users'
messages). It renders them with source-trust sanitizing and secret
redaction, then writes a `firstTurnQueuedInput` receipt
(`paperclip.first-turn-queued-input.v1`) into the run context. A second
call for the same run returns the same messages.
`runAcceptsFirstTurnQueuedInput` is true for a running native (or
preparing native) run that has no receipt and no accepted provider turn.
- `packages/paperclip-runner/src/native-session-runtime.ts`: new
`getFirstTurnQueuedInput` option. The runner calls it immediately before
the first `session.startTurn` and appends the result to `task.prompt`
(or to `events` for a resumed-session continuation envelope). It is not
called for the disposition-only recovery turn.
- `native-session-executor.ts`: passes the claim to the runner. A failed
seal commits nothing, so the executor logs it and starts the turn
without the messages. The messages stay queued.
- `queued-comment-delivery.ts`: a message in a run's receipt counts as
delivered only after the provider accepted that run's turn
(`turn.accepted` or the provider's `turn.started`), and only if the
message was not edited after the seal. Release promotion, Send, and
later first turns use this rule. `explicit-native-continuation.ts` uses
the same rule for stopped native runs.
- `routes/issues.ts` Interrupt: when the target run has not sealed its
first turn and has no accepted turn, the route does not record interrupt
intent and does not cancel the run. The response has
`executionWait.reason = "first_turn_pending"`. Other runs are
interrupted as before.
- Queue views (`routes/issues.ts` and the wake-queue adapter) hide
messages that the active run took into its first turn. Queue edits keep
those ids on the wake, so a run that fails before acceptance still
leaves them queued.
- `run-state.ts`: the issue-control run projection includes the receipt,
so the queue poll can hide taken messages.
- `doc/execution-semantics.md`: new section "Queued messages before a
native run's first turn".

## Verification

- Red before the fix, green after:
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "first
turn|already started"`: Interrupt on a preparing run and on a claimed
native run before `turn.start` does not stop the run, and the first turn
gets both messages in order. Before the fix, both cases cancelled the
run.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts -t "first-turn queued input"`: the
claim runs immediately before `turn.start`, and the first turn's prompt
ends with the queued messages. Before the fix, the claim was never
called.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/first-turn-queued-input.test.ts`: a run that
fails before `turn.start` leaves the message queued. The next run
delivers it, and after that no later run or copied retry context takes
it again. Before the delivery rule, the "exactly once" and "accepted
then failed" cases failed.
- Safety cases that pass before and after: Interrupt still cancels a run
whose provider accepted a turn (including a run saved by the previous
release with no receipt) and a run whose first-turn input is already
sealed. The existing "explicit interrupt fallback" cases for preparing
and native runs without a responsible user still pass.
- Speed: the seal is one short transaction before `turn.start`. In the
embedded-Postgres test it took 5.5 ms. It removes a full second
preparation (54 s on staging) from the message-to-first-token time.
- Related suites: `src/modules/wake-queue`,
`explicit-native-continuation`, `issue-queued-comment-queue`,
`issue-queued-comments-routes`, `queued-comment-consumption`,
`heartbeat/run-state` (888 tests), `native-session-executor.test.ts`
(1306 tests), and the full `native-session-runtime.test.ts` (238 tests)
pass. `pnpm --filter @paperclipai/server typecheck` passes.
- Not done: no live run against a real provider or sandbox. Staging was
read only.

## Risks

- Previous-release state: a deferred wake saved by the previous release
has the same shape and is taken normally. A run saved by the previous
release has no receipt. If it is resumed before its first turn, it seals
on the new release. If its provider already accepted a turn, Interrupt
stops it as before. A run saved by the previous release while it was in
`turn.start` but before acceptance counts as unsealed, so in that short
window an Interrupt does not stop it and the message is delivered at the
turn boundary.
- Rollback: if the server rolls back after a run sealed and its turn was
accepted, the old release does not read the receipt and can deliver
those messages a second time.
- A first turn that does not use the fresh-turn path (restart
continuation, disposition-only recovery) does not call the claim. Its
messages are delivered at the turn boundary as before.
- A message that arrives after the seal waits for the turn boundary.
Interrupt stops the run as before.
- No schema migration. The receipt is a key in the run context.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution, read-only
browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:44:18 -06:00
DottaandClaude Opus 5.5 cedf335c98 fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records
that turn as failed with `provider_turn_lost_on_restore` and continues
on a same-conversation continuation turn
> - Restart recovery blocks a run as
`provider_checkpoint_permanently_failed` when its checkpoint carries a
failed run terminal and no pending restart interruption
> - While a continuation is running, the checkpoint keeps the lost
turn's failed run terminal and names the continuation as the active
turn. The pending-interruption check returns nothing once a turn is
active
> - So a second restart during the continuation read the run as
permanently failed, blocked replay, and needed the board
> - This pull request treats that failed terminal as the restart loss it
records whenever an active turn follows a restart-lost terminal turn
> - The benefit is that runs survive repeated restarts during their
continuations, including M1 parents and their subtasks

## Linked Issues or Issue Description

Refs #20148

**What happened?**
Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master
8e3b64a257), t05 (claude M1 parent):
1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each
time it continued, and the third continuation `turn_lab_92b03dc7…` was
active.
2. At 00:48:29 the next restart blocked it:
`native_restart_recovery_blocked` /
`provider_checkpoint_permanently_failed`, with replay blocked.
3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from
the lost turn), `activeTurnId` set to the continuation, and a last
terminal turn fingerprinted `provider_turn_lost_on_restore`.

Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass
with the same shape, and its continuation had already proposed a
semantic result. Both stayed blocked. The parent's Retry was held on the
blocked child.

**Expected behavior**
A failed run terminal that records a restart loss, followed by an active
continuation turn, is not a permanent failure. Recovery claims
`resume_dead_runner`. The runtime then reconciles the active turn, and a
held result settles a lost turn as completed. A real provider failure
followed by a later turn still blocks.

**Steps to reproduce**
1. Checkpoint a run whose last terminal turn is a restart loss, whose
`terminal` is failed, and whose `activeTurnId` is a later continuation
(optionally with a semantic result).
2. Claim restart recovery. Before this change it blocks with
`provider_checkpoint_permanently_failed`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `contracts/native-session-backend.ts` (runner): the new
`nativeRestartContinuationInFlight` is true when all of these hold:
  - the checkpoint has an active turn;
  - its last terminal turn is a different turn;
- that turn's fingerprint is a failed, result-less, recoverable
`provider_turn_lost_on_restore`.
- `native-session-runtime.ts`: recovery clears that stale failed
terminal for a continuation in flight, as it already does for a
checkpointed interruption. Before, it threw
`provider_checkpoint_failed_terminal` before it reconciled the active
turn.
- `native-restart-recovery.ts` (server): `checkpointFailed` now also
requires that no restart continuation be in flight. It uses the same
contract function through the vendor boundary.
- `native-session-runtime.test.ts`: an end-to-end recovery test. A
checkpoint with the stale failed terminal and an active continuation
resolves on the continuation's result without starting a new turn.
- `native-runner-restart-recovery.integration.test.ts`:
  - a codex case of the stale-terminal shape as saved in production;
  - an ACPX case with a held result;
- a held case where an active turn follows a real provider failure
(`model_not_found`).

## Verification

- Red/green: on master, the stale-terminal case is blocked with
`provider_checkpoint_permanently_failed`. With the change it claims
`resume_dead_runner`. The real-failure case stays blocked on both.
- Red/green (runtime): without the runtime change, the end-to-end test
throws `provider_checkpoint_failed_terminal`. With it, the run
completes.
- `vitest run` over the restart-recovery and failed-workspace-recovery
test files in `server/src/services/native-runtime`: 5 files, 305 tests
passed (14 skipped).
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 385 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm
run typecheck:typescript` (`packages/paperclip-runner`): passed.

## Risks

- Low risk. The exception applies only when the recorded failure is
itself a recoverable restart loss and a later turn is in flight. Every
other recovery gate is unchanged: the provider-attempt budget,
controller takeover and process proofs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:28:12 -06:00
DottaandClaude Opus 5.5 b43cccbb08 fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the runner reports
it as `provider_turn_lost_on_restore` and the controller starts one
same-conversation continuation. A later restart that loses the
continuation is answered the same way
> - With back-to-back restarts, the controller's checkpoint can miss a
continuation that the runner already accepted. On the next restore the
controller starts a fresh continuation, and the restored runner then
reports the unrecorded earlier continuation as lost
> - The event consumer treated that stale lost terminal as the run's
terminal, so the run failed while its new continuation was already
running
> - Recovery then refused to replay the failed run, and the task needed
a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but
the parent stayed `blocked` on that subtask
> - This pull request makes the consumer skip a restart-lost terminal
for any turn other than the continuation it started
> - The benefit is that back-to-back restarts no longer turn a
recoverable lost turn into a failed run that needs Retry

## Linked Issues or Issue Description

Refs #20107

**What happened?**
Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade
between builds plus repeated restarts) at 070365404e. In task t02
(claude M1), the codex subtask CHA-7 failed with
`native_provider_terminal_failed: The runner restored the conversation
after process loss, but the previous turn is no longer active.` The
sequence:
1. Restore 1 lost turn `01a1bd17…`, and the controller started
continuation `01a1bd18-15ec…` (runner command 14). A restart followed
within a second, before the controller's checkpoint recorded that
continuation.
2. Restore 2 saw the checkpointed interruption of `01a1bd17…` and
started a fresh continuation, `01a1bd18-4cc4…` (command 24). The
restored runner also reported the unrecorded continuation
`01a1bd18-15ec…` as `provider_turn_lost_on_restore`.
3. Because a continuation was already started, the consumer returned
that stale terminal as the run's terminal. The run failed, the new
continuation was stopped (command 25), and automatic recovery refused to
replay it (`uncertain_provider_action`, `preserve_without_replay_v1`).
4. The parent CHA-2 asked CHA-7 to resume by comment, but an agent
comment cannot authorize a blocked replay, so the wake stayed
`deferred_issue_execution`. The parent's own Retry succeeded, but CHA-2
stayed `blocked` on CHA-7 (`R3:retry_did_not_complete`).

**Expected behavior**
A restart-lost terminal for an older, unrecorded continuation is
superseded by the continuation this restore started. The run continues
on that continuation and needs no Retry.

**Steps to reproduce**
1. Checkpoint a native session whose last terminal is a restart-lost
turn and whose active turn is null, so recovery starts a continuation.
2. Have the restored session report another turn (the unrecorded earlier
continuation) as `provider_turn_lost_on_restore`, either before or after
the new continuation starts.
3. Before this change, the run ends on that stale terminal.

**Paperclip version or commit**
070365404e (wave); branch rebased on current master.

## What Changed

- `native-session-runtime.ts`:
  - Records the turn id of the restart continuation it starts.
- New `supersededByRestartContinuation`, used by the consumer's
`resumeInterruptedTurn`, keeps consuming past a
`provider_turn_lost_on_restore` `turn.failed` when all of these hold:
    - the run was recovered from a checkpointed interruption;
- the failed turn is neither the checkpointed interrupted turn nor the
started continuation;
    - a continuation was actually started.
- The check first waits for the startup decision, because the consumer
starts before the continuation does.
- `native-session-runtime.test.ts`: a red/green test, with the stale
lost terminal arriving both after and before the continuation starts.

## Verification

- Red/green:
  - On master, both test cases end the run on the stale terminal.
- With the change, both resolve on the continuation's result. The
continuation is started once, and the stale `turn.failed` is still
appended to the journal.
- `vitest run` over the restart, recovery, continuation and terminal
test files in `packages/paperclip-runner`, including all of
`native-session-runtime.test.ts`: 9 files, 380 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip applies only to restart-lost failures of turns
other than the checkpointed interrupted turn and the started
continuation, and only after a continuation actually started. Real
provider failures, a loss of the continuation itself, and runs without a
checkpointed interruption keep their current handling.
- If startup fails, the startup decision still settles. The consumer
then falls back to the existing behavior and the run fails as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:44:13 -06:00
DottaandClaude Opus 5.5 412d49e48d fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path

- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.

## Linked Issues or Issue Description

Refs #18879

**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.

**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.

**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.

## What Changed

-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.

## Verification

- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at 070365404e, still failed for 3 Claude
runs. The cause was the install path: the saved sidecar path pointed
into the old build's worktree. Fixed in 72e020e13b; a second K14 rerun
has been requested.

## Risks

- **The restored conversation runs on the new provider pack.** That is
already the case for re-bound suspended sessions. Session start still
verifies every artifact and the restored provider identity against the
qualified command digest. A changed upstream release still fails strict.
- **Old provider process.** For an active turn, lost-turn recovery waits
for the provider-lifetime fence before restoring. For an open session,
behavior matches the same-build restart, which reopens without the
fence.
- **A state that really is incompatible** (an upstream provider release
change):
- **Between turns**, the common upgrade case: the next run's checkpoint
carries `allow_replacement_after_resume_failure`, so the refusal opens a
fresh provider session automatically, with no Retry. A new test,
`continues in a fresh session when a new run's saved session is
incompatible…`, pins this in `native-session-runtime.test.ts`.
- **Mid-turn, same run:** this still stops for a board Retry, by design.
The effects of the lost turn are unknown, and the automatic replacement
sweep (`decideNativeReplacement`) only replaces a session when it has a
complete effect inventory.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 16:34:23 -06:00
DottaandClaude Opus 5.5 dcaada5f59 fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.

## Linked Issues or Issue Description

Refs #19852 (startup sub-spans), #19846 (TTFT baseline).

**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.

**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.

**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.

**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.

## What Changed

- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.

## Verification

- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.

## Risks

- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:40:51 -06:00
DottaandClaude Opus 5.5 2468b3f907 fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart loses a turn, the native runtime sends
a same-conversation continuation (#19501, #19699, #19771)
> - A restart can also kill shell or background processes that earlier
turns started
> - In three restart storms, the continuation agent could not inspect
its killed `bench.sh` and set the task to blocked
> - The continuation prompt told the agent not to "blindly rerun an
action whose outcome is unknown", so the agent did not re-run the
command
> - This pull request tells the agent that earlier processes may be
gone, and to check the workspace outputs and re-run a command whose
expected output is missing
> - The benefit is that a task can finish after a restart instead of
blocking on a process that no longer exists

## Linked Issues or Issue Description

Refs #19501, #19699, #19771

**What happened?**

Restart chaos workload B5 (Codex and Claude) failed this way three
times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started
`bench.sh` in the background. A restart killed it. The restart
continuation agent tried to find its tool session and got "process
inspection is denied". It then blocked the task. The comment read:
"Blocked by the Paperclip runtime restart: the benchmark session is
unavailable…". It added that it "cannot verify whether the original run
is still active or recover its output, so I have not rerun it".

**Expected behavior**

The agent knows it is in a resumed session after a restart. It checks
the workspace for the command's outputs. If the expected output is
missing, it re-runs the command and finishes the task.

**Root cause**

`packages/paperclip-runner/src/native-session-runtime.ts`,
`restartContinuation`: the prompt did not say that processes from
earlier turns may be gone. It told the agent to inspect each unfinished
command's "current state" and not to "blindly rerun an action whose
outcome is unknown". With the process gone, the agent could not inspect
it, so it blocked.

## What Changed

- The restart continuation prompt now says:
  - This is a resumed session after a Paperclip control-plane restart.
- Shell commands and background processes from earlier turns may no
longer be running, and their tool sessions may be gone. Do not block
only because one cannot be found or inspected.
- Check the workspace for the outputs or artifacts the command should
have produced. If the expected output is present, use it. If the
expected output is missing, re-run the command.
- Do not repeat completed work. Before repeating an action with external
effects (a push, a message, a deployment), check whether it already took
effect. If you cannot tell, report the blocker.
- The prompt does not say that re-running is always safe. Process
teardown (#19727) has not merged.
- `doc/execution-semantics.md`: one additive sentence.

## Verification

- `native-session-runtime.test.ts`: the three restart continuation tests
(Codex, ACPX, OpenCode) now check the new guidance. They were red before
the prompt change and are green after it.
- Guard: "uses the admitted session cwd in the task envelope" now checks
that fresh turns and ordinary resumed turns do not contain the restart
guidance.
- `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/
src/live/runnerd-restart-continuation.test.ts` in
`packages/paperclip-runner`: 603 passed.
- `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for
`packages/paperclip-runner` are clean.
- Not run: a live restart-chaos replay with a real model.

## Risks

- Low. This changes only the prompt text of the restart continuation.
Ordinary turns do not change.
- An agent may now re-run a command whose earlier run had partial side
effects. The prompt asks it to check external effects first, and to
re-run only when the expected output is missing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 03:46:20 -06:00
DottaandClaude Opus 5.5 ef83d566ca fix(native): ask for the result once after a restart continuation ends without one (#19771)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs continue a turn that a control-plane restart lost, in
the same conversation (#19501, #19699)
> - In restart storms 5, 6b and 8, ACPX Claude runs crossed restarts,
ran the continuation, and then failed with
`native_session_retry_exhausted`
> - The continuation completed, but Claude ended it with "I'll wait for
the notification when it completes" for a background command, and did
not call `paperclip_finish`
> - The next same-run attempt then sent no turn at all and failed with
"session returned no semantic result"
> - This pull request makes the continuation ask for the result, and
asks once more in the same attempt if the continuation still ends
without one
> - The benefit is that a restart-interrupted task settles without a
Retry, and the restarts that the run survived cannot block the request

## Linked Issues or Issue Description

Refs #19501, #19699, #19720

**What happened?**

Storm 8, run 84c532f7 and its Retry 0faead66 (ACPX Claude, task CHA-5).
The steps are the same in both runs:

1. `resume_dead_runner`, then `session.resumed`, then `turn.failed` with
`provider_turn_lost_on_restore` (recoverable).
2. The restart continuation runs. Claude starts the bench in the
background and ends the turn with "I'll wait for the notification when
it completes". The turn completes with no `paperclip_finish`.
3. Lifecycle: "provider turn completed without a semantic result;
same-run disposition recovery persisted" (attempt 2).
4. 30 s later, attempt 3 restores the session. It sends no turn. After
about 2 s it fails with `native_finalization_missing: session returned
no semantic result`, and the attempt count makes it
`native_session_retry_exhausted`.

**Expected behavior**

The continuation tells the agent to finish and report its result. If the
continuation still completes without a result, Paperclip asks for the
disposition one time. Restarts that the run already survived do not
block that request. An agent that never reports a result still fails
closed.

**Root cause**

- `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts`
`startTurn` (lines 224 and 248): the restart continuation uses the
driver's one-shot disposition marker
(`dispositionOnlyRecoveryAvailable`). #19699 uses the marker on purpose,
so that a crash can adopt the accepted continuation turn. The marker
then stays consumed with the continuation turn id.
-
`packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts`
(line 226): only a *lost* continuation gives the marker back. A
*completed* continuation with no result does not.
- `packages/paperclip-runner/src/native-session-runtime.ts`
(`dispositionRecoveryWasSubmitted`, about line 2761): on the next
attempt, the consumed marker looks like a disposition request that
already ran. Recovery replays the continuation's terminal and starts no
turn. So the disposition request never happens.
- `server/src/services/native-runtime/native-session-executor.ts` (lines
9755-9761 and `nativeSessionFailureDisposition` at 7768): the
`semantic_result_missing` same-run retry charges a provider attempt on
the same counter as restart resumes. In this run, attempt 3 was the last
allowed attempt.
- The continuation prompt (`restartContinuation`) did not tell the agent
to report a result, or not to stop and wait for a background
notification.

## What Changed

- Codex-protocol driver (used by Codex and by ACPX through runnerd):
when a restart continuation completes without a semantic result, the
driver gives back the one-shot disposition marker. The disposition
request is not a continuation, so it consumes the marker for good.
- Native session runtime: after that, the same attempt sends one
disposition request (`Paperclip semantic-result recovery after a restart
continuation ... invoke exactly one paperclip_finish or
paperclip_block`). It does this only when the driver shows that the
continuation borrowed the marker and gave it back. Other drivers keep
the current behavior.
- Restart continuation prompt: it now says to finish in this turn, not
to end the turn to wait for a background command or a notification, and
to report one `paperclip_finish` or `paperclip_block`.
- `doc/execution-semantics.md`: one additive paragraph.
- No server change. The disposition request runs in the same attempt, so
it uses no provider attempt. This avoids `native-restart-recovery.ts`
(#19747).

## Verification

Red before the fix, green after:

- `codex-app-server-driver.recovery.test.ts` "keeps one disposition
request after a restart continuation completes without a result". Red:
marker stays consumed on `turn-2`. Green: the marker is given back, one
disposition turn starts, and a resultless disposition turn does not
re-arm.
- `native-session-runtime.test.ts` "settles a restart continuation that
ended without a result through one disposition turn (Codex driver)".
Real Codex-protocol driver with a fake transport. Red: only one
`turn/start`, then `native_finalization_missing`. Green: the disposition
turn `turn-3` reports the result and the run completes.
- `native-session-runtime.test.ts` "asks once for the disposition when a
restart continuation ends without a result" and "fails closed when the
driver keeps the continuation's disposition marker".

Also run:

- `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/`
in `packages/paperclip-runner`: 593 passed.
- `src/live/runnerd-restart-continuation.test.ts` (real runnerd and fake
Codex, killed runner): 5 passed.
- `tsc --noEmit` for `packages/paperclip-runner`.
- `pnpm verify` exits 2 because the doc change is a global CI input.
Required CI supplies the broad coverage.

Not run: a live replay of storm 8 t05 with Claude. The chaos harness
needs provider keys from the operator's shell.

## Risks

- Low to medium. A run whose continuation ends without a result now gets
one more provider turn in the same attempt. The bound is one disposition
request per continuation. Each continuation still needs an admitted
restart, and the provider-attempt budget bounds those.
- If the controller crashes after the continuation completes but before
its checkpoint, the checkpoint keeps the stricter consumed marker. That
case fails closed as before.
- The prompt text changes for restart continuations. It does not change
a protocol schema or a profile digest.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 03:22:37 -06:00
DottaandClaude Opus 5.5 fc7e3a6503 fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action

## Linked Issues or Issue Description

Refs #19659, Refs #19516, Refs #19334

**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).

The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".

**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.

**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.

**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.

## What Changed

- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.

## Verification

Red before the fix, green after (each test first run against unchanged
source from `origin/master` 644ee66f5d):

- `native-session-runtime.test.ts`, test "never replaces a session after
a deterministic, non-destructive recovery refusal". Red: replacement was
attempted (`TypeError: Invalid value used as weak map key` from the stub
replacement). Green: typed error, and `openReplacementSession`,
`openSession` and `openRun` are not called.
- `harness-driver-backend.test.ts`, test "keeps the runner's closed
class of a failed recovery". Red: the typed error did not exist. Green:
both new classes are read from the command result and from the typed
error. The same message text without the type stays unclassified.
- `tests/codex_provider.rs` (real durable executor and fake Codex
app-server):
-
`attach_with_a_pending_tool_call_is_classified_as_a_pending_side_effect`.
Red: `left: None, right: Some("provider_side_effect_pending")`.
-
`saved_codex_thread_attaches_after_a_release_changes_its_feature_toggles`
(protected sandbox change). Red: `left: None, right:
Some("provider_attach_refused")`.
- `native-session-executor.test.ts`:
- Test "stops same-run retries for an operator while a saved
side-effecting call has no result". Red: `expected
'native_session_interrupted' to be
'native_provider_side_effect_pending'`.
- Test "retains the local controller checkpoint until remote continuity
archiving succeeds: failed/timeout". Red: the message had no detail.
Green: the message has the exit status and output, a token in the output
is redacted, and the local runner and control-plane files are intact.

Safety and regression checks:
- New test "still replaces a session after an unclassified resume
failure when the checkpoint allows it". It passes before and after: a
transport drop still reaches replacement.
- Existing test "types a runner-proven incompatible session when
same-session recovery is required" still passes.
- `native-session-resume.test.ts`: a source stopped with
`native_provider_side_effect_pending` is not rebound.
- `run-completion.test.ts` "replays saved user intent through normal
Retry …" passes with the detailed archive message.
- Runner Vitest: `native-session-runtime.test.ts` and
`harness-driver-backend.test.ts` (266 passed);
`runnerd-codex-transport-warm-attachment.test.ts` and
`codex-app-server-driver.recovery.test.ts` (74 passed).
- Rust: `--lib` (560 passed, plus one ACPX timing test that failed under
load and passed when rerun alone), `codex_provider` (118 passed),
`native_provider_backend` (45 passed), `durable_recovery` (3 passed).

Not run: a live staging recovery at this revision. The local server
typecheck could not resolve the unbuilt runner package, so required CI
provides typecheck.

## Risks

- A run that a replacement session would have continued after an attach
refusal now stops for an operator. The refusal repeats on every retry,
so the replacement was the only way past it. It also destroyed saved
state, which is what blocked the staging tasks. A board Retry still
continues the task in a fresh session in the same workspace. This is the
existing `native_provider_state_incompatible` behavior.
- A new failure code `native_provider_side_effect_pending` appears in
run records and recovery actions. It joins
`EXECUTION_RECONCILIATION_CAUSES`, so the UI shows the reconciliation
flow, not Retry.
- An attach refusal used to reach an automatic replacement; it now needs
one board Retry. This is intended: the replacement archived saved state
and failed part way on staging.
- The pending-side-effect class means "no result delivered to the
runner". The controller replays a recorded tool outcome only after
attach. A call whose outcome the server already recorded therefore also
stops for an operator. That is conservative; the reconciliation shows
the recorded outcome.
- Only a pending tool call classifies an attach blocker. Other blockers
(active turn, ambiguous turn start, active goal) keep their current
behavior.
- The archive error message is longer. The one exact-match consumer now
accepts the `code: detail` form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser reads of staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:25:54 -06:00
DottaandClaude Opus 5.5 ece09c156a fix(runner): keep an accepted result when ACPX reports its turn lost on restore (#19691)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native Codex, OpenCode, and ACPX runs survive control-plane
restarts: runnerd restores the conversation, and the control plane
continues the task from its checkpoint
> - #19658 made recovery keep a durable result when a Codex restore
loses the turn that proposed it. It covered the recovery read path only
> - ACPX reports the lost turn differently. After the restored
conversation reopens, a live `turn.failed` arrives with
`provider_turn_lost_on_restore`
> - That live terminal still turned an accepted `done` result into a
failed run. The run ended `failed` (`adapter_failed`), and the finished
work needed a Retry
> - This pull request moves the rule into the shared terminal mapping. A
restore-lost turn with a held result for that exact turn settles as
completed, whichever path reports it
> - The benefit is that ACPX and OpenCode tasks that already finished
keep their accepted result after a restart, as Codex tasks now do

## Linked Issues or Issue Description

Refs #19658, #19525, #19501, #19539

**What happened?**
In a restart chaos wave, an ACPX Claude task (workload A1) wrote its
brief and proposed a `done` result (`run.result.proposed`, then a
completed `paperclip_finish` tool call). Then a K1 and a K6 crash loop
hit the control plane, and `resume_dead_runner` ran at controller
generations 2 and 3. The restored ACPX conversation emitted
`turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`)
for the checkpointed turn. The control plane then emitted
`run.result.accepted` with the `done` result, but `run.terminal` came
back `runTerminalState: failed` / `turnTerminalState: failed` /
`reportedWorkDisposition: done`. The run ended `failed` /
`adapter_failed`, "Native session failed".

**Steps to reproduce**
1. Start a native ACPX (or OpenCode) task through runnerd.
2. Let the agent call `paperclip_finish`, and let the control plane
persist the proposal.
3. Kill the control plane and runner before the turn ends.
4. Let restart recovery resume the run. The restored runner reports the
turn lost on restore as a live terminal.

**Expected behavior**
Recovery finalizes the accepted result, and the run succeeds.

**Actual behavior**
ACPX and OpenCode use the same Codex harness session as Codex
(`codex-native-backend.ts`). Recovery restored the held result for the
turn from the durable proposal. But the live `turn/completed`
notification with `status: failed` and `provider_turn_lost_on_restore`
went through `mapTerminalTurn`, which emitted `turn.failed`. The backend
maps `turn.failed` to `runTerminalState: failed`. #19658 handled only
the `thread/read` reconciliation path.

## What Changed

- `codex-session-terminal.ts`: a new
`settleRestoreLostTurnWithHeldResult` helper. When a terminal is
`failed` with `provider_turn_lost_on_restore` and the session holds the
semantic result that this exact turn proposed, the turn settles as
`completed`. `mapTerminalTurn` applies it first, so live notifications
and recovery reads follow one rule.
- `codex-session-terminal.ts`: the helper also covers
`acpx_active_turn_recovery_closed`. Runnerd's ACPX backend
(`acpx_provider_backend.rs`, `restore_session_if_needed`) emits that
code instead of restoring the conversation when its own state already
recorded the accepted result. It is the same restart loss, with the
result already accepted. #19525 already keeps accepted results when
close cannot prove suspension, and session close still handles
`provider_exit_unconfirmed`.
- `codex-harness-session.ts`: `reconcile()` keeps only the missing-turn
case from #19658. The restore-lost read case and the prior-terminal
check now use the shared helper.
- `doc/execution-semantics.md`: the restart paragraph now states that
the rule covers Codex, OpenCode, and ACPX, and both report paths.

## Verification

- Red before the fix, with the real Codex native backend
(`createCodexNativeSessionBackend`) and a fake transport through
`executeNativeSession`: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/native-session-runtime.test.ts -t "journaled
completion proposal"`. The new `restore-lost live` scenario failed.
Green after the fix: the run completes with the journaled result on
`turn-1` and a succeeded terminal, and no `turn/start` is sent. The
`restore-lost read` scenario passes on both (#19658 already covered it).
- Driver red/green:
`src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "live
restore-lost"`. With a held result, the live lost terminal was
`turn.failed` before the fix and is `turn.completed` after it. With no
held result, it stays `turn.failed` with the lost cause, so the #19501
continuation is unchanged.
- The driver case was extended to `acpx_active_turn_recovery_closed`
(red with a held result before the extension, then green). It was found
in a live replay where the run ended `failed`/`adapter_failed` with an
accepted `done` result.
- Suites: `src/drivers/codex/` and `src/native-session-runtime.test.ts`
(580 tests), plus `src/backends/` and
`src/live/runnerd-restart-continuation.test.ts`, pass.
`typecheck:typescript` passes.
- Live chaos replays (ACPX Claude): results and a separate new signature
are in a comment below.

## Risks

- The rule needs a held result whose turn id equals the lost turn.
Recovery restores that result only from the durable
`run.result.proposed` for the checkpointed turn, so a result for another
turn does not settle it.
- A restore-lost turn without a held result keeps its failure and the
bounded same-conversation continuation.
- Real provider failures and other terminal states are unchanged.
- No schema, profile, pack contract, or database changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 21:07:20 -06:00
DottaandClaude Opus 5.5 a894b11a2b fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run's close must prove that its runner durably suspended
before the session's next run may start. If it cannot, the runtime
quarantines the session (`NativeSessionCloseUnrecoverableError`)
> - In a restart chaos storm, an ACPX Claude run succeeded and its
normal close quarantined its own session. The follow-up run on that
session failed 2 s later with `native_session_cleanup_quarantined`, and
no control-plane restart came between the two runs
> - The close sent `runner.suspend` once, the ACPX sidecar rejected it
right after the turn completed, and the barrier gave up at the first
rejected command. No log line showed the quarantine or its cause
> - This pull request retries a rejected suspension within the close
deadline, and logs every cleanup quarantine with its cause when it is
created
> - The benefit is that a successful run no longer blocks its own
follow-up because of a one-time suspension rejection, and any remaining
quarantine can be traced

## Linked Issues or Issue Description

**What happened?**
Run A (ACPX Claude) succeeded. The run log of its close shows:
- `runner did not prove durable suspension before checkpoint`
- `native_session_settlement_incomplete
{"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}`

The runner journal shows `runner.drain` completed, then `runner.suspend`
failed with `failed to suspend ACPX provider: ACPX sidecar command
session.suspend was rejected (retryable=false,
classification=unclassified)`. The follow-up run B on the same session
was created 200 ms after A finished and failed 2 s later at admission
(`retryQuarantinedSessionCleanups`, operator recovery branch). The
server log had no line for the quarantine. One Retry completed the task.

**Steps to reproduce**
1. Run an ACPX task whose sidecar rejects the first `session.suspend`
after the turn completes.
2. Start a follow-up run on the same session.

**Expected behavior**
The close asks again, proves suspension, and the follow-up runs. If a
quarantine is still needed, it is logged with its cause.

**Actual behavior**
The close fails on the first rejection, quarantines the session, and the
follow-up fails with no logged cause.

## What Changed

- `runner/.../durable/runner.rs` (`process_command`): a failed
`runner.suspend`, on first execution or replay, now keeps the runner in
`Continue` instead of `Suspend`. Before, the runner recorded `suspended`
over a provider whose session was still open, then exited. The RC-34
journal shows exactly that: runner lifecycle `suspended`, ACPX provider
`session_open`. A failed `runner.shutdown` still stops the runner.
- `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend`
now lifts the connection's terminal-lifecycle fence, so the next pending
command (the fresh suspend) is delivered on the same connection.
Completed suspensions and every shutdown keep the fence. New test:
"delivers a fresh suspension after a rejected one on the same
connection". Before the fix, the retry was never sent and the test timed
out waiting for the frame.
- `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a
rejected `runner.suspend`, the barrier sends a fresh suspend command, up
to 3 attempts, 250 ms apart, inside the same close deadline. An exited
runner or a spent budget still fails closed. A rejection changes no
state: the runner returns before it records suspension.
- `native-session-runtime.ts`: new
`onNativeSessionCleanupQuarantined(listener)`. It is called when a
cleanup quarantine is created or escalates to operator recovery, with
company, run, issue, session, `operatorRecoveryRequired`, a bounded
cause (error name, code, message) and bounded settlement facts.
- `server`: `logNativeCleanupQuarantines` logs each quarantine as a
warning. The subscription is process-wide and installed once, so closes
that fail during shutdown, after HTTP closes, are still logged.

## Verification

- Red before the change: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected
suspension"`.
  - "retries a rejected suspension…" got `false`.
  - "stops retrying…" saw 1 attempt instead of 3.
- Green after the change: both pass.
- Rust: new test
`rejected_suspend_keeps_the_runner_serving_its_controller` covers a
failed suspend, its replay, and an accepted retry with a fresh command.
`failed_lifecycle_commands_replay_their_terminal_transition` now expects
`Continue` for suspend and `Shutdown` for shutdown. The `pnpm run
check:runner` unit suites pass (559). `acpx_park_readiness` has two
timing tests (`< 500ms`) that fail at this host load on unchanged master
as well.
- New runtime test: "announces every cleanup quarantine with its cause
when it is created" (operator-required quarantine, cause and settlement
reported once). `src/native-session-runtime.test.ts` passes (212).
- The suspension cases in `runnerd-codex-transport.test.ts` pass, except
"proves local suspension after an event backlog… (1024 suffix deltas)".
That case also times out on unchanged master at this host load (27–35);
I confirmed this by stashing the change and rerunning.
- Server `tsc --noEmit` passes after the runner package build.

## Risks

- A provider that keeps rejecting suspension now gets three tries
instead of one, within the existing close deadline. The final outcome
and quarantine are unchanged.
- The cause of the ACPX sidecar's first rejection is not proven. The
logged evidence fits the sidecar's "not at a safe suspension point"
guard (`turnId`, tools or inputs still set) racing the just-completed
turn. This change makes close tolerate it. It does not change the
sidecar.
- Log volume: one warning per quarantine.
- The Rust change ships in the runner binary. Hosted images need a
rebuild to pick it up.
- After a failed suspend, the runner stays alive with its provider. If
every retry fails, close still treats suspension as unproven and kills
the owned runner process, exactly as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19532, #19609.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 20:06:54 -06:00
DottaandClaude Opus 5.5 785ead5c7a fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native provider session close fails, the runner keeps a
cleanup owner or quarantine in process memory. Its admission gate
refuses new sessions for that task (scoped to the task since #19532)
> - The server could not see that state. A task's Retry was accepted
(202), its run was dispatched, and the runner refused it at once with
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`
> - The user saw a Retry that failed instantly, and nothing said what to
do
> - This pull request lets the server read the runner's per-task cleanup
gate and hold the Retry with a clear reason. The Retry starts
automatically when the close is released
> - The benefit is that a Retry is never dispatched only to fail on a
block that is already known

## Linked Issues or Issue Description

**What happened?**
After two hard restarts, a Codex run's close stayed incomplete. The
board Retry for that task was accepted, and its run failed in about 1 s
with `native_session_cleanup_quarantined: prior session cleanup remains
incomplete`. The next Retry did the same.

**Steps to reproduce**
1. Let a native session close fail or stay pending for a task.
2. Press Retry on that task.

**Expected behavior**
The Retry waits with a clear reason, and starts when the cleanup
finishes. If the cleanup needs an operator, Retry is not offered and the
one working action is shown.

**Actual behavior**
The Retry is accepted and its run fails at once.

## What Changed

- `native-session-runtime.ts`:
- New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a
read-only view of the in-process admission gate for one task: `pending`
(a retained close or a bounded automatic close recovery still owns the
task's provider resources), `operator_required`, or `null`. It grants no
authority.
- New `onNativeSessionCleanupSettled(listener)`. It is called whenever a
cleanup owner or quarantine entry is released.
- `explicit-native-continuation.ts` (the continuation gate for a Retry
or a new message):
- `pending`: the input is held with `local_cleanup` ("Retry is waiting
for the previous execution to finish cleanup. Your message is saved.").
It starts automatically when the close is released.
- `operator_required`: handled exactly like a recorded
`native_session_cleanup_quarantined` run. Retry is not offered. A new
message runs the existing stopped-process proof
(`verifyStoppedNativeSessionForContinuation`), which retires the runner
quarantine when the old processes are proven stopped.
- `native-cleanup-release-wake.ts` (registered in
`server/src/index.ts`): each cleanup release wakes deferred execution
for the company in a transaction, so a held Retry re-evaluates.
- `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime
functions.

## Verification

- Red before the gate (server change stashed): `pnpm --filter
@paperclipai/server exec vitest run
src/services/quarantined-task-retry.test.ts -t "holds a Retry"`
dispatched a successor run while the task's close was retained
(`expected [ { … } ] to have a length of +0 but got 1`).
- Green after the gate:
- The Retry is held (`deferred_issue_execution`, `executionWait.reason =
local_cleanup`), and no successor is dispatched.
  - A different task is not blocked.
- When the close settles, the release is announced, and the next pass
dispatches exactly one successor.
- The whole `quarantined-task-retry.test.ts` file passes (28 tests).
`src/native-session-runtime.test.ts` passes. Server `tsc --noEmit`
passes after the runner package build.
- Live chaos replays: results are added in a comment.

## Risks

- The gate reads process memory, so it applies only where the server and
the runner session share a process (local runner). Other deployments
keep today's behavior.
- A held Retry re-evaluates on the release wake (sent in its own
transaction) or on the existing 30 s deferred-execution pass. When the
bounded automatic close budget is spent with nothing else scheduled, the
view reports `operator_required`. Retry is then no longer offered, and a
new message reaches the stopped-process recovery instead of waiting
forever.
- No schema or API shape changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #19532.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 18:52:46 -06:00
DottaandPaperclip 19c2bebc9c fix(runner): settle durable provider exits on open streams (#19657)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Runner keeps the native session and its event stream until a turn
ends.
> - A provider can exit with a terminal error while its event stream
stays open.
> - Waiting for that stream to close leaves the run and its resource
owner active.
> - The existing durable failure receipt already identifies the failed
current turn, including an authoritative replay.
> - This pull request uses that event to enter the existing failure and
cleanup path.
> - Recovery and later idle retirement can then proceed from a real
terminal result.

## Linked Issues or Issue Description

Refs #18988 for native resource retirement. This is a separate
terminal-event defect found during failed-turn recovery qualification. A
search of open and closed PRs found no duplicate `provider_exited` fix.
The related merged #12596 changes Rust test polling; it does not change
this TypeScript session owner.

**What happened?**

`executeNativeSession` records a nonrecoverable `provider_exited` event,
but can continue to wait on an open event iterator. A proposed result
does not authorize success, and stream EOF may never arrive. The run
cannot reach its existing failure and resource cleanup path.

**Expected behavior**

Settle the committed or authoritative duplicate terminal failure for
this exact run, session and provider. Close through the existing owner.
Preserve recoverable goals, completed-turn and identity guards. A
duplicate must not add another committed event count.

**Steps to reproduce**

1. Supply a native session whose iterator emits a current,
nonrecoverable, unexpected `provider_exited` failure.
2. Keep that iterator open until the session owner closes it.
3. Observe that the old implementation does not settle within the
existing test bound.
4. Apply the guard below. The same execution fails with the typed
terminal error and calls the existing close owner once.

**Paperclip version or commit**

The original regression was reproduced against master source
`05a19768ae4d337a4ae88ecdbc259f251406487c`. Current candidate
`f4bb8f1b68e2793062dd640ee56b87a9208b4fd9` normally merges fetched
master `07a548073ab48f13cb93b2fdc31553a5da88a070`. All unrelated master
paths are preserved exactly. The two-file PR diff contains only the
guarded settlement and its tests.

## What Changed

- Add a guard after the existing durable event receipt. It requires a
committed or duplicate receipt, a nonrecoverable unexpected exit, no
completed current or previous turn, and exact provider/run/session
identity.
- Throw the existing typed provider failure so the existing failure and
close owners run. Add no scheduler, polling loop, provider parser or
cleanup owner. Event counting remains committed-only.
- Cover open streams with and without a result proposal, authoritative
duplicate replay, all six safety predicates, and a recoverable goal that
later completes. Preserve accepted-result and close-once assertions.

## Verification

- The original open-stream regression fails before the production
change. Review then found that the duplicate test released EOF before
asserting settlement. The corrected duplicate control fails on the prior
production code at its original timely-settlement assertion; it passes
after accepting the authoritative duplicate. All 21 selected exit,
safety, goal and compaction controls pass.
- The normal durable transport validates authenticated current
run/session/turn/item identity and canonical replay equality before
delivery. The App port validates current authority and binding; its
durable event transaction returns duplicate only for matching source
identity and canonical payload. No absent or conflicting event
authorizes settlement. The six negative consumer tests use a
deliberately permissive mock port; they prove this consumer guard, not
that production accepts foreign events.
- The first remote Typecheck job found five errors in new mocks that
accessed PRP fields without narrowing a native-or-PRP event. Both mocks
now explicitly reject the wrong event shape. Complete original CI
evidence is retained. No cast, suppression or narrowed test compiler
program is used.
- On current integrated candidate `f4bb8f1`, normal Runner emission and
the full normal `tsconfig.test.json` source/test program pass. All 26
selected cases pass: the same 21 controls plus all five structured
provider-failure/model-retry rows affected by the incoming master
change. The other 197 cases were filtered, not counted as passing.
- Before master integration, all 411 declaration paths and bytes were
identical across the settlement change. Incoming master independently
adds one OpenCode declaration and changes three OpenCode declarations;
those deltas are recorded separately. This fix changes no runtime
declaration. No Rust source or binary is changed by this PR.
- Actual guarded workers recorded zero PG sockets, active listeners or
blocked attempts. All known worker processes/groups closed. The working
tree is clean after qualification. No install, Rust build, App run,
provider connection or PostgreSQL suite was part of the final freshness
loop.
- Fresh exact-head remote CI and complete bot reviews remain required. A
local whole-repository test/build, successful task retry or
live-provider acceptance is not claimed.

Run the selected controls from a normal dependency installation:

```sh
pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts --testNamePattern='settles committed or duplicate provider exits while their stream stays open|retains EOF settlement when provider exit fails the|preserves a recoverable provider-exit goal until its later authoritative completion|preserves a durable session failure when its stream closes|preserves session-goal failure priority|context compaction|never continues task work when a restart loses a compaction turn|preserves structured provider failure and model retry classification' --maxWorkers=1 --no-file-parallelism
```

## Risks

- A false terminal classification could stop useful work. The guard
requires the existing durable receipt and explicit current-turn failure
fields; recoverable and completed-turn controls remain covered. An exact
durable duplicate may settle the same failure without repeating
accounting.
- Close can still fail. This change reuses the existing close owner and
does not manufacture resource retirement or clear retained failure debt.
- This proves Runner failure settlement under a controlled session
boundary. It does not establish successful task retry, whole-tenant
quiet, live provider closure, or deployed suspension.

## Model Used

OpenAI `gpt-6.1-sol` produced the delegated fix and qualification.
OpenAI Codex, based on GPT-6, performed managing technical review and
integration. Both used repository tools and code execution. The
coordinator's exact backend snapshot and context-window size were not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run the selected tests locally and they pass
- [x] I have added tests for the changed behavior
- [x] I have reviewed documentation; no command or public contract
changes require a documentation edit
- [x] I have considered and documented the risks above
- [ ] All Paperclip CI gates are green — fresh exact-head checks pending
- [ ] Greptile is 5/5 with no open findings — fresh exact-head review
pending
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 18:10:33 -06:00
DottaandClaude Opus 5.5 7941f6b051 fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model

## Linked Issues or Issue Description

**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".

**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.

**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".

## What Changed

- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.

## Verification

- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.

## Risks

- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 17:08:04 -06:00
DottaandClaude Opus 5.5 d4d3383c97 fix(runner): scope cleanup quarantine to the session's task and conversation (#19532)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Codex, ACPX) execute through the runner's
`executeNativeSession`, which gates each new session behind cleanup of
earlier sessions
> - When a session close fails in a way that needs operator recovery
(`NativeSessionCloseUnrecoverableError`), the runner keeps a quarantine
entry for it
> - The admission gate keyed that quarantine by `[companyId, backend
kind, backend name]`, so one bad close blocked every later run of the
same backend in the whole company
> - In a restart chaos wave (two hard kills about 20 s apart), one Codex
session's close became unrecoverable. After that, every Codex run and
every Retry in the company failed in about 1 s with
`native_session_cleanup_quarantined`
> - This pull request scopes the gate to the resources that a retained
owner can still hold: its task (issue) and its provider conversation
(normalized session)
> - The benefit is that one failed close no longer stops unrelated
tasks, while the task and conversation that are really at risk stay
protected

## Linked Issues or Issue Description

**What happened?**
After two hard control-plane restarts about 20 s apart, one Codex
session close became unrecoverable. From then on, every Codex execution
in the company failed at admission with
`native_session_cleanup_quarantined: prior session cleanup requires
operator recovery…` (`retryQuarantinedSessionCleanups`). Board Retries
for four unrelated tasks were accepted and then failed at once.

**Steps to reproduce**
1. Run two native Codex tasks in one company (local runner).
2. Make one session's close throw `NativeSessionCloseUnrecoverableError`
(in the wave: a protocol failure after a second hard kill, then an
unsettled runner suspension).
3. Start or retry a different Codex task in the same company.

**Expected behavior**
Only work that shares a resource with the retained owner waits. Other
tasks run normally.

**Actual behavior**
Every later Codex run in the company fails within about 1 s, until the
server process restarts.

## What Changed

- `packages/paperclip-runner/src/native-session-runtime.ts`: the cleanup
domain is now `{ key, resources }`. `key` is the old backend key
(company, backend kind and name, optional remote cleanup scope).
`resources` are the session's issue and its normalized session (or its
run when it has no normalized session).
- The admission gate (`retryQuarantinedSessionCleanups`) and its
retained close owners block a new session only when the backend key
matches and at least one resource matches.
- The exact retirement paths (`completeRetainedNativeSessionCleanup`,
`completeTerminatedRemoteNativeSessionCleanup`,
`completeTerminatedLocalNativeSessionCleanup`) read the unchanged
backend key, so their behavior does not change.
- New regression test: an unrecoverable close on session A does not
block session B of another task. A session with the same issue, or with
the same normalized session, stays blocked.

Why this scope is safe: the resources that a retained owner can still
hold are its runner state root and its workspace. The server derives the
runner state root from the session scope (company, agent, workspace,
provider, normalized session). The workspace belongs to the task.
Different tasks already run concurrently on the same backend, so a
different task shares no resource with the retained owner that normal
concurrency does not already share.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/paperclip-runner exec
vitest run src/native-session-runtime.test.ts -t "unrelated tasks of the
same backend"` failed with `native_session_cleanup_quarantined` for the
unrelated task (the RC-20 error).
- Green after the fix: the same test passes. The full file passes
(211/211).
- `pnpm --filter @paperclipai/paperclip-runner typecheck` passes.

## Risks

- A retained owner can no longer block a different task that shares a
project workspace. Concurrent tasks in a shared workspace are already
allowed, so this adds no new kind of overlap.
- The same task and the same conversation stay blocked. A Retry of the
task whose close failed can still be accepted and then fail at
admission. The trigger of that close is handled separately.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 12:39:31 -06:00
DottaandClaude Opus 5.5 a0cba70f18 fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep their provider conversation when the control plane
restarts
> - A local Codex turn lost in a restart already continues once in the
same conversation (#15297)
> - ACPX (Claude and the other ACPX profiles) and OpenCode took a
different path: the runner closed the lost turn as a provider failure
> - The task then blocked with "Provider turn failed", and the user's
queued follow-up stayed unread
> - A restart must not need a user action when continuing is safe
> - This pull request gives ACPX and OpenCode the same recoverable
restart-loss cause and the same single continuation as Codex
> - The benefit is that work interrupted by a controller restart
finishes on its own

## Linked Issues or Issue Description

Refs #15297 (Codex continuation), #19399 (restart-interrupted turn
hook), #19471 (harness fix used for the live reruns).

**What happened?**

After a control-plane restart during an active native ACPX Claude turn,
the task ended `blocked` with `native_provider_terminal_failed: Provider
turn failed`. The queued follow-up comment was not consumed. No
continuation started. The next run then failed with
`native_session_cleanup_quarantined`. A native OpenCode run killed
mid-turn ended the same way (`turn.failed` with `error: null`).

**Expected behavior**

When the old provider process is gone and the conversation can be
restored, continue the task once in the same conversation. When that is
not safe, record an honest restart cause instead of a generic provider
failure.

**Steps to reproduce**

1. Start a native ACPX Claude task on a local runner.
2. Submit a follow-up comment while the turn is active.
3. Restart the control plane while the turn is still running.
4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id
everyday-workflows.runner-acpx-claude.local.recover-controller`.

**Root cause**

- `runner-core/src/acpx_provider_backend.rs`
`restore_session_if_needed`: a saved `turn_active` state always closed
the session. It recorded `turn.failed` with only a top-level `code:
acpx_active_turn_recovery_closed` and no `error` object. It also emitted
`run.terminal`.
- The runnerd transport forwards only `error`, so the controller saw
`error: null` and reported generic `provider_turn_failed`.
- `runner-core/src/provider_backend.rs` gave the recoverable
`provider_turn_lost_on_restore` cause only to `provider_label ==
"codex"`. OpenCode shares that backend.
- `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The
runtime continuation gate accepted only `provider.kind === "codex"`.
- Two follow-on defects appeared once the continuation ran live:
- The restored ACPX provider rejected the controller's result for the
lost turn's open semantic call. That call never settled.
- The restored provider's MCP request ids restart. The controller keys
semantic calls by `hash(runId, callId)`. The continuation's first
Paperclip tool call reused an earlier id, was treated as already
answered, and hung. After the result was accepted,
`runner_settlement_incomplete` failed the run.

## What Changed

- ACPX runner: for an active turn lost to a runner restart, prove that
the original provider lifetime lost its fence quorum. Then restore the
same conversation and record `turn.failed` with `error: { code:
"provider_turn_lost_on_restore", recoverable: true }`. Keep the run open
for the controller's continuation. An accepted result, an unanswered
runtime request, or an active goal keeps the old close behavior. So do
an unproven lifetime fence and a failed restore.
- ACPX runner fallback close: add a structured `error`
(`acpx_active_turn_recovery_closed`, not recoverable, with an honest
restart message), so the cause survives the transport.
- ACPX runner: track open semantic calls durably. On restore, retire the
lost turn's open calls. Acknowledge their controller results as
`retired` and do not send them to the new provider.
- ACPX sidecar: when the sidecar restores an existing session, prefix
forwarded semantic `callId`s with a per-process namespace
(`r<hex>:<id>`). Fresh sessions keep raw ids.
- OpenCode runner: give OpenCode the same recoverable cause and open run
as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays
excluded because its agent runs remotely.
- Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and
`opencode_server`. The runtime continuation gate accepts `acpx` and
`opencode`. All other Codex safety rules are unchanged: one
continuation, admitted dead-runner restart only, no compaction, goal,
semantic result, or pending runtime request.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.

## Verification

Red, then green, at each layer:

- Rust `tests/acpx_warm_attachment.rs`
`restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`.
Red: status `closed`. Then red on the lost call's result: "tool result
does not match a pending provider call". Green after the fix.
- Rust `tests/native_provider_backend.rs`
`restart_lost_opencode_turn_records_a_recoverable_cause`, with the
qualified OpenCode launch fixture. Red: `error` was `null`. Green after
the fix.
- Rust unit test
`restart_lost_turns_continue_only_for_locally_restored_providers`
(Codex, OpenCode yes; Cursor Cloud no).
- TS `native-session-runtime.test.ts` "continues a restart-lost
acpx_runtime / opencode_server turn once". Red:
`NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green:
exactly one continuation, which does not resend the task.
- TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are
namespaced and resolve.
- Server `native-runner-restart-recovery.integration.test.ts`: a
checkpointed `acpx_runtime` or `opencode_server` restart interruption is
claimed as `resume_dead_runner`. Red without the driver gate. Green with
it.
- Two `native_provider_backend.rs` tests asserted the old
close-on-restore result. They now assert the new contract. Their real
invariant still holds: the delivered response is never replayed.

Suites run locally on the final revision:

- runner-core: `cargo test --no-fail-fast acpx`, `--test
native_provider_backend` (38), `--test acpx_warm_attachment`, `--test
codex_provider restart_lost`
- runner TS: `native-session-runtime`, `runnerd-restart-continuation`,
Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass)
- server: `native-runner-restart-recovery.integration.test.ts` (80
passed)
- `pnpm --filter "@paperclipai/paperclip-runner..." build` (all
generated-contract checks pass)

Live runner E2E (local, real Claude via ACPX, macOS arm64):

- `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS
twice in a row on the final runner code. The restart crossed an active
turn. The task finished `done`, the queued follow-up was consumed, the
ZIP download was verified, and there were no failure codes.
- `stock-harness.runner-acpx-claude.local.continuity-restart`
regression: PASS.
- Earlier failing reruns are kept as evidence of the follow-on defects:
one harness `kill EPERM` while stopping the server (unrelated to this
change), one hang on the colliding call id, and one
`runner_settlement_incomplete`.

## Risks

- A continuation can run more work in the restored conversation. Its
prompt requires inspecting state before repeating an uncertain action,
as for Codex.
- The ACPX restore waits up to 5 s for the original provider lifetime
fence before it falls back to the old close.
- The ACPX provider state file gains two fields with defaults. Older
saved states still load. A downgraded runner cannot read state that a
newer runner wrote with these fields.
- Restored ACPX sessions now forward prefixed call ids. Only fresh
sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor,
Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves
them only by unit tests and code review.
- OpenCode is covered by the runner fixture and gate tests. The live
OpenCode restart-chaos cell was not rerun here.
- `runner-core`
`unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
is flaky when run in parallel. It also fails with the new path disabled,
so it predates this change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use, code execution, and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 11:39:29 -06:00
Nicky LeachandClaude Opus 5.5 83c6832ccb fix(build): typecheck package, Playwright, and e2e test files in CI (#18839)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner, plugin, and Playwright tests verify how those agents run,
connect, and appear in the board
> - Most package tsconfigs exclude tests, and Vitest strips types, so a
type error in those tests fails nothing
> - The tests drifted to 803 real type errors, and some of those errors
hid assertions that checked nothing
> - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check
the server test tree and four test projects, but not these files
> - This pull request adds a required check for these test files, fixes
every error, and fails CI if a test file is in no program
> - The benefit is that a type error in any test file outside
`server/src` now fails CI

## Linked Issues or Issue Description

Refs #18739
Refs #18738

**What happened?**

517 test files were in no program that required CI checks. A type error
in those files did not fail CI.

**Expected behavior**

CI fails on a type error in any test file.

**Steps to reproduce**

Add `export const value: number = "text";` to a test file under
`packages/paperclip-runner/src` on master. CI stays green, because
`tsconfig.json` excludes test files and Vitest strips types.

**Measured gap**

Each row is a program with its tests included, on one thread. File and
error counts are for head `7d4b5e907a`. Memory peaks were measured
before the two latest merges of master.

| Group | Test files | Real errors | Peak |
| --- | ---: | ---: | ---: |
| `paperclip-runner` Node source | 290 | 644 | 1.09 GiB |
| `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB |
| `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB |
| `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB |
| 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB |
| `plugins/sdk` | 22 | 15 | 0.41 GiB |
| `plugin-workspace-diff` | 5 | 0 | 0.37 GiB |
| `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB |
| `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB |
| `packages/db` scripts | 1 | 0 | 1.01 GiB |
| 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB |
| `tests/runner-acceptance` | 3 | 0 | 0.58 GiB |
| `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB |
| `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a |
| **Total** | **517** | **803** | |

27 of the 803 errors came from master while this pull request was open.
Each merge of master brought new test code that no check had read.

Errors that were config errors, not real errors:

- One TS2688 for each provider, because the root install does not
install provider dependencies.
- 9 in the Runner Node program (TS7016 and TS6059).
- 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that
config was invalid.
- 213 in the Playwright suites (`@/*` aliases and Vite types).
- About 60 in `tests/e2e` (import extensions and aliases).

`packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig
excludes its test, but `cli/tsconfig.json` reaches that test through an
import. `server/test-baselines` also has no gap, because
`tests/lifecycle-baseline` reads it.

## What Changed

- Added `scripts/typecheck-test-programs.mjs` and its tests. The
`Typecheck + Release Registry` job runs it. It checks each program in
`PROGRAMS` on one thread.
- The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file
outside `server/src` is in no program. The compiler file list decides
this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists.
- Added a `tsconfig.test.json` to each uncovered package, and
`tests/tsconfig.playwright.json`.
- Added `tests/e2e` to `PROJECTS` in
`scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a
`tsconfig.layers.json`.
- Added `scripts/typecheck-sandbox-providers`. It is a small workspace
that installs provider SDKs from one lockfile. I did not add a lockfile
inside each provider, because
`scripts/build-standalone-public-packages.mjs` uses a frozen install
when one exists, and that changes what a release resolves.
- Corrected `packages/paperclip-runner/tsconfig.browser.json` for
TypeScript 7.
- Set `rootDir` to `..` in
`packages/paperclip-runner/tsconfig.test.json`, because one Runner test
imports from `packages/adapter-utils/src`. The program does not emit
files, so this changes no build output.
- Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the
version the other providers use. Vitest 3 requires `tinypool ^1.1.1`,
which has two critical advisories (GHSA-85c8-ppgw-ccpr and
GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have
the same result on both versions: 189 passed, and the same 12
`file-sync` tests failed.
- Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave
its two `z.record` calls a key schema. This is the only production
change, and it stays in this pull request. See "Production change for
review".
- Renamed `AttentionInteractionResolver.test.ts` to
`AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the
`.test.tsx` beside it.
- Fixed all 803 errors, with changes to test files and test support
only. The fix for `contracts/native-execution.test.ts` is the same as
the fix in paperclipai/paperclip#18739, so those two changes merge
cleanly.
- Updated `doc/DEVELOPING.md`.

This pull request does not edit `server/scripts/typecheck-layers.mjs`.
paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both
merged, and this branch includes them.

### Production change for review

The new lockfile made dependency review flag `zod@3.25.76` as a
high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version
is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to
validate operator config. The zod 3 source does not compile against zod
4, so a typecheck-only override cannot fix this.

The existing tests do not prove this change, because they also passed
with the invalid one-argument `z.record`. So I parsed 45 inputs with the
same schema on zod 3.25.76 and zod 4.6.7:

| Result | Inputs | Detail |
| --- | ---: | --- |
| Identical | 23 | Includes defaults, valid config, and a `__proto__`
key |
| Same outcome, new error wording | 21 | For example, `Invalid url` is
now `Invalid URL`. `validateConfig` shows this text to operators. |
| Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was
valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. |

I did not test this against a real cluster.

The decision is to keep this change in this pull request. Since then,
paperclipai/paperclip#17364 has moved the rest of the repository to zod
4.6.x, so the Kubernetes plugin now matches it.

### Behavior change for review

`reducedMotion` under `use` is not a Playwright option. It had no effect
in 11 configs and 2 `test.use` calls, so those suites ran with
`no-preference`. The options are now under `contextOptions`, so reduced
motion now takes effect. `app-shell-design.spec.ts` asserts a real
transition, so it now asks for `no-preference` itself.

The baseline manifest has `snapshotCount: 0`, so this invalidates no
published baseline. A frozen trial reference PNG can differ.

### Findings

These are product questions. I did not change them, and I did not hide
them with casts:

1. **`PrpEvent` and its schema disagree.** The TypeScript type in
`src/protocol/replay-contract.ts` requires `normalizedSessionId`. The
`required` list in `protocol/schemas/event.schema.json` does not include
it. `validatePrpEvent` accepts events that the type forbids.
2. **`requestCheckboxConfirmation` is stricter than its runtime.** It
uses the output type, so plugin authors must pass `continuationPolicy`
and `allowDeclineReason`, although the schema gives them defaults.
`z.input` would fix this.
3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.**
The quarantine domain comes from `descriptor.kind`, but the release uses
`proof.backend.kind`. If they differ,
`completeRetainedNativeSessionCleanup` returns 0 and the quarantine
stays. The production caller passes `"runner"` today, so this is not a
live bug.
4. **`CodexHarnessSession.close(input?)` is optional, but the contract
requires `{ reason }`.**

These are tests that verified nothing. I corrected them, and they pass:

- `backends/native-instruction-measurement.test.ts` filtered on
`placement === "core"`, which is not a valid placement. It measured an
empty tool list. It now measures 14 tools.
- `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`,
which is not an event type.
- The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts`
ignored two overrides, so
`expect(prepareSandbox).not.toHaveBeenCalled()` could not fail.
- `live/live-session.test.ts` set `agent: "pi"` before a spread that
overwrote it.
- `native-distribution-integrity.test.ts` passed an AbortSignal where
`openCommand` expects `{ signal }`.

These are gaps that this pull request does not close:

- **The root lockfile still has some old zod versions.**
paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x.
`pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`,
which are below the patched 4.6.6.
`packages/paperclip-runner/scripts/pi-distribution/package.json` pins
`zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency
review does not flag them, because this pull request does not add them.
- **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts`
returned early for GitHub, so four later `provider === "github"`
branches never ran. I removed them. They included the private key file
and paste match, and the reconnect flow. Nothing asserts those behaviors
in that spec now.
- **Provider tests do not run in CI.** The root `vitest.config.ts` lists
only Daytona. The other 8 providers, `bridge-template` included, are
typechecked now, but CI does not run their tests.
- **One `packages/db` test never runs.**
`scripts/clean-poisoned-claude-sessions.test.ts` is outside the
`include` of `packages/db/vitest.config.ts`.
- **`plugin-orchestration-smoke-example` does not collect.**
`@paperclipai/shared` is not linked in its `node_modules`. It passes 3
of 3 with an alias.

## Verification

Run these commands:

```sh
node scripts/typecheck-test-programs.mjs --low-memory
pnpm --filter @paperclipai/server typecheck
node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations
node --test scripts/typecheck-test-programs.test.mjs
```

Results on macOS (arm64), Node 26.12. CI uses Node 24, and its
`Typecheck + Release Registry` job also passes. At head `7d4b5e907a`,
the test programs, `tests/e2e`, and the script tests pass again with 0
errors. The times and peaks below come from the run before the two
latest merges of master:

| Check | Result |
| --- | --- |
| Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak
1.78 GiB |
| Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41
GiB |
| Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds,
peak 1.59 GiB |
| Script tests | 12 passed |
| Injected type error in each of 23 places | Each one failed its program
with TS2322 |
| Coverage check | 0 unchecked files, 10 seconds |

I did not run this in a 4 GiB container. The peaks above are `maximum
resident set size` for the largest single process.

**Runtime check of the fixes.** Types do not prove that a test still
works, so I ran the suites before the two latest merges of master:

| Suite | Result |
| --- | --- |
| Runner, full suite | 4,885 passed, 3 failed, 13 skipped |
| Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped |
| Plugin SDK | 200 passed |
| `kv-demo-mcp-server` | 12 passed |
| Renamed UI test and its sibling | 4 passed |

All 15 failures exist without these fixes or come from the local
machine:

- **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and
`cursor-cloud-driver.test.ts` has 2. They time out in a full local run,
and all 133 tests in those files with `codex-security-config.test.ts`
pass when run alone. They fail in the same way with those two files at
their unedited versions.
- **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same
way before and after the fixes. The first failure is that
`/proc/self/fd` does not exist on macOS.

After each of the two latest merges of master, I ran only the Runner
test files that I changed:

| Merge | Result |
| --- | --- |
| `885c249cb9`, 8 files | 534 passed |
| `7d4b5e907a`, 7 files | 672 passed, 16 failed |

The 16 failures are in the two `source-reconciliation` test files. They
fail in the same way with master's unedited versions of those files. 15
of them fail with `ACPX sidecar exited at initialize`, so I think my
local build of the Rust and sidecar binaries is out of date. I did not
rebuild them to check. CI builds them from scratch, and all Runner jobs
pass in CI at `7d4b5e907a`.

**Not run:**

- `tests/e2e` and the 8 Playwright suites need a server, so most of them
have only compile checks. `playwright test --list` loads 308 tests from
the main e2e config.
- `packages/plugins/sandbox-providers/kubernetes/test/integration` needs
a cluster, so it has only compile checks.
- I did not run the full `pnpm test:run` or root `pnpm build`. Required
CI runs them.

**Reduced motion.** I ran 6 motion-sensitive Storybook specs in system
Chrome with a temporary config, once with `reduce` and once with
`no-preference`. Both runs had 45 passed and the same 5 failed. So
reduced motion did not change those results. The 5 failures do not
depend on motion. At least one comes from the browser, because system
Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0
0)`. I did not run the other Storybook specs. The `storybook-visual`
label runs that workflow.

## Risks

- **A new lockfile to maintain.** A change to a provider `package.json`
now fails the typecheck job until someone refreshes
`scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure
message gives the command.
- **The job has a new install.** The step downloads provider SDKs, so it
depends on the npm registry. It uses `--frozen-lockfile` and
`--ignore-scripts`.
- **The local install replaces provider `node_modules`.** A contributor
with a standalone provider install must reinstall that provider after
running the script.
- **Reduced motion can change Storybook results.** See "Behavior change
for review". That workflow is not part of required CI.
- **Kubernetes plugin validation changed.** A release of this plugin
ships zod 4. Error wording changes, and very large integers are now
rejected. See "Production change for review".
- **Master moves fast.** Each merge of master brought new test code with
type errors, 27 so far. After this merges, the check reports those
errors in the pull request that adds them.

## Model Used

- Provider: Anthropic Claude, through Claude Code
- Model: Claude Opus 5.5 (`claude-opus-5-5`)
- Capabilities: tool use, code execution, and subagents for the type
fixes in separate files

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived
details. The app generated the branch name.
- [ ] I have run tests locally and they pass. See the failures explained
in Verification.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 07:33:59 -08:00
DottaandClaude Opus 5.5 6ca7b151f7 feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses

## Linked Issues or Issue Description

Refs paperclipai/paperclip#19273

**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.

**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.

**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.

## What Changed

- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.

## Verification

- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.

## Risks

- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:07:12 -06:00
DottaandPaperclip 22735cfcea fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path

> - Paperclip manages work done by AI agents.
> - A native turn can yield to a saved human question or approval.
> - The provider must stop, and its terminal event must be saved.
> - The old drain waits for both the event transaction and its later
observers.
> - A slow observer can make a saved terminal look unproved.
> - This change separates the committed receipt from owned observer
work.
> - Checkpoint and result acceptance still wait for that work to settle.

## Linked Issues or Issue Description

**What happened?**

A native governed wait can fail with
`governed_wait_provider_terminal_unproven` after its exact interrupted
terminal is already in PostgreSQL. The event port waits for postcommit
accounting, output and projection callbacks before it returns the
receipt. Those callbacks can cross the provider drain deadline.

**What did you expect to happen?**

A committed terminal proves the event transaction. Its observers remain
separate work. A checkpoint or result must not be accepted until those
observers settle. A pending human-only question must keep its audience
and continuation policy.

**Steps to reproduce**

Run the real PostgreSQL control-plane regression. Commit a same-turn
interrupted terminal. Hold its postcommit observer beyond the existing
provider drain. The parent code reports an unproved provider terminal.
The fixed code retains the committed receipt and joins the observer
before checkpoint or result acceptance.

No overlapping repair was found in the current master or open-PR search.
Related [#18977](https://github.com/paperclipai/paperclip/pull/18977)
covers final conversation-response selection. It does not change this
receipt boundary. The source reproduction is consistent with the
observed failure. The exact hosted provider code is unavailable, so it
does not prove that every observed failure used this branch.

## What Changed

- Add an explicit terminal-only option to the internal event port.
Ordinary appends keep their existing behavior.
- Return that receipt only after the normal transaction and company,
agent, run, session, source and revocation checks.
- Retain the full observer promise. Join it before checkpoint and result
writes. Keep its original rejection and observe it after an aborted
join.
- Reuse the existing provider drain and checkpoint deadlines. Preserve
duplicate and restart replay without another model turn.
- Document the receipt and duplicate-observer replay boundaries. Keep
human-only and no-continuation cards unchanged.
- Assert the actual Codex facade requests deferral only for its
interrupted terminal. Isolate empty skill inventory in monitor-only
process fixtures while retaining their real scheduler, lifecycle,
selection snapshot and three-second deadline.

## Verification

- Meaningful parent red: actual native runtime and real PostgreSQL event
port. The terminal row commits, but a held observer causes
`NativeProviderTerminalFailure` with
`governed_wait_provider_terminal_unproven`.
- Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full
runtime suite, 203 passed. Real PostgreSQL control-plane, timing and
monitor suites, 103 passed. Runner TypeScript and full layered server
typechecks passed on the unchanged candidate source tree. The controls
cover uncommitted and foreign evidence, explicit false and failed
terminals, held and rejecting observers, abort and controller
revocation, duplicate receipts, and new-port recovery. The real pending
human-only card stays unanswered with continuation policy `none`.
Recovery does not call `startTurn` again.
- `pnpm -r typecheck`, the production server build (including the real
Runner release binary), module boundaries and vendor dependency
verification passed on `6d75`; the follow-up changes only tests and
documentation. Diff checks pass on the current candidate. The prior
`399ad3dc2b` repository-wide `test:run` was stopped as obsolete during
the general-server phase after preserving its progress (exit 130,
incomplete); its repository build was not reached. Its six completed
chat shards passed 1,904 cases, but this is prior-head evidence and not
a full-suite pass. Full current-head CI is required before merge.
- CI attempt 1 on `6d75` failed one real three-second monitor-settlement
assertion; the aggregate verify gate consequently failed. Other jobs
passed. The actual CI merge tree was
`3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An
unchanged full-file replay reproduced three timeouts. Original-deadline
inspection found the legacy process still preparing; local phase timing
measured cold skill inventory at 6.629 seconds. The original CI did not
retain that phase detail, so its exact subcause remains unproved. The
fixture correction supplies empty skill entries for these skill-free
process tests and still invokes the selection-snapshot callback. It
preserves all assertions and the three-second deadline, with no new
drain wait. All 21 cases pass; deliberately restoring the old
blocked-status rejection still fails the real blocked-check case with
409. No CI retry was issued.
- The prior current-head native review was 4/5 with no production
correctness finding. Its requested direct facade-options assertion and
duplicate-observer wording correction are included. Current-head CI is
green (55 success, 3 skipped), including required verify and E2E;
current-head Greptile is 5/5 with no actionable findings. The old native
advisory remains tied to `6d75`, and its requested findings are
addressed. No new native assessment is claimed or required by
protection.
- The first draft broke three existing transaction-start timing
controls. The no-debt path now keeps the original synchronous start, and
all three pass. Two draft type errors were corrected. A further real-PG
red found an extra default-append assertion after commit. Removing it
restores the original committed receipt after a late abort; the same
aborted signal still cannot write a checkpoint or result. Original setup
and assertion failures remain in private verification receipts.
- Normal merge of current master
`3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The
affected production source blobs did not change in that baseline merge.
- Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL
instance and deterministic provider-boundary fixtures. It makes no model
call. No deployed replay or original-task recovery is claimed.

## Risks

Deferred observers must remain owned. A missed join could accept
incomplete accounting or output. A rejected or stalled observer must
prevent acceptance, and an abort must prevent late checkpoint/result
writes. The tests exercise these boundaries. The port keeps the existing
best-effort trace-publication handling. This change does not turn a PRP
terminal into physical-stop or accounting-completion proof.

## Model Used

OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not
expose the reasoning mode or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 14:11:03 -06:00
Devin FoleyandPaperclip 4e29f00139 fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner keeps one ACPX provider conversation across the
per-turn runs of a task. Each run cold-attaches the session that the
previous run settled.
> - The saved ACPX state binds the digest of the launch profile that
wrote it. Every deploy ships a new provider pack, so that digest
changes.
> - After a deploy, the next run of such a task fails `run.attach` with
`ACPX durable launch profile digest does not match runner startup`.
Every same-run retry fails the same way.
> - On one instance this happened seven times. Six tasks used their
retry budget and ended blocked, or in progress with no runner. In the
seventh, the scheduled same-run resume never ran at all: the task stayed
`in_progress` with nothing running, and its monitor did not fire.
> - This pull request re-binds a settled session to the upgraded launch
profile, types the remaining mismatches so the server stops retrying
them, and makes a scheduled remote resume survive the stale-lock sweep.
> - The benefit is that a task continues its conversation after a
deploy. When it cannot, it stops with a clear next action instead of
stalling.

## Linked Issues or Issue Description

Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes).
Refs #15795, an open change to the same stale-lock sweep function. It
changes run teardown, not lock cleanup, so a second merge needs at most
a small rebase.

**What happened?**

A per-turn native task used provider `acpx` with Claude, driver
`acpx_runtime`, on a remote sandbox.

1. Run A succeeded and settled its provider session. The next run with
the same build cold-attached it.
2. The instance deployed a new build.
3. A board approval woke run B in a fresh sandbox. Run B failed after
about five minutes: `native_session_recovery_failed: Error: PRP command
run.attach failed: {...
"result":{"code":"command_execution_failed","message":"ACPX durable
launch profile digest does not match runner startup"}}`.
4. The coordinator recorded `retryable_failure`, attempt 1,
`native_session_interrupted`, recovery mode `exact_checkpoint_resume`,
recovery state `resuming_session`, and a retry 30 seconds later.
5. Two seconds after the failure, the stale-lock sweep cleared the
task's execution and checkout locks, because run B was `failed`.
6. Forty minutes later the attempt was still 1. The scheduled monitor
had not fired. The task stayed `in_progress` with nothing running.

**Expected behavior**

A session that the previous run settled continues after a deploy. If it
cannot continue, the run stops at once with a clear next action. A
scheduled resume either runs or fails visibly.

**Steps to reproduce**

1. Save a suspended ACPX session under one launch profile.
2. Change the sidecar bytes and the provider-pack authority at the same
paths.
3. Cold-attach the next run in a new runner. Restore fails with the
digest mismatch.
4. For the stall: leave a remote native run `failed` with a due
`retryable_failure` coordinator, run the stale-lock sweep, then run
restart recovery for that run. It returns `blocked:
issue_execution_lock_changed` and writes nothing.

The new Rust and PostgreSQL tests run these steps.

**Paperclip version or commit**

`master` at `b0c4e345267`.

**Deployment mode**

Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies
to every ACPX agent with an ordinary startup.

## What Changed

### Root causes

- **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the
provider-pack authority, the sidecar command and arguments, each
artifact path and SHA-256, and any candidate admission. For a remote
sandbox the authority is the provider-pack manifest digest, which covers
the Runner source revision and dist digest. So every deploy changes it.
`AcpxDurableState` stores only the digest, and `validate()` rejects any
difference. Only a Kimi candidate run binding had a carve-out.
- **Silent stall.** The failure write keeps the task locks for
`retryable_failure` and clears them only for `terminal_failure`. The
stale-lock sweep treats any `failed` run as cleanable, so it cleared
them. A remote retry goes through `claimNativeRestartRecoveries`. That
claim requires `issues.execution_run_id` to name the run, so it returned
`blocked: issue_execution_lock_changed` with no coordinator write and no
deadline. The live recovery queue then dropped the entry. The local
claim (`claimNativeSessionResumptions`) already reclaims a free lock, so
only remote runs stalled.
- **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while
any native run for it has a coordinator that is not `committed` or
`terminal_failure`. Run B's `retryable_failure` held it. That is by
design; it ends when the resume runs.

### Runner (Rust)

- `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore
re-binds a saved session to the current launch profile only when all of
these are true:
- The lifecycle is `suspended` and an identity exists. There is no
active turn, pending runtime request, undelivered provider event, or
unconfirmed provider exit.
  - The startup has no candidate admission.
- The saved descriptor's sidecar command and arguments equal this
startup's.
- Any other digest mismatch keeps failing with the same message, plus
`classification: "provider_state_incompatible"` in the durable command
result. `DurableRunnerError` carries an optional closed classification
for this.
- A saved descriptor that names an agent release this build does not
qualify (ACPX version, agent server package or version, or command
digest) also gets the class. `validate_session()` and its message are
unchanged; restore only adds the class.

### Runner (TypeScript)

- `runnerRecoveryFailureClass` reads the classification from the
authenticated `runnerCommandResult`. The Codex app-server driver (which
also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on
as `failureClass`.
- If replacement after a failed resume is not allowed,
`executeNativeSession` throws `NativeProviderStateIncompatibleError`
(message unchanged: `native_session_recovery_failed: ...`). If
replacement is allowed, the existing path opens a fresh session in the
same run, as before.

### Server

- `nativeSessionFailureSourceCode` maps that error class to
`native_provider_state_incompatible` with `instanceof`, not text. The
disposition is permanent: `terminal_failure` at once, the task blocks,
and the board owns the recovery action. Its next action says that a
Retry continues in a fresh provider session with the same workspace.
- `findNativeSessionResumeRun` also returns the barrier's `errorCode`.
`rebindNativeSessionCheckpoint` does not resume a source with
`native_provider_state_incompatible`. The existing fallback in
`buildNativeExecutionWithCheckpoint` then starts a fresh normalized
session with the full task bootstrap.
- `sweepStaleIssueLocks` keeps both locks while a referenced run's
coordinator owns a result-less `retryable_failure`, unless the task is
done or cancelled. Its update also requires that no referenced run has a
live status at the write, because a resume claim can return the run to
running after the status read.
- `claimNativeRestartRecoveries` reclaims a free execution lock for a
`retryable_failure` resume, like the local claim does, when the checkout
lock is free or held by this run and no other run for the task is
queued, running or scheduled. It records
`native.recovery.execution_reclaimed`. A lock held by another run still
blocks.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` and
`doc/execution-semantics.md`.

### Safety argument for the re-bind

What the binding protects, and what this change keeps:

- **Candidate admission** carries company, model, platform and expiry
authority. A candidate startup never re-binds. The Kimi run-binding
carve-out and all candidate rejections are unchanged.
- **The agent that loads the conversation** stays exact. `validate()`
runs on the saved descriptor before the digest check and requires this
build's qualified agent profile: ACPX version, agent server package and
version, and `commandDigest`. The saved identity must still match that
profile and the provider mode.
- **In-flight provider work** is never re-bound. A turn, runtime
request, undelivered event or unconfirmed exit keeps the strict binding,
so no old provider process or request continues under a new build. The
next launch only loads a settled conversation.
- **Launch shape.** A changed sidecar command path or arguments still
rejects. Only the content at the same paths may change.
- **Artifacts** are not trusted from the saved state. The current
profile comes from the trusted controller at startup, and session start
verifies every artifact's bytes and the restored provider identity, as
before.

What changes: a new provider-pack authority, or new sidecar or Node
bytes, at the same paths no longer invalidates a settled conversation.
The saved digest cannot show which part changed, so the rule does not
depend on it. I did not add state fields to tell authority changes from
artifact changes, because the rule would treat both the same.

One existing assertion changes.
`released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile`
expected a settled ordinary Kimi session to reject a changed authority,
artifact digest, or saved digest. Those three cases moved to a new
positive test. Command, arguments, every candidate grant field, a
missing profile and run binding still reject.

### How a task in the reported state recovers after this deploys

The task has old-digest saved state, a `retryable_failure` coordinator
at attempt 1, and cleared locks.

1. Startup recovery, and the live recovery queue, run restart recovery
for the due run. It reclaims the free execution lock and claims the run.
The resume runs now.
2. Run B's failed `run.attach` is a durable outcome of B's exact run
authority. If the resume replays that outcome instead of executing it
again, it fails with the old, untyped text. The remaining attempts then
fail the same way, and the run ends `terminal_failure` with
`native_session_retry_exhausted`. The task blocks with a board recovery
action. This takes about a minute.
3. If the task blocks, press **Retry** once. Retry selects run B as the
source (it has process identity and controller evidence, so the #18860
skip does not apply) and rebinds B's checkpoint, which holds A's
provider session. The new run cold-attaches under the new Runner, and
restore re-binds the old digest because the session is suspended and
quiescent. The conversation continues. Run B's failed attach never wrote
the state file, so the state is still A's settled state.
4. Tasks that already exhausted their retries (blocked, or in progress
after `native_session_retry_exhausted`) need the same single Retry.
Their earlier attempts ran under the old Runner.

Wait for the task to show blocked before pressing Retry. A Retry pressed
while the resume is running joins that run.

## Verification

- Rust, failing first:
`settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed
on `master` with the production message.
`unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed
on the missing classification. Both pass now.
- `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test
binaries pass. The library has 471 passed and 1 ignored.
- `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner
typecheck` (TypeScript, generated contracts, `cargo check`): pass.
- Server, failing first: `native-scheduled-resume-lock.test.ts`
(PostgreSQL). On `master` the sweep cleared the locks and restart
recovery returned `issue_execution_lock_changed`. Both negative controls
(another run's lock, a queued run) already passed. All four tests pass
now.
- `pnpm --filter @paperclipai/server typecheck`: pass.
- Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`,
`src/drivers/codex`, `src/drivers/provider-failure.test.ts`,
`src/live/runnerd-codex-transport*.test.ts`,
`src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011
tests pass. New: the typed error when same-session recovery is required,
and the recovery class reader and pass-through.
- Server vitest: the native executor, status corpus and provider-pack
admission suites (1,268 tests) pass. New: the classifier makes the typed
error permanent, and the same text stays retryable. Checkpoint selection
returns `errorCode`, and an incompatible source starts a fresh session
with the same workspace (unit and PostgreSQL).
- Server vitest over the recovery area: stale-lock sweep, native
resumption, process recovery, failed-workspace recovery, restart
recovery (integration, diskless, preparation), controller recovery,
startup retry, startup cleanup, recovered-owner follow-up, session
resume, runtime selection and recovery deadlines. 926 passed. Four
`heartbeat-process-recovery` tests failed in that loaded run. Three pass
when I run them alone. `redacts opaque environment-bound credentials
from Sentry diagnostics: result` fails the same way on unmodified
`master` on this host (a local environment value is redacted).
- Review round 1: a new sweep test returns the run to running before the
clear. It cleared the locks without the write recheck and keeps them
with it. The strict Rust restore test covers an older agent server
release and an older ACPX release. The ACPX backend and durable runner
unit tests, the lock, sweep, resume projection and controller recovery
suites, `cargo fmt --check` and the server typecheck pass. In the second
full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s
for a child exit on this loaded host. They do not use the changed code
and passed in the first full run.
- After the rebase onto the current `master`: the new lock test,
failed-workspace recovery, stale-lock sweep and session resume suites
pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and
`git diff --check` pass.
- CI on `929ef4508d2`: in the first run, `isolated-repository-rehome`
hit its 15 s test timeout, `idle-completed-history` could not reach its
fixture runtime service, and e2e shard 3 was cancelled by a runner
shutdown. A rerun of the failed jobs passed. All 55 checks pass.
- Not verified: a live remote sandbox deploy, and whether run B's
same-run resume replays its recorded `run.attach` failure or executes it
again. The Retry path is tested at source selection and checkpoint
rebind (server) and at cold attach (Runner). It is not tested end to end
through a sandbox.

## Risks

- A settled session now continues under new sidecar bytes. If a sidecar
change made an old ACPX record unreadable, session start fails as any
failed resume does. It does not corrupt state, and the run follows its
normal recovery policy.
- The sweep now keeps locks for a failed native run with a scheduled
resume. If that resume can never be claimed (for example the agent stays
paused), the task keeps the lock until the coordinator settles or the
task closes. Before, the sweep cleared it, and the resume still never
ran.
- If another run holds the task's lock, a scheduled remote resume still
stays blocked and the coordinator stays `retryable_failure`. This change
does not settle that case. It needs a separate decision about
superseding the resume.
- `native_provider_state_incompatible` is a new failure code. No UI or
migration change is needed. Older servers never see it.
- No schema migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
(shell, file edits, test runs). The runtime does not expose the context
window or reasoning configuration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 16:30:58 -08:00
DottaandPaperclip 3910352c92 feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives each provider durable execution, tool, and completion
contracts.
> - Cursor Cloud currently has a direct adapter outside that native
contract.
> - Cloud work needs exact ownership across HTTP, SSE, Stop, and
recovery.
> - This PR adds a qualified Cloud API driver and verified proxy to
Runner.
> - Product setup and hosted placement remain separate dependent
changes.

## Linked Issues or Issue Description

Refs #5664, #11803, #17299, and #17471. Product integration is #18069.
Managed-only API placement is #18310. The related direct-adapter
model/transcript work is not replaced. No duplicate native Cursor Cloud
integration was found.

## What Changed

- Add native `cursor_cloud` / `cursor_cloud_api` with explicit
model/repository validation, bounded HTTP/SSE, durable per-turn
checkpoints and a qualified Node proxy launched through the Rust facade.
- Commit terminal custody before publication/release; coalesce
consumers, preserve ambiguous create/Stop/recovery fences, reject late
tools, and distinguish definitely-unsent oversized requests from
ambiguous responses.
- Support scoped cold attachment, current authority, reconnect and
verified artifact staging. Keep unsupported warm/external and Windows
paths closed.
- Preserve early source cursors and verify immutable legacy prefixes
before suffix replay, including supported compaction/masking. Optional
proof publication failure falls back to complete cold verification.
- Preserve canonical records at the durable native producer with
JSON-safe privacy and the existing 8MiB bound; drain admitted successor
flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and
retain S3 and legacy CLI checks.

## Verification

Core head `37c442ec8dd4`: all 66 check contexts are terminal
successful/neutral/skipped, including required `ci / verify` and `ci /
e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads.
Dependent Product `81e828d20a77` and Top `dd2959689627` include the
later client handoff correction and are running their own fresh gates.

Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and
normal Runner build passed. Runtime Top `9af1a5de42a2`: build and
recursive typecheck passed. Earlier source-scoped evidence includes 398
focused tests, five relay tests and token/type gates. The 251-assertion,
7/8-suite run failed physical PostgreSQL teardown; the unchanged stable
WSP suite passed 128 through the documented wrapper.

The downstream fixture corrections passed 7/7 browser and 121/121
startup tests without timeout/oracle relaxation; a broken-unsubscribe
control fails. Hosted runtime `9af1a5de42a2` has a passing normal build
and recursive typecheck. Latest dependent heads additionally contain a
pure client fix for stale live/history overlap, with 48 focused tests,
UI typecheck and token gates passing. Its new client is not deployed in
the frozen hosted image; server/native/API/provider source is unchanged
by these four downstream files.

Fresh hosted Linux admission on `9af1a5de42a2` retired its owned
runner/proxy/socket with zero prompts. The saved configuration survived
reopening, and Test authenticated the bound key/catalog/selected model
without a prompt. Artifact with `claude-sonnet-4-5` completed,
downloaded the exact 27 bytes, matched 121 canonical identities against
complete logs and independently reached FINISHED. The active Stop retry
was clicked within its immutable boundary, produced one canonical
interrupted turn and acknowledged run-only cancellation, retained a
valid 35.076s quiet window (maximum gap 4.584s), and independently
remained CANCELLED/artifact-absent beyond the original 180-second
command deadline. All provider-observer credential callbacks, DB clients
and loaders closed; accounting is settled and explicitly unpriced. The
first fresh Stop retains its failed 10.373s sampling gap and every
sample.

On the upgraded9af server, the historical F226 artifact was reopened and
downloaded with its exact original hash after its provider agent had
been deleted/GET404 earlier today. Historical F226 full Artifact/Stop
cleanup, Question, corrected Plan and reconnect retain their original
identities. Deleting the three newly created disposable provider agents
and the new-case post-delete grade remain blocked on explicit user
approval after automatic review rejected expanding the two original
approved IDs. No new-case cleanup or complete fresh matrix pass is
claimed; this is separate from the directly observed task/Stop behavior
and retained exact cleanup proof.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier startup-title, title-save,
custody/Stop and deletion failures with unknown causes and bounded
recoveries. `dea023` Artifact passed 17 checks but Stop failed its false
warning and 50 duplicate log identities. Pre-repair ACK failure and
original Core CI failure remain recorded; retry alone establishes no
cause or full local-suite pass. Provider dollar cost stays UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. The caller’s
explicit values are submitted and behavior checked live. Unpriced
provider dollar cost stays UNKNOWN.

Only qualified per-turn controllers are admitted. Uncertain work and
incomplete cleanup stay fenced. Existing damaged/duplicate archives
remain failed evidence. Prefix proof adds bounded object-store I/O; the
8MiB producer bound does not guarantee future providers’ output fits.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:38:57 -06:00
Nicky LeachandPaperclip f8a888a072 feat(runner): define bounded startup lifetime contracts (#18490)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native sessions and plugin workers have separate startup lifetimes.
> - A caller timeout does not prove that a worker or an accepted host
write has stopped.
> - Shared startup code needs an exact deadline and an actual settlement
contract.
> - This PR adds those contracts and tests them before any production
caller uses them.
> - The dependent integration in #18405 will use the contracts with
verified cleanup.
> - This split lets reviewers assess the lifetime rules separately from
runtime activation.

## Linked Issues or Issue Description

Refs #18405. This PR extracts the foundation from that change. The
integration will depend on this PR.

**Problem or motivation**

A bounded startup must distinguish the caller's wait from actual worker
completion. A late worker reply or an accepted host request can outlive
the original RPC promise. A shorter timeout must not discard that
responsibility or authorize a replacement run.

**Proposed solution**

Add one Core execution context, optional Runner startup deadlines, and
worker settlement and retirement APIs. Test the contracts with real
child-process IPC. Keep production context creation, transport
projection, capability negotiation, durable cleanup, and Stop/recovery
integration in #18405.

**Alternatives considered**

One combined PR mixed the lifetime contract with its activation and
cleanup policy. Activating a shorter deadline before the complete
transport and cleanup path is unsafe. Splitting only the SDK
declarations would not provide a useful behavioral review boundary.

**Roadmap alignment**

This supports the existing plugin, sandbox execution, and bounded
recovery capabilities. It adds no new product workflow, endpoint,
migration, provider or runtime dependency.

## What Changed

- Add `NativeExecutionContext`. It retains the selected input, binds one
transport target, rejects work after cancellation or expiry, and tracks
actual worker settlement. Expiry alone does not complete pending work.
- Add worker startup groups. A caller timeout retains the group's
responsibility. Retirement revokes obsolete invocation scopes, waits for
accepted host handlers, and retires only the verified group's
bookkeeping. Each cleanup wait is bounded; an unfinished handler remains
pending for a later retry.
- Add an optional Runner startup signal and absolute deadline for
session open and retained attachment. Clean up abort listeners after
synchronous failures. Existing callers keep their default timeout path.
- Expose the remaining confirmed controller-lease budget without a new
query or renewal timer.
- Add optional SDK context and preparation-retirement fields. No
production bridge sends or advertises them in this slice.
- Keep the required lease mock update with the new `remainingMs()`
contract so this foundation typechecks independently.

Production activation files, including the native executor and
environment task bridge, match the current master base. No production
caller constructs `NativeExecutionContext` or passes the new Runner
startup option. This PR adds no durable cleanup producer and claims **no
standalone TTFT improvement**. The dependent integration must enable the
context, transport and verified cleanup together.

## Verification

Initial split verification on
`8ebc8a2f9c6c02e90f920356138b3742a4067b28`, based on
`9a3f395737e7a92faf99971459979061d423531a`:

- From `server`: `../node_modules/.bin/vitest run
src/services/native-runtime/native-execution-context.test.ts
src/services/native-runtime/native-execution-context-worker.test.ts
src/services/native-runtime/native-execution-lease.test.ts` — **17
passed**. Includes real child IPC, caller timeout with late reply,
worker death, accepted host-handler draining, obsolete cleanup
invocation rejection, unrelated work preservation, context scope,
expiry, and confirmed lease budget.
- From `packages/paperclip-runner`: `../../node_modules/.bin/vitest run
src/native-session-runtime.test.ts` — **188 passed**. Includes existing
callers without the new option and new
synchronous-failure/expired-deadline cases for fresh and retained
sessions.
- Direct TypeScript checks for `packages/plugins/sdk/tsconfig.json` and
`packages/paperclip-runner/tsconfig.json` — passed. Built only their
shared/eval TypeScript prerequisites.
- From `server`: `../node_modules/.bin/tsc --noEmit` — passed.
- Checked the partition against the reviewed combined candidate. The
twelve selected files match exactly. The thirteenth file carries only
the required test-mock update. The production executor, bridge and
target wiring remain equal to the base.
- `git diff --check`, added-line secret/private-data scan, and
added-code-comment scan — passed. No generated files or run artifacts
are committed.
- The first old-base split passed its focused tests but failed server
typechecking because the unchanged executor test mock lacked
`remainingMs()`. The original failure is retained. Moving that one-line
mock update into this foundation resolved the split dependency; the
final current-master checks above pass.
- Full local CI suites, Rust builds, deployments and provider calls were
not run, as requested. GitHub CI will provide full regression coverage.
Current-head GitHub CI, standard review and Apex are complete; exact
acceptance is recorded below.

Review follow-up `d7635f4d280dab757a3666703737bcda955c9361` fixes three
reproduced lifetime edge cases. A retiring startup group rejects new
calls and shares one retirement promise; failed bounded retirement
remains retryable. Transport identity uses a fixed field tuple rather
than object-property order. Finishing a context rejects pending lookup
waits without aborting an already-started backend or pretending the
worker completed. All three new regressions failed before the fix; the
updated foundation selection passes **20 tests**, including actual IPC
and failed-retirement retry. Server typechecking and added-line checks
pass. These replace the earlier 17-case selection, rather than adding 20
unique cases to it. The original failures and passing reruns are
retained.

Apex follow-up `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` keeps removed
worker handles reachable while they own startup calls, accepted host
handlers or retirement work. Drain/retirement includes active and
retained handles; completion callbacks remove settled handles. Stop and
replacement remove only the exact handle they observed. This adds no
database/network operation, polling loop or generic shutdown wait. The
SDK, Runner, context and transport bridge are unchanged.

Seven additional IPC regressions failed on the prior manager and pass
after the fix, covering removal/replacement, Stop-all, host-handler
settlement, bounded retirement failure/retry, pruning, and replacement
ownership. The updated foundation selection passes **27 tests**, with
server typechecking passing; these supersede the earlier 20-case
selection. A separate real-PostgreSQL/child-IPC characterization uses
the integration's actual SDK host, storage registry and run-control path
with the exact corrected manager bytes. Before the fix, cleanup released
run ownership while an accepted host write was blocked, then the late
write restored obsolete metadata. After the fix, ownership and cleanup
remain pending until the write completes; cleanup then removes the stale
metadata before releasing ownership. The original failure, corrected
result and source snapshots are preserved. Final head
`cb8860bab5ab9d6357a352e11f276be0d0cdb26d` passes GitHub CI (53
successful checks, 2 expected skips, Snyk success), standard Greptile
**5/5** and fresh Apex **5/5**. All review threads are resolved and the
PR is conflict-free.

## Risks

The shared Runner helper now observes synchronous operation failures
through its asynchronous cleanup path. The affected runtime file's 188
tests pass, but full regression coverage still depends on CI.

The new lifetime APIs require honest settlement signals. A future caller
must not treat a deadline, worker crash or Stop intent as verified
retirement. Production activation remains dependent on the complete
integration and its cleanup tests in #18405. This foundation introduces
no shorter production startup budget or new cleanup policy by itself.

The context's transport projection is process-local. It must not be
persisted or used as a general authorization cache. No production caller
supplies it in this PR. There is no database migration, new dependency,
or deployment requirement for merging this foundation.

## Model Used

OpenAI Codex, based on GPT-6, assisted with repository inspection, the
split, tests and command execution. The exact serving model ID and
context-window size were not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal issue
identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes — this
description documents the inactive contracts; operational activation
documentation remains with #18405
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on `cb8860bab5`
- [x] Current-head standard Greptile and Apex are 5/5 with no unresolved
findings
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 08:22:42 -08:00
DottaandPaperclip fe3e0f2d5f Preserve rejected native completion reports as failures (#18227)
Keep rejected declared completion reports on the existing permanent native failure path. Preserve rejection across restoration and order permanent failure ahead of terminal goal facts, goal results, and continuation while retaining the original provider terminal.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-29 18:49:01 -05:00
DottaandPaperclip 4808a3a2b1 fix(heartbeat): retry operator Stop after a transaction deadlock (#17895)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/69/changes/paperclip/69
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operator Stop must revoke task authority before it stops provider
execution.
> - Its transaction locks the task and then the run. A concurrent writer
can hold the run while it waits for the task.
> - PostgreSQL can abort the Stop transaction with a confirmed deadlock.
The cancel API then returns 500.
> - This pull request retries only that aborted database transaction, at
most three times.
> - Provider Stop and activity publication stay after commit, so the
retry cannot repeat those effects.

## Linked Issues or Issue Description

Refs #17639. Its agent-chat project browser test completed the product
assertions, then failed during cleanup when the cancel API returned 500.
A real PostgreSQL lock-cycle test reproduces the same `40P01` error in
the Stop transaction.

Related: #13911 addresses audit foreign-key lock contention. This change
handles a different concurrent run-writer path without changing the
schema or task/run lock order.

Stack base: #17639. This is a separate correctness repair for the sleep
migration stack, not another polling migration.

## What Changed

- Control the runner timeout regression fixture's clock so it reaches
the pending append before expiry; retain production cancellation and
late-append assertions.

- Retry the Stop authority transaction only after PostgreSQL reports
`40P01`. Make at most three attempts, with short bounded backoff.
- Discard rolled-back local run state, claim flags, and queued activity
publications before each retry. Re-read current authority under fresh
locks.
- Keep provider cancellation and activity publication outside the
retried transaction. Preserve other errors and the existing native
lock-conflict response.
- Add a real two-transaction deadlock test. Add server-side failure
tests for retry exhaustion and errors that must not retry.
- Let run-control tests use a validated disposable PostgreSQL database
URL.

## Verification

Published head: `60ba892ae9fc5faaf8cd0833694b4a66c13323c3`, targeting
master after #17639 merged at
`1ed0af41867ba40fd1d23292d395588d18c2fcbe`. The normal ancestry merge
preserves the complete five-path tested cancellation patch
byte-for-byte, including the review repair, and all four unrelated
current master paths. The production retry logic remains unchanged from
this slice's reviewed implementation; the follow-up adds coverage and
its audit.

- **228 distinct complete cases pass:** all 48 run-control cases and all
180 native-session runtime cases. The two new targeted cases are
included in 48 and are not counted again. Server typechecking, build and
module-boundary checks also pass.
- The new real PostgreSQL cases reach a failed native run's correlated
Stop claim, cancellation fence, board handoff retirement and activity
write before raising `40P01`. One case preserves retry authority and
commits exactly one cancellation activity/live notice. The other changes
coordinator authority between attempts, returns 409, preserves the
failed run and handoff, and performs no provider/workspace cleanup.
- A negative control removes only the publication-list reset. It fails
with two live notices for one durable activity. Production was restored
byte-for-byte before final validation. The existing actual
two-connection deadlock, bounded attempts, non-deadlock rejection and
post-commit provider behavior remain covered.
- Preserve the initial new-assertion typecheck failure (optional
live-event payload) and the corrected final pass. Service-local test
files are typechecked; `src/__tests__` is excluded. The negative-control
harness initially expected an untruncated Vitest assertion line; the
actual named 1-versus-2 failure and restoration were inspected before
correcting its result matcher.
- [Current-parent acceptance and failure
ledger](https://github.com/paperclipai/paperclip/blob/60ba892ae9fc5faaf8cd0833694b4a66c13323c3/doc/plans/2026-10-28-cancellation-deadlock-acceptance.md)
records current scope, commands, isolated fixtures, mutation evidence
and limits. Earlier combined-stack counts retain their original revision
scope. The existing controlled runner timeout/late-append fixture is
retained and passes within the complete 180-case runtime suite.
- This addresses the prior Ponytail request to exercise native claimed
state before deadlock rollback and re-evaluate authority afterward. The
advisory loop-invariant snapshot/named-attempt-bound note remains a
maintainability suggestion, not a runtime defect. No production loop
refactor or new worker duty is introduced.
- **All 58 current checks are clean; Greptile and Ponytail both rate
this exact head 5/5.** Ponytail explicitly confirms that the
native-claim coverage request is resolved and says Ship. The formerly
queued dependency/quality review workflow has completed successfully.
Old `fffc`/parent checks and reviews are not current passes; the former
pending Snyk status is not called successful. Actual failing Ponytail
findings must be fixed. Missing/stopped/paused turns without a verdict
are authorized nonblockers. Active unfinished reviews retain their
original per-head timer, with a 25-minute monitor cutoff respecting the
30-minute maximum.
- No full local repository matrix, live provider execution, external
alarm, physical host/database suspension or savings claim is made. No
schema, dependency, workflow, shared scheduler contract or shared OS
resource changes are introduced.

## Risks

- Persistent contention still returns an error after the third attempt.
This change does not promise to eliminate every possible lock cycle.
- Retrying a write after an ambiguous connection failure would be
unsafe. The retry accepts only PostgreSQL's confirmed deadlock abort
code; it does not accept transport errors.
- No schema or public API contract changes. Current task/run authority
and cancellation eligibility are checked again on each attempt.

## Model Used

OpenAI GPT-6 via Codex, with shell, GitHub CLI, TypeScript, PostgreSQL,
and Playwright tools. The exact serving model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:46:58 -05:00
Nicky Leach f5b6721ae0 Use the admitted session directory in native task prompts (#17310) 2026-10-23 05:48:09 -07:00
DottaandPaperclip d84061b923 fix(runner): recover journaled completion proposals (#16962)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native runs preserve provider sessions across controller restarts.
> - A completion event can reach durable storage before the next session
checkpoint.
> - Recovery advanced the event cursor but lost the saved completion
proposal.
> - The completed turn then appeared to need another provider turn to
report its result.
> - This change restores the exact active turn's proposal before
advancing the cursor.
> - Recovery can finalize completed work without repeating its external
actions.

## Linked Issues or Issue Description

**What happened?**

After a controller restart, a native run could fail with
`native_finalization_missing` even though `run.result.proposed` was
already durable. The recovered driver skipped that event when its cursor
advanced, so the result was absent from its state. Automatic disposition
retries could then encounter outstanding tool receipts and exhaust
recovery.

**Expected behavior**

Restore the saved result for the exact active turn. Wait for provider
termination, then use normal finalization. Do not start another provider
turn to repeat completed work.

**Steps to reproduce**

1. Start a native Codex task and save its active-turn checkpoint.
2. Persist its completion proposal, then lose the controller before
another checkpoint.
3. Recover the same run after the provider completes. Before this
change, finalization reports a missing result.

**Paperclip version or commit**

Reproduced on `92b238137b`. The repair is based on `7c247243e5`.

**Deployment mode**

Native Runner with Codex. The incident used a remote sandbox; the
regression covers controller event recovery.

Related: #12845 and #15297. The older #12241 addresses provider process
continuity; this change repairs the controller's saved event projection.

## What Changed

- Restore valid completion proposals for the checkpoint's exact active
turn during event replay.
- Reject conflicting, invalid, or incorrectly bound proposals. Ignore
proposals from another turn and prevent adoption after a newer turn
starts.
- Keep provider termination and normal finalization as separate
requirements.
- Add eight behavioral cases using the real Codex driver and a simulated
transport, and document the recovery contract and verification scope.

## Verification

- Red: the new restart regression failed with
`native_finalization_missing` before the production change.
- Green: 254 tests passed across the native runtime, backend adapter,
and Codex recovery suites. The new cases check no repeated provider
work, no early finalization, duplicate proposals, conflicts, invalid
proposals, and stale turn/session bindings.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- `pnpm -r typecheck` and `pnpm build` passed locally. All CI gates
passed on `2ac025a15c`; Greptile rated that commit 5/5 with no
outstanding findings. The full local `pnpm test:run` was started after
completing the embedded PostgreSQL installation, then stopped while
still running once the complete CI test matrix passed; it is not
reported as a completed local run.
- Real local acceptance passed on `2ac025a15c` with the production Rust
Runner transport, Codex CLI `0.160.0`, and `gpt-6.1-sol` on macOS arm64.
Persisted the completion proposal while retaining the older active-turn
checkpoint; let the provider finish without persisting its later
controller events; stopped the exact owned runner/provider; recovered
the same provider session and turn. Observed one accepted result, one
durable `turn.start` command, and one line in the receipt file. Hosted
sandbox restart and recovery of the original stopped task have not been
performed.
- Initial setup failures are retained: dependency drift caused
diff-viewer type errors, and skipped PostgreSQL postinstall links
prevented database startup. Dependency alignment and the package
postinstall repaired those local prerequisites. Probe setup iterations
failed on a direct-provider response schema, invalid probe input fields,
and an original provider that still held its writer lock. Acceptance
uses the production transport, validated input, and exact-owner
shutdown. No production code changes were made for those probe setup
failures.

## Risks

- A stale proposal could finalize the wrong work. Recovery requires the
checkpoint's active turn and matching run, session, and source. It
rejects conflicting results and waits for provider termination.
- The change uses the existing paginated recovery scan. It adds no
database query per event or new retry allowance.
- No schema migration or change to task authorization is included.

## Model Used

OpenAI GPT-6, running as Codex with reasoning, tool use, and code
execution. The exact served model snapshot and context-window size are
not exposed in this session. The live acceptance probe uses
`gpt-6.1-sol`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 09:09:22 -05:00
DottaandPaperclip d855e550ad Finish Copilot native runtime on current master (#15701)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner controls provider processes, permissions and task
completion.
> - Copilot needs its pinned distribution, native message identities and
accepted semantic completion receipts.
> - The release must preserve saved-token access and task lifecycle
guarantees.
> - This pull request will keep the current completion and recovery
fixes while adding the Copilot runtime.
> - The benefit is a reviewable part of the Copilot production release.

## Linked Issues or Issue Description

Refs #14434. Supersedes the runtime work in #15560. This is the first PR
in the stack; #15703, #15705 and #15706 follow. Related alternative
adapter work: #15881. This change uses the shared Runner path.

## What Changed

- Pin Copilot CLI 1.0.88 and bind the integrated implementation as
profile45.
- Keep native instruction, permission, receipt, process ownership and
cancellation boundaries.
- Preserve current master completion continuation and remote
verification behavior. Other provider release identities stay unchanged.

## Verification

Final reviewed source
[`4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5`](https://github.com/paperclipai/paperclip/commit/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5)
includes master `cde944ed75648adfb569e86133adc8b118b55dd6`. The combined
four-part PR tree matches it except for the CI-owned lockfile. Every
prior public PR head remains an ancestor. Released master migrations
remain unchanged; Copilot adds migration361. No unrelated provider is
enabled.

Copilot CLI1.0.88 and profile47 retain digest
`sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`.
The pending qualification marker is removed. The [qualification
record](https://github.com/paperclipai/paperclip/blob/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5/doc/plans/2026-10-20-copilot-qualification-v47.json)
reports **20/20 Product cells,154 assertions;7/7 protocol cases,46
assertions** against frozen installed application
`c2616370800f5f6df45246b0890ba35a7cd2faab`. Both environments use the
exact authenticated model `gpt-5.6-luna`. Every passing Product case
confirms cleanup. Automatic behavior retries are disabled. Original
failed attempts remain recorded.

The ordinary installed remote journey passed saved-token selection,
authenticated model discovery, normal UI save/reopen, metadata-only
Test, task completion, accepted receipt and an exact19-byte browser
download. An open model catalog caused no new discovery requests
during117seconds. Actual installed runtime smokes passed on macOS ARM64,
macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token
browser entry was not exercised; the encrypted connection was created
through the normal API.

The original Daytona file attempt timed out during delayed semantic
dispatch. Its precise stalled layer remains unresolved. Independent
ordinary delivery and the other19 cells passed before one instrumented
recheck passed8/8 with unchanged source,prompt,assertions and deadlines.
No causal latency repair is claimed.

After current-master reconciliation,129 connection tests,59 runner
tests,72 shared eval tests and88 recovery/workspace tests pass.
Migration generation/safety, unchanged profile identity and UI token
gates pass. Prior required CI was green. Fresh exact-head required CI
and E2E are green for all four updated heads. Each has a fresh
Greptile5/5 review and zero unresolved review threads. The [final
normal-package cloud
build](https://github.com/paperclipai/paperclip/actions/runs/39414239704)
is running. Recursive typecheck,full CI test shards,build and all8
browser shards pass. These checks cover the final source; no additional
local full-suite pass is claimed.

The [frozen image
build](https://github.com/paperclipai/paperclip/actions/runs/39334401638)
and [original normal package
build](https://github.com/paperclipai/paperclip/actions/runs/39334407471)
passed. The exact image is published and independently retrieved
anonymously. No separate full-image scan is claimed. All18 original
normal archives were scanned with zero selected credentials. Docker
builds run in the cloud. Provider USD remains unknown and
GitHub-attributed.

**Qualification is complete on the recorded frozen build. Normal
protected merge, automatic master canary publication and final
shipped-package local/Daytona canaries remain. No production deployment
is claimed.**

## Risks

Shared Runner behavior can affect active tasks. Exact profile admission
rejects stale runtime identities. The final campaign and installed
canaries remain release gates.

## Model Used

OpenAI Codex, based on GPT-6. The exact serving model ID and context
window are not exposed in this task. Used for code changes, review and
tool-assisted verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 06:31:47 -05:00
Nicky LeachandPaperclip fa9f3f8a08 feat(runner): support preconfigured user homes for Codex and ACPX (#16486)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Runner launches the selected provider and keeps durable session
state.
> - Runner currently prepares private provider homes for each session.
> - Operators can already have working login, skills, and MCP
configuration on the execution machine.
> - This PR adds an explicit mode that uses that existing setup.
> - Runner keeps its task bridge and session ownership while leaving the
user's configuration in place.

## Linked Issues or Issue Description

**Problem or motivation:** Rebuilding a provider home is unnecessary
when the execution account is already configured.

**Proposed solution:** Add `homeMode: "preconfigured"` to Runner adapter
config and runnerd transport options. Keep `"isolated"` as the default.
Initially support native Codex and ACPX Claude.

**Alternatives considered:** Continue copying user configuration into a
private home, or add an approval workflow. Both add work outside this
feature's scope.

**Roadmap alignment:** This extends the existing cross-provider Runner.
It does not add another runtime.

Related work: #14056 covers managed Codex credentials. #14474 covers
command environment delivery. This mode uses user-owned credentials and
retains the existing launch boundaries.

## What Changed

- Add a closed v7 launch contract for the selected home mode. Existing
contracts retain their behavior.
- Keep homeMode under operator control. Reject task overrides and
agent-authenticated changes, including removal through replacements,
adapter changes, or rollback. Ignore stored task values and exclude it
from automatic hire inheritance.
- Skip managed AI login, assigned-skill materialization, and external
MCP setup in preconfigured mode. Preserve the Runner task/completion
bridge.
- Keep protocol state and provider lifetime fences scoped to the Runner
session. Different sessions can share a home.
- Fail preconfigured ACPX planning/read-only launches before provider
admission until a read-only user-home launch is qualified. Use isolated
mode or native Codex for planning.
- Reject ACPX approval requests. Keep native Codex on `never`. Local
permission rules and hooks remain effective. Initially require the
default Runner ACPX `approve-all` setting; other Runner policies are
unqualified in this mode and fail before launch.
- Keep the internal ACPX bridge in a stable session namespace,
preserving an existing user MCP server named `paperclip`.
- Reject preconfigured ACPX Codex before launch until a session
permission profile can enforce Runner-state denial. Remove the
unprotected full-access mode switch; use native Codex or isolated mode
for that model family.
- Bump Pi to profile 23 because its attestation covers shared Runner
source changed here. Preserve profile 22 for historical recovery, and
update Docker/package verification inputs.
- Fence recovery on mode and home bindings. Fail missing
provider-history recovery without starting a replacement session.
Exclude the external Codex home from backups and credential cleanup.
- Discover remote HOME on the execution machine. Do not project
controller HOME or XDG paths onto it.
- Separate task instructions from Codex ACP isolation and update the
generated provider digest.
- Keep remote configured-environment manifests consistent when dropping
controller paths. Preserve exact Claude model IDs through per-launch
environment settings, without editing the user home or patching the
provider.

## Verification

- Targeted local checks cover native contracts and launch allowlists,
user-config preservation, provider admission, permissions, session
recovery, remote environment projection, heartbeat preparation, and
operator-only configuration. Runner, shared, and server TypeScript
checks and the Rust binary build pass; generated ACPX profiles are
current.
- Following the latest master integration, all 757 server
configuration/executor tests and 115 Runner admission/recovery tests
passed. Additional review-fix checks passed: 103 planning contract/host
tests, 78 provider-backend tests, 95 route/preparation/inheritance
tests, 43 task-validator tests, and 43 workspace-policy tests. Counts
describe separate runs and can overlap.
- The real pinned Claude ACP/SDK regression reproduces exact-model
aliasing without the fix (`claude-sonnet-5-5` becomes `sonnet`) and
passes with the fix. All 89 adapter tests pass. User settings remain
byte-for-byte unchanged.
- Localhost-only Claude SDK probes with synthetic credentials verify the
private bridge runs, local `ask` reaches the existing reject callback,
local `deny` blocks without a callback, and a user MCP server named
`paperclip` remains available. No user model allowlist is required.
- Review regressions cover fail-closed provider admission, saved-mode
recovery before routing, controller-home path removal without
configured-environment manifest drift, pre-launch rejection of
unqualified read-only ACPX launches, and denial of task/agent changes to
the operator-owned home mode.
- Before rebase, `d4d4ee2aebe4cb88c4f4d9ef8e22fed4b36b87be` received
standard Greptile 5/5 with no unresolved threads. Prior Apex findings
are fixed and covered by regressions. No manual Greptile retrigger is
requested for this rebase, per reviewer direction.
- The prior PR head and master hit the same GitHub runner encoding crash
in workspace-b on three PR attempts, including diagnostic logging
([prior PR
CI](https://github.com/paperclipai/paperclip/actions/runs/38818474563/job/116714538405),
[prior master
CI](https://github.com/paperclipai/paperclip/actions/runs/38819528748/job/116712423193)).
Rebased onto `origin/master` at `c3ffa3f0dd` to pull in its bridge-test
index correction. The rebased head
`edd153db410bea063b645e0000aa68649f287b00` passes all CI checks ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/38824676914)),
including workspace-b. Greptile's automatic standard review is 5/5 on
this exact head, with no unresolved threads; no manual review was
requested.
- Local workspace-b investigation passed 911 shared tests and 3,868
tests across the remaining 15 workspace projects. The catalog run
separately hit an existing ignored
`.agents/skills/design-guide/SKILL.md` description-budget violation in
this checkout; that file is untracked, outside the PR, and left
untouched. Repeating shared tests with CI reporting enabled also passed.
- Rebase verification: the final Git tree exactly matches the previously
reviewed branch combined with current master. All 192 Runner
admission/contract tests, 61 server preparation/provider tests, and 68
bridge tests pass, as do Runner/server TypeScript and generated-profile
checks. The real Claude adapter fixture hit an `ENOTEMPTY` cleanup race
on its initial run; all 89 adapter tests passed on retry.
- Apex on rebased head `edd153db` returned 2/5: ACPX Codex selected full
access without enforcing Runner-state denial, and agent configuration
replacement could remove the operator mode. Both are fixed in
`26e14dbbf`. Preconfigured ACPX Codex is temporarily unqualified;
re-enabling it requires session permission enforcement on thread/turn
operations and a real filesystem-denial probe. The config guard covers
effective replacements, adapter changes, and rollback while allowing
ordinary agent merges and operator removal. All 190 focused Runner
tests, 103 server tests, both typechecks, and generated-profile checks
pass. Standard review and CI passed before the next Apex request.
- CI on `26e14dbbf` passed 1,967 Runner-shard tests but hit the
previously observed `ENOTEMPTY` fixture cleanup race after successful
Claude model admission. `eb83284fb` adds bounded filesystem retries to
that fixture's teardown; all 87 adapter tests pass. Assertions and
production cleanup behavior are unchanged. Head
`eb83284fb42ee19a15d9a93b106c1bbb07e1695e` passes all 59 CI checks ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/38829500857))
and standard Greptile at 5/5 with no unresolved threads. Apex also
completed at **5/5** on that exact head ([Apex
result](https://github.com/paperclipai/paperclip/pull/16486#issuecomment-6201559527)).
All review threads are resolved, the branch has no merge conflicts, and
the PR remains unmerged.
- No live paid-provider or remote-machine smoke was performed. The
lockfile remains unchanged under the repository's refresh-bot policy;
local dependency installation resolved the changed patch hash.

## Risks

This explicit opt-in grants Codex tools access to the execution home
(read-only in planning), including credentials. Native Codex separately
denies Runner-owned protocol state. ACPX Claude retains its existing
provider trust boundary, and the isolated default is unchanged.
Providers may refresh credentials and update their own metadata.

User configuration can block tools or the completion reporter. Existing
blocked results and failure diagnostics remain the recovery path. Edit
the local settings and retry; no new approval workflow is added.

External provider history remains on the execution machine and is not
backed up by Runner. Remote HOME discovery requires `getent` and fails
explicitly when unavailable. Other harnesses need separate
qualification.

The Codex ACP bundle digest changes. Existing sessions tied to the old
bundle may need a fresh session. No database migration is required.

## Model Used

OpenAI Codex, GPT-6, with reasoning and code execution. The exact
runtime model ID was not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the feature issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 17:57:58 -07:00
DottaandPaperclip fc3b15537b fix(runner): match call_api to existing agent REST permissions (#16040)
Route native call_api through existing agent REST permissions. Preserve admitted credentials across restarts, protect sensitive replies and private snapshots, and compare saved completion with provider history through the same redaction projection. Add permission, privacy and recovery regressions and update the access map and run-log guide.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-13 07:34:15 -05:00
DottaandPaperclip b5342febe5 fix(runner): require current-turn completion after connection continuations (#15514)
## Thinking Path

> - Paperclip manages agents, tasks, permissions, and execution budgets.
> - Native tasks can resume after a connection decision in the same
provider conversation.
> - Each turn still needs an accepted completion report.
> - Compact continuation messages did not explain that reports from
earlier turns cannot finish the new turn.
> - The connection evaluator could also grade before the final reply was
stored or reject valid unavailable-access wording.
> - This PR clarifies the current-turn report requirement and fixes
those observation boundaries.
> - The original connection instructions and strict native completion
gate stay in place.

## Linked Issues or Issue Description

Refs #15489. The reduction remains draft while this separate repair is
qualified. Refs #15471 for the earlier connection continuation work.

## What Changed

- Add a current-turn completion reminder to compact continuation inputs.
- Keep final prose insufficient for completion. Preserve permissions and
retry policy.
- Wait for the final successful task run's saved, attributed decline
reply within the existing deadline.
- Use one bounded explanation matcher for both decline checks.
- Wait for a recorded tool-action rejection to dispatch its bound
continuation, with strict company, task, agent and source-run checks.
- Retain the exact grading input before later API refreshes.
- Add failure and delay regressions and update the Runner and evaluator
docs.

## Verification

The fresh comparison has **15/15 original passes on each variant**: 15
unchanged pass pairs, zero new failures, and no pending pair. There are
30 case attempts and **65 actual agent runs** (baseline 33; candidate
32). All runs succeeded. All 30 cleanups passed. No model attempt was
retried.

| Profile | Baseline | Candidate |
| --- | --- | --- |
| Native Codex `gpt-5.6-sol` | 5/5 | 5/5 |
| ACPX Claude `claude-sonnet-5` | 5/5 | 5/5 |
| OpenCode `openrouter/deepseek/deepseek-v4-flash-0731` | 5/5 | 5/5 |

Each profile covers service approval, service decline, connection
decline, provider decline, and selection of the second provider. The
saved replies, approved briefings, decisions, fixture observations,
final task states, and native completion records were inspected. All 18
saved decline-grade snapshots match their original captured inputs and
checks. Result, API snapshot, and final ledger run sets agree.

- [Candidate
campaign](https://github.com/paperclipai/paperclip/actions/runs/37711658378)
· [public candidate
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711658378-1/index.html)
- [Baseline
campaign](https://github.com/paperclipai/paperclip/actions/runs/37711675579)
· [public baseline
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711675579-1/index.html)
- Candidate source: `79905343bba280d462765faad19a26e7f179259e`. Baseline
source: `7c5e120158f1385a1fc5f20be41f66c58a605534`. Both use master
context `fc6304dfe5f2e446e09bd052a7b45f51e930f250`. The trusted workflow
source is separately frozen at
`dd777f4b7343305c4e6f44c422f44a1d78e12e4f`.
- Both variants use the same evaluator, fixtures, models, permissions,
720-second cell deadline, and 1,000-cent company and agent hard stops.
The only production difference is the compact continuation reminder.
- Suite hash:
`aee30b74b4d38ada08777798db0932fbc64e368bb427d29caa8cfd86c7f59747`.
Definition hash:
`ea9e17f54fe0af1acbb2337ddfab8a7e92db8488bd0f59510a63095ca0229b60`.
- Provider-free transport capture: startup/resume input stays at 55,726
bytes. Compact continuation input grows from 53,334 to 53,535 bytes. The
42-tool catalog stays unchanged. These are Paperclip input bytes, not
complete vendor prompt tokens.
- Focused evaluator tests: 31 pass. Native contract, transport delivery,
and session tests: 194 pass. Evaluator support: 1,819 TypeScript tests
and 128 Node tests pass, with one intentional skip.
- Full build, workspace typecheck, and evaluator typecheck pass.
Current-head CI passes all required gates. The current rollup has 51
successful check runs, two intentional Storybook skips, and a successful
Snyk status. Review is 5/5 with zero unresolved threads.
- CI attempt 1 had one initial runtime-fixture health timeout. The exact
test and its full 164-test file pass locally. One CI shard retry passed.
The original CI failure, its dependent verify failure, and the retry
remain visible in [CI
history](https://github.com/paperclipai/paperclip/actions/runs/37711235060).
- The broad local `pnpm test:run` attempt was interrupted after about 49
minutes (exit 130). It recorded one failure in the unchanged Zep
memory-connector disabled-setting test. That test and the full 388-test
tool-access file pass in separate local checks; current-head CI also
passes. The local cause is not established, and this broad local attempt
is **not** claimed as passing. Three earlier local failures also pass in
their isolated checks; their original logs remain retained.
- The first two setup admissions were cancelled before provider jobs to
include the review correction. They made no provider calls. The
completed campaigns above are the first and only model attempts for
these corrected variants.

Cost evidence stays separate from behavior. Original result summaries
report only OpenCode amounts: baseline $0.039192096 and candidate
$0.063221620. Final run ledgers also retain estimates for Claude
(baseline $1.232439000; candidate $1.333611200) and Codex (baseline
$2.340324400; candidate $2.129335600). These estimates do not replace
the original summaries. Local and GitHub runtime are unmetered here.
Invoices are unknown. This is not a cheaper or faster claim.

## Risks

- A single matched trial cannot prove general equivalence or causation.
The reminder is an instruction change, not a new completion enforcement
rule.
- Candidate OpenCode service-decline finished within one continuous run;
its baseline used two. That pair passed the task outcome, but it does
not qualify the reminder on a resumed decline turn. No extra paid run
was used to replace it.
- The text matcher is bounded evidence of an explanation. It does not
prove reasoning or consumption of feedback. Bounded stdout excerpts do
not prove that every extra attempted tool call is absent.
- Missing saved replies or continuations still fail at the original
deadline. Failed native completion remains a failure even when final
prose is correct.
- The unresolved local-suite discrepancy above remains a validation
limit. Full remote CI and both focused local reproductions pass.
- The original connection instructions stay in place. These results do
not qualify the reduction in #15489. Its original 11/15 versus 12/15
grades and two new failing pairs remain unchanged.

## Model Used

OpenAI Codex, based on GPT-6. The exact deployment ID and context window
are not exposed in this session. Capabilities used: reasoning,
repository editing, code execution, test inspection and eval analysis.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run the focused local tests listed above and they pass; the
interrupted broad local run is disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 04:52:12 -05:00
DottaandPaperclip fd8c6b920a fix(native): resume connection tasks after approval decisions (#15471)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native tasks can pause while a human decides whether to allow a
connection action.
> - The next turn needs both the saved decision and complete accounting
for the previous turn.
> - Cancellation could discard final usage, and complete direct Claude
API receipts could remain unpriced.
> - A stale blocked or review report could also request approval again
after the original card was declined.
> - This pull request retains shutdown accounting and rejects approval
waits bound to an already resolved action.
> - The benefit is reliable continuation with the existing budget and
approval controls.

## Linked Issues or Issue Description

Refs #15420. Related: #15312 addresses requester ownership during
dispatch. This change addresses receipt capture and final-response
validation.

## What Changed

- Retain usage events after native cancellation. Continue to reject late
provider messages and work.
- Drain same-turn accounting and terminal events for at most fifteen
seconds after a durable governed wait. Keep incomplete accounting
blocked.
- Estimate complete, unpriced, direct Anthropic API receipts for the
exact `claude-sonnet-5` model. Record the rate version and assumptions.
Use the one-hour cache-write rate when the receipt lacks cache TTL.
- Bind stale approval reports to exact interaction, action-request, or
invocation IDs in the same company, task, agent, and run. Cover blocked,
review, and response-wake reports. Keep independent reviews valid.
- Fence checkpoint and result writes after a controller detaches for
restart, including operations waiting for a database lock. Reject stale
successful returns before certifying accounting.
- Allow bounded subscription teardown only after retaining an actual
provider terminal.
- Journal the exact governed-wait trigger and disposition before
provider interruption. Recover that wait independently of a later saved
answer, replay retained accounting, and reject mismatched or unproven
terminal evidence.
- Give settling governed turns a bounded window before shutdown detaches
their controller.
- Keep fuzzy external app matches alongside installed capability matches
instead of forcing an unrelated provider question for a generic query.
- Return up to twenty exact active catalog tool names after an invalid
request, after eligibility checks; still reject the request without
granting access or creating an approval.
- Require retained provider terminal proof before settling a governed
wait, including when complete usage arrives before stream
closure/error/timeout. Retain harmless numbered cancellation events so
restart replay stays contiguous.
- Isolate accounting-test OpenCode config from the host plugin
directory.
- Add regression coverage and document the accounting, restart and
connection-search behavior.

## Verification

Current PR source: `0cf08efd75f8fb23f7989beda6dbda92587088bf`. The live
matrix below measured frozen `09a776bcb1f77422156f8be11e13f8c29f43e7f7`;
later review fixes are verified separately and do not relabel those
runs.

- Repository typecheck and build pass.
- Current runner runtime and cancellation suites: 191 tests pass. Six
new regressions cover stream end/error/timeout without provider-stop
proof and contiguous cancellation acknowledgement/request replay; all
six failed before the fix. The existing bounded cleanup case now
explicitly supplies terminal proof. All 31 adapter accounting tests pass
with isolated fixture config.
- New checkpoint-rebinding and approval-criterion suites: 65 tests pass;
runner HTTP integration: 29 tests pass. Unchanged executor/control-plane
suites: 640 tests pass; database-backed connection suites: 68 tests
pass.
- Current retained evidence verification covers 222 file hashes across
all fifteen original result artifacts. The three-case [published
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37651896525-1/index.html)
and all eight screenshot hashes verify. The [three-case recovery
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37662587356-1/index.html)
and all six screenshots also verify; the nine-result campaign did not
publish.
- Evaluation support: 1,805 Vitest tests pass, one skipped; 128 Node
tests pass. Eval typecheck and catalog discovery pass.
- Full local repository unit run was interrupted before the follow-up
edits after three tool-access failures and one runner HTTP failure.
Those failures pass in isolation; the 09a full run was interrupted after
one rapid Slack callback-ordering failure and seven skill-service
failures. All eight pass both isolated and with full-runner environment
settings, and all 74 skill-service tests pass together; the subsequent
full run reported two 15-second OpenCode accounting timeouts and was
stopped with exit 130 to apply review fixes. The timeouts reproduce
while copying this host’s 61 MB OpenCode config. All 31 tests pass after
isolating config inside each fixture without increasing timeouts or
changing assertions. A complete local full-suite pass is not claimed.
Repository-wide CI also passes on the final review-fix head in [run
37669185952](https://github.com/paperclipai/paperclip/actions/runs/37669185952).
Earlier database-skipped diagnostics and the older ENFILE run are
retained and are not full-suite passing evidence.
- Three-case live campaign
[37651896525](https://github.com/paperclipai/paperclip/actions/runs/37651896525)
passes all three original grades on frozen source 09a: 55/55 checks,
eight succeeded run records, complete accounting receipts, matching
checkpoint identities and no pending approvals. Campaign
[37653533353](https://github.com/paperclipai/paperclip/actions/runs/37653533353)
adds nine original passes (173/173 checks, eighteen succeeded records)
on the identical source. Its other three jobs failed before runner
assignment or any step while GitHub could not load the paid environment;
those original infrastructure failures are retained. Campaign
[37662587356](https://github.com/paperclipai/paperclip/actions/runs/37662587356)
completes only those unstarted cells: all three original grades pass
(57/57 checks, six succeeded run records). All fifteen exact cases now
pass on source 09a: eight FAIL → PASS, seven PASS → PASS, zero new
overall failures and zero pending pairs. Total current evidence: 285/285
checks and thirty-two succeeded run records, complete accounting
receipts, matching checkpoint identities, no pending approvals or retry
records. Earlier campaigns retain forty-seven additional run records and
two known same-run recovery attempts; actual provider-call counts and
invoices remain unknown. The nine-result campaign skipped publication
and its public URL returns 403; original artifacts remain retained.
Previous ba9 campaign
[37645656840](https://github.com/paperclipai/paperclip/actions/runs/37645656840)
completed 2 PASS / 1 FAIL: Claude restart/approval and Codex decline
pass, while OpenCode resumes but times out searching for exact tool
names and never creates the access card. That failure and incomplete
cancelled-run accounting remain preserved; the new catalog error
guidance targets this observed dead end. Campaign
[37642957312](https://github.com/paperclipai/paperclip/actions/runs/37642957312)
remains 0 PASS / 3 FAIL and exposed the now-corrected cross-run marker
leak and unknown-criterion approval gap. The original baseline remains 7
PASS / 8 FAIL, first repair 2 PASS / 3 FAIL, and second repair 0 PASS /
3 FAIL. All fifteen selected cases are qualified by their original
grades in this bounded trial. These live grades belong to 09a. Its
twelve governed-wait checkpoints retain matching same-turn terminal
fingerprints, but passing artifacts omit detailed event journals; the
later six adversarial regressions qualify the new terminal-proof and
replay guards separately. Final-head repository CI passes. [Fresh
Greptile
review](https://github.com/paperclipai/paperclip/pull/15471#issuecomment-6044369780)
is 5/5, confirms both findings are fixed, and reports no new actionable
issues. All review threads are resolved and the PR has no merge
conflicts.

## Risks

- Governed cancellation drains accounting for up to fifteen seconds.
Restart detachment gives a settling batch up to twenty seconds to
finish. An incomplete receipt or unproven provider terminal still
prevents successful qualification.
- Claude prices are estimates, not invoices. The estimate assumes
standard global API pricing and uses a conservative cache-write rate.
Unsupported models, billers, and billing modes remain unpriced.
- Approval identity matching must remain scoped to the current run and
the requested approval. It does not authorize execution of a declined
call.
- Original baseline and final candidate have different merged master
context. Exact-case outcomes are before/after observations, not isolated
causal attribution to this repair.
- No schema migration, fixture, oracle or grader change. Search-result
guidance now treats fuzzy external matches as suggestions. Existing app
authorization and provider-consent checks remain required.

## Model Used

- OpenAI GPT-6 through Codex, with code editing, terminal tools, and
test execution. The exact deployment identifier and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (191 current runtime tests
and 31 accounting tests; interrupted full-suite history and CI coverage
are disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-07 14:43:07 -05:00
DottaandPaperclip 63f3aa2dbf fix(runner): continue restart-interrupted Codex turns (#15297)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs retain their provider conversation across server
restarts.
> - A dead runner can restore a Codex conversation after its active turn
is lost.
> - The old recovery path synthesized a failed task result from that
interruption.
> - The task then required operator action even though its conversation
and workspace were available.
> - This pull request preserves the interruption cause and uses the
admitted restart attempt for one continuation in the same conversation.
> - The agent can reconcile unfinished actions and complete the current
request without resending the original task.

## Linked Issues or Issue Description

Related: #12845 added native restart recovery. #15042 covers admission
during shutdown. #14796 covers legacy shutdown recovery. This change
covers a lost native Codex turn after successful conversation
restoration.

**What happened?**

After a server restart killed the local runner, Paperclip restored the
saved Codex thread. Runnerd found that the old turn was no longer
active. It synthesized a failed terminal and a needs-review result from
the last progress message. The transport also discarded the terminal
error when it reconstructed thread history. The task failed instead of
continuing.

**Expected behavior**

After proving that the old process stopped and admitting a bounded
recovery attempt, resume the current request in the same conversation.
Preserve the workspace. Inspect unfinished actions before proceeding.
Keep real provider failures, accepted results, intentional stops,
unknown unreconciled effects, and exhausted attempts subject to their
existing rules.

**Steps to reproduce**

1. Start a local native Codex run and leave its turn active.
2. Kill the isolated runner and provider processes, as can happen during
a server restart.
3. Restore the same provider thread with no active turn.
4. Observe the synthetic task failure. The new real-process regression
reproduces this boundary with a scripted provider.

## What Changed

- Record an explicit recoverable process-loss cause without inventing a
task result.
- Preserve terminal errors and prior turns in reconstructed provider
history. Recover the authoritative saved result when adopting an
accepted continuation.
- Send one continuation in the same conversation for an admitted
dead-runner recovery. Require reconciliation of unfinished commands and
external actions.
- Persist the interrupted terminal before submission and retain the
existing recovery marker across another controller loss.
- Keep provider attempt limits, terminal failures, and intentional
cancellation behavior.
- Add red/green regressions, real process-kill coverage, restart
checkpoint coverage, and retry-budget coverage. Document the behavior
and run-log evidence.

## Verification

- Red: the new native runtime regression rejected with
`NativeProviderTerminalFailure` on the original code; the Rust restore
regression found a missing recovery cause.
- Red/green: if restoring the conversation fails and replacement is
allowed, the replacement receives the full task and fresh-session
handoff. Both prepared and legacy execution inputs are covered.
- Red: a second controller crash after the provider accepted the
continuation caused an extra `turn/start`. The regression now proves
there are exactly two submissions total: the original and its
continuation.
- Green: focused runtime, backend, driver recovery, and real-process
restart suites (207 tests). After the final history/result changes,
driver recovery and real-process restart suites passed again (36 tests).
- Green: complete Codex transport suite (186 tests), server restart
classification/database integration suites (34 tests), and Rust Codex
provider suite (92 passed, 2 ignored).
- Full `pnpm -r typecheck` and `pnpm build` passed on `60141e649`. The
subsequent replacement-prompt guard passed the Runner TypeScript check
and the complete runtime plus process-restart suites (148 tests).
- Local full-suite attempt: `pnpm test:run` reported two failures in the
untouched chat integration suite. Both passed individually, and the
complete chat suite passed on rerun (1,063 tests). After all remote test
shards passed, the duplicate serial local run was stopped with SIGINT;
it is not claimed as a full local-suite pass.
- Latest-head CI (`b1297dcd4`): 55 successful checks and 4 intentionally
skipped checks, including all test shards, typecheck, build, native
Runner verification, end-to-end tests, and canary dry run. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/37405346303).
- Greptile: 5/5 on the latest head, with no unresolved review threads.
The PR is mergeable.
- The process tests use the real runner binary and a scripted Codex
provider. They do not call a live model service.

## Risks

- This changes local Codex recovery after process loss. A continuation
can execute more work in the retained conversation. Its prompt requires
state inspection before repeating an uncertain action; the system does
not replay tool calls.
- Recovery shares the existing three-attempt budget and one-shot
continuation marker. Real failures and older unmarked failed checkpoints
are not reopened.
- No database migration or API change is required.

## Model Used

- OpenAI Codex, based on GPT-6. The exact model ID and context-window
size are not exposed in this session.
- Capabilities: reasoning, source inspection, tool use, code editing,
code execution, and test analysis.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 07:49:47 -05:00
DottaandPaperclip 2ec82c5774 fix(runner): preserve task context when tool connections change (#14963)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use tools through company-scoped connections and provider
sessions.
> - Resolving a tool connection currently forces a fresh session even
when the provider can load new tools into the existing conversation.
> - A fresh provider conversation can receive too little history to
continue the task.
> - This pull request adds explicit tool-refresh capabilities and uses
them in both runner paths.
> - Fresh attempts receive bounded task history with source IDs and
retrieval instructions.
> - The benefit is that agents can continue the same task after a
connection changes.

## Linked Issues or Issue Description

**What happened?**

A resolved tool connection forced a fresh provider conversation. The new
conversation could lose the original goal and prior answers. Claude also
rejected resume when only the MCP server set changed.

**Expected behavior**

Resume the provider conversation when its harness can refresh tools.
When a fresh session is required, supply enough bounded history to
continue the task. Preserve company, agent, task, workspace, model,
instruction, and skill checks.

**Steps to reproduce**

1. Start a conversation and agree on a task and its constraints.
2. Request and connect a tool needed for the task.
3. Continue the conversation after the connection resolves.
4. Check that the agent remembers the task and can use the new tool.

**Paperclip version or commit**

The bug was reproduced on master at `c46e41e81`. This branch is rebased
on current master.

**Deployment mode**

Self-hosted server. Both legacy adapters and the native runner are
affected.

Related public work: Refs #13282 for task-backed conversations. Refs
#13057 for the broader session-compaction proposal. Refs #14659 for
another report about local CLI session continuity. This change fixes
tool-connection continuation. Provider authentication repairs keep their
existing recovery behavior.

## What Changed

- Expose tool-refresh support in native harness descriptors and legacy
adapter metadata.
- Request tool refresh after connection resolution. Keep provider
authentication repair as a fresh-session wake.
- Reload current tools and credentials while retaining supported Claude,
Codex, Grok, and other provider conversations.
- Allow MCP-only changes during qualified native recovery. Keep all
other compatibility checks.
- Refresh managed-provider and ACPX tool bindings when attaching a new
run.
- Add a fresh-session handoff for both runner paths. Bound database
reads, excerpts, and the final packet to 24,000 bytes.
- Include the original request, recent messages, decisions, plans, prior
answers, and source IDs. Mark omitted content. Apply reset boundaries,
wake cutoffs, quarantine, and secret redaction.
- Add regression tests and document the capabilities and handoff
behavior.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed. Rust formatting passed.
- Final review fixes passed 314 server tests, 333 adapter utility tests,
139 native-session runtime tests, and 12 managed-provider Rust tests.
They verify historical quarantine, raised budgets across attachment, no
history reads on successful resume, and handoff delivery on fresh retry.
- Broader branch verification also passed 1,401 adapter utility tests,
1,047 runner TypeScript tests, 43 Grok adapter tests, and 311 Rust core
tests.
- Live Claude CLI and Grok ACP probes preserved the provider session ID,
recalled a prior task constraint, and called a newly added read-only MCP
tool.
- GitHub CI passed on `b21486d18084a7aa4cafbe8e012f7cad6585d9cc`: 55
successful checks and 4 skipped checks. This includes all test shards,
all eight browser shards, runner checks, and the Grok clean public npm
install canary. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/37057514976).
- A full local test attempt encountered a separate Git snapshot timeout.
All affected local suites passed after the final edits, and the full CI
test gates passed.
- Review the capability matrix in `packages/paperclip-runner/README.md`.
Repeat the four reproduction steps with a supported provider and with an
unsupported harness.

## Risks

- Provider tool refresh can fail. Existing recovery falls back to a
fresh conversation where policy permits it.
- A new transport can replace an old process while preserving the
provider conversation. Tests cover current credentials and unchanged
identity.
- Long history can omit older context. Explicit markers and source IDs
let the agent retrieve needed context within task scope.
- Unknown and unqualified harnesses use the fresh-session path. No
database migration is required.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and live
provider testing. The exact model ID and context-window size are not
exposed in this session. Claude and Grok also ran as test subjects.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-02 15:11:44 -05:00
DottaandPaperclip 94e8dec56b fix(runner): preserve tool outcomes through shutdown and restart (#14734)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner sends authorized tool calls to the server and saves their
results.
> - A provider turn can stop while a server write is still running.
> - The old shutdown path invented a failed result that could conflict
with the real result.
> - Truncated execution input and incomplete recovery records made the
failure harder to diagnose.
> - This pull request preserves exact inputs and actual outcomes through
shutdown and restart.
> - Tests force the race and crash boundaries so safe retries do not
repeat writes.

## Linked Issues or Issue Description

**What happened?**

Stopping a turn during a server tool call could record a false failure,
then reject the actual result as a conflict. The diagnostic input
formatter could truncate instruction content before execution. A crash
during saved-result delivery could leave that delivery permanently
indeterminate. Cleanup could hide the first failure, and a retry could
overwrite earlier run logs.

**Expected behavior**

Keep dispatched tools pending until their actual result is known.
Preserve accepted input bytes. Accept identical result delivery without
failing the task. Reject conflicting results with enough evidence to
diagnose them. Recover saved-result delivery without repeating the
business operation.

**Steps to reproduce**

1. Hold an instruction update at the filesystem commit barrier.
2. Stop its provider turn before the server returns the result.
3. Release the write, deliver its result, and replay the same result.
4. Repeat with a restart before and after the delivery receipt is saved.
5. Check that there is one write and one audit row, and that the exact
result survives.

**Paperclip version or commit**

The change was developed from `44736c9c7` and rebased onto `0e5830887`.

**Deployment mode**

Self-hosted server with the native runner. Tests use local runner
processes, scripted providers, and PostgreSQL.

Related work: #12353 added durable semantic tool receipts; #12384 added
durable Codex tool recovery; #12404 bound semantic tools to ACPX
sessions. #14633 covers separate native-provider cancellation and
qualification work. This PR addresses server semantic-tool outcomes and
their durable delivery. No duplicate fix was found. AgentMail discovery
is outside this PR.

## What Changed

- Close turn admission without inventing results for dispatched tools.
Keep pending calls and accept late actual results.
- Accept identical result replay with a diagnostic warning. Include call
identity and both result hashes in real conflict errors.
- Preserve exact execution arguments. Reject prohibited or oversized
input before dispatch. Keep diagnostic previews redacted and bounded.
- Commit instruction-attempt evidence before the filesystem write. Save
completed mutation receipts so concurrent and restarted duplicates
return the first result. Recheck authorization before replay. An attempt
without a completed result stays unknown and cannot execute again.
Definite pre-write failures save and replay their original error without
another write.
- Recover an interrupted saved-result delivery only for backends with
durable result receipts. Never replay an ordinary business operation
with an unknown outcome.
- Preserve the initiating error when cleanup also fails. Record
incomplete settlement evidence. Propagate typed unknown-outcome errors
through the native tool wrapper without creating a false completed tool
result.
- Append run-log attempts and restore the durable log before appending
after local file loss. Reject incomplete restores. Publish a restored
prefix only if the destination is absent so concurrent attempts cannot
overwrite new lines.
- Add deterministic race, crash, replay, authorization, exact-content,
and log-restoration tests. Document their assertions in
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Current head: `7e088f4c7fba8ebabf98ae95485a5753b013d489`. All 55
applicable checks pass; four conditional/manual checks are skipped. This
includes build, typecheck, Rust, both runner TypeScript shards, server
and workspace tests, all eight browser shards, isolated runner
compilation, and the clean-install release dry run. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/36746101110).
- Greptile reviewed this exact head at 5/5 with zero new findings. All
three earlier review threads are resolved.
- Focused local verification includes 11 instruction integration tests,
23 surrounding authority/tool tests, 26 run-log tests, and 169
controller/driver tests. The post-rebase controller/transport/runtime
selection passed 415 tests. The full Rust release suite passed 617 tests
with two ignored. The real-process SIGKILL recovery test passed three
consecutive runs.
- The fault matrix in
`packages/paperclip-runner/docs/durable-recovery.md` uses explicit
barriers, real PostgreSQL rollback, durable journal reloads, and killed
runner processes. It covers late results, identical and conflicting
replay, exact long content, concurrent log restoration, lost commit
acknowledgements, and definite failure replay after the original CAS
base becomes valid again. No paid model calls are needed.
- Full local recursive typecheck and build passed during implementation.
Server typecheck and the runner TypeScript build passed after the review
fixes. The broad local repository test run was stopped after repeated
database startup timeouts. Four timing/launch failures in an earlier
broad runner run passed focused reruns without changed assertions or
timeouts. These are local verification limitations; the complete
current-head CI suite is green. An earlier CI workspace job received an
infrastructure shutdown signal; its current-head replacement passed.

## Risks

- A stopped turn can remain blocked when a dispatched operation has no
proven result. The system does not guess its outcome or rerun its
effect.
- Conflicting results still fail settlement. Existing failed or
conflicting journals are not repaired automatically.
- Accepted semantic input is limited to 480 KiB of encoded JSON to fit
the encrypted transport. Larger input fails before execution.
- Instruction filesystem writes and database receipts are not one atomic
storage operation. A separately committed attempt and audit record
survive rollback. An attempt without a completed success or definite
pre-write failure receipt remains blocked as an unknown outcome. It is
not replayed or reported as success.
- Run-log restoration now reads the durable object before appending when
the local log is missing. Failed or incomplete reads reject the append.
- No schema migration, dependency change, workflow change, or AgentMail
change is included.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and test
analysis. The exact served model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; the broad
local run limitation is recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-30 11:54:56 -05:00
DottaandFry 3ca196b0a6 feat(agents): persist agent files across tasks without revision history (#14420)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent needs personal files across tasks and sessions.
> - AGENTS.md is one file in that directory. Supporting files need the
same persistence.
> - The Instructions Editor and agent runs must share one current
directory.
> - Concurrent runs should apply only the files they change. The last
sync of the same file wins.
> - This pull request uses existing file transport and removes temporary
copies after sync.
> - Old instruction-only sessions keep their restore contract. New saves
do not create revision history.

## Linked Issues or Issue Description

Refs #14325. This replaces its revision-oriented design with persistent
agent files. Keep #14325 unmerged.

Transport prerequisite #14416 merged first at
`d172197117a14b80a1eb2d2835a0e7cce2679656`. This PR now targets master
and remains below 100 changed files.

Related work: #4513 and #8798 cover instruction tooling. This change
handles run synchronization, cross-task personal files, browser editing,
and old-session restoration.

## What Changed

- Keep one current directory per company and agent. Point AGENT_HOME at
a temporary working copy for each active run. Keep task files and
provider HOME separate.
- Restore text, binary files, and nested folders through workspace
transport. Exclude remote agent files from task Git snapshots with a
self-ignoring file inside the reserved runtime directory; never write
through repository-controlled Git metadata.
- Collect after the provider and child processes have stopped. Keep
resumable conversation state.
- Apply changed and deleted files under the agent lock. The last sync
wins for the same file. Unrelated concurrent changes survive.
- Remove temporary copies after successful sync, rejected sync, and
staging failure. Register ownership before copying so restart recovery
can remove interrupted preparation. Retry transient synchronization up
to three times. Preserve the original remote lease reference until
deletion succeeds; restart cleanup never acquires a replacement sandbox.
Do not create captured directories or a conflict-review queue for new
runs.
- Keep browser editing, stale-draft protection, and streaming binary
downloads. Keep the instruction entry and text editor limited to 1 MiB.
- Keep historical agent-folder sync failures on their affected runs
instead of repeating them above current saved instructions. Preserve
legacy candidate review and current browser-save errors. Avoid duplicate
quota warnings while retaining separate sync failures when they describe
a different problem.
- Require target-scoped caller grants for peer instruction access, while
preserving self edits, responsible-user checks, and protected-change
consent.
- Treat full storage as a nonblocking run warning, never an agent pause
or run-admission failure. Restore already-over-quota saved folders so
ordinary agent cleanup can recover; warn on each run until cleanup. The
run detail view shows the warning.
- Allow 256 MiB per file, 2 GiB per directory, and 100,000 entries. Hash
large files as streams. Check editor-save quotas with metadata instead
of hashing unrelated files.
- Preserve old native inputs, instruction-only copies, paths, digests,
and pending legacy candidates. Adopt old revision heads once. New writes
do not append history rows.
- Add idempotent migration 0287 and verify upgrades from the preview
tables and receipts.
- Add nine interactive stories under **Agents / Persistent files**,
including automatic incoming edits, stale browser drafts, and
storage-limit diagnostics.

## Verification

- Merge candidate: `4f5390107ec6ffd80a76d1d2e85530e66f21d079`, after
merging current master and the landed transport prerequisite.
Integration required no manual conflict resolution; the feature remains
99 changed files. Full workspace typecheck, production build, token
gates, and 715 focused tests passed on this merge candidate. Fresh
Greptile review is 5/5 with no unresolved findings. All 55 checks
passed, with four conditional skips, including the build, typecheck,
browser E2E, and canary dry run. A single retry recovered four jobs
interrupted by runner shutdowns; no source changes were required.
- Historical-warning UI fix: all 6,834 UI tests across 640 files passed,
including regression coverage for three old failures, legacy preserved
edits, and warnings scoped to the affected run. Full workspace
typecheck, production build, Storybook build, and token gates passed.
Browser-verified Storybook playtests passed for Historical Failures
After Successful Save, Storage Limit, and Full Storage Run Warning.
- Review follow-ups at `4e20c9fb2`: all 18 focused tests passed,
including external Git directories, linked worktrees, symlinks,
hardlinks, and distinct I/O failures alongside storage warnings. Server
and UI typechecks, token gates, and the production build passed.
- Storage warning regressions at `0724f3012`: all 33 directory tests and
all five heartbeat-list tests passed, with no skips in their successful
runs. They cover repeated runs while full, an already-over-quota saved
folder, cleanup, warnings retained after unrelated save failures, and
bounded warnings in large result JSON. Server typecheck passed after the
final warning fixes.
- Full workspace typecheck, production build, and token gates passed
during this follow-up. Product E2E harness: 631 tests passed across 52
files; harness typecheck passed. Earlier native session/context and
directory/legacy collection suites passed 537 tests; Runner
unit/transport suites passed 329 tests.
- **Real E2E at `0724f3012` (before this follow-up):** legacy local
Codex and native Daytona Codex each passed six tasks, one server
restart, seven independent assertions, and cleanup verification. Both
prove browser-to-agent edits, agent-to-browser edits, nested/binary
restoration, per-file last-sync-wins, a successful run after an
oversized save rejection, and cleanup clearing the warning.
- Native local Codex also passed the six-task quota flow before the
final warning-retention fixes. That pass began at `918d1ed02` while the
bounded-result warning fix was being edited, so it is not claimed as
exact-final-head evidence. Its final-head rerun failed during embedded
PostgreSQL bootstrap before any provider run: the macOS host had 87,365
of 87,381 SysV semaphores occupied. No unrelated services or kernel
limits were changed.
- The final-source report intentionally records **2/3 cells passed**,
preserving the blocked native-local attempt:
`tests/runner-e2e/results/agent-files-quota-final-20260928-report/`.
Earlier failed attempts and provenance notes remain under
`tests/runner-e2e/results/agent-files-quota-final-20260928-input/` and
the original campaign directories.
- Daytona used immutable image
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:5643f0d801417cae3581833a1a3bc6715b325e028602738d2652c44cac5dc6bf`
and its exact Linux runner binary. Controller source is `0724f3012`;
image source is recorded separately.
- Legacy-session compatibility and all three ACP Stop/resume browser
regressions passed on the prior validated feature head
`169fab46d5af21caa2269b4c1b29b69c933a6951`. They assert the same
provider session is retained and interrupted writes are not replayed.
Migration upgrade tests also passed earlier.
- Nine interactive stories are under **Agents / Persistent files**,
including **Full Storage Run Warning**. Its playtest and visual browser
inspection passed; the warning states that runs continue and the editor
remains available.
- Prior-head checks on `4e20c9fb2`: 55 passed, two conditional jobs
skipped, no failures or pending checks. All eight browser E2E shards and
their aggregate passed. Fresh Greptile review is 5/5 with no findings;
all review threads are resolved, the security scan passed, and GitHub
reports no merge conflicts.
- The broad local follow-up test run was interrupted after host
semaphore exhaustion affected isolated PostgreSQL instances. It also
encountered the existing macOS long-path fixture failure and two timeout
failures. This is not a claim that the full local suite passed. Logs are
retained; focused storage/warning tests passed.

## Risks

- A later sync can overwrite an earlier edit to the same file, including
a saved browser edit. There is no text merge or retained version. This
is the intended last-sync-wins policy.
- A save that exceeds a storage limit is rejected and its temporary copy
is discarded. The run itself continues normally, and later runs restore
the last saved files with a warning until cleanup. Transient sync
failures get bounded retries. An I/O failure partway through a sync can
leave some files updated; a failed receipt does not claim whole-folder
success.
- Larger folders increase copy time, network traffic, and temporary disk
usage. Active runs still need working copies. Terminal runs do not
accumulate archives. Operators must provision disk for agents and
configured concurrency; these limits are not company-wide quotas.
- A restored old native session remains instruction-only until a fresh
session starts. Its original conflict fence and existing pending
candidates remain compatible.
- Provider processes close at the collection boundary. Conversation
resume remains available, but warm process reuse is lost.
- Backups must include the instance filesystem and database. External
bundles keep their existing behavior until explicitly moved to managed
storage.

## Model Used

OpenAI Codex, GPT-6 family. The session does not expose a more specific
model ID or context-window size. Reasoning, code execution, and browser
tools assisted this change. Real provider E2E uses `gpt-5.6-sol`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Fry (Paperclip) <noreply@paperclip.ing>
2026-09-29 08:25:56 -05:00
DottaandPaperclip 992f720262 fix: make runner task context ownership explicit (#13753)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100). -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task descriptions, comments, continuation data, skills, and
execution rules enter several agent adapters.
> - The same source can be rendered by more than one automatic input
carrier.
> - Failed resumes can also rebuild input from stale or compact context.
> - This pull request gives each Paperclip-owned source one delivery
owner and preserves the required transport boundaries.
> - It adds deterministic adapter, interaction, runner, and browser
tests for these boundaries.
> - The benefit is more predictable context delivery with explicit
evidence for later live qualification.

## Linked Issues or Issue Description

Related: #13144 removes a duplicate environment payload and bounds wake
lists. Related: #11360 addresses Hermes resume behavior. This pull
request preserves compatible active-session formats while repairing
context ownership and stale question creation.

**What happened?**

Task descriptions and comments could enter more than one automatic
context block. Native transports could wrap a complete model input in a
second task envelope. Some legacy and gateway adapters could omit the
owned assignment on ordinary tasks or rebuild a failed resume with stale
compact context. A continuation could also request a question after
newer human comments had arrived.

**Expected behavior**

Each task or comment source has one automatic model-facing owner.
Distinct comment IDs and repeated wording remain distinct. Fresh
fallback attempts rebuild the required full context. A question request
is rejected when newer queued human direction makes it stale. Harness
access policy remains owned by execution configuration.

**Steps to reproduce**

1. Build a task with a description and current comments.
2. Capture the actual adapter or runner input.
3. Compare source ownership and task-envelope nesting.
4. Queue a human comment before a continuation requests a question.
5. Trigger a failed resume and inspect the fresh retry input.
6. Run the focused adapter, interaction, runner, and browser checks.

## What Changed

- Add shared prompt-section selection at the provider-attempt boundary.
- Deliver owned assignment context through native, legacy CLI, ACP,
gateway, cloud, Pi, Kimi, Grok, Gemini, OpenCode, Cursor, OpenClaw, and
Hermes paths.
- Rebuild full or compact context after resume recovery changes the
attempt. Add native and Claude ACP tests of actual recovery requests.
- Preserve custom templates, loaded instruction files, execution
policies, and older active-session formats.
- Record continuation source metadata and reject stale question creation
under the issue-row lock.
- Add explicit Product E2E context-integrity profiles, prerequisite
gates, credential-isolation checks, and report fixtures.
- Bypass service-worker forwarding for same-origin Vite development
modules. A real Chromium test fails with resource exhaustion before the
repair and passes after it. Production asset caching keeps its existing
policy.
- Add browser diagnostics and service-worker module-loading regressions.
- Add an explicit zero-retry eval option. The default retry behavior
remains unchanged. Each campaign records its effective policy.
- Remove the model-facing working-directory sentence from four prompt
builders. Existing workspace, sandbox, permission, and custom-template
configuration remains unchanged.
- Align the everyday workflow assertion with the current 47-entry
catalog.

Compared with current upstream master, the branch carries the
context-ownership implementation and its tests, the explicit
context-integrity catalog and evidence harness, and the focused browser
regression checks.

## Verification

**Merge assessment:** focused regression evidence supports merge. This
is not full completion of the original broad qualification matrix. The
maintainer has authorized merge after fresh verification of the master
integration.

- Current head: `bbd52f82114eabf09bc7b1a7e97d54a5b43bbc00`. This
integrates current master `2f585ef26a1814fa209715242d1ca791b63e4c4e`.
All 14 conflicts are resolved. Cancellation checks, workspace
finalization, native Grok support, and both sets of tests are retained.
- Current-head Greptile: **5/5**, with no blocking findings. The review
names this exact commit. All **59 reported checks are terminal: 55
successful, 4 skipped, zero pending or failing**. This includes the full
root general and serialized suites, separate runner checks, typecheck,
build, canary, browser E2E, Docker, and security checks. The successful
legacy security status is included in that total.
- After integration: workspace typecheck and full build passed. Separate
runner checks passed: **2,160 TypeScript tests (10 skipped), 582 Rust
tests, and 39 preparation checks**. Other passing checks include 621
Product E2E harness units, 376 focused shared/adapter tests, 160
real-database/API tests, 86 Hermes tests, 18 browser-support checks, and
Product E2E typechecking. The complete root suite passed in CI. The
duplicate local monolithic root run was stopped after that CI result; it
is not counted as a completed local pass.
- New native recovery coverage retains full assignment, completion
contract, and explicit skill selection after safe replacement, for old
and prepared input formats. Full native session test file: **136/136
passed**.
- New Claude ACP coverage captures actual fresh, resumed, and
missing-session fallback requests. It verifies one assignment copy,
comment order, identical text under distinct comment IDs, and full
fallback context. Full file: **33/33 passed**. Both affected TypeScript
checks passed.
- Existing deterministic tests cover source revisions, approval and
trust boundaries, completion validation, custom templates, compatible
sessions, standalone driver wrapping, and maintained adapter transport
requests.
- Provider-free browser support: **17/17 passed** after the master
merge. Service-worker unit tests: **33/33 passed**. The module-overload
regression failed before the repair and passed after it in real
Chromium.

### Fresh live comparisons

The new batch ran exactly four Product E2E attempts. **All four passed
on the first attempt; no retries.** Each has six terminal matchers plus
the existing browser lifecycle and invariant checks.

| Exact case ID | Control | Candidate |
|---|---|---|
| `core-compatibility.runner-codex.local.plan-revise-accept` | Passed |
Passed |
|
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`
| Passed | Passed |

The plan case checks a revised canonical plan and revision-bound
approval before completion. The question case restarts the server before
submitting the answer, then verifies the continuation completes.

Control source is `dfa4e1bda8d50a1a01746603251a9128dbe9d0d6`. Candidate
source is `79fcdb5dece501d28064ea9da306603881b46f0c`. They use identical
frozen definitions and provider versions: Codex `0.156.0` with
`gpt-5.6-sol`; ACPX `0.13.1` / Claude ACP `0.73.0` with
`claude-sonnet-5`. The September 24 head added master browser recovery
and test-only changes. The September 28 head also integrates newer
master changes, including cancellation, workspace finalization, and
native Grok. These are frozen-source live results, not exact-head live
runs.

The candidate received one description copy where the control initially
received three. The submitted initial plan envelopes were 7,969 versus
19,097 characters. Question envelopes were 7,592 versus 18,919. These
are structural measurements, not whole-provider token or dollar savings.

### Earlier evidence and failed attempts

- The preceding fresh batch has four effective passing pairs: OpenCode
comment continuation and assigned skill, native Codex comment
continuation, and native Claude comment continuation. It retains **11
attempts: eight passed and three failed**.
- Original failures remain recorded: missing local PostgreSQL library
links before task creation; host-sleep cleanup after task/page checks
passed; and a Claude **control** session-open rejection before a model
turn. Setup was repaired identically on both worktrees. The permitted
unchanged infrastructure retries passed. The underlying Claude provider
startup error was not retained and remains unknown.
- Older R2 retains **17 passes and one failure** across 18 attempts,
including eight both-pass native/legacy Codex/Claude pairs. Its OpenCode
blank-page failure led to the service-worker repair. R2 is historical
evidence: master changed the native fixed prompt and removed duplicate
wake environment data afterward.
- The September 24 CI run initially failed one unrelated preview
readiness test (`ECONNREFUSED` on its local fixture). Its test and
production code match master. Isolated local verification passed **28
tests, 3 skipped**. One unchanged CI retry passed the full shard: **831
passed, 1 skipped**, including all **31 preview-exposure tests**. The
aggregate CI gate passed afterward. The precise startup cause remains
unknown; a port race is a hypothesis, not a proved cause.

### Limits

The original wider profile/workflow matrix, repeated trials, and remote
Daytona qualification are incomplete. These results support a focused
merge recommendation, not statistical equivalence or universal harness
qualification. Some usage receipts are missing in both variants, so no
token or dollar savings are claimed. The $500 ceiling was preserved
using conservative allowances; failed attempts and unknown charges
remain in the ledger.

Reproduce the focused additions with `pnpm exec vitest run
packages/adapters/claude-local/src/server/acp.test.ts` and `pnpm
--filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts`. Full checks use `pnpm -r
typecheck`, `pnpm test:run`, `pnpm build`, and the separate runner
checks. Paid evals require the frozen definitions, profiles, and
credentials; do not use `--all` as a substitute for the selected cases.

## Risks

- Context placement changes can affect model behavior. Deterministic
checks cover the selected paths, but live qualification remains
incomplete.
- The stale-question guard can reject a request when queued human
comments arrived during the run. This is intended.
- New stored inputs and model envelopes retain compatibility readers for
older active sessions.
- Custom templates may intentionally repeat content.
- Removing a model-facing working-directory sentence does not change
filesystem, command, sandbox, or permission configuration.
- The worker bypass applies only to same-origin development module
paths. Cache-policy tests preserve private-response handling and
production asset caching. Mounted HTTP fixture changes remain test-only.
- This PR does not claim measured token savings or statistical
equivalence across every harness.

## Model Used

OpenAI Codex, exact model gpt-6-astra, with repository tools and code
execution. Bounded supporting work used gpt-5.6-luna and gpt-6-luna. The
serving context-window size is not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the required issue fields
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run the focused local checks and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect these changes
- [x] I have considered and documented risks above
- [x] All current-head Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
for the current head
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-28 14:49:14 -05:00
DottaandPaperclip 96bf004a79 fix: use persisted state for lifecycle continuation and retry budgets (#13888)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its control plane decides when a task can continue, wait, stop, or
complete.
> - Legacy continuation could change when an agent changed its wording
without changing task state.
> - Shared attempt counts also let repair and infrastructure retries
affect each other's limits.
> - This pull request uses persisted state and separate, bounded
allowances for these decisions.
> - If automatic repair stops, the task explains what happened and
offers a guarded retry.
> - Paired tests and real-provider evaluations verify that Stop,
approvals, ownership, and spending limits remain authoritative.

## Linked Issues or Issue Description

Related work: Refs #13761, Refs #11126, Refs #13610. These cover
obsolete continuation dispatch and retry storms. Open and closed issues
and PRs were searched for related lifecycle, continuation, and retry
work.

**What happened?**
Legacy continuation depended on English wording and progress heuristics.
Repair, failure retry, and productive continuation could consume shared
counts. When bounded repair stopped, the task showed a technical
recovery message without a clear next action.

**Expected behavior**
Persisted disposition and owned execution paths determine the next
action. Missing disposition prompts bounded agent repair. Explicit work
mode determines planning mode. Narrative changes and raw activity counts
cannot replenish allowances. An exhausted repair shows a readable
notice. An explicit retry checks current controls and preserves the
assigned agent.

**Steps to reproduce**
Run `pnpm test:lifecycle-baseline`. The paired probes keep structured
state constant while varying completion, planning, blocker, and progress
prose. Run the explicit `lifecycle-baseline` and
`continuation-accounting` Product E2E suites for real-provider coverage.
In Storybook, open **Design previews / Recovery notice** to inspect the
production component's normal, pending, acknowledged, unavailable,
failure, and mobile states.

## What Changed

- Hide the image attachment button, icon, and drop/paste hint in answer
composers. Image paste and drop support remains available.
- Merge current master and retain both browser regression sets. Use a
production-stamped service worker in the offline recovery browser
fixture.
- Share one state-based legacy continuation decision across immediate,
delayed, and recovered dispatch. Bind bounded repairs to their source
run and episode.
- Remove title and description wording from work-mode authority. Agents
can still write requested plans in execution mode.
- Persist separate failure-retry and productive-continuation counters.
Disposition repair and resource waits cannot consume or reset those
allowances.
- Validate delayed repair identity, then recheck current gates before
provider dispatch. Fence native startup cancellation.
- Show **Agent needs attention**, a plain-language explanation, **Retry
agent**, and expandable details in both task interfaces. Report request
progress, acknowledgement, and errors inline.
- Store typed recovery notice metadata. Recognize older active notices
only through exact stored action and run IDs. Notice text never grants
retry authority.
- Use the existing recovery-action endpoint for retry. Recheck current
action, status, owner, agent availability, dependencies, active runs,
pending questions and confirmations, approvals, pause controls, and
budget. Duplicate requests do not wake twice.
- Add component, page, route, database, contract, and Storybook
coverage. Keep the scenario inventory and executable evals here.
Historical reports and snapshots live in the [commit-pinned
paperclip-evals
archive](https://github.com/paperclipai/paperclip-evals/blob/ce3e5afcd4a1184650f586a2b5b8be5874c66c8b/experiments/2026-09-lifecycle-authority/README.md).
- Preserve unsaved project fields while the same project URL changes to
its canonical alias. Do not reuse data across projects or companies.
This separate fix addresses the repeated repository-editor browser
failure without changing the browser test.
- Keep the development service worker from intercepting Vite module
reloads. Update the connection-intent browser fixture to record progress
and completion through the agent API.

## Verification

Merge preparation on September 25, commit
`c1e8e4b7ddd9fbc4913ed55ce21b8e12906c2f97`:

- Merged master `bd2030932` and resolved the browser test-list conflict
by keeping both sets of regressions.
- Deterministic lifecycle baseline: 1,090/1,090 assertions passed; no
failures, skips, or missing selected evidence. Unit 423, runner 184,
database integration 397, grading 86.
- Browser support: 17/17 passed. The offline recovery test first failed
with an unstamped development worker, then passed with the production
stamp. Its assertions are unchanged.
- Focused interaction UI and offline fallback tests: 19/19 passed.
Verified the custom-answer composer in Storybook: no attachment controls
or hint; entering an answer enables Next.
- Recursive typecheck, production build, token gates, and diff checks
passed. The worktree is clean. No new real-provider campaign was run.
- Current CI and review: [Current PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/36166011243):
55 successful checks and two optional Storybook skips. Greptile scored
this exact commit 5/5. Hiding the question attachment controls is an
intentional UI change; paste/drop remains available.

Earlier recovery UI verification, commit
`21be0fec0e90e86b6d662b8ee4831847cd041cdb`:

- Recursive typecheck, production build, token gates, and diff checks
passed.
- Focused UI coverage: 338 tests passed across six suites (336 before
the interaction guard, with the two affected suites rerun at 149 passed
after it). Covers both task interfaces, the real page mutation,
pending/error acknowledgement, stale state, and unavailable controls.
- Recovery database integration: 352 tests passed before the interaction
guard. The complete recovery-action and mutation-route suites passed 181
tests after it. The two new pending question/confirmation regressions
failed before the fix and passed afterward, including
resolved-interaction controls. Shared validator suite: 31 passed. E2E
catalog suites: 34 passed.
- Browser inspection passed for light/dark themes, mobile layout,
expandable details, pending retry, acknowledgement, failure, and
disabled retry. Storybook renders the production component; its request
is simulated.
- The broad local run hit two chat callback-order wait failures and was
stopped after all CI unit/database/runner shards passed. Both local
failures passed when rerun without the competing full-suite process.
- CI exposed a repeated project-repository draft-loss race during
canonical redirects. A new unit regression failed before the fix; all
nine project-page tests now pass, including controls for other projects
and companies. Both unchanged repository browser tests passed against a
fresh local server. UI typecheck, production UI build, and token gates
passed after this fix.
- [Earlier PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/36072486798)
on `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: 55 successful checks, two
optional Storybook skips, and no failed or pending checks. The
repository browser shard passed with the production fix. Greptile is 5/5
on this exact commit with no unresolved review threads. The PR is
mergeable.

Historical, source-qualified lifecycle evidence:

- Lifecycle baseline: 1,074 assertions. Native session coverage: 447
tests. Product E2E support: 515 tests. Browser support: 11 tests. Full
earlier verification is retained in the archive.
- [Real-provider campaign: 8/8 passed, zero
retries](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35881382080-1/index.html),
source `e88d210417280140b44a36449027290adcb1aeaa`. Evidence and cleanup
checks passed. This includes deliberately exhausted repair cases that
correctly remain blocked; it does not mean every task finished Done.
This campaign predates the recovery UI change.
- Archive migration verified all 16 original JSON files byte-for-byte
and all 24 checksum entries. App tests do not need private archive
access. [Archive PR
#27](https://github.com/paperclipai/paperclip-evals/pull/27) is merged.

## Risks

- Agents that omit durable disposition receive at most two repair
attempts by default. Prose-only completion exposes missing state rather
than silently changing scheduling.
- A retry is an explicit board action. The server rechecks current
controls. A successful response confirms the task returned to To do; it
does not claim that the provider has already started.
- Existing notice metadata remains valid. Only older active notices with
matching structured evidence receive the new UI. Historical notices
without that evidence keep their existing rendering. No schema migration
is required.
- Old run records require conservative retry accounting. Tests cover old
counters, alternating retry lanes, restarts, and exhausted repairs.
- Historical snapshots require private `paperclip-evals` access. The app
index retains public campaign links. Live campaigns qualify specific
sources and scenarios; no new real-provider campaign has run for the
recovery UI commit.

> This fixes existing lifecycle and recovery behavior and does not
duplicate planned core work.

## Model Used

OpenAI GPT-6 through Codex assisted implementation, reasoning, code
execution, and review. The exact serving model ID and context window are
not exposed in this task. Historical real-provider evaluations used
Codex model `gpt-5.6-sol`, separately from the implementation assistant.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 15:28:11 -07:00
DottaandPaperclip 43acbcc398 fix(runner): preserve sessions and complete question and approval continuations (#13655)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task state to provider sessions.
> - Follow-up turns must retain provider memory and carry new user
direction.
> - Lost session IDs caused repeated context and extra input tokens.
> - Native question answers and approval races could leave valid work
blocked.
> - This pull request repairs those paths and adds regression coverage.
> - Agents can continue accepted work without repeating the conversation
or losing the user's answer.

## Linked Issues or Issue Description

Refs #13574. That merged PR shortened continuation prompts and moved
question instructions into tool documentation. This change preserves
sessions and fixes failures exposed by broader testing. Related runtime
work: #13408 and #13410.

**What happened?**

Native follow-up turns could lose the provider session ID. Completion
guidance could replace the original task with its latest comment. Claude
native questions could remain pending after the user answered. Approval
during a running tool call could suspend the run before the tool
response arrived. Onboarding and chat handoff instructions also caused
repeated planning or missing plan documents.

**Expected behavior**

Reuse a valid provider session. Send only new events when that session
already has the history. Preserve the task requirements and apply later
user direction. Store the question answer and deliver it to the waiting
run. Finish governed tool responses before suspending. Execute the
accepted plan without asking for the same approval again.

**Steps to reproduce**

Run the continuation, local-session-integrity, first-task, and
agent-chat suites with native Codex and Claude. Include
provider-question-bridge, accept-while-running, and plan-handoff.

**Paperclip version or commit**

This branch is based on master d54b75011. The active full catalog run
tests 4e75881db. Later review fixes have separate regression coverage.

**Deployment mode**

Isolated local instances and Daytona sandboxes in the existing Runner
full-stack E2E harness.

## What Changed

- Retain provider session identity across turns and late usage
snapshots. Send new continuation events on session reuse, with full
context available for a fresh session.
- Preserve task requirements and later direction in completion guidance.
Return the current contract revision after a stale completion
submission.
- Bridge native Claude questions to saved Paperclip cards. Submit
answers through the saved card and resume the same run.
- Delay governed suspension until tool results settle. Add a
deterministic test barrier for approval during an active run.
- Clarify free-text question examples, explicit onboarding plans, and
execution of accepted chat plans.
- Fix continuation readiness, verified output evidence, and declared
screenshot collection.
- Qualify the legacy Claude test CLI at 2.1.277. The old 2.1.19 CLI did
not discover mounted skills. Update the existing workflow pin and
isolated launcher together.
- Refresh the Daytona image lockfile integrity pin after reviewing
master patch updates.
- Carry continuation mode as runtime metadata instead of inferring it
from user-visible text. Install the test Claude CLI without lifecycle
scripts.

## Verification

- Targeted paid verification: 20/20 cases passed across
local-environment campaigns before the rebase.
[Report](https://pages.paperclip.ing/runner-e2e-seven-fixes-35397904249/).
- Harness checks: 379 unit tests passed; harness typecheck passed.
- Latest-head PR checks: 55 passed, two intentionally skipped. Greptile
is 5/5; the security scan passes.
- Review regressions: 350 executor tests and 204 session/driver tests
passed. A script-free Claude install was verified with the actual CLI.
- Full catalog, including the explicit-only everyday suite: [run
35417932353](https://github.com/paperclipai/paperclip/actions/runs/35417932353).
Completed: **164/205 passed; 41 failed**. [Full dashboard and failure
investigation](https://pages.paperclip.ing/runner-e2e-full-catalog-35417932353/).
Includes 204 case artifacts and one pre-case GitHub authorization
timeout; missing evidence is not scored as a pass. The full run tested
`4e75881db`; Final-head metadata/CLI smoke cases both passed. In the
separate [six infrastructure
retries](https://github.com/paperclipai/paperclip/actions/runs/35419769343),
the GitHub timeout case passed and all five Docker preflight failures
repeated. [Follow-up
dashboard](https://pages.paperclip.ing/runner-e2e-full-catalog-35417932353/follow-up/).
- Full local typecheck and build passed on the rebased branch. The full
local unit run completed with 657 passing files, two test timeouts and
one suite setup timeout. All three affected files passed when rerun in
isolation (84 tests). The first full local run was not clean.
- Focused regression coverage includes the live question bridge,
same-run response delivery, UI routing, stale revisions, approval
overlap, and session reuse.

## Full-catalog follow-ups

- Test infrastructure: 14 Claude everyday cells probe an absent host
CLI; six cells failed pre-task GitHub/Docker qualification (GitHub
passes on retry; all five Docker cases repeat; the workflow preflight
allowlist omits their case IDs); five ACPX Codex cells cannot create
sandbox namespaces.
- Runtime: four OpenCode completion-criteria mismatches masked by
shutdown errors, one service-approval suspension failure; three Daytona
recovery failures encounter existing skill files; one duplicate
completion wake.
- Confirmed test defects: question pagination and a noncanonical plan
document key.
- Product/behavior: mismatched visible/required question sets, an
attachment instead of the requested task document, one lone-option
onboarding question, early completion instead of review, and a Codex
Mini completion-schema failure.
- The report job itself fails on trusted master’s stale patch/lock
configuration. The linked report is rebuilt with the shared renderer
from original cell results and public fixture screenshots; it excludes
private snapshots, logs and traces.

These are investigated follow-ups, not silently regraded passes.
First-task passed 51/52. The PR checks are green independently of the
broader catalog’s behavioral/infrastructure failures.

## Risks

- Session reuse depends on a valid provider identity and context
coverage. Fresh-session fallback and reset tests cover this boundary.
- Native question delivery spans saved interaction state and a live
provider run. Tests cover duplicate events, closed runs, and same-run
answers.
- Provider behavior varies. The full paid catalog may expose failures
beyond these targeted fixes; those results will be reported without
relaxing valid approval or output checks.
- The legacy Claude version update is limited to test infrastructure. No
database migration is included.

## Model Used

OpenAI Codex, GPT-6 family, with repository inspection, code execution,
and browser/E2E tools. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks and all
three timeout-file reruns pass; full-run timeout caveat above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-19 07:42:57 -05:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
DottaandPaperclip 250deab910 fix(runner): keep healthy native sessions alive (#13261)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native sessions keep a provider process and its work alive across
control-plane operations.
> - A hidden 15-minute turn deadline stopped work even when the agent
timeout was zero.
> - A one-hour runner lifetime and fixed connection lease added two more
limits.
> - Recovery also rejected goal commands because it reconstructed their
startup summary with the wrong protocol version.
> - This pull request removes implicit duration limits and renews
authenticated leases in the harness.
> - Healthy sessions can continue without model action or a user
interface change.

## Linked Issues or Issue Description

Refs #13092 and #12845. Related: #13163 covers sandbox recovery after
app restarts; this change covers session duration and lease renewal.

**What happened?**

A native Codex session stopped after 15 minutes while a tool was still
running. The agent had `timeoutSec: 0`. Recovery then rejected a
`session.goal.get` startup command with `invalid provider startup
ownership fence`.

**Expected behavior**

An unlimited session keeps working while its provider and authenticated
controller remain healthy. Lease maintenance is transparent. Explicit
timeouts, cancellation and revoked authority still take effect.

**Steps to reproduce**

Start a native session with `timeoutSec: 0` and run a tool beyond 15
minutes. Before this fix, the runtime cancels the turn. A recovery
startup that uses a goal command also exposes the protocol-version
mismatch.

## What Changed

- Honor the agent turn timeout. Zero disables the timer. Long explicit
durations use timer chunks to avoid Node timer overflow.
- Default native runner lifetime to unlimited. Keep bounded startup,
reconnect and control-operation deadlines.
- Renew leases over the authenticated connection. Persist renewal before
the reply. Validate identity, epoch and expiry. Handle duplicate
requests and a lost reply on reconnect.
- Freeze renewal during warm ownership transitions and terminal
handling.
- Validate persisted goal startup commands with protocol v2.
- Add duration, renewal, ownership, recovery and real-process regression
tests. Update runner protocol and recovery docs.
- Add no UI components or controls. Renewal requires no model output or
user action.

## Verification

Current head: `348e369c35c5da8bb8be378f4b35dcf6f40882e7`. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34649767113).

- All 32 checks pass on this head. The two Storybook checks are skipped
as expected. CI includes full build, typecheck, runner verification,
browser suites, server tests and the canary package dry run.
- Greptile reports 5/5 on this head. All review threads are resolved,
and the security scan passes.
- Passed `pnpm -r typecheck` and `pnpm build` locally.
- Passed 219 native-runtime and controller tests, including fake-clock
tests for three weeks of renewal and 30-day explicit timeouts. Six
denial tests confirm that renewal cannot extend expired, revoked or
mismatched authority.
- Passed 337 executor, cancellation and restart-recovery tests, plus 278
Rust runner-core library tests.
- Passed a real runner with a silent fake Codex provider across its
original lease expiry. Runner PID, provider PID, thread and active turn
stayed unchanged. Warm-attach recovery tests also pass.
- Passed all 83 plugin-worker tests and 159 of 161 workspace-runtime
tests locally. The two remaining assertions passed with a canonical
macOS temporary directory, as did the changed runtime fixture. The full
affected server shard passes in CI.
- An unchanged GitHub callback-ordering test failed once in CI, passed
locally in isolation, and passed its one test-shard retry. The final CI
summary is successful.
- The full local `pnpm test:run` sweep was interrupted after dependency
setup failures and load-related timeouts. Identified failing suites
passed in isolated reruns after the dependency repair. The complete test
matrix passed remotely in CI.

## Risks

- Deploy the controller and runner together to enable renewal. Older
peers keep their existing bounded lease behavior.
- Unlimited runtime permits long resource use until completion, explicit
cancellation, configured timeout or loss of valid authority.
- Lease renewal changes authenticated protocol handling. Regression
tests cover stale, revoked and mismatched authority, lost replies and
warm handoff behavior.
- Simulated multi-week tests and a real lease-boundary test do not
constitute a weeks-long production soak.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution
and TypeScript/Rust test tools. The exact backend revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 17:09:43 -05:00
DottaandPaperclip 663c44cb2b fix: continue conversations after confirmed remote runner stop (#13254)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Users can stop a run and send another message on the same task.
> - Remote runners need evidence from their sandbox provider that
execution stopped.
> - Local process checks cannot prove that a remote process exited.
> - This pull request records provider stop receipts and uses them for
conversation admission.
> - New user messages can proceed after confirmed cleanup without
repeating interrupted actions.

## Linked Issues or Issue Description

Refs #13237 and #13239. Related: #13163 covers app-restart recovery;
this change covers an explicit stop followed by a new user message.

**What happened?**

A stopped remote Claude ACP task kept its execution hold after Daytona
cleanup succeeded. Native runners also rejected remote process
identities and retained stale session cleanup gates. A message sent
during cleanup could stay deferred after the sandbox stopped.

**Expected behavior**

After the provider confirms that the old execution stopped, a new user
message starts a fresh turn. Pending user messages must not need another
message to trigger admission. Prior action outcomes remain recorded.

**Steps to reproduce**

1. Start a long-running task in Daytona with a legacy Claude ACP or
native ACP runner.
2. Cancel the run while its tool is active.
3. Send a new message immediately, or after cleanup completes.
4. Observe the execution hold despite the old sandbox having stopped.

**Paperclip version or commit**

Reproduced on master at 7b829efdf6. The
branch is rebased on current master.

## What Changed

- Add optional provider stop receipts to sandbox release and destroy
hooks. Old plugins remain compatible.
- Bind receipts to the company, run, lease, and provider resource.
Failed cleanup cannot supply stop authority.
- Acknowledge legacy remote cancellation after confirmed termination.
- Admit native user continuations using remote receipts instead of host
process checks.
- Retire only the settled cleanup owner matching the stopped company,
run, and provider resource. Isolate cleanup gates between remote
sandboxes, including two sandboxes owned by one run.
- Reconsider user messages deferred during remote cleanup through normal
admission, including successful later cleanup retries.
- Preserve receipts through cleanup retries and inline cleanup after
failed startup.
- Permit provider destruction after a terminal remote checkpoint
failure. Busy ownership still blocks destruction.
- Add regression tests and update execution semantics.
- Give the real preview fixture ten seconds for cold startup. Reuse
release-mode Rust artifacts for the filtered parity checks, avoiding a
duplicate debug test build that exhausted CI disk twice; test filters
and assertions are unchanged.

## Verification

- Recursive typecheck and build passed.
- Targeted server, Daytona plugin, and native runtime tests passed. They
cover missing or mismatched receipts, failed cleanup, local process
protection, exact cleanup ownership, and a message sent during cleanup.
- Live Daytona tests passed for legacy Claude ACP, native per-turn,
native warm, and a newly created native runner using disposable
sandboxes. Each original run was cancelled; its explicit follow-up
completed with no execution hold. The disposable case confirmed a new
sandbox after deletion.
- Native tests used the provider's Opus 5 selector, `opus[1m]`, and the
Linux runner bundle from the sandbox image. Existing checkpoint/sync
finalization warnings remained visible before the native follow-ups
reached committed success. This change does not repair those separate
warnings or guarantee recovery of uncheckpointed files.
- A direct live provider test also passed with the final delete-wait
change: the destroy hook returned its receipt only after Daytona
reported the sandbox destroyed. The final Daytona plugin suite passed
all 153 tests; its build passed.
- After rebasing on master, 284 targeted server/plugin tests and 313
native executor tests passed. The native session runtime suite passed
all 128 tests.
- The review regression passed all 145 tests across the continuation,
environment runtime, and pending-cleanup sweep suites. Recursive
typecheck and build passed again after that fix.
- The security review's exact-resource finding is fixed. Cleanup
completion requires the same provider-resource scope used during session
creation. All 128 native runtime tests, 61 server continuation/cleanup
tests, recursive typecheck, and build passed after this fix. The
two-sandboxes-in-one-run regression proves one receipt cannot retire the
other quarantine.
- Greptile reviewed the final commit at 5/5, the security scan passed,
and no review threads remain unresolved. The final native executor suite
also passed all 313 tests.
- The full local suite reached 10,616 passing tests before stopping on
four failures. All four now pass in focused reruns: the final-code
continuation/teardown suites, the attachment suite, and the native
session test after building its required fake-provider binary. The
original full run overlapped source edits and did not reach the
remaining groups; it is not counted as a full-suite pass.
- The preview exposure suite passed all 25 applicable tests (three
Linux-only tests skipped locally) after the startup allowance change.
Both release-mode Rust parity commands passed locally. All final-head PR
checks passed, including full server/workspace/browser test groups, full
native runner verification, build, typecheck, canary dry-run, and the
aggregate gates.

## Risks

- A provider must return a receipt only after confirmed termination.
Incorrect provider claims could permit overlapping execution.
- Older providers without receipts retain the existing hold. Missing
evidence, failed cleanup, active ownership, pauses, approvals, and
budgets still block admission.
- This change preserves unknown action outcomes and old checkpoints. It
does not authorize replay or alter historical runs.
- No database migration or telemetry contract change is required.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
and browser tools. The exact backend revision and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 15:18:37 -05:00
DottaandPaperclip ca96e1eb0a fix(runner): keep streaming after task completion tools (#13108)
Keep receiving provider events after paperclip_finish, drain pending event persistence, and select the final assistant answer after the provider turn ends. Preserve cancellation, failure, and governed-wait behavior.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-09 15:22:00 -05:00
DottaandPaperclip fac07b42ad fix(runner): preserve durable native session authority across recovery (#13092)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner carries tool results and task output to the
control plane.
> - A lost connection must not change which run owns a result.
> - A session must not become reusable while provider output is still
pending.
> - This pull request adds strict recovery evidence and bounded drain
barriers.
> - It preserves current PRP version negotiation and session-goal
support.
> - The benefit is safer reuse of native sessions after a transport
failure.

## Linked Issues or Issue Description

Refs #13038. This is the first of two stacked pull requests. It contains
the native runtime prerequisites. The second pull request contains the
experimental chat-channel integration. It preserves the provider
identity and typed terminal-failure contracts in #13074 and the durable
recovery work in #13075.

**What happened?**

Native session failures could leave retained provider events, incomplete
tool results, or warm handoff state that was not safe to reuse. A later
run could observe output from an earlier authority.

**Expected behavior**

Recovery must preserve exact run, tool, process, artifact, and lease
evidence. Uncertain or corrupt state must fail closed. A successful
close must prove that retained provider output is settled.

**Steps to reproduce**

Run the transport and control-plane regressions. They hold and drop
authenticated frames, fail durable writes, and restart fresh controllers
and runner processes with retained state. Provider executables are local
test fixtures.

## What Changed

- Preserve pending provider cleanup and semantic-result evidence across
session close and restart.
- Add an authenticated warm handoff with exact old and new identities,
durable receipts, and completion acknowledgement.
- Drain retained provider events under the cumulative acknowledgement
fence.
- Reject corrupt tool-result contracts without unsafe provider replay or
reusable checkpoints.
- Keep ordinary PRP v1 sessions and current session-goal behavior.
Require negotiated PRP v2 and acknowledged native session evidence
before warm authority rotation.
- Preserve late semantic inputs and exact durable result receipts until
close can prove settlement.
- Add transport, crash-window, artifact, checkpoint, and final-output
regressions.
- Deduplicate resolved execution delivery under the current issue lock.
Reuse the exact existing successor after concurrent scans or a lost
acknowledgement. Preserve newer operator evidence.
- Persist idle provider integrity/capacity failures before process
retirement, retain permanent model-rejection classification, and keep
external question identifiers out of task instructions.
- Expose only the context source on native status events. Keep thin
dispatch projections compatible without exposing the complete context.

## Verification

- Review-fix revision: 128 runtime-context/native-session tests, five
idle-failure/adjacent Rust cases, 24 warm crash-window cases, three
startup-notification/close cases, and five attach/backlog cases passed.
The security and idle-failure cases were first reproduced failing.
- Prior merged revision: runner production build, TypeScript typecheck,
complete Rust workspace tests and formatting passed; 272 focused runner
tests and two real PostgreSQL regressions passed.
- Earlier full runner runs and CI Build failed on missing
semantic-result fixture receipts, stale local provider fixture bytes,
startup-notification ordering, and a confirmation-loss fixture that
could accidentally send its final ACK. Each cause was reproduced and
corrected without relaxing production authority or close assertions.
These earlier runs are retained as failures, not represented as passing
verification.
- The first local repository-wide run failed before later phases because
the isolated install omitted PostgreSQL's native-library aliases; it
also encountered an unrelated occupied-port fixture. Those results are
retained, not represented as a passing run.
- Exact `335b2ee52709afb3885d4d6ebb2a3ece4b5864d6`: the complete runner
suite passed 1,888 tests, with 10 existing skips. The full Rust release
workspace passed with serial test scheduling. The unchanged parallel
Rust run hit the five-second 300-descendant fixture deadline; that
failure is retained. No deadline or assertion was relaxed.
- The resolved-execution regression suite passed 57 tests, including
concurrent delivery, lost acknowledgement, superseded authority, and
newer operator evidence. Plain server typecheck passed. The
duplicate-delivery cases were first reproduced failing.
- Prior exact `335b2ee52709afb3885d4d6ebb2a3ece4b5864d6` CI passed all
required jobs and Greptile reported 5/5. Its local general-server run
passed 7,208 tests but failed one responsibility fixture; later phases
did not run. The fixture started the next wake while its bounded handoff
was active. It also used nonexistent comment IDs, which hid the current
stored-message-author identity rule. The updated tests use real message
authors, preserve task ownership, and await exact automatic handoffs. No
production identity policy changed.
- Current head `aa39275a1f300f7d1a0b16cd0885eea567cff6b0` includes
current master and the native context-source projection. The focused
identity/status cohort passed 27 tests and plain server typecheck
passed. Fresh full repository tests, types, build, required CI, and
Greptile review are pending. Final results will be updated before merge.
- This is deterministic local-provider evidence. It is not a claim of
complete live-provider qualification.

## Risks

- This changes authenticated recovery and close ordering. The TypeScript
transport and runner binary must be built from the same revision.
- Failed or incomplete evidence intentionally prevents reuse and can
require a fresh run.
- PRP v1 ordinary/cold sessions remain supported. A v1 connection lease
cannot upgrade in place. A current v2-capable runner held on a v1 lease
was qualified through owned-process retirement/join, fresh bootstrap on
the same old authority, v2 observation/ACK, then warm rotation. Legacy
binary replacement and adopted-owner migration are not qualified by that
test; rollout must not present them as automatic same-lease upgrades.
- This pull request has no database migration or chat-channel
activation. The second pull request keeps the channel feature
experimental.

## Model Used

OpenAI Codex assisted with implementation, tool execution, tests, and
reconciliation. The existing implementation records OpenAI `gpt-6-astra`
assistance. The current environment does not report a context-window
size. No private reasoning traces are included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-09 11:02:59 -05:00
DottaandPaperclip 6681104692 fix(runner): preserve provider identity and terminal failures (#13074)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner translates provider events into durable task execution.
> - Provider notifications can refer to another thread or a descendant.
> - Root validation treated these notifications as fatal, and later
layers could lose the original failure.
> - This pull request classifies event identity and preserves structured
terminal failures.
> - The server can then choose safe recovery without weakening tool
authority.

## Linked Issues or Issue Description

Refs #13038. This change incorporates the shared protocol-integrity and
bounded-cleanup prerequisites from that PR. It does not include the chat
feature. Related failure classification work: #13028.

**What happened?**
An informational provider notification for another thread could
terminate the root session. A failed stream could then become a
missing-result error and lose its cause.

**Expected behavior**
Ignore unrelated informational notices with bounded diagnostics. Reject
invalid authoritative events. Preserve the original failure code and
recovery meaning through cleanup.

**Steps to reproduce**
Run a native Codex task. Deliver a notification for an unrelated thread,
or close the stream after a structured failure. Inspect the root outcome
and recorded failure.

**Paperclip version or commit**
Reproduced before e20010472. This branch includes the current
session-goal contract from that commit.

**Deployment mode**
Built from source. Native runner with Codex.

## What Changed

- Classify root, provider-confirmed descendant, stale, unrelated, and
invalid provider events.
- Keep tool requests bound to their original execution authority.
- Preserve typed failures through transport, session, and durable
control-plane cleanup.
- Keep bounded cleanup failures separate from the primary execution
failure.
- Add compatible shared contracts for continuation context, execution
status, and explicit reconciliation. The dependent PR adds their server
and UI consumers.

## Verification

- Runner TypeScript and ACPX suites: 1,733 passed, 7 skipped. Node
contracts: 38 passed.
- Real provider-process fixtures cover 300 descendant identities across
restart, the 4,096-identity capacity boundary, and rejection of
continuation after terminal acknowledgement and restart.
- Repository build, typecheck, and full `pnpm test:run` passed on the
rebased stack (18,448 tests passed, 49 skipped). The full Rust workspace
passed with `--test-threads=1`; parallel execution exposed an existing
fixture port-reservation race. All latest-head CI checks passed. One
unchanged artifact-document concurrency test failed on the first CI run
and passed on its single rerun.
- Added notification, streaming failure, protocol integrity, cleanup
quarantine, and durable failure tests.

## Risks

Provider event classification must retain the new session-goal behavior
on master. Descendant notifications must never gain root tool authority.
Shared contract fields are additive. This PR does not migrate data or
start replacement provider work.

## Model Used

OpenAI GPT-6 through Codex. The exact deployment ID and context window
were not exposed. Used reasoning, tool use, code execution, and browser
automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-09 08:53:07 -05:00
DottaandPaperclip 2043e0c735 fix: repair runner configuration, macOS execution, and artifact galleries (#13062)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent adapters select a provider, a model, and a runtime.
> - Runner conversion rejected existing Claude agents. The model list
mixed providers.
> - The native Claude runner rejected custom models and could not launch
on macOS.
> - This pull request fixes conversion, model selection, and verified
macOS execution.
> - It also groups configuration fields consistently across adapters and
opens artifact images in the task gallery.
> - Operators can change an agent configuration and run the selected
model on their Mac.

## Linked Issues or Issue Description

**What happened?**

Converting an existing Claude agent to Paperclip Runner failed with a
Codex-only restriction. ACPX Claude showed unrelated models and required
`claude-sonnet-5`. Its native runtime rejected macOS. Configuration
mixed common model settings with process controls. Artifact cards
labeled “Open gallery” navigated to attachment URLs instead of opening
the task gallery.

**Expected behavior**

Conversion keeps agent identity and compatible settings. ACPX Claude
uses the normal Claude catalog and accepts typed model IDs. Codex uses
the native runner. The verified Claude runtime can launch on macOS ARM64
and x64. Common configuration sections place the same fields together
across adapters. Artifact images open in the shared task gallery with
navigation and downloads.

**Steps to reproduce**

1. Open the configuration of an existing Claude agent.
2. Convert it to Paperclip Runner.
3. Select ACPX Claude and a different catalog model or a typed model ID.
4. Save the agent and run a disposable task on macOS.
5. Inspect configuration and advanced run-policy controls across
adapters.

**Paperclip version or commit**

The bugs were reproduced on `165ca56a22adb60e5fda56045442d9c8498116a8`.
This branch was rebased onto `7ed122911`.

**Deployment mode**

Built from source. Local test-drive instance on macOS ARM64 with an
isolated database.

Related work: #11798 addresses unsupported ACP session options in the
existing adapter path. #13048 addresses working-folder preservation.
This change fixes native runner configuration and launch behavior.

## What Changed

- Remove the Codex-only conversion restriction. Preserve agent identity,
instructions, directories, credentials, and compatible model settings.
Reset incompatible sessions while retaining history.
- Show ACPX Claude and native Codex as distinct provider choices. Remove
ACPX Codex from advertised configuration. Normalize legacy
configurations before fresh runs without rewriting historical run
descriptors.
- Select model catalogs and cache entries by provider. Support refresh
and typed model IDs. Pass exact Claude IDs through session creation,
model changes, and recovery.
- Add verified macOS ARM64 and x64 Claude SDK snapshots. Bound
executable allocation and total snapshot size. Preserve package checks,
dependency isolation, process ownership, cancellation, and Linux
descriptor loading.
- Probe local runtime readiness. Report remote platform checks as
incomplete until the remote runner verifies its runtime.
- Surface actual model rejection and allow correction and retry.
- Repair missing ACPX goal-capability helpers exposed by the post-rebase
live test. Persist and restore the optional capability without breaking
session startup.
- Put Agent identity first and intentionally remove the Capabilities
editor, as requested. This is removal of UI editing, not relocation:
preserve existing capability metadata and API compatibility without
adding another editor. Use the themed select for configurable permission
modes, with normal text instead of monospace.
- Put model and provider under Adapter. Give environment variables their
own section. Fold command and arguments under Configuration. Fold
lifecycle, timeout, and interrupt grace under Advanced Run Policy. Hide
single-option permission controls.

- Open image and video artifact cards in the existing task gallery,
including cards in the artifacts panel. Chat attachment images use the
same gallery. Preserve standalone media previews and download links.

## Verification

- Rebased focused UI/API/database suites: 293 tests passed.
- Rebased native runtime and ACPX suites: 242 passed, 7 skipped.
- Repository typecheck, build, and token gates passed for the runner
changes. Gallery follow-up UI typecheck, build, and token gates also
passed.
- Follow-up UI suites passed (86 tests), packaging checks passed (14
tests), and the final focused runtime suites passed (126 passed, 7
skipped).
- Linux container isolation and lifecycle fixtures passed before rebase
(57 passed, 2 skipped). Rust ACPX provider-session tests passed after
rebase (8 tests).
- Browser tests completed actual Claude and native Codex tasks on macOS
ARM64. They covered conversion, catalog refresh, a non-default catalog
model, a typed `haiku` ID, save/reload, cancel, follow-up session
continuity, invalid-model errors, and recovery.
- Final-revision live tests completed a typed Claude task, a follow-up
with the same provider session, and a native Codex task on macOS ARM64.
- Browser tests confirmed the moved interrupt-grace field saves and
survives reload. Cross-adapter tests cover Claude, Codex, Gemini,
process, gateway, and schema forms.
- Full local run: 7,080 passed, 30 skipped, and two timeouts. Both
timeout suites passed on isolated rerun (84 tests); the failures were
the plugin login-worker exit diagnostic and the runner real-server
vertical slice.
- Final follow-up checks: 50 registry tests and 45 snapshot/installation
tests passed (6 platform-specific skips). Oversized executable rejection
is covered before allocation or reading; unsupported-platform tests
invoke the real installation probe.
- Runner head `ddb5101c483a297f74875ab96b3c66035b002d50`: all CI gates
green, including full runner verification, repository build, typecheck,
general/serialized server suites, browser tests, and canary dry run. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34286178670).
- Greptile: 5/5 on that runner head. All four review threads resolved.
Superagent, Socket, and Snyk checks green.
- After snapshot hardening, another real Claude task completed on this
Mac using the rebuilt runtime.

- Gallery follow-up: 148 focused tests passed, covering artifact
selection, shared attachment collections, deduplication, image/video
cards, standalone previews, downloads, and closing. Live browser
verification completed on the settings follow-up: artifact selection,
6-image pagination with wrapping, download action, and closing all
stayed on the same task URL. All checks passed on gallery head
`96136da58ff195bf6ca00b281eb3022ad12d7bd8`: [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34287987536).
Greptile returned 5/5 on that exact head with no unresolved threads.

- Final settings polish: 96 focused tests, UI typecheck/build, and token
gates passed. A real browser walkthrough verified readable permission
options, identity placement, Capabilities removal, and permission
save/reload. Original test-agent permission mode restored. All 31 checks
passed on final head `e46540d6bf32bfb0566dca16b2f4a75ba437618c`: [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34292797886).
Greptile returned 5/5 with no unresolved threads.

## Risks

- Capabilities intentionally has no editable UI field after this change.
Existing values remain readable and API-compatible; removing the field
does not erase stored metadata.

- macOS launch now copies verified package files into private snapshots.
The implementation must retain isolation and clean up snapshots on exit.
- Runtime provider or model changes reset the current session.
Historical runs remain available.
- The macOS x64 SDK executable digest was verified, but a live Intel Mac
run was not available. Linux verification used container fixtures, not a
real Claude task.
- Remote environment tests report a warning when only the platform has
been checked. They do not claim package readiness from the server host.

## Model Used

OpenAI Codex, based on GPT-6. The exact served model identifier and
context-window limit are not exposed in this session. Used reasoning,
repository inspection, code execution, Rust and TypeScript tests, and
browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites and both
timeout suites on rerun; full-run counts above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-08 19:10:09 -05:00
DottaandPaperclip 7ed122911b Add end-to-end session goals to Paperclip Runner
Add capability-aware slash-goal controls, durable provider goal state, PRP v2 negotiation, autonomous goal execution, and safe local session recovery. Integrate with current master, preserve provider session identity, and verify the browser goal/chat/replacement/clear workflow and unsupported-agent rejection.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-08 16:18:47 -05:00
Dotta 0c1e7504c0 fix(runner): persist warm Daytona workspaces across turns (#12904)
## Thinking Path

> - Daytona preserves a stopped sandbox filesystem, but deleting or
replacing a sandbox removes its only remote copy.
> - Warm reuse therefore improves latency but cannot be Paperclip's
durability boundary.
> - The host execution workspace must remain authoritative after every
successful turn, while same-run recovery must avoid overwriting
unexported remote work.
> - Result proposal, workspace export/merge, and terminal completion
need a durable, replayable ordering so a crash never starts a duplicate
provider turn.
> - A paid browser acceptance suite must exercise both legacy Codex and
Runner Codex for three real turns on one continuously warm Daytona
sandbox.

## Linked Issues or Issue Description

Refs #12901.

Runner Codex did not previously export successful Daytona workspace
changes back to the authoritative host workspace. That made warm reuse
depend on Daytona's remote filesystem and left deleted/replacement
sandboxes without a reliable reconstruction path. The existing paid
fixture also lacked a focused three-turn continuity case for both Codex
adapters.

## What Changed

- Persist versioned, atomic native workspace-sync descriptors and
durable seeds in `PAPERCLIP_HOME`, without credentials or a database
migration.
- Classify fresh, warm, replacement, and same-run-recovery workspace
preparation explicitly; ambiguous lease/root/digest evidence fails
closed.
- Finalize native workspace export/merge after semantic result proposal
and before run completion, with idempotent replay that never submits a
second provider turn.
- Surface legacy Codex workspace restoration failures instead of masking
them, while preserving an earlier provider error when both fail.
- Keep healthy reusable Daytona leases warm for legacy and native
adapters, stamp finalized workspace generations, and retain existing
cleanup behavior for per-turn or unhealthy leases.
- Preserve Runner Codex's provider process/session across warm turns,
including bounded post-terminal tail draining and exact authority
rotation.
- Add the exact paid `daytona-warm-continuity` matrix:
  - `legacy-codex × daytona × warm-three-turn`
  - `runner-codex × daytona × warm-three-turn`
- Drive all three turns through the browser, verify ordered file
continuity and stable lease/workspace/runtime identities, capture
per-turn timings, and delete the sandbox immediately after assertions.
- Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no
package script was added.

## Verification

- `pnpm typecheck` — passed, including migration safety (no migration
added)
- Focused server/runner Vitest coverage — 144 passed
- `pnpm test:e2e:runner:unit` — 114 passed
- `pnpm test:e2e:runner:typecheck` — passed
- `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed,
1 helper ignored
- `native-session-executor.test.ts` — 139 passed, including safe
fail-closed cleanup after remote runner identity capture failure
- Paid local browser acceptance, exact post-rebase Linux/amd64 runner
binary:
- Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- [Protected paid GitHub Actions
campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033)
against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2:
- Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
- Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
  - Merge/enforcement, S3 history, and Pages publication jobs passed
- Paid result artifacts were scanned for both provider credentials;
neither secret was present.
- Current PR checks — 31 passed, 1 expected Storybook skip; Greptile
5/5; Superagent security scan passed
- `git diff --check origin/master...HEAD` — passed
- Confirmed no `package.json`, lockfile, migration, or SQL changes.

## Risks

- Workspace synchronization now sits on the terminal-success path, so a
remote export failure deliberately prevents false success. Retryable
state retains its lease/seed; loss of the only unexported remote copy
fails closed.
- Warm provider reuse has strict identity and quiescence checks.
Mismatched or ambiguous evidence blocks reuse rather than risking
concurrent provider work.
- The paid suite incurs Daytona and Codex cost only in the existing
protected scheduled/manual workflow and explicitly destroys its sandbox
after each cell.

## Model Used

OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real
browser E2E execution, Rust/TypeScript test execution, and GitHub
Actions diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked an existing issue or described the issue
in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name contains no internal ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have documented the dedicated suite invocation without adding a
package script
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green on the current revision
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the current revision
- [x] I will address all reviewer comments before requesting merge
2026-09-06 08:16:12 -05:00