mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
98ce93dd4e56182071e8dfd2f9c046f3d819102c
56
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
289b402c26 |
fix(runner): settle a saved governed wait whose parked turn was lost on restore (#20519)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run can ask the board a question and park on that saved (governed) wait > - PR #20494 made a saved wait settle after a control-plane restart for Codex, which reports the parked turn as interrupted > - Claude runs through ACPX. After a kill -9 restart, Runnerd restores the conversation and reports the parked turn as `turn.failed` with `provider_turn_lost_on_restore` > - The governed-wait drain and the saved-replay check rejected every failed terminal as unproven, so the saved question failed and the task needed a human Retry > - Runnerd emits `provider_turn_lost_on_restore` only after it proves that the old provider lifetime ended. That is the same proof as an interrupted terminal > - This pull request accepts that recoverable restart loss as proof for a saved governed wait, and only that failure > - The benefit is that a Claude question survives a control-plane kill the same way a Codex question does ## Linked Issues or Issue Description Refs #20494. Related: #20498 (Claude lost cancel reply; this change does not touch that code). **What happened?** Restart chaos wave w-20261110T094930-s214 (Claude, Q1, PV1 and PV2 with `--provider-fault-restart K1`) failed 2 of 2 Q1 tasks with `R3:retry_needed`. The asking run ended `failed` with `native_provider_terminal_failed`. Its events show the question saved, the K1, recovery with `native_governed_wait_settlement_pending`, and then `turn.failed` with `provider_turn_lost_on_restore` for the parked turn. The thrown `providerCode` was `governed_wait_provider_terminal_unproven`. **What did you expect?** The saved question settles after the restart. The asking run ends `succeeded`. The board answer wakes the agent without a Retry. **Steps to reproduce** Run the harness with `--adapters claude --workloads Q1 --faults PV1,PV2 --provider-fault-restart K1 --provider-fault-delays 3 --provider-fault-restart-after-seconds 1 --concurrency 1 --tasks 4`. The new unit-test variants `lost-on-restore-after-restart` and `lost-on-restore-saved` failed with the same `providerCode` before this change. ## What Changed - `native-session-runtime.ts`: add `governedWaitTerminalProven(event)`. A failed terminal proves the stop only when `isNativeRestartInterruption(error)` is true (code `provider_turn_lost_on_restore`, `recoverable: true`). Other terminals keep the old rule (`providerTerminalObserved !== false`). - The governed-wait accounting drain and the saved-replay check both use that helper. - Tests: two new variants of the #20494 test (a lost terminal after the restart, and a lost terminal already saved before the next recovery). Two new rejection cases: a non-restart failed terminal and a non-recoverable restart loss still fail with `governed_wait_provider_terminal_unproven`. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime`: 265 tests pass. The two new settle variants failed before the fix. - `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` passes. - A live restart chaos wave on this branch (Claude Q1, PV1/PV2 + K1) is linked in a comment below when it finishes. ## Risks - Low risk. The change accepts one more terminal as proof for a saved governed wait. That terminal comes only from Runnerd after it proves the old provider lifetime ended, and the code already trusts it to continue a restart-lost turn (`nativeRestartInterruptedTurnId`). - A failed terminal with any other code, or a non-recoverable restart loss, is still rejected. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
ca472d543a |
fix(runner): settle a saved governed wait whose shutdown drain left sequence gaps (#20494)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run can ask the board a question and park on that saved (governed) wait > - After the question is saved, the runner interrupts the provider and drains its usage and terminal for a bounded time > - The drain keeps only usage, the terminal and cancellation facts of the parked turn. It discards every other numbered provider event > - If the control plane restarts before the drain settles, recovery replays the saved journal. That replay required strictly contiguous `sourceSeq`, so the discarded events made a valid saved wait look like the wrong conversation > - This pull request accepts a gap after the saved trigger only when every event after the gap is a drain-retained fact of the same turn. Identity checks do not change > - The benefit is that the asking run settles its saved wait after a restart instead of recording a failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261110T063253-s208`, tasks t02 (Codex Q1, PV2: provider terminal dropped) and t03 (Codex Q1, PV1: provider terminal held), each with K1 (`kill -9` of the control plane about 3-4 s after the fault). The question, the board answer and the final brief were all correct. But the asking run was recorded `failed: native_session_interrupted`, and the grader failed `T:asking_run_settled`. **Cause** The run log of the asking run shows `failureDetail.message: native_governed_wait_replay_mismatch`. It is thrown at `packages/paperclip-runner/src/native-session-runtime.ts:2344` (master). The persisted runner events of the asking run have `sourceSeq` 33..40 and then 43 (t02), and 41, 42 and then 45 (t03). The two missing numbers are events that the accounting drain (`consumeTurn` → `settleDurableResult`, around line 1206 on master) read and discarded, because they are not usage, a terminal or a cancellation fact. Runs without a restart have the same gap (for example `w-20261110T032350-s201/t01`: 34, then 37). Those runs settle without replay, so the gap was not seen before. With a held or dropped terminal and a kill before the shutdown settlement receipt, recovery must replay. Its strict contiguity check then fails on a gap that the runner made itself. **Expected behavior** The question was saved before the provider was interrupted. Recovery must settle that same saved wait, as #20336 does after verified shutdown. Recovery must not start a new model turn. **Related** Refs #20336 (settle saved handoffs after verified shutdown). Not related in cause or code: #20477, #20482, #20483. ## What Changed - `isGovernedDrainRetainedEvent` (new, in `native-session-runtime.ts`) names the facts that the governed-wait drain persists: usage, the turn terminal, the interrupt acknowledgement, cancelled or expired runtime requests, and a failed or cancelled `run.terminal`, all for the parked turn. The drain now uses it in place of the same inline condition. Drain behavior does not change. - Governed-wait recovery replay accepts a `sourceSeq` gap only when: - the trigger is the first replayed event (a gap before it still fails), - sequence numbers strictly increase, - every event after the first gap is a drain-retained fact of the trigger's turn. - The run, normalized session and source instance identity checks do not change. Any mismatch still fails with `native_governed_wait_replay_mismatch`. - `doc/execution-semantics.md`: the governed-wait text said that shutdown keeps a contiguous journal. It now describes the gaps and the replay rule. Note for the sibling Claude/ACPX `turn.cancel` work: this change touches `settleDurableResult` only in the drain filter line (same condition, shared helper). It does not change cancel, the drain timeout or `settleCancelledTurn`. ## Verification - Red/green: `packages/paperclip-runner/src/native-session-runtime.test.ts` - `settles a saved governed wait whose drain left sequence gaps before a restart (terminal-after-restart)` - `settles a saved governed wait whose drain left sequence gaps before a restart (late-terminal-saved)` The first run uses the real drain. It discards two numbered events, so the history is trigger, acknowledgement, then usage with a gap. The stream then ends without a terminal, as with PV1/PV2 plus K1. On master, recovery rejects with `native_governed_wait_replay_mismatch` (the live error). With this change, recovery settles the saved wait as yielded. It does not call `resolveGovernedWait` again. It starts no second turn. The terminal is appended once. When the terminal arrived late and is already saved, recovery does not cancel the provider again. A further recovery of the settled checkpoint does not submit or settle again. - Fail closed: `keeps rejecting an unproven governed-wait replay with a sequence gap: …` covers a gap before the trigger, a foreign run, a foreign session, a foreign source instance, usage of another turn, a non-accounting event after a gap, and an accounting event followed by a non-accounting event. - `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts`: 261 passed. `npx tsc --noEmit -p .`: clean. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t governed`: 12 passed. - `pnpm verify` returned exit 2 (test file outside resolved graphs). Required CI supplies the broad coverage. - Live restart-chaos wave: see the comment below with the wave id and result. ## Risks - Low. The rule loosens only the sequence check, and only after the saved trigger, and only for facts that the drain itself retains and that carry no work authority. Identity checks stay strict. A non-accounting event after a gap still fails closed. - No change to the normal start path or turn startup (fast-turn rule). - No schema, protocol or env change. The pack `contractDigest` and provider `commandDigest` do not change. No Daytona image rebuild is necessary. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
0e325e8d52 |
fix(runner): settle saved handoffs after verified shutdown (#20336)
## Thinking Path > - Paperclip manages AI agents and their work. > - An agent can save a question or approval handoff before its provider stops. > - A missing provider terminal notification does not prove that the saved work failed. > - Runner already verifies provider exit, event drain, tool settlement and suspension. > - This change uses that proof to settle the same saved handoff. > - Unproven shutdown holds continuation and explains that the work is saved. > - Recovery must never repeat the completed work or allow overlapping execution. ## Linked Issues or Issue Description Refs #20330. That PR extends the Codex terminal drain window. This change handles the separate case where no terminal arrives but the owned shutdown can prove stopped execution. **What happened?** A native run saved a human handoff, accepted cancellation, and later stopped its provider. The missing terminal notification still made the attempt fail and show “Couldn’t finish this attempt.” **Expected behavior** Keep the saved handoff. Accept its yielded result after either an authoritative provider terminal or verified shutdown of the exact execution. If shutdown remains uncertain, retain the handoff and explain the hold without offering a duplicate Retry. **Steps to reproduce** Save a governed wait, acknowledge interruption, and omit the provider terminal. The new regression uses the real Runner binary and a controlled Codex process. The old runtime rejects the handoff instead of asking the existing shutdown owner to settle it. **Deployment mode** Native Runner with Codex. Local verification used macOS arm64 and Node 26.4.0. Hosted sandbox recovery has not been qualified. ## What Changed - Add an optional driver shutdown-settlement operation. Runner returns proof only after its existing stop, drain, tool settlement and suspension barrier passes. - Bind the receipt to the run, normalized session and turn. Persist it with the saved wait and validate it during recovery. - Complete the same yielded handoff after verified shutdown. Retire the stopped session from warm reuse. Do not invent a provider terminal or complete usage. - Retain a named shutdown-pending condition under the existing bounded same-run recovery policy. Preserve actual provider failures and integrity failures. - Show saved-work shutdown guidance in task and inbox presentation. Keep genuine approvals authoritative and disable duplicate Retry. - Document the completion and restart contract. The checkpoint field is additive; older checkpoints retain the terminal-or-shutdown path. ## Verification - Red: the new runtime regression failed against the original implementation because verified shutdown was never requested. Server presentation and error-classification regressions also failed before the fix. - Red: both inbox shutdown states showed generic text before the inbox change. - Green: Runner runtime, shutdown and harness-backend suites pass 301 tests. Tests cover missing terminals, failed stop proof, wrong identity, cancellation, late proof, genuine provider failure, and restart after receipt persistence. Recovery completes once without another model turn. - Green: the native executor and task-presentation suites pass 1,371 tests. Execution projection, inbox and blocker presentation pass 125 tests. Two test-authoring mistakes were corrected: an integrity-error text expectation and an assertion for a nonexistent projection field. Their initial failures remain in local logs. - Runner TypeScript and full server layered type checks pass. Token gates and diff whitespace checks pass. - Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one disposable local workspace, one tool call, and one deliberately dropped terminal notification. The saved file, bound settlement receipt, one accepted yielded result, one provider turn, confirmed provider exit and suspended Runner all passed. The tool handler and persistence port were controlled. This does not claim a full server or hosted journey. - The full browser question-and-answer cell stopped before execution because OPENAI_API_KEY was absent. No model work ran in that attempt. Hosted sandbox recovery and the full browser continuation journey remain unverified. - `pnpm verify` returned exit 2 because the execution-spec change requires broad CI. Required CI will supply that coverage; a full local repository suite was not repeated. - Review correction in `8553da244e`: a failed trailing-event write could hide a provider failure. A new regression reproduced red, then passed after all ordinary drain errors retained the named pending condition. No settlement receipt or result can be accepted after that error. The live missing-terminal check passed again on the corrected runtime. - The real PostgreSQL control-plane regression and three existing governed-wait controls pass. The new case persists one accepted yielded result, a bound shutdown receipt, and the original pending human question without a provider terminal. - First CI revision `7c2cb9b1`: test-program typecheck found a missing `PrpEvent` annotation in the new fixture. The annotation is fixed and the Runner test-program typecheck passes. Runner support also timed out in the unchanged `registered roots remain protected even when equal to genuine public static bytes` case after 30 seconds. One unchanged local diagnostic passed in 3.69 seconds overall. The hosted timeout cause remains unknown; no timeout, fixture, or assertion was weakened. [Original failed support job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133). - Rebased onto the merged #20330 and current recovery changes. The combined branch passes 1,402 executor/presentation tests. Both prompt and 18-second provider terminals still pass. A warm session with no terminal passes after the full 30-second drain and verified stop/suspension. Its initial overly broad assertion rejected a nonterminal turn event; the corrected assertion rejects every terminal type. The rebased CI gate was cancelled before any runner or step started; its cause is unconfirmed and it supplies no validation credit. Final revision `b49ec5a144` starts fresh CI and review. - Final-head CI attempt 1 failed before creating jobs. GitHub reports an internal server error (correlation `2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty runner-selection outputs for the aggregate jobs. One unchanged-commit retry was requested; the initial admission failure remains recorded. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42673159868). - **Current blocker:** final head `b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile review and zero unresolved threads, but required CI is not complete. GitHub reports an active [Actions and Checks API incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt 2 ran only part of the full selected graph. Build, canary, Rust, sandbox providers, admitted browser shards and other completed test jobs passed. The E2E aggregate correctly failed because `policy` and `e2e_shards` had `abandoned` results. Server shard 5/12 received a runner shutdown signal at 18:36:55 UTC and its test step was cancelled; its job still reports in progress. The shutdown cause is not established. The separate review workflow had the same pre-execution internal-error class and passed one bounded retry. No further retry was submitted during the incident. Resume with the missing/cancelled coverage after GitHub recovers; do not count the absent typecheck, Runner or support jobs as passes. [Failed coverage gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236), [interrupted server shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174). ## Risks - A missing terminal can use one additional bounded shutdown interval. Unknown shutdown still blocks continuation. - Provider terminal and usage accounting keep their separate meaning. The receipt proves stopped execution and settled tool outcomes, not model success. - Recovery rejects a foreign receipt, a later turn, an active turn with a settled receipt, and an actual failed provider terminal. - There is no database migration. Older checkpoint shapes remain readable. Drivers without this optional proof retain a named pending state. - This PR does not deploy the fix or repair an existing hosted task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and local code execution. The exact serving model ID and context window are not exposed in this session. Live test inference used gpt-6.1-sol. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
42bc238aa5 |
fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records it as `provider_turn_lost_on_restore` and continues in a same-conversation continuation turn > - #20118 taught the event consumer to skip a stale restart-lost terminal of another turn once a continuation is running. It applied only when the checkpoint already recorded the interruption > - When the checkpoint instead names the lost turn as active (the drain hit right after the first turn was submitted), the consumer starts the continuation on the first lost terminal. If the same lost terminal is then delivered again, it was taken as the run's terminal and failed the run > - This pull request applies the skip to any recovered run once its continuation has started. For a checkpointed interruption it still waits for the startup decision first > - The benefit is that a turn lost on restore always continues, however often its lost terminal is delivered ## Linked Issues or Issue Description Refs #20118 **What happened?** Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master |
||
|
|
b54317d3fb |
fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.
## Linked Issues or Issue Description
**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.
**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.
**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.
**Paperclip version or commit**
Staging revision
|
||
|
|
cedf335c98 |
fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the run records that turn as failed with `provider_turn_lost_on_restore` and continues on a same-conversation continuation turn > - Restart recovery blocks a run as `provider_checkpoint_permanently_failed` when its checkpoint carries a failed run terminal and no pending restart interruption > - While a continuation is running, the checkpoint keeps the lost turn's failed run terminal and names the continuation as the active turn. The pending-interruption check returns nothing once a turn is active > - So a second restart during the continuation read the run as permanently failed, blocked replay, and needed the board > - This pull request treats that failed terminal as the restart loss it records whenever an active turn follows a restart-lost terminal turn > - The benefit is that runs survive repeated restarts during their continuations, including M1 parents and their subtasks ## Linked Issues or Issue Description Refs #20148 **What happened?** Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master |
||
|
|
b43cccbb08 |
fix(runner): keep a restart continuation when a stale earlier continuation is reported lost (#20118)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a restart loses a native run's active turn, the runner reports it as `provider_turn_lost_on_restore` and the controller starts one same-conversation continuation. A later restart that loses the continuation is answered the same way > - With back-to-back restarts, the controller's checkpoint can miss a continuation that the runner already accepted. On the next restore the controller starts a fresh continuation, and the restored runner then reports the unrecorded earlier continuation as lost > - The event consumer treated that stale lost terminal as the run's terminal, so the run failed while its new continuation was already running > - Recovery then refused to replay the failed run, and the task needed a Retry. In an M1 parent/subtask task, the parent's Retry succeeded but the parent stayed `blocked` on that subtask > - This pull request makes the consumer skip a restart-lost terminal for any turn other than the continuation it started > - The benefit is that back-to-back restarts no longer turn a recoverable lost turn into a failed run that needs Retry ## Linked Issues or Issue Description Refs #20107 **What happened?** Restart-chaos K14 wave w-20261108T213600-s9803 (local, an upgrade between builds plus repeated restarts) at |
||
|
|
412d49e48d |
fix(runner): resume an ACPX turn interrupted by a deploy instead of refusing the launch digest (#20112)
## Thinking Path
- The rule for upgrade-safe saved state: work saved by the old build
must resume under the new build.
- The K14 upgrade-mid-run chaos wave (`838e9bf3d2` → `f789f3e2af`) broke
that rule for Claude. The same heartbeat run resumed after the upgrade
restart, and the runner's resume probe (`runner.drain`) failed with
`ACPX durable launch profile digest does not match runner startup`
(`provider_state_incompatible`). t03 and t04's Claude subtask both
needed a board Retry. Codex crossed the upgrade cleanly.
- The ACPX launch profile digest covers the provider-pack authority and
the sidecar artifact hashes, so every deploy changes it.
- #18879 already re-binds a suspended session to the new digest. A
deploy that restarts the runner mid-turn leaves the same run with an
active turn, or with an open session, which is the state a same-build
lost-turn restore leaves behind. Neither was re-bound.
- A same-build restart already recovers both states safely: it reopens
an open session, and recovers an active turn through
`restore_conversation_after_lost_turn`, which is guarded by the
provider-lifetime fence. An upgraded build should take the same path.
## Linked Issues or Issue Description
Refs #18879
**What happened?**
A deploy restarted the runner while a Claude (ACPX) task was mid-run.
When the same run resumed under the new build, restore refused the saved
state with `provider_state_incompatible`. The task stopped and needed a
manual Retry, which started a fresh provider session and dropped the
conversation.
**Steps to reproduce**
1. Start a Claude task on build A and let it reach an active turn.
2. Upgrade to build B, which ships a new provider pack or sidecar build,
with a graceful restart while the turn runs.
3. The same run's resume fails with `ACPX durable launch profile digest
does not match runner startup`.
**Expected behavior**
The resumed run recovers exactly as it would after a same-build restart.
The conversation is restored, or the lost turn is closed through the
normal recovery path, with no Retry.
## What Changed
-
`packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs`:
- `rebinds_settled_session` now also re-binds the saved digest for an
**interrupted turn or open session of the same heartbeat run**
(`rebinds_interrupted_turn`).
- The re-bind requires exactly the same-build recovery preconditions: no
accepted semantic result, no pending runtime request, no unconfirmed
provider exit, and a dormant goal. It also keeps the existing
requirements: no candidate admission, and the same sidecar command and
arguments.
- Recovery then runs unchanged. The provider-lifetime fence still
decides whether the conversation is restored or the lost turn is closed.
- **Relocated installs:** an upgrade can install the same launch files
under a new directory, as an `npx` versioned cache, a new image layout
or a new checkout does. The K14 harness moves between worktrees. The
re-bind now accepts a launch of the same shape: the same executable and
sidecar file names, and identical non-path arguments. It then records
this startup's runner-owned paths in the saved descriptor. A renamed
executable or a changed argument stays strict.
- An active turn or open session saved by a *different* run stays
strict, as before. So do suspended sessions with undelivered events, and
every qualified-release binding (ACPX, agent server and command digest).
- Tests (Rust, `acpx_provider_backend`):
-
`interrupted_turn_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
red on master. An old-digest saved state with an active turn restores,
the digest is re-bound, and identity and command digest are kept.
-
`open_session_of_the_same_run_rebinds_to_the_upgraded_runner_launch_profile`:
restore gets past the saved-state check, to provider start.
- `saved_sessions_rebind_to_an_upgraded_build_installed_at_a_new_path`:
red on master and on the first fix, with the production error. It covers
an interrupted turn of the same run and a settled session attached by
the next run, both moved to a new install directory.
-
`interrupted_turn_after_upgrade_stays_strict_when_lost_turn_recovery_is_unsafe`:
each unsafe guard still fails with `provider_state_incompatible` and
leaves the saved state untouched.
- The existing strict tests for other runs are unchanged and still pass.
- `packages/paperclip-runner/docs/durable-recovery.md`: documents the
extended re-bind.
- `tests/acpx_warm_attachment.rs`: the end-to-end test
`restores_a_turn_lost_to_a_runner_upgrade_under_the_new_launch_profile`,
from the Phase 2 session's parallel fix. It fails on master with the
production error and passes here.
## Verification
- `cargo test -p paperclip-runner-core --lib acpx_provider_backend`: 44
passed.
- The full `paperclip-runner-core` lib suite: 585 passed. The 3 failures
are timing-based `task_memory` kill-monitor tests on macOS; that file is
untouched (last changed 5 days ago). CI runs them on Linux.
- `cargo fmt --check`: clean.
- Live: the first K14 rerun, at
|
||
|
|
dcaada5f59 |
fix(native): carry each warm run's own completion contract (#19874)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm runner (`lifecycleMode: warm`) keeps runnerd and the provider
process alive between runs, so a follow-up message can start fast.
> - Each run has its own completion contract. The revision changes with
each follow-up, and the agent must name that revision when it finishes.
> - On a warm attachment the transport rebuilt the `run.attach` payload
from its constructor options. Those options belong to the first run.
Every later run kept the first run's contract.
> - The OpenCode proxy checks the finish claim locally against that
stale contract. Every warm OpenCode follow-up was refused ("Semantic
result completion contract revision does not match"), and the agent
looped until the run timed out.
> - This pull request carries each run's own contract into the warm
attachment, and lets runnerd adopt it without a cold provider resume.
> - The benefit: warm follow-ups finish correctly, and they keep the
live provider process, which is the reason to be warm.
## Linked Issues or Issue Description
Refs #19852 (startup sub-spans), #19846 (TTFT baseline).
**What existing behavior does this improve?**
Warm cross-run attachment for native Codex and OpenCode agents.
**Current behavior**
In a local measurement wave with `lifecycleMode: warm`, 4 of 4 OpenCode
follow-ups looped for 8+ minutes. Each finish call was refused with
"Semantic result completion contract revision does not match". The saved
`runAttachTemplate` after the attachment still had revision `1`, while
the follow-up turn named revision `2`. Codex hid the defect because its
finish claim goes through the controller's tool path, which uses the
current input.
**Proposed behavior**
The attachment carries the attaching run's contract. Runnerd updates the
contract in place for Codex and OpenCode and keeps the provider process.
The OpenCode proxy receives the current contract with the run identity
on `turn/start`.
**Reason and benefit**
Warm follow-ups are the largest TTFT lever in the baseline. A
correctness-only fix (cold-resume the provider when the contract
changes) would make every warm follow-up relaunch Codex or OpenCode,
because the revision changes on every follow-up.
## What Changed
- `attachRun` carries `completionContract` through `NativeSession`, the
harness backend, the Codex harness session and the runnerd transport.
`executeNativeSession` passes the current input's contract. The
transport uses it instead of its first-run `resumeCompletionContract`.
- Runnerd (`codex_provider.rs`, `attach_run_in_place`): a changed
contract no longer forces a cold resume for `codex` and `opencode`.
Codex app-server never receives the contract, and the OpenCode proxy now
receives it per turn. Other providers keep the cold-resume boundary. A
changed tool set still forces it.
- Runnerd sends `paperclipCompletionContract` with each OpenCode
`turn/start`, next to the existing `paperclipRunId`.
- The OpenCode proxy passes that contract to `attachRun` when the run
changes. `OpenCodeHarnessSession.attachRun` replaces its envelope's
contract and validates it.
## Verification
- `src/live/runnerd-codex-transport-warm-attachment.test.ts` (new case):
real runnerd and fake Codex. After a warm attachment with a new
contract, the saved `runAttachTemplate` has the new revision and the
Codex pid is unchanged.
- `src/cli/opencode-proxy-input.test.ts`: the real proxy bundle serves a
warm second run with revision `proxy-v2`. The second
`paperclip/runResult` is produced and names `proxy-v2`. Without the fix
the finish call is refused.
- Each test fails without its part of the fix: the proxy test without
the proxy change; the transport test without the transport change (stale
saved revision); and, with `PAPERCLIP_ATTACH_TRANSITION_RUNNER` pointed
at a runnerd built without the `codex_provider.rs` change, the Codex pid
changes.
- `cargo test -p paperclip-runner-core --lib warm` (11 tests),
`native-session-runtime.test.ts` (232), the OpenCode driver, backend and
warm-attachment suites, and `tsc --noEmit` for the runner package pass.
## Risks
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Saved `runAttachTemplate` values carry forward,
and the next attachment writes the current contract. A new runnerd with
an old OpenCode proxy behaves as today: the proxy ignores the new
parameter and keeps the stale contract. An old runnerd sends no
parameter, so a new proxy keeps its launch contract.
- In-place adoption widens the warm-retention path for Codex and
OpenCode to changed contracts. The idle, accounting and pending-work
checks still apply first.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
2468b3f907 |
fix(native): tell a restart continuation that earlier processes may be gone (#19804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart loses a turn, the native runtime sends a same-conversation continuation (#19501, #19699, #19771) > - A restart can also kill shell or background processes that earlier turns started > - In three restart storms, the continuation agent could not inspect its killed `bench.sh` and set the task to blocked > - The continuation prompt told the agent not to "blindly rerun an action whose outcome is unknown", so the agent did not re-run the command > - This pull request tells the agent that earlier processes may be gone, and to check the workspace outputs and re-run a command whose expected output is missing > - The benefit is that a task can finish after a restart instead of blocking on a process that no longer exists ## Linked Issues or Issue Description Refs #19501, #19699, #19771 **What happened?** Restart chaos workload B5 (Codex and Claude) failed this way three times: storm 6b t14, s7111 t16 and storm 10 t12. The agent started `bench.sh` in the background. A restart killed it. The restart continuation agent tried to find its tool session and got "process inspection is denied". It then blocked the task. The comment read: "Blocked by the Paperclip runtime restart: the benchmark session is unavailable…". It added that it "cannot verify whether the original run is still active or recover its output, so I have not rerun it". **Expected behavior** The agent knows it is in a resumed session after a restart. It checks the workspace for the command's outputs. If the expected output is missing, it re-runs the command and finishes the task. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts`, `restartContinuation`: the prompt did not say that processes from earlier turns may be gone. It told the agent to inspect each unfinished command's "current state" and not to "blindly rerun an action whose outcome is unknown". With the process gone, the agent could not inspect it, so it blocked. ## What Changed - The restart continuation prompt now says: - This is a resumed session after a Paperclip control-plane restart. - Shell commands and background processes from earlier turns may no longer be running, and their tool sessions may be gone. Do not block only because one cannot be found or inspected. - Check the workspace for the outputs or artifacts the command should have produced. If the expected output is present, use it. If the expected output is missing, re-run the command. - Do not repeat completed work. Before repeating an action with external effects (a push, a message, a deployment), check whether it already took effect. If you cannot tell, report the blocker. - The prompt does not say that re-running is always safe. Process teardown (#19727) has not merged. - `doc/execution-semantics.md`: one additive sentence. ## Verification - `native-session-runtime.test.ts`: the three restart continuation tests (Codex, ACPX, OpenCode) now check the new guidance. They were red before the prompt change and are green after it. - Guard: "uses the admitted session cwd in the task envelope" now checks that fresh turns and ordinary resumed turns do not contain the restart guidance. - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/ src/live/runnerd-restart-continuation.test.ts` in `packages/paperclip-runner`: 603 passed. - `tsc -p tsconfig.json` and `tsc -p tsconfig.test.json` for `packages/paperclip-runner` are clean. - Not run: a live restart-chaos replay with a real model. ## Risks - Low. This changes only the prompt text of the restart continuation. Ordinary turns do not change. - An agent may now re-run a command whose earlier run had partial side effects. The prompt asks it to check external effects first, and to re-run only when the expected output is missing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ef83d566ca |
fix(native): ask for the result once after a restart continuation ends without one (#19771)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn that a control-plane restart lost, in the same conversation (#19501, #19699) > - In restart storms 5, 6b and 8, ACPX Claude runs crossed restarts, ran the continuation, and then failed with `native_session_retry_exhausted` > - The continuation completed, but Claude ended it with "I'll wait for the notification when it completes" for a background command, and did not call `paperclip_finish` > - The next same-run attempt then sent no turn at all and failed with "session returned no semantic result" > - This pull request makes the continuation ask for the result, and asks once more in the same attempt if the continuation still ends without one > - The benefit is that a restart-interrupted task settles without a Retry, and the restarts that the run survived cannot block the request ## Linked Issues or Issue Description Refs #19501, #19699, #19720 **What happened?** Storm 8, run 84c532f7 and its Retry 0faead66 (ACPX Claude, task CHA-5). The steps are the same in both runs: 1. `resume_dead_runner`, then `session.resumed`, then `turn.failed` with `provider_turn_lost_on_restore` (recoverable). 2. The restart continuation runs. Claude starts the bench in the background and ends the turn with "I'll wait for the notification when it completes". The turn completes with no `paperclip_finish`. 3. Lifecycle: "provider turn completed without a semantic result; same-run disposition recovery persisted" (attempt 2). 4. 30 s later, attempt 3 restores the session. It sends no turn. After about 2 s it fails with `native_finalization_missing: session returned no semantic result`, and the attempt count makes it `native_session_retry_exhausted`. **Expected behavior** The continuation tells the agent to finish and report its result. If the continuation still completes without a result, Paperclip asks for the disposition one time. Restarts that the run already survived do not block that request. An agent that never reports a result still fails closed. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-harness-session.ts` `startTurn` (lines 224 and 248): the restart continuation uses the driver's one-shot disposition marker (`dispositionOnlyRecoveryAvailable`). #19699 uses the marker on purpose, so that a crash can adopt the accepted continuation turn. The marker then stays consumed with the continuation turn id. - `packages/paperclip-runner/src/drivers/codex/codex-session-terminal.ts` (line 226): only a *lost* continuation gives the marker back. A *completed* continuation with no result does not. - `packages/paperclip-runner/src/native-session-runtime.ts` (`dispositionRecoveryWasSubmitted`, about line 2761): on the next attempt, the consumed marker looks like a disposition request that already ran. Recovery replays the continuation's terminal and starts no turn. So the disposition request never happens. - `server/src/services/native-runtime/native-session-executor.ts` (lines 9755-9761 and `nativeSessionFailureDisposition` at 7768): the `semantic_result_missing` same-run retry charges a provider attempt on the same counter as restart resumes. In this run, attempt 3 was the last allowed attempt. - The continuation prompt (`restartContinuation`) did not tell the agent to report a result, or not to stop and wait for a background notification. ## What Changed - Codex-protocol driver (used by Codex and by ACPX through runnerd): when a restart continuation completes without a semantic result, the driver gives back the one-shot disposition marker. The disposition request is not a continuation, so it consumes the marker for good. - Native session runtime: after that, the same attempt sends one disposition request (`Paperclip semantic-result recovery after a restart continuation ... invoke exactly one paperclip_finish or paperclip_block`). It does this only when the driver shows that the continuation borrowed the marker and gave it back. Other drivers keep the current behavior. - Restart continuation prompt: it now says to finish in this turn, not to end the turn to wait for a background command or a notification, and to report one `paperclip_finish` or `paperclip_block`. - `doc/execution-semantics.md`: one additive paragraph. - No server change. The disposition request runs in the same attempt, so it uses no provider attempt. This avoids `native-restart-recovery.ts` (#19747). ## Verification Red before the fix, green after: - `codex-app-server-driver.recovery.test.ts` "keeps one disposition request after a restart continuation completes without a result". Red: marker stays consumed on `turn-2`. Green: the marker is given back, one disposition turn starts, and a resultless disposition turn does not re-arm. - `native-session-runtime.test.ts` "settles a restart continuation that ended without a result through one disposition turn (Codex driver)". Real Codex-protocol driver with a fake transport. Red: only one `turn/start`, then `native_finalization_missing`. Green: the disposition turn `turn-3` reports the result and the run completes. - `native-session-runtime.test.ts` "asks once for the disposition when a restart continuation ends without a result" and "fails closed when the driver keeps the continuation's disposition marker". Also run: - `npx vitest run src/native-session-runtime.test.ts src/drivers/codex/` in `packages/paperclip-runner`: 593 passed. - `src/live/runnerd-restart-continuation.test.ts` (real runnerd and fake Codex, killed runner): 5 passed. - `tsc --noEmit` for `packages/paperclip-runner`. - `pnpm verify` exits 2 because the doc change is a global CI input. Required CI supplies the broad coverage. Not run: a live replay of storm 8 t05 with Claude. The chaos harness needs provider keys from the operator's shell. ## Risks - Low to medium. A run whose continuation ends without a result now gets one more provider turn in the same attempt. The bound is one disposition request per continuation. Each continuation still needs an admitted restart, and the provider-attempt budget bounds those. - If the controller crashes after the continuation completes but before its checkpoint, the checkpoint keeps the stricter consumed marker. That case fails closed as before. - The prompt text changes for restart continuations. It does not change a protocol schema or a profile digest. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
fc7e3a6503 |
fix(recovery): never replace a native session after a deterministic refusal (#19733)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs keep a saved provider session, and same-run
recovery resumes it after a failure
> - When resume fails and the checkpoint allows it, recovery archives
the saved continuity state and opens a fresh replacement session
> - Recovery used that path for every resume failure, including
deterministic refusals that left the saved state unchanged and repeat on
every retry
> - In five staging tasks a refusal (a protected launch argument, or a
predecessor tool call with no result) escalated into a replacement. The
remote archive move failed part way, the next attempt found no runner
state, and each task was blocked for workspace repair
> - This pull request stops before replacement after a closed-class
refusal, keeps the saved state, and hands the run to an operator with a
named reason
> - The benefit is that a deterministic refusal can no longer damage
saved state or open a replacement over an unsettled external action
## Linked Issues or Issue Description
Refs #19659, Refs #19516, Refs #19334
**What happened?**
Five Codex tasks on Daytona sandboxes failed on staging. The first
attempt failed with a deterministic refusal:
- `run.attach cannot change protected launch arguments` (four tasks,
fixed separately in #19659), or
- `native_runner_source_semantic_reconciliation_required` with a
predecessor `paperclip_call_assigned_tool` call still pending (one
task).
The same-run retry then opened a replacement session.
`archiveContinuityState` failed with
`runner_continuity_break_archive_failed` and no detail. The third
attempt failed with `runner_remote_state_unavailable: exit=1`, and the
task was blocked with "Workspace repair required".
**Steps to reproduce**
1. Save a native session whose checkpoint policy is
`allow_replacement_after_resume_failure`.
2. Make the resume fail with a refusal that does not change state (an
attach refusal, or a pending tool call).
3. Same-run recovery opens a replacement session and archives the saved
continuity state.
**Expected behavior**
A deterministic refusal stops the run for an operator. The saved state
stays intact. A run with a pending side-effecting call never reaches
replacement. Transport drops keep the existing retry behavior.
**Root cause**
`executeNativeSession` checked only the checkpoint's replacement policy.
The closed failure class (#19334) only had an effect when replacement
was forbidden. Attach refusals and pending tool calls had no class at
all, and the source-reconciliation refusal was a plain `Error`.
## What Changed
- Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach`
refusals that would change the durable Codex profile or a protected
launch argument carry the closed classification
`provider_attach_refused`. An attach blocked by
`durable_pending_tool_calls` carries `provider_side_effect_pending`.
These refusals already leave the saved state unchanged.
- Contracts (`harness-driver.ts`): new closed classes
`PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed
`NativeRunnerSourceReconciliationRequiredError` replaces the plain
`Error` at the five `native_runner_source_*_reconciliation_required`
throw sites. `runnerRecoveryFailureClass` reads the authenticated
command-result classification or the typed error, never message text.
- Runtime (`native-session-runtime.ts`): after either class, recovery
never opens a replacement session, whatever the checkpoint policy.
- An attach refusal ends with the existing
`NativeProviderStateIncompatibleError`, which maps to the existing
operator path `native_provider_state_incompatible`.
- A pending side effect ends with the new
`NativeProviderSideEffectPendingError`.
- Server (`native-session-executor.ts`, `native-session-resume.ts`,
shared `EXECUTION_RECONCILIATION_CAUSES`):
`native_provider_side_effect_pending` is a permanent failure and an
execution-reconciliation cause, so the board gets a reconciliation
(record whether the action ran) instead of a generic Retry. The
reconciled continuation does not rebind the saved session, which still
refuses attachment. It starts a fresh provider session in the same
workspace. The saved session and workspace are unchanged.
- Archive (`archiveContinuityState`): a failed remote move now records
the bounded (512 characters), redacted exit status and output in
`runner_continuity_break_archive_failed: exit=… timedOut=… <output>`.
The remote move already ran before the local move on `master`, so a
failed remote move leaves the local state in place.
`restart-interrupted-turn.ts` accepts the detailed message for its
reattachment-loss retry.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe the new
rule.
## Verification
Red before the fix, green after (each test first run against unchanged
source from `origin/master`
|
||
|
|
ece09c156a |
fix(runner): keep an accepted result when ACPX reports its turn lost on restore (#19691)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex, OpenCode, and ACPX runs survive control-plane restarts: runnerd restores the conversation, and the control plane continues the task from its checkpoint > - #19658 made recovery keep a durable result when a Codex restore loses the turn that proposed it. It covered the recovery read path only > - ACPX reports the lost turn differently. After the restored conversation reopens, a live `turn.failed` arrives with `provider_turn_lost_on_restore` > - That live terminal still turned an accepted `done` result into a failed run. The run ended `failed` (`adapter_failed`), and the finished work needed a Retry > - This pull request moves the rule into the shared terminal mapping. A restore-lost turn with a held result for that exact turn settles as completed, whichever path reports it > - The benefit is that ACPX and OpenCode tasks that already finished keep their accepted result after a restart, as Codex tasks now do ## Linked Issues or Issue Description Refs #19658, #19525, #19501, #19539 **What happened?** In a restart chaos wave, an ACPX Claude task (workload A1) wrote its brief and proposed a `done` result (`run.result.proposed`, then a completed `paperclip_finish` tool call). Then a K1 and a K6 crash loop hit the control plane, and `resume_dead_runner` ran at controller generations 2 and 3. The restored ACPX conversation emitted `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`) for the checkpointed turn. The control plane then emitted `run.result.accepted` with the `done` result, but `run.terminal` came back `runTerminalState: failed` / `turnTerminalState: failed` / `reportedWorkDisposition: done`. The run ended `failed` / `adapter_failed`, "Native session failed". **Steps to reproduce** 1. Start a native ACPX (or OpenCode) task through runnerd. 2. Let the agent call `paperclip_finish`, and let the control plane persist the proposal. 3. Kill the control plane and runner before the turn ends. 4. Let restart recovery resume the run. The restored runner reports the turn lost on restore as a live terminal. **Expected behavior** Recovery finalizes the accepted result, and the run succeeds. **Actual behavior** ACPX and OpenCode use the same Codex harness session as Codex (`codex-native-backend.ts`). Recovery restored the held result for the turn from the durable proposal. But the live `turn/completed` notification with `status: failed` and `provider_turn_lost_on_restore` went through `mapTerminalTurn`, which emitted `turn.failed`. The backend maps `turn.failed` to `runTerminalState: failed`. #19658 handled only the `thread/read` reconciliation path. ## What Changed - `codex-session-terminal.ts`: a new `settleRestoreLostTurnWithHeldResult` helper. When a terminal is `failed` with `provider_turn_lost_on_restore` and the session holds the semantic result that this exact turn proposed, the turn settles as `completed`. `mapTerminalTurn` applies it first, so live notifications and recovery reads follow one rule. - `codex-session-terminal.ts`: the helper also covers `acpx_active_turn_recovery_closed`. Runnerd's ACPX backend (`acpx_provider_backend.rs`, `restore_session_if_needed`) emits that code instead of restoring the conversation when its own state already recorded the accepted result. It is the same restart loss, with the result already accepted. #19525 already keeps accepted results when close cannot prove suspension, and session close still handles `provider_exit_unconfirmed`. - `codex-harness-session.ts`: `reconcile()` keeps only the missing-turn case from #19658. The restore-lost read case and the prior-terminal check now use the shared helper. - `doc/execution-semantics.md`: the restart paragraph now states that the rule covers Codex, OpenCode, and ACPX, and both report paths. ## Verification - Red before the fix, with the real Codex native backend (`createCodexNativeSessionBackend`) and a fake transport through `executeNativeSession`: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts -t "journaled completion proposal"`. The new `restore-lost live` scenario failed. Green after the fix: the run completes with the journaled result on `turn-1` and a succeeded terminal, and no `turn/start` is sent. The `restore-lost read` scenario passes on both (#19658 already covered it). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "live restore-lost"`. With a held result, the live lost terminal was `turn.failed` before the fix and is `turn.completed` after it. With no held result, it stays `turn.failed` with the lost cause, so the #19501 continuation is unchanged. - The driver case was extended to `acpx_active_turn_recovery_closed` (red with a held result before the extension, then green). It was found in a live replay where the run ended `failed`/`adapter_failed` with an accepted `done` result. - Suites: `src/drivers/codex/` and `src/native-session-runtime.test.ts` (580 tests), plus `src/backends/` and `src/live/runnerd-restart-continuation.test.ts`, pass. `typecheck:typescript` passes. - Live chaos replays (ACPX Claude): results and a separate new signature are in a comment below. ## Risks - The rule needs a held result whose turn id equals the lost turn. Recovery restores that result only from the durable `run.result.proposed` for the checkpointed turn, so a result for another turn does not settle it. - A restore-lost turn without a held result keeps its failure and the bounded same-conversation continuation. - Real provider failures and other terminal states are unchanged. - No schema, profile, pack contract, or database changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a894b11a2b |
fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run's close must prove that its runner durably suspended
before the session's next run may start. If it cannot, the runtime
quarantines the session (`NativeSessionCloseUnrecoverableError`)
> - In a restart chaos storm, an ACPX Claude run succeeded and its
normal close quarantined its own session. The follow-up run on that
session failed 2 s later with `native_session_cleanup_quarantined`, and
no control-plane restart came between the two runs
> - The close sent `runner.suspend` once, the ACPX sidecar rejected it
right after the turn completed, and the barrier gave up at the first
rejected command. No log line showed the quarantine or its cause
> - This pull request retries a rejected suspension within the close
deadline, and logs every cleanup quarantine with its cause when it is
created
> - The benefit is that a successful run no longer blocks its own
follow-up because of a one-time suspension rejection, and any remaining
quarantine can be traced
## Linked Issues or Issue Description
**What happened?**
Run A (ACPX Claude) succeeded. The run log of its close shows:
- `runner did not prove durable suspension before checkpoint`
- `native_session_settlement_incomplete
{"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}`
The runner journal shows `runner.drain` completed, then `runner.suspend`
failed with `failed to suspend ACPX provider: ACPX sidecar command
session.suspend was rejected (retryable=false,
classification=unclassified)`. The follow-up run B on the same session
was created 200 ms after A finished and failed 2 s later at admission
(`retryQuarantinedSessionCleanups`, operator recovery branch). The
server log had no line for the quarantine. One Retry completed the task.
**Steps to reproduce**
1. Run an ACPX task whose sidecar rejects the first `session.suspend`
after the turn completes.
2. Start a follow-up run on the same session.
**Expected behavior**
The close asks again, proves suspension, and the follow-up runs. If a
quarantine is still needed, it is logged with its cause.
**Actual behavior**
The close fails on the first rejection, quarantines the session, and the
follow-up fails with no logged cause.
## What Changed
- `runner/.../durable/runner.rs` (`process_command`): a failed
`runner.suspend`, on first execution or replay, now keeps the runner in
`Continue` instead of `Suspend`. Before, the runner recorded `suspended`
over a provider whose session was still open, then exited. The RC-34
journal shows exactly that: runner lifecycle `suspended`, ACPX provider
`session_open`. A failed `runner.shutdown` still stops the runner.
- `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend`
now lifts the connection's terminal-lifecycle fence, so the next pending
command (the fresh suspend) is delivered on the same connection.
Completed suspensions and every shutdown keep the fence. New test:
"delivers a fresh suspension after a rejected one on the same
connection". Before the fix, the retry was never sent and the test timed
out waiting for the frame.
- `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a
rejected `runner.suspend`, the barrier sends a fresh suspend command, up
to 3 attempts, 250 ms apart, inside the same close deadline. An exited
runner or a spent budget still fails closed. A rejection changes no
state: the runner returns before it records suspension.
- `native-session-runtime.ts`: new
`onNativeSessionCleanupQuarantined(listener)`. It is called when a
cleanup quarantine is created or escalates to operator recovery, with
company, run, issue, session, `operatorRecoveryRequired`, a bounded
cause (error name, code, message) and bounded settlement facts.
- `server`: `logNativeCleanupQuarantines` logs each quarantine as a
warning. The subscription is process-wide and installed once, so closes
that fail during shutdown, after HTTP closes, are still logged.
## Verification
- Red before the change: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected
suspension"`.
- "retries a rejected suspension…" got `false`.
- "stops retrying…" saw 1 attempt instead of 3.
- Green after the change: both pass.
- Rust: new test
`rejected_suspend_keeps_the_runner_serving_its_controller` covers a
failed suspend, its replay, and an accepted retry with a fresh command.
`failed_lifecycle_commands_replay_their_terminal_transition` now expects
`Continue` for suspend and `Shutdown` for shutdown. The `pnpm run
check:runner` unit suites pass (559). `acpx_park_readiness` has two
timing tests (`< 500ms`) that fail at this host load on unchanged master
as well.
- New runtime test: "announces every cleanup quarantine with its cause
when it is created" (operator-required quarantine, cause and settlement
reported once). `src/native-session-runtime.test.ts` passes (212).
- The suspension cases in `runnerd-codex-transport.test.ts` pass, except
"proves local suspension after an event backlog… (1024 suffix deltas)".
That case also times out on unchanged master at this host load (27–35);
I confirmed this by stashing the change and rerunning.
- Server `tsc --noEmit` passes after the runner package build.
## Risks
- A provider that keeps rejecting suspension now gets three tries
instead of one, within the existing close deadline. The final outcome
and quarantine are unchanged.
- The cause of the ACPX sidecar's first rejection is not proven. The
logged evidence fits the sidecar's "not at a safe suspension point"
guard (`turnId`, tools or inputs still set) racing the just-completed
turn. This change makes close tolerate it. It does not change the
sidecar.
- Log volume: one warning per quarantine.
- The Rust change ships in the runner binary. Hosted images need a
rebuild to pick it up.
- After a failed suspend, the runner stays alive with its provider. If
every retry fails, close still treats suspension as unproven and kills
the owned runner process, exactly as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19609.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
785ead5c7a |
fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
19c2bebc9c |
fix(runner): settle durable provider exits on open streams (#19657)
## Thinking Path > - Paperclip manages AI agents and their work. > - Runner keeps the native session and its event stream until a turn ends. > - A provider can exit with a terminal error while its event stream stays open. > - Waiting for that stream to close leaves the run and its resource owner active. > - The existing durable failure receipt already identifies the failed current turn, including an authoritative replay. > - This pull request uses that event to enter the existing failure and cleanup path. > - Recovery and later idle retirement can then proceed from a real terminal result. ## Linked Issues or Issue Description Refs #18988 for native resource retirement. This is a separate terminal-event defect found during failed-turn recovery qualification. A search of open and closed PRs found no duplicate `provider_exited` fix. The related merged #12596 changes Rust test polling; it does not change this TypeScript session owner. **What happened?** `executeNativeSession` records a nonrecoverable `provider_exited` event, but can continue to wait on an open event iterator. A proposed result does not authorize success, and stream EOF may never arrive. The run cannot reach its existing failure and resource cleanup path. **Expected behavior** Settle the committed or authoritative duplicate terminal failure for this exact run, session and provider. Close through the existing owner. Preserve recoverable goals, completed-turn and identity guards. A duplicate must not add another committed event count. **Steps to reproduce** 1. Supply a native session whose iterator emits a current, nonrecoverable, unexpected `provider_exited` failure. 2. Keep that iterator open until the session owner closes it. 3. Observe that the old implementation does not settle within the existing test bound. 4. Apply the guard below. The same execution fails with the typed terminal error and calls the existing close owner once. **Paperclip version or commit** The original regression was reproduced against master source `05a19768ae4d337a4ae88ecdbc259f251406487c`. Current candidate `f4bb8f1b68e2793062dd640ee56b87a9208b4fd9` normally merges fetched master `07a548073ab48f13cb93b2fdc31553a5da88a070`. All unrelated master paths are preserved exactly. The two-file PR diff contains only the guarded settlement and its tests. ## What Changed - Add a guard after the existing durable event receipt. It requires a committed or duplicate receipt, a nonrecoverable unexpected exit, no completed current or previous turn, and exact provider/run/session identity. - Throw the existing typed provider failure so the existing failure and close owners run. Add no scheduler, polling loop, provider parser or cleanup owner. Event counting remains committed-only. - Cover open streams with and without a result proposal, authoritative duplicate replay, all six safety predicates, and a recoverable goal that later completes. Preserve accepted-result and close-once assertions. ## Verification - The original open-stream regression fails before the production change. Review then found that the duplicate test released EOF before asserting settlement. The corrected duplicate control fails on the prior production code at its original timely-settlement assertion; it passes after accepting the authoritative duplicate. All 21 selected exit, safety, goal and compaction controls pass. - The normal durable transport validates authenticated current run/session/turn/item identity and canonical replay equality before delivery. The App port validates current authority and binding; its durable event transaction returns duplicate only for matching source identity and canonical payload. No absent or conflicting event authorizes settlement. The six negative consumer tests use a deliberately permissive mock port; they prove this consumer guard, not that production accepts foreign events. - The first remote Typecheck job found five errors in new mocks that accessed PRP fields without narrowing a native-or-PRP event. Both mocks now explicitly reject the wrong event shape. Complete original CI evidence is retained. No cast, suppression or narrowed test compiler program is used. - On current integrated candidate `f4bb8f1`, normal Runner emission and the full normal `tsconfig.test.json` source/test program pass. All 26 selected cases pass: the same 21 controls plus all five structured provider-failure/model-retry rows affected by the incoming master change. The other 197 cases were filtered, not counted as passing. - Before master integration, all 411 declaration paths and bytes were identical across the settlement change. Incoming master independently adds one OpenCode declaration and changes three OpenCode declarations; those deltas are recorded separately. This fix changes no runtime declaration. No Rust source or binary is changed by this PR. - Actual guarded workers recorded zero PG sockets, active listeners or blocked attempts. All known worker processes/groups closed. The working tree is clean after qualification. No install, Rust build, App run, provider connection or PostgreSQL suite was part of the final freshness loop. - Fresh exact-head remote CI and complete bot reviews remain required. A local whole-repository test/build, successful task retry or live-provider acceptance is not claimed. Run the selected controls from a normal dependency installation: ```sh pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts --testNamePattern='settles committed or duplicate provider exits while their stream stays open|retains EOF settlement when provider exit fails the|preserves a recoverable provider-exit goal until its later authoritative completion|preserves a durable session failure when its stream closes|preserves session-goal failure priority|context compaction|never continues task work when a restart loses a compaction turn|preserves structured provider failure and model retry classification' --maxWorkers=1 --no-file-parallelism ``` ## Risks - A false terminal classification could stop useful work. The guard requires the existing durable receipt and explicit current-turn failure fields; recoverable and completed-turn controls remain covered. An exact durable duplicate may settle the same failure without repeating accounting. - Close can still fail. This change reuses the existing close owner and does not manufacture resource retirement or clear retained failure debt. - This proves Runner failure settlement under a controlled session boundary. It does not establish successful task retry, whole-tenant quiet, live provider closure, or deployed suspension. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated fix and qualification. OpenAI Codex, based on GPT-6, performed managing technical review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the selected tests locally and they pass - [x] I have added tests for the changed behavior - [x] I have reviewed documentation; no command or public contract changes require a documentation edit - [x] I have considered and documented the risks above - [ ] All Paperclip CI gates are green — fresh exact-head checks pending - [ ] Greptile is 5/5 with no open findings — fresh exact-head review pending - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
7941f6b051 |
fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model
## Linked Issues or Issue Description
**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".
**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.
**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".
## What Changed
- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.
## Verification
- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.
## Risks
- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
d4d3383c97 |
fix(runner): scope cleanup quarantine to the session's task and conversation (#19532)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Codex, ACPX) execute through the runner's
`executeNativeSession`, which gates each new session behind cleanup of
earlier sessions
> - When a session close fails in a way that needs operator recovery
(`NativeSessionCloseUnrecoverableError`), the runner keeps a quarantine
entry for it
> - The admission gate keyed that quarantine by `[companyId, backend
kind, backend name]`, so one bad close blocked every later run of the
same backend in the whole company
> - In a restart chaos wave (two hard kills about 20 s apart), one Codex
session's close became unrecoverable. After that, every Codex run and
every Retry in the company failed in about 1 s with
`native_session_cleanup_quarantined`
> - This pull request scopes the gate to the resources that a retained
owner can still hold: its task (issue) and its provider conversation
(normalized session)
> - The benefit is that one failed close no longer stops unrelated
tasks, while the task and conversation that are really at risk stay
protected
## Linked Issues or Issue Description
**What happened?**
After two hard control-plane restarts about 20 s apart, one Codex
session close became unrecoverable. From then on, every Codex execution
in the company failed at admission with
`native_session_cleanup_quarantined: prior session cleanup requires
operator recovery…` (`retryQuarantinedSessionCleanups`). Board Retries
for four unrelated tasks were accepted and then failed at once.
**Steps to reproduce**
1. Run two native Codex tasks in one company (local runner).
2. Make one session's close throw `NativeSessionCloseUnrecoverableError`
(in the wave: a protocol failure after a second hard kill, then an
unsettled runner suspension).
3. Start or retry a different Codex task in the same company.
**Expected behavior**
Only work that shares a resource with the retained owner waits. Other
tasks run normally.
**Actual behavior**
Every later Codex run in the company fails within about 1 s, until the
server process restarts.
## What Changed
- `packages/paperclip-runner/src/native-session-runtime.ts`: the cleanup
domain is now `{ key, resources }`. `key` is the old backend key
(company, backend kind and name, optional remote cleanup scope).
`resources` are the session's issue and its normalized session (or its
run when it has no normalized session).
- The admission gate (`retryQuarantinedSessionCleanups`) and its
retained close owners block a new session only when the backend key
matches and at least one resource matches.
- The exact retirement paths (`completeRetainedNativeSessionCleanup`,
`completeTerminatedRemoteNativeSessionCleanup`,
`completeTerminatedLocalNativeSessionCleanup`) read the unchanged
backend key, so their behavior does not change.
- New regression test: an unrecoverable close on session A does not
block session B of another task. A session with the same issue, or with
the same normalized session, stays blocked.
Why this scope is safe: the resources that a retained owner can still
hold are its runner state root and its workspace. The server derives the
runner state root from the session scope (company, agent, workspace,
provider, normalized session). The workspace belongs to the task.
Different tasks already run concurrently on the same backend, so a
different task shares no resource with the retained owner that normal
concurrency does not already share.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/paperclip-runner exec
vitest run src/native-session-runtime.test.ts -t "unrelated tasks of the
same backend"` failed with `native_session_cleanup_quarantined` for the
unrelated task (the RC-20 error).
- Green after the fix: the same test passes. The full file passes
(211/211).
- `pnpm --filter @paperclipai/paperclip-runner typecheck` passes.
## Risks
- A retained owner can no longer block a different task that shares a
project workspace. Concurrent tasks in a shared workspace are already
allowed, so this adds no new kind of overlap.
- The same task and the same conversation stay blocked. A Retry of the
task whose close failed can still be accepted and then fail at
admission. The trigger of that close is handled separately.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a0cba70f18 |
fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r<hex>:<id>`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
83c6832ccb |
fix(build): typecheck package, Playwright, and e2e test files in CI (#18839)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner, plugin, and Playwright tests verify how those agents run, connect, and appear in the board > - Most package tsconfigs exclude tests, and Vitest strips types, so a type error in those tests fails nothing > - The tests drifted to 803 real type errors, and some of those errors hid assertions that checked nothing > - paperclipai/paperclip#18739 and paperclipai/paperclip#18738 check the server test tree and four test projects, but not these files > - This pull request adds a required check for these test files, fixes every error, and fails CI if a test file is in no program > - The benefit is that a type error in any test file outside `server/src` now fails CI ## Linked Issues or Issue Description Refs #18739 Refs #18738 **What happened?** 517 test files were in no program that required CI checks. A type error in those files did not fail CI. **Expected behavior** CI fails on a type error in any test file. **Steps to reproduce** Add `export const value: number = "text";` to a test file under `packages/paperclip-runner/src` on master. CI stays green, because `tsconfig.json` excludes test files and Vitest strips types. **Measured gap** Each row is a program with its tests included, on one thread. File and error counts are for head `7d4b5e907a`. Memory peaks were measured before the two latest merges of master. | Group | Test files | Real errors | Peak | | --- | ---: | ---: | ---: | | `paperclip-runner` Node source | 290 | 644 | 1.09 GiB | | `paperclip-runner` surfaces | 9 | 12 | 0.42 GiB | | `paperclip-runner` devtools and examples | 14 | 0 | 0.68 GiB | | `sandbox-providers/kubernetes` | 21 | 65 | 0.44 GiB | | 8 other sandbox providers and `bridge-template` | 32 | 4 | 0.34 GiB | | `plugins/sdk` | 22 | 15 | 0.41 GiB | | `plugin-workspace-diff` | 5 | 0 | 0.37 GiB | | `plugin-orchestration-smoke-example` | 1 | 1 | 0.15 GiB | | `kv-demo-mcp-server` | 3 | 0 | 0.13 GiB | | `packages/db` scripts | 1 | 0 | 1.01 GiB | | 8 Playwright suites under `tests/` | 52 | 25 | 1.72 GiB | | `tests/runner-acceptance` | 3 | 0 | 0.58 GiB | | `tests/e2e`, through layer declarations | 63 | 37 | at most 1.59 GiB | | `ui` `AttentionInteractionResolver.test.tsx` | 1 | 0 | n/a | | **Total** | **517** | **803** | | 27 of the 803 errors came from master while this pull request was open. Each merge of master brought new test code that no check had read. Errors that were config errors, not real errors: - One TS2688 for each provider, because the root install does not install provider dependencies. - 9 in the Runner Node program (TS7016 and TS6059). - 2 in `tsconfig.browser.json`. TypeScript 7 removed `baseUrl`, so that config was invalid. - 213 in the Playwright suites (`@/*` aliases and Vite types). - About 60 in `tests/e2e` (import extensions and aliases). `packages/plugins/create-paperclip-plugin` has no gap. Its tsconfig excludes its test, but `cli/tsconfig.json` reaches that test through an import. `server/test-baselines` also has no gap, because `tests/lifecycle-baseline` reads it. ## What Changed - Added `scripts/typecheck-test-programs.mjs` and its tests. The `Typecheck + Release Registry` job runs it. It checks each program in `PROGRAMS` on one thread. - The script fails if a tracked `*.test.ts(x)` or `*.spec.ts(x)` file outside `server/src` is in no program. The compiler file list decides this, because `include` drops `foo.test.tsx` when `foo.test.ts` exists. - Added a `tsconfig.test.json` to each uncovered package, and `tests/tsconfig.playwright.json`. - Added `tests/e2e` to `PROJECTS` in `scripts/typecheck-test-projects.mjs`, with a `tsconfig.json` and a `tsconfig.layers.json`. - Added `scripts/typecheck-sandbox-providers`. It is a small workspace that installs provider SDKs from one lockfile. I did not add a lockfile inside each provider, because `scripts/build-standalone-public-packages.mjs` uses a frozen install when one exists, and that changes what a release resolves. - Corrected `packages/paperclip-runner/tsconfig.browser.json` for TypeScript 7. - Set `rootDir` to `..` in `packages/paperclip-runner/tsconfig.test.json`, because one Runner test imports from `packages/adapter-utils/src`. The program does not emit files, so this changes no build output. - Moved the Kubernetes provider from `vitest ^3.2.4` to `^4.1.8`, the version the other providers use. Vitest 3 requires `tinypool ^1.1.1`, which has two critical advisories (GHSA-85c8-ppgw-ccpr and GHSA-5gmw-xhrv-c9v3). The first patched version is 2.1.2. Its tests have the same result on both versions: 189 passed, and the same 12 `file-sync` tests failed. - Moved the Kubernetes provider from `zod ^3.24.2` to `^4.6.6`, and gave its two `z.record` calls a key schema. This is the only production change, and it stays in this pull request. See "Production change for review". - Renamed `AttentionInteractionResolver.test.ts` to `AttentionInteractionResolver.replace.test.ts`, so TypeScript reads the `.test.tsx` beside it. - Fixed all 803 errors, with changes to test files and test support only. The fix for `contracts/native-execution.test.ts` is the same as the fix in paperclipai/paperclip#18739, so those two changes merge cleanly. - Updated `doc/DEVELOPING.md`. This pull request does not edit `server/scripts/typecheck-layers.mjs`. paperclipai/paperclip#18739 and paperclipai/paperclip#18738 have both merged, and this branch includes them. ### Production change for review The new lockfile made dependency review flag `zod@3.25.76` as a high-severity advisory (GHSA-mrhx-c7xc-m3w5). The first patched version is 4.6.6, so 3.x has no fix. The published Kubernetes plugin uses zod to validate operator config. The zod 3 source does not compile against zod 4, so a typecheck-only override cannot fix this. The existing tests do not prove this change, because they also passed with the invalid one-argument `z.record`. So I parsed 45 inputs with the same schema on zod 3.25.76 and zod 4.6.7: | Result | Inputs | Detail | | --- | ---: | --- | | Identical | 23 | Includes defaults, valid config, and a `__proto__` key | | Same outcome, new error wording | 21 | For example, `Invalid url` is now `Invalid URL`. `validateConfig` shows this text to operators. | | Different outcome | 1 | `jobTtlSecondsAfterFinished: 2 ** 60` was valid and is now rejected. The new limit is `Number.MAX_SAFE_INTEGER`. | I did not test this against a real cluster. The decision is to keep this change in this pull request. Since then, paperclipai/paperclip#17364 has moved the rest of the repository to zod 4.6.x, so the Kubernetes plugin now matches it. ### Behavior change for review `reducedMotion` under `use` is not a Playwright option. It had no effect in 11 configs and 2 `test.use` calls, so those suites ran with `no-preference`. The options are now under `contextOptions`, so reduced motion now takes effect. `app-shell-design.spec.ts` asserts a real transition, so it now asks for `no-preference` itself. The baseline manifest has `snapshotCount: 0`, so this invalidates no published baseline. A frozen trial reference PNG can differ. ### Findings These are product questions. I did not change them, and I did not hide them with casts: 1. **`PrpEvent` and its schema disagree.** The TypeScript type in `src/protocol/replay-contract.ts` requires `normalizedSessionId`. The `required` list in `protocol/schemas/event.schema.json` does not include it. `validatePrpEvent` accepts events that the type forbids. 2. **`requestCheckboxConfirmation` is stricter than its runtime.** It uses the output type, so plugin authors must pass `continuationPolicy` and `allowDeclineReason`, although the schema gives them defaults. `z.input` would fix this. 3. **`RetainedRunnerdMaintenanceInput.backend.kind` is a free string.** The quarantine domain comes from `descriptor.kind`, but the release uses `proof.backend.kind`. If they differ, `completeRetainedNativeSessionCleanup` returns 0 and the quarantine stays. The production caller passes `"runner"` today, so this is not a live bug. 4. **`CodexHarnessSession.close(input?)` is optional, but the contract requires `{ reason }`.** These are tests that verified nothing. I corrected them, and they pass: - `backends/native-instruction-measurement.test.ts` filtered on `placement === "core"`, which is not a valid placement. It measured an empty tool list. It now measures 14 tools. - `drivers/acpx/codex-acpx-driver.test.ts` checked for `usage.updated`, which is not an event type. - The `hostFixture()` helper in `drivers/acpx/runtime-host.test.ts` ignored two overrides, so `expect(prepareSandbox).not.toHaveBeenCalled()` could not fail. - `live/live-session.test.ts` set `agent: "pi"` before a spread that overwrote it. - `native-distribution-integrity.test.ts` passed an AbortSignal where `openCommand` expects `{ signal }`. These are gaps that this pull request does not close: - **The root lockfile still has some old zod versions.** paperclipai/paperclip#17364 moved the direct zod dependencies to 4.6.x. `pnpm-lock.yaml` still resolves `zod@3.25.76`, `4.1.11`, and `4.4.3`, which are below the patched 4.6.6. `packages/paperclip-runner/scripts/pi-distribution/package.json` pins `zod` to `3.25.76`, and some agent SDKs resolve `4.4.3`. Dependency review does not flag them, because this pull request does not add them. - **Dead GitHub coverage.** `chat-adapters-ui-providers.spec.ts` returned early for GitHub, so four later `provider === "github"` branches never ran. I removed them. They included the private key file and paste match, and the reconnect flow. Nothing asserts those behaviors in that spec now. - **Provider tests do not run in CI.** The root `vitest.config.ts` lists only Daytona. The other 8 providers, `bridge-template` included, are typechecked now, but CI does not run their tests. - **One `packages/db` test never runs.** `scripts/clean-poisoned-claude-sessions.test.ts` is outside the `include` of `packages/db/vitest.config.ts`. - **`plugin-orchestration-smoke-example` does not collect.** `@paperclipai/shared` is not linked in its `node_modules`. It passes 3 of 3 with an alias. ## Verification Run these commands: ```sh node scripts/typecheck-test-programs.mjs --low-memory pnpm --filter @paperclipai/server typecheck node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations node --test scripts/typecheck-test-programs.test.mjs ``` Results on macOS (arm64), Node 26.12. CI uses Node 24, and its `Typecheck + Release Registry` job also passes. At head `7d4b5e907a`, the test programs, `tests/e2e`, and the script tests pass again with 0 errors. The times and peaks below come from the run before the two latest merges of master: | Check | Result | | --- | --- | | Test programs, `--low-memory` | Exit 0, 0 errors, 42 seconds, peak 1.78 GiB | | Server layers, low memory | Exit 0, 0 errors, 38 seconds, peak 3.41 GiB | | Test projects, including `tests/e2e` | Exit 0, 0 errors, 13 seconds, peak 1.59 GiB | | Script tests | 12 passed | | Injected type error in each of 23 places | Each one failed its program with TS2322 | | Coverage check | 0 unchecked files, 10 seconds | I did not run this in a 4 GiB container. The peaks above are `maximum resident set size` for the largest single process. **Runtime check of the fixes.** Types do not prove that a test still works, so I ran the suites before the two latest merges of master: | Suite | Result | | --- | --- | | Runner, full suite | 4,885 passed, 3 failed, 13 skipped | | Sandbox providers, 10 packages | 939 passed, 12 failed, 36 skipped | | Plugin SDK | 200 passed | | `kv-demo-mcp-server` | 12 passed | | Renamed UI test and its sibling | 4 passed | All 15 failures exist without these fixes or come from the local machine: - **3 Cursor Cloud timeouts.** `artifact-delivery.test.ts` has 1 and `cursor-cloud-driver.test.ts` has 2. They time out in a full local run, and all 133 tests in those files with `codex-security-config.test.ts` pass when run alone. They fail in the same way with those two files at their unedited versions. - **12 Kubernetes `file-sync.test.ts` failures.** They fail in the same way before and after the fixes. The first failure is that `/proc/self/fd` does not exist on macOS. After each of the two latest merges of master, I ran only the Runner test files that I changed: | Merge | Result | | --- | --- | | `885c249cb9`, 8 files | 534 passed | | `7d4b5e907a`, 7 files | 672 passed, 16 failed | The 16 failures are in the two `source-reconciliation` test files. They fail in the same way with master's unedited versions of those files. 15 of them fail with `ACPX sidecar exited at initialize`, so I think my local build of the Rust and sidecar binaries is out of date. I did not rebuild them to check. CI builds them from scratch, and all Runner jobs pass in CI at `7d4b5e907a`. **Not run:** - `tests/e2e` and the 8 Playwright suites need a server, so most of them have only compile checks. `playwright test --list` loads 308 tests from the main e2e config. - `packages/plugins/sandbox-providers/kubernetes/test/integration` needs a cluster, so it has only compile checks. - I did not run the full `pnpm test:run` or root `pnpm build`. Required CI runs them. **Reduced motion.** I ran 6 motion-sensitive Storybook specs in system Chrome with a temporary config, once with `reduce` and once with `no-preference`. Both runs had 45 passed and the same 5 failed. So reduced motion did not change those results. The 5 failures do not depend on motion. At least one comes from the browser, because system Chrome prints `oklch(55.6% 0 0)` where the test expects `oklch(0.556 0 0)`. I did not run the other Storybook specs. The `storybook-visual` label runs that workflow. ## Risks - **A new lockfile to maintain.** A change to a provider `package.json` now fails the typecheck job until someone refreshes `scripts/typecheck-sandbox-providers/pnpm-lock.yaml`. The failure message gives the command. - **The job has a new install.** The step downloads provider SDKs, so it depends on the npm registry. It uses `--frozen-lockfile` and `--ignore-scripts`. - **The local install replaces provider `node_modules`.** A contributor with a standalone provider install must reinstall that provider after running the script. - **Reduced motion can change Storybook results.** See "Behavior change for review". That workflow is not part of required CI. - **Kubernetes plugin validation changed.** A release of this plugin ships zod 4. Error wording changes, and very large integers are now rejected. See "Production change for review". - **Master moves fast.** Each merge of master brought new test code with type errors, 27 so far. After this merges, the check reports those errors in the pull request that adds them. ## Model Used - Provider: Anthropic Claude, through Claude Code - Model: Claude Opus 5.5 (`claude-opus-5-5`) - Capabilities: tool use, code execution, and subagents for the type fixes in separate files ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details. The app generated the branch name. - [ ] I have run tests locally and they pass. See the failures explained in Verification. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
4e29f00139 |
fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. **What happened?** A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. **Expected behavior** A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. **Steps to reproduce** 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. **Paperclip version or commit** `master` at `b0c4e345267`. **Deployment mode** Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - **Silent stall.** The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - **Candidate admission** carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - **The agent that loads the conversation** stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - **In-flight provider work** is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - **Launch shape.** A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - **Artifacts** are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press **Retry** once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3910352c92 |
feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives each provider durable execution, tool, and completion contracts. > - Cursor Cloud currently has a direct adapter outside that native contract. > - Cloud work needs exact ownership across HTTP, SSE, Stop, and recovery. > - This PR adds a qualified Cloud API driver and verified proxy to Runner. > - Product setup and hosted placement remain separate dependent changes. ## Linked Issues or Issue Description Refs #5664, #11803, #17299, and #17471. Product integration is #18069. Managed-only API placement is #18310. The related direct-adapter model/transcript work is not replaced. No duplicate native Cursor Cloud integration was found. ## What Changed - Add native `cursor_cloud` / `cursor_cloud_api` with explicit model/repository validation, bounded HTTP/SSE, durable per-turn checkpoints and a qualified Node proxy launched through the Rust facade. - Commit terminal custody before publication/release; coalesce consumers, preserve ambiguous create/Stop/recovery fences, reject late tools, and distinguish definitely-unsent oversized requests from ambiguous responses. - Support scoped cold attachment, current authority, reconnect and verified artifact staging. Keep unsupported warm/external and Windows paths closed. - Preserve early source cursors and verify immutable legacy prefixes before suffix replay, including supported compaction/masking. Optional proof publication failure falls back to complete cold verification. - Preserve canonical records at the durable native producer with JSON-safe privacy and the existing 8MiB bound; drain admitted successor flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and retain S3 and legacy CLI checks. ## Verification Core head `37c442ec8dd4`: all 66 check contexts are terminal successful/neutral/skipped, including required `ci / verify` and `ci / e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads. Dependent Product `81e828d20a77` and Top `dd2959689627` include the later client handoff correction and are running their own fresh gates. Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and normal Runner build passed. Runtime Top `9af1a5de42a2`: build and recursive typecheck passed. Earlier source-scoped evidence includes 398 focused tests, five relay tests and token/type gates. The 251-assertion, 7/8-suite run failed physical PostgreSQL teardown; the unchanged stable WSP suite passed 128 through the documented wrapper. The downstream fixture corrections passed 7/7 browser and 121/121 startup tests without timeout/oracle relaxation; a broken-unsubscribe control fails. Hosted runtime `9af1a5de42a2` has a passing normal build and recursive typecheck. Latest dependent heads additionally contain a pure client fix for stale live/history overlap, with 48 focused tests, UI typecheck and token gates passing. Its new client is not deployed in the frozen hosted image; server/native/API/provider source is unchanged by these four downstream files. Fresh hosted Linux admission on `9af1a5de42a2` retired its owned runner/proxy/socket with zero prompts. The saved configuration survived reopening, and Test authenticated the bound key/catalog/selected model without a prompt. Artifact with `claude-sonnet-4-5` completed, downloaded the exact 27 bytes, matched 121 canonical identities against complete logs and independently reached FINISHED. The active Stop retry was clicked within its immutable boundary, produced one canonical interrupted turn and acknowledged run-only cancellation, retained a valid 35.076s quiet window (maximum gap 4.584s), and independently remained CANCELLED/artifact-absent beyond the original 180-second command deadline. All provider-observer credential callbacks, DB clients and loaders closed; accounting is settled and explicitly unpriced. The first fresh Stop retains its failed 10.373s sampling gap and every sample. On the upgraded9af server, the historical F226 artifact was reopened and downloaded with its exact original hash after its provider agent had been deleted/GET404 earlier today. Historical F226 full Artifact/Stop cleanup, Question, corrected Plan and reconnect retain their original identities. Deleting the three newly created disposable provider agents and the new-case post-delete grade remain blocked on explicit user approval after automatic review rejected expanding the two original approved IDs. No new-case cleanup or complete fresh matrix pass is claimed; this is separate from the directly observed task/Stop behavior and retained exact cleanup proof. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier startup-title, title-save, custody/Stop and deletion failures with unknown causes and bounded recoveries. `dea023` Artifact passed 17 checks but Stop failed its false warning and 50 duplicate log identities. Pre-repair ACK failure and original Core CI failure remain recorded; retry alone establishes no cause or full local-suite pass. Provider dollar cost stays UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. The caller’s explicit values are submitted and behavior checked live. Unpriced provider dollar cost stays UNKNOWN. Only qualified per-turn controllers are admitted. Uncertain work and incomplete cleanup stay fenced. Existing damaged/duplicate archives remain failed evidence. Prefix proof adds bounded object-store I/O; the 8MiB producer bound does not guarantee future providers’ output fits. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
f8a888a072 |
feat(runner): define bounded startup lifetime contracts (#18490)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native sessions and plugin workers have separate startup lifetimes. > - A caller timeout does not prove that a worker or an accepted host write has stopped. > - Shared startup code needs an exact deadline and an actual settlement contract. > - This PR adds those contracts and tests them before any production caller uses them. > - The dependent integration in #18405 will use the contracts with verified cleanup. > - This split lets reviewers assess the lifetime rules separately from runtime activation. ## Linked Issues or Issue Description Refs #18405. This PR extracts the foundation from that change. The integration will depend on this PR. **Problem or motivation** A bounded startup must distinguish the caller's wait from actual worker completion. A late worker reply or an accepted host request can outlive the original RPC promise. A shorter timeout must not discard that responsibility or authorize a replacement run. **Proposed solution** Add one Core execution context, optional Runner startup deadlines, and worker settlement and retirement APIs. Test the contracts with real child-process IPC. Keep production context creation, transport projection, capability negotiation, durable cleanup, and Stop/recovery integration in #18405. **Alternatives considered** One combined PR mixed the lifetime contract with its activation and cleanup policy. Activating a shorter deadline before the complete transport and cleanup path is unsafe. Splitting only the SDK declarations would not provide a useful behavioral review boundary. **Roadmap alignment** This supports the existing plugin, sandbox execution, and bounded recovery capabilities. It adds no new product workflow, endpoint, migration, provider or runtime dependency. ## What Changed - Add `NativeExecutionContext`. It retains the selected input, binds one transport target, rejects work after cancellation or expiry, and tracks actual worker settlement. Expiry alone does not complete pending work. - Add worker startup groups. A caller timeout retains the group's responsibility. Retirement revokes obsolete invocation scopes, waits for accepted host handlers, and retires only the verified group's bookkeeping. Each cleanup wait is bounded; an unfinished handler remains pending for a later retry. - Add an optional Runner startup signal and absolute deadline for session open and retained attachment. Clean up abort listeners after synchronous failures. Existing callers keep their default timeout path. - Expose the remaining confirmed controller-lease budget without a new query or renewal timer. - Add optional SDK context and preparation-retirement fields. No production bridge sends or advertises them in this slice. - Keep the required lease mock update with the new `remainingMs()` contract so this foundation typechecks independently. Production activation files, including the native executor and environment task bridge, match the current master base. No production caller constructs `NativeExecutionContext` or passes the new Runner startup option. This PR adds no durable cleanup producer and claims **no standalone TTFT improvement**. The dependent integration must enable the context, transport and verified cleanup together. ## Verification Initial split verification on `8ebc8a2f9c6c02e90f920356138b3742a4067b28`, based on `9a3f395737e7a92faf99971459979061d423531a`: - From `server`: `../node_modules/.bin/vitest run src/services/native-runtime/native-execution-context.test.ts src/services/native-runtime/native-execution-context-worker.test.ts src/services/native-runtime/native-execution-lease.test.ts` — **17 passed**. Includes real child IPC, caller timeout with late reply, worker death, accepted host-handler draining, obsolete cleanup invocation rejection, unrelated work preservation, context scope, expiry, and confirmed lease budget. - From `packages/paperclip-runner`: `../../node_modules/.bin/vitest run src/native-session-runtime.test.ts` — **188 passed**. Includes existing callers without the new option and new synchronous-failure/expired-deadline cases for fresh and retained sessions. - Direct TypeScript checks for `packages/plugins/sdk/tsconfig.json` and `packages/paperclip-runner/tsconfig.json` — passed. Built only their shared/eval TypeScript prerequisites. - From `server`: `../node_modules/.bin/tsc --noEmit` — passed. - Checked the partition against the reviewed combined candidate. The twelve selected files match exactly. The thirteenth file carries only the required test-mock update. The production executor, bridge and target wiring remain equal to the base. - `git diff --check`, added-line secret/private-data scan, and added-code-comment scan — passed. No generated files or run artifacts are committed. - The first old-base split passed its focused tests but failed server typechecking because the unchanged executor test mock lacked `remainingMs()`. The original failure is retained. Moving that one-line mock update into this foundation resolved the split dependency; the final current-master checks above pass. - Full local CI suites, Rust builds, deployments and provider calls were not run, as requested. GitHub CI will provide full regression coverage. Current-head GitHub CI, standard review and Apex are complete; exact acceptance is recorded below. Review follow-up `d7635f4d280dab757a3666703737bcda955c9361` fixes three reproduced lifetime edge cases. A retiring startup group rejects new calls and shares one retirement promise; failed bounded retirement remains retryable. Transport identity uses a fixed field tuple rather than object-property order. Finishing a context rejects pending lookup waits without aborting an already-started backend or pretending the worker completed. All three new regressions failed before the fix; the updated foundation selection passes **20 tests**, including actual IPC and failed-retirement retry. Server typechecking and added-line checks pass. These replace the earlier 17-case selection, rather than adding 20 unique cases to it. The original failures and passing reruns are retained. Apex follow-up `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` keeps removed worker handles reachable while they own startup calls, accepted host handlers or retirement work. Drain/retirement includes active and retained handles; completion callbacks remove settled handles. Stop and replacement remove only the exact handle they observed. This adds no database/network operation, polling loop or generic shutdown wait. The SDK, Runner, context and transport bridge are unchanged. Seven additional IPC regressions failed on the prior manager and pass after the fix, covering removal/replacement, Stop-all, host-handler settlement, bounded retirement failure/retry, pruning, and replacement ownership. The updated foundation selection passes **27 tests**, with server typechecking passing; these supersede the earlier 20-case selection. A separate real-PostgreSQL/child-IPC characterization uses the integration's actual SDK host, storage registry and run-control path with the exact corrected manager bytes. Before the fix, cleanup released run ownership while an accepted host write was blocked, then the late write restored obsolete metadata. After the fix, ownership and cleanup remain pending until the write completes; cleanup then removes the stale metadata before releasing ownership. The original failure, corrected result and source snapshots are preserved. Final head `cb8860bab5ab9d6357a352e11f276be0d0cdb26d` passes GitHub CI (53 successful checks, 2 expected skips, Snyk success), standard Greptile **5/5** and fresh Apex **5/5**. All review threads are resolved and the PR is conflict-free. ## Risks The shared Runner helper now observes synchronous operation failures through its asynchronous cleanup path. The affected runtime file's 188 tests pass, but full regression coverage still depends on CI. The new lifetime APIs require honest settlement signals. A future caller must not treat a deadline, worker crash or Stop intent as verified retirement. Production activation remains dependent on the complete integration and its cleanup tests in #18405. This foundation introduces no shorter production startup budget or new cleanup policy by itself. The context's transport projection is process-local. It must not be persisted or used as a general authorization cache. No production caller supplies it in this PR. There is no database migration, new dependency, or deployment requirement for merging this foundation. ## Model Used OpenAI Codex, based on GPT-6, assisted with repository inspection, the split, tests and command execution. The exact serving model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal issue identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — this description documents the inactive contracts; operational activation documentation remains with #18405 - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on `cb8860bab5` - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fe3e0f2d5f |
Preserve rejected native completion reports as failures (#18227)
Keep rejected declared completion reports on the existing permanent native failure path. Preserve rejection across restoration and order permanent failure ahead of terminal goal facts, goal results, and continuation while retaining the original provider terminal. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
4808a3a2b1 |
fix(heartbeat): retry operator Stop after a transaction deadlock (#17895)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/69/changes/paperclip/69 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operator Stop must revoke task authority before it stops provider execution. > - Its transaction locks the task and then the run. A concurrent writer can hold the run while it waits for the task. > - PostgreSQL can abort the Stop transaction with a confirmed deadlock. The cancel API then returns 500. > - This pull request retries only that aborted database transaction, at most three times. > - Provider Stop and activity publication stay after commit, so the retry cannot repeat those effects. ## Linked Issues or Issue Description Refs #17639. Its agent-chat project browser test completed the product assertions, then failed during cleanup when the cancel API returned 500. A real PostgreSQL lock-cycle test reproduces the same `40P01` error in the Stop transaction. Related: #13911 addresses audit foreign-key lock contention. This change handles a different concurrent run-writer path without changing the schema or task/run lock order. Stack base: #17639. This is a separate correctness repair for the sleep migration stack, not another polling migration. ## What Changed - Control the runner timeout regression fixture's clock so it reaches the pending append before expiry; retain production cancellation and late-append assertions. - Retry the Stop authority transaction only after PostgreSQL reports `40P01`. Make at most three attempts, with short bounded backoff. - Discard rolled-back local run state, claim flags, and queued activity publications before each retry. Re-read current authority under fresh locks. - Keep provider cancellation and activity publication outside the retried transaction. Preserve other errors and the existing native lock-conflict response. - Add a real two-transaction deadlock test. Add server-side failure tests for retry exhaustion and errors that must not retry. - Let run-control tests use a validated disposable PostgreSQL database URL. ## Verification Published head: `60ba892ae9fc5faaf8cd0833694b4a66c13323c3`, targeting master after #17639 merged at `1ed0af41867ba40fd1d23292d395588d18c2fcbe`. The normal ancestry merge preserves the complete five-path tested cancellation patch byte-for-byte, including the review repair, and all four unrelated current master paths. The production retry logic remains unchanged from this slice's reviewed implementation; the follow-up adds coverage and its audit. - **228 distinct complete cases pass:** all 48 run-control cases and all 180 native-session runtime cases. The two new targeted cases are included in 48 and are not counted again. Server typechecking, build and module-boundary checks also pass. - The new real PostgreSQL cases reach a failed native run's correlated Stop claim, cancellation fence, board handoff retirement and activity write before raising `40P01`. One case preserves retry authority and commits exactly one cancellation activity/live notice. The other changes coordinator authority between attempts, returns 409, preserves the failed run and handoff, and performs no provider/workspace cleanup. - A negative control removes only the publication-list reset. It fails with two live notices for one durable activity. Production was restored byte-for-byte before final validation. The existing actual two-connection deadlock, bounded attempts, non-deadlock rejection and post-commit provider behavior remain covered. - Preserve the initial new-assertion typecheck failure (optional live-event payload) and the corrected final pass. Service-local test files are typechecked; `src/__tests__` is excluded. The negative-control harness initially expected an untruncated Vitest assertion line; the actual named 1-versus-2 failure and restoration were inspected before correcting its result matcher. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/60ba892ae9fc5faaf8cd0833694b4a66c13323c3/doc/plans/2026-10-28-cancellation-deadlock-acceptance.md) records current scope, commands, isolated fixtures, mutation evidence and limits. Earlier combined-stack counts retain their original revision scope. The existing controlled runner timeout/late-append fixture is retained and passes within the complete 180-case runtime suite. - This addresses the prior Ponytail request to exercise native claimed state before deadlock rollback and re-evaluate authority afterward. The advisory loop-invariant snapshot/named-attempt-bound note remains a maintainability suggestion, not a runtime defect. No production loop refactor or new worker duty is introduced. - **All 58 current checks are clean; Greptile and Ponytail both rate this exact head 5/5.** Ponytail explicitly confirms that the native-claim coverage request is resolved and says Ship. The formerly queued dependency/quality review workflow has completed successfully. Old `fffc`/parent checks and reviews are not current passes; the former pending Snyk status is not called successful. Actual failing Ponytail findings must be fixed. Missing/stopped/paused turns without a verdict are authorized nonblockers. Active unfinished reviews retain their original per-head timer, with a 25-minute monitor cutoff respecting the 30-minute maximum. - No full local repository matrix, live provider execution, external alarm, physical host/database suspension or savings claim is made. No schema, dependency, workflow, shared scheduler contract or shared OS resource changes are introduced. ## Risks - Persistent contention still returns an error after the third attempt. This change does not promise to eliminate every possible lock cycle. - Retrying a write after an ambiguous connection failure would be unsafe. The retry accepts only PostgreSQL's confirmed deadlock abort code; it does not accept transport errors. - No schema or public API contract changes. Current task/run authority and cancellation eligibility are checked again on each attempt. ## Model Used OpenAI GPT-6 via Codex, with shell, GitHub CLI, TypeScript, PostgreSQL, and Playwright tools. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
f5b6721ae0 | Use the admitted session directory in native task prompts (#17310) | ||
|
|
d84061b923 |
fix(runner): recover journaled completion proposals (#16962)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs preserve provider sessions across controller restarts. > - A completion event can reach durable storage before the next session checkpoint. > - Recovery advanced the event cursor but lost the saved completion proposal. > - The completed turn then appeared to need another provider turn to report its result. > - This change restores the exact active turn's proposal before advancing the cursor. > - Recovery can finalize completed work without repeating its external actions. ## Linked Issues or Issue Description **What happened?** After a controller restart, a native run could fail with `native_finalization_missing` even though `run.result.proposed` was already durable. The recovered driver skipped that event when its cursor advanced, so the result was absent from its state. Automatic disposition retries could then encounter outstanding tool receipts and exhaust recovery. **Expected behavior** Restore the saved result for the exact active turn. Wait for provider termination, then use normal finalization. Do not start another provider turn to repeat completed work. **Steps to reproduce** 1. Start a native Codex task and save its active-turn checkpoint. 2. Persist its completion proposal, then lose the controller before another checkpoint. 3. Recover the same run after the provider completes. Before this change, finalization reports a missing result. **Paperclip version or commit** Reproduced on `92b238137b`. The repair is based on `7c247243e5`. **Deployment mode** Native Runner with Codex. The incident used a remote sandbox; the regression covers controller event recovery. Related: #12845 and #15297. The older #12241 addresses provider process continuity; this change repairs the controller's saved event projection. ## What Changed - Restore valid completion proposals for the checkpoint's exact active turn during event replay. - Reject conflicting, invalid, or incorrectly bound proposals. Ignore proposals from another turn and prevent adoption after a newer turn starts. - Keep provider termination and normal finalization as separate requirements. - Add eight behavioral cases using the real Codex driver and a simulated transport, and document the recovery contract and verification scope. ## Verification - Red: the new restart regression failed with `native_finalization_missing` before the production change. - Green: 254 tests passed across the native runtime, backend adapter, and Codex recovery suites. The new cases check no repeated provider work, no early finalization, duplicate proposals, conflicts, invalid proposals, and stale turn/session bindings. - `node scripts/check-maps.mjs` and `git diff --check` passed. - `pnpm -r typecheck` and `pnpm build` passed locally. All CI gates passed on `2ac025a15c`; Greptile rated that commit 5/5 with no outstanding findings. The full local `pnpm test:run` was started after completing the embedded PostgreSQL installation, then stopped while still running once the complete CI test matrix passed; it is not reported as a completed local run. - Real local acceptance passed on `2ac025a15c` with the production Rust Runner transport, Codex CLI `0.160.0`, and `gpt-6.1-sol` on macOS arm64. Persisted the completion proposal while retaining the older active-turn checkpoint; let the provider finish without persisting its later controller events; stopped the exact owned runner/provider; recovered the same provider session and turn. Observed one accepted result, one durable `turn.start` command, and one line in the receipt file. Hosted sandbox restart and recovery of the original stopped task have not been performed. - Initial setup failures are retained: dependency drift caused diff-viewer type errors, and skipped PostgreSQL postinstall links prevented database startup. Dependency alignment and the package postinstall repaired those local prerequisites. Probe setup iterations failed on a direct-provider response schema, invalid probe input fields, and an original provider that still held its writer lock. Acceptance uses the production transport, validated input, and exact-owner shutdown. No production code changes were made for those probe setup failures. ## Risks - A stale proposal could finalize the wrong work. Recovery requires the checkpoint's active turn and matching run, session, and source. It rejects conflicting results and waits for provider termination. - The change uses the existing paginated recovery scan. It adds no database query per event or new retry allowance. - No schema migration or change to task authorization is included. ## Model Used OpenAI GPT-6, running as Codex with reasoning, tool use, and code execution. The exact served model snapshot and context-window size are not exposed in this session. The live acceptance probe uses `gpt-6.1-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
d855e550ad |
Finish Copilot native runtime on current master (#15701)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner controls provider processes, permissions and task completion. > - Copilot needs its pinned distribution, native message identities and accepted semantic completion receipts. > - The release must preserve saved-token access and task lifecycle guarantees. > - This pull request will keep the current completion and recovery fixes while adding the Copilot runtime. > - The benefit is a reviewable part of the Copilot production release. ## Linked Issues or Issue Description Refs #14434. Supersedes the runtime work in #15560. This is the first PR in the stack; #15703, #15705 and #15706 follow. Related alternative adapter work: #15881. This change uses the shared Runner path. ## What Changed - Pin Copilot CLI 1.0.88 and bind the integrated implementation as profile45. - Keep native instruction, permission, receipt, process ownership and cancellation boundaries. - Preserve current master completion continuation and remote verification behavior. Other provider release identities stay unchanged. ## Verification Final reviewed source [`4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5`](https://github.com/paperclipai/paperclip/commit/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5) includes master `cde944ed75648adfb569e86133adc8b118b55dd6`. The combined four-part PR tree matches it except for the CI-owned lockfile. Every prior public PR head remains an ancestor. Released master migrations remain unchanged; Copilot adds migration361. No unrelated provider is enabled. Copilot CLI1.0.88 and profile47 retain digest `sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`. The pending qualification marker is removed. The [qualification record](https://github.com/paperclipai/paperclip/blob/4f3f1af807d2ed21adabb04c2f27f4df9c4bc8d5/doc/plans/2026-10-20-copilot-qualification-v47.json) reports **20/20 Product cells,154 assertions;7/7 protocol cases,46 assertions** against frozen installed application `c2616370800f5f6df45246b0890ba35a7cd2faab`. Both environments use the exact authenticated model `gpt-5.6-luna`. Every passing Product case confirms cleanup. Automatic behavior retries are disabled. Original failed attempts remain recorded. The ordinary installed remote journey passed saved-token selection, authenticated model discovery, normal UI save/reopen, metadata-only Test, task completion, accepted receipt and an exact19-byte browser download. An open model catalog caused no new discovery requests during117seconds. Actual installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token browser entry was not exercised; the encrypted connection was created through the normal API. The original Daytona file attempt timed out during delayed semantic dispatch. Its precise stalled layer remains unresolved. Independent ordinary delivery and the other19 cells passed before one instrumented recheck passed8/8 with unchanged source,prompt,assertions and deadlines. No causal latency repair is claimed. After current-master reconciliation,129 connection tests,59 runner tests,72 shared eval tests and88 recovery/workspace tests pass. Migration generation/safety, unchanged profile identity and UI token gates pass. Prior required CI was green. Fresh exact-head required CI and E2E are green for all four updated heads. Each has a fresh Greptile5/5 review and zero unresolved review threads. The [final normal-package cloud build](https://github.com/paperclipai/paperclip/actions/runs/39414239704) is running. Recursive typecheck,full CI test shards,build and all8 browser shards pass. These checks cover the final source; no additional local full-suite pass is claimed. The [frozen image build](https://github.com/paperclipai/paperclip/actions/runs/39334401638) and [original normal package build](https://github.com/paperclipai/paperclip/actions/runs/39334407471) passed. The exact image is published and independently retrieved anonymously. No separate full-image scan is claimed. All18 original normal archives were scanned with zero selected credentials. Docker builds run in the cloud. Provider USD remains unknown and GitHub-attributed. **Qualification is complete on the recorded frozen build. Normal protected merge, automatic master canary publication and final shipped-package local/Daytona canaries remain. No production deployment is claimed.** ## Risks Shared Runner behavior can affect active tasks. Exact profile admission rejects stale runtime identities. The final campaign and installed canaries remain release gates. ## Model Used OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this task. Used for code changes, review and tool-assisted verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fa9f3f8a08 |
feat(runner): support preconfigured user homes for Codex and ACPX (#16486)
## Thinking Path > - Paperclip manages AI agents and their work. > - Runner launches the selected provider and keeps durable session state. > - Runner currently prepares private provider homes for each session. > - Operators can already have working login, skills, and MCP configuration on the execution machine. > - This PR adds an explicit mode that uses that existing setup. > - Runner keeps its task bridge and session ownership while leaving the user's configuration in place. ## Linked Issues or Issue Description **Problem or motivation:** Rebuilding a provider home is unnecessary when the execution account is already configured. **Proposed solution:** Add `homeMode: "preconfigured"` to Runner adapter config and runnerd transport options. Keep `"isolated"` as the default. Initially support native Codex and ACPX Claude. **Alternatives considered:** Continue copying user configuration into a private home, or add an approval workflow. Both add work outside this feature's scope. **Roadmap alignment:** This extends the existing cross-provider Runner. It does not add another runtime. Related work: #14056 covers managed Codex credentials. #14474 covers command environment delivery. This mode uses user-owned credentials and retains the existing launch boundaries. ## What Changed - Add a closed v7 launch contract for the selected home mode. Existing contracts retain their behavior. - Keep homeMode under operator control. Reject task overrides and agent-authenticated changes, including removal through replacements, adapter changes, or rollback. Ignore stored task values and exclude it from automatic hire inheritance. - Skip managed AI login, assigned-skill materialization, and external MCP setup in preconfigured mode. Preserve the Runner task/completion bridge. - Keep protocol state and provider lifetime fences scoped to the Runner session. Different sessions can share a home. - Fail preconfigured ACPX planning/read-only launches before provider admission until a read-only user-home launch is qualified. Use isolated mode or native Codex for planning. - Reject ACPX approval requests. Keep native Codex on `never`. Local permission rules and hooks remain effective. Initially require the default Runner ACPX `approve-all` setting; other Runner policies are unqualified in this mode and fail before launch. - Keep the internal ACPX bridge in a stable session namespace, preserving an existing user MCP server named `paperclip`. - Reject preconfigured ACPX Codex before launch until a session permission profile can enforce Runner-state denial. Remove the unprotected full-access mode switch; use native Codex or isolated mode for that model family. - Bump Pi to profile 23 because its attestation covers shared Runner source changed here. Preserve profile 22 for historical recovery, and update Docker/package verification inputs. - Fence recovery on mode and home bindings. Fail missing provider-history recovery without starting a replacement session. Exclude the external Codex home from backups and credential cleanup. - Discover remote HOME on the execution machine. Do not project controller HOME or XDG paths onto it. - Separate task instructions from Codex ACP isolation and update the generated provider digest. - Keep remote configured-environment manifests consistent when dropping controller paths. Preserve exact Claude model IDs through per-launch environment settings, without editing the user home or patching the provider. ## Verification - Targeted local checks cover native contracts and launch allowlists, user-config preservation, provider admission, permissions, session recovery, remote environment projection, heartbeat preparation, and operator-only configuration. Runner, shared, and server TypeScript checks and the Rust binary build pass; generated ACPX profiles are current. - Following the latest master integration, all 757 server configuration/executor tests and 115 Runner admission/recovery tests passed. Additional review-fix checks passed: 103 planning contract/host tests, 78 provider-backend tests, 95 route/preparation/inheritance tests, 43 task-validator tests, and 43 workspace-policy tests. Counts describe separate runs and can overlap. - The real pinned Claude ACP/SDK regression reproduces exact-model aliasing without the fix (`claude-sonnet-5-5` becomes `sonnet`) and passes with the fix. All 89 adapter tests pass. User settings remain byte-for-byte unchanged. - Localhost-only Claude SDK probes with synthetic credentials verify the private bridge runs, local `ask` reaches the existing reject callback, local `deny` blocks without a callback, and a user MCP server named `paperclip` remains available. No user model allowlist is required. - Review regressions cover fail-closed provider admission, saved-mode recovery before routing, controller-home path removal without configured-environment manifest drift, pre-launch rejection of unqualified read-only ACPX launches, and denial of task/agent changes to the operator-owned home mode. - Before rebase, `d4d4ee2aebe4cb88c4f4d9ef8e22fed4b36b87be` received standard Greptile 5/5 with no unresolved threads. Prior Apex findings are fixed and covered by regressions. No manual Greptile retrigger is requested for this rebase, per reviewer direction. - The prior PR head and master hit the same GitHub runner encoding crash in workspace-b on three PR attempts, including diagnostic logging ([prior PR CI](https://github.com/paperclipai/paperclip/actions/runs/38818474563/job/116714538405), [prior master CI](https://github.com/paperclipai/paperclip/actions/runs/38819528748/job/116712423193)). Rebased onto `origin/master` at `c3ffa3f0dd` to pull in its bridge-test index correction. The rebased head `edd153db410bea063b645e0000aa68649f287b00` passes all CI checks ([CI run](https://github.com/paperclipai/paperclip/actions/runs/38824676914)), including workspace-b. Greptile's automatic standard review is 5/5 on this exact head, with no unresolved threads; no manual review was requested. - Local workspace-b investigation passed 911 shared tests and 3,868 tests across the remaining 15 workspace projects. The catalog run separately hit an existing ignored `.agents/skills/design-guide/SKILL.md` description-budget violation in this checkout; that file is untracked, outside the PR, and left untouched. Repeating shared tests with CI reporting enabled also passed. - Rebase verification: the final Git tree exactly matches the previously reviewed branch combined with current master. All 192 Runner admission/contract tests, 61 server preparation/provider tests, and 68 bridge tests pass, as do Runner/server TypeScript and generated-profile checks. The real Claude adapter fixture hit an `ENOTEMPTY` cleanup race on its initial run; all 89 adapter tests passed on retry. - Apex on rebased head `edd153db` returned 2/5: ACPX Codex selected full access without enforcing Runner-state denial, and agent configuration replacement could remove the operator mode. Both are fixed in `26e14dbbf`. Preconfigured ACPX Codex is temporarily unqualified; re-enabling it requires session permission enforcement on thread/turn operations and a real filesystem-denial probe. The config guard covers effective replacements, adapter changes, and rollback while allowing ordinary agent merges and operator removal. All 190 focused Runner tests, 103 server tests, both typechecks, and generated-profile checks pass. Standard review and CI passed before the next Apex request. - CI on `26e14dbbf` passed 1,967 Runner-shard tests but hit the previously observed `ENOTEMPTY` fixture cleanup race after successful Claude model admission. `eb83284fb` adds bounded filesystem retries to that fixture's teardown; all 87 adapter tests pass. Assertions and production cleanup behavior are unchanged. Head `eb83284fb42ee19a15d9a93b106c1bbb07e1695e` passes all 59 CI checks ([CI run](https://github.com/paperclipai/paperclip/actions/runs/38829500857)) and standard Greptile at 5/5 with no unresolved threads. Apex also completed at **5/5** on that exact head ([Apex result](https://github.com/paperclipai/paperclip/pull/16486#issuecomment-6201559527)). All review threads are resolved, the branch has no merge conflicts, and the PR remains unmerged. - No live paid-provider or remote-machine smoke was performed. The lockfile remains unchanged under the repository's refresh-bot policy; local dependency installation resolved the changed patch hash. ## Risks This explicit opt-in grants Codex tools access to the execution home (read-only in planning), including credentials. Native Codex separately denies Runner-owned protocol state. ACPX Claude retains its existing provider trust boundary, and the isolated default is unchanged. Providers may refresh credentials and update their own metadata. User configuration can block tools or the completion reporter. Existing blocked results and failure diagnostics remain the recovery path. Edit the local settings and retry; no new approval workflow is added. External provider history remains on the execution machine and is not backed up by Runner. Remote HOME discovery requires `getent` and fails explicitly when unavailable. Other harnesses need separate qualification. The Codex ACP bundle digest changes. Existing sessions tied to the old bundle may need a fresh session. No database migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning and code execution. The exact runtime model ID was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fc3b15537b |
fix(runner): match call_api to existing agent REST permissions (#16040)
Route native call_api through existing agent REST permissions. Preserve admitted credentials across restarts, protect sensitive replies and private snapshots, and compare saved completion with provider history through the same redaction projection. Add permission, privacy and recovery regressions and update the access map and run-log guide. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
b5342febe5 |
fix(runner): require current-turn completion after connection continuations (#15514)
## Thinking Path > - Paperclip manages agents, tasks, permissions, and execution budgets. > - Native tasks can resume after a connection decision in the same provider conversation. > - Each turn still needs an accepted completion report. > - Compact continuation messages did not explain that reports from earlier turns cannot finish the new turn. > - The connection evaluator could also grade before the final reply was stored or reject valid unavailable-access wording. > - This PR clarifies the current-turn report requirement and fixes those observation boundaries. > - The original connection instructions and strict native completion gate stay in place. ## Linked Issues or Issue Description Refs #15489. The reduction remains draft while this separate repair is qualified. Refs #15471 for the earlier connection continuation work. ## What Changed - Add a current-turn completion reminder to compact continuation inputs. - Keep final prose insufficient for completion. Preserve permissions and retry policy. - Wait for the final successful task run's saved, attributed decline reply within the existing deadline. - Use one bounded explanation matcher for both decline checks. - Wait for a recorded tool-action rejection to dispatch its bound continuation, with strict company, task, agent and source-run checks. - Retain the exact grading input before later API refreshes. - Add failure and delay regressions and update the Runner and evaluator docs. ## Verification The fresh comparison has **15/15 original passes on each variant**: 15 unchanged pass pairs, zero new failures, and no pending pair. There are 30 case attempts and **65 actual agent runs** (baseline 33; candidate 32). All runs succeeded. All 30 cleanups passed. No model attempt was retried. | Profile | Baseline | Candidate | | --- | --- | --- | | Native Codex `gpt-5.6-sol` | 5/5 | 5/5 | | ACPX Claude `claude-sonnet-5` | 5/5 | 5/5 | | OpenCode `openrouter/deepseek/deepseek-v4-flash-0731` | 5/5 | 5/5 | Each profile covers service approval, service decline, connection decline, provider decline, and selection of the second provider. The saved replies, approved briefings, decisions, fixture observations, final task states, and native completion records were inspected. All 18 saved decline-grade snapshots match their original captured inputs and checks. Result, API snapshot, and final ledger run sets agree. - [Candidate campaign](https://github.com/paperclipai/paperclip/actions/runs/37711658378) · [public candidate report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711658378-1/index.html) - [Baseline campaign](https://github.com/paperclipai/paperclip/actions/runs/37711675579) · [public baseline report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711675579-1/index.html) - Candidate source: `79905343bba280d462765faad19a26e7f179259e`. Baseline source: `7c5e120158f1385a1fc5f20be41f66c58a605534`. Both use master context `fc6304dfe5f2e446e09bd052a7b45f51e930f250`. The trusted workflow source is separately frozen at `dd777f4b7343305c4e6f44c422f44a1d78e12e4f`. - Both variants use the same evaluator, fixtures, models, permissions, 720-second cell deadline, and 1,000-cent company and agent hard stops. The only production difference is the compact continuation reminder. - Suite hash: `aee30b74b4d38ada08777798db0932fbc64e368bb427d29caa8cfd86c7f59747`. Definition hash: `ea9e17f54fe0af1acbb2337ddfab8a7e92db8488bd0f59510a63095ca0229b60`. - Provider-free transport capture: startup/resume input stays at 55,726 bytes. Compact continuation input grows from 53,334 to 53,535 bytes. The 42-tool catalog stays unchanged. These are Paperclip input bytes, not complete vendor prompt tokens. - Focused evaluator tests: 31 pass. Native contract, transport delivery, and session tests: 194 pass. Evaluator support: 1,819 TypeScript tests and 128 Node tests pass, with one intentional skip. - Full build, workspace typecheck, and evaluator typecheck pass. Current-head CI passes all required gates. The current rollup has 51 successful check runs, two intentional Storybook skips, and a successful Snyk status. Review is 5/5 with zero unresolved threads. - CI attempt 1 had one initial runtime-fixture health timeout. The exact test and its full 164-test file pass locally. One CI shard retry passed. The original CI failure, its dependent verify failure, and the retry remain visible in [CI history](https://github.com/paperclipai/paperclip/actions/runs/37711235060). - The broad local `pnpm test:run` attempt was interrupted after about 49 minutes (exit 130). It recorded one failure in the unchanged Zep memory-connector disabled-setting test. That test and the full 388-test tool-access file pass in separate local checks; current-head CI also passes. The local cause is not established, and this broad local attempt is **not** claimed as passing. Three earlier local failures also pass in their isolated checks; their original logs remain retained. - The first two setup admissions were cancelled before provider jobs to include the review correction. They made no provider calls. The completed campaigns above are the first and only model attempts for these corrected variants. Cost evidence stays separate from behavior. Original result summaries report only OpenCode amounts: baseline $0.039192096 and candidate $0.063221620. Final run ledgers also retain estimates for Claude (baseline $1.232439000; candidate $1.333611200) and Codex (baseline $2.340324400; candidate $2.129335600). These estimates do not replace the original summaries. Local and GitHub runtime are unmetered here. Invoices are unknown. This is not a cheaper or faster claim. ## Risks - A single matched trial cannot prove general equivalence or causation. The reminder is an instruction change, not a new completion enforcement rule. - Candidate OpenCode service-decline finished within one continuous run; its baseline used two. That pair passed the task outcome, but it does not qualify the reminder on a resumed decline turn. No extra paid run was used to replace it. - The text matcher is bounded evidence of an explanation. It does not prove reasoning or consumption of feedback. Bounded stdout excerpts do not prove that every extra attempted tool call is absent. - Missing saved replies or continuations still fail at the original deadline. Failed native completion remains a failure even when final prose is correct. - The unresolved local-suite discrepancy above remains a validation limit. Full remote CI and both focused local reproductions pass. - The original connection instructions stay in place. These results do not qualify the reduction in #15489. Its original 11/15 versus 12/15 grades and two new failing pairs remain unchanged. ## Model Used OpenAI Codex, based on GPT-6. The exact deployment ID and context window are not exposed in this session. Capabilities used: reasoning, repository editing, code execution, test inspection and eval analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused local tests listed above and they pass; the interrupted broad local run is disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fd8c6b920a |
fix(native): resume connection tasks after approval decisions (#15471)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native tasks can pause while a human decides whether to allow a connection action. > - The next turn needs both the saved decision and complete accounting for the previous turn. > - Cancellation could discard final usage, and complete direct Claude API receipts could remain unpriced. > - A stale blocked or review report could also request approval again after the original card was declined. > - This pull request retains shutdown accounting and rejects approval waits bound to an already resolved action. > - The benefit is reliable continuation with the existing budget and approval controls. ## Linked Issues or Issue Description Refs #15420. Related: #15312 addresses requester ownership during dispatch. This change addresses receipt capture and final-response validation. ## What Changed - Retain usage events after native cancellation. Continue to reject late provider messages and work. - Drain same-turn accounting and terminal events for at most fifteen seconds after a durable governed wait. Keep incomplete accounting blocked. - Estimate complete, unpriced, direct Anthropic API receipts for the exact `claude-sonnet-5` model. Record the rate version and assumptions. Use the one-hour cache-write rate when the receipt lacks cache TTL. - Bind stale approval reports to exact interaction, action-request, or invocation IDs in the same company, task, agent, and run. Cover blocked, review, and response-wake reports. Keep independent reviews valid. - Fence checkpoint and result writes after a controller detaches for restart, including operations waiting for a database lock. Reject stale successful returns before certifying accounting. - Allow bounded subscription teardown only after retaining an actual provider terminal. - Journal the exact governed-wait trigger and disposition before provider interruption. Recover that wait independently of a later saved answer, replay retained accounting, and reject mismatched or unproven terminal evidence. - Give settling governed turns a bounded window before shutdown detaches their controller. - Keep fuzzy external app matches alongside installed capability matches instead of forcing an unrelated provider question for a generic query. - Return up to twenty exact active catalog tool names after an invalid request, after eligibility checks; still reject the request without granting access or creating an approval. - Require retained provider terminal proof before settling a governed wait, including when complete usage arrives before stream closure/error/timeout. Retain harmless numbered cancellation events so restart replay stays contiguous. - Isolate accounting-test OpenCode config from the host plugin directory. - Add regression coverage and document the accounting, restart and connection-search behavior. ## Verification Current PR source: `0cf08efd75f8fb23f7989beda6dbda92587088bf`. The live matrix below measured frozen `09a776bcb1f77422156f8be11e13f8c29f43e7f7`; later review fixes are verified separately and do not relabel those runs. - Repository typecheck and build pass. - Current runner runtime and cancellation suites: 191 tests pass. Six new regressions cover stream end/error/timeout without provider-stop proof and contiguous cancellation acknowledgement/request replay; all six failed before the fix. The existing bounded cleanup case now explicitly supplies terminal proof. All 31 adapter accounting tests pass with isolated fixture config. - New checkpoint-rebinding and approval-criterion suites: 65 tests pass; runner HTTP integration: 29 tests pass. Unchanged executor/control-plane suites: 640 tests pass; database-backed connection suites: 68 tests pass. - Current retained evidence verification covers 222 file hashes across all fifteen original result artifacts. The three-case [published report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37651896525-1/index.html) and all eight screenshot hashes verify. The [three-case recovery report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37662587356-1/index.html) and all six screenshots also verify; the nine-result campaign did not publish. - Evaluation support: 1,805 Vitest tests pass, one skipped; 128 Node tests pass. Eval typecheck and catalog discovery pass. - Full local repository unit run was interrupted before the follow-up edits after three tool-access failures and one runner HTTP failure. Those failures pass in isolation; the 09a full run was interrupted after one rapid Slack callback-ordering failure and seven skill-service failures. All eight pass both isolated and with full-runner environment settings, and all 74 skill-service tests pass together; the subsequent full run reported two 15-second OpenCode accounting timeouts and was stopped with exit 130 to apply review fixes. The timeouts reproduce while copying this host’s 61 MB OpenCode config. All 31 tests pass after isolating config inside each fixture without increasing timeouts or changing assertions. A complete local full-suite pass is not claimed. Repository-wide CI also passes on the final review-fix head in [run 37669185952](https://github.com/paperclipai/paperclip/actions/runs/37669185952). Earlier database-skipped diagnostics and the older ENFILE run are retained and are not full-suite passing evidence. - Three-case live campaign [37651896525](https://github.com/paperclipai/paperclip/actions/runs/37651896525) passes all three original grades on frozen source 09a: 55/55 checks, eight succeeded run records, complete accounting receipts, matching checkpoint identities and no pending approvals. Campaign [37653533353](https://github.com/paperclipai/paperclip/actions/runs/37653533353) adds nine original passes (173/173 checks, eighteen succeeded records) on the identical source. Its other three jobs failed before runner assignment or any step while GitHub could not load the paid environment; those original infrastructure failures are retained. Campaign [37662587356](https://github.com/paperclipai/paperclip/actions/runs/37662587356) completes only those unstarted cells: all three original grades pass (57/57 checks, six succeeded run records). All fifteen exact cases now pass on source 09a: eight FAIL → PASS, seven PASS → PASS, zero new overall failures and zero pending pairs. Total current evidence: 285/285 checks and thirty-two succeeded run records, complete accounting receipts, matching checkpoint identities, no pending approvals or retry records. Earlier campaigns retain forty-seven additional run records and two known same-run recovery attempts; actual provider-call counts and invoices remain unknown. The nine-result campaign skipped publication and its public URL returns 403; original artifacts remain retained. Previous ba9 campaign [37645656840](https://github.com/paperclipai/paperclip/actions/runs/37645656840) completed 2 PASS / 1 FAIL: Claude restart/approval and Codex decline pass, while OpenCode resumes but times out searching for exact tool names and never creates the access card. That failure and incomplete cancelled-run accounting remain preserved; the new catalog error guidance targets this observed dead end. Campaign [37642957312](https://github.com/paperclipai/paperclip/actions/runs/37642957312) remains 0 PASS / 3 FAIL and exposed the now-corrected cross-run marker leak and unknown-criterion approval gap. The original baseline remains 7 PASS / 8 FAIL, first repair 2 PASS / 3 FAIL, and second repair 0 PASS / 3 FAIL. All fifteen selected cases are qualified by their original grades in this bounded trial. These live grades belong to 09a. Its twelve governed-wait checkpoints retain matching same-turn terminal fingerprints, but passing artifacts omit detailed event journals; the later six adversarial regressions qualify the new terminal-proof and replay guards separately. Final-head repository CI passes. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/15471#issuecomment-6044369780) is 5/5, confirms both findings are fixed, and reports no new actionable issues. All review threads are resolved and the PR has no merge conflicts. ## Risks - Governed cancellation drains accounting for up to fifteen seconds. Restart detachment gives a settling batch up to twenty seconds to finish. An incomplete receipt or unproven provider terminal still prevents successful qualification. - Claude prices are estimates, not invoices. The estimate assumes standard global API pricing and uses a conservative cache-write rate. Unsupported models, billers, and billing modes remain unpriced. - Approval identity matching must remain scoped to the current run and the requested approval. It does not authorize execution of a declined call. - Original baseline and final candidate have different merged master context. Exact-case outcomes are before/after observations, not isolated causal attribution to this repair. - No schema migration, fixture, oracle or grader change. Search-result guidance now treats fuzzy external matches as suggestions. Existing app authorization and provider-consent checks remain required. ## Model Used - OpenAI GPT-6 through Codex, with code editing, terminal tools, and test execution. The exact deployment identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (191 current runtime tests and 31 accounting tests; interrupted full-suite history and CI coverage are disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
63f3aa2dbf |
fix(runner): continue restart-interrupted Codex turns (#15297)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs retain their provider conversation across server restarts. > - A dead runner can restore a Codex conversation after its active turn is lost. > - The old recovery path synthesized a failed task result from that interruption. > - The task then required operator action even though its conversation and workspace were available. > - This pull request preserves the interruption cause and uses the admitted restart attempt for one continuation in the same conversation. > - The agent can reconcile unfinished actions and complete the current request without resending the original task. ## Linked Issues or Issue Description Related: #12845 added native restart recovery. #15042 covers admission during shutdown. #14796 covers legacy shutdown recovery. This change covers a lost native Codex turn after successful conversation restoration. **What happened?** After a server restart killed the local runner, Paperclip restored the saved Codex thread. Runnerd found that the old turn was no longer active. It synthesized a failed terminal and a needs-review result from the last progress message. The transport also discarded the terminal error when it reconstructed thread history. The task failed instead of continuing. **Expected behavior** After proving that the old process stopped and admitting a bounded recovery attempt, resume the current request in the same conversation. Preserve the workspace. Inspect unfinished actions before proceeding. Keep real provider failures, accepted results, intentional stops, unknown unreconciled effects, and exhausted attempts subject to their existing rules. **Steps to reproduce** 1. Start a local native Codex run and leave its turn active. 2. Kill the isolated runner and provider processes, as can happen during a server restart. 3. Restore the same provider thread with no active turn. 4. Observe the synthetic task failure. The new real-process regression reproduces this boundary with a scripted provider. ## What Changed - Record an explicit recoverable process-loss cause without inventing a task result. - Preserve terminal errors and prior turns in reconstructed provider history. Recover the authoritative saved result when adopting an accepted continuation. - Send one continuation in the same conversation for an admitted dead-runner recovery. Require reconciliation of unfinished commands and external actions. - Persist the interrupted terminal before submission and retain the existing recovery marker across another controller loss. - Keep provider attempt limits, terminal failures, and intentional cancellation behavior. - Add red/green regressions, real process-kill coverage, restart checkpoint coverage, and retry-budget coverage. Document the behavior and run-log evidence. ## Verification - Red: the new native runtime regression rejected with `NativeProviderTerminalFailure` on the original code; the Rust restore regression found a missing recovery cause. - Red/green: if restoring the conversation fails and replacement is allowed, the replacement receives the full task and fresh-session handoff. Both prepared and legacy execution inputs are covered. - Red: a second controller crash after the provider accepted the continuation caused an extra `turn/start`. The regression now proves there are exactly two submissions total: the original and its continuation. - Green: focused runtime, backend, driver recovery, and real-process restart suites (207 tests). After the final history/result changes, driver recovery and real-process restart suites passed again (36 tests). - Green: complete Codex transport suite (186 tests), server restart classification/database integration suites (34 tests), and Rust Codex provider suite (92 passed, 2 ignored). - Full `pnpm -r typecheck` and `pnpm build` passed on `60141e649`. The subsequent replacement-prompt guard passed the Runner TypeScript check and the complete runtime plus process-restart suites (148 tests). - Local full-suite attempt: `pnpm test:run` reported two failures in the untouched chat integration suite. Both passed individually, and the complete chat suite passed on rerun (1,063 tests). After all remote test shards passed, the duplicate serial local run was stopped with SIGINT; it is not claimed as a full local-suite pass. - Latest-head CI (`b1297dcd4`): 55 successful checks and 4 intentionally skipped checks, including all test shards, typecheck, build, native Runner verification, end-to-end tests, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37405346303). - Greptile: 5/5 on the latest head, with no unresolved review threads. The PR is mergeable. - The process tests use the real runner binary and a scripted Codex provider. They do not call a live model service. ## Risks - This changes local Codex recovery after process loss. A continuation can execute more work in the retained conversation. Its prompt requires state inspection before repeating an uncertain action; the system does not replay tool calls. - Recovery shares the existing three-attempt budget and one-shot continuation marker. Real failures and older unmarked failed checkpoints are not reopened. - No database migration or API change is required. ## Model Used - OpenAI Codex, based on GPT-6. The exact model ID and context-window size are not exposed in this session. - Capabilities: reasoning, source inspection, tool use, code editing, code execution, and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2ec82c5774 |
fix(runner): preserve task context when tool connections change (#14963)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use tools through company-scoped connections and provider sessions. > - Resolving a tool connection currently forces a fresh session even when the provider can load new tools into the existing conversation. > - A fresh provider conversation can receive too little history to continue the task. > - This pull request adds explicit tool-refresh capabilities and uses them in both runner paths. > - Fresh attempts receive bounded task history with source IDs and retrieval instructions. > - The benefit is that agents can continue the same task after a connection changes. ## Linked Issues or Issue Description **What happened?** A resolved tool connection forced a fresh provider conversation. The new conversation could lose the original goal and prior answers. Claude also rejected resume when only the MCP server set changed. **Expected behavior** Resume the provider conversation when its harness can refresh tools. When a fresh session is required, supply enough bounded history to continue the task. Preserve company, agent, task, workspace, model, instruction, and skill checks. **Steps to reproduce** 1. Start a conversation and agree on a task and its constraints. 2. Request and connect a tool needed for the task. 3. Continue the conversation after the connection resolves. 4. Check that the agent remembers the task and can use the new tool. **Paperclip version or commit** The bug was reproduced on master at `c46e41e81`. This branch is rebased on current master. **Deployment mode** Self-hosted server. Both legacy adapters and the native runner are affected. Related public work: Refs #13282 for task-backed conversations. Refs #13057 for the broader session-compaction proposal. Refs #14659 for another report about local CLI session continuity. This change fixes tool-connection continuation. Provider authentication repairs keep their existing recovery behavior. ## What Changed - Expose tool-refresh support in native harness descriptors and legacy adapter metadata. - Request tool refresh after connection resolution. Keep provider authentication repair as a fresh-session wake. - Reload current tools and credentials while retaining supported Claude, Codex, Grok, and other provider conversations. - Allow MCP-only changes during qualified native recovery. Keep all other compatibility checks. - Refresh managed-provider and ACPX tool bindings when attaching a new run. - Add a fresh-session handoff for both runner paths. Bound database reads, excerpts, and the final packet to 24,000 bytes. - Include the original request, recent messages, decisions, plans, prior answers, and source IDs. Mark omitted content. Apply reset boundaries, wake cutoffs, quarantine, and secret redaction. - Add regression tests and document the capabilities and handoff behavior. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. Rust formatting passed. - Final review fixes passed 314 server tests, 333 adapter utility tests, 139 native-session runtime tests, and 12 managed-provider Rust tests. They verify historical quarantine, raised budgets across attachment, no history reads on successful resume, and handoff delivery on fresh retry. - Broader branch verification also passed 1,401 adapter utility tests, 1,047 runner TypeScript tests, 43 Grok adapter tests, and 311 Rust core tests. - Live Claude CLI and Grok ACP probes preserved the provider session ID, recalled a prior task constraint, and called a newly added read-only MCP tool. - GitHub CI passed on `b21486d18084a7aa4cafbe8e012f7cad6585d9cc`: 55 successful checks and 4 skipped checks. This includes all test shards, all eight browser shards, runner checks, and the Grok clean public npm install canary. [CI run](https://github.com/paperclipai/paperclip/actions/runs/37057514976). - A full local test attempt encountered a separate Git snapshot timeout. All affected local suites passed after the final edits, and the full CI test gates passed. - Review the capability matrix in `packages/paperclip-runner/README.md`. Repeat the four reproduction steps with a supported provider and with an unsupported harness. ## Risks - Provider tool refresh can fail. Existing recovery falls back to a fresh conversation where policy permits it. - A new transport can replace an old process while preserving the provider conversation. Tests cover current credentials and unchanged identity. - Long history can omit older context. Explicit markers and source IDs let the agent retrieve needed context within task scope. - Unknown and unqualified harnesses use the fresh-session path. No database migration is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and live provider testing. The exact model ID and context-window size are not exposed in this session. Claude and Grok also ran as test subjects. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
94e8dec56b |
fix(runner): preserve tool outcomes through shutdown and restart (#14734)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runner sends authorized tool calls to the server and saves their results. > - A provider turn can stop while a server write is still running. > - The old shutdown path invented a failed result that could conflict with the real result. > - Truncated execution input and incomplete recovery records made the failure harder to diagnose. > - This pull request preserves exact inputs and actual outcomes through shutdown and restart. > - Tests force the race and crash boundaries so safe retries do not repeat writes. ## Linked Issues or Issue Description **What happened?** Stopping a turn during a server tool call could record a false failure, then reject the actual result as a conflict. The diagnostic input formatter could truncate instruction content before execution. A crash during saved-result delivery could leave that delivery permanently indeterminate. Cleanup could hide the first failure, and a retry could overwrite earlier run logs. **Expected behavior** Keep dispatched tools pending until their actual result is known. Preserve accepted input bytes. Accept identical result delivery without failing the task. Reject conflicting results with enough evidence to diagnose them. Recover saved-result delivery without repeating the business operation. **Steps to reproduce** 1. Hold an instruction update at the filesystem commit barrier. 2. Stop its provider turn before the server returns the result. 3. Release the write, deliver its result, and replay the same result. 4. Repeat with a restart before and after the delivery receipt is saved. 5. Check that there is one write and one audit row, and that the exact result survives. **Paperclip version or commit** The change was developed from `44736c9c7` and rebased onto `0e5830887`. **Deployment mode** Self-hosted server with the native runner. Tests use local runner processes, scripted providers, and PostgreSQL. Related work: #12353 added durable semantic tool receipts; #12384 added durable Codex tool recovery; #12404 bound semantic tools to ACPX sessions. #14633 covers separate native-provider cancellation and qualification work. This PR addresses server semantic-tool outcomes and their durable delivery. No duplicate fix was found. AgentMail discovery is outside this PR. ## What Changed - Close turn admission without inventing results for dispatched tools. Keep pending calls and accept late actual results. - Accept identical result replay with a diagnostic warning. Include call identity and both result hashes in real conflict errors. - Preserve exact execution arguments. Reject prohibited or oversized input before dispatch. Keep diagnostic previews redacted and bounded. - Commit instruction-attempt evidence before the filesystem write. Save completed mutation receipts so concurrent and restarted duplicates return the first result. Recheck authorization before replay. An attempt without a completed result stays unknown and cannot execute again. Definite pre-write failures save and replay their original error without another write. - Recover an interrupted saved-result delivery only for backends with durable result receipts. Never replay an ordinary business operation with an unknown outcome. - Preserve the initiating error when cleanup also fails. Record incomplete settlement evidence. Propagate typed unknown-outcome errors through the native tool wrapper without creating a false completed tool result. - Append run-log attempts and restore the durable log before appending after local file loss. Reject incomplete restores. Publish a restored prefix only if the destination is absent so concurrent attempts cannot overwrite new lines. - Add deterministic race, crash, replay, authorization, exact-content, and log-restoration tests. Document their assertions in `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Current head: `7e088f4c7fba8ebabf98ae95485a5753b013d489`. All 55 applicable checks pass; four conditional/manual checks are skipped. This includes build, typecheck, Rust, both runner TypeScript shards, server and workspace tests, all eight browser shards, isolated runner compilation, and the clean-install release dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/36746101110). - Greptile reviewed this exact head at 5/5 with zero new findings. All three earlier review threads are resolved. - Focused local verification includes 11 instruction integration tests, 23 surrounding authority/tool tests, 26 run-log tests, and 169 controller/driver tests. The post-rebase controller/transport/runtime selection passed 415 tests. The full Rust release suite passed 617 tests with two ignored. The real-process SIGKILL recovery test passed three consecutive runs. - The fault matrix in `packages/paperclip-runner/docs/durable-recovery.md` uses explicit barriers, real PostgreSQL rollback, durable journal reloads, and killed runner processes. It covers late results, identical and conflicting replay, exact long content, concurrent log restoration, lost commit acknowledgements, and definite failure replay after the original CAS base becomes valid again. No paid model calls are needed. - Full local recursive typecheck and build passed during implementation. Server typecheck and the runner TypeScript build passed after the review fixes. The broad local repository test run was stopped after repeated database startup timeouts. Four timing/launch failures in an earlier broad runner run passed focused reruns without changed assertions or timeouts. These are local verification limitations; the complete current-head CI suite is green. An earlier CI workspace job received an infrastructure shutdown signal; its current-head replacement passed. ## Risks - A stopped turn can remain blocked when a dispatched operation has no proven result. The system does not guess its outcome or rerun its effect. - Conflicting results still fail settlement. Existing failed or conflicting journals are not repaired automatically. - Accepted semantic input is limited to 480 KiB of encoded JSON to fit the encrypted transport. Larger input fails before execution. - Instruction filesystem writes and database receipts are not one atomic storage operation. A separately committed attempt and audit record survive rollback. An attempt without a completed success or definite pre-write failure receipt remains blocked as an unknown outcome. It is not replayed or reported as success. - Run-log restoration now reads the durable object before appending when the local log is missing. Failed or incomplete reads reject the append. - No schema migration, dependency change, workflow change, or AgentMail change is included. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and test analysis. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; the broad local run limitation is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3ca196b0a6 |
feat(agents): persist agent files across tasks without revision history (#14420)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent needs personal files across tasks and sessions. > - AGENTS.md is one file in that directory. Supporting files need the same persistence. > - The Instructions Editor and agent runs must share one current directory. > - Concurrent runs should apply only the files they change. The last sync of the same file wins. > - This pull request uses existing file transport and removes temporary copies after sync. > - Old instruction-only sessions keep their restore contract. New saves do not create revision history. ## Linked Issues or Issue Description Refs #14325. This replaces its revision-oriented design with persistent agent files. Keep #14325 unmerged. Transport prerequisite #14416 merged first at `d172197117a14b80a1eb2d2835a0e7cce2679656`. This PR now targets master and remains below 100 changed files. Related work: #4513 and #8798 cover instruction tooling. This change handles run synchronization, cross-task personal files, browser editing, and old-session restoration. ## What Changed - Keep one current directory per company and agent. Point AGENT_HOME at a temporary working copy for each active run. Keep task files and provider HOME separate. - Restore text, binary files, and nested folders through workspace transport. Exclude remote agent files from task Git snapshots with a self-ignoring file inside the reserved runtime directory; never write through repository-controlled Git metadata. - Collect after the provider and child processes have stopped. Keep resumable conversation state. - Apply changed and deleted files under the agent lock. The last sync wins for the same file. Unrelated concurrent changes survive. - Remove temporary copies after successful sync, rejected sync, and staging failure. Register ownership before copying so restart recovery can remove interrupted preparation. Retry transient synchronization up to three times. Preserve the original remote lease reference until deletion succeeds; restart cleanup never acquires a replacement sandbox. Do not create captured directories or a conflict-review queue for new runs. - Keep browser editing, stale-draft protection, and streaming binary downloads. Keep the instruction entry and text editor limited to 1 MiB. - Keep historical agent-folder sync failures on their affected runs instead of repeating them above current saved instructions. Preserve legacy candidate review and current browser-save errors. Avoid duplicate quota warnings while retaining separate sync failures when they describe a different problem. - Require target-scoped caller grants for peer instruction access, while preserving self edits, responsible-user checks, and protected-change consent. - Treat full storage as a nonblocking run warning, never an agent pause or run-admission failure. Restore already-over-quota saved folders so ordinary agent cleanup can recover; warn on each run until cleanup. The run detail view shows the warning. - Allow 256 MiB per file, 2 GiB per directory, and 100,000 entries. Hash large files as streams. Check editor-save quotas with metadata instead of hashing unrelated files. - Preserve old native inputs, instruction-only copies, paths, digests, and pending legacy candidates. Adopt old revision heads once. New writes do not append history rows. - Add idempotent migration 0287 and verify upgrades from the preview tables and receipts. - Add nine interactive stories under **Agents / Persistent files**, including automatic incoming edits, stale browser drafts, and storage-limit diagnostics. ## Verification - Merge candidate: `4f5390107ec6ffd80a76d1d2e85530e66f21d079`, after merging current master and the landed transport prerequisite. Integration required no manual conflict resolution; the feature remains 99 changed files. Full workspace typecheck, production build, token gates, and 715 focused tests passed on this merge candidate. Fresh Greptile review is 5/5 with no unresolved findings. All 55 checks passed, with four conditional skips, including the build, typecheck, browser E2E, and canary dry run. A single retry recovered four jobs interrupted by runner shutdowns; no source changes were required. - Historical-warning UI fix: all 6,834 UI tests across 640 files passed, including regression coverage for three old failures, legacy preserved edits, and warnings scoped to the affected run. Full workspace typecheck, production build, Storybook build, and token gates passed. Browser-verified Storybook playtests passed for Historical Failures After Successful Save, Storage Limit, and Full Storage Run Warning. - Review follow-ups at `4e20c9fb2`: all 18 focused tests passed, including external Git directories, linked worktrees, symlinks, hardlinks, and distinct I/O failures alongside storage warnings. Server and UI typechecks, token gates, and the production build passed. - Storage warning regressions at `0724f3012`: all 33 directory tests and all five heartbeat-list tests passed, with no skips in their successful runs. They cover repeated runs while full, an already-over-quota saved folder, cleanup, warnings retained after unrelated save failures, and bounded warnings in large result JSON. Server typecheck passed after the final warning fixes. - Full workspace typecheck, production build, and token gates passed during this follow-up. Product E2E harness: 631 tests passed across 52 files; harness typecheck passed. Earlier native session/context and directory/legacy collection suites passed 537 tests; Runner unit/transport suites passed 329 tests. - **Real E2E at `0724f3012` (before this follow-up):** legacy local Codex and native Daytona Codex each passed six tasks, one server restart, seven independent assertions, and cleanup verification. Both prove browser-to-agent edits, agent-to-browser edits, nested/binary restoration, per-file last-sync-wins, a successful run after an oversized save rejection, and cleanup clearing the warning. - Native local Codex also passed the six-task quota flow before the final warning-retention fixes. That pass began at `918d1ed02` while the bounded-result warning fix was being edited, so it is not claimed as exact-final-head evidence. Its final-head rerun failed during embedded PostgreSQL bootstrap before any provider run: the macOS host had 87,365 of 87,381 SysV semaphores occupied. No unrelated services or kernel limits were changed. - The final-source report intentionally records **2/3 cells passed**, preserving the blocked native-local attempt: `tests/runner-e2e/results/agent-files-quota-final-20260928-report/`. Earlier failed attempts and provenance notes remain under `tests/runner-e2e/results/agent-files-quota-final-20260928-input/` and the original campaign directories. - Daytona used immutable image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:5643f0d801417cae3581833a1a3bc6715b325e028602738d2652c44cac5dc6bf` and its exact Linux runner binary. Controller source is `0724f3012`; image source is recorded separately. - Legacy-session compatibility and all three ACP Stop/resume browser regressions passed on the prior validated feature head `169fab46d5af21caa2269b4c1b29b69c933a6951`. They assert the same provider session is retained and interrupted writes are not replayed. Migration upgrade tests also passed earlier. - Nine interactive stories are under **Agents / Persistent files**, including **Full Storage Run Warning**. Its playtest and visual browser inspection passed; the warning states that runs continue and the editor remains available. - Prior-head checks on `4e20c9fb2`: 55 passed, two conditional jobs skipped, no failures or pending checks. All eight browser E2E shards and their aggregate passed. Fresh Greptile review is 5/5 with no findings; all review threads are resolved, the security scan passed, and GitHub reports no merge conflicts. - The broad local follow-up test run was interrupted after host semaphore exhaustion affected isolated PostgreSQL instances. It also encountered the existing macOS long-path fixture failure and two timeout failures. This is not a claim that the full local suite passed. Logs are retained; focused storage/warning tests passed. ## Risks - A later sync can overwrite an earlier edit to the same file, including a saved browser edit. There is no text merge or retained version. This is the intended last-sync-wins policy. - A save that exceeds a storage limit is rejected and its temporary copy is discarded. The run itself continues normally, and later runs restore the last saved files with a warning until cleanup. Transient sync failures get bounded retries. An I/O failure partway through a sync can leave some files updated; a failed receipt does not claim whole-folder success. - Larger folders increase copy time, network traffic, and temporary disk usage. Active runs still need working copies. Terminal runs do not accumulate archives. Operators must provision disk for agents and configured concurrency; these limits are not company-wide quotas. - A restored old native session remains instruction-only until a fresh session starts. Its original conflict fence and existing pending candidates remain compatible. - Provider processes close at the collection boundary. Conversation resume remains available, but warm process reuse is lost. - Backups must include the instance filesystem and database. External bundles keep their existing behavior until explicitly moved to managed storage. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific model ID or context-window size. Reasoning, code execution, and browser tools assisted this change. Real provider E2E uses `gpt-5.6-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Fry (Paperclip) <noreply@paperclip.ing> |
||
|
|
992f720262 |
fix: make runner task context ownership explicit (#13753)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100). --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task descriptions, comments, continuation data, skills, and execution rules enter several agent adapters. > - The same source can be rendered by more than one automatic input carrier. > - Failed resumes can also rebuild input from stale or compact context. > - This pull request gives each Paperclip-owned source one delivery owner and preserves the required transport boundaries. > - It adds deterministic adapter, interaction, runner, and browser tests for these boundaries. > - The benefit is more predictable context delivery with explicit evidence for later live qualification. ## Linked Issues or Issue Description Related: #13144 removes a duplicate environment payload and bounds wake lists. Related: #11360 addresses Hermes resume behavior. This pull request preserves compatible active-session formats while repairing context ownership and stale question creation. **What happened?** Task descriptions and comments could enter more than one automatic context block. Native transports could wrap a complete model input in a second task envelope. Some legacy and gateway adapters could omit the owned assignment on ordinary tasks or rebuild a failed resume with stale compact context. A continuation could also request a question after newer human comments had arrived. **Expected behavior** Each task or comment source has one automatic model-facing owner. Distinct comment IDs and repeated wording remain distinct. Fresh fallback attempts rebuild the required full context. A question request is rejected when newer queued human direction makes it stale. Harness access policy remains owned by execution configuration. **Steps to reproduce** 1. Build a task with a description and current comments. 2. Capture the actual adapter or runner input. 3. Compare source ownership and task-envelope nesting. 4. Queue a human comment before a continuation requests a question. 5. Trigger a failed resume and inspect the fresh retry input. 6. Run the focused adapter, interaction, runner, and browser checks. ## What Changed - Add shared prompt-section selection at the provider-attempt boundary. - Deliver owned assignment context through native, legacy CLI, ACP, gateway, cloud, Pi, Kimi, Grok, Gemini, OpenCode, Cursor, OpenClaw, and Hermes paths. - Rebuild full or compact context after resume recovery changes the attempt. Add native and Claude ACP tests of actual recovery requests. - Preserve custom templates, loaded instruction files, execution policies, and older active-session formats. - Record continuation source metadata and reject stale question creation under the issue-row lock. - Add explicit Product E2E context-integrity profiles, prerequisite gates, credential-isolation checks, and report fixtures. - Bypass service-worker forwarding for same-origin Vite development modules. A real Chromium test fails with resource exhaustion before the repair and passes after it. Production asset caching keeps its existing policy. - Add browser diagnostics and service-worker module-loading regressions. - Add an explicit zero-retry eval option. The default retry behavior remains unchanged. Each campaign records its effective policy. - Remove the model-facing working-directory sentence from four prompt builders. Existing workspace, sandbox, permission, and custom-template configuration remains unchanged. - Align the everyday workflow assertion with the current 47-entry catalog. Compared with current upstream master, the branch carries the context-ownership implementation and its tests, the explicit context-integrity catalog and evidence harness, and the focused browser regression checks. ## Verification **Merge assessment:** focused regression evidence supports merge. This is not full completion of the original broad qualification matrix. The maintainer has authorized merge after fresh verification of the master integration. - Current head: `bbd52f82114eabf09bc7b1a7e97d54a5b43bbc00`. This integrates current master `2f585ef26a1814fa209715242d1ca791b63e4c4e`. All 14 conflicts are resolved. Cancellation checks, workspace finalization, native Grok support, and both sets of tests are retained. - Current-head Greptile: **5/5**, with no blocking findings. The review names this exact commit. All **59 reported checks are terminal: 55 successful, 4 skipped, zero pending or failing**. This includes the full root general and serialized suites, separate runner checks, typecheck, build, canary, browser E2E, Docker, and security checks. The successful legacy security status is included in that total. - After integration: workspace typecheck and full build passed. Separate runner checks passed: **2,160 TypeScript tests (10 skipped), 582 Rust tests, and 39 preparation checks**. Other passing checks include 621 Product E2E harness units, 376 focused shared/adapter tests, 160 real-database/API tests, 86 Hermes tests, 18 browser-support checks, and Product E2E typechecking. The complete root suite passed in CI. The duplicate local monolithic root run was stopped after that CI result; it is not counted as a completed local pass. - New native recovery coverage retains full assignment, completion contract, and explicit skill selection after safe replacement, for old and prepared input formats. Full native session test file: **136/136 passed**. - New Claude ACP coverage captures actual fresh, resumed, and missing-session fallback requests. It verifies one assignment copy, comment order, identical text under distinct comment IDs, and full fallback context. Full file: **33/33 passed**. Both affected TypeScript checks passed. - Existing deterministic tests cover source revisions, approval and trust boundaries, completion validation, custom templates, compatible sessions, standalone driver wrapping, and maintained adapter transport requests. - Provider-free browser support: **17/17 passed** after the master merge. Service-worker unit tests: **33/33 passed**. The module-overload regression failed before the repair and passed after it in real Chromium. ### Fresh live comparisons The new batch ran exactly four Product E2E attempts. **All four passed on the first attempt; no retries.** Each has six terminal matchers plus the existing browser lifecycle and invariant checks. | Exact case ID | Control | Candidate | |---|---|---| | `core-compatibility.runner-codex.local.plan-revise-accept` | Passed | Passed | | `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` | Passed | Passed | The plan case checks a revised canonical plan and revision-bound approval before completion. The question case restarts the server before submitting the answer, then verifies the continuation completes. Control source is `dfa4e1bda8d50a1a01746603251a9128dbe9d0d6`. Candidate source is `79fcdb5dece501d28064ea9da306603881b46f0c`. They use identical frozen definitions and provider versions: Codex `0.156.0` with `gpt-5.6-sol`; ACPX `0.13.1` / Claude ACP `0.73.0` with `claude-sonnet-5`. The September 24 head added master browser recovery and test-only changes. The September 28 head also integrates newer master changes, including cancellation, workspace finalization, and native Grok. These are frozen-source live results, not exact-head live runs. The candidate received one description copy where the control initially received three. The submitted initial plan envelopes were 7,969 versus 19,097 characters. Question envelopes were 7,592 versus 18,919. These are structural measurements, not whole-provider token or dollar savings. ### Earlier evidence and failed attempts - The preceding fresh batch has four effective passing pairs: OpenCode comment continuation and assigned skill, native Codex comment continuation, and native Claude comment continuation. It retains **11 attempts: eight passed and three failed**. - Original failures remain recorded: missing local PostgreSQL library links before task creation; host-sleep cleanup after task/page checks passed; and a Claude **control** session-open rejection before a model turn. Setup was repaired identically on both worktrees. The permitted unchanged infrastructure retries passed. The underlying Claude provider startup error was not retained and remains unknown. - Older R2 retains **17 passes and one failure** across 18 attempts, including eight both-pass native/legacy Codex/Claude pairs. Its OpenCode blank-page failure led to the service-worker repair. R2 is historical evidence: master changed the native fixed prompt and removed duplicate wake environment data afterward. - The September 24 CI run initially failed one unrelated preview readiness test (`ECONNREFUSED` on its local fixture). Its test and production code match master. Isolated local verification passed **28 tests, 3 skipped**. One unchanged CI retry passed the full shard: **831 passed, 1 skipped**, including all **31 preview-exposure tests**. The aggregate CI gate passed afterward. The precise startup cause remains unknown; a port race is a hypothesis, not a proved cause. ### Limits The original wider profile/workflow matrix, repeated trials, and remote Daytona qualification are incomplete. These results support a focused merge recommendation, not statistical equivalence or universal harness qualification. Some usage receipts are missing in both variants, so no token or dollar savings are claimed. The $500 ceiling was preserved using conservative allowances; failed attempts and unknown charges remain in the ledger. Reproduce the focused additions with `pnpm exec vitest run packages/adapters/claude-local/src/server/acp.test.ts` and `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts`. Full checks use `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`, and the separate runner checks. Paid evals require the frozen definitions, profiles, and credentials; do not use `--all` as a substitute for the selected cases. ## Risks - Context placement changes can affect model behavior. Deterministic checks cover the selected paths, but live qualification remains incomplete. - The stale-question guard can reject a request when queued human comments arrived during the run. This is intended. - New stored inputs and model envelopes retain compatibility readers for older active sessions. - Custom templates may intentionally repeat content. - Removing a model-facing working-directory sentence does not change filesystem, command, sandbox, or permission configuration. - The worker bypass applies only to same-origin development module paths. Cache-policy tests preserve private-response handling and production asset caching. Mounted HTTP fixture changes remain test-only. - This PR does not claim measured token savings or statistical equivalence across every harness. ## Model Used OpenAI Codex, exact model gpt-6-astra, with repository tools and code execution. Bounded supporting work used gpt-5.6-luna and gpt-6-luna. The serving context-window size is not exposed in this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the required issue fields - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run the focused local checks and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect these changes - [x] I have considered and documented risks above - [x] All current-head Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups for the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
96bf004a79 |
fix: use persisted state for lifecycle continuation and retry budgets (#13888)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its control plane decides when a task can continue, wait, stop, or complete. > - Legacy continuation could change when an agent changed its wording without changing task state. > - Shared attempt counts also let repair and infrastructure retries affect each other's limits. > - This pull request uses persisted state and separate, bounded allowances for these decisions. > - If automatic repair stops, the task explains what happened and offers a guarded retry. > - Paired tests and real-provider evaluations verify that Stop, approvals, ownership, and spending limits remain authoritative. ## Linked Issues or Issue Description Related work: Refs #13761, Refs #11126, Refs #13610. These cover obsolete continuation dispatch and retry storms. Open and closed issues and PRs were searched for related lifecycle, continuation, and retry work. **What happened?** Legacy continuation depended on English wording and progress heuristics. Repair, failure retry, and productive continuation could consume shared counts. When bounded repair stopped, the task showed a technical recovery message without a clear next action. **Expected behavior** Persisted disposition and owned execution paths determine the next action. Missing disposition prompts bounded agent repair. Explicit work mode determines planning mode. Narrative changes and raw activity counts cannot replenish allowances. An exhausted repair shows a readable notice. An explicit retry checks current controls and preserves the assigned agent. **Steps to reproduce** Run `pnpm test:lifecycle-baseline`. The paired probes keep structured state constant while varying completion, planning, blocker, and progress prose. Run the explicit `lifecycle-baseline` and `continuation-accounting` Product E2E suites for real-provider coverage. In Storybook, open **Design previews / Recovery notice** to inspect the production component's normal, pending, acknowledged, unavailable, failure, and mobile states. ## What Changed - Hide the image attachment button, icon, and drop/paste hint in answer composers. Image paste and drop support remains available. - Merge current master and retain both browser regression sets. Use a production-stamped service worker in the offline recovery browser fixture. - Share one state-based legacy continuation decision across immediate, delayed, and recovered dispatch. Bind bounded repairs to their source run and episode. - Remove title and description wording from work-mode authority. Agents can still write requested plans in execution mode. - Persist separate failure-retry and productive-continuation counters. Disposition repair and resource waits cannot consume or reset those allowances. - Validate delayed repair identity, then recheck current gates before provider dispatch. Fence native startup cancellation. - Show **Agent needs attention**, a plain-language explanation, **Retry agent**, and expandable details in both task interfaces. Report request progress, acknowledgement, and errors inline. - Store typed recovery notice metadata. Recognize older active notices only through exact stored action and run IDs. Notice text never grants retry authority. - Use the existing recovery-action endpoint for retry. Recheck current action, status, owner, agent availability, dependencies, active runs, pending questions and confirmations, approvals, pause controls, and budget. Duplicate requests do not wake twice. - Add component, page, route, database, contract, and Storybook coverage. Keep the scenario inventory and executable evals here. Historical reports and snapshots live in the [commit-pinned paperclip-evals archive](https://github.com/paperclipai/paperclip-evals/blob/ce3e5afcd4a1184650f586a2b5b8be5874c66c8b/experiments/2026-09-lifecycle-authority/README.md). - Preserve unsaved project fields while the same project URL changes to its canonical alias. Do not reuse data across projects or companies. This separate fix addresses the repeated repository-editor browser failure without changing the browser test. - Keep the development service worker from intercepting Vite module reloads. Update the connection-intent browser fixture to record progress and completion through the agent API. ## Verification Merge preparation on September 25, commit `c1e8e4b7ddd9fbc4913ed55ce21b8e12906c2f97`: - Merged master `bd2030932` and resolved the browser test-list conflict by keeping both sets of regressions. - Deterministic lifecycle baseline: 1,090/1,090 assertions passed; no failures, skips, or missing selected evidence. Unit 423, runner 184, database integration 397, grading 86. - Browser support: 17/17 passed. The offline recovery test first failed with an unstamped development worker, then passed with the production stamp. Its assertions are unchanged. - Focused interaction UI and offline fallback tests: 19/19 passed. Verified the custom-answer composer in Storybook: no attachment controls or hint; entering an answer enables Next. - Recursive typecheck, production build, token gates, and diff checks passed. The worktree is clean. No new real-provider campaign was run. - Current CI and review: [Current PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36166011243): 55 successful checks and two optional Storybook skips. Greptile scored this exact commit 5/5. Hiding the question attachment controls is an intentional UI change; paste/drop remains available. Earlier recovery UI verification, commit `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: - Recursive typecheck, production build, token gates, and diff checks passed. - Focused UI coverage: 338 tests passed across six suites (336 before the interaction guard, with the two affected suites rerun at 149 passed after it). Covers both task interfaces, the real page mutation, pending/error acknowledgement, stale state, and unavailable controls. - Recovery database integration: 352 tests passed before the interaction guard. The complete recovery-action and mutation-route suites passed 181 tests after it. The two new pending question/confirmation regressions failed before the fix and passed afterward, including resolved-interaction controls. Shared validator suite: 31 passed. E2E catalog suites: 34 passed. - Browser inspection passed for light/dark themes, mobile layout, expandable details, pending retry, acknowledgement, failure, and disabled retry. Storybook renders the production component; its request is simulated. - The broad local run hit two chat callback-order wait failures and was stopped after all CI unit/database/runner shards passed. Both local failures passed when rerun without the competing full-suite process. - CI exposed a repeated project-repository draft-loss race during canonical redirects. A new unit regression failed before the fix; all nine project-page tests now pass, including controls for other projects and companies. Both unchanged repository browser tests passed against a fresh local server. UI typecheck, production UI build, and token gates passed after this fix. - [Earlier PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/36072486798) on `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: 55 successful checks, two optional Storybook skips, and no failed or pending checks. The repository browser shard passed with the production fix. Greptile is 5/5 on this exact commit with no unresolved review threads. The PR is mergeable. Historical, source-qualified lifecycle evidence: - Lifecycle baseline: 1,074 assertions. Native session coverage: 447 tests. Product E2E support: 515 tests. Browser support: 11 tests. Full earlier verification is retained in the archive. - [Real-provider campaign: 8/8 passed, zero retries](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35881382080-1/index.html), source `e88d210417280140b44a36449027290adcb1aeaa`. Evidence and cleanup checks passed. This includes deliberately exhausted repair cases that correctly remain blocked; it does not mean every task finished Done. This campaign predates the recovery UI change. - Archive migration verified all 16 original JSON files byte-for-byte and all 24 checksum entries. App tests do not need private archive access. [Archive PR #27](https://github.com/paperclipai/paperclip-evals/pull/27) is merged. ## Risks - Agents that omit durable disposition receive at most two repair attempts by default. Prose-only completion exposes missing state rather than silently changing scheduling. - A retry is an explicit board action. The server rechecks current controls. A successful response confirms the task returned to To do; it does not claim that the provider has already started. - Existing notice metadata remains valid. Only older active notices with matching structured evidence receive the new UI. Historical notices without that evidence keep their existing rendering. No schema migration is required. - Old run records require conservative retry accounting. Tests cover old counters, alternating retry lanes, restarts, and exhausted repairs. - Historical snapshots require private `paperclip-evals` access. The app index retains public campaign links. Live campaigns qualify specific sources and scenarios; no new real-provider campaign has run for the recovery UI commit. > This fixes existing lifecycle and recovery behavior and does not duplicate planned core work. ## Model Used OpenAI GPT-6 through Codex assisted implementation, reasoning, code execution, and review. The exact serving model ID and context window are not exposed in this task. Historical real-provider evaluations used Codex model `gpt-5.6-sol`, separately from the implementation assistant. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
43acbcc398 |
fix(runner): preserve sessions and complete question and approval continuations (#13655)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner connects task state to provider sessions. > - Follow-up turns must retain provider memory and carry new user direction. > - Lost session IDs caused repeated context and extra input tokens. > - Native question answers and approval races could leave valid work blocked. > - This pull request repairs those paths and adds regression coverage. > - Agents can continue accepted work without repeating the conversation or losing the user's answer. ## Linked Issues or Issue Description Refs #13574. That merged PR shortened continuation prompts and moved question instructions into tool documentation. This change preserves sessions and fixes failures exposed by broader testing. Related runtime work: #13408 and #13410. **What happened?** Native follow-up turns could lose the provider session ID. Completion guidance could replace the original task with its latest comment. Claude native questions could remain pending after the user answered. Approval during a running tool call could suspend the run before the tool response arrived. Onboarding and chat handoff instructions also caused repeated planning or missing plan documents. **Expected behavior** Reuse a valid provider session. Send only new events when that session already has the history. Preserve the task requirements and apply later user direction. Store the question answer and deliver it to the waiting run. Finish governed tool responses before suspending. Execute the accepted plan without asking for the same approval again. **Steps to reproduce** Run the continuation, local-session-integrity, first-task, and agent-chat suites with native Codex and Claude. Include provider-question-bridge, accept-while-running, and plan-handoff. **Paperclip version or commit** This branch is based on master |
||
|
|
422287eecd |
fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner connects task messages, provider execution, and task outcomes. > - First-time user tests exposed gaps in recovery, completion permissions, message delivery, and Stop behavior. > - These gaps left usable output hidden, completed work waiting for bookkeeping, or safe work unable to continue. > - This pull request fixes the shared lifecycle and receipt paths while preserving process ownership and action checks. > - Users can continue work with accurate task state and durable messages. ## Linked Issues or Issue Description **What happened?** A stopped local Codex execution could remain blocked even after its processes had stopped and its complete transcript proved that no external action needed replay. Claude under Conservative permissions could fail to call task completion tools. Recovery could reuse an assistant item ID and overwrite prior output. A delivered comment could remain marked uncertain after navigation. Stop could look like Pause or a new recovery incident. Workspace contention could look like cancellation. A direct reply reopening Done could enter a clarification loop. **Expected behavior** Recover automatically only with verified termination and complete action receipts. Preserve answers and messages. Keep task completion available under Conservative permissions without broad tool access. Show crashes as Blocked, actual human decisions as In Review, and ordinary workspace contention as waiting. Stop the current response and allow a new direction. **Steps to reproduce** 1. Create ordinary response tasks with local Codex and Claude Code, then send follow-up messages through the task composer. 2. Interrupt a disposable local Codex runner during text-only work. Verify automatic continuation and retained output. 3. Stop a response, send a new request, answer a clarification, and reopen completed work with another message. 4. Navigate or reload while a comment submission is pending. Confirm the exact persisted request receipt settles it without removing newer draft text. 5. Run two tasks in a shared Daytona workspace. Confirm waiting does not appear as failure. **Paperclip version or commit** Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`. Current integration base: `6cef9743c`. Both operator-interruption and workspace-waiting guards are preserved; native restart and legacy permission rules remain documented. **Deployment mode** An isolated source-built test-drive instance, with real local Codex and Claude Code providers and disposable Daytona environments. Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163. This PR addresses additional failures from ordinary task journeys, including controller restart handoff and repeated warm sandbox setup. Historical task status reconciliation is excluded. ## What Changed - Persist runner ownership immediately at spawn and resume an explicitly adopted runner even when the controller crashed before the first driver checkpoint. Detach the controller safely across graceful restarts, including session startup. Prevent an old finalizer from suspending or signaling an adopted runner. Checkpoint idle warm sessions before shutdown. Preserve the same run and queued follow-up messages. - Scope saved legacy queue successor checks to the queue owner while preserving ordinary task locks, operator identity, assignment gates, and exactly-once delivery. - Preserve managed Codex credential files when an old session is detached for restart; normal owned cleanup still copies refreshed auth back and removes the scoped copy. - Reuse the bound warm shared sandbox and fully verify an existing staged provider pack before using it. This avoids repeated uploads when the pack is already valid. - Add a narrow local Codex replacement path with stopped-process proof, a closed transcript inventory, exact completion receipts, and fresh-session lineage. Preserve no-replay holds when evidence is incomplete. Recovery may clear only the same run's recorded Blocked status version; manual re-blocking and dependency changes invalidate that receipt, while queued comments do not. Later blocks stop scheduled, queued, and final dispatch; queued/final checks re-read dependencies even when the task status stays In Progress. - Permit only task delivery and human-input tools through the isolated Claude runner's exact task bridge. - Scope assistant item identity to the provider turn and ignore only authority-free Codex skill-change notifications during startup. - Reconcile composer submissions by client request ID across response loss, navigation, and reload. Retain text typed during delivery. - Keep acknowledged run-only Stop neutral and show workspace contention as waiting. Project exhausted native failures as Blocked. - Restore the guarded task-page retry action for failed legacy runs, including the server-supported explicit new-attempt path for stopped conversation adapters. Preserve native/process recovery holds and avoid promising Retry while a decision or execution gate hides it. - Refresh delivered artifacts and handle direct user replies that reopen completed work without a clarification loop. - Check the embedded PostgreSQL PID, data directory, and actual port before connecting or migrating. - Document accepted behavior and add focused regressions at lifecycle, route, transcript, and UI boundaries. ## Verification - Final head `fece606ac2` passes the complete GitHub CI matrix: **34 green checks, two expected Storybook skips, no failures or pending checks**, including `ci / verify`, `ci / e2e`, full runner verification, typecheck, build, every server/workspace shard, and all browser shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/34727183287). Greptile is **5/5 with no open findings**. The final two commits only refine test fixtures; both affected suites pass 24/24 locally and in CI, with server typecheck green. - Complete local Vitest coverage uses the canonical groups/shards: all 635 general server suites, all 145 serialized suites, and all workspace packages. The aggregate began on `0a8001c18` while the final queue fix arrived: 23,903 passed, five failed, 87 skipped. The five port/socket/timing failures passed unchanged in follow-ups (60 tests in the exposure/file suites and 412 tests covering the serialized failures and unrun tails). The final queue/operator-identity suites separately passed 52/52. This is aggregate coverage plus explicit reruns, not a pristine single-command final-head run. - After integration with current master, queue/operator-identity/continuation suites passed 162/162 and affected UI suites passed 140/140. ACP Stop/continuation and legacy task/Inbox/message browser suites passed 9/9, including both task recovery Retry and thread Try again, automatic saved-message delivery, exactly one new run, Done, and retained output after reload. The default process Stop/Pause/Resume browser case passed (the native-provider case is opt-in and skipped by default). The complete Board attachment/receipt browser suite passed 11/11 on a disposable instance, covering both composers, exact receipts after lost responses, no replay, bound attachments, and newer drafts after reload. - Blocking-intent regressions cover pre-existing Blocked, a mismatched run/cause, an explicit manual re-block, changed dependencies, a queued comment after failure, and a block arriving between scheduling and provider dispatch. The negative cases reproduced before the fix. All 478 affected executor/recovery/dispatch tests passed; both database suites ran separately after availability-probe skips in the first combined command. The final late-dependency check passed all 143 affected recovery/dispatch tests (zero skips) after two new negative cases reproduced the bug. - Focused runtime regressions cover awaited runner ownership publication, authenticated adoption before the first checkpoint, old-finalizer detachment, idle and busy warm-session shutdown, rejected checkpoint propagation, provider-pack verification, and managed-Codex credential preservation. Four managed credential detachment cases reproduced the bug before the fix; normal owned cleanup still succeeds exactly once. - Live local Claude: SIGKILL 2.6 seconds into startup recovered the same run automatically in 53 seconds, then a normal follow-up completed in 24 seconds. SIGTERM 2.5 seconds into startup preserved the same run (54 seconds) and its queued follow-up (21 seconds). Answers remained visible and the task reached Done. - Live Claude Daytona: a warm follow-up retained its sandbox and fell from 121 seconds to 44 seconds. A separate cold turn took 127 seconds; after controller shutdown and checkpointing, its follow-up completed in 33 seconds with the same sandbox, workspace, native session, and runner. Both answers remained visible and the task was Done. - Other live journeys covered task completion and follow-up with local and Daytona Codex, local Codex crash recovery, Stop then new direction, clarification response, live artifact refresh, and shared-workspace waiting. - Validation limits: the opt-in native composer Stop/Pause→subtree Resume fixture exposes terminal/result ordering and subtree-cancellation attribution bugs that can leave a child task blocked; that new finding is assigned to a separate follow-up and is not claimed fixed here. Default CI skips this optional native-provider fixture. Managed-Codex credential handoff and the queue-agent integration use automated regression evidence. Cold custom provider-pack uploads still add startup latency. ## Risks - Automatic replacement remains deliberately narrow: local Codex, verified stopped identities, unchanged retained state, and a complete text/completion-only turn. Unknown actions, partial history, or changed ownership remain blocked. - Claude completion permission handling changes an upstream package patch. The exact isolated task bridge must remain pinned; unrelated tools keep their existing permissions. - New task failure projection changes user-visible status. No historical status backfill or database migration is included. - This is a broad lifecycle fix across server and UI. Live proof covers graceful local Claude restart during startup and idle Claude Daytona session recovery across controller shutdown. Live abrupt SIGKILL during local Claude startup also recovered the same run. Unknown ownership or missing action evidence still blocks reuse. Cold custom provider-pack uploads still add startup latency; this change avoids unnecessary repeat uploads. ## Model Used OpenAI GPT-6 (Codex), with reasoning, code execution, browser automation, and tool use. The exact hosted model ID and context window are not exposed in this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
250deab910 |
fix(runner): keep healthy native sessions alive (#13261)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions keep a provider process and its work alive across control-plane operations. > - A hidden 15-minute turn deadline stopped work even when the agent timeout was zero. > - A one-hour runner lifetime and fixed connection lease added two more limits. > - Recovery also rejected goal commands because it reconstructed their startup summary with the wrong protocol version. > - This pull request removes implicit duration limits and renews authenticated leases in the harness. > - Healthy sessions can continue without model action or a user interface change. ## Linked Issues or Issue Description Refs #13092 and #12845. Related: #13163 covers sandbox recovery after app restarts; this change covers session duration and lease renewal. **What happened?** A native Codex session stopped after 15 minutes while a tool was still running. The agent had `timeoutSec: 0`. Recovery then rejected a `session.goal.get` startup command with `invalid provider startup ownership fence`. **Expected behavior** An unlimited session keeps working while its provider and authenticated controller remain healthy. Lease maintenance is transparent. Explicit timeouts, cancellation and revoked authority still take effect. **Steps to reproduce** Start a native session with `timeoutSec: 0` and run a tool beyond 15 minutes. Before this fix, the runtime cancels the turn. A recovery startup that uses a goal command also exposes the protocol-version mismatch. ## What Changed - Honor the agent turn timeout. Zero disables the timer. Long explicit durations use timer chunks to avoid Node timer overflow. - Default native runner lifetime to unlimited. Keep bounded startup, reconnect and control-operation deadlines. - Renew leases over the authenticated connection. Persist renewal before the reply. Validate identity, epoch and expiry. Handle duplicate requests and a lost reply on reconnect. - Freeze renewal during warm ownership transitions and terminal handling. - Validate persisted goal startup commands with protocol v2. - Add duration, renewal, ownership, recovery and real-process regression tests. Update runner protocol and recovery docs. - Add no UI components or controls. Renewal requires no model output or user action. ## Verification Current head: `348e369c35c5da8bb8be378f4b35dcf6f40882e7`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/34649767113). - All 32 checks pass on this head. The two Storybook checks are skipped as expected. CI includes full build, typecheck, runner verification, browser suites, server tests and the canary package dry run. - Greptile reports 5/5 on this head. All review threads are resolved, and the security scan passes. - Passed `pnpm -r typecheck` and `pnpm build` locally. - Passed 219 native-runtime and controller tests, including fake-clock tests for three weeks of renewal and 30-day explicit timeouts. Six denial tests confirm that renewal cannot extend expired, revoked or mismatched authority. - Passed 337 executor, cancellation and restart-recovery tests, plus 278 Rust runner-core library tests. - Passed a real runner with a silent fake Codex provider across its original lease expiry. Runner PID, provider PID, thread and active turn stayed unchanged. Warm-attach recovery tests also pass. - Passed all 83 plugin-worker tests and 159 of 161 workspace-runtime tests locally. The two remaining assertions passed with a canonical macOS temporary directory, as did the changed runtime fixture. The full affected server shard passes in CI. - An unchanged GitHub callback-ordering test failed once in CI, passed locally in isolation, and passed its one test-shard retry. The final CI summary is successful. - The full local `pnpm test:run` sweep was interrupted after dependency setup failures and load-related timeouts. Identified failing suites passed in isolated reruns after the dependency repair. The complete test matrix passed remotely in CI. ## Risks - Deploy the controller and runner together to enable renewal. Older peers keep their existing bounded lease behavior. - Unlimited runtime permits long resource use until completion, explicit cancellation, configured timeout or loss of valid authority. - Lease renewal changes authenticated protocol handling. Regression tests cover stale, revoked and mismatched authority, lost replies and warm handoff behavior. - Simulated multi-week tests and a real lease-boundary test do not constitute a weeks-long production soak. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution and TypeScript/Rust test tools. The exact backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
663c44cb2b |
fix: continue conversations after confirmed remote runner stop (#13254)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Users can stop a run and send another message on the same task.
> - Remote runners need evidence from their sandbox provider that
execution stopped.
> - Local process checks cannot prove that a remote process exited.
> - This pull request records provider stop receipts and uses them for
conversation admission.
> - New user messages can proceed after confirmed cleanup without
repeating interrupted actions.
## Linked Issues or Issue Description
Refs #13237 and #13239. Related: #13163 covers app-restart recovery;
this change covers an explicit stop followed by a new user message.
**What happened?**
A stopped remote Claude ACP task kept its execution hold after Daytona
cleanup succeeded. Native runners also rejected remote process
identities and retained stale session cleanup gates. A message sent
during cleanup could stay deferred after the sandbox stopped.
**Expected behavior**
After the provider confirms that the old execution stopped, a new user
message starts a fresh turn. Pending user messages must not need another
message to trigger admission. Prior action outcomes remain recorded.
**Steps to reproduce**
1. Start a long-running task in Daytona with a legacy Claude ACP or
native ACP runner.
2. Cancel the run while its tool is active.
3. Send a new message immediately, or after cleanup completes.
4. Observe the execution hold despite the old sandbox having stopped.
**Paperclip version or commit**
Reproduced on master at
|
||
|
|
ca96e1eb0a |
fix(runner): keep streaming after task completion tools (#13108)
Keep receiving provider events after paperclip_finish, drain pending event persistence, and select the final assistant answer after the provider turn ends. Preserve cancellation, failure, and governed-wait behavior. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
fac07b42ad |
fix(runner): preserve durable native session authority across recovery (#13092)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner carries tool results and task output to the control plane. > - A lost connection must not change which run owns a result. > - A session must not become reusable while provider output is still pending. > - This pull request adds strict recovery evidence and bounded drain barriers. > - It preserves current PRP version negotiation and session-goal support. > - The benefit is safer reuse of native sessions after a transport failure. ## Linked Issues or Issue Description Refs #13038. This is the first of two stacked pull requests. It contains the native runtime prerequisites. The second pull request contains the experimental chat-channel integration. It preserves the provider identity and typed terminal-failure contracts in #13074 and the durable recovery work in #13075. **What happened?** Native session failures could leave retained provider events, incomplete tool results, or warm handoff state that was not safe to reuse. A later run could observe output from an earlier authority. **Expected behavior** Recovery must preserve exact run, tool, process, artifact, and lease evidence. Uncertain or corrupt state must fail closed. A successful close must prove that retained provider output is settled. **Steps to reproduce** Run the transport and control-plane regressions. They hold and drop authenticated frames, fail durable writes, and restart fresh controllers and runner processes with retained state. Provider executables are local test fixtures. ## What Changed - Preserve pending provider cleanup and semantic-result evidence across session close and restart. - Add an authenticated warm handoff with exact old and new identities, durable receipts, and completion acknowledgement. - Drain retained provider events under the cumulative acknowledgement fence. - Reject corrupt tool-result contracts without unsafe provider replay or reusable checkpoints. - Keep ordinary PRP v1 sessions and current session-goal behavior. Require negotiated PRP v2 and acknowledged native session evidence before warm authority rotation. - Preserve late semantic inputs and exact durable result receipts until close can prove settlement. - Add transport, crash-window, artifact, checkpoint, and final-output regressions. - Deduplicate resolved execution delivery under the current issue lock. Reuse the exact existing successor after concurrent scans or a lost acknowledgement. Preserve newer operator evidence. - Persist idle provider integrity/capacity failures before process retirement, retain permanent model-rejection classification, and keep external question identifiers out of task instructions. - Expose only the context source on native status events. Keep thin dispatch projections compatible without exposing the complete context. ## Verification - Review-fix revision: 128 runtime-context/native-session tests, five idle-failure/adjacent Rust cases, 24 warm crash-window cases, three startup-notification/close cases, and five attach/backlog cases passed. The security and idle-failure cases were first reproduced failing. - Prior merged revision: runner production build, TypeScript typecheck, complete Rust workspace tests and formatting passed; 272 focused runner tests and two real PostgreSQL regressions passed. - Earlier full runner runs and CI Build failed on missing semantic-result fixture receipts, stale local provider fixture bytes, startup-notification ordering, and a confirmation-loss fixture that could accidentally send its final ACK. Each cause was reproduced and corrected without relaxing production authority or close assertions. These earlier runs are retained as failures, not represented as passing verification. - The first local repository-wide run failed before later phases because the isolated install omitted PostgreSQL's native-library aliases; it also encountered an unrelated occupied-port fixture. Those results are retained, not represented as a passing run. - Exact `335b2ee52709afb3885d4d6ebb2a3ece4b5864d6`: the complete runner suite passed 1,888 tests, with 10 existing skips. The full Rust release workspace passed with serial test scheduling. The unchanged parallel Rust run hit the five-second 300-descendant fixture deadline; that failure is retained. No deadline or assertion was relaxed. - The resolved-execution regression suite passed 57 tests, including concurrent delivery, lost acknowledgement, superseded authority, and newer operator evidence. Plain server typecheck passed. The duplicate-delivery cases were first reproduced failing. - Prior exact `335b2ee52709afb3885d4d6ebb2a3ece4b5864d6` CI passed all required jobs and Greptile reported 5/5. Its local general-server run passed 7,208 tests but failed one responsibility fixture; later phases did not run. The fixture started the next wake while its bounded handoff was active. It also used nonexistent comment IDs, which hid the current stored-message-author identity rule. The updated tests use real message authors, preserve task ownership, and await exact automatic handoffs. No production identity policy changed. - Current head `aa39275a1f300f7d1a0b16cd0885eea567cff6b0` includes current master and the native context-source projection. The focused identity/status cohort passed 27 tests and plain server typecheck passed. Fresh full repository tests, types, build, required CI, and Greptile review are pending. Final results will be updated before merge. - This is deterministic local-provider evidence. It is not a claim of complete live-provider qualification. ## Risks - This changes authenticated recovery and close ordering. The TypeScript transport and runner binary must be built from the same revision. - Failed or incomplete evidence intentionally prevents reuse and can require a fresh run. - PRP v1 ordinary/cold sessions remain supported. A v1 connection lease cannot upgrade in place. A current v2-capable runner held on a v1 lease was qualified through owned-process retirement/join, fresh bootstrap on the same old authority, v2 observation/ACK, then warm rotation. Legacy binary replacement and adopted-owner migration are not qualified by that test; rollout must not present them as automatic same-lease upgrades. - This pull request has no database migration or chat-channel activation. The second pull request keeps the channel feature experimental. ## Model Used OpenAI Codex assisted with implementation, tool execution, tests, and reconciliation. The existing implementation records OpenAI `gpt-6-astra` assistance. The current environment does not report a context-window size. No private reasoning traces are included. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
6681104692 |
fix(runner): preserve provider identity and terminal failures (#13074)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The runner translates provider events into durable task execution.
> - Provider notifications can refer to another thread or a descendant.
> - Root validation treated these notifications as fatal, and later
layers could lose the original failure.
> - This pull request classifies event identity and preserves structured
terminal failures.
> - The server can then choose safe recovery without weakening tool
authority.
## Linked Issues or Issue Description
Refs #13038. This change incorporates the shared protocol-integrity and
bounded-cleanup prerequisites from that PR. It does not include the chat
feature. Related failure classification work: #13028.
**What happened?**
An informational provider notification for another thread could
terminate the root session. A failed stream could then become a
missing-result error and lose its cause.
**Expected behavior**
Ignore unrelated informational notices with bounded diagnostics. Reject
invalid authoritative events. Preserve the original failure code and
recovery meaning through cleanup.
**Steps to reproduce**
Run a native Codex task. Deliver a notification for an unrelated thread,
or close the stream after a structured failure. Inspect the root outcome
and recorded failure.
**Paperclip version or commit**
Reproduced before
|
||
|
|
2043e0c735 |
fix: repair runner configuration, macOS execution, and artifact galleries (#13062)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent adapters select a provider, a model, and a runtime. > - Runner conversion rejected existing Claude agents. The model list mixed providers. > - The native Claude runner rejected custom models and could not launch on macOS. > - This pull request fixes conversion, model selection, and verified macOS execution. > - It also groups configuration fields consistently across adapters and opens artifact images in the task gallery. > - Operators can change an agent configuration and run the selected model on their Mac. ## Linked Issues or Issue Description **What happened?** Converting an existing Claude agent to Paperclip Runner failed with a Codex-only restriction. ACPX Claude showed unrelated models and required `claude-sonnet-5`. Its native runtime rejected macOS. Configuration mixed common model settings with process controls. Artifact cards labeled “Open gallery” navigated to attachment URLs instead of opening the task gallery. **Expected behavior** Conversion keeps agent identity and compatible settings. ACPX Claude uses the normal Claude catalog and accepts typed model IDs. Codex uses the native runner. The verified Claude runtime can launch on macOS ARM64 and x64. Common configuration sections place the same fields together across adapters. Artifact images open in the shared task gallery with navigation and downloads. **Steps to reproduce** 1. Open the configuration of an existing Claude agent. 2. Convert it to Paperclip Runner. 3. Select ACPX Claude and a different catalog model or a typed model ID. 4. Save the agent and run a disposable task on macOS. 5. Inspect configuration and advanced run-policy controls across adapters. **Paperclip version or commit** The bugs were reproduced on `165ca56a22adb60e5fda56045442d9c8498116a8`. This branch was rebased onto `7ed122911`. **Deployment mode** Built from source. Local test-drive instance on macOS ARM64 with an isolated database. Related work: #11798 addresses unsupported ACP session options in the existing adapter path. #13048 addresses working-folder preservation. This change fixes native runner configuration and launch behavior. ## What Changed - Remove the Codex-only conversion restriction. Preserve agent identity, instructions, directories, credentials, and compatible model settings. Reset incompatible sessions while retaining history. - Show ACPX Claude and native Codex as distinct provider choices. Remove ACPX Codex from advertised configuration. Normalize legacy configurations before fresh runs without rewriting historical run descriptors. - Select model catalogs and cache entries by provider. Support refresh and typed model IDs. Pass exact Claude IDs through session creation, model changes, and recovery. - Add verified macOS ARM64 and x64 Claude SDK snapshots. Bound executable allocation and total snapshot size. Preserve package checks, dependency isolation, process ownership, cancellation, and Linux descriptor loading. - Probe local runtime readiness. Report remote platform checks as incomplete until the remote runner verifies its runtime. - Surface actual model rejection and allow correction and retry. - Repair missing ACPX goal-capability helpers exposed by the post-rebase live test. Persist and restore the optional capability without breaking session startup. - Put Agent identity first and intentionally remove the Capabilities editor, as requested. This is removal of UI editing, not relocation: preserve existing capability metadata and API compatibility without adding another editor. Use the themed select for configurable permission modes, with normal text instead of monospace. - Put model and provider under Adapter. Give environment variables their own section. Fold command and arguments under Configuration. Fold lifecycle, timeout, and interrupt grace under Advanced Run Policy. Hide single-option permission controls. - Open image and video artifact cards in the existing task gallery, including cards in the artifacts panel. Chat attachment images use the same gallery. Preserve standalone media previews and download links. ## Verification - Rebased focused UI/API/database suites: 293 tests passed. - Rebased native runtime and ACPX suites: 242 passed, 7 skipped. - Repository typecheck, build, and token gates passed for the runner changes. Gallery follow-up UI typecheck, build, and token gates also passed. - Follow-up UI suites passed (86 tests), packaging checks passed (14 tests), and the final focused runtime suites passed (126 passed, 7 skipped). - Linux container isolation and lifecycle fixtures passed before rebase (57 passed, 2 skipped). Rust ACPX provider-session tests passed after rebase (8 tests). - Browser tests completed actual Claude and native Codex tasks on macOS ARM64. They covered conversion, catalog refresh, a non-default catalog model, a typed `haiku` ID, save/reload, cancel, follow-up session continuity, invalid-model errors, and recovery. - Final-revision live tests completed a typed Claude task, a follow-up with the same provider session, and a native Codex task on macOS ARM64. - Browser tests confirmed the moved interrupt-grace field saves and survives reload. Cross-adapter tests cover Claude, Codex, Gemini, process, gateway, and schema forms. - Full local run: 7,080 passed, 30 skipped, and two timeouts. Both timeout suites passed on isolated rerun (84 tests); the failures were the plugin login-worker exit diagnostic and the runner real-server vertical slice. - Final follow-up checks: 50 registry tests and 45 snapshot/installation tests passed (6 platform-specific skips). Oversized executable rejection is covered before allocation or reading; unsupported-platform tests invoke the real installation probe. - Runner head `ddb5101c483a297f74875ab96b3c66035b002d50`: all CI gates green, including full runner verification, repository build, typecheck, general/serialized server suites, browser tests, and canary dry run. [CI run](https://github.com/paperclipai/paperclip/actions/runs/34286178670). - Greptile: 5/5 on that runner head. All four review threads resolved. Superagent, Socket, and Snyk checks green. - After snapshot hardening, another real Claude task completed on this Mac using the rebuilt runtime. - Gallery follow-up: 148 focused tests passed, covering artifact selection, shared attachment collections, deduplication, image/video cards, standalone previews, downloads, and closing. Live browser verification completed on the settings follow-up: artifact selection, 6-image pagination with wrapping, download action, and closing all stayed on the same task URL. All checks passed on gallery head `96136da58ff195bf6ca00b281eb3022ad12d7bd8`: [CI run](https://github.com/paperclipai/paperclip/actions/runs/34287987536). Greptile returned 5/5 on that exact head with no unresolved threads. - Final settings polish: 96 focused tests, UI typecheck/build, and token gates passed. A real browser walkthrough verified readable permission options, identity placement, Capabilities removal, and permission save/reload. Original test-agent permission mode restored. All 31 checks passed on final head `e46540d6bf32bfb0566dca16b2f4a75ba437618c`: [CI run](https://github.com/paperclipai/paperclip/actions/runs/34292797886). Greptile returned 5/5 with no unresolved threads. ## Risks - Capabilities intentionally has no editable UI field after this change. Existing values remain readable and API-compatible; removing the field does not erase stored metadata. - macOS launch now copies verified package files into private snapshots. The implementation must retain isolation and clean up snapshots on exit. - Runtime provider or model changes reset the current session. Historical runs remain available. - The macOS x64 SDK executable digest was verified, but a live Intel Mac run was not available. Linux verification used container fixtures, not a real Claude task. - Remote environment tests report a warning when only the platform has been checked. They do not claim package readiness from the server host. ## Model Used OpenAI Codex, based on GPT-6. The exact served model identifier and context-window limit are not exposed in this session. Used reasoning, repository inspection, code execution, Rust and TypeScript tests, and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites and both timeout suites on rerun; full-run counts above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
7ed122911b |
Add end-to-end session goals to Paperclip Runner
Add capability-aware slash-goal controls, durable provider goal state, PRP v2 negotiation, autonomous goal execution, and safe local session recovery. Integrate with current master, preserve provider session identity, and verify the browser goal/chat/replacement/clear workflow and unsupported-agent rejection. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
0c1e7504c0 |
fix(runner): persist warm Daytona workspaces across turns (#12904)
## Thinking Path > - Daytona preserves a stopped sandbox filesystem, but deleting or replacing a sandbox removes its only remote copy. > - Warm reuse therefore improves latency but cannot be Paperclip's durability boundary. > - The host execution workspace must remain authoritative after every successful turn, while same-run recovery must avoid overwriting unexported remote work. > - Result proposal, workspace export/merge, and terminal completion need a durable, replayable ordering so a crash never starts a duplicate provider turn. > - A paid browser acceptance suite must exercise both legacy Codex and Runner Codex for three real turns on one continuously warm Daytona sandbox. ## Linked Issues or Issue Description Refs #12901. Runner Codex did not previously export successful Daytona workspace changes back to the authoritative host workspace. That made warm reuse depend on Daytona's remote filesystem and left deleted/replacement sandboxes without a reliable reconstruction path. The existing paid fixture also lacked a focused three-turn continuity case for both Codex adapters. ## What Changed - Persist versioned, atomic native workspace-sync descriptors and durable seeds in `PAPERCLIP_HOME`, without credentials or a database migration. - Classify fresh, warm, replacement, and same-run-recovery workspace preparation explicitly; ambiguous lease/root/digest evidence fails closed. - Finalize native workspace export/merge after semantic result proposal and before run completion, with idempotent replay that never submits a second provider turn. - Surface legacy Codex workspace restoration failures instead of masking them, while preserving an earlier provider error when both fail. - Keep healthy reusable Daytona leases warm for legacy and native adapters, stamp finalized workspace generations, and retain existing cleanup behavior for per-turn or unhealthy leases. - Preserve Runner Codex's provider process/session across warm turns, including bounded post-terminal tail draining and exact authority rotation. - Add the exact paid `daytona-warm-continuity` matrix: - `legacy-codex × daytona × warm-three-turn` - `runner-codex × daytona × warm-three-turn` - Drive all three turns through the browser, verify ordered file continuity and stable lease/workspace/runtime identities, capture per-turn timings, and delete the sandbox immediately after assertions. - Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no package script was added. ## Verification - `pnpm typecheck` — passed, including migration safety (no migration added) - Focused server/runner Vitest coverage — 144 passed - `pnpm test:e2e:runner:unit` — 114 passed - `pnpm test:e2e:runner:typecheck` — passed - `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed, 1 helper ignored - `native-session-executor.test.ts` — 139 passed, including safe fail-closed cleanup after remote runner identity capture failure - Paid local browser acceptance, exact post-rebase Linux/amd64 runner binary: - Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created, resumed, resumed`; 10/10 matchers; cleanup passed - Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created, resumed, resumed`; 10/10 matchers; cleanup passed - [Protected paid GitHub Actions campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033) against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2: - Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`; evidence and cleanup passed - Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`; evidence and cleanup passed - Merge/enforcement, S3 history, and Pages publication jobs passed - Paid result artifacts were scanned for both provider credentials; neither secret was present. - Current PR checks — 31 passed, 1 expected Storybook skip; Greptile 5/5; Superagent security scan passed - `git diff --check origin/master...HEAD` — passed - Confirmed no `package.json`, lockfile, migration, or SQL changes. ## Risks - Workspace synchronization now sits on the terminal-success path, so a remote export failure deliberately prevents false success. Retryable state retains its lease/seed; loss of the only unexported remote copy fails closed. - Warm provider reuse has strict identity and quiescence checks. Mismatched or ambiguous evidence blocks reuse rather than risking concurrent provider work. - The paid suite incurs Daytona and Codex cost only in the existing protected scheduled/manual workflow and explicitly destroys its sandbox after each cell. ## Model Used OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real browser E2E execution, Rust/TypeScript test execution, and GitHub Actions diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked an existing issue or described the issue in-PR - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have documented the dedicated suite invocation without adding a package script - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green on the current revision - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups on the current revision - [x] I will address all reviewer comments before requesting merge |