mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
be34a2d9d28ad2ba9fdf40d176b7e0a45393cc44
7325
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude
|
be34a2d9d2 |
test(restart-chaos): two servers on one database (MS1-MS3) and a cross-server invariant (#20518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) tests how runs survive control-plane faults > - Every wave runs one server, so no wave can reach a race between two servers on one database > - Hosted Cloud deploys by rolling overlap: a new server's startup recovery and sweeps run beside the old server's live runs > - #20508 fixed such a race (another server's orphan sweep failing a run whose owner took its lease back) with a database test only > - This pull request adds three two-server faults (MS1–MS3) and a cross-server invariant > - The benefit is that cross-server ownership races can be reproduced and graded in a live wave ## Linked Issues or Issue Description Refs #20508 (the race this harness can now reach live). **What is missing?** The harness starts one control plane per wave. A fault replaces or freezes that one server. Nothing starts a second server on the same database while the first still owns runs, so cross-server ownership bugs are found only by database tests. **How production runs (checked before choosing the scenarios)** - Hosted Cloud runs on Railway with Neon Postgres. A deploy is a rolling A/B overlap: B starts on the same database, A serves until B is ready, then A drains. Defaults are a 5 s overlap and up to 90 s of drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`, lines 374–375). The chaos plan names the same shape as K12 (`doc/plans/2026-11-06-control-plane-restart-chaos.md`, line 103). It also records B's startup recovery hitting a lock on a run A still owned (lines 229–230). - This repository has no replica count or deploy strategy of its own. `docker/ecs-task-definition.json` describes one task, and the real Cloud deploy lives in another repository. The self-hosting docs do not describe several servers on one database. - Server-side ownership: per-boot controller lease of 90 s (`server/src/services/controller-instance-lease.ts`), legacy run lease of 60 s, native executor lease of 20 min, and the orphan sweep in `server/src/services/heartbeat/recovery.ts`. So MS1 models B's overlap beside a live A, MS2 models A paused (for example a host sleep) past its controller lease while B is up, and MS3 models the rolling upgrade itself. ## What Changed - New `tests/restart-chaos/multi-server.ts`: - MS1: B boots on the same database, instance directory, storage and secrets, on its own port. It stays up 60–150 s beside a live A, then gets SIGTERM. - MS2: B boots. A and its tree are frozen (TM2's freeze without aging) for 100–150 s, past the 90 s controller lease, then thawed. B stays up 30 s more, then stops. - MS3: B boots, then A gets K2 and a replacement on A's port. B stops 20–45 s after the replacement is ready. - B is always stopped once started, also when a step fails. - `checkCrossServer`, graded once per wave with an MS fault, over the wave database and the server process windows (pid, start, exit, freezes): - `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while its owner was alive and not frozen (#20508's shape included). The owner's own lease loss counts only without a prior freeze past the lease. - `MS_double_executor`: a run recorded a second runner identity before the first was shown stopped. - Unattributable ownership failures are listed, not charged. - `supervisor.ts`: records every server's process window and freezes, spawns and stops B (`<wave>/server-b.log`), and runs the MS steps. TM freezes are recorded too. - `types.ts`, `scheduler.ts`, `journey-matrix.ts`, `config.ts`: MS1–MS3 fault kinds, seeded holds, and validation (local only; no storm, crash point or K14). - `main.ts`: writes `<wave>/cross-server.json` and fails the wave on a violation. - README section "Two servers, one database (MS1–MS3)", with the production evidence and the limits. Test-only. No `package.json` or `.github` changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 473 tests pass. - New `multi-server.test.ts` (14 tests): option validation and seeded holds; MS1/MS2/MS3 step order and timing with fake hooks; B stopped when a step fails; the invariant for a live owner, a frozen or exited owner, #20508's thaw-then-write shape, lease loss with and without a freeze past the lease, unattributed owners, and double executors. - New `multi-server-db.test.ts` (embedded Postgres): a run still `running` with `native_execution_ownership_unverified` (#20508's marker) is read at its write time and charged as `MS_foreign_failure`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave `w-20261110T105448-s301` on head `a9ec5a4502` (master product code + this harness), `--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3`: 3 of 3 pass, all crossed their fault. B booted on the shared database in about 10 s each time. MS2 froze A for 104 s; A stayed up after the thaw. Cross-server invariant: 0 violations, 0 unattributed. ## Risks - Test-harness code only. - B runs on the same host and PID namespace as A, so it can check A's PIDs directly, which a Railway replica cannot. These faults are a lower bound on cross-server exposure. The README says so. - MS2 outlasts the 90 s controller lease, not the 20 min native executor lease. - MS faults refuse TM2/TM3 in the same wave, and fail when B exits before its planned stop. - `MS_foreign_failure` uses the run's `finished_at` (or `updated_at` for a still-running row) and the owner recorded on the run. A run whose owner changed after the failure could be misattributed; the evidence file keeps the owner and windows for review. - `MS_double_executor` reads runner identity events; it cannot see a second executor that never recorded one. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c0593b69f1 |
fix(accounting): close a remote-Runner turn completed across a session restore as a lower bound (#20524)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs settle their charge once the final usage receipt is complete. > - #20517 fixed one case for the local Runner. If a restart stops the provider before a turn's final usage report, and the restored Runner then reports that turn completed, the turn now closes as a lower bound instead of waiting 10 minutes. > - Remote Runners deliver usage through a separate producer-owned receiver. That receiver dropped `session.resumed`, so the same turn there still waited for the 10-minute closure timeout. > - This pull request applies the #20517 rule to that receiver. > - The benefit is that remote ACPX and OpenCode runs settle promptly with an honest lower-bound label after a mid-turn restore. ## Linked Issues or Issue Description Refs #20517. **What happened?** After #20517, `session.resumed` reached turn accounting only through the local Runner's fallback recorder. The remote producer receiver (`native-usage-delivery.ts`) and the session-authority filter (`isNativeUsageAccountingEvent`) both dropped it. A remote ACPX or OpenCode turn that was open across a session restore, and then reported terminal without a complete report, left the receipt incomplete until the closure timeout. **Expected behavior** Same rule as the local Runner. Such a turn closes as a lower bound right away. A complete report that arrives after the restore is still used, and a turn that starts after the restore stays pending until its own report is complete. **Steps to reproduce** `native-usage-delivery.integration.test.ts > a turn restored mid-turn on a remote Runner > closes the turn as a lower bound, also after the controller restarts between restore and terminal` fails on master: the receipt stays `accountingReceiptReady: false`. ## What Changed - **Receiver (`createNativeUsageReceiver`).** It now accepts `session.resumed` for turn-reporting providers (ACPX, OpenCode) while a turn is open. It journals the restore as a receipt like any other producer event, so turn accounting marks the open turn restored. A replacement controller rebuilds the same state from that journal. - **Restores never fail delivery.** A restore from another run or source, or one with no open turn, is ignored, not rejected. An unacknowledged lifecycle event would otherwise replay without end. - **Session-authority filter.** `acceptAccountingEvent` now uses a new `isNativeUsageDeliveryEvent`: the usage events plus `session.resumed`. A restore for another run is ignored there too. - **Recovery routing is unchanged.** `acceptRecoveryAccountingEvent` and `onRecoveryAccountingEvent` keep the usage-only `isNativeUsageAccountingEvent`, so a source run's restore never requires a proven recovery source. - **Turn accounting** gets a `hasOpenTurn()` helper. - **Docs:** `doc/SPEC-implementation.md` now covers both Runners. ## Verification - `native-usage-delivery.integration.test.ts`, describe `a turn restored mid-turn on a remote Runner` (real PostgreSQL): | Test | On master | Here | |---|---|---| | `closes the turn as a lower bound, also after the controller restarts between restore and terminal`: the restore is journaled by one receiver and the terminal arrives at a rebuilt one; replaying everything again is idempotent | fails | passes | | `keeps the turn's own final report when it arrives after the restore` (guard) | passes | passes | | `keeps a turn started after the restore pending, and journals nothing for a restore with no open turn` (guard) | passes | passes | - `native-session-executor.test.ts > binds retained warm transport usage to the successor run authority` covers routing through the session-authority filter: - a current-run restore reaches the successor's capture; - a draining-run restore reaches the drained capture; - an unrelated-run restore resolves with no capture; - recovery routing ignores a source restore. It fails when the filter is reverted to `isNativeUsageAccountingEvent`. - Related suites pass: `native-usage-delivery.integration` and `native-usage-accounting` (46 tests), and the usage, accounting, receipt and authority cases in `native-session-executor.test.ts` (224 tests). Server typecheck passes. - **The live remote path is unproven.** No Daytona wave ran because the Daytona Runner image has not been rebuilt yet. The local Runner counterpart (#20517) passed restart-chaos L1, seed 20386, 4/4 (`w-20261110T104332-s20386`). ## Risks - **Upgrade safety.** Usage state saved by current master has no journaled restore and rebuilds as before. A receiver that predates this change replays a journaled restore as an ignored event, which is the behaviour on master. A restore that an older controller already acknowledged is not replayed, so that run waits for the closure timeout as it does today. - **One extra receipt per mid-turn restore**, only for ACPX and OpenCode and only while a turn is open. Nothing is added before `turn.start`: a restore with no open turn returns immediately. - **Codex and other providers are unaffected.** They do not use turn accounting. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
16c30ac32b |
fix(runner): relaunch a provider whose startup only timed out without a reply (#20521)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. After a restart, a new runner relaunches the provider (for example the Codex app server). A durable startup fence stops a second runner from launching a provider while an unadmitted one may still run. > - #19784 lets a new runner retire a fence whose recorded provider process and group have provably exited. That rule covers a launch that a kill cut off. It deliberately keeps a failed initialization closed, because that can be a real provider failure. > - A relaunch whose reply never came within the runner's 30 s request deadline is saved as the same failed initialization. A stalled or frozen host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and Codex, and their monotonic clocks keep running while they are stopped. > - Every later runner then refused to restore with "provider startup ownership remains unadmitted", and the user had to click Retry. The owner's rule is that a run interrupted by a restart or sleep must resume by itself. > - This pull request records whether initialization failed only because the provider gave no reply, and lets a later runner retire that fence once the provider is provably gone. > - The benefit is that a stall during a provider relaunch continues automatically, while a real provider rejection still fails as before. ## Linked Issues or Issue Description Refs #19784 **What happened?** Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run `c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a TM3 kill followed the thaw. The next three runners each failed `runner.drain` with "provider startup ownership remains unadmitted". Recovery stopped with `native_session_retry_exhausted`, and the task needed Retry. **What did you expect?** The new runner relaunches the provider and the run continues. **Steps to reproduce** The new test `restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply` (runner-core `tests/codex_provider.rs`). The fake Codex stalls its first `thread/resume` reply past the 30 s deadline. On `origin/master`, the saved attempt is `initialization_failed` / `thread_open`, and the next runner fails with "provider startup ownership remains unadmitted". ## What Changed Design: - The deadline that passes is the runner's own 30 s request deadline. When a startup request (`initialize`, `thread/start`/`thread/resume`, `thread/read`) reaches that deadline with the provider still running and silent, the runner sets `noReply: true` on the durable `initialization_failed` startup record. - Not `noReply`: a provider error response, a provider that exited before it replied, or closed stdout. - `retire_orphaned_startup_attempt` retires an `initialization_failed` fence only when `noReply` is set and the recorded process and its process group are both gone (`ESRCH`). This is the same proof as `spawned` in #19784. A rejection, an exit, a live process or group, and spawn or admission failures stay fenced. - Validation accepts `noReply` only on `initialization_failed` at the `initialize`, `thread_open` or `thread_read` stage. - Bounded: a runner restores once (`restore_checked` / `restore_error`), so the loop is one relaunch per recovery attempt. Recovery's existing resume budget (`native_session_retry_exhausted`, seen after three attempts in the wave) stops a provider that never replies. - Upgrade: `noReply` is `#[serde(default, skip_serializing_if = "Not::not")]`. A new runner reads state saved by master as `noReply: false`, so that state keeps today's behaviour and stays fenced. - Rollback: an old runner (`deny_unknown_fields`) rejects state that holds a `noReply` fence and fails closed. That run needs Retry, which is today's outcome for the same fence. The field is written only while a timed-out fence is present, and admission clears it, so all other state is unchanged for old runners. - Normal path: nothing is added before `turn.start`. The cost is one in-memory flag per request; no extra I/O. Files: - `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and the `no_reply` field on `ProviderStartupObservation::Failed`. - `runner-core/src/provider_backend.rs`: the `noReply` field, its validation, and the retirement rule, with unit tests: - a no-reply fence is retired at `initialize`, `thread_open` and `thread_read`; - it is kept while the provider may still run; - a rejection or a state saved without the field stays fenced. - `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch `--stall-first-thread-resume-ms`. - `runner-core/tests/codex_provider.rs`: the end-to-end red/green test. - `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2 claim. A stopped process's monotonic clock keeps running, so TM2 is also a long process stall. ## Verification - `cargo test -p paperclip-runner-core --test codex_provider restart_relaunches_a_provider_whose_relaunch_timed_out` is red on `origin/master` (the next runner refuses restore) and green here, in about 31 s. - `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This includes `failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`, where a real "no rollout found" rejection still never restarts. - `cargo fmt -p paperclip-runner-core -- --check` is clean. - A restart-chaos proof wave on seed 426 will follow in a comment. ## Risks - A provider that hangs for good is relaunched once per recovery attempt until the existing resume budget stops it. Before, it failed on the first attempt. This costs at most a few extra 30 s attempts, and they are bounded. - Rollback to an older runner while a `noReply` fence exists fails closed for that one run, which needs Retry, as it does today. - No database, server or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, cargo and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
23fa37f5bb |
fix(recovery): treat a retired prepared remote epoch as nothing to recover (#20491)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A restart must never leave a task stuck: an interrupted run must resume on its own or have a working Retry > - When a graceful restart interrupts a remote-workspace run that is still preparing, the run keeps a `prepared` reference to its workspace epoch, and a preserve-without-replay hold blocks the task > - The unsaved-files reconciler then retires that epoch with `abandonUnverifiedRemoteWorkspaceEpoch` and writes a `paperclip.remote-workspace-abandoned-epoch.v1` receipt. The workspace content is the last committed save again > - Recovery never read that receipt. The held-preparation reconciler waited for a repair receipt that could never exist, and restore recovery refused the source as `source_binding_invalid`. The task stayed blocked, and a UI Retry returned 202 and started nothing > - This pull request adds one strict shared rule: an abandoned-epoch receipt for the run's exact prepared epoch and allocation means there is nothing to recover. Every recovery path now uses it > - The benefit is that the task resumes automatically from the last committed save after the restart, whichever recovery path owns the hold ## Linked Issues or Issue Description Refs #20072 (replay a preparation a restart interrupted), #20299 (storage outage waits). **What existing behavior does this improve?** Recovery of a remote-workspace run that a graceful restart interrupted during preparation, after its unsaved epoch was retired. **Subsystem affected** Server: recovery (held-preparation reconciler, workspace restore recovery, the workspace repair check). **Current behavior** These are Daytona R1 restart-chaos waves (Paperclip-sized workspace, `--storage s3`, K2 graceful restart at +110 s, during the ~730 MiB first save): - `w-20261110T052533-s8805` and `w-20261110T062703-s8805` (ST2 slow store), held-preparation path: both runs ended `interrupted` in `executionStage: preparing`. The reference was `{ state: "prepared", epoch: 1 }`, and the workspace had moved to `{ epoch: 2, location: null, dirty: false, writerRunId: null }` with an abandoned-epoch receipt for epoch 1. The `legacy_interrupted_preparation_hold` reconciler never settled the hold (`I2_settled` violations), because `isInterruptedNativePreparation` requires `hasVerifiedWorkspaceRepair`, and that check accepts only a `saved` reference. Grader: `R3:retry_no_new_run`. - `w-20261110T091639-s11936` (ST1 outage), restore-recovery path: the run also had `workspaceRestoreFailure: "restore_failed"` and `workspaceRestoreRecovery`. Restore recovery owns that hold. `inspectWorkspaceRecoverySource` threw `source_binding_invalid` / `workspace_reference_mismatch` with retry class `service_required`, and the hold says Retry "succeeds only after Paperclip is updated". Grader: `R3:retry_no_new_run`. **Proposed behavior** `isAbandonedPreparedRemoteWorkspace(db, run)` is true only when all of these hold: - The run has a `prepared` remote reference and no native workspace sync reference. - The workspace authority is at a later epoch, and the run is not its writer. - The workspace has an abandoned-epoch receipt for the same run, the same epoch, and the same lease, provider lease and remote cwd. That rule is used in three places: - `hasVerifiedWorkspaceRepair` (`workspace-recovery-receipt.ts`): the retired epoch counts as repaired. This covers held-preparation replay, the restore-failure check in `settleInterruptedNativeBootstrap`, the execution blocker, and explicit Retry admission. `verifiedWorkspaceRepairReceipt`, which returns a receipt id, is unchanged. - `inspectWorkspaceRecoverySource` (`workspace-recovery-source.ts`): the retired source is reported as `source_unavailable` / `use_alternative_strategy`, not as a binding failure that needs a service update. - `heldInterruptedPreparationWhere`: a restore-recovery hold is listed for the held-preparation reconciler only when the matching receipt exists (SQL pre-filter). The settle then re-checks the full rule in JS before it acts. **Reason and benefit** The retired epoch held only preparation files: the hold proof requires that no provider work started. Its unsaved files were already given up by the reconciler that wrote the receipt. Waiting for a repair, or for a service update, cannot finish. ## What Changed - New `server/src/services/remote-workspace-abandoned-preparation.ts`: `isAbandonedPreparedRemoteWorkspace` (the strict rule) and `abandonedPreparedRemoteWorkspaceExists` (its SQL pre-filter). - `workspace-recovery-receipt.ts`: `hasVerifiedWorkspaceRepair` also accepts the rule. - `workspace-recovery-source.ts`: a retired prepared source is `source_unavailable` / `use_alternative_strategy`. - `interrupted-preparation-hold-settlement.ts`: restore-recovery holds are listed only with a matching receipt. - `interrupted-preparation-hold-settle.test.ts`: held-preparation replay (before and after the first save committed); a restore-recovery hold in the exact shape saved by the previous build (`restore_failed`, `workspaceRestoreRecovery`, exhausted `source_binding_invalid`) that now settles and replays; inspection reporting `source_unavailable`; and the rule refusing a workspace still at the run's epoch, a run still the writer, a native sync reference, another run's receipt, or another allocation's receipt. Not changed: `run-completion.ts` and the graceful-drain path in `recovery.ts`. ## Verification - `cd server && npx vitest run src/__tests__/interrupted-preparation-hold-settle.test.ts src/services/task-retry.test.ts`: 101 pass. - Red before: with the three recovery files at master, 4 tests fail (held-preparation replay ×2, the restore-recovery replay, and the inspection result). Green after. - Neighbouring recovery suites: see the comment below. - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Chaos rerun on a build with this change: see the comment below. ## Risks - Previous-release story (upgrade-safe run state): no persisted shape changes. Holds, run references and receipts written by the previous build are read as they are. The restore-recovery test uses the exact shape the previous build saved. A task already stuck in that state settles on the next reconciler pass after deploy. - The rule needs an exact receipt (run, epoch, lease, provider lease, cwd) and a later workspace epoch, so a real recovery is never skipped. Any mismatch keeps today's behavior. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
c2423e30e8 |
test(restart-chaos): provider faults across K2/K4 restarts, per-workload message, per-run speed split (#20515)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) proves that runs recover when the control plane restarts > - PR #20494 fixed a saved governed wait whose provider terminal was late or lost across a restart. The harness proved it only with a K1 (kill -9) restart > - The harness could not restart with K2 (graceful SIGTERM) or K4 (outage hold) after a provider fault, so the graceful shutdown drain was not tested > - S1 needs the `interrupt_ack` message and Q1 needs `terminal`, but one wave could set only one message > - G1 was listed as a provider-fault workload, but its agent ends its own turn. No rule ever applies, and each G1 task waits until its deadline > - Speed breaches could not be explained after a wave, because the instance database is deleted > - This pull request adds K2/K4 provider-fault restarts, a message for each workload, removes G1, and keeps a speed split for each run > - The benefit is that one wave can test #20494 across graceful and held restarts. Also, a cold-start breach now shows whether Paperclip or the provider was slow ## Linked Issues or Issue Description Refs #20494 **What happened?** The harness accepted only `--provider-fault-restart K1`. A wave with `--workloads G1` and a provider fault planned tasks that could not apply the fault. Waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211 show `applied=0` for G1 with Codex and Claude. The board then waited for a restart that did not come. **What did you expect?** A provider fault can cross every restart kind that the harness supports. The harness rejects a workload that does not interrupt a provider turn. **Steps to reproduce** Run `--adapters codex --workloads G1 --faults PV1 --provider-fault-restart K1 --concurrency 1`. The Codex shim logs `bound` and then `exited` with code 0. It logs no `interrupt_requested`. The task fails `R1:no_rest_within_budget`. ## What Changed - `--provider-fault-restart` accepts K1, K2 and K4. A comma list (for example `K4,K4,K2,K2`) cycles across PV1/PV2 tasks in plan order. K4 holds the control plane down for 120 s. - `--provider-fault-message` accepts `WORKLOAD=message` entries next to the default, for example `S1=interrupt_ack,terminal`. - G1 is removed from `PROVIDER_FAULT_WORKLOADS`. The config check that told G1 to add a restart is removed, because G1 is now rejected. - `speed.json` keeps a `runs` list. Each entry has `bootMs` (Paperclip share), `queueMs`, `providerFirstActivityMs` (turn accepted to first activity) and the pre-turn spans. - README documents the new flags, the G1 limit, and the speed split. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 434 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave w-20261110T092735-s213 (Codex, seed 213, concurrency 1). All 4 tasks pass R1–R6, T and `provider_fault_applied`: - t01 S1 PV2 drop `interrupt_ack` + K4 (120 s hold) - t02 Q1 PV1 delay `terminal` 18 s + K4 (120 s hold) - t03 S1 PV1 delay `interrupt_ack` 18 s + K2 - t04 Q1 PV2 drop `terminal` + K2 - That wave wrote the new `runs` split in `speed.json`. Cold boot was 1.3–1.7 s and provider first activity was 2.0–2.6 s. ## Risks - Low risk. Only the test harness changes. Product code does not change. - Old `plan.json` files keep `providerFaultRestart: "K1"`. That value is still valid, so replays of old waves do not change. - A wave that used G1 with provider faults now fails at parse time. Before, it ran and failed at the deadline. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
289b402c26 |
fix(runner): settle a saved governed wait whose parked turn was lost on restore (#20519)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run can ask the board a question and park on that saved (governed) wait > - PR #20494 made a saved wait settle after a control-plane restart for Codex, which reports the parked turn as interrupted > - Claude runs through ACPX. After a kill -9 restart, Runnerd restores the conversation and reports the parked turn as `turn.failed` with `provider_turn_lost_on_restore` > - The governed-wait drain and the saved-replay check rejected every failed terminal as unproven, so the saved question failed and the task needed a human Retry > - Runnerd emits `provider_turn_lost_on_restore` only after it proves that the old provider lifetime ended. That is the same proof as an interrupted terminal > - This pull request accepts that recoverable restart loss as proof for a saved governed wait, and only that failure > - The benefit is that a Claude question survives a control-plane kill the same way a Codex question does ## Linked Issues or Issue Description Refs #20494. Related: #20498 (Claude lost cancel reply; this change does not touch that code). **What happened?** Restart chaos wave w-20261110T094930-s214 (Claude, Q1, PV1 and PV2 with `--provider-fault-restart K1`) failed 2 of 2 Q1 tasks with `R3:retry_needed`. The asking run ended `failed` with `native_provider_terminal_failed`. Its events show the question saved, the K1, recovery with `native_governed_wait_settlement_pending`, and then `turn.failed` with `provider_turn_lost_on_restore` for the parked turn. The thrown `providerCode` was `governed_wait_provider_terminal_unproven`. **What did you expect?** The saved question settles after the restart. The asking run ends `succeeded`. The board answer wakes the agent without a Retry. **Steps to reproduce** Run the harness with `--adapters claude --workloads Q1 --faults PV1,PV2 --provider-fault-restart K1 --provider-fault-delays 3 --provider-fault-restart-after-seconds 1 --concurrency 1 --tasks 4`. The new unit-test variants `lost-on-restore-after-restart` and `lost-on-restore-saved` failed with the same `providerCode` before this change. ## What Changed - `native-session-runtime.ts`: add `governedWaitTerminalProven(event)`. A failed terminal proves the stop only when `isNativeRestartInterruption(error)` is true (code `provider_turn_lost_on_restore`, `recoverable: true`). Other terminals keep the old rule (`providerTerminalObserved !== false`). - The governed-wait accounting drain and the saved-replay check both use that helper. - Tests: two new variants of the #20494 test (a lost terminal after the restart, and a lost terminal already saved before the next recovery). Two new rejection cases: a non-restart failed terminal and a non-recoverable restart loss still fail with `governed_wait_provider_terminal_unproven`. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime`: 265 tests pass. The two new settle variants failed before the fix. - `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` passes. - A live restart chaos wave on this branch (Claude Q1, PV1/PV2 + K1) is linked in a comment below when it finishes. ## Risks - Low risk. The change accepts one more terminal as proof for a saved governed wait. That terminal comes only from Runnerd after it proves the old provider lifetime ended, and the code already trusts it to continue a restart-lost turn (`nativeRestartInterruptedTurnId`). - A failed terminal with any other code, or a non-recoverable restart loss, is still rejected. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
37808f2a4e |
fix(accounting): close a local-Runner turn completed across a session restore as a lower bound (#20517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs record usage receipts, and accounting settles a run's charge once its final receipt is complete. > - ACPX Claude reports provisional zero usage until a turn ends, then one final report. > - A restart can stop the provider after the agent's finish call and before that final report. The restored Runner then resumes the session and reports the same turn completed with no complete report. > - The receipt stayed incomplete, so the accepted run waited for the 10-minute closure timeout and then settled as unpriced. Its spend was missing from company totals in the meantime. > - This pull request closes such a turn as a lower bound, the same way a turn the Runner fails as lost on restore is already closed. > - The benefit is that an accepted run settles right away with its observed usage and an honest lower-bound label. ## Linked Issues or Issue Description Refs #20484 (restart-chaos L1 contention follow-up). **What happened?** After #20484 fixed the database contention, restart-chaos L1 (`--seed 20386 --db-pressure 4`, codex and claude, K1) reached 4/4 tasks done. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. A replay on master with `--keep` (`w-20261110T100747-s20386`) reproduced it, and I read the kept database: - The pending run (`71f71d10`, ACPX Claude) succeeded and had its result accepted. Its usage had `accountingReceiptReady: false`, zero tokens, `costStatus: unpriced`, and no cost event. - Its only turn started before K1, and every usage report in it was provisional (`runDeltaComplete: false`). K1 struck right after the agent's `paperclip_finish` call (source seq 118), before the turn's final report. - On restart, recovery verified the local process stopped and resumed the session (`session.resumed`, seq 119). The restored Runner then reported the same turn `turn.completed` (seq 124) with no usage report. - The other Claude run in that wave lost its turn as `provider_turn_lost_on_restore`, ran a new turn live with full usage, and settled at once. **Expected behavior** An accepted run whose open turn's final report was lost with the provider process settles promptly. It records its observed usage as a lower bound, like a turn lost on restore. A complete report that does arrive after the restore is still used. **Steps to reproduce** `server/src/services/native-runtime/native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound` uses the event sequence from the kept database. It fails on master because the snapshot stays incomplete. ## What Changed - `createNativeTurnAccounting` observes `session.resumed` and marks every open turn as restored. A terminal event for a restored turn without a complete report marks it lost. The snapshot then closes it as a lower bound (`accountingUsageLost`), through the existing lost-on-restore path. A new `turn.started` clears the mark. - A lost terminal turn keeps its identity in the snapshot, so the executor's terminal-turn match certifies the final checkpoint as complete. This came from review. - The native session executor includes `session.resumed` when it rebuilds turn state from the run's event log after a controller restart. - `doc/SPEC-implementation.md` documents the case next to the existing lost-on-restore rule. ## Verification - Red on master, green here: `native-usage-accounting.test.ts > closes a turn that completed across a session restore without its final report, as a lower bound`. It also checks that replaying the history twice gives the same snapshot. - Red on the first revision, green after the review fix: `native-session-executor.test.ts > certifies a turn completed across a session restore as a lower bound (history replay: false|true)`. It drives the restored sequence through executor completion, both live and from the event history. - New guards that pass before and after: - `keeps a restored turn's own final report when it arrives after the restore`: a complete report after resume is used, not marked lost. - `keeps an incomplete turn pending when the session resumed before it started`. - The existing `keeps an incomplete provider report pending instead of trusting placeholder zeroes` still passes. - Related suites pass: `native-usage-accounting`, `native-usage-delivery.integration`, `closed-native-receipt-accounting` (107 tests), and the usage, accounting and receipt cases in `native-session-executor.test.ts` (151 tests). Server typecheck passes. - Live, restart-chaos L1, seed 20386, `--db-pressure 4`, codex and claude, K1, head `a6e521f06c`, clean worktree: `w-20261110T104332-s20386` passed 4/4. R1–R6 and the T-oracle passed for every task. `company_spend_settled` reported `pending_runs=0` (summary 85.30823 = cost events 85.30823), with zero serialization failures under about 397k pressure updates. On master the same seed failed every task on `company_spend_settled pending_runs=1` (`w-20261110T100747-s20386`, among others). ## Risks - A restored turn without a complete report now settles at its observed usage, flagged as a lower bound, instead of after 10 minutes as "closure timed out". In both cases the final report never arrived. The money recorded is the same observed usage; only the timing and the provenance label differ. - Scope: local Runner only. The executor fallback-recorder path sees `session.resumed`. The producer-owned delivery path for remote runners (`native-usage-delivery.ts`, `isNativeUsageAccountingEvent`) still drops it, so a remote ACPX/OpenCode turn restored mid-turn still waits for the closure timeout. That is a follow-up. - No schema or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.10 |
||
|
DottaandClaude
|
8b3348c6ba |
fix(runner): classify Claude's rejected selected model as an unavailable model (#20513)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner agents on the Claude (ACPX) provider run with the
model the user saves on the agent
> - When the provider cannot run that model, Paperclip should classify
the run as `native_provider_model_rejected`, ask the user to choose a
model, and let Retry finish the task
> - Claude Code accepts an unknown model at session start and reports it
only on the first turn, as an AIR `request` failure with a fixed title.
The runner reported it as an unknown provider failure
> - So the run failed as `native_provider_terminal_failed`, and the task
stopped with `provider_effect_inventory_unavailable` ("Automatic
recovery stopped. Recorded work is preserved…") without naming the model
> - This pull request classifies that Claude title as an unavailable
model, the category the native runtime already turns into
`native_provider_model_rejected`
> - The benefit is that an unknown Claude model asks for a model choice
and names it, as Codex does after #20488
## Linked Issues or Issue Description
**What happened?**
Config wave `w-20261110T094430-s207-cf` on `master` (
|
||
|
DottaandClaude Opus 5.5
|
57def75d01 |
fix(run-log): wait for a just-written journal head instead of failing the claim (#20509)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode, run logs use the `s3_segments` store: an object byte journal whose `head.json` names the current writer > - Each run start (and each resume after a restart) claims the journal: it reads the head and publishes the next generation with a conditional write > - On an S3-compatible store with read-after-write lag, the head that the previous generation just published can read as `NoSuchKey` > - The claim then tries to create a new head, the store rejects the create because the head exists, and the run fails with `object_journal_writer_conflict` > - This pull request makes the claim wait for the head to become readable when its read said "missing" but the store says it exists > - The benefit is that a run that resumes right after a restart survives a store that shows new objects a few seconds late ## Linked Issues or Issue Description **What happened?** In restart-chaos wave w-20261110T085328-s701 (local, A1/C1, K1/K2, `--storage s3 --storage-faults ST7`), 3 of 4 tasks failed R3 with `retry_needed (adapter_failed)`. Every failed run has `failureCause: object_journal_writer_conflict` and the error "The output journal is owned by another writer" at `object-byte-journal.ts` `publish` from `object-run-log-store.ts` `begin`. All six failures happened in the ST7 window (GET/HEAD of a just-written key answers 404 `NoSuchKey` for 24 s), right after back-to-back K1 restarts. The proxy lagged 34 `GetObject` reads. **Expected behavior** A resuming run claims the run log and continues. A store that is slow to show a new object must not look like another writer. **Steps to reproduce** Claim a journal, then claim it again while the store answers `NoSuchKey` for `head.json` for a short time. The second claim fails with `ObjectJournalConflictError`. ## What Changed - `server/src/services/object-byte-journal.ts`: `claim()` reads the head and claims from it. When the head read as missing and the create fails as a conflict, it waits with backoff (250 ms doubling to 4 s, up to 60 s) and claims again from the head it then reads. A conflict on a head it could read, or a deletion tombstone, still fails at once. - `server/src/services/object-byte-journal.test.ts`: a new test hides a just-written head for three reads. Before this change the second claim fails with "The output journal is owned by another writer". After it, the claim gets the next generation and keeps the committed bytes. ## Verification - `cd server && npx vitest run src/services/object-byte-journal.test.ts src/services/object-run-log-store-tail-read.test.ts src/services/object-run-log-store.test.ts src/services/run-log-store-cancellation.test.ts src/services/run-log-store.test.ts`: 83 pass. The new test fails before the fix and passes after. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. - Live: an ST7 rerun on a build with this fix follows in a comment. ## Risks - A claim on a journal that a truly concurrent first writer just created waits up to 60 s for that head and then claims after it. The caller already asserted execution authority, so it is the owner; it would claim after that head anyway once the head is readable. - A store that never shows the head makes the claim fail after 60 s instead of at once. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
7964433cd0 |
test(restart-chaos): arm a provider-fault Stop on the first live provider turn (#20512)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) injects provider message faults (PV1–PV3) to test recovery > - A board Stop task (S1) with a PV `interrupt_ack` fault needs the Stop to reach a live provider turn > - The harness clicked Stop only after a seeded wall-clock window, so a fast agent finished first and nothing was injected > - That task then failed `provider_fault_applied`, which made a clean wave exit 1 with no product failure > - This pull request arms the Stop on the first live provider turn for PV tasks > - The benefit is that PV S1 tasks inject their fault every time, so a wave result means what it says ## Linked Issues or Issue Description Refs #20498 (the product fix whose live waves showed this harness gap). **What happened?** In waves `w-20261110T082702-s212` and `w-20261110T091057-s213` (Claude, `--workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack`), task t03 (S1) failed only `provider_fault_applied` with `applied=0`. Its run finished before the seeded Stop window, so the Stop was missed and the fault never fired. Every product rule (R1–R6, U1) passed. The wave still exited 1. **Expected behavior** A PV S1 task injects its fault on every run. A missed injection is a harness gap, not a product failure. **Steps to reproduce** Run `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4`. The second S1 task often finishes before its Stop window. ## What Changed - `observer.ts`: the run-event cursor now tracks a live provider turn (`turn.started` with no `turn.completed/failed/cancelled/interrupted` yet). Run samples expose it as `providerTurnActive`. - New `stopArmed()`: a task with a provider fault arms its Stop on the first poll that shows a live provider turn. Other tasks keep the seeded wall-clock window, unchanged. - A runner replacement or stop clears the live-turn mark, as it already clears open tools. - A PV Stop still counts as missed when the run ends with no live turn seen and the task rests after the window. - The continue comment still waits for the seeded window and `STOP_CONTINUE_AFTER_MS`. - README note under the provider fault section. No product code changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. - New tests in `observer.test.ts`: - `S1 with a provider fault: stops on the first live provider turn, not the wall-clock window` - `S1 with a provider fault: does not stop after a turn terminal, and stops once when the next turn starts` - `S1 with a provider fault: records a missed Stop when the run ends before any live turn and the task rests after the window` - `noteLivenessEvent: a runner replacement or stop clears the live provider turn` - `stopArmed: a provider-fault Stop needs a live provider turn; others keep the window` - Red against master `observer.ts`: the first three new tests fail. Green with the change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - No live wave was run (the wave queue was full); the coordinator asked for a test-only change. ## Risks - Low. This is test-harness code only. - A PV S1 Stop can now land earlier in the turn than before. That is the intended test point for `interrupt_ack`. - A turn shorter than the poll interval (`--poll-ms`, 2–5 s) can still be missed. The README states this limit. - If a provider never records `turn.started`, the PV Stop waits until the task rests and is then recorded as missed, the same result as before. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
dc9971bdb8 |
fix(native): keep another server's orphan sweep off a lease its owner took back (#20508)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an executor lease on its `native_run_finalizations` row. #20474 lets the same owner take back its own lease when the lease lapsed only because the host slept. > - That take-back is safe only if every other actor that acts on a lapsed lease records its action under the coordinator row lock first. The plan `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md` lists the actors that do not. Gap 2 is another server's orphan sweep. > - The sweep reads the lease as lapsed. Later it writes the ownership-unverified marker on the run, without the coordinator lock and without a new check of the lease. If the owner wakes and takes its lease back between the read and the write, a healthy run gets the marker. > - The board then shows "automatic recovery is blocked" on a run that works. After the next host sleep, the owner cannot take its lease back, so the turn is interrupted and the user must click Retry. The owner's rule is that a task interrupted by sleep must resume by itself. > - This pull request makes the sweep write its marker under the coordinator row lock, and only while no executor lease on the run is live. > - The benefit is that exactly one side wins: the sweep marks a run whose owner did not come back, and it leaves alone a run whose owner took its lease back. ## Linked Issues or Issue Description Refs #20474 **What happened?** Two servers share one database. Server A's host sleeps, so the executor lease of its native run lapses. Server B's orphan sweep reads the lapsed lease. Server A wakes and takes its lease back (#20474). Server B then writes `native_execution_ownership_unverified` on the run that A still owns. **What did you expect?** Server B writes the marker only if the lease is still lapsed when it writes. One side wins. **Steps to reproduce** `server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` holds the owner's take-back open, runs the real `heartbeatService(db).reapOrphanedRuns()` until its write waits on a lock, then commits the take-back. Before this change the run ends with `errorCode: native_execution_ownership_unverified` and the owner's next take-back fails with `native_session_lease_lost`. ## What Changed - `setRunStatusFromLive` (`server/src/services/heartbeat/run-lifecycle.ts`) has a new option `onlyWhileNativeLeaseLapsed`. It locks the coordinator row first. This is the same order that `lockNativeExecutionLeaseForReacquire` uses. The UPDATE then requires that no executor lease on the run is live. - `markNativeOwnershipUnverified` (`server/src/services/heartbeat/recovery.ts`) takes `leaseLapsed`. The orphan sweep in `recoverActiveExecutions` sets it when it read the lease as lapsed or unset. Other callers do not change. They keep the same call shape. - A new regression test, red before the fix: `server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`. It also checks that the sweep still marks a lapsed run whose owner did not come back, and that the late owner stays fenced out. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: gap 2 is marked closed without the epoch design. ## Verification - `cd server && pnpm exec vitest run src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` — red on `origin/master` (the run keeps the marker), green with this change. - `cd server && pnpm exec vitest run src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts src/services/heartbeat/recovery.test.ts src/services/native-runtime/native-execution-lease-reacquire.test.ts src/__tests__/heartbeat-status-preserving-telemetry.test.ts` — 129 passed. - `cd server && pnpm exec tsc --noEmit -p tsconfig.json` — clean. - Restart-chaos time-fault wave on this branch: see the wave comment below. ## Risks - Low risk. Only the orphan sweep's marker write changes, and only for a run whose lease it read as lapsed. The sweep now takes the coordinator row lock before the run row. The owner's take-back uses the same order, so the two cannot deadlock. - If the owner took the lease back, the sweep now writes nothing and appends no event. The next sweep sees a live lease and defers, as before. - The other gaps in the plan (workspace finalization, shared-workspace holder detection, board copyback confirmation) do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1a89fca1ce |
test(restart-chaos): add --rr-stop-on recovering so RR follow-up stops land inside recovery (#20511)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - #20480 added repeated restarts (RR1–RR3): a K1, then more restarts while the replacement recovers. > - In the first four live waves, no follow-up stop landed inside recovery. Every stop saw `/api/ready` report `ready`. > - The saved receipts show why. `/api/ready` reports `recovering` only 15–80 ms before `/api/health` answers, and `ready` 0.25–0.5 s after it. A gap measured from health, even 1 s, always lands after recovery. > - This pull request adds `--rr-stop-on recovering`. Each follow-up stop fires on the first `recovering` status, polled from the replacement's spawn. > - The benefit is that RR waves now test a restart that interrupts recovery itself. ## Linked Issues or Issue Description Refs #20480 **What is the problem?** RR follow-up stops wait a gap from the replacement's health. On a local host recovery ends about 0.3 s after health, so no stop in four waves (14 tasks, 18 follow-up stops) landed inside recovery. **What is the solution?** Add `--rr-stop-on recovering`. It replaces the gap: the harness polls `/api/ready` every 50 ms from spawn and stops the replacement on its first `recovering` status. This is the phase poll that K6 already uses. **Why is it needed?** The deploy loops of 2026-11-08 and 2026-11-09 restarted the server while it recovered interrupted runs. The harness must reach that window to test it. ## What Changed - `repeated-restarts.ts`: `--rr-stop-on recovering` is parsed by `parseRepeatedRestartOptions`. It needs an RR fault and does not combine with `--rr-gap-seconds` or `--replay`. `runRepeatedRestarts` takes `stopOn`: with it, no gap is slept, each stop waits on the new `waitForStopPhase` hook, and only the last replacement is waited on to health. - `supervisor.ts`: `waitForStopPhase` reuses the K6 phase poll (`waitForCrashLoopPhase`, 50 ms). The record carries `stopOn`. - `types.ts`, `config.ts`, `main.ts`: the `rrStopOn` option, its flag, and one line to pass it to the supervisor. - The evidence shows each stop's `/api/ready` status (`phaseAtStop`). With `stopOn`, `gapMs` is null, and the earlier entries have no `healthyAt`. - README: the option and the measured timings. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. New tests cover option parsing, stops on the first `recovering` with no sleep, health waits only on the last replacement, a `ready` stop when recovery was too short to see, an exit before a status, and a missing watcher. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - A live wave from this branch (`--workloads A1,B5,C1 --faults RR1,RR3 --rr-stop-on recovering`, codex, 4 tasks) follows; its results go to a comment. ## Risks - Harness-only change; no product code changes. RR waves without the flag are unchanged. - A stop on `recovering` lands before the replacement answers health. This is the K6 behavior, and the server log shows it. A replacement that never reports a status fails the fault after the 5-minute ready timeout, as K6 does. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
a09093a5c5 |
fix(native): resume a run whose turn.start a restart cut off before Codex created the turn (#20507)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run sends `turn.start` to its runner, and the runner sends `turn/start` to Codex. The runner marks the start "ambiguous" before it sends the request > - A control-plane restart can kill the runner and Codex while that request is in flight. After the restart, neither journal says whether Codex created the turn, and the resumed thread has no active turn > - The restored runner then failed closed on every same-run resume with "cannot safely recover an ambiguous Codex turn start without an active replacement turn". The answer never reached the agent > - The next run on the task was refused at the runner-state fence (`prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`) before it asked for a runner. Retry after that run then waited forever on `process_identity_missing` for a process that never existed > - This pull request lets the runner prove from Codex's own thread history that the start never created a turn, and then resume and start the turn once. It also lets Retry continue after a run that the runner-state fence refused before launch > - The benefit is that a turn start cut off by a restart continues by itself when Codex never saw it, and a person's Retry is no longer a dead end ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T033426-s108`. Command: `--env local --adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Task t03 (Q2): the answer run `92a57f50` resumed after the first restart. A second K1 restart landed 0.25 s after `session.resumed`, while its `turn.start` was in flight. Each same-run resume then failed with `PRP command session.snapshot failed: cannot safely recover an ambiguous Codex turn start without an active replacement turn`. When the prerequisite finished, the next run `0e6aa9d1` failed before launch with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The accepted Retry (202) never created a run: the task stayed on `retryWait.reason=process_identity_missing` until the wave timed out (`R3:retry_no_new_run`). **Expected behavior** If Codex never created the turn, the run resumes and starts it once by itself. If that cannot be proven, a person's Retry starts a new run instead of waiting forever. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-turn-start.test.ts` on master (case "before" fails with the wave error). ## What Changed - Runner (`codex_provider.rs`, `provider_backend.rs`): when the restored provider state has an ambiguous turn start, no active turn before the loss, and the resumed thread has no active turn, the runner reads the newest turn of the thread (`thread/turns/list`, limit 1, metadata only, the same request `read_thread` already sends). If there is no turn, or the newest turn is one Paperclip already settled, Codex never created the turn. The runner clears the ambiguity, emits a `harness.diagnostic` with code `ambiguous_turn_start_not_created`, and restores an idle session. The old `turn.start` stays `execution_indeterminate` and is never executed again. The same-run resume then starts the run's turn once, through the existing "recovered session without an active turn" path. - Any other history (a turn Paperclip never settled, an in-progress turn, an incomplete page, a non-Codex provider) keeps today's fail-closed error. The probe runs only on this recovery path. The normal turn path does no new work before `turn.start`. - Server (`native-local-process-stop.ts`): `nativeRunNeverLaunched` also accepts a failed native run whose error is the runner-state fence refusal `runner_state_identity_mismatch: prior_owner_*` (or `legacy_prior_owner_*`). The executor throws it only while it migrates another run's saved state, before it requests a launch. The existing evidence check still applies: any launch request, identity record, `adapter.invoke` or runner event keeps the run held. Other runner-state errors (for example `authority_indeterminate`) stay held. - Fake Codex (`fake-codex-app-server.rs`): keeps the thread's turn history in a `<state-file>.turns` sidecar, so `thread/turns/list` returns history like real Codex (newest first, honours `limit`). New `--hang-turn-start <file>` holds `turn/start` "before" or "after" the thread records the turn, to model a kill in flight. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-turn-start.test.ts`. A real runner and fake Codex start a turn; both processes are killed while `turn/start` is in flight. "before": the resumed run completes, the journal holds the old `turn.start` (`execution_indeterminate`) and one new completed `turn.start` with the same task text, and Codex saw exactly one more `turn/start`. "after": recovery still fails closed and no new `turn/start` is sent. - Rust: `ambiguous_turn_start_that_never_created_a_turn_recovers_and_starts_once` and `ambiguous_turn_start_that_created_an_unsettled_turn_still_fails_closed`. Both rewrite saved provider state the way a master runner wrote it, so they also cover state saved before this change. - `explicit-native-continuation.test.ts`: five new rows for the runner-state fence. Admitted: refused before launch (with and without a coordinator). Held on `process_identity_missing`: refused after a launch request, refused after a runner event, and a run that failed its own `authority_indeterminate` check. - `docs/durable-recovery.md`: describes both behaviours. ## Verification - Red/green, real runnerd: on master (the staged master runnerd), `runnerd-codex-transport-cut-turn-start.test.ts` "before" fails with `PRP command runner.drain failed: ... cannot safely recover an ambiguous Codex turn start without an active replacement turn`, the wave error. With this change, both cases pass. - Red/green, Rust: without the `provider_backend.rs` change, `cargo test -p paperclip-runner-core --test codex_provider ambiguous_turn_start_that` fails the "never created" test. With it, both pass. The full `cargo test -p paperclip-runner-core` passes. - Red/green, server: without the `native-local-process-stop.ts` change, the two "refused at the prior-owner runner-state fence" rows fail with `process_identity_missing`. With it, all 14 rows pass. - Related suites pass: 11 Runner transport files, 419 tests (`runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `-cut-turn-start`, `-settlement`, `-suspension`, `runnerd-restart-continuation`, `runnerd-saved-continuation`); server `explicit-native-continuation`, `quarantined-task-retry` and `issue-confirm-run-stopped-routes`, 3 files, 470 tests; full `cargo test -p paperclip-runner-core`. - `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit` and server `tsc --noEmit` pass. - Pack `contractDigest` is unchanged: `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). - Live proof: the wave command above on this branch. Results follow in a comment. **No duplicate provider work.** The runner starts a new turn only when Codex's own history proves the cut-off start created nothing. The old command is never executed again (the runner refuses an `indeterminate` command). A recorded turn that Paperclip never settled keeps the run failed closed. ## Risks - Low-medium risk. The runner change runs only while it restores a session whose saved state has an ambiguous turn start and no active turn. Before this change, that run could never continue. - The proof relies on Codex recording a turn in the thread before it runs anything. If Codex accepted the request but died before it wrote the turn, the runner treats the start as not created and starts the turn once. Codex writes the user turn to the thread when it starts the turn, before it asks the model for output, so no model output or tool call is expected without a recorded turn. - If the thread's newest turn is not in the runner's settled set (for example after a provider identity epoch rollover forgot older identities), the runner keeps failing closed, as today. - **Open decision (not changed here):** when Codex did record the cut-off turn, it may have done work. This pull request keeps the run failed closed and does not replay or adopt that turn. Continuing is a person's decision through Retry or a new message. - The server change trusts the executor's own error text `runner_state_identity_mismatch: prior_owner_*`, together with the absence of every launch and runner event. A run that requested a launch keeps every stop proof. - **Runner release:** the pack `contractDigest` does not change. Local runners use the runnerd built with the app, so the fix applies with this release. - Not covered: the related `authority_indeterminate` variant in the same wave (t04, Q1) is a separate cause; its Retry already worked in that wave. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20477, #20482 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.9 |
||
|
DottaandClaude Opus 5.5
|
7eae4bc2ea |
test(restart-chaos): add ST4 partial multipart and ST7 read-after-write lag storage faults (#20504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` puts a fault proxy between the server and a loopback MinIO (ST1–ST3, ST5/ST5R) > - Remote workspace snapshots over 16 MiB upload as S3 multipart uploads, and saves are read back right after they are written > - No fault hit single upload parts or the completion, and no fault made a just-written object briefly invisible > - Nothing checked that a wave left no orphaned multipart uploads or that every committed snapshot reads back > - This pull request adds ST4 (partial multipart) and ST7 (read-after-write lag), and a teardown store audit > - The benefit is that waves can prove snapshot saves and restores survive part failures and read lag without data loss or leaked uploads ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults treat every request the same. They cannot target one multipart part or the completion, and they cannot lag reads behind writes. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** A wave cannot fail the Nth `UploadPart` of a snapshot upload, cut a part body mid-transfer, or fail `CompleteMultipartUpload`. A wave cannot make a `GET` or `HEAD` answer `NoSuchKey` right after a `PUT`. No wave checks for incomplete multipart uploads or unreadable committed snapshots. **Proposed behavior** `--storage-faults ST4` fails a seeded subset of snapshot upload parts (500/503, reset, or reset after 1–4 MiB of the body), and sometimes the completion. `--storage-faults ST7` answers 404 `NoSuchKey` to reads of an object for 5–30 s after its write succeeded. With either fault, the wave reads every committed workspace snapshot back from MinIO (size and SHA-256 against its row) and, after the server stops, lists incomplete multipart uploads under the wave prefix. Either problem fails the wave. **Reason and benefit** These are the storage failures that can lose a workspace or leak storage cost without any visible error. ## What Changed - `storage-s3-ops.ts` (new): names the S3 operation from the method and query, and reads the part number. - `storage-multipart.ts` (new): ST4 rule planning. - `storage-lag.ts` (new): ST7 rule planning, and a tracker of successful `PutObject` and `CompleteMultipartUpload` writes per key. - `storage-audit.ts` (new): teardown audit (committed snapshot read-back, incomplete multipart uploads), `storage-audit.json`, and a `Storage audit` section in `SUMMARY.md`. MinIO lists multipart uploads only for an exact key or no prefix, so the audit lists the wave-owned bucket and filters by the wave prefix. - `storage-proxy.ts`: rules can match `operations` and `partNumbers`. New faults `truncate` (reset after N body bytes) and `read_after_write_lag`. `storage-proxy.json` reports `lagged`. - `storage-chaos.ts`, `main.ts`: registry lines for ST4/ST7 and the audit calls. - README: ST4/ST7 rows and the audit. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 423 tests pass. The new `storage-multipart-lag.test.ts` fails before this change (the modules and kinds do not exist) and passes after. It includes real MinIO tests with the AWS SDK: part 2 fails once with 500, a part body is cut after 1 MiB, the completion fails once, the audit finds the incomplete upload and then finds none after completion; a just-written object answers `NoSuchKey` to `GET` and 404 to `HEAD` until the lag passes; the snapshot check reports `sha256_mismatch` and `object_missing`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves run from master after merge (local ST7, Daytona R1 ST4 and ST7). Results go in a follow-up comment. ## Risks - Test harness only. No product code changes. - The audit runs only on waves with ST4 or ST7, so other storage waves do not change. - The snapshot check verifies the stored ciphertext against its row. It does not decrypt, because the key lives in the instance. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0b20ea6fe9 |
fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Before a run starts provider work, Paperclip reserves its budget and binds a usage recorder. Both write the run's `heartbeat_runs` row. > - Run-log appends, output progress and lifecycle events update that same row while the run prepares and streams. > - #20385 fixed this race for the standalone usage-receipt write only. Budget admission and the recorder handoff still take a repeatable-read snapshot before they lock the run row. > - Each concurrent row update makes that snapshot stale. Four retries run out with SQLSTATE `40001`, and the run ends `adapter_failed` with raw SQL as its error. > - This pull request applies the #20385 pattern (READ COMMITTED under the existing locks) to those writers and to the final dispatch gate. It also gives exhausted retries a bounded cause. > - The benefit is that busy runs start and account normally, and a remaining failure names its cause. ## Linked Issues or Issue Description Refs #20385. Builds on #20483 (bounded failure cause and `withoutSqlStatementText`). **What happened?** The restart-chaos L1 workload with `--db-pressure 4` (harness writers that update each running run row, one row per statement, like a fast stream) failed on master. In one wave, 0 of 4 tasks passed. Every first run failed in `reserveRunBudget` before spawn. In an earlier wave, a run failed in `createRunUsageRecorder`. The stack in both cases is `could not serialize access due to concurrent update` (`40001`) on the run-row `FOR UPDATE`. The board `runError` showed the full `select ... from heartbeat_runs ... for update` statement. The server log line `heartbeat execution failed` dropped the database cause. Realism: the harness writers are synthetic, but the product has the same writers on the same row. `appendHeartbeatRunEvent` increments `next_event_seq`, output progress updates `last_output_*`, and lifecycle events land while a run prepares. On a hosted database the window between snapshot and row lock is wider than locally, so a busy run reaches the same race without the harness. **Expected behavior** Budget admission and the recorder handoff commit while other writers update the run row. Budget invariants stay intact: one reservation per run, a reached hard stop still refuses work, and committed holds count against later admissions. Usage receipts are journaled once. If retries still run out, the run shows a bounded cause without SQL, and the server log records the SQLSTATE. **Steps to reproduce** Run `server/src/services/run-accounting-contention.test.ts` against the parent commit. The deterministic cases commit a real `appendHeartbeatRunEvent` after the writer's transaction checks the company and before it locks the run. Six of the eight cases fail with `40001`. ## What Changed - `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow the shared admission fences, the agent `FOR SHARE` lock and the run-row `FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic). The `budget_reservations_run_idx` unique index and the run lock still prevent a second hold. - The final dispatch gate's accounting phase (`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks` in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its ownership phase already read the same staleness facts at READ COMMITTED under the same issue and run locks. Retry promotion and stale-run cancellation keep their repeatable-read snapshot. The first L1 proof wave found this writer exhausting `40001` once the other three were fixed. - `createRunUsageRecorder` runs its receipt-source handoff and its capture-failure fence at READ COMMITTED. Both lock the run row first, the same as `persistUsageReceipt` after #20385. - `withOperationAccountingTransaction` raises `AccountingContentionError` when retries run out. It has a bounded message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the database error, and a warning logs both. - `failRun` shows that bounded message. The code stays `adapter_failed`, so chat and recovery retry paths still apply. #20483's `failureCause` records `40001`. All other SQL text handling comes from #20483 (`withoutSqlStatementText`), unchanged here. - `doc/DATABASE.md` documents the isolation choice and the failure contract. ### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write run-scoped rows | Writer | Status | Notes | | --- | --- | --- | | `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ COMMITTED under the run lock. | | `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in waves (`budget-reservations.ts:90`). | | `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here | Seen in waves (`usage-receipts.ts:572`). | | `createRunUsageRecorder` capture-failure fence (`persistFailure`) | Fixed here | Same shape: run lock first, runs mid-stream. | | Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe | A default READ COMMITTED transaction with `FOR UPDATE` on the coordinator row. It cannot raise `40001`. The wave errors there were `native_finalization_missing`, a symptom of the failed reservation (no coordinator existed yet). | | Final dispatch gate, accounting phase (`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`, `fea0ee5a`). Its ownership phase already used READ COMMITTED under the same locks with the same staleness reads. | | Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`, `cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at repeatable read: `loadGateFacts` combines wakes, comments, dependencies, tree holds and repair state in one decision. They act on queued or scheduled runs, not streaming ones. Not seen failing in waves. If retries run out, they now fail with the bounded cause. | | Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up | Seen with `40001` in a wave log. Leaves the run pending; `reconcileRunCosts` retries it durably. Multi-aggregate (coordinator, receipts, images, closure). | | Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) | Safe | Insert `cost_events` only; the run row is referenced, not locked or updated. | | Budget enforcement and evaluation (`budgets.ts`), billing reconciliation, accounting-integrity repair | Safe | Lock policy, invoice or company rows, not a streaming run row. Run reads are plain selects. | | Browser use, decisions, prompt-injection, fast responses | Safe | Write their own invocation/reservation rows, not `heartbeat_runs`. | | Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or no run-scoped writes. | ## Verification - Red on the parent commit, green on this branch: `server/src/services/run-accounting-contention.test.ts` (real embedded PostgreSQL). - `reserveRunBudget > reserves once when run events commit between its company check and its run lock` (red: `40001`) - `reserveRunBudget > reserves every run while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > counts every committed hold, so successive admissions never exceed the budget` (red) - `reserveRunBudget > still refuses a run once the hard stop is reached` (red: `40001` instead of `409`) - `createRunUsageRecorder > binds its receipt source when run events commit between its company check and its run lock` (red: `40001`; green asserts the spooled receipt is journaled exactly once) - `createRunUsageRecorder > binds while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > never creates a second reservation for one run when two dispatches race` (invariant; passes before and after) - `final dispatch gate > admits and hands off a reserved run when run events commit between its company check and its run lock` (red: `AccountingContentionError` after 4 × `40001`) - `createRunUsageRecorder > persists its capture-failure fence once when run events commit between its company check and its run lock` - `final dispatch gate > admits while back-to-back writers update the run row` (red: `40001`) - `exhausted retries > fail with a bounded, SQL-free cause that keeps the SQLSTATE` (new contract) - `run-completion.test.ts > saves a bounded cause and keeps the retryable generic code when accounting retries are exhausted` (saved code, message and `failureCause`, and log fields, through `failRun`) - Related suites pass: `usage-receipts-read`, `admission-independent-of-cleanup`, `budget-live-attempts`, `idle-accounting-work`, `budget-enforcement-scoping`, `closed-native-receipt-accounting`, `budget-evaluation-work`, `run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126 run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown exceeded 30 s on this loaded host, the known local flake). - `pnpm --filter @paperclipai/server typecheck` passes. - Latency of `reserveRunBudget` on the normal path (no contention, embedded PostgreSQL, 350 measured reservations per run, 3 runs each): before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50 14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the statements are the same, only the isolation level changes. Under contention it removes the retry backoff. - Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and claude, K1): - Master before this change: `w-20261110T050818-s20386`, 0/4. Every first run failed in `reserveRunBudget` with `40001`. - Head `90530789b0` (first three writers fixed): `w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude failures (t02, t03) trace to the dispatch gate, now fixed. Those runs already showed the bounded cause, with no SQL on the board. The failure was logged with `sqlState` 40001. No `reserveRunBudget` or recorder `40001` remained. - Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`. All 4 tasks reached done with no retry and R1–R6 passed. There were no failed runs and zero `40001` in the server log, under 310k pressure updates. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. That pending run is t02's continuation run `601ae80a`, cancelled before provider work as `issue_terminal_status`, whose accounting debt was not yet settled when graded about 15 s later. It is not a contention failure. Caveat: I edited the worktree during this wave (`worktree_changed_mid_wave`), so a clean wave on the final head follows. - Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`. All 4 tasks reached done with no retry, and R1–R6 passed for each. There were no failed runs and zero serialization failures in the server log, under about 523k pressure updates. Each task still fails only the T-oracle `company_spend_settled` (`pending_runs=1`). It is the same cancelled pre-provider continuation run (`issue_terminal_status`), whose accounting debt was still unsettled at grading. That is a separate settlement-timing follow-up, not database contention. Master was 0/4 on the same seed. ## Risks - READ COMMITTED gives each statement its own snapshot. Admission reads policy, scope and exposure in separate statements. The shared company/project fences and the agent share lock exclude concurrent pause and policy edits, and exposure stays one statement, so each read is at least as fresh as the old snapshot. - Concurrent admissions of different runs stay best-effort, as `doc/SPEC-implementation.md` already states ("concurrent starts can exceed the same observed balance"). This change does not widen that window: each statement snapshot is newer than the old transaction snapshot. It does not add a company mutex. - A run that fails after exhausted retries now records `accounting_contention` instead of `adapter_failed`. Retry rules that key on `adapter_failed` spawn-like messages did not match this failure before, so no automatic retry changes. - No schema, migration or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
914e3ae14f |
fix(runner): settle ACPX turns when the cancel reply is lost (#20498)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Claude agents run through the native Runner and the ACPX sidecar > - A saved question or a board Stop cancels the active turn with `turn.cancel` > - The Runner waited 30 s for the sidecar's reply to that cancel, and the Runner serves controller commands one at a time > - When that reply was lost, the controller's shutdown commands timed out first, so the run failed and the task needed a human Retry > - This pull request bounds that wait, keeps the transport open, and lets the provider's own turn terminal settle the turn > - The benefit is that a lost cancel reply on Claude settles the same way it already does on Codex (#20336) ## Linked Issues or Issue Description Refs #20336 (Codex equivalent: settle a saved handoff after verified shutdown). Related, no overlap: #20477, #20482, #20483. **What happened?** The restart-chaos harness dropped the ACPX sidecar's reply to the Runner's `turn.cancel` (fault PV2, target `interrupt_ack`). A Claude task that saved a question (Q1) failed with "The waiting state is saved; confirming shutdown of the previous execution is still pending". Its next attempt failed `native_session_cleanup_quarantined`. A Claude task stopped by the board (S1) ended `blocked`. Retry was accepted but started no run. Cause, in `packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs` (`interrupt_turn`, master line ~800): the cancel used the ordinary sidecar request with the 30 s bound from `acpx_provider_backend.rs:539`. While it waited: - `turn.stop` and `runner.suspend` stayed queued. The controller's 15 s governed drain plus its 10 s close grace ran out first (`runner did not durably suspend before checkpoint`, suspend command `pending`). - At 30 s, `fail_closed` killed the sidecar and dropped the provider's buffered turn terminal. The poll path then recorded `turn.failed` (`acpx_provider_transport_failed`) and `provider_exit_unconfirmed` (`acpx_provider_backend.rs:2716-2738`). `turn.stop` returned `already_settled`, and `runner.suspend` refused with `ACPX provider lifetime cleanup is not yet proven` (`acpx_provider_backend.rs:2613-2621`). Codex passes the same fault because its Runner does not block on the interrupt reply. **Expected behavior** A lost cancel reply must not fail the run. The saved question must settle as `succeeded`. A board Stop must end the run as cancelled with a proven suspension, so the next turn can start. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4` Failing wave on master: `w-20261110T060120-s207` (t04, t05). **Deployment mode** Native Runner with the Claude ACPX profile, local. Hosted sandboxes run the same Runner code. ## What Changed - `AcpxSidecarTransport::request_turn_cancel`: sends `turn.cancel` and waits only to a caller deadline. On a missing reply it keeps the transport usable and retains the request ID, so a late reply is skipped in both the response and the event-poll paths. A rejected reply and transport errors keep their existing fail-closed result. - `AcpxProviderSession::interrupt_turn`: waits at most 5 s (or the shorter configured request bound) for the cancel reply. On a missing reply it retires the runtime from warm reuse and returns. The provider's turn terminal still arrives through polling, and `turn.stop` still reaps a provider that never stops. - `AcpxProviderSession::deliver_tool_result`: a board Stop can land while a semantic tool runs. The session's own `turn.cancel` then retires the call in the sidecar before its result arrives, and the turn terminal may still be buffered. The sidecar rejects the result as stale (`ACPX_TOOL_CALL_STALE`). When this session sent `turn.cancel` for that turn, the Runner now keeps the durable result, like a result for an already closed turn. Every other rejection still fails closed. (`AcpxSidecarTransport::request_tool_resolve` reports whether a rejection was the stale-call case.) - A repeated interruption while a cancel is unanswered sends nothing and returns at once (review finding). - `fake-acpx-sidecar`: new `--cancel-reply lost|late` flag. After that cancel, `tool.resolve` is rejected as stale. The provider ends the turn, and the reply is dropped or written just before the next request. - Three regression tests (below) and a short section in `packages/paperclip-runner/docs/durable-recovery.md`. No TypeScript, server, schema or protocol change. The normal start path is unchanged; nothing waits before turn start. ## Verification Red/green (red runs used the master versions of `acpx_provider_session.rs` and `acpx_sidecar_transport.rs`): - `cargo test -p paperclip-runner-core --test acpx_warm_attachment a_lost_cancellation_reply_still_settles_the_turn_and_suspends_the_runner` - Red: after 30.4 s, `ACPX turn interrupt failed: ACPX sidecar request timed out at turn.cancel` (the live error). - Green: `turn.interrupt` returns in about 5 s, then `turn.interrupted` and a `cancelled` `run.terminal` (no `turn.failed`), `turn.stop`, and `runner.suspend` completes. - `cargo test -p paperclip-runner-core --test acpx_provider_turns lost_cancellation_reply_keeps_the_turn_terminal_and_the_transport` - Red: `interrupt_turn` fails at the request bound. - Green, for a lost reply and for a late reply: the terminal is polled, warm reuse is refused, and the next request (`session.close`) succeeds after the late reply arrives. A late reply causes no second settle and no second prompt. - `cargo test -p paperclip-runner-core --test acpx_provider_turns a_tool_result_for_a_call_retired_by_our_cancellation_does_not_fail_the_transport` - Red (guard disabled): `ACPX sidecar command tool.resolve was rejected (retryable=false, classification=provider_tool_call_retired)`, the same error as live wave task t03 below. - Green: the result is kept, the pending tool clears, and the turn terminal is polled after it. - `cargo test -p paperclip-runner-core` (whole crate): all pass. Existing `turns-tool-resolution-rejected` coverage still fails closed when no cancel was sent. `cargo fmt --check` passes. - Pack contract digest: unchanged (`sha256:3164357d…` before and after, from `scripts/contracts/contract-fingerprints.mjs`). - Live wave 1, `w-20261110T072706-s211` (commit `d91bff1e9f`, the first fix only), `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --tasks 4`: 3 of 4 pass. Both Q1 tasks pass (the asking run `succeeded`, no failed run, answer in the brief). One S1 passes. S1 task t03 failed: Stop landed during a `write_task_document` call and hit the stale tool-result error above. The second commit fixes that case. - Live wave 2, `w-20261110T082702-s212` on head `76dd57515d`, same flags: every task passes R1–R6 and U1. Both Q1 tasks pass with the fault applied (asking run `succeeded`, no failed run). S1 t01 passes with the fault applied (Stop took effect, task `done`, no Retry). S1 t03 fails only `provider_fault_applied` (`applied=0`): the agent finished before the Stop window, so nothing was injected and the task proves nothing either way. - Live wave 3, `w-20261110T091057-s213` on head `76dd57515d`, same flags: the same result. t01 (S1), t02 and t04 (Q1) pass with the fault applied; t01's Stop took effect (`status=cancelled`), task `done`, no Retry. t03 again fails only `provider_fault_applied` (`applied=0`, the run finished before the Stop window); all its R1–R6 and U checks pass. - Across waves 2 and 3 on the final head: 6 of 6 tasks with the fault applied pass (4 Q1, 2 S1). No product rule failed. The two misses are harness timing (no injection), not product failures. ## Risks - A healthy cancel reply that takes longer than 5 s now retires the warm runtime. The next turn starts a fresh provider process for that session. The observed healthy reply time is about 1 s. - If the provider never stops and the reply is lost, the turn stays active until `turn.stop` terminates the provider and proves the lifetime fence. If that proof fails, the existing settlement-pending and quarantine states are unchanged; this PR adds no success path without proof. - Runner binary change: hosted Daytona images need a normal image rebuild to pick up the fix. The pack `contractDigest` does not change, so existing image packs stay reusable. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
82abb3ea04 |
fix(native): classify a Codex API-key unknown model as a model rejection (#20488)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner agents run on a provider (here Codex) with a model the
user saves on the agent
> - When the provider rejects that model, Paperclip should classify the
run as `native_provider_model_rejected`, ask the user to choose a model,
and let Retry finish the task
> - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON
envelope). With API-key auth, Codex returns a different terminal error
for an unknown model
> - So the run failed as a generic `adapter_failed` ("Native session
failed"), and recovery opened a reconciliation incident ("Recovery
needed") for a turn that never did any work
> - This pull request recognizes the real API-key shape, bound to the
configured model, and names the rejected model on the task
> - The benefit is that the task asks for a model choice instead of a
reconciliation, and Retry completes the task after the user picks a
valid model
## Linked Issues or Issue Description
**What happened?**
The restart-chaos config fault CF3 `model_id` saves a Codex agent with
the model `chaos-missing-model-<nonce>`. The run stays "running" for
about 60 seconds, then fails with `errorCode: adapter_failed`, "Native
session failed". The task shows "Recovery needed", cause
`native_continuation_requires_reconciliation`. The grader fails as
`CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01
and `w-20261110T053831-s205-cf` t02).
Root cause: the wave runs Codex with an OpenAI API key. I drove the
bundled `codex app-server` (0.160.0) directly with an unknown model:
- With API-key auth, Codex retries the Responses API 404 ten times
(about 59 s), then fails the turn with `{"message":"unexpected status
404 Not Found: The model \`<id>\` does not exist or you do not have
access to it., url: https://api.openai.com/v1/responses, cf-ray: …,
request id:
req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`.
- With ChatGPT auth, the turn fails in about 3 s with `message` =
`{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The
'<id>' model is not supported when using Codex with a ChatGPT
account.","code":"model_not_found"}}`. Paperclip already classified this
one.
`createNativeProviderFailureObservation` parsed only the JSON envelope,
and the runner's provider-failure category for an HTTP 404 is `unknown`.
The terminal therefore stayed unclassified.
**Expected behavior**
The run is `native_provider_model_rejected`, the task shows `userAction:
"choose_model"` and names the model, no reconciliation incident opens,
and Retry completes the task after the user picks a valid model.
**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex
--references model_id --tasks 1 --concurrency 1 --restarts K2 --seed
206` on `master` (with `OPENAI_API_KEY` set).
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
8a8abdb7b9 |
fix(recovery): retire the ledger row of an abandoned workspace save (#20502)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save their files to object storage. Each save
writes a row in the `execution_workspace_snapshots` ledger. The row
stays `pending` until the upload publishes it.
> - A restart that kills the server during a save leaves the save
unfinished. The ownership reconciler settles the save operation as
`remote_workspace_save_abandoned`. It does not touch the ledger row.
> - The row then stays `pending` for good, and its `updated_at` never
moves. The `remote_snapshot_ledger` ownership kind has no settle path,
so every later ownership report flags it as an `I2_settled` violation.
> - This pull request retires that row to a terminal state once no retry
can reopen its save, and gives `remote_snapshot_ledger` a settle path.
> - The benefit is a clean ownership report after a restart during a
save, and a ledger that tells the truth about uploads that will never
finish.
## Linked Issues or Issue Description
**What happened?**
A Daytona R1 × K1 restart-chaos wave (`w-20261110T071047-s7101`,
`--storage s3`) killed the server during the initial workspace save of 3
runs. The save operations ended `failed` /
`remote_workspace_save_abandoned`. Their `execution_workspace_snapshots`
rows stayed `pending` permanently, with no object key and no upload
attempt. The workspaces later committed newer snapshots (authority epoch
3, row epoch 1). Every ownership report afterwards flagged 3
`I2_settled` violations of kind `remote_snapshot_ledger`.
**Expected behavior**
When a save is abandoned and its run and writer epoch have ended, its
pending ledger row reaches a terminal state, and the ownership report
shows no `I2_settled` violation for it.
**Steps to reproduce**
1. Run a Daytona R1 × K1 wave with `--storage s3` and K1 at about +65 s,
so the restart lands inside the initial remote workspace save.
2. Let the runs finish or fail and the workspace commit a newer
snapshot.
3. Read `execution_workspace_snapshots`: the abandoned rows are still
`pending`. The server log shows `I2_settled` violations for
`remote_snapshot_ledger` on each report.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
1402e92d1f |
fix(native): take back an executor lease that lapsed only because the host slept (#20474)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The lease lasts 20 minutes, and the owner renews it every 5 minutes. > - When a laptop sleeps for longer than the lease, the wall clock and the database clock move past `lease_expires_at`. The server's monotonic timers barely move. > - After the wake, the next renewal finds its own lease expired. It interrupts a healthy turn with `native_session_lease_lost`, and the user must click Retry. Nobody else took the lease. > - Several background consumers act on an expired lease without changing its owner. So expiry alone cannot simply be ignored for the owner. > - This pull request makes takeover explicit. The same owner can take back its own lapsed lease only when no other actor has recorded anything on the row or the run. > - The benefit is that a run interrupted only by host sleep continues with no Retry. A real takeover still fences the late owner out. ## Linked Issues or Issue Description **What happened?** The restart-chaos time fault TM2 stops the server and runner tree and ages every database time by one hour, as a stand-in for a laptop that slept. After the thaw, the next 5-minute renewal of the native execution lease found the lease expired. It interrupted the turn (`turn.interrupted` at 300 s, `native_session_lease_lost`). The failure settlement was then refused with the same code, and the UI showed "Couldn't finish this attempt" with Retry. The owner, attempt, boot and pid were unchanged. Evidence: wave `w-20261110T051535-s304` t01 (TM2, 1 h) failed R3. t02 in the same wave (TM1, a 113 s server-only freeze with no clock jump) passed, so the failure follows the clock jump and not the pause. **Is this a harness artifact?** No. TM2 ages the stored rows, and a real sleep moves the clock forward. Both produce the same comparison: a stored `lease_expires_at` is earlier than `clock_timestamp()` after the wake. The server's renewal timer and local deadline use the monotonic clock. On macOS and Linux that clock does not count suspended time, so they do not notice the sleep in either case. The first database check after the wake is the renewal predicate `lease_expires_at > clock_timestamp()`, and it fails in both cases. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, keeps running and finishes without Retry. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304` with the harness from #20458. Refs #20458, #20500 ## What Changed - `native-execution-lease.ts`: add `nativeExecutionLeaseHeldCondition` and `lockNativeExecutionLeaseForReacquire`. A lease is held when it is live. For a local run, it is also held when it lapsed and is untouched. "Untouched" means all of these are true: - the claimed generation matches; - the phase is `observed` and no result is accepted; - `recovery_state`, `recovery_request_id`, `failure_code` and `next_attempt_at` are null; - `controller_host_hash` is not the board's attested-stopped stamp; - the run is still `running`, without the ownership-unverified error code and without a `nativeStartupCleanup` marker. - The caller's WHERE already matches the owner token, attempt and controller boot/pid/start. - The lock helper locks the coordinator row. If the lease lapsed, it also takes a share lock on the run row. The predicate then reads the latest committed run, and an in-flight run write cannot slip past it. - Renewal (`startNativeSessionExecutionLeaseRenewal` / `renewNativeSessionExecutionLease`) takes back a lapsed lease when the executor passes its claimed generation. Remote targets never do. A deadlock (40P01) on that rare path retries at the next tick, as a statement timeout already does. - `native-session-executor.ts`: the executor passes its generation to renewal. Failure settlement and the final release use the same lock and condition. After a result is accepted, a lapsed lease can only be relinquished (set to null), never renewed, because workspace finalization may act on it. - New `native-execution-lease-reacquire.test.ts` (embedded Postgres, 24 cases). - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: step 2, which adds explicit claim tokens for the consumers this PR does not yet move. Consumer inventory (readers of `native_run_finalizations.lease_expires_at` that can act on a lapsed lease): - Restart recovery and takeover (`claimNativeRestartRecoveries`): claims first. It writes `recovery_state`, `lease_owner` or `controller_*` under the coordinator lock. - Finalizer and cleanup claims: claim first (`lease_owner`). - Failed-run owner release, environment settlement, observed-workspace preservation, retained-session cleanup: require a terminal run status, `committed`, or a recovery state. The rule refuses all of these. - Finalization reconciler, live recovery: require `retryable_failure`, a field the rule checks. - Board controller-stopped confirmation: writes `controller_host_hash`, which the rule checks. - Startup cleanup promotion: writes the run's `nativeStartupCleanup` marker, which the rule checks. It also applies only to runner-task leases. - Another server's orphan sweep: writes the run's ownership-unverified code. The rule checks it when the marker lands first. The reverse order is deferred (see Risks). - Workspace finalization copyback: acts on a lapsed post-result lease without a coordinator write. Post-result leases are therefore never renewed (deferred to step 2). - Accepted-result and token lease settlement, checkpoints, cancellation delivery, admission and the other readers: act on live leases only, or are read-only. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts` passes (24/24) on this branch. With the source files from `origin/master`, 5 fail, including "lets the same owner take back a lease that lapsed untouched" and "renews from the shared timer after the sleep instead of interrupting the turn". - (a) The owner takes back its lapsed lease when no other actor claimed it. Also covered: the shared 5-minute timer, rows saved by the previous release, and the post-result relinquish. - (b) 14 cases where a consumer claimed first fence the owner out and leave the row unchanged. A release over a recorded recovery claim is refused. A different boot (a deploy mid-run), a different generation, a missing generation and a remote target are refused. - (c) Races have exactly one winner: a claimant and the owner lock the coordinator in each order, an in-flight run status write races the owner, and 8 simultaneous starts each have one winner. - Existing suites pass: `native-execution-lease.test.ts`, `native-session-executor.test.ts`, `native-runner-restart-recovery`, `native-runner-restart-lease-renewal`, `native-runner-restart-preparation` and `native-runner-restart-checkpoint-cost` integration tests. - Live proof: `B5 × TM2 --time-age-hours 1 --tasks 2 --seed 304` with `--server-from` this branch. Wave `w-20261110T083444-s304` passed 2/2, with no Retry and no `native_session_lease_lost`. - Both runs ran past the 300 s renewal after the one-hour aging, at the exact point where `origin/master` fails. t01 ran 310 s and t02 ran 319 s. Each was a single run that succeeded. - The harness used the aging exclusion from #20500 (merged): process start identity is not aged. Without that exclusion, TM2 also moves `controller_process_started_at`. The live owner then fails its identity check, which a real sleep never causes. An earlier wave on the unpatched harness, `w-20261110T072106-s304`, failed 0/2 for that reason only. ## Risks - Previous release: no persisted shape changes and no migration. Rows saved by the previous release have the same columns, and a null `controller_host_hash` is accepted. A deploy that lands mid-run cannot take back the old server's lease, because the boot and pid differ. - On the normal path, a live lease takes the same single coordinator lock as before. The run share lock is taken only after a lapse, and nothing is added before `turn.start`. - Deferred to step 2 (see the plan): - The orphan sweep of another server process can read the row before the owner takes the lease back and write after. That run would carry an ownership-unverified error while it is live. - Shared-workspace holder detection reads a silent run with lapsed leases as gone. - Post-result leases are not renewed. - Remote leases are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing, sub-agent code audit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
182a7cd4fe |
test(restart-chaos): keep process start identity out of TM2/TM3 aging (#20500)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) checks recovery under faults. TM2 (host sleep) and TM3 (aged outage) move every database time back by an age. > - The aging also moved process start times: `controller_process_started_at`, `process_started_at`, and `processStartedAt` copies in JSON. > - A process start is identity, not a deadline. The server reads its own start from the OS once and compares it with the stamp an owner saved. A real sleep never changes it. > - After a shift, every live owner looked like a different process and failed its identity check, whatever its lease said. This harness artifact hid whether a lease fix works (#20474). > - This pull request keeps process start identity unaged and makes the grader's unshift skip it too. > - The benefit is that TM2/TM3 failures now come only from deadlines that really lapse, as on a real laptop. ## Linked Issues or Issue Description Refs #20458 (adds TM1–TM3) and #20474 (the lease fix whose proof wave exposed this). **What happened?** A TM2 wave against #20474 still failed at the 300 s renewal. The fix takes back a lapsed lease only for the same owner (token, attempt, generation, boot, pid and process start). The aging had moved the saved `controller_process_started_at` back by one hour. The live process still reported its real start, so the identity comparison failed. **Expected behavior** Aging moves deadlines and event times. It never moves facts that a real sleep cannot change, such as when a process started. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304 --server-from <#20474 checkout>`. Before this change it fails 0/2 (w-20261110T072106-s304). ## What Changed - `time-faults.ts`: add `isProcessIdentityTime`. `agingStatements` skips columns named for a process start. `paperclip_shift_json` keeps JSON keys named for a process start (any spelling) unchanged. `unshiftJson` skips the same keys, so the grader does not move them forward. - `time-faults.test.ts`: a unit test for the statements and the unshift. - `time-faults-db.test.ts`: the embedded-Postgres aging test now checks that a `controller_process_started_at` column and `processStartedAt` / `processStartedAtMs` JSON keys keep their values. - `README.md`: one paragraph under **Aging** that says why. ## Verification - `npx vitest run -c tests/restart-chaos/vitest.config.ts tests/restart-chaos/time-faults-db.test.ts tests/restart-chaos/time-faults.test.ts`: 26/26 pass. Without the `time-faults.ts` change, 2 fail (the new unit test, and the DB aging test's identity check). - `npx tsc --noEmit -p tests/restart-chaos` is clean. - Test-only change. No product code. ## Risks - Low. Only the harness changes. A column or key whose name contains "process started at" is no longer aged. If such a field were really a deadline, it would no longer lapse, but none is. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2f59b9c67c |
test(chaos): fire a planned restart while the server is mid S3 transfer (#20451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness restarts the control plane while agent runs are in flight, and `--storage s3` runs the server against a loopback MinIO behind a fault proxy > - Planned restarts fire by time or by run events. A time-based restart seldom lands while the server is moving bytes to or from the store > - The riskiest storage moments are a workspace snapshot upload, a snapshot restore, and a run-log journal write. A restart in the middle of one must not lose or corrupt the workspace or the log > - This pull request adds a storage-triggered restart: the planned fault fires the moment the proxy sees a matching S3 request, and the proxy holds that request so the restart interrupts it > - The benefit is that restart-during-transfer becomes a repeatable wave with receipts that name the interrupted request ## Linked Issues or Issue Description **Subsystem affected** Restart-chaos test harness (`tests/restart-chaos`), cloud storage mode. **Problem or motivation** A K1/K2/K4 restart fires on a timer or on a run trigger. With `--storage s3`, the timer almost never lands during a snapshot upload, a snapshot download, or a run-log write. So the harness does not prove that a restart in the middle of a transfer is safe. **Proposed solution** Add `--storage-trigger <target>`. At its planned time, each task's fault is armed. The next server request that matches the target fires it. The proxy holds the request open (up to 60 s) while the harness snapshots the tasks and restarts the server. **Alternatives considered** A new `--triggers storage` id. It needs edits to the observer (near its 1,000-line limit), the trigger registry and the delay table. The storage trigger is a wave-level mode in a new module instead. ## What Changed - New `tests/restart-chaos/storage-trigger.ts`: targets `snapshot-upload`, `snapshot-part` (multipart `UploadPart`), `snapshot-download` and `run-log`; the `StorageTrigger` that arms faults, fires on a match, holds the request and records receipts. - `storage-proxy.ts`: an `observe(listener)` hook. The listener sees every request and can hold it (the existing latency path). - `main.ts`, `config.ts`, `types.ts`: registry lines only. `--storage-trigger` (needs `--storage s3` and `--triggers time`); a due fault is armed instead of fired; the wave writes `storage-trigger.json`. - `storage-trigger.json` records the matching request (method, path with upload ids hidden, declared bytes, outcome `aborted` or `completed`, status), every other request in flight, and the fault id. A fault that never matched is recorded with `missedReason`. - README: "Restart during a transfer". ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (25 files, 304 tests pass). - Red/green: `storage-trigger.test.ts` against the master `storage-proxy.ts` fails 3 of 6 tests (no hook, so nothing is held or fired); it passes with this change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local run-log, Daytona snapshot upload and download) are recorded in a PR comment as they finish. ## Risks - Low risk. Test harness only. Without `--storage-trigger`, the proxy has no listener and behaves as before. - A replay re-arms its faults at their observed offsets. It fires on the next matching request, not on the exact same request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7fef6321fe |
test(restart-chaos): remove storage fault window time from speed budgets (#20501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed against local budgets (`wave-speed.ts`) > - The speed grade already removes restart outage time (fault start to server ready) from spans and restart recoveries > - It did not remove storage fault window time (ST1 outage, ST5 rejected credentials, and so on) > - A restart recovery that correctly waits for the store inside a storage fault window then counts as a speed breach > - This pull request removes the overlap with storage fault windows the same way, and shows the removed time on its own line > - The benefit is that storage waves fail only on real slowness, not on correct waiting ## Linked Issues or Issue Description **What existing behavior does this improve?** The wave speed grade treats time spent inside a storage fault window as product slowness. **Subsystem affected** tests/restart-chaos (speed grade). **Current behavior** The ST5 waves w-20261110T050044-s503 and w-20261110T062313-s506 exited 1 on speed breaches only. Their `restart_recovery_control_plane` p95 was 62–67 s against a 4 s budget. The recoveries were inside an ST5R window and correctly waited for a valid key. **Proposed behavior** Spans and restart recoveries do not count time inside storage fault windows (from the `storage-faults.json` receipts). Restart outage and storage windows that overlap are removed once. `SUMMARY.md` shows the storage time that was removed, and `speed.json` has `storageFaultWindows` and `storageFaultRemovedMs`. **Reason and benefit** Storage waves must fail on real product failures and real slowness, not on the harness's own fault windows. ## What Changed - `wave-speed.ts`: `storageFaultWindows()` turns receipts into windows (an open window ends now). `mergeWindows()` joins overlapping windows, so `outageFreeClock()` removes shared time once. `collectWaveSpeed()` takes the storage receipts and removes their time from spans; `gradeWaveSpeed()` removes it from restart recoveries. The storage share is reported apart from the restart outage share. - `main.ts`: one line passes `storageChaos.receipts` to `collectWaveSpeed()`. - README: one paragraph in the Cloud storage section. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/wave-speed.test.ts`: 20 pass. The 3 new tests fail before this change (`storageFaultWindows is not a function`, overlap removed twice, span not trimmed) and pass after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. ## Risks - Test harness only. No product code changes. - ST2 (slow store) windows are also removed from speed budgets. Slowness during a slow-store window is expected, so this is intended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
a14ae9b5c9 |
fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent starts, Paperclip prepares the task workspace and runs the project's configured setup commands (for example the workspace provision command) > - When that command fails, the run fails as `setup_failed`, and the task stops with a Retry > - The task page shows only "Failed" and a generic recovery notice. The real error (the command, its exit status and output) is visible only on the run page > - The user cannot see from the task which setting is broken or what to change > - This pull request records a bounded, named cause for a failed setup command and shows it on the task page > - The benefit is that the task names the broken setting and what to change, and classification, retry and recovery stay the same ## Linked Issues or Issue Description **What happened?** The restart-chaos config fault wave CF1 sets a project `workspaceStrategy.provisionCommand` to `bash ./scripts/provision-worktree.sh`, a script that does not exist. The run fails as `setup_failed` with `Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed: bash: ./scripts/provision-worktree.sh: No such file or directory`. The task page shows "Failed" and "Paperclip automatically retried continuation … still has no live execution path". The grader fails the task as `CF1/provision_command:message_only_in_run_detail`. **Expected behavior** The task page names the failed setup step, the configured command, its exit status and a short tail of its output, and says what to change. Retry behavior stays the same. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks 2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
52b906b112 |
fix(native): let Retry continue after a run that failed before native selection (#20482)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before Paperclip starts a new message or a Retry after a failed native run, it proves that the run's local process stopped > - A native run can fail before native selection, for example when a budget reservation is refused or a preparation query fails. That run never had a runner, a native session or a process ID > - The stop gate still waited for a stop record of that process. Retry returned 202, then the task stayed on `retryWait.reason=process_identity_missing` and no run started > - Native selection writes the run's native issue, native session, runner instance and coordinator in one transaction. A runner launch needs all of them, and the executor records `native.process_start_requested` before any spawn > - This pull request treats a settled run with none of that evidence as having nothing to stop > - The benefit is that Retry and new messages continue after a failure that happened before the agent started ## Linked Issues or Issue Description **What happened?** In local restart-chaos database-load wave `w-20261110T050818-s20386`, all 4 tasks failed before any agent process started. One example is t01, run `f7e2ee55`. A preparation query failed, and the run ended `adapter_failed` with no native session, no runner instance, no process ID and no launch event. Retry returned 202. For 16 or more minutes, `retryWait` then stayed on `process_identity_missing` and no run started. **Expected behavior** If the previous run never had a process, there is nothing to verify, so Retry and new input continue. **Steps to reproduce** 1. Make a native run fail before native selection (for example, a refused budget reservation or a failed query). 2. Click Retry, or send a message. Before this change, the input waits on `process_identity_missing` and does not continue. ## What Changed - `native-local-process-stop.ts`: `nativeRunRefusedBeforeLaunch` (renamed `nativeRunNeverLaunched`) now also proves "never launched" for a settled native run in this state: - no process ID and no process group; - no native issue, no native session and no runner instance (native selection writes them together). A coordinator row requires the native issue by foreign key, so the native-issue check also rules out a coordinator; - no launch request, no identity record, no `adapter.invoke` and no runner-authored event (any event with a source event ID). The existing case, a run that the cleanup quarantine refused before launch, does not change. - `explicit-native-continuation.test.ts`: the guard "failed for another reason without a process record" now separates real pre-selection failures from the shapes that must stay held. - Admitted: a run that failed before it requested a launch. This case is red on `master`. - Still held on `process_identity_missing`: a run that has a runner event, a run with a prepared native session, and a run with a coordinator, for `failed`, `timed_out` and `interrupted` runs. A `timed_out` run that failed before launch is admitted. The chat suite's `unreconciled` case (a native issue and phase, no session) also stays held. ## Verification - Red/green: on `master`, `routes input after a run failed before it requested a launch` fails with `process_identity_missing`. It passes with this change. - `vitest run src/services/explicit-native-continuation.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/issue-confirm-run-stopped-routes.test.ts`: 3 files, 462 tests pass. This includes every existing "keeps the hold for missing_process" and historical-run case. - Server `tsc --noEmit` passes. ## Risks - Low risk. The new proof needs structural evidence that no runner could exist: no native selection and no launch or runner event. A run that reached native selection, or that has any launch evidence, keeps every existing stop proof. - The proof does not depend on the build: a runner from any build needs the native session and the runner instance. Saved runs from older builds that were selected keep those fields, so they stay held exactly as before. - Relates to #20477, which fixes the restart window inside an answer run's attachment. This pull request is independent of it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19829 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.8 |
||
|
DottaandClaude
|
3e572475c8 |
fix(native): re-attach an answer run whose run.attach a restart cut off (#20477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person answers an agent's saved question, Paperclip starts an answer run. For a per-turn session, that run starts a new runner and sends `run.attach` to bind the saved conversation to the run > - The runner journals each command before its effect. A restart that kills it inside that window leaves `run.attach` with an unknown outcome. The restored runner settles it as `execution_indeterminate` and never executes it again > - The controller's same-run resume waited on that old command, so every resume failed with "PRP command run.attach failed". The answer never reached the agent. A Retry or a stall followed > - A kill during the attachment's provider launch can also leave a startup fence that no later runner retires, so every later command fails with "provider startup ownership remains unadmitted" > - This pull request attaches again with a new command when nothing after the cut-off attachment started provider work. It also lets the runner retire that orphaned startup fence > - The benefit is that an answer given just before a restart continues on its own ## Linked Issues or Issue Description **What happened?** Local restart-chaos journey wave `w-20261110T054406-s108` on master `2c2186706e`. Command: `--adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Result: 1 of 4 passed. - t02 (Q1, K1 2.7 s after the answer). Answer run `0b357a1f` resumed after the restart. Each resume failed with `native_session_recovery_failed: PRP command run.attach failed: execution_indeterminate`. Then "Automatic recovery stopped", and the task needed a Retry. - t01 (Q2, K1 1.1 s after the answer). Answer run `82c60af9` failed the same way on three same-run resumes. Its provider state kept `lifecycle: "prepared"` with a `spawned` startup attempt for `run.attach`, and that provider process was gone. Each later drain and suspend failed with "provider startup ownership remains unadmitted". The successor run then refused with `retirement_refused=journal_commands_unsettled(run.attach.failed)`. The accepted Retry created no run (`retry_no_new_run`). The earlier wave `w-20261110T033426-s108` hit the same window at different points. It showed `authority_indeterminate` (Q1) and an ambiguous `turn.start` stall (Q2). See Risks for what this pull request does not cover. **Expected behavior** An answer run that a restart interrupts before it starts provider work resumes by itself. It does not need a Retry. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-attach.test.ts` on master. Both cases fail with the error from the wave. ## What Changed - `runnerd-codex-transport.ts`: the same-run resume waits on the recovered `run.attach` as before. If that wait fails, the new `cutOffRunAttachmentPayload` checks the journal. It queues one new `run.attach` with the same payload, and waits on it, only when all of these are true: - the latest `run.attach` has an unknown outcome (`indeterminate`, or `failed` with `execution_indeterminate`); - it is a cold attachment (no `paperclipNextAuthority`), with no warm transition and no staged attachment; - every later command is recovery maintenance, as the control plane's shared `isRunnerRecoveryMaintenanceCommand` defines it (`session.snapshot`, `session.goal.get`, `runner.drain`, `runner.suspend`, `runner.shutdown`, `turn.stop`); - no committed event shows provider work (`turn.accepted`, `turn.started`, `item.started`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created` or `mcp_app.tool_input`). Any other failure is thrown unchanged. - `provider_backend.rs` (Runner): `retire_orphaned_startup_attempt` also treats a `prepared` state with a saved thread and a nonzero process generation as restorable. `attach_run` already documents this state as the settled checkpoint that `run.attach` restores. The other fence rules do not change. The fence is retired only before the spawn, or when the recorded process and its group have exited. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-attach.test.ts`. A real runner and the fake Codex server run an asking run, then a cold answer-run attachment. The test kills both processes and rewinds both journals to "`run.attach` journaled, outcome not recorded", then resumes. One case has a startup fence and one does not. - A unit test for `cutOffRunAttachmentPayload`. It covers each maintenance command, and the cases where it must not re-attach (a definite answer, a pending command, a warm or staged attachment, any work command, a turn event). - The Rust orphaned-startup test also covers `prepared`. A new test shows that a `prepared` state with no saved thread keeps its fence. - `docs/durable-recovery.md`: describes both behaviors. ## Verification - Red/green, real runnerd: - On master, `runnerd-codex-transport-cut-attach.test.ts` fails both cases with `PRP command run.attach failed: ... execution_indeterminate`, the error from the wave. - With only the transport change, the case without a fence passes. The fence case fails with `provider startup ownership remains unadmitted`, the t01 error. - With both changes, both cases pass. - Red/green, Rust: `cargo test -p paperclip-runner-core --lib orphaned` fails `restart_retires_a_spawned_startup_whose_provider_process_exited` for `prepared` without the runner change. With it, all 10 tests pass. - Related transport suites, 8 files and 387 tests, pass: `runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `runnerd-restart-continuation` and `runnerd-saved-continuation`. `tsc --noEmit` for the Runner package passes. - Live proof: the same wave command on this branch. Results follow in a comment. **No duplicate provider work.** The new attachment is sent only when the journal proves that no provider work followed the cut-off one. No command after it can start or steer work: only the shared recovery-maintenance commands are allowed, and none of them starts a turn. No turn, item, tool-input or runtime-request event is committed. `run.attach` itself starts no turn. In the Codex and ACPX runners, `attach_run` refuses to attach unless the session is settled. That means no active or ambiguous turn, no pending tool call and no live active goal. The old command is never executed again, because the runner refuses an `indeterminate` command. The new attachment is a new command with a new ID. ## Risks - Low-medium risk. The new path runs only after a same-run resume has already failed on an unknown-outcome attachment. Before this change, that run could not continue. - If an attachment's partial effect had changed provider state, the runner's own settled-session check refuses the new attachment. The run then fails as it does today. - **Runner release:** the Runner change does not change the pack `contractDigest`. It is `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). Local runners use the runnerd built with the app, so the fix takes effect with this release. A remote image pack (for example Daytona) that was built before this change keeps the old fence rule until a new image or pack is built and qualified. No release or pin bump is part of this pull request. - **Mixed versions during an upgrade:** - New server with an old runner: the server re-attaches. A run without a startup fence continues. With a fence, the old runner refuses the new attachment, the same failure as today. - Old server with a new runner: the runner retires the fence, but the old server still fails on the unknown-outcome attachment, as today. - Neither mix adds a new failure. Saved state from older builds needs no migration. - Not covered here, and still open: - A `turn.start` with an unknown outcome where the resumed thread has no active turn (wave `w-20261110T033426-s108` t03). The runner deliberately fails closed for that case. - A warm-runner attachment that may already have been delivered (`authority_indeterminate`). - A Retry that waits on `process_identity_missing` for a run that failed before any process started. A separate pull request will cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20146, #19784, #20148 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
ca472d543a |
fix(runner): settle a saved governed wait whose shutdown drain left sequence gaps (#20494)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run can ask the board a question and park on that saved (governed) wait > - After the question is saved, the runner interrupts the provider and drains its usage and terminal for a bounded time > - The drain keeps only usage, the terminal and cancellation facts of the parked turn. It discards every other numbered provider event > - If the control plane restarts before the drain settles, recovery replays the saved journal. That replay required strictly contiguous `sourceSeq`, so the discarded events made a valid saved wait look like the wrong conversation > - This pull request accepts a gap after the saved trigger only when every event after the gap is a drain-retained fact of the same turn. Identity checks do not change > - The benefit is that the asking run settles its saved wait after a restart instead of recording a failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261110T063253-s208`, tasks t02 (Codex Q1, PV2: provider terminal dropped) and t03 (Codex Q1, PV1: provider terminal held), each with K1 (`kill -9` of the control plane about 3-4 s after the fault). The question, the board answer and the final brief were all correct. But the asking run was recorded `failed: native_session_interrupted`, and the grader failed `T:asking_run_settled`. **Cause** The run log of the asking run shows `failureDetail.message: native_governed_wait_replay_mismatch`. It is thrown at `packages/paperclip-runner/src/native-session-runtime.ts:2344` (master). The persisted runner events of the asking run have `sourceSeq` 33..40 and then 43 (t02), and 41, 42 and then 45 (t03). The two missing numbers are events that the accounting drain (`consumeTurn` → `settleDurableResult`, around line 1206 on master) read and discarded, because they are not usage, a terminal or a cancellation fact. Runs without a restart have the same gap (for example `w-20261110T032350-s201/t01`: 34, then 37). Those runs settle without replay, so the gap was not seen before. With a held or dropped terminal and a kill before the shutdown settlement receipt, recovery must replay. Its strict contiguity check then fails on a gap that the runner made itself. **Expected behavior** The question was saved before the provider was interrupted. Recovery must settle that same saved wait, as #20336 does after verified shutdown. Recovery must not start a new model turn. **Related** Refs #20336 (settle saved handoffs after verified shutdown). Not related in cause or code: #20477, #20482, #20483. ## What Changed - `isGovernedDrainRetainedEvent` (new, in `native-session-runtime.ts`) names the facts that the governed-wait drain persists: usage, the turn terminal, the interrupt acknowledgement, cancelled or expired runtime requests, and a failed or cancelled `run.terminal`, all for the parked turn. The drain now uses it in place of the same inline condition. Drain behavior does not change. - Governed-wait recovery replay accepts a `sourceSeq` gap only when: - the trigger is the first replayed event (a gap before it still fails), - sequence numbers strictly increase, - every event after the first gap is a drain-retained fact of the trigger's turn. - The run, normalized session and source instance identity checks do not change. Any mismatch still fails with `native_governed_wait_replay_mismatch`. - `doc/execution-semantics.md`: the governed-wait text said that shutdown keeps a contiguous journal. It now describes the gaps and the replay rule. Note for the sibling Claude/ACPX `turn.cancel` work: this change touches `settleDurableResult` only in the drain filter line (same condition, shared helper). It does not change cancel, the drain timeout or `settleCancelledTurn`. ## Verification - Red/green: `packages/paperclip-runner/src/native-session-runtime.test.ts` - `settles a saved governed wait whose drain left sequence gaps before a restart (terminal-after-restart)` - `settles a saved governed wait whose drain left sequence gaps before a restart (late-terminal-saved)` The first run uses the real drain. It discards two numbered events, so the history is trigger, acknowledgement, then usage with a gap. The stream then ends without a terminal, as with PV1/PV2 plus K1. On master, recovery rejects with `native_governed_wait_replay_mismatch` (the live error). With this change, recovery settles the saved wait as yielded. It does not call `resolveGovernedWait` again. It starts no second turn. The terminal is appended once. When the terminal arrived late and is already saved, recovery does not cancel the provider again. A further recovery of the settled checkpoint does not submit or settle again. - Fail closed: `keeps rejecting an unproven governed-wait replay with a sequence gap: …` covers a gap before the trigger, a foreign run, a foreign session, a foreign source instance, usage of another turn, a non-accounting event after a gap, and an accounting event followed by a non-accounting event. - `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts`: 261 passed. `npx tsc --noEmit -p .`: clean. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t governed`: 12 passed. - `pnpm verify` returned exit 2 (test file outside resolved graphs). Required CI supplies the broad coverage. - Live restart-chaos wave: see the comment below with the wave id and result. ## Risks - Low. The rule loosens only the sequence check, and only after the saved trigger, and only for facts that the drain itself retains and that carry no work authority. Identity checks stay strict. A non-accounting event after a gap still fails closed. - No change to the normal start path or turn startup (fast-turn rule). - No schema, protocol or env change. The pack `contractDigest` and provider `commandDigest` do not change. No Daytona image rebuild is necessary. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1e13c58a4e |
fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat run settlement records why a run failed, and the board shows that record. > - Runs that fail without a named code settle as `adapter_failed` (or `setup_failed`) with no cause. > - A user, an operator, and the restart chaos harness cannot tell what went wrong from such a run. > - Some run errors also show raw SQL statement text to users. > - This pull request records a bounded `failureCause` code for generic failures, shows it on the board, and keeps SQL text out of run errors. > - The benefit is that a generic failure now names its cause, for example "adapter failed (40001)", without leaking messages, SQL, or secrets. ## Linked Issues or Issue Description Refs #20385 (the nested-cause extraction this change reuses). Related: #20421 (restart chaos universal checks U1-U4). **What happened?** The restart chaos audit of 79 saved waves found 30 failed runs with `errorCode: "adapter_failed"` and a null or generic error. Every U3 ("diagnosable") warning was `adapter_failed`. Nobody could tell what failed. Some board run errors also showed raw SQL statement text from wrapped database errors (Drizzle `Failed query: ... params: ...`). **Expected behavior** A run that fails with a generic code names a short, safe cause. The board shows the cause next to the stop reason. Run error text never shows SQL statements or their parameters. **Steps to reproduce** Make an adapter throw a wrapped database error (an `Error` whose nested `cause` has `code: "40001"` and whose message has a Drizzle `Failed query:` statement). Before this change, the run settles as `adapter_failed` with no cause, and `heartbeat_runs.error` contains the SQL statement and parameters. The new tests in `server/src/services/heartbeat/run-completion.test.ts` reproduce this. ## What Changed - Runner: move the #20385 event-commit cause extraction into `safeErrorCauseCode` (`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and export it. The Runner's own diagnostics use the same function, unchanged in behavior. - Shared: add the `failureCause` contract (`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]` token of at most 80 characters, `normalizeRunFailureCause`, `readRunFailureCause`, and `withoutSqlStatementText`. - Server: add `server/src/services/heartbeat/run-failure-cause.ts`. Settlement records `resultJson.failureCause` only when the code is generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same. - Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`, else `provider_<category>`, else the first safe code in the error and up to three nested causes (`40001`, `ECONNRESET`), else the error class name, else `unknown_error`. - Adapter results without a code (`completeRun`): the error family, `signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`, `adapter_reported_error`, or `unknown_error`. - The failure log lines (`heartbeat execution failed`, `heartbeat execution setup failed`, and a new `heartbeat run failed` warning for generic adapter results) include `failureCause` and `errorCode`. - Server: replace SQL statement text in run error text with "Database query failed (<cause>)" at all three settlement paths, including the cancelled branch. The execution blocker's `runError`, the attention inbox (failed-run excerpt and metadata), and continuation summaries apply the same replacement on read, so older rows also stop showing SQL there. - Server: Retry admission and failed-preparation replay accept only a closed set of `resultJson` keys. `failureCause` is added to each of those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`, `cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is a diagnostic code with no execution or replay authority, so a failed-preparation run stays retryable. - Server: `runsForIssue` returns a validated top-level `failureCause`. It is null for older rows, non-string JSON values, and malformed values. - UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed (40001)". Older runs keep "adapter failed". The Storybook run-ledger fixture has a failed run with a cause. - Restart chaos harness: run samples keep `failureCause`. U3 treats a generic code with a specific `failureCause` as diagnosable. A bare `Error`, `unknown_error`, or the no-code fallback `adapter_reported_error` still counts as vague. U3 stays a warning. - Docs: `doc/run-log-events.md` and the harness README describe the field and the SQL replacement. ## Verification - Red before the fix: the four new `run-completion.test.ts` cases failed against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL text in `error`). - Green: - `server/src/services/heartbeat/run-completion.test.ts`: all tests passed (real embedded PostgreSQL). New cases cover a nested `40001` three levels deep for execution and setup failures, SQL and a secret parameter absent from `error` and `resultJson`, the bound (an 81-character code and a SQL "code" fall back to the class name), `exit_code_137` for a code-less adapter result, a cancelled result with SQL text, no cause for a named code, and the log line. On this loaded host the suite's `afterAll` database teardown hit its 30 s hook limit after every test passed (the known local teardown limit, also seen in #20385). CI runs the suite. - `server/src/services/heartbeat/run-failure-cause.test.ts` (4), `server/src/__tests__/activity-service.test.ts` (recorded, legacy, malformed, overlong, boolean, and number cause projections), `server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL row). - `packages/shared/src/run-failure-cause.test.ts` (6), `ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos `universal.test.ts` and `observer.test.ts`. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`: 207 passed (the stall-diagnostic cases still read `40001` and `ECONNRESET`). - Typecheck: shared, runner, server (all layers), UI, restart chaos harness. `pnpm check:token-gates`: all gates clean. - `pnpm verify` returned exit 2 (global doc input changed); required CI supplies broad coverage. - CI on `6295f8d` found that the new key made failed-preparation runs ineligible for Retry (`chat-channels.integration.test.ts` saved board-message Retry got 409; `workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries). Both reproduced locally and pass after the allowlist fix, with the other 459 Retry cases in `chat-channels.integration.test.ts` and all 25 in `workspace-recovery-preparation-dispatch.test.ts`. - No chaos waves were run. Storybook was not opened in a browser; the fixture typechecks and the label is covered by the UI test. ## Risks - Low risk. `errorCode` values do not change, so retry, recovery, and comparison logic is unaffected. - No schema change and no migration. `failureCause` is a new optional `resultJson` key. Rows saved by the previous release have no key and render as before; a new row read by an old server is ignored. - The SQL replacement only acts on a Drizzle `Failed query:` marker, or on `$n` placeholder text with a statement shape. Ordinary error text does not change. When it acts, it also drops the text after the statement. - Code that checks `resultJson` against a closed key list must allow `failureCause`. I found and updated the four lists on the Retry path; a list I missed would refuse a Retry for a run with a generic failure. - This is the run log / instance database path. It adds no Telemetry event. - The server failure log line still logs the raw error message (unchanged behavior); only the board-facing text is changed. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file editing) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
5c9641f141 |
test(restart-chaos): add repeated restarts (RR1-RR3): a restart inside the previous restart's recovery (#20480)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Every existing fault restarts the server once per task. > - Real incidents on 2026-11-08 and 2026-11-09 were deploy loops. The server restarted again 5–60 s after it came back, while recovery of interrupted runs was still in progress. > - No wave puts a second restart inside the first restart's recovery window. > - This pull request adds a fault family, repeated restarts (RR1–RR3). It does not change product code. > - The benefit is that waves now test recovery that is itself interrupted. ## Linked Issues or Issue Description **What is the problem?** The chaos harness restarts the server only once per fault. Production deploy loops restart it again while it recovers, and no wave tests that. **What is the solution?** Add three repeated-restart faults. Each one starts as a K1 and then stops the replacement again after a seeded gap from its health. **Why is it needed?** Recovery of interrupted runs can itself be interrupted. A second restart in that window can leave a lease, a claim or a wake half done. Refs #20458 (the TM fault family this one follows). ## What Changed - Add `RR1`, `RR2` and `RR3` to `--faults` (`repeated-restarts.ts`): - **RR1 double crash**: K1, then one more `SIGKILL` 3–20 s after the replacement reports healthy. - **RR2 crash, then graceful**: K1, then one `SIGTERM` (as K2) 3–20 s after health. - **RR3 crash loop**: 3–4 `SIGKILL`s in all, 10–30 s after each health. The last replacement stays up. - Add `--rr-gap-seconds <n>` or `<min>-<max>` (1–300). It replaces every RR gap. RR faults do not combine with `--storm-minutes`. - The gaps are seeded per task (`restartSteps` in `plan.json`, also in `--matrix` plans). They are drawn only for RR tasks, so the plans of existing seeds do not change. `--replay` restarts again after the recorded gaps. - The supervisor runs the whole sequence as one fault receipt. The grader therefore sees one fault window, from the first kill to the last replacement's ready. Crossing, R1's outage allowance, R5, the T-oracle, task deadlines and the recovery speed budget treat it as one fault. No check is relaxed. - Each stop and replacement is recorded with its signal, gap, PID, the `/api/ready` status just before the stop, and its stopped, restarted and healthy times. The record goes to `<wave>/repeated-restarts/<faultId>.json` (also written when the sequence fails part way), to the receipt (`repeatedRestart` in `faults.json` and in each failure's `startup-phases.json`), and to the timeline (`repeated_restart_stop`, `repeated_restart_healthy`). - Small registry additions: `types.ts`, `config.ts`, `scheduler.ts`, `journey-matrix.ts`, `main.ts`, `evidence.ts`, `supervisor.ts`, and the README section next to TM. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 33 files, 402 tests pass. The new `repeated-restarts.test.ts` covers option parsing, seeded planning for each kind, unchanged plans for other faults, steps in the task schedule, the matrix and replay, the stop-and-replace loop on a fake clock (gaps measured from health, `SIGTERM` for RR2, an exit during a gap, shutdown), and one crossing window for the sequence. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - `pnpm verify` asks for broader coverage (the README is a global CI input), so required CI supplies it. - Live waves run from master after merge, as the merge rule for test-only harness PRs allows. Their results go to a follow-up comment. ## Risks - Harness-only change; no product code changes. Waves without RR faults are unchanged: the plan draws, the fault receipt and the grader paths are the same. - During an RR sequence the server stays "not ready" for the harness, so no other fault fires and the observer does not click Retry until the last replacement is ready. This is the same rule as for K6. - A replacement that exits on its own during a gap fails the fault as a harness error, with the stops so far recorded. The wave then recovers the server as after any failed fault. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
53c9be8d41 |
Reuse identical captured workspace archives (#20476)
Reuse the current committed snapshot when a fresh capture matches its manifest and plaintext archive hash and size. Preserve the original encryption binding and verify each new run receipt for recovery, concurrent replay and storage request statistics. Verified with 338 focused save and recovery tests, server typecheck and build, all final-head CI checks, and Greptile 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
179694de7d |
test(chaos): provider message faults for Claude (ACPX) and across a K1 restart (#20454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) proves that agent work survives control-plane restarts and provider faults > - PR #20432 added provider message faults (PV1 delay, PV2 drop, PV3 duplicate), but only for Codex, and only in waves without restarts > - Claude runs through the ACPX sidecar, whose provider pack and sidecar are digest-verified, so the Codex executable shim cannot reach it > - A late or missing terminal can also arrive across a control-plane restart, which no wave could exercise > - This pull request adds a Claude (ACPX) seam with no product change, and a K1 restart that fires after a provider fault applies > - The benefit is that governed waits, Stop and approvals are now tested under provider message faults for both main adapters, also across a kill -9 ## Linked Issues or Issue Description Refs #20432, #20330, #20336 **What is the problem?** Provider message faults covered only Codex. They could not run with restart faults. PV3, the `interrupt_ack` target, and the S1 and G1 workloads under PV1/PV2 never ran live. **What is the expected outcome?** The harness can delay, drop or duplicate one targeted provider message for Claude and Codex. A PV1/PV2 fault can straddle a K1 restart, and the grade states what a pass means. **Why is this the right layer?** This is test-harness work. The product is not changed. The seam replaces only a file the runner already hashes and verifies at launch, inside a per-wave stand-in package. ## What Changed - `acpx-provider-faults.ts`: builds a per-wave stand-in runner package under `<repo>/node_modules/.restart-chaos-<wave>/` (symlinks to the real package, except a generated `dist/cli/acpx-runtime-sidecar.cjs`). It also parses `--provider-fault-restart`. - `provider-shim/acpx-sidecar-hook.mjs`: a server-only resolve hook that loads `runnerd-codex-transport` (source `src/live/*.ts` or built `dist/live/*.js`) from the stand-in package. Product code then hashes the stand-in sidecar, and the runner verifies and launches it. The module's own imports resolve to the real package. - `provider-shim/acpx-fault-tap.cjs`: the stand-in installs this tap and then runs the real sidecar bundle in the same process, as the runner's verified CommonJS loader does. For Claude, `terminal` is the provider's ACP `session/prompt` response after a `session/cancel`. `interrupt_ack` is the sidecar's `turn.cancel` response, because ACP's cancel has no acknowledgement. The provider process is still spawned and admitted by product code. - `--provider-fault-restart K1 [--provider-fault-restart-after-seconds n]`: each PV1/PV2 task also kills -9 the control plane n seconds (default 3) after its fault applied, through a new internal `provider` trigger. PV3 and planned K cells stay rejected. - `withProviderFaultCheck`: with a restart, the check passes only if the fault applied once, the K1 started after it, and for PV1 no release came first. The ordinary R1–R6 grade and the workload T-oracle still decide whether the task recovered. - `config.ts`: provider faults accept `--adapters codex,claude`. G1 with a provider fault needs `--provider-fault-restart K1`, because its oracle grades the board decision after a restart. - README: the Claude seam, the target table per adapter, what a pass means across a restart, and the Claude PV1 delay window. - Unit tests: the ACP and runner-pipe taps, the generated stand-in with a fixture sidecar, the hook against a fake runner package, planning, config validation, and the restart check. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (all pass after rebase on master) - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - Live waves (local, one at a time through the wave-slot protocol): | Wave | Scenario | Result | | --- | --- | --- | | w-20261110T040202-s202 | Claude Q1, PV1 terminal 18 s | Task passed its Q1 oracle; `provider_fault_applied` failed: the runner settled the cancelled turn ~3.6 s after the cancel and closed the sidecar before the release, so 18 s is a drop for Claude (documented) | | w-20261110T041224-s203 | Claude Q1, PV1 terminal 1 s and 3 s | 2/2 pass (held and released) | | w-20261110T041355-s204 | Claude Q1 and S1, PV2 terminal | 2/2 pass | | w-20261110T043354-s205 | Codex Q1, PV3 terminal (duplicate) | 1/1 pass | | w-20261110T050407-s206 | Codex S1 and G1, PV1/PV2 terminal | 0/4: S1 terminal never sent (Codex exits after Stop, so `terminal` cannot apply; documented, use `interrupt_ack`); G1 needs a restart by design (now rejected without `--provider-fault-restart K1`). No product failure. | | queued | Codex+Claude Q1/S1 PV1/PV2 `interrupt_ack`; Codex+Claude Q1/G1 PV1/PV2 + K1 | Runs from master after merge (queue is ~90 min deep) | ## Risks - Low product risk: no product code changes. - The seam depends on the runner computing the local sidecar path from the transport module URL and on the ACPX provider bootstrap marker string. If either changes, the tap logs nothing and every Claude PV task fails `provider_fault_applied`. It does not pass silently. - The stand-in package is written under the workspace `node_modules` (gitignored) and removed at wave end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.7 nightly/v2026.1110.0-nightly.0 |
||
|
DottaandClaude
|
233f615644 |
fix(heartbeat): resume legacy runs a graceful shutdown stopped before dispatch (#20461)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs agent turns and recovers them after a
server restart
> - A graceful restart can stop a legacy run (for example a `process`
adapter agent) while the run is still in setup, before the adapter
starts
> - That run was parked with `legacy_execution_requires_reconciliation`,
so automatic recovery stopped
> - A `process` or `http` adapter cannot continue that hold through
Retry, so the accepted Retry waited forever and the task stayed blocked
> - This pull request proves "no adapter work started" for that case and
resumes the run automatically
> - The benefit is that a restart during setup no longer strands the
task, and Retry starts a run for holds that an older server saved
## Linked Issues or Issue Description
**What happened?**
The restart-chaos harness ran workload Q3 (a board approves an agent's
app action late) with a `process` adapter agent. A graceful restart
(fault K2) fired right after the approval. The agent's follow-up run was
interrupted with `server_shutdown_interrupted` 0.2 s after it started,
while it was still in setup. Automatic recovery stopped with
`legacy_execution_requires_reconciliation`. The board clicked Retry. The
server accepted the Retry (HTTP 202), but no run started. The task
showed "Retry is waiting for execution checks" (`retryWait.reason =
execution_recovery`) forever.
**Expected behavior**
A run that a graceful shutdown stops before any adapter work must resume
automatically. An accepted Retry must start a run.
**Steps to reproduce**
1. Run a `process` adapter agent on a task.
2. Send SIGTERM to the server while the run is still in setup (before
the adapter starts).
3. Restart the server. The task is blocked with
`legacy_execution_requires_reconciliation`.
4. Click Retry. The request is accepted, but no run starts.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
693de27c90 |
fix(storage): wait out expired or rotated S3 credentials and re-resolve the key (#20456)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode, run logs, provider traces, attachments and remote workspace snapshots go to S3 > - The server uses the default AWS credential chain, and real keys expire or rotate while runs are in flight > - #20299 made runs wait out a storage outage, but a 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch` was a permanent `adapter_failed` > - The SDK also caches a key until its stated expiration (forever for a key with none), so a rotated key was never picked up > - This pull request treats credential rejections as a storage wait and re-resolves the key from the credential chain after a rejection > - The benefit is that runs survive key expiry and rotation without failed attempts or a manual Retry ## Linked Issues or Issue Description Refs #20299 (storage outage wait). Harness that found this: #20455. **What existing behavior does this improve?** Runs wait out S3 outages (5xx, timeouts, resets) but fail at once when the store rejects the server's credentials. **Subsystem affected** server storage (S3 provider) and heartbeat run completion. **Current behavior** Restart-chaos wave w-20261110T041556-s501 (`--storage s3 --storage-faults ST5`): one 41 s window of 403 `ExpiredToken` failed 4 of 6 tasks. Each run ended `adapter_failed` with "The provided token has expired.", and the automatic retries failed within one second, so the tasks needed a board Retry (R3 `retry_did_not_complete`). **Proposed behavior** A credential rejection is a transient infrastructure condition. Critical-path storage calls wait it out like an outage, and a run that stops on it gets the `storage_unavailable` wait retry. After each rejection the S3 client re-resolves its key with `forceRefresh`, so the next attempt signs with a rotated key. The run error names the credentials. If they never come back, the exhausted-wait message tells the operator to check the storage service and its credentials. **Reason and benefit** Expiring or rotated keys are normal in cloud deployments. They must not fail work. ## What Changed - `storage-unavailable.ts`: `ExpiredToken`, `ExpiredTokenException`, `TokenRefreshRequired`, `InvalidToken`, `InvalidAccessKeyId`, `InvalidClientTokenId` and `SignatureDoesNotMatch` answered by the store with 400/403 count as storage unavailable. `AccessDenied` and a credential chain that cannot load any key (`CredentialsProviderError`) stay permanent. New credential-specific run message; the exhausted message also names credentials. - `s3-credential-refresh.ts` (new): S3 client middleware that calls the client's credential provider with `forceRefresh` after a rejection, once per credential generation (requests that signed with an already-replaced key do not refresh again), then rethrows so the existing storage wait owns the retry. - `s3-provider.ts`: installs the middleware on both S3 clients. - `run-completion.ts`: uses the credential-aware message. ## Verification - `cd server && npx vitest run src/__tests__/storage-s3-credential-rotation.test.ts src/storage/ src/__tests__/storage-outage-wait.test.ts src/__tests__/storage-s3-transient-retry.test.ts src/__tests__/storage-s3-failure-diagnostics.test.ts src/services/heartbeat/run-completion.test.ts` (186 pass). - The new `storage-s3-credential-rotation.test.ts` fails 5 of 6 cases on master (red) and passes with this change (green). It covers a real key rotation through the default credential chain, 403 windows for each code, the message, and `AccessDenied` staying permanent. - `cd server && npx tsc --noEmit` passes. - `s3-credential-refresh.test.ts`: two parallel requests signed with the old key, rejected 50 ms and 400 ms apart, cause one forced lookup (3 before the generation check, 2 after). - Restart-chaos rerun on a build with this fix and #20455: w-20261110T050044-s503 (`--env local --workloads A1,C1 --faults K1,K2 --tasks 8 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90`) passed 8/8 tasks (R1–R6 and T) with a 39 s `ExpiredToken` window and an 85 s real key rotation (76 credential 403s). Before the fix, w-20261110T041556-s501 failed 4/6 tasks. ## Risks - Scope: calls with a Readable body (streamed `putObject`, buffered multipart) cannot be resent, so they still fail on the first rejection, as they do for outages. The middleware has already re-resolved the key, so the caller's next attempt (the run-level wait retry for run work) uses the new key. - A key that the store rejects for good (for example a wrong secret) now waits up to the storage wait budget (5 min in the call, then up to 10 run retries) before the board shows the exhausted message, instead of failing at once. - `forceRefresh` re-runs the credential chain after a rejection. Calls are coalesced, so a burst of rejections causes one refresh. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
bf1524c5f8 |
test(restart-chaos): pin each wave's API URL to its own loopback server (#20462)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness starts an isolated Paperclip server for
each wave and runs real agents against it
> - Agents find the server through `PAPERCLIP_API_URL`, which the server
derives at startup
> - On a developer machine with a public URL for the main instance (for
example a tailnet origin), the wave server derived that public URL
instead of its own loopback origin
> - So `process` agents sent the run key for the temporary instance to a
different host, which returned 502
> - This pull request pins each wave server's `PAPERCLIP_API_URL` to its
own loopback origin and removes the Q3 script workaround
> - The benefit is that a wave's agent credentials only go to the wave's
own server
## Linked Issues or Issue Description
**What happened?**
In isolated restart-chaos wave instances, `process` agents received
`PAPERCLIP_API_URL` set to the machine's public tailnet origin. That
origin serves the developer's main instance, not the wave. The agent's
run key for the temporary instance was sent to that host, which returned
502. The Q3 agent script worked around this with a hard-coded loopback
base URL.
**Expected behavior**
A wave's agents call the wave's own server.
**Steps to reproduce**
1. Configure a public URL (for example a tailnet origin) for the
developer's main Paperclip instance on the machine.
2. Run a restart-chaos wave with a `process` agent that reads
`PAPERCLIP_API_URL`.
3. The agent's API calls go to the public origin, not to the wave's
`127.0.0.1` port.
**Paperclip version or commit**
master at
canary/v2026.1110.0-canary.6
|
||
|
DottaandClaude
|
400bb1f0f9 |
test(restart-chaos): one-row db pressure, bound accepted-Retry waits in U1, live L1 results (#20472)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness grades real agent tasks on an isolated control plane. > - #20421 added the L1 contention workload, `--db-pressure`, and the universal checks U1–U4. > - The first live L1 waves showed two harness gaps and one missed product shape. > - The pressure writers locked several run rows per statement and deadlocked with product transactions. A real stream does not do that. > - U1 counted an accepted Retry that waits on the system as a live owner forever. Ten tasks sat 8 minutes in that state with no run. > - This pull request fixes both gaps, adds the owed direct test for `collectContentionEvidence`, and records the live wave results. ## Linked Issues or Issue Description Refs #20421, #20385. **What happened?** The live L1 waves (`w-20261110T040531-s20385` on the #20385 parent, `w-20261110T044338-s20385` and `w-20261110T050818-s20386` on master) produced harness deadlocks (40P01) from multi-row pressure statements. U1 passed tasks whose accepted Retry waited on `process_identity_missing` for 16 minutes without a run. **Expected behavior** Pressure writers behave like a run-log stream (one row per statement). U1 flags an accepted Retry that waits on the system past the queue bound. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <results dirs>` over the saved waves, or the L1 wave command in the README. ## What Changed - `contention.ts`: each pressure statement updates one running run row. The old statement updated every running row at once and deadlocked with product transactions. - `universal.ts` (U1): an accepted Retry that waits on the system (`retryWait`) is an owner only for the queue bound (5 min of server-ready time). While it waits, the Retry button, failed notice and blocker do not count as actionable items, because the person already acted. A scheduled retry is still a timer. - `contention-db.test.ts`: a real PostgreSQL test for `collectContentionEvidence` (receipt count, duplicate producer events, terminal receipt, accounting fields, company scope). - README: live wave results table, updated audit numbers and the U1 rule. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 28 files, 345 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Audit over 79 saved waves (1,620 tasks, 1,483 passing): U1 17 flags, none on a passing task or an all-pass wave. The longest accepted-Retry wait on a passing task was 278 s. U2 1, U3 30 (3 on passing tasks, warning only), U4 0. - Live waves (codex and claude, `--workloads L1 --faults K1 --concurrency 4 --tasks 4 --db-pressure 4`): - `w-20261110T040531-s20385`, #20385 parent: 117 receipt-write serialization failures (40001) in the server log; 1 of 4 tasks passed. Three runs failed `adapter_failed` after 40001 exhausted in `createRunUsageRecorder`, `reserveRunBudget` and native finalization. - `w-20261110T044338-s20385`, master `8004ad069e` with the harness: 2 serialization failures; 1 of 4 passed. Failures included 40001 in `createRunUsageRecorder` and deadlocks caused by the old multi-row writers (fixed here). - `w-20261110T050818-s20386`, same build with one-row writers: 0 of 4 passed. Every first run failed in `reserveRunBudget` before spawn, and every accepted Retry then waited on `process_identity_missing` without starting a run. - No wave reached the runner's commit stall, so none showed `native_event_replay_conflict`. ## Risks - U1 can now fail a task whose accepted Retry waits more than 8 minutes. No passing task in 79 saved waves came close (longest 278 s). - One-row writers produce fewer statements per second than the multi-row statement did; `contention-db.test.ts` still reproduces the #20385 receipt race with them. - Test-only change; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2444658c6f |
test(restart-chaos): add ST5 credential rejection and ST5R key rotation storage faults (#20455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` already puts a fault proxy between the server and a loopback MinIO (ST1 outage, ST2 latency, ST3 errors) > - Cloud deployments use the default AWS credential chain, and keys can expire or rotate during a run > - The harness had no fault for a rejected or rotated key, so this failure class was untested > - This pull request adds ST5 (a window of 403 credential rejections from the proxy) and ST5R (a real key rotation at MinIO) > - The benefit is that waves can prove runs wait out credential problems and pick up rotated keys ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults cover outages, latency and 5xx errors only. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** No wave can reject the server's S3 credentials or rotate the store's key. **Proposed behavior** `--storage-faults ST5` answers every request in a window with 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch`. `--storage-faults ST5R` restarts MinIO with a new root key (the old key stops working) and gives the server the new key through its credential source only when the window ends. **Reason and benefit** The first local ST5 wave (w-20261110T041556-s501) shows the gap: a 41 s `ExpiredToken` window failed 4 of 6 tasks with `adapter_failed`, and the automatic retries failed within one second. The product fix is a separate PR. ## What Changed - `tests/restart-chaos/storage-credentials.ts` (new): ST5 and ST5R rule planning, a wave-owned `AWS_CONFIG_FILE` profile whose `credential_process` prints the current key, MinIO restart with a new root key on the same loopback port, and a proxy wrapper that holds traffic during the restart and publishes the new key when the window ends. - `storage-proxy.ts`: the `expired_credentials` fault takes an S3 error code; a `credential_rotation` marker passes traffic through. - `storage-chaos.ts`, `types.ts`, `config.ts`, `main.ts`: registry lines for ST5/ST5R and the rotation wiring. - README: ST5/ST5R rows and the credential-source description. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (28 files, 328 tests pass). The new `storage-credentials.test.ts` fails before this change (the kinds do not exist) and passes after. It includes a real MinIO test: after a rotation the old key gets 403 `InvalidAccessKeyId`, a client with a cached key fails, and the same client succeeds after it re-resolves its profile. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave: `--env local --workloads A1,C1 --faults K1,K2 --tasks 6 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90` ran and injected 62 credential 403s (storage-proxy.json). ## Risks - Test harness only. No product code changes. - ST5R restarts MinIO. Requests are held for up to 5 s during the restart, so a slow restart can also show as a short 502 blip. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
8809f44b5e | Connect Codex and Claude login to agent computers (#20417) | ||
|
DottaandClaude Opus 5.5
|
95cb3c94c4 |
test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). **What existing behavior does this improve?** The ST2 slow-store fault in the restart-chaos storage proxy. **Subsystem affected** Test harness (`tests/restart-chaos`). **Current behavior** ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. **Proposed behavior** ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. **Reason and benefit** Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
30d99f4e32 |
test(restart-chaos): add time faults (TM1-TM3): freeze, sleep with aged rows, aged outage (#20458)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Each wave runs for minutes on a fresh instance. Nothing time-bound ever expires during a wave. > - Production failures on 2026-11-09 were about time: an OAuth refresh after more than 1 h made pending approvals invalid (#20340), and a timeout used the wrong window (#20330). > - This pull request adds a fourth fault family, time faults (TM1–TM3). The faults make leases, deadlines and waits expire during a run or across a restart. They do not change product code. > - The benefit is that waves now find failures that occur only when time passes. The first waves found one: a run whose turn completes after a host sleep fails, and the task needs Retry. ## Linked Issues or Issue Description Refs #20340 Refs #20330 **What is the problem?** The chaos harness never moves the clock, so it cannot find failures where a lease, a deadline or a pending wait expires. **What is the solution?** Add three time faults. Each one makes stored deadlines lapse in a way that a real host sees: a process pause, a host sleep, or a long outage. **Why is it needed?** Two days of chaos testing did not find the failures of #20340 and #20330, because no wave lasted long enough for a deadline to pass. ## What Changed - Add `TM1`, `TM2` and `TM3` to `--faults` (`time-faults.ts`, `time-faults-config.ts`): - **TM1 freeze**: send `SIGSTOP` to the server process only, for 95–150 s, then send `SIGCONT`. The runner and Postgres continue. The pause is longer than the controller-instance lease (90 s) and the execution-control deadline (60 s). - **TM2 sleep**: send `SIGSTOP` to the server and all of its descendants. While they are stopped, move every database time back by the age. Then send `SIGCONT`. This is a stand-in for a laptop sleep. - **TM3 aged outage**: send `SIGKILL` to the server tree (as K1). While it is down, move every database time back by the age. Then start a replacement. With `--workloads G1 --triggers approval`, an approval stays pending for a day across a restart. - Add `--time-age-hours` (1–72, default 26). TM faults run only with `--env local`, and not with storms, crash points or K14. - Aging is one transaction. It moves every `timestamp` column, and every time inside a `jsonb` column (ISO strings in their own format, and epoch milliseconds in the wave's window), back by the same interval. Triggers are off for the shift. Copies of a time in JSON (for example the comment `updatedAt` in a continuation receipt) still match their columns. The first wave showed that a shift of the columns only causes false failures. - The identity, state and time columns of the time-bound tables are written before the shift to `<wave>/time-faults/<faultId>.json`. Per-table counts go to the fault receipt (`timeFault`) and the timeline (`time_fault_aged`). - The observer and grader read through `TimeShiftedApi`. It adds the age back to each time that a shift moved, so R4, R5, U1–U4 and the speed budgets use real time. - Make small additions to the registries: `types.ts` (fault kinds, receipt field, option), `config.ts`, `scheduler.ts` (a seeded freeze length only for TM tasks, so the plans of existing seeds do not change), `supervisor.ts` (the freeze and aging hooks), `main.ts` (one line for the API) and the README. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 30 files, 368 tests pass. The new tests are: - `time-faults.test.ts`: option parsing, freeze order, Postgres exclusion, statement generation, and reading times back at their real time. - `time-faults-db.test.ts`: aging runs on real embedded Postgres. The test checks that trigger rewrites are suppressed, that JSON copies match their columns, that text columns do not change, that the evidence file is written, and that an age the wave nearly outlived is refused. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local, codex), each run from a separate worktree and one wave per slot: | Wave | Scenario | Result | | --- | --- | --- | | `w-20261110T031929-s301` (A1,C1 × TM1–TM3) | First run, before JSON aging | t01: TM1 missed, because A1 finished first. t02: pass. t03 (C1): its turn completed after TM2 and failed with `native_session_lease_lost`; a follow-up comment woke a new run. After TM3, that run failed with `continuation_user_authorization_missing` and needed Retry. That second failure was a harness artifact: the receipt kept the comment's `updatedAt` in JSON, which the first version did not shift. JSON aging fixes it. | | `w-20261110T041910-s303` (B5 × TM1–TM3) | Freeze, sleep, aged outage | t01 and t02 crossed TM1, then TM2. Both turns completed after the thaw and failed with `native_session_lease_lost`, so both needed Retry (R3). This is a **product failure**; see below. t03 (TM3, 26 h aged outage): **pass**, with no Retry. | | `w-20261110T050125-s302` (G1 × TM3, `--triggers approval`) | Aged wait: an approval pending for 26 h across a restart | **Pass** (2/2). Both approvals were still pending after the restart and the board decided after the fault: t01 rejected, so the task is cancelled; t02 approved, so it is done with exactly one subtask. No Retry; U1–U4 pass. | | `w-20261110T051535-s304` (B5, TM2 with `--time-age-hours 1`, then TM1; one task at a time) | Attribute the failure | t01 (TM2 sleep, only 1 h): its turn completed after the thaw, then failed with `native_session_lease_lost`, so the task needed Retry. This reproduces the product failure. t02 (TM1 alone, 113 s server-only freeze): **pass**. The failure follows the database clock jump, not the pause. | **Product failure found (not fixed in this PR).** A local native run whose host sleeps longer than the 20-minute execution lease loses its completed work. After the wake, the provider finishes the turn and the brief is saved. Then result acceptance (`native-session-executor.ts`, the transaction at about line 12031) requires `native_run_finalizations.lease_expires_at > clock_timestamp()`. The database clock has moved on, so the run fails with `native_session_lease_lost`, and the task shows "Couldn't finish this attempt … Retry". No other boot claimed the lease: the owner, attempt and controller identity are unchanged, and the server's monotonic deadline did not expire. The lease is fenced by wall-clock expiry alone. A safe fix changes lease fencing semantics, which is a heartbeat-boundary decision, so I report it here rather than include a fix in this harness PR. Reproduce with `--workloads B5 --faults TM2` (`--time-age-hours 1` is enough; see `w-20261110T051535-s304`). ## Risks - Harness-only change; no product code changes. Waves that do not use TM faults are unchanged. `TimeShiftedApi` returns responses unchanged when no shift was applied. - `SIGSTOP` goes only to the wave server's own process tree. The tree is found from the server PID, and Postgres is excluded. Every frozen process gets `SIGCONT` in a `finally` block and in a process `exit` hook. - Aging is a model: the database becomes what it would be if everything happened the age earlier. Times outside Postgres (in-memory timers, runner state files), times inside `text` columns, and HMACs over a time are not moved. The README lists these limits. Agent JWT expiry is not covered: native runs mint a token for each tool call. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
549ca46dae |
fix(runner): retain Claude GitHub broker settings through the Rust sidecar (#20463)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Shared agents use the instructing person's GitHub identity for managed commands. > - Native Claude starts through the Rust daemon and a TypeScript ACPX sidecar. > - The Rust launch filter kept the command launchers but removed their broker transport. > - Shell commands then reported `GitHub capability_missing`, although connector tools worked. > - This pull request retains that transport for managed Claude launches. > - Each command can acquire its current identity through the existing broker checks. ## Linked Issues or Issue Description **What happened?** Native Claude could use GitHub connector tools but could not authenticate `git` or `gh`. The Rust daemon removed the broker URL and session/bridge capabilities before starting the ACPX sidecar. Reconnecting the account did not repair this process environment. **Expected behavior** Managed shell commands use the current authorized GitHub identity. The provider receives broker capabilities, while raw GitHub tokens remain confined to individual commands. **Steps to reproduce** 1. Start native Claude through the Rust Runner with managed GitHub launchers. 2. Run `gh api user` and a private repository ref lookup through Bash. 3. Observe `capability_missing` and no request to the broker. **Paperclip version or commit** Reproduced from source at `2c2186706ef4bb352b662f9d8faebefe47fb52ac`. The repair is based on `a5e01634f69f18dfe10a0bf709df02d9e6a2135b`. **Deployment mode** Native Claude through ACPX. Local acceptance used macOS arm64 and Node 25.6.1. The reported failure also affected a Linux sandbox. Related: #18212 preserved launcher paths. #18850 repaired the TypeScript filter. This change repairs the preceding Rust filter. The duplicate search found no open repair for this boundary. ROADMAP.md already lists managed personal GitHub identities. ## What Changed - Retain the broker URL and both broker capabilities when Claude has a nonempty managed launcher directory. - Keep raw GitHub tokens, unrelated secrets, empty launcher configurations, and sibling ACPX providers outside that exception. - Exercise the regression through the public sidecar initialize protocol in the existing Rust integration suite. Document the restart requirement. The Rust sidecar transport owns this filter. Credential resolution, company access, approval, and run identity policies keep their existing owners. No schema or API changes are needed. ## Verification - Red/green: the subprocess regression failed on the missing `PAPERCLIP_GITHUB_BROKER_URL` before the fix, then passed with all three broker settings and the negative controls. - Passed before moving the regression: 615 Rust core tests, with 3 existing ignored tests. After moving it into the integration suite, all 20 sidecar integration tests and the 9 remaining sidecar unit controls pass. The moved regression was also run against the unfixed source and failed on the missing broker URL. - Passed: 85 TypeScript environment, launcher, sandbox, and configured-environment controls. These include operation identity changes, revocation, recovery, and exclusion of capabilities from persisted environment data. - Passed: Runner Rust typecheck and format check, plus the Runner build through `node scripts/agent-preflight.mjs --skip-rust-debug`. - Live before/after: real `claude-opus-5-5` through the actual Rust daemon and built ACPX sidecar reproduced the failure with the old binary. Neither shell command reached the broker. With the fixed binary, `gh` identified the connected account and `git` read private repository refs. A second turn retained a workspace file and authenticated again. After the controlled broker denied access, the next command failed without reusing the prior token. No GitHub writes were made. - Live release cutover: saved a session with the unfixed daemon, suspended it, and restored it with the repaired daemon on `aa9f5b30614c011ebc3fe4509e11173b766c1fea`. The same session and workspace marker survived. `gh` authenticated after restoration, then failed when the controlled broker revoked access. - Live acceptance used an Anthropic API credential and a loopback broker with an authorized operator GitHub credential. This isolates the repaired transport; it does not qualify the hosted subscription credential or production grant resolver. - The first PR review gate did not recognize the in-source Rust test. The regression now lives in the existing integration test file, and the same test-coverage gate passes locally. - `pnpm verify` returned its documented exit 2 and required broad CI coverage. The final head passed the full CI typecheck, build, and test lanes. - Greptile reviewed `64cca69b76073cc3185b28334465dee48fbadb29` at 5/5 with no actionable findings or open review threads. Its check passed on that head. The same head is mergeable with no conflicts. - Final head `64cca69b76073cc3185b28334465dee48fbadb29` passed [full PR CI](https://github.com/paperclipai/paperclip/actions/runs/42761401396), [Windows Rust](https://github.com/paperclipai/paperclip/actions/runs/42761401117), and [isolated Runner compilation](https://github.com/paperclipai/paperclip/actions/runs/42761400973). Typecheck, build, all test shards, Runner checks, browser checks, and aggregate verification passed. No checks are pending or failed; five conditional jobs were skipped. ## Risks - The Claude sidecar now receives the intended broker capabilities. The existing broker must continue to enforce active-run identity and current connection access. Raw GitHub tokens remain excluded from the provider launch environment. - Previous-release checkpoint and profile formats carry forward unchanged. A running provider must retire and restart normally to inherit the repaired environment. No profile reset or data migration is required. - Hosted recovery requires delivery of the updated Runner binary. This PR does not deploy or retry existing tasks. Live acceptance has not exercised Linux sandbox confinement or a GitHub write. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning. Used repository inspection, code editing, and shell execution. The context window size is not exposed. Anthropic `claude-opus-5-5` was exercised as the acceptance-test provider. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a5e01634f6 |
feat(ui): ask self-hosted users to star Paperclip on GitHub after 5 done tasks (#20453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Many teams run Paperclip self-hosted, so the project grows when those users tell others about it > - A GitHub star is the cheapest way for a user to do that, but the app never asks > - Asking too early is noise. A user who has finished a few tasks knows whether Paperclip helps them > - This pull request adds a one-time, dismissable modal that asks self-hosted users for a star after their company has 5 done tasks > - The benefit is more visibility for the project, and no repeated prompt for the user ## Linked Issues or Issue Description **Subsystem affected** Board UI (dialogs shown in the main app shell). **Problem or motivation** Self-hosted users have no prompt to star the repository. Users who like Paperclip often do not think of it. A prompt at the right moment helps other teams find the project. **Proposed solution** Show a small modal once the selected company has 5 tasks with status "done". The modal shows the Paperclip mark, the headline "You completed 5 tasks!", a short body, a "Star on GitHub" button and a "Not now" button. It shows only once per browser. Closing it in any way counts as seen. **Alternatives considered** A permanent banner or sidebar link. Both stay on screen after the user decides, so they cost attention every day. A one-time modal costs attention once. **Roadmap alignment** This is a small UI addition. It does not overlap with planned core work in ROADMAP.md. No related public issues or PRs found (searched for "star", "github star"). ## What Changed - Added `GitHubStarPrompt` (`ui/src/components/GitHubStarPrompt.tsx`). It reads the health endpoint and shows only on self-hosted instances (never on Cloud), never during onboarding, and only after 5 done tasks in the selected company. It waits for any other open dialog to close. It stores a "seen" flag in `localStorage` when it opens, so a reload does not show it again. - Mounted the prompt in `BoardDialogs`. - Exported the modal-detection helpers (`MODAL_SELECTOR`, `useModalOpen`) from `AnnouncementWell` so both components use the same rule. - Added `PaperclipMark`, the mark alone from the existing lockup geometry. The lockup and the mark now share one path constant. - Added the `issues.starPromptDone` query key. - Added tests for when the prompt opens and when it stays hidden. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/GitHubStarPrompt.test.tsx src/components/AnnouncementWell.test.tsx` - `pnpm check:token-gates` (all gates clean) - `pnpm typecheck` - Manual: on a self-hosted dev instance, mark 5 tasks in one company as done. The modal opens. "Star on GitHub" opens github.com/paperclipai/paperclip in a new tab. Reload: the modal does not open again. To see it again, delete the `paperclip.github-star-prompt.seen` key from `localStorage`. ## Risks - Low risk. The change is UI only and additive. - The prompt runs one small issues query (limit 5) while it is still eligible. After it is seen, the query is disabled. - If `localStorage` is not available, the prompt treats itself as already seen and does not show. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model id `claude-opus-5-5`, run through Claude Code with tool use (shell, file edits). Standard context window; extended thinking at low effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: squadbot <squadbot@squadbots-Mac-mini.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.5 |
||
|
DottaandClaude
|
2c2186706e |
test(restart-chaos): human-in-the-loop journeys (Q2, Q3), journey triggers and a journey x fault matrix (#20448)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane while real agents work, and grades each task. > - Almost every harness workload is "the agent does a file task and finishes". Few workloads have a person in the loop. > - On 2026-11-09, production failures came from journeys with a person in the loop: a saved answer that could not reach a dependency-blocked task (#20363), and an app-action approval that a routine OAuth refresh or a one-hour deadline made invalid (#20340). > - No workload exercised these journeys, so no wave could catch them. > - This pull request adds journey workloads with deterministic oracles and journey-specific fault triggers. It also adds a journey × fault matrix and a way to run a wave against an older build. > - The benefit: one command crosses every journey with every fault kind, and the same wave proves red before a fix and green after it. ## Linked Issues or Issue Description Refs #20363, Refs #20340, Refs #20336, Refs #20432 ## What Changed - New workload **Q2** (`journeys.ts`, `journey-driver.ts`): the agent saves a question with its native `request_human_input` tool. The board adds an unfinished `backlog` prerequisite, marks the task `blocked` by it, and answers through the task-thread respond route. The oracle requires that the answer reaches the agent in bounded answer mode while the task stays blocked, that the prerequisite relation stays, that no run has `issue_dependencies_blocked`, and that no stale Retry notice stays at rest. Then the board finishes the prerequisite and the agent finishes the task. - New workload **Q3**: a harness-owned OAuth 2.1 + MCP app on loopback (`journey-oauth-app.ts`, discovery, dynamic client registration, PKCE, rotating refresh tokens, one write tool). The board connects it through the Apps routes and sets the tool to Ask first for a scripted `process` agent. When the review is pending, the harness moves the review's saved timestamps and the token expiry back 61 minutes in the wave's own database, runs Check connection (which refreshes the token through the app), and approves. The oracle requires one review, a refresh between request and decision, an accepted approval, exactly one `tools/call` with the reviewed arguments, an `executed` request, and an agent report that carries a receipt only the provider result contains. - New triggers `asked` and `answered` fire a restart fault 0–3 s after the saved request is first seen and after the board's response. They work for the existing **Q1** saved question (#20432) too. The board answers before an `answered` fault fires and after every other planned fault. - New `--matrix` option: one task per workload × adapter × (a `K0` no-fault baseline, each restart fault at each of the workload's own triggers, each provider message fault). `K0` is a new task fault kind with no fault. - New `--server-from <checkout>` option: every server start and restart of the wave runs another prepared checkout, for example the parent of a fix, with the same harness code. - Q3 setup allows unpriced usage on the wave's company budget policy, because a scripted process run records usage with no price and the default policy pauses the whole company. - README documents the journeys, the triggers, the matrix and the red-before-fix recipe. Unit tests cover the oracles (pass and the pre-fix failure signatures), the driver's ordering, the matrix plan, option validation and the OAuth app. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 295 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Live local waves, codex, one wave at a time (shared machine lock), `--concurrency 2`, at most 3 tasks: | Wave | Server build | Command | Result | | --- | --- | --- | --- | | `w-20261110T024308-s106` | this branch (master `5570debc5b` base) | `--workloads Q3 --faults K1 --matrix --seed 106` | 3/3 pass (K0, K1^asked, K1^answered); the app saw one refresh before each decision and one `tools/call` | | `w-20261110T024447-s106` | `--server-from` parent of #20340 (`0e325e8d52`) | same | 3/3 fail as expected: approve returns 409 `action_expired` ("Tool review has expired"), no `tools/call`, task stays `in_review` | | `w-20261110T024644-s107` | this branch | `--workloads Q2 --faults K1 --matrix --seed 107` | K1^asked and K1^answered pass. The K0 cell failed only because the oracle then required a comment; the agent had reported `ANSWER-…` in its bounded turn's run result 14 s after the answer, while blocked. The oracle now accepts either. | | `w-20261110T030341-s107` | `--server-from` parent of #20363 (`eebdfe71b1`) | `--workloads Q2 --faults K1 --matrix --seed 107` | 3/3 fail as expected: `answer_reached_agent_while_blocked`. No run starts between the answer and the finished prerequisite (6 minutes of ready server). | | `w-20261110T033426-s108` | this branch, rebased onto #20432 | `--workloads Q1,Q2 --faults K1 --triggers asked,answered --tasks 4 --seed 108` | Q1^asked and Q2^asked pass. Q1^answered and Q2^answered fail on a product gap (below). | | `w-20261110T020037-s102` | this branch, before the rebase | `--workloads Q1,Q2,Q3 --faults K1 --triggers asked,answered --tasks 3 --seed 102` | Q1 with `asked` and `answered` K1 restarts: both done with the answer. | Waves `…-s107` (pre-fix) and `…-s106` (pre-fix) used the same harness code as the green waves. Only the server build changed. **New product failure (not fixed here).** A K1 restart 0.3–3 s after the board's answer can lose the answer turn's native run: - `w-20261110T033426-s108/t04` (Q1, K1 2.9 s after the answer): run `a40cc8f7` resumes with `resume_dead_runner`, then fails `runner_state_identity_mismatch: authority_indeterminate` (`adapter_failed`). The task shows "Couldn't finish this attempt. Your message is saved." and needs Retry. Retry works. - `w-20261110T033426-s108/t03` (Q2, blocked task, K1 0.27 s after the answer): run `5b5355de` is `process_lost`. Its bounded retry `92a57f50` fails `native_session_recovery_failed` (`session.snapshot` failed). The Retry click starts `0e6aa9d1`, which fails `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The task then waits with `retryWait.reason=process_identity_missing` and stays blocked without the answer. - Suspected path: native runner restart reconciliation of the runner state identity (`resume_dead_runner`, journal command settlement) in `server/src/services/native-runtime/` and `packages/paperclip-runner/`. Host load was 8–14 during the wave. ## Risks - Test harness only. No product code changes. - Q2 depends on the model following its instructions. The oracle accepts the answer in either a comment or the bounded turn's run result, because both were seen. - Q3 shifts persisted timestamps only in the wave's isolated database. It does not shift unrelated tables, so it simulates elapsed time for the review and the token, not for the whole instance. - `K0` and the new triggers add values to shared registries (`types.ts`, `scheduler.ts`). Existing plans keep their meaning. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.4 |
||
|
DottaandClaude Opus 5.5
|
120017e39d |
fix(workspaces): read saved workspace files in place instead of extracting the whole archive (#20442)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a cloud task finishes, its workspace is saved as an encrypted
snapshot and its sandbox is released. The board's Files view then reads
that saved copy
> - Saved Files access downloaded, decrypted and extracted the whole
archive for every request, and refused archives over 512 MiB with
`too_large`
> - A finished task in a Paperclip-sized repository saves a snapshot of
about 730 MiB, so its files could not be listed or opened at all
> - The archive format already records each file's size and SHA-256, and
the snapshot records the manifest digest, so single files can be read
and verified without the rest of the archive
> - This pull request reads the saved archive in place, one verified
file at a time
> - The benefit is that users can open any file of a large finished
workspace quickly, with bounded memory and no whole-archive transfer
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves on Daytona with a real Paperclip clone, the
committed workspace snapshots were 763,944,535 bytes. After the sandbox
was released, every saved Files read returned 422 `too_large` (506 reads
in total), because `createSavedWorkspaceLoader` refuses archives over
`maxArchiveBytes` (512 MiB). The board Files view uses the same loader,
so it fails the same way.
**Expected behavior**
A user can list and open any single file of a saved workspace of
Paperclip-repository size without the server extracting or buffering the
whole archive. Memory and time stay bounded per request, and integrity
is still verified.
**Steps to reproduce**
1. Run a cloud task whose workspace is a large repository, so the final
snapshot exceeds 512 MiB.
2. Let the task finish and its sandbox be released.
3. Open the task's Files view, or call `GET
/api/issues/:id/file-resources/content?workspace=execution&path=…`: it
returns 422 `too_large`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
5ab98b0063 |
test(restart-chaos): add database contention workload and universal checks U1-U4 (#20421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks on an isolated control plane and grades them. > - Two production failures on 2026-11-09 passed the harness. Its tasks made little output and little accounting, so database write contention never happened (#20385). Its grading was per workload, so a task stuck for a reason no scenario targeted was not caught (#20361). > - The harness needs a contention workload and checks that apply to every task. > - This pull request adds the L1 workload, a `--db-pressure` knob, and the universal checks U1–U4. > - The benefit is that the #20385 failure class and unexplained stuck or misleading tasks now fail a wave. ## Linked Issues or Issue Description Refs #20385, #20383, #20361. **What happened?** The harness ran real embedded PostgreSQL, but no task streamed enough run-log output while usage receipts committed. The #20385 receipt race (SQLSTATE 40001, retained outbox, `native_event_replay_conflict`) could not occur. Also, no rule checked every task for an owner or an actionable item, for retry loops, for diagnosable failures, or for stale notices. **Expected behavior** A chaos wave can create run-row contention between run-log writes and accounting writes and check that accounting stays exact. Every task of every wave gets scenario-independent liveness and notice checks. **Steps to reproduce** Run an L1 wave with `--db-pressure 4` against the parent of the #20385 merge (red), then against master (green). Commands are in Verification. ## What Changed - `contention.ts` (new): workload L1 (five 3,000-line shell outputs as five tool calls, then a brief), its accounting oracle, and `RunRowPressure`, the `--db-pressure <writers>` knob. The writers run `update heartbeat_runs set updated_at = updated_at` on the wave company's running runs, in the wave's own isolated database only. - L1 oracle: no `native_event_replay_conflict` or `runner_event_commit_stalled`; each producer event journaled once; a terminal receipt for each succeeded native run; finished runs accounted; per-run cost equals its cost events; the board's spend equals the cost-event sum with no runs pending accounting; and proof that the task produced contention. - `universal.ts` (new): U1 liveness, U2 no silent retry loop, U3 diagnosable failures, U4 no stale notices. `grader.ts` grades them on every task. U1, U2 and U4 fail a task. U3 is a warning. - `invariants.ts`: `readUniversalDbEvidence` reads failed-run error text and recovery-action causes from the wave database at wave end, for U3. - `universal-audit.ts` (new): runs U1–U4 over saved wave evidence to measure false positives and bound headroom. - `contention-db.test.ts` (new): on real PostgreSQL, under the pressure writers, the pre-#20385 repeatable-read receipt write exhausts its retries with 40001 and the current writer commits. - Small registry edits: `types.ts` (L1 id, `dbPressure`, optional `contention` and `universal` fields), `workloads.ts` (L1 entry), `config.ts` (`--db-pressure`, L1 not a default), `main.ts` (start and stop the writers, read database evidence before teardown), `ledger.ts` (universal verdicts and warnings in the ledger and `SUMMARY.md`). - README: "Database contention" and "Universal checks" sections, with the audit table and bound reasons. ## Verification - `contention-db.test.ts` (real embedded PostgreSQL): with `--db-pressure` writers (4), the pre-#20385 repeatable-read receipt write exhausted its retries with 40001 in 8 of 8 attempts; the current READ COMMITTED `persistUsageReceipt` committed 8 of 8. This is the harness-level red/green for #20385. - Live L1 waves (`--env local --adapters codex,claude --workloads L1 --faults K1 --concurrency 4 --tasks 4 --seed 20385 --db-pressure 4`), red on the #20385 parent `bf818280f2` with this harness overlaid and green on this branch: queued behind the shared one-wave-at-a-time lock and not yet run. The first red attempt failed at startup (the fresh checkout lacked a built plugin SDK; no tasks ran). Live results are not claimed here. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: all harness unit tests pass, including the new `universal.test.ts`, `contention.test.ts` and `contention-db.test.ts`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Universal-check audit over 74 saved waves (1,592 tasks): U1 7 flags (one C1 stall and six R1 runs queued 8 min behind a retry row, the #20361 shape), U2 1, U3 22, U4 0. None of U1, U2 or U4 flags a passing task or an all-pass wave. U3 flags 3 passing tasks (all `adapter_failed`), so it ships as a warning. ## Risks - U1, U2 and U4 can now fail a task that passed R1–R6 and T. The audit found no such case in 70 saved waves. The bounds are environment-configurable. - `--db-pressure` is opt-in and writes no values. It writes only to the wave's own temporary database. - Other agents edit the same harness. This change keeps registry edits small; `grader.ts` and `invariants.ts` changes are additive. - No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0c7a628fd1 |
test(chaos): read remote R1 files from the execution workspace and record why a read failed (#20440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - In replays w-20261110T010031-s9913 and w-20261110T012332-s9950 the tasks recovered (R1, R3, R6 pass), but all 506 content reads returned 422, even after the closing save left the workspace clean (`snapshotId` set, no writer, `location: null`) > - The harness discarded each error, so the waves could not show which 422 the server returned > - This pull request asks for the execution workspace explicitly and records the last error per file in the task record and the timeline > - The benefit is that the next wave names the exact cause instead of a blind `module_present` failure ## Linked Issues or Issue Description Follow-up to #20348. **What happened?** Every file read for a finished Daytona R1 task returned 422, and the response was discarded. **Expected behavior** Failed reads leave their reason in the evidence. **Actual behavior** Only the outcome, a `null` file, was kept. ## What Changed - File reads add `workspace=execution`, so the server reads the issue's execution workspace and not the `auto` choice. - `readRemoteWorkspaceFile` and `captureRemoteWorkspaceFiles` report each failure through a callback. The observer keeps the last message per file in `record.remoteFileErrors` and logs `remote_file_read_failed` once per distinct message. - The remote-workspace file reader stays in `remote-workspace-files.ts`; `observer.ts` is at 993 lines. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 263 passed. The test server now requires `workspace=execution`, so the existing remote tests were red before, and a new test records the 422 `workspace_unavailable` message. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - If a Daytona task's files only exist in a project workspace rather than its execution workspace, an explicit `execution` selector reports that clearly instead of falling back. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.3 |
||
|
DottaandClaude Opus 5.5
|
d064bd21e4 |
perf(plugins): pick each lifecycle resource's head in one pass (#20406)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins receive resource lifecycle events (agent, project and agent-runtime changes) through a durable inbox > - The inbox lists an event only when it is the first unacknowledged event of its resource, so each resource delivers in order > - The list query checked that rule with a `NOT EXISTS` for every unacknowledged event. Each check walked the resource's whole acknowledged history > - A resource with a long acknowledged history and a backlog behind a deferred head costs history × backlog probes. On the hosted staging instance three copies of this statement ran for more than 23 minutes each, on a database that was at 6–8 of 8 vCPU all day > - This pull request picks the deliverable head of each resource in one pass > - The benefit is a list call that is linear in the company's events, and database CPU that other requests can use ## Linked Issues or Issue Description **What existing behavior does this improve?** `pluginLifecycleInbox(...).list()` in `server/src/services/plugin-lifecycle-inbox.ts`. The delivery worker calls it in a loop and when it checks for idle work. **Subsystem affected** Plugin lifecycle delivery (server service, PostgreSQL query). **Current behavior** Neon "Active queries" for the staging database on 2026-11-09 showed three copies of this statement at 1,381–1,401 s each. In the same view, trivial primary-key reads waited 100–280 ms. Locally, one resource with 2,000 acknowledged events and a 2,000-event backlog takes 3.5 s for one `list()` call. The cost grows with acknowledged history × backlog. **Proposed behavior** Only the head of a resource is deliverable: its unacknowledged creation if it has one, otherwise its lowest unacknowledged event. This is exactly the set that the old `firstForResource` predicate admits. The query now selects these heads with `DISTINCT ON (resource_type, resource_id)` ordered by `(action <> 'create'), id`, and then applies the page cursor, order and limit. **Reason and benefit** The same fixture lists in 8–24 ms instead of 3.5 s. Long-running copies of this statement no longer hold database CPU that every request needs. **Breaking changes** None. `acknowledge()` keeps its per-event eligibility check. ## What Changed - `pluginLifecycleInbox.list()` uses a `lifecycle_heads` CTE with `DISTINCT ON` to pick one head per resource, then filters by the `afterId` cursor and orders by id with the limit. - New test in `server/src/__tests__/resource-lifecycle-events.test.ts`: one resource with 2,000 acknowledged events and a 2,000-event backlog, plus a second resource. It asserts the exact heads in id order and a list time below 1 s (3.5 s before, 8–24 ms after). ## Verification - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts -t "acknowledged backlog"` — red on `origin/master` (3,546 ms), green with the change (24 ms, 8 ms). - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-assignments.test.ts src/__tests__/agent-runtime-hooks.test.ts src/__tests__/project-instance-secrets.test.ts` — the existing ordering, paging, per-plugin isolation, backfilled-creation and late-commit tests pass. (One `agent-runtime-hooks` case is timing-sensitive on a loaded machine and also fails on `origin/master` there. It passed 22/22 on rerun.) - `pnpm verify` selection and CI. - After deploy: Neon "Active queries" no longer shows long-running `resource_lifecycle_events … NOT EXISTS` statements. ## Risks - Low risk. The result set is the same: per resource, the unacknowledged creation, otherwise the lowest unacknowledged id. The cursor still applies after the head is chosen, as before, so paging past failed heads does not skip later events. - Each call still reads the company's events of the requested types once. A company with millions of acknowledged events still pays one linear pass per call. A per-plugin acknowledgment watermark would remove that and is out of scope. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser, code execution). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0934b834c4 |
perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path > - Paperclip manages agent work and preserves its workspace files. > - Online migration captures local files and publishes remote workspace authority. > - Each capture verifies encrypted readback, restores the archive and compares the current source. > - A cohort kept every completed parent payload until the whole cohort published. > - A larger cohort could retain many complete copies while later members still worked. > - This pull request releases only finished parent payloads before a capture worker starts another member. > - Nested-child proof, final fresh checks and atomic publication keep their existing owners. ## Linked Issues or Issue Description Refs: #20418, #20437, #20441. **What happened?** A cohort retained each member's capture archive, verified archive, comparison archive and restored tree until atomic publication. The capture worker limit did not limit that retained parent data. **Expected behavior** Release completed parent payloads once their consumers finish. Keep files that a later publication check still needs. **Steps to reproduce** Run the added held-peer regression against the previous owner. It completes one parent, holds a later member and finds all four completed parent paths still present. ## What Changed - Release three named parent archives and the restored tree after a successful online cohort capture, before that worker takes another member. - Keep parent scratch roots and all nested-child verification areas. Keep every final fresh child and publication check. - Retain payloads for archived repository intents. Use the existing writable-tree cleanup for read-only restored directories. - Add real filesystem regressions for early release, retained child proof, late child mutation and the final archived-intent path check. Update the source-fence guide. This branch incorporates the final #20441 owner and both distinct saved-relative-path regressions. Its scratch-specific production change is nine added lines and one removed line. Single imports and saved journal/archive formats stay unchanged. ## Verification - Canonical Node24 preflight passed. The Runner build came from the content cache. After the current-master merge, the preflight checked 1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin SDK prerequisites passed. - Genuine RED on the unchanged #20441 owner: one held-peer case failed at four retained payload paths; the archived-intent control passed. The run took21.14s, with221 unrelated cases intentionally unselected. The first sandbox-restricted setup attempt hit DNS errors and was stopped; the normal network-qualified preflight then passed. - Initial treatment: four selected PostgreSQL/filesystem cases passed in38.61s, with219 intentionally unselected. The held peer observes released parent payloads and retained child areas. A later child mutation still refuses the whole cohort and publishes no authority or activity. - The archived-intent control observes all parent payloads at the actual final saved-relative-path check. The changed path refuses publication and keeps task/repository state intact. This exercises its actual singleton owner; archived-only sources do not enter the online cohort API. - After a normal merge of current master and the final #20441 test, the five affected cases and server TypeScript passed. No full local matrix was repeated. Initial CI failed the inherited recovered-upload assertion because the new retention test leaked its archive and filesystem spies. That case passed alone but failed in the adjacent three-case sequence (two pass, one fail); scoped `finally` cleanup made the same sequence pass three of three in17.15s, with191 unrelated cases intentionally unselected. No upstream assertion or production code changed. Server TypeScript passed again. All required [CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511) passed on final normal-merge head `604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and one neutral; no pending or failing checks). The previously failing server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with no open threads. The merge resolves the duplicate #20441 squash insertion; both tested owner and test files are byte-for-byte identical to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior head received [Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885); its archived-cohort coverage warning was [answered using the actual public admission boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017). - `git diff --check` passed. No hosted cleanup, migration, provider action or deployment was performed. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication' pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reduces retained parent data. It does not establish a total disk bound. Active capture/encryption/readback files and retained child areas still need measured capacity. - Cleanup failure uses the existing worker abort and peer-join path before publication. Original source files, uploaded objects and journals remain intact. - Archived repository intents still need their restored payload at publication. Those payloads stay retained. No capture limit, budget or integrity guard changes. - Saved operations and archive identities carry forward unchanged. Resume still uses the existing full source and authenticated object checks. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f00f2b83e |
fix(workspaces): recognize committed imports in linked source scope (#20444)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration fences every local Git source that a capture can reach. > - A linked worktree can remain on disk after its workspace is migrated to cloud storage. > - The source inspector lists local workspace rows, so it then treats that linked worktree as unregistered. > - This pull request uses the existing canonical import journal and snapshot proof to recognize its exact original physical root. > - The benefit is that a later linked workspace can migrate without weakening source ownership or stopping agents. ## Linked Issues or Issue Description **What happened?** After one linked Git workspace commits its canonical migration, its local files and Git registration can remain. The workspace row now uses `cloud_sandbox`. A later migration of a related local workspace fails with `online_workspace_source_scope_unproven` because the original linked root is no longer in the local-row list. **Expected behavior** The inspector can recognize the exact original physical root from the same-company successful migration journal and its committed or superseded snapshot. Unknown or changed provenance must still fail. **Steps to reproduce** 1. Create two local Git workspaces with a shared worktree registration. 2. Complete canonical migration for one workspace while retaining its original local files. 3. Inspect or migrate the other workspace. Before this fix, the inspector returns no source scope. **Paperclip version or commit** Reproduced on `f73ff4ace9` with real PostgreSQL, Git, filesystem, and canonical snapshot storage. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:online_workspace_source_scope_unproven` Refs #20418, #20436, #20441. This fix changes source registration only. ## What Changed - Let the existing imported-source owner match optional exact linked physical roots in addition to its existing nested-child paths. - Require each added linked root to have a finished successful same-company journal, an exact committed or superseded snapshot binding, and the same manifest digest. - Include those proven physical roots in the online scope and its source fingerprint checks. - Retain Git root, backlink, pointer, ownership, company, foreign writer, source lock, and publication reinspection guards. - Keep existing nested-child behavior and all persisted schemas unchanged. Previous-release canonical imports remain readable. No runtime authority comes from an arbitrary provider path or remote authority. ## Verification - Genuine baseline RED: both committed and superseded imported sibling cases return a null source scope after a real successful import. - GREEN: 17 focused real-PostgreSQL/FS/Git tests pass. They cover both imported states, the original issue source fence, changed journal/snapshot/manifest/unfinished provenance, a foreign local alias, unregistered links, backlink/pointer corruption, and a listing change during upload. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'imported linked Git source|foreign local writer|unproven linked|revalidates the linked Git listing|nested Git wrapper.*shared|nested Git wrapper blocked'`. - Server layered typecheck, module boundaries, and `git diff --check` pass. - Actual controlled staging canary: the canonical API committed both targeted workspace snapshots in 310.249 seconds. The owned job closed with exit 0 and all owned processes joined. A separate read-only inventory confirmed that both workspaces left the legacy provider set. This used a reviewed private administrative artifact; the serving app image and normal release path stayed unchanged. - Final head `1759ac6f40a1b35f79587b246de8b36e7a371537`: 56 successful checks, 4 intentional skips, and 1 neutral status. Greptile and Ponytail both report 5/5 with no findings or unresolved threads. - Initial CI had unchanged Runner-support and workspace-recovery timeouts, plus an unrelated heartbeat plugin-environment timing/fixture failure. The same owning inputs passed on the sibling PR. Each failed job then passed one targeted retry on unchanged product bytes. The original failures remain recorded in the CI history. ## Risks - This recognizes a retained original root only from the existing committed import ledger. It does not treat arbitrary cloud workspace paths as local ownership. - Each physical root still passes the existing Git identity and foreign-scope checks. Publication re-inspects the source scope under the existing source locks. - This adds no object reads, provider calls, recurring work, capture algorithm changes, or data exclusion. The existing import query keeps its current 10,240-row limit and runs during explicit migration inspection only. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1110.0-canary.2 |
||
|
DottaandClaude
|
1b52f359dd |
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0991d9bda1 |
fix(workspaces): protect migration sources and add guarded recovery service (#20443)
## Thinking Path > - Paperclip manages agents and their task workspaces. > - Online migration saves files before it publishes remote workspace authority. > - A saved migration journal can survive the importing process. > - Git cache expiration did not count that journal as a consumer. > - Expiration could remove a source and change its fingerprint while its import was incomplete. > - This PR protects unfinished imports and exposes a guarded internal service transition for an already affected cohort. > - New imports can preserve the current sources while all old recovery data stays retained. ## Linked Issues or Issue Description **What happened?** Git cache expiration could archive and remove a workspace after an import process ended but before its saved cohort finished. The saved source fingerprint then differed from the current archived workspace. Exact cohort resume correctly refused that changed source. Reopening the workspace cannot recreate the old metadata or fingerprint. **Expected behavior** An unfinished migration remains a durable consumer after process loss. If an older server already expired some sources, one canonical transition can fail the exact unpublished cohort without losing its snapshots, objects, source history, or recovery obligations. New attempts use the current verified sources. **Steps to reproduce** 1. Start a cohort import of eight Git worktrees with local changes. 2. Interrupt it after four snapshot uploads. 3. Let an older server expire two idle source worktrees. 4. Resume the saved cohort. It refuses the changed archived sources. Related: #20429 preserves exact cohort identity when independent companion imports change only the scope proof. #20424 preserves prepared recovery state for archived sources. This PR handles completed source expiration, not a scope-only change. Public searches found related expiration work and no matching compensation owner. ## What Changed - Count saved migration markers and unfinished migration journals as Git expiration consumers. Keep failed migration retention holds as consumers too. - Expose `supersedeExpiredRemoteWorkspaceMigrationCohort` as an internal service API for a privileged operator with a real database handle and reviewed company/cohort inputs. It is called directly by the PostgreSQL/Git test and reviewed one-time administrative qualification. It adds no CLI mode or HTTP route. The existing CLI still refuses a damaged cohort; this PR does not claim automatic CLI recovery. - Require an exact unpublished cohort of up to eight members. It requires completed expiration and an unexpired restore recipe for each missing member. The other members must retain their saved source fingerprints. - Recheck current source and recovery state under source, lifecycle, finalizer, and row locks. One transaction fails all old journals, writes audit records, and removes only their exact matched active markers. - Keep old cohort proofs, operations, snapshots, and object references. Genuine failure receipts use the existing failed-save retention hold. This transition does not report migration success. - Reuse the existing bounded source-lock queue when `sourceLockWaitMs` is set. Keep fast refusal by default. Cancellation targets only this waiter, and current checks still run after admission. - Record predecessor IDs on new attempts and reserve a snapshot generation above the existing ledger maximum. New success does not retire the old hold. - Document the owning transition and retention rules. ## Verification - The two new journal-consumer cases failed under the old expiration predicate and passed under the new predicate. - The full new recovery case passed in 21.12 seconds with real PostgreSQL locks, Git repositories, encrypted local object storage, and archive restores. It covers four retained uploads, two real expirations, unchanged old resume refusal, authority change refusal, transaction rollback, a held native finalizer, idempotence, eight new imports, dirty/untracked file restores, and a retention pass 30 days later with no object deletion. - Server TypeScript and module-boundary checks passed. - On `7cec8c915acfad2d08874eaf26abaa852f560219`, the queued-admission regression failed before forwarding the option (24.07 seconds), then passed (22.25 seconds). It verifies a real held project lock, cancellation that preserves the holder and all journals, release into full compensation, retention, and eight restored fresh imports. Current server TypeScript and module/diff checks passed. The only production follow-up changes are the optional queue input and its existing-owner call. - The earlier `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c` CI run had one Runner support failure: the unchanged credential-free Python/Pi calibration exceeded its outer 25-second subprocess deadline. That job passed 6,926 other tests and skipped four. The timeout is retained as a nonpass. The changed logical head requires fresh CI and review; no test deadline was raised. - The affected cohort, Git expiration, and eligibility suites passed all 148 cases in 228.41 seconds on `aa589f3f447e36d0277c32f8b44cbe5e84a673be`. Merged current master `a9d0946e64`; the only conflict retained both independent added test blocks. The complete merged cohort suite passed all 30 cases in 70.00 seconds on `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c`, including upstream larger-cohort controls. Final-head server TypeScript, incremental preflight, module-boundary, and diff checks passed. Unchanged expiration/eligibility evidence remains valid. Fresh `809ab704e619812d8f1bf38cf5de9c65893861cc` CI and review are pending. - On `809ab704e619812d8f1bf38cf5de9c65893861cc`, the complete recovery fixture passed in 21.38 seconds after adding one valid archived recipe with no restore point. It refuses with `migration_supersession_expiration_unverified` and keeps all workspaces, journals, and snapshots unchanged. The same case then verifies queue cancellation/release, full compensation, retained history, fresh imports, and restored bytes. Server TypeScript passed. This follow-up changes only the test and service-scope documentation; product bytes remain identical to qualified `7cec8c915acfad2d08874eaf26abaa852f560219`. - Earlier fixture attempts did not pass. They exposed the selection of a held source and an incorrect expected refusal string. The fixture now selects eligible sources and asserts the canonical authority refusal. Production guards were not relaxed. - Live qualification of the exact owning product bytes completed in 230.419 seconds. All eight old operations became failed with auditable supersession lineage. All eight old snapshot rows remained byte-for-byte identical, and their object references remain retained. The existing source-lock queue waited 1.151 seconds for a normal finalizer handoff. The three complete source/recovery passes took about 218 seconds including their transaction, which explains why the earlier 180-second total budget was insufficient for that measured workload. No source proof or publication guard was skipped. Compensation made no provider or object-storage calls. - Earlier live qualifications remain nonpasses: the 180-second attempt did not finish; its instrumented successor completed source checks and refused a busy source lock before mutation. The successful bounded-queue attempt retained those failure records. Fresh archived/online imports are a separate subsequent step and are not claimed complete here. - The local checks do not claim browser acceptance, app deployment, fleet enrollment, or a completed zero-downtime cutover. ## Risks - This owner mutates migration bookkeeping. It preserves all-or-nothing row comparisons and current source/recovery locks. A changed journal, authority, writer, finalizer, expired recipe, or partial transaction refuses the transition. - Historical snapshots and objects remain pinned after new success. This intentionally retains storage. Release requires a separate reviewed action. - An older importer cannot use the successor generation contract. Use the containing canonical owner for recovery. Normal app delivery must also carry the expiration guard; a private maintenance graph does not patch a running server's cleanup worker. - The migration owner crosses 1,000 lines. The bounded compensation transition remains beside its private source inspection, fingerprint, fence, reservation, and publication contracts. Moving it now would require exporting or duplicating those safety owners. No unrelated refactor or new framework is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source editing, Git, and local execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
29a66d679e |
fix(workspaces): verify archived repository intents before publication (#20441)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration moves local files into durable, authenticated snapshots. > - Archived Git workspaces can retain an unmaterialized repository request after their local tree is removed. > - Admission cannot prove filesystem absence from the removed tree. > - This pull request checks strict request candidates against the private restored Git tree and the authenticated snapshot readback. > - Publication rechecks current repository rows without changing the request or task history. ## Linked Issues or Issue Description **What happened?** An archived workspace with an authenticated restoration recipe and an unresolved, unmaterialized repository request fails migration with `repository_preparation_unresolved`. Its original directory no longer exists. **Expected behavior** Migration can preserve the request when the actual private restored tree proves its target absent and every parent safe. It must refuse partial materialization, symlinks, or changed request state. **Steps to reproduce** 1. Archive a Git workspace with a valid saved recipe and a pending repository request. 2. Remove its original local workspace directory. 3. Run canonical workspace migration. The previous implementation refuses before it can inspect the restored tree. **Paperclip version or commit** Reproduced on `1ca3e8ea2a759e511ab2d2659339c6c770a7a2ba`. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:repository_preparation_unresolved` Refs #20416, #20424, #20436. These fixes retain their source, recovery, and ownership guards. ## What Changed - Separate strict repository request metadata from filesystem absence proof. - Admit only strict candidates backed by a canonical archived Git recipe to private materialization. - Verify target absence and safe parents before a new upload and after authenticated archive restore. - Lock and recheck current repository rows before authority publication, including recovered uploaded archives. - Preserve previous-release recipes, request rows, task selection, history, provider references, and unresolved recovery obligations. No persisted schema or public option changes. ## Verification - Genuine baseline RED: both pending and needs-repair archived requests fail on unchanged master. - GREEN: 27 focused real-PostgreSQL/FS/Git tests pass. Includes partial/symlink refusal before upload, request mutation during upload, recovered-upload readback, and existing native F/source checks. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'archived repository intent|unmaterialized repository request|unresolved repository request|pending repository|unselected archived recipe|archived prepared source|competing native F owner'`. - After rebasing onto current master, all six new cases pass on `0b6854789acc067a44e697acebdc04e51b9afe88`. Server layered typecheck, module boundaries, and `git diff --check` pass. A review follow-up adds one pending-path mutation regression (1/1 real-PG pass) without changing product code. Fresh required CI runs on final head `2a30a4b444108488ac8b51d0c46c340ee15227b9`; fresh Greptile is 5/5. Server7 had an unchanged trace-parent assertion failure; that same shard passed on the previous product-identical head and the linked-source sibling PR. The original failure remains recorded. The single targeted server7 retry passed. Final head has 56 successful checks, 4 intentional skips, and 1 neutral review status. ## Risks - Metadata admission is only a candidate check. A candidate still fails unless the private authenticated filesystem proves absence. - Publication keeps repository rows locked while it checks their current paths against the owned restored tree. A real pending-relativePath mutation now tests this final check. Existing workspace/source/native locks remain in place. - Absence is proved only in SAVED data reconstructed from the retained recipe and snapshot. This does not prove that a request never materialized before archival. Existing Git restore points can exclude ignored `.paperclip-repositories` paths outside recorded checkouts. This PR does not change that preexisting coverage or recover bytes that were never saved. - The original recipe, restore point and checkout references remain unchanged. No files are recovered into or removed from the original directory. No request is marked ready or restored. Partial target files present in reconstructed saved data remain blocked and retain their state. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |