mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
263ef18098ef47e58405f0137117745165a526aa
7309
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude
|
263ef18098 |
test(chaos): retry a rejected model's run the way its task marker does
A rejected model asks the user to choose a model (userAction choose_model) instead of offering the task-level Retry. Its "Model unavailable" marker offers Retry, which retries that run on the agent. The config wave now uses that action after it restores the model. Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
d2e8494e41 |
fix(native): match only the terminal httpConnectionFailed 404
Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
21befa1c5c |
fix(native): classify a Codex API-key unknown model as a model rejection
With API-key auth, Codex (0.160.0) answers an unknown configured model
with a Responses API 404, retries it for about a minute, and fails the
turn with "unexpected status 404 Not Found: The model `<id>` does not
exist or you do not have access to it." and codexErrorInfo
httpConnectionFailed/404. The observation only recognized the ChatGPT
400 envelope, so the run failed as a generic adapter_failed ("Native
session failed") and recovery opened a reconciliation incident for a
turn that never did work.
Recognize that terminal shape, bound to the configured model, as
native_provider_model_rejected with a closed api_key/404 diagnostic. The
task then asks the user to choose a model, and Retry runs with the
repaired model. The task interruption also carries the rejected model id
so the task page names it.
Co-Authored-By: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude
|
182a7cd4fe |
test(restart-chaos): keep process start identity out of TM2/TM3 aging (#20500)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) checks recovery under faults. TM2 (host sleep) and TM3 (aged outage) move every database time back by an age. > - The aging also moved process start times: `controller_process_started_at`, `process_started_at`, and `processStartedAt` copies in JSON. > - A process start is identity, not a deadline. The server reads its own start from the OS once and compares it with the stamp an owner saved. A real sleep never changes it. > - After a shift, every live owner looked like a different process and failed its identity check, whatever its lease said. This harness artifact hid whether a lease fix works (#20474). > - This pull request keeps process start identity unaged and makes the grader's unshift skip it too. > - The benefit is that TM2/TM3 failures now come only from deadlines that really lapse, as on a real laptop. ## Linked Issues or Issue Description Refs #20458 (adds TM1–TM3) and #20474 (the lease fix whose proof wave exposed this). **What happened?** A TM2 wave against #20474 still failed at the 300 s renewal. The fix takes back a lapsed lease only for the same owner (token, attempt, generation, boot, pid and process start). The aging had moved the saved `controller_process_started_at` back by one hour. The live process still reported its real start, so the identity comparison failed. **Expected behavior** Aging moves deadlines and event times. It never moves facts that a real sleep cannot change, such as when a process started. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304 --server-from <#20474 checkout>`. Before this change it fails 0/2 (w-20261110T072106-s304). ## What Changed - `time-faults.ts`: add `isProcessIdentityTime`. `agingStatements` skips columns named for a process start. `paperclip_shift_json` keeps JSON keys named for a process start (any spelling) unchanged. `unshiftJson` skips the same keys, so the grader does not move them forward. - `time-faults.test.ts`: a unit test for the statements and the unshift. - `time-faults-db.test.ts`: the embedded-Postgres aging test now checks that a `controller_process_started_at` column and `processStartedAt` / `processStartedAtMs` JSON keys keep their values. - `README.md`: one paragraph under **Aging** that says why. ## Verification - `npx vitest run -c tests/restart-chaos/vitest.config.ts tests/restart-chaos/time-faults-db.test.ts tests/restart-chaos/time-faults.test.ts`: 26/26 pass. Without the `time-faults.ts` change, 2 fail (the new unit test, and the DB aging test's identity check). - `npx tsc --noEmit -p tests/restart-chaos` is clean. - Test-only change. No product code. ## Risks - Low. Only the harness changes. A column or key whose name contains "process started at" is no longer aged. If such a field were really a deadline, it would no longer lapse, but none is. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2f59b9c67c |
test(chaos): fire a planned restart while the server is mid S3 transfer (#20451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness restarts the control plane while agent runs are in flight, and `--storage s3` runs the server against a loopback MinIO behind a fault proxy > - Planned restarts fire by time or by run events. A time-based restart seldom lands while the server is moving bytes to or from the store > - The riskiest storage moments are a workspace snapshot upload, a snapshot restore, and a run-log journal write. A restart in the middle of one must not lose or corrupt the workspace or the log > - This pull request adds a storage-triggered restart: the planned fault fires the moment the proxy sees a matching S3 request, and the proxy holds that request so the restart interrupts it > - The benefit is that restart-during-transfer becomes a repeatable wave with receipts that name the interrupted request ## Linked Issues or Issue Description **Subsystem affected** Restart-chaos test harness (`tests/restart-chaos`), cloud storage mode. **Problem or motivation** A K1/K2/K4 restart fires on a timer or on a run trigger. With `--storage s3`, the timer almost never lands during a snapshot upload, a snapshot download, or a run-log write. So the harness does not prove that a restart in the middle of a transfer is safe. **Proposed solution** Add `--storage-trigger <target>`. At its planned time, each task's fault is armed. The next server request that matches the target fires it. The proxy holds the request open (up to 60 s) while the harness snapshots the tasks and restarts the server. **Alternatives considered** A new `--triggers storage` id. It needs edits to the observer (near its 1,000-line limit), the trigger registry and the delay table. The storage trigger is a wave-level mode in a new module instead. ## What Changed - New `tests/restart-chaos/storage-trigger.ts`: targets `snapshot-upload`, `snapshot-part` (multipart `UploadPart`), `snapshot-download` and `run-log`; the `StorageTrigger` that arms faults, fires on a match, holds the request and records receipts. - `storage-proxy.ts`: an `observe(listener)` hook. The listener sees every request and can hold it (the existing latency path). - `main.ts`, `config.ts`, `types.ts`: registry lines only. `--storage-trigger` (needs `--storage s3` and `--triggers time`); a due fault is armed instead of fired; the wave writes `storage-trigger.json`. - `storage-trigger.json` records the matching request (method, path with upload ids hidden, declared bytes, outcome `aborted` or `completed`, status), every other request in flight, and the fault id. A fault that never matched is recorded with `missedReason`. - README: "Restart during a transfer". ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (25 files, 304 tests pass). - Red/green: `storage-trigger.test.ts` against the master `storage-proxy.ts` fails 3 of 6 tests (no hook, so nothing is held or fired); it passes with this change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local run-log, Daytona snapshot upload and download) are recorded in a PR comment as they finish. ## Risks - Low risk. Test harness only. Without `--storage-trigger`, the proxy has no listener and behaves as before. - A replay re-arms its faults at their observed offsets. It fires on the next matching request, not on the exact same request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7fef6321fe |
test(restart-chaos): remove storage fault window time from speed budgets (#20501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed against local budgets (`wave-speed.ts`) > - The speed grade already removes restart outage time (fault start to server ready) from spans and restart recoveries > - It did not remove storage fault window time (ST1 outage, ST5 rejected credentials, and so on) > - A restart recovery that correctly waits for the store inside a storage fault window then counts as a speed breach > - This pull request removes the overlap with storage fault windows the same way, and shows the removed time on its own line > - The benefit is that storage waves fail only on real slowness, not on correct waiting ## Linked Issues or Issue Description **What existing behavior does this improve?** The wave speed grade treats time spent inside a storage fault window as product slowness. **Subsystem affected** tests/restart-chaos (speed grade). **Current behavior** The ST5 waves w-20261110T050044-s503 and w-20261110T062313-s506 exited 1 on speed breaches only. Their `restart_recovery_control_plane` p95 was 62–67 s against a 4 s budget. The recoveries were inside an ST5R window and correctly waited for a valid key. **Proposed behavior** Spans and restart recoveries do not count time inside storage fault windows (from the `storage-faults.json` receipts). Restart outage and storage windows that overlap are removed once. `SUMMARY.md` shows the storage time that was removed, and `speed.json` has `storageFaultWindows` and `storageFaultRemovedMs`. **Reason and benefit** Storage waves must fail on real product failures and real slowness, not on the harness's own fault windows. ## What Changed - `wave-speed.ts`: `storageFaultWindows()` turns receipts into windows (an open window ends now). `mergeWindows()` joins overlapping windows, so `outageFreeClock()` removes shared time once. `collectWaveSpeed()` takes the storage receipts and removes their time from spans; `gradeWaveSpeed()` removes it from restart recoveries. The storage share is reported apart from the restart outage share. - `main.ts`: one line passes `storageChaos.receipts` to `collectWaveSpeed()`. - README: one paragraph in the Cloud storage section. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/wave-speed.test.ts`: 20 pass. The 3 new tests fail before this change (`storageFaultWindows is not a function`, overlap removed twice, span not trimmed) and pass after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. ## Risks - Test harness only. No product code changes. - ST2 (slow store) windows are also removed from speed budgets. Slowness during a slow-store window is expected, so this is intended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
a14ae9b5c9 |
fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent starts, Paperclip prepares the task workspace and runs the project's configured setup commands (for example the workspace provision command) > - When that command fails, the run fails as `setup_failed`, and the task stops with a Retry > - The task page shows only "Failed" and a generic recovery notice. The real error (the command, its exit status and output) is visible only on the run page > - The user cannot see from the task which setting is broken or what to change > - This pull request records a bounded, named cause for a failed setup command and shows it on the task page > - The benefit is that the task names the broken setting and what to change, and classification, retry and recovery stay the same ## Linked Issues or Issue Description **What happened?** The restart-chaos config fault wave CF1 sets a project `workspaceStrategy.provisionCommand` to `bash ./scripts/provision-worktree.sh`, a script that does not exist. The run fails as `setup_failed` with `Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed: bash: ./scripts/provision-worktree.sh: No such file or directory`. The task page shows "Failed" and "Paperclip automatically retried continuation … still has no live execution path". The grader fails the task as `CF1/provision_command:message_only_in_run_detail`. **Expected behavior** The task page names the failed setup step, the configured command, its exit status and a short tail of its output, and says what to change. Retry behavior stays the same. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks 2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
52b906b112 |
fix(native): let Retry continue after a run that failed before native selection (#20482)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before Paperclip starts a new message or a Retry after a failed native run, it proves that the run's local process stopped > - A native run can fail before native selection, for example when a budget reservation is refused or a preparation query fails. That run never had a runner, a native session or a process ID > - The stop gate still waited for a stop record of that process. Retry returned 202, then the task stayed on `retryWait.reason=process_identity_missing` and no run started > - Native selection writes the run's native issue, native session, runner instance and coordinator in one transaction. A runner launch needs all of them, and the executor records `native.process_start_requested` before any spawn > - This pull request treats a settled run with none of that evidence as having nothing to stop > - The benefit is that Retry and new messages continue after a failure that happened before the agent started ## Linked Issues or Issue Description **What happened?** In local restart-chaos database-load wave `w-20261110T050818-s20386`, all 4 tasks failed before any agent process started. One example is t01, run `f7e2ee55`. A preparation query failed, and the run ended `adapter_failed` with no native session, no runner instance, no process ID and no launch event. Retry returned 202. For 16 or more minutes, `retryWait` then stayed on `process_identity_missing` and no run started. **Expected behavior** If the previous run never had a process, there is nothing to verify, so Retry and new input continue. **Steps to reproduce** 1. Make a native run fail before native selection (for example, a refused budget reservation or a failed query). 2. Click Retry, or send a message. Before this change, the input waits on `process_identity_missing` and does not continue. ## What Changed - `native-local-process-stop.ts`: `nativeRunRefusedBeforeLaunch` (renamed `nativeRunNeverLaunched`) now also proves "never launched" for a settled native run in this state: - no process ID and no process group; - no native issue, no native session and no runner instance (native selection writes them together). A coordinator row requires the native issue by foreign key, so the native-issue check also rules out a coordinator; - no launch request, no identity record, no `adapter.invoke` and no runner-authored event (any event with a source event ID). The existing case, a run that the cleanup quarantine refused before launch, does not change. - `explicit-native-continuation.test.ts`: the guard "failed for another reason without a process record" now separates real pre-selection failures from the shapes that must stay held. - Admitted: a run that failed before it requested a launch. This case is red on `master`. - Still held on `process_identity_missing`: a run that has a runner event, a run with a prepared native session, and a run with a coordinator, for `failed`, `timed_out` and `interrupted` runs. A `timed_out` run that failed before launch is admitted. The chat suite's `unreconciled` case (a native issue and phase, no session) also stays held. ## Verification - Red/green: on `master`, `routes input after a run failed before it requested a launch` fails with `process_identity_missing`. It passes with this change. - `vitest run src/services/explicit-native-continuation.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/issue-confirm-run-stopped-routes.test.ts`: 3 files, 462 tests pass. This includes every existing "keeps the hold for missing_process" and historical-run case. - Server `tsc --noEmit` passes. ## Risks - Low risk. The new proof needs structural evidence that no runner could exist: no native selection and no launch or runner event. A run that reached native selection, or that has any launch evidence, keeps every existing stop proof. - The proof does not depend on the build: a runner from any build needs the native session and the runner instance. Saved runs from older builds that were selected keep those fields, so they stay held exactly as before. - Relates to #20477, which fixes the restart window inside an answer run's attachment. This pull request is independent of it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19829 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.8 |
||
|
DottaandClaude
|
3e572475c8 |
fix(native): re-attach an answer run whose run.attach a restart cut off (#20477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person answers an agent's saved question, Paperclip starts an answer run. For a per-turn session, that run starts a new runner and sends `run.attach` to bind the saved conversation to the run > - The runner journals each command before its effect. A restart that kills it inside that window leaves `run.attach` with an unknown outcome. The restored runner settles it as `execution_indeterminate` and never executes it again > - The controller's same-run resume waited on that old command, so every resume failed with "PRP command run.attach failed". The answer never reached the agent. A Retry or a stall followed > - A kill during the attachment's provider launch can also leave a startup fence that no later runner retires, so every later command fails with "provider startup ownership remains unadmitted" > - This pull request attaches again with a new command when nothing after the cut-off attachment started provider work. It also lets the runner retire that orphaned startup fence > - The benefit is that an answer given just before a restart continues on its own ## Linked Issues or Issue Description **What happened?** Local restart-chaos journey wave `w-20261110T054406-s108` on master `2c2186706e`. Command: `--adapters codex --workloads Q1,Q2 --faults K1 --triggers answered --tasks 4 --concurrency 2 --seed 108`. Result: 1 of 4 passed. - t02 (Q1, K1 2.7 s after the answer). Answer run `0b357a1f` resumed after the restart. Each resume failed with `native_session_recovery_failed: PRP command run.attach failed: execution_indeterminate`. Then "Automatic recovery stopped", and the task needed a Retry. - t01 (Q2, K1 1.1 s after the answer). Answer run `82c60af9` failed the same way on three same-run resumes. Its provider state kept `lifecycle: "prepared"` with a `spawned` startup attempt for `run.attach`, and that provider process was gone. Each later drain and suspend failed with "provider startup ownership remains unadmitted". The successor run then refused with `retirement_refused=journal_commands_unsettled(run.attach.failed)`. The accepted Retry created no run (`retry_no_new_run`). The earlier wave `w-20261110T033426-s108` hit the same window at different points. It showed `authority_indeterminate` (Q1) and an ambiguous `turn.start` stall (Q2). See Risks for what this pull request does not cover. **Expected behavior** An answer run that a restart interrupts before it starts provider work resumes by itself. It does not need a Retry. **Steps to reproduce** 1. Run the wave command above on master. 2. Or run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport-cut-attach.test.ts` on master. Both cases fail with the error from the wave. ## What Changed - `runnerd-codex-transport.ts`: the same-run resume waits on the recovered `run.attach` as before. If that wait fails, the new `cutOffRunAttachmentPayload` checks the journal. It queues one new `run.attach` with the same payload, and waits on it, only when all of these are true: - the latest `run.attach` has an unknown outcome (`indeterminate`, or `failed` with `execution_indeterminate`); - it is a cold attachment (no `paperclipNextAuthority`), with no warm transition and no staged attachment; - every later command is recovery maintenance, as the control plane's shared `isRunnerRecoveryMaintenanceCommand` defines it (`session.snapshot`, `session.goal.get`, `runner.drain`, `runner.suspend`, `runner.shutdown`, `turn.stop`); - no committed event shows provider work (`turn.accepted`, `turn.started`, `item.started`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created` or `mcp_app.tool_input`). Any other failure is thrown unchanged. - `provider_backend.rs` (Runner): `retire_orphaned_startup_attempt` also treats a `prepared` state with a saved thread and a nonzero process generation as restorable. `attach_run` already documents this state as the settled checkpoint that `run.attach` restores. The other fence rules do not change. The fence is retired only before the spawn, or when the recorded process and its group have exited. - Tests: - New real-runnerd test `runnerd-codex-transport-cut-attach.test.ts`. A real runner and the fake Codex server run an asking run, then a cold answer-run attachment. The test kills both processes and rewinds both journals to "`run.attach` journaled, outcome not recorded", then resumes. One case has a startup fence and one does not. - A unit test for `cutOffRunAttachmentPayload`. It covers each maintenance command, and the cases where it must not re-attach (a definite answer, a pending command, a warm or staged attachment, any work command, a turn event). - The Rust orphaned-startup test also covers `prepared`. A new test shows that a `prepared` state with no saved thread keeps its fence. - `docs/durable-recovery.md`: describes both behaviors. ## Verification - Red/green, real runnerd: - On master, `runnerd-codex-transport-cut-attach.test.ts` fails both cases with `PRP command run.attach failed: ... execution_indeterminate`, the error from the wave. - With only the transport change, the case without a fence passes. The fence case fails with `provider startup ownership remains unadmitted`, the t01 error. - With both changes, both cases pass. - Red/green, Rust: `cargo test -p paperclip-runner-core --lib orphaned` fails `restart_retires_a_spawned_startup_whose_provider_process_exited` for `prepared` without the runner change. With it, all 10 tests pass. - Related transport suites, 8 files and 387 tests, pass: `runnerd-codex-transport`, `-attachment-recovery`, `-cold-restore`, `-warm-attachment`, `-release-resume`, `-cut-attach`, `runnerd-restart-continuation` and `runnerd-saved-continuation`. `tsc --noEmit` for the Runner package passes. - Live proof: the same wave command on this branch. Results follow in a comment. **No duplicate provider work.** The new attachment is sent only when the journal proves that no provider work followed the cut-off one. No command after it can start or steer work: only the shared recovery-maintenance commands are allowed, and none of them starts a turn. No turn, item, tool-input or runtime-request event is committed. `run.attach` itself starts no turn. In the Codex and ACPX runners, `attach_run` refuses to attach unless the session is settled. That means no active or ambiguous turn, no pending tool call and no live active goal. The old command is never executed again, because the runner refuses an `indeterminate` command. The new attachment is a new command with a new ID. ## Risks - Low-medium risk. The new path runs only after a same-run resume has already failed on an unknown-outcome attachment. Before this change, that run could not continue. - If an attachment's partial effect had changed provider state, the runner's own settled-session check refuses the new attachment. The run then fails as it does today. - **Runner release:** the Runner change does not change the pack `contractDigest`. It is `sha256:3164357d…` before and after (`scripts/contracts/contract-fingerprints.mjs`). Local runners use the runnerd built with the app, so the fix takes effect with this release. A remote image pack (for example Daytona) that was built before this change keeps the old fence rule until a new image or pack is built and qualified. No release or pin bump is part of this pull request. - **Mixed versions during an upgrade:** - New server with an old runner: the server re-attaches. A run without a startup fence continues. With a fence, the old runner refuses the new attachment, the same failure as today. - Old server with a new runner: the runner retires the fence, but the old server still fails on the unknown-outcome attachment, as today. - Neither mix adds a new failure. Saved state from older builds needs no migration. - Not covered here, and still open: - A `turn.start` with an unknown outcome where the resumed thread has no active turn (wave `w-20261110T033426-s108` t03). The runner deliberately fails closed for that case. - A warm-runner attachment that may already have been delivered (`authority_indeterminate`). - A Retry that waits on `process_identity_missing` for a run that failed before any process started. A separate pull request will cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20146, #19784, #20148 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
ca472d543a |
fix(runner): settle a saved governed wait whose shutdown drain left sequence gaps (#20494)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run can ask the board a question and park on that saved (governed) wait > - After the question is saved, the runner interrupts the provider and drains its usage and terminal for a bounded time > - The drain keeps only usage, the terminal and cancellation facts of the parked turn. It discards every other numbered provider event > - If the control plane restarts before the drain settles, recovery replays the saved journal. That replay required strictly contiguous `sourceSeq`, so the discarded events made a valid saved wait look like the wrong conversation > - This pull request accepts a gap after the saved trigger only when every event after the gap is a drain-retained fact of the same turn. Identity checks do not change > - The benefit is that the asking run settles its saved wait after a restart instead of recording a failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261110T063253-s208`, tasks t02 (Codex Q1, PV2: provider terminal dropped) and t03 (Codex Q1, PV1: provider terminal held), each with K1 (`kill -9` of the control plane about 3-4 s after the fault). The question, the board answer and the final brief were all correct. But the asking run was recorded `failed: native_session_interrupted`, and the grader failed `T:asking_run_settled`. **Cause** The run log of the asking run shows `failureDetail.message: native_governed_wait_replay_mismatch`. It is thrown at `packages/paperclip-runner/src/native-session-runtime.ts:2344` (master). The persisted runner events of the asking run have `sourceSeq` 33..40 and then 43 (t02), and 41, 42 and then 45 (t03). The two missing numbers are events that the accounting drain (`consumeTurn` → `settleDurableResult`, around line 1206 on master) read and discarded, because they are not usage, a terminal or a cancellation fact. Runs without a restart have the same gap (for example `w-20261110T032350-s201/t01`: 34, then 37). Those runs settle without replay, so the gap was not seen before. With a held or dropped terminal and a kill before the shutdown settlement receipt, recovery must replay. Its strict contiguity check then fails on a gap that the runner made itself. **Expected behavior** The question was saved before the provider was interrupted. Recovery must settle that same saved wait, as #20336 does after verified shutdown. Recovery must not start a new model turn. **Related** Refs #20336 (settle saved handoffs after verified shutdown). Not related in cause or code: #20477, #20482, #20483. ## What Changed - `isGovernedDrainRetainedEvent` (new, in `native-session-runtime.ts`) names the facts that the governed-wait drain persists: usage, the turn terminal, the interrupt acknowledgement, cancelled or expired runtime requests, and a failed or cancelled `run.terminal`, all for the parked turn. The drain now uses it in place of the same inline condition. Drain behavior does not change. - Governed-wait recovery replay accepts a `sourceSeq` gap only when: - the trigger is the first replayed event (a gap before it still fails), - sequence numbers strictly increase, - every event after the first gap is a drain-retained fact of the trigger's turn. - The run, normalized session and source instance identity checks do not change. Any mismatch still fails with `native_governed_wait_replay_mismatch`. - `doc/execution-semantics.md`: the governed-wait text said that shutdown keeps a contiguous journal. It now describes the gaps and the replay rule. Note for the sibling Claude/ACPX `turn.cancel` work: this change touches `settleDurableResult` only in the drain filter line (same condition, shared helper). It does not change cancel, the drain timeout or `settleCancelledTurn`. ## Verification - Red/green: `packages/paperclip-runner/src/native-session-runtime.test.ts` - `settles a saved governed wait whose drain left sequence gaps before a restart (terminal-after-restart)` - `settles a saved governed wait whose drain left sequence gaps before a restart (late-terminal-saved)` The first run uses the real drain. It discards two numbered events, so the history is trigger, acknowledgement, then usage with a gap. The stream then ends without a terminal, as with PV1/PV2 plus K1. On master, recovery rejects with `native_governed_wait_replay_mismatch` (the live error). With this change, recovery settles the saved wait as yielded. It does not call `resolveGovernedWait` again. It starts no second turn. The terminal is appended once. When the terminal arrived late and is already saved, recovery does not cancel the provider again. A further recovery of the settled checkpoint does not submit or settle again. - Fail closed: `keeps rejecting an unproven governed-wait replay with a sequence gap: …` covers a gap before the trigger, a foreign run, a foreign session, a foreign source instance, usage of another turn, a non-accounting event after a gap, and an accounting event followed by a non-accounting event. - `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts`: 261 passed. `npx tsc --noEmit -p .`: clean. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t governed`: 12 passed. - `pnpm verify` returned exit 2 (test file outside resolved graphs). Required CI supplies the broad coverage. - Live restart-chaos wave: see the comment below with the wave id and result. ## Risks - Low. The rule loosens only the sequence check, and only after the saved trigger, and only for facts that the drain itself retains and that carry no work authority. Identity checks stay strict. A non-accounting event after a gap still fails closed. - No change to the normal start path or turn startup (fast-turn rule). - No schema, protocol or env change. The pack `contractDigest` and provider `commandDigest` do not change. No Daytona image rebuild is necessary. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1e13c58a4e |
fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat run settlement records why a run failed, and the board shows that record. > - Runs that fail without a named code settle as `adapter_failed` (or `setup_failed`) with no cause. > - A user, an operator, and the restart chaos harness cannot tell what went wrong from such a run. > - Some run errors also show raw SQL statement text to users. > - This pull request records a bounded `failureCause` code for generic failures, shows it on the board, and keeps SQL text out of run errors. > - The benefit is that a generic failure now names its cause, for example "adapter failed (40001)", without leaking messages, SQL, or secrets. ## Linked Issues or Issue Description Refs #20385 (the nested-cause extraction this change reuses). Related: #20421 (restart chaos universal checks U1-U4). **What happened?** The restart chaos audit of 79 saved waves found 30 failed runs with `errorCode: "adapter_failed"` and a null or generic error. Every U3 ("diagnosable") warning was `adapter_failed`. Nobody could tell what failed. Some board run errors also showed raw SQL statement text from wrapped database errors (Drizzle `Failed query: ... params: ...`). **Expected behavior** A run that fails with a generic code names a short, safe cause. The board shows the cause next to the stop reason. Run error text never shows SQL statements or their parameters. **Steps to reproduce** Make an adapter throw a wrapped database error (an `Error` whose nested `cause` has `code: "40001"` and whose message has a Drizzle `Failed query:` statement). Before this change, the run settles as `adapter_failed` with no cause, and `heartbeat_runs.error` contains the SQL statement and parameters. The new tests in `server/src/services/heartbeat/run-completion.test.ts` reproduce this. ## What Changed - Runner: move the #20385 event-commit cause extraction into `safeErrorCauseCode` (`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and export it. The Runner's own diagnostics use the same function, unchanged in behavior. - Shared: add the `failureCause` contract (`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]` token of at most 80 characters, `normalizeRunFailureCause`, `readRunFailureCause`, and `withoutSqlStatementText`. - Server: add `server/src/services/heartbeat/run-failure-cause.ts`. Settlement records `resultJson.failureCause` only when the code is generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same. - Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`, else `provider_<category>`, else the first safe code in the error and up to three nested causes (`40001`, `ECONNRESET`), else the error class name, else `unknown_error`. - Adapter results without a code (`completeRun`): the error family, `signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`, `adapter_reported_error`, or `unknown_error`. - The failure log lines (`heartbeat execution failed`, `heartbeat execution setup failed`, and a new `heartbeat run failed` warning for generic adapter results) include `failureCause` and `errorCode`. - Server: replace SQL statement text in run error text with "Database query failed (<cause>)" at all three settlement paths, including the cancelled branch. The execution blocker's `runError`, the attention inbox (failed-run excerpt and metadata), and continuation summaries apply the same replacement on read, so older rows also stop showing SQL there. - Server: Retry admission and failed-preparation replay accept only a closed set of `resultJson` keys. `failureCause` is added to each of those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`, `cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is a diagnostic code with no execution or replay authority, so a failed-preparation run stays retryable. - Server: `runsForIssue` returns a validated top-level `failureCause`. It is null for older rows, non-string JSON values, and malformed values. - UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed (40001)". Older runs keep "adapter failed". The Storybook run-ledger fixture has a failed run with a cause. - Restart chaos harness: run samples keep `failureCause`. U3 treats a generic code with a specific `failureCause` as diagnosable. A bare `Error`, `unknown_error`, or the no-code fallback `adapter_reported_error` still counts as vague. U3 stays a warning. - Docs: `doc/run-log-events.md` and the harness README describe the field and the SQL replacement. ## Verification - Red before the fix: the four new `run-completion.test.ts` cases failed against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL text in `error`). - Green: - `server/src/services/heartbeat/run-completion.test.ts`: all tests passed (real embedded PostgreSQL). New cases cover a nested `40001` three levels deep for execution and setup failures, SQL and a secret parameter absent from `error` and `resultJson`, the bound (an 81-character code and a SQL "code" fall back to the class name), `exit_code_137` for a code-less adapter result, a cancelled result with SQL text, no cause for a named code, and the log line. On this loaded host the suite's `afterAll` database teardown hit its 30 s hook limit after every test passed (the known local teardown limit, also seen in #20385). CI runs the suite. - `server/src/services/heartbeat/run-failure-cause.test.ts` (4), `server/src/__tests__/activity-service.test.ts` (recorded, legacy, malformed, overlong, boolean, and number cause projections), `server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL row). - `packages/shared/src/run-failure-cause.test.ts` (6), `ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos `universal.test.ts` and `observer.test.ts`. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`: 207 passed (the stall-diagnostic cases still read `40001` and `ECONNRESET`). - Typecheck: shared, runner, server (all layers), UI, restart chaos harness. `pnpm check:token-gates`: all gates clean. - `pnpm verify` returned exit 2 (global doc input changed); required CI supplies broad coverage. - CI on `6295f8d` found that the new key made failed-preparation runs ineligible for Retry (`chat-channels.integration.test.ts` saved board-message Retry got 409; `workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries). Both reproduced locally and pass after the allowlist fix, with the other 459 Retry cases in `chat-channels.integration.test.ts` and all 25 in `workspace-recovery-preparation-dispatch.test.ts`. - No chaos waves were run. Storybook was not opened in a browser; the fixture typechecks and the label is covered by the UI test. ## Risks - Low risk. `errorCode` values do not change, so retry, recovery, and comparison logic is unaffected. - No schema change and no migration. `failureCause` is a new optional `resultJson` key. Rows saved by the previous release have no key and render as before; a new row read by an old server is ignored. - The SQL replacement only acts on a Drizzle `Failed query:` marker, or on `$n` placeholder text with a statement shape. Ordinary error text does not change. When it acts, it also drops the text after the statement. - Code that checks `resultJson` against a closed key list must allow `failureCause`. I found and updated the four lists on the Retry path; a list I missed would refuse a Retry for a run with a generic failure. - This is the run log / instance database path. It adds no Telemetry event. - The server failure log line still logs the raw error message (unchanged behavior); only the board-facing text is changed. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file editing) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
5c9641f141 |
test(restart-chaos): add repeated restarts (RR1-RR3): a restart inside the previous restart's recovery (#20480)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Every existing fault restarts the server once per task. > - Real incidents on 2026-11-08 and 2026-11-09 were deploy loops. The server restarted again 5–60 s after it came back, while recovery of interrupted runs was still in progress. > - No wave puts a second restart inside the first restart's recovery window. > - This pull request adds a fault family, repeated restarts (RR1–RR3). It does not change product code. > - The benefit is that waves now test recovery that is itself interrupted. ## Linked Issues or Issue Description **What is the problem?** The chaos harness restarts the server only once per fault. Production deploy loops restart it again while it recovers, and no wave tests that. **What is the solution?** Add three repeated-restart faults. Each one starts as a K1 and then stops the replacement again after a seeded gap from its health. **Why is it needed?** Recovery of interrupted runs can itself be interrupted. A second restart in that window can leave a lease, a claim or a wake half done. Refs #20458 (the TM fault family this one follows). ## What Changed - Add `RR1`, `RR2` and `RR3` to `--faults` (`repeated-restarts.ts`): - **RR1 double crash**: K1, then one more `SIGKILL` 3–20 s after the replacement reports healthy. - **RR2 crash, then graceful**: K1, then one `SIGTERM` (as K2) 3–20 s after health. - **RR3 crash loop**: 3–4 `SIGKILL`s in all, 10–30 s after each health. The last replacement stays up. - Add `--rr-gap-seconds <n>` or `<min>-<max>` (1–300). It replaces every RR gap. RR faults do not combine with `--storm-minutes`. - The gaps are seeded per task (`restartSteps` in `plan.json`, also in `--matrix` plans). They are drawn only for RR tasks, so the plans of existing seeds do not change. `--replay` restarts again after the recorded gaps. - The supervisor runs the whole sequence as one fault receipt. The grader therefore sees one fault window, from the first kill to the last replacement's ready. Crossing, R1's outage allowance, R5, the T-oracle, task deadlines and the recovery speed budget treat it as one fault. No check is relaxed. - Each stop and replacement is recorded with its signal, gap, PID, the `/api/ready` status just before the stop, and its stopped, restarted and healthy times. The record goes to `<wave>/repeated-restarts/<faultId>.json` (also written when the sequence fails part way), to the receipt (`repeatedRestart` in `faults.json` and in each failure's `startup-phases.json`), and to the timeline (`repeated_restart_stop`, `repeated_restart_healthy`). - Small registry additions: `types.ts`, `config.ts`, `scheduler.ts`, `journey-matrix.ts`, `main.ts`, `evidence.ts`, `supervisor.ts`, and the README section next to TM. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 33 files, 402 tests pass. The new `repeated-restarts.test.ts` covers option parsing, seeded planning for each kind, unchanged plans for other faults, steps in the task schedule, the matrix and replay, the stop-and-replace loop on a fake clock (gaps measured from health, `SIGTERM` for RR2, an exit during a gap, shutdown), and one crossing window for the sequence. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - `pnpm verify` asks for broader coverage (the README is a global CI input), so required CI supplies it. - Live waves run from master after merge, as the merge rule for test-only harness PRs allows. Their results go to a follow-up comment. ## Risks - Harness-only change; no product code changes. Waves without RR faults are unchanged: the plan draws, the fault receipt and the grader paths are the same. - During an RR sequence the server stays "not ready" for the harness, so no other fault fires and the observer does not click Retry until the last replacement is ready. This is the same rule as for K6. - A replacement that exits on its own during a gap fails the fault as a harness error, with the stops so far recorded. The wave then recovers the server as after any failed fault. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
53c9be8d41 |
Reuse identical captured workspace archives (#20476)
Reuse the current committed snapshot when a fresh capture matches its manifest and plaintext archive hash and size. Preserve the original encryption binding and verify each new run receipt for recovery, concurrent replay and storage request statistics. Verified with 338 focused save and recovery tests, server typecheck and build, all final-head CI checks, and Greptile 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
179694de7d |
test(chaos): provider message faults for Claude (ACPX) and across a K1 restart (#20454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) proves that agent work survives control-plane restarts and provider faults > - PR #20432 added provider message faults (PV1 delay, PV2 drop, PV3 duplicate), but only for Codex, and only in waves without restarts > - Claude runs through the ACPX sidecar, whose provider pack and sidecar are digest-verified, so the Codex executable shim cannot reach it > - A late or missing terminal can also arrive across a control-plane restart, which no wave could exercise > - This pull request adds a Claude (ACPX) seam with no product change, and a K1 restart that fires after a provider fault applies > - The benefit is that governed waits, Stop and approvals are now tested under provider message faults for both main adapters, also across a kill -9 ## Linked Issues or Issue Description Refs #20432, #20330, #20336 **What is the problem?** Provider message faults covered only Codex. They could not run with restart faults. PV3, the `interrupt_ack` target, and the S1 and G1 workloads under PV1/PV2 never ran live. **What is the expected outcome?** The harness can delay, drop or duplicate one targeted provider message for Claude and Codex. A PV1/PV2 fault can straddle a K1 restart, and the grade states what a pass means. **Why is this the right layer?** This is test-harness work. The product is not changed. The seam replaces only a file the runner already hashes and verifies at launch, inside a per-wave stand-in package. ## What Changed - `acpx-provider-faults.ts`: builds a per-wave stand-in runner package under `<repo>/node_modules/.restart-chaos-<wave>/` (symlinks to the real package, except a generated `dist/cli/acpx-runtime-sidecar.cjs`). It also parses `--provider-fault-restart`. - `provider-shim/acpx-sidecar-hook.mjs`: a server-only resolve hook that loads `runnerd-codex-transport` (source `src/live/*.ts` or built `dist/live/*.js`) from the stand-in package. Product code then hashes the stand-in sidecar, and the runner verifies and launches it. The module's own imports resolve to the real package. - `provider-shim/acpx-fault-tap.cjs`: the stand-in installs this tap and then runs the real sidecar bundle in the same process, as the runner's verified CommonJS loader does. For Claude, `terminal` is the provider's ACP `session/prompt` response after a `session/cancel`. `interrupt_ack` is the sidecar's `turn.cancel` response, because ACP's cancel has no acknowledgement. The provider process is still spawned and admitted by product code. - `--provider-fault-restart K1 [--provider-fault-restart-after-seconds n]`: each PV1/PV2 task also kills -9 the control plane n seconds (default 3) after its fault applied, through a new internal `provider` trigger. PV3 and planned K cells stay rejected. - `withProviderFaultCheck`: with a restart, the check passes only if the fault applied once, the K1 started after it, and for PV1 no release came first. The ordinary R1–R6 grade and the workload T-oracle still decide whether the task recovered. - `config.ts`: provider faults accept `--adapters codex,claude`. G1 with a provider fault needs `--provider-fault-restart K1`, because its oracle grades the board decision after a restart. - README: the Claude seam, the target table per adapter, what a pass means across a restart, and the Claude PV1 delay window. - Unit tests: the ACP and runner-pipe taps, the generated stand-in with a fixture sidecar, the hook against a fake runner package, planning, config validation, and the restart check. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (all pass after rebase on master) - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - Live waves (local, one at a time through the wave-slot protocol): | Wave | Scenario | Result | | --- | --- | --- | | w-20261110T040202-s202 | Claude Q1, PV1 terminal 18 s | Task passed its Q1 oracle; `provider_fault_applied` failed: the runner settled the cancelled turn ~3.6 s after the cancel and closed the sidecar before the release, so 18 s is a drop for Claude (documented) | | w-20261110T041224-s203 | Claude Q1, PV1 terminal 1 s and 3 s | 2/2 pass (held and released) | | w-20261110T041355-s204 | Claude Q1 and S1, PV2 terminal | 2/2 pass | | w-20261110T043354-s205 | Codex Q1, PV3 terminal (duplicate) | 1/1 pass | | w-20261110T050407-s206 | Codex S1 and G1, PV1/PV2 terminal | 0/4: S1 terminal never sent (Codex exits after Stop, so `terminal` cannot apply; documented, use `interrupt_ack`); G1 needs a restart by design (now rejected without `--provider-fault-restart K1`). No product failure. | | queued | Codex+Claude Q1/S1 PV1/PV2 `interrupt_ack`; Codex+Claude Q1/G1 PV1/PV2 + K1 | Runs from master after merge (queue is ~90 min deep) | ## Risks - Low product risk: no product code changes. - The seam depends on the runner computing the local sidecar path from the transport module URL and on the ACPX provider bootstrap marker string. If either changes, the tap logs nothing and every Claude PV task fails `provider_fault_applied`. It does not pass silently. - The stand-in package is written under the workspace `node_modules` (gitignored) and removed at wave end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.7 nightly/v2026.1110.0-nightly.0 |
||
|
DottaandClaude
|
233f615644 |
fix(heartbeat): resume legacy runs a graceful shutdown stopped before dispatch (#20461)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat service runs agent turns and recovers them after a
server restart
> - A graceful restart can stop a legacy run (for example a `process`
adapter agent) while the run is still in setup, before the adapter
starts
> - That run was parked with `legacy_execution_requires_reconciliation`,
so automatic recovery stopped
> - A `process` or `http` adapter cannot continue that hold through
Retry, so the accepted Retry waited forever and the task stayed blocked
> - This pull request proves "no adapter work started" for that case and
resumes the run automatically
> - The benefit is that a restart during setup no longer strands the
task, and Retry starts a run for holds that an older server saved
## Linked Issues or Issue Description
**What happened?**
The restart-chaos harness ran workload Q3 (a board approves an agent's
app action late) with a `process` adapter agent. A graceful restart
(fault K2) fired right after the approval. The agent's follow-up run was
interrupted with `server_shutdown_interrupted` 0.2 s after it started,
while it was still in setup. Automatic recovery stopped with
`legacy_execution_requires_reconciliation`. The board clicked Retry. The
server accepted the Retry (HTTP 202), but no run started. The task
showed "Retry is waiting for execution checks" (`retryWait.reason =
execution_recovery`) forever.
**Expected behavior**
A run that a graceful shutdown stops before any adapter work must resume
automatically. An accepted Retry must start a run.
**Steps to reproduce**
1. Run a `process` adapter agent on a task.
2. Send SIGTERM to the server while the run is still in setup (before
the adapter starts).
3. Restart the server. The task is blocked with
`legacy_execution_requires_reconciliation`.
4. Click Retry. The request is accepted, but no run starts.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
693de27c90 |
fix(storage): wait out expired or rotated S3 credentials and re-resolve the key (#20456)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode, run logs, provider traces, attachments and remote workspace snapshots go to S3 > - The server uses the default AWS credential chain, and real keys expire or rotate while runs are in flight > - #20299 made runs wait out a storage outage, but a 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch` was a permanent `adapter_failed` > - The SDK also caches a key until its stated expiration (forever for a key with none), so a rotated key was never picked up > - This pull request treats credential rejections as a storage wait and re-resolves the key from the credential chain after a rejection > - The benefit is that runs survive key expiry and rotation without failed attempts or a manual Retry ## Linked Issues or Issue Description Refs #20299 (storage outage wait). Harness that found this: #20455. **What existing behavior does this improve?** Runs wait out S3 outages (5xx, timeouts, resets) but fail at once when the store rejects the server's credentials. **Subsystem affected** server storage (S3 provider) and heartbeat run completion. **Current behavior** Restart-chaos wave w-20261110T041556-s501 (`--storage s3 --storage-faults ST5`): one 41 s window of 403 `ExpiredToken` failed 4 of 6 tasks. Each run ended `adapter_failed` with "The provided token has expired.", and the automatic retries failed within one second, so the tasks needed a board Retry (R3 `retry_did_not_complete`). **Proposed behavior** A credential rejection is a transient infrastructure condition. Critical-path storage calls wait it out like an outage, and a run that stops on it gets the `storage_unavailable` wait retry. After each rejection the S3 client re-resolves its key with `forceRefresh`, so the next attempt signs with a rotated key. The run error names the credentials. If they never come back, the exhausted-wait message tells the operator to check the storage service and its credentials. **Reason and benefit** Expiring or rotated keys are normal in cloud deployments. They must not fail work. ## What Changed - `storage-unavailable.ts`: `ExpiredToken`, `ExpiredTokenException`, `TokenRefreshRequired`, `InvalidToken`, `InvalidAccessKeyId`, `InvalidClientTokenId` and `SignatureDoesNotMatch` answered by the store with 400/403 count as storage unavailable. `AccessDenied` and a credential chain that cannot load any key (`CredentialsProviderError`) stay permanent. New credential-specific run message; the exhausted message also names credentials. - `s3-credential-refresh.ts` (new): S3 client middleware that calls the client's credential provider with `forceRefresh` after a rejection, once per credential generation (requests that signed with an already-replaced key do not refresh again), then rethrows so the existing storage wait owns the retry. - `s3-provider.ts`: installs the middleware on both S3 clients. - `run-completion.ts`: uses the credential-aware message. ## Verification - `cd server && npx vitest run src/__tests__/storage-s3-credential-rotation.test.ts src/storage/ src/__tests__/storage-outage-wait.test.ts src/__tests__/storage-s3-transient-retry.test.ts src/__tests__/storage-s3-failure-diagnostics.test.ts src/services/heartbeat/run-completion.test.ts` (186 pass). - The new `storage-s3-credential-rotation.test.ts` fails 5 of 6 cases on master (red) and passes with this change (green). It covers a real key rotation through the default credential chain, 403 windows for each code, the message, and `AccessDenied` staying permanent. - `cd server && npx tsc --noEmit` passes. - `s3-credential-refresh.test.ts`: two parallel requests signed with the old key, rejected 50 ms and 400 ms apart, cause one forced lookup (3 before the generation check, 2 after). - Restart-chaos rerun on a build with this fix and #20455: w-20261110T050044-s503 (`--env local --workloads A1,C1 --faults K1,K2 --tasks 8 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90`) passed 8/8 tasks (R1–R6 and T) with a 39 s `ExpiredToken` window and an 85 s real key rotation (76 credential 403s). Before the fix, w-20261110T041556-s501 failed 4/6 tasks. ## Risks - Scope: calls with a Readable body (streamed `putObject`, buffered multipart) cannot be resent, so they still fail on the first rejection, as they do for outages. The middleware has already re-resolved the key, so the caller's next attempt (the run-level wait retry for run work) uses the new key. - A key that the store rejects for good (for example a wrong secret) now waits up to the storage wait budget (5 min in the call, then up to 10 run retries) before the board shows the exhausted message, instead of failing at once. - `forceRefresh` re-runs the credential chain after a rejection. Calls are coalesced, so a burst of rejections causes one refresh. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
bf1524c5f8 |
test(restart-chaos): pin each wave's API URL to its own loopback server (#20462)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness starts an isolated Paperclip server for
each wave and runs real agents against it
> - Agents find the server through `PAPERCLIP_API_URL`, which the server
derives at startup
> - On a developer machine with a public URL for the main instance (for
example a tailnet origin), the wave server derived that public URL
instead of its own loopback origin
> - So `process` agents sent the run key for the temporary instance to a
different host, which returned 502
> - This pull request pins each wave server's `PAPERCLIP_API_URL` to its
own loopback origin and removes the Q3 script workaround
> - The benefit is that a wave's agent credentials only go to the wave's
own server
## Linked Issues or Issue Description
**What happened?**
In isolated restart-chaos wave instances, `process` agents received
`PAPERCLIP_API_URL` set to the machine's public tailnet origin. That
origin serves the developer's main instance, not the wave. The agent's
run key for the temporary instance was sent to that host, which returned
502. The Q3 agent script worked around this with a hard-coded loopback
base URL.
**Expected behavior**
A wave's agents call the wave's own server.
**Steps to reproduce**
1. Configure a public URL (for example a tailnet origin) for the
developer's main Paperclip instance on the machine.
2. Run a restart-chaos wave with a `process` agent that reads
`PAPERCLIP_API_URL`.
3. The agent's API calls go to the public origin, not to the wave's
`127.0.0.1` port.
**Paperclip version or commit**
master at
canary/v2026.1110.0-canary.6
|
||
|
DottaandClaude
|
400bb1f0f9 |
test(restart-chaos): one-row db pressure, bound accepted-Retry waits in U1, live L1 results (#20472)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness grades real agent tasks on an isolated control plane. > - #20421 added the L1 contention workload, `--db-pressure`, and the universal checks U1–U4. > - The first live L1 waves showed two harness gaps and one missed product shape. > - The pressure writers locked several run rows per statement and deadlocked with product transactions. A real stream does not do that. > - U1 counted an accepted Retry that waits on the system as a live owner forever. Ten tasks sat 8 minutes in that state with no run. > - This pull request fixes both gaps, adds the owed direct test for `collectContentionEvidence`, and records the live wave results. ## Linked Issues or Issue Description Refs #20421, #20385. **What happened?** The live L1 waves (`w-20261110T040531-s20385` on the #20385 parent, `w-20261110T044338-s20385` and `w-20261110T050818-s20386` on master) produced harness deadlocks (40P01) from multi-row pressure statements. U1 passed tasks whose accepted Retry waited on `process_identity_missing` for 16 minutes without a run. **Expected behavior** Pressure writers behave like a run-log stream (one row per statement). U1 flags an accepted Retry that waits on the system past the queue bound. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <results dirs>` over the saved waves, or the L1 wave command in the README. ## What Changed - `contention.ts`: each pressure statement updates one running run row. The old statement updated every running row at once and deadlocked with product transactions. - `universal.ts` (U1): an accepted Retry that waits on the system (`retryWait`) is an owner only for the queue bound (5 min of server-ready time). While it waits, the Retry button, failed notice and blocker do not count as actionable items, because the person already acted. A scheduled retry is still a timer. - `contention-db.test.ts`: a real PostgreSQL test for `collectContentionEvidence` (receipt count, duplicate producer events, terminal receipt, accounting fields, company scope). - README: live wave results table, updated audit numbers and the U1 rule. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 28 files, 345 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Audit over 79 saved waves (1,620 tasks, 1,483 passing): U1 17 flags, none on a passing task or an all-pass wave. The longest accepted-Retry wait on a passing task was 278 s. U2 1, U3 30 (3 on passing tasks, warning only), U4 0. - Live waves (codex and claude, `--workloads L1 --faults K1 --concurrency 4 --tasks 4 --db-pressure 4`): - `w-20261110T040531-s20385`, #20385 parent: 117 receipt-write serialization failures (40001) in the server log; 1 of 4 tasks passed. Three runs failed `adapter_failed` after 40001 exhausted in `createRunUsageRecorder`, `reserveRunBudget` and native finalization. - `w-20261110T044338-s20385`, master `8004ad069e` with the harness: 2 serialization failures; 1 of 4 passed. Failures included 40001 in `createRunUsageRecorder` and deadlocks caused by the old multi-row writers (fixed here). - `w-20261110T050818-s20386`, same build with one-row writers: 0 of 4 passed. Every first run failed in `reserveRunBudget` before spawn, and every accepted Retry then waited on `process_identity_missing` without starting a run. - No wave reached the runner's commit stall, so none showed `native_event_replay_conflict`. ## Risks - U1 can now fail a task whose accepted Retry waits more than 8 minutes. No passing task in 79 saved waves came close (longest 278 s). - One-row writers produce fewer statements per second than the multi-row statement did; `contention-db.test.ts` still reproduces the #20385 receipt race with them. - Test-only change; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2444658c6f |
test(restart-chaos): add ST5 credential rejection and ST5R key rotation storage faults (#20455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` already puts a fault proxy between the server and a loopback MinIO (ST1 outage, ST2 latency, ST3 errors) > - Cloud deployments use the default AWS credential chain, and keys can expire or rotate during a run > - The harness had no fault for a rejected or rotated key, so this failure class was untested > - This pull request adds ST5 (a window of 403 credential rejections from the proxy) and ST5R (a real key rotation at MinIO) > - The benefit is that waves can prove runs wait out credential problems and pick up rotated keys ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults cover outages, latency and 5xx errors only. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** No wave can reject the server's S3 credentials or rotate the store's key. **Proposed behavior** `--storage-faults ST5` answers every request in a window with 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch`. `--storage-faults ST5R` restarts MinIO with a new root key (the old key stops working) and gives the server the new key through its credential source only when the window ends. **Reason and benefit** The first local ST5 wave (w-20261110T041556-s501) shows the gap: a 41 s `ExpiredToken` window failed 4 of 6 tasks with `adapter_failed`, and the automatic retries failed within one second. The product fix is a separate PR. ## What Changed - `tests/restart-chaos/storage-credentials.ts` (new): ST5 and ST5R rule planning, a wave-owned `AWS_CONFIG_FILE` profile whose `credential_process` prints the current key, MinIO restart with a new root key on the same loopback port, and a proxy wrapper that holds traffic during the restart and publishes the new key when the window ends. - `storage-proxy.ts`: the `expired_credentials` fault takes an S3 error code; a `credential_rotation` marker passes traffic through. - `storage-chaos.ts`, `types.ts`, `config.ts`, `main.ts`: registry lines for ST5/ST5R and the rotation wiring. - README: ST5/ST5R rows and the credential-source description. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (28 files, 328 tests pass). The new `storage-credentials.test.ts` fails before this change (the kinds do not exist) and passes after. It includes a real MinIO test: after a rotation the old key gets 403 `InvalidAccessKeyId`, a client with a cached key fails, and the same client succeeds after it re-resolves its profile. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave: `--env local --workloads A1,C1 --faults K1,K2 --tasks 6 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90` ran and injected 62 credential 403s (storage-proxy.json). ## Risks - Test harness only. No product code changes. - ST5R restarts MinIO. Requests are held for up to 5 s during the restart, so a slow restart can also show as a short 502 blip. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
8809f44b5e | Connect Codex and Claude login to agent computers (#20417) | ||
|
DottaandClaude Opus 5.5
|
95cb3c94c4 |
test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). **What existing behavior does this improve?** The ST2 slow-store fault in the restart-chaos storage proxy. **Subsystem affected** Test harness (`tests/restart-chaos`). **Current behavior** ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. **Proposed behavior** ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. **Reason and benefit** Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
30d99f4e32 |
test(restart-chaos): add time faults (TM1-TM3): freeze, sleep with aged rows, aged outage (#20458)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Each wave runs for minutes on a fresh instance. Nothing time-bound ever expires during a wave. > - Production failures on 2026-11-09 were about time: an OAuth refresh after more than 1 h made pending approvals invalid (#20340), and a timeout used the wrong window (#20330). > - This pull request adds a fourth fault family, time faults (TM1–TM3). The faults make leases, deadlines and waits expire during a run or across a restart. They do not change product code. > - The benefit is that waves now find failures that occur only when time passes. The first waves found one: a run whose turn completes after a host sleep fails, and the task needs Retry. ## Linked Issues or Issue Description Refs #20340 Refs #20330 **What is the problem?** The chaos harness never moves the clock, so it cannot find failures where a lease, a deadline or a pending wait expires. **What is the solution?** Add three time faults. Each one makes stored deadlines lapse in a way that a real host sees: a process pause, a host sleep, or a long outage. **Why is it needed?** Two days of chaos testing did not find the failures of #20340 and #20330, because no wave lasted long enough for a deadline to pass. ## What Changed - Add `TM1`, `TM2` and `TM3` to `--faults` (`time-faults.ts`, `time-faults-config.ts`): - **TM1 freeze**: send `SIGSTOP` to the server process only, for 95–150 s, then send `SIGCONT`. The runner and Postgres continue. The pause is longer than the controller-instance lease (90 s) and the execution-control deadline (60 s). - **TM2 sleep**: send `SIGSTOP` to the server and all of its descendants. While they are stopped, move every database time back by the age. Then send `SIGCONT`. This is a stand-in for a laptop sleep. - **TM3 aged outage**: send `SIGKILL` to the server tree (as K1). While it is down, move every database time back by the age. Then start a replacement. With `--workloads G1 --triggers approval`, an approval stays pending for a day across a restart. - Add `--time-age-hours` (1–72, default 26). TM faults run only with `--env local`, and not with storms, crash points or K14. - Aging is one transaction. It moves every `timestamp` column, and every time inside a `jsonb` column (ISO strings in their own format, and epoch milliseconds in the wave's window), back by the same interval. Triggers are off for the shift. Copies of a time in JSON (for example the comment `updatedAt` in a continuation receipt) still match their columns. The first wave showed that a shift of the columns only causes false failures. - The identity, state and time columns of the time-bound tables are written before the shift to `<wave>/time-faults/<faultId>.json`. Per-table counts go to the fault receipt (`timeFault`) and the timeline (`time_fault_aged`). - The observer and grader read through `TimeShiftedApi`. It adds the age back to each time that a shift moved, so R4, R5, U1–U4 and the speed budgets use real time. - Make small additions to the registries: `types.ts` (fault kinds, receipt field, option), `config.ts`, `scheduler.ts` (a seeded freeze length only for TM tasks, so the plans of existing seeds do not change), `supervisor.ts` (the freeze and aging hooks), `main.ts` (one line for the API) and the README. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 30 files, 368 tests pass. The new tests are: - `time-faults.test.ts`: option parsing, freeze order, Postgres exclusion, statement generation, and reading times back at their real time. - `time-faults-db.test.ts`: aging runs on real embedded Postgres. The test checks that trigger rewrites are suppressed, that JSON copies match their columns, that text columns do not change, that the evidence file is written, and that an age the wave nearly outlived is refused. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local, codex), each run from a separate worktree and one wave per slot: | Wave | Scenario | Result | | --- | --- | --- | | `w-20261110T031929-s301` (A1,C1 × TM1–TM3) | First run, before JSON aging | t01: TM1 missed, because A1 finished first. t02: pass. t03 (C1): its turn completed after TM2 and failed with `native_session_lease_lost`; a follow-up comment woke a new run. After TM3, that run failed with `continuation_user_authorization_missing` and needed Retry. That second failure was a harness artifact: the receipt kept the comment's `updatedAt` in JSON, which the first version did not shift. JSON aging fixes it. | | `w-20261110T041910-s303` (B5 × TM1–TM3) | Freeze, sleep, aged outage | t01 and t02 crossed TM1, then TM2. Both turns completed after the thaw and failed with `native_session_lease_lost`, so both needed Retry (R3). This is a **product failure**; see below. t03 (TM3, 26 h aged outage): **pass**, with no Retry. | | `w-20261110T050125-s302` (G1 × TM3, `--triggers approval`) | Aged wait: an approval pending for 26 h across a restart | **Pass** (2/2). Both approvals were still pending after the restart and the board decided after the fault: t01 rejected, so the task is cancelled; t02 approved, so it is done with exactly one subtask. No Retry; U1–U4 pass. | | `w-20261110T051535-s304` (B5, TM2 with `--time-age-hours 1`, then TM1; one task at a time) | Attribute the failure | t01 (TM2 sleep, only 1 h): its turn completed after the thaw, then failed with `native_session_lease_lost`, so the task needed Retry. This reproduces the product failure. t02 (TM1 alone, 113 s server-only freeze): **pass**. The failure follows the database clock jump, not the pause. | **Product failure found (not fixed in this PR).** A local native run whose host sleeps longer than the 20-minute execution lease loses its completed work. After the wake, the provider finishes the turn and the brief is saved. Then result acceptance (`native-session-executor.ts`, the transaction at about line 12031) requires `native_run_finalizations.lease_expires_at > clock_timestamp()`. The database clock has moved on, so the run fails with `native_session_lease_lost`, and the task shows "Couldn't finish this attempt … Retry". No other boot claimed the lease: the owner, attempt and controller identity are unchanged, and the server's monotonic deadline did not expire. The lease is fenced by wall-clock expiry alone. A safe fix changes lease fencing semantics, which is a heartbeat-boundary decision, so I report it here rather than include a fix in this harness PR. Reproduce with `--workloads B5 --faults TM2` (`--time-age-hours 1` is enough; see `w-20261110T051535-s304`). ## Risks - Harness-only change; no product code changes. Waves that do not use TM faults are unchanged. `TimeShiftedApi` returns responses unchanged when no shift was applied. - `SIGSTOP` goes only to the wave server's own process tree. The tree is found from the server PID, and Postgres is excluded. Every frozen process gets `SIGCONT` in a `finally` block and in a process `exit` hook. - Aging is a model: the database becomes what it would be if everything happened the age earlier. Times outside Postgres (in-memory timers, runner state files), times inside `text` columns, and HMACs over a time are not moved. The README lists these limits. Agent JWT expiry is not covered: native runs mint a token for each tool call. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
549ca46dae |
fix(runner): retain Claude GitHub broker settings through the Rust sidecar (#20463)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Shared agents use the instructing person's GitHub identity for managed commands. > - Native Claude starts through the Rust daemon and a TypeScript ACPX sidecar. > - The Rust launch filter kept the command launchers but removed their broker transport. > - Shell commands then reported `GitHub capability_missing`, although connector tools worked. > - This pull request retains that transport for managed Claude launches. > - Each command can acquire its current identity through the existing broker checks. ## Linked Issues or Issue Description **What happened?** Native Claude could use GitHub connector tools but could not authenticate `git` or `gh`. The Rust daemon removed the broker URL and session/bridge capabilities before starting the ACPX sidecar. Reconnecting the account did not repair this process environment. **Expected behavior** Managed shell commands use the current authorized GitHub identity. The provider receives broker capabilities, while raw GitHub tokens remain confined to individual commands. **Steps to reproduce** 1. Start native Claude through the Rust Runner with managed GitHub launchers. 2. Run `gh api user` and a private repository ref lookup through Bash. 3. Observe `capability_missing` and no request to the broker. **Paperclip version or commit** Reproduced from source at `2c2186706ef4bb352b662f9d8faebefe47fb52ac`. The repair is based on `a5e01634f69f18dfe10a0bf709df02d9e6a2135b`. **Deployment mode** Native Claude through ACPX. Local acceptance used macOS arm64 and Node 25.6.1. The reported failure also affected a Linux sandbox. Related: #18212 preserved launcher paths. #18850 repaired the TypeScript filter. This change repairs the preceding Rust filter. The duplicate search found no open repair for this boundary. ROADMAP.md already lists managed personal GitHub identities. ## What Changed - Retain the broker URL and both broker capabilities when Claude has a nonempty managed launcher directory. - Keep raw GitHub tokens, unrelated secrets, empty launcher configurations, and sibling ACPX providers outside that exception. - Exercise the regression through the public sidecar initialize protocol in the existing Rust integration suite. Document the restart requirement. The Rust sidecar transport owns this filter. Credential resolution, company access, approval, and run identity policies keep their existing owners. No schema or API changes are needed. ## Verification - Red/green: the subprocess regression failed on the missing `PAPERCLIP_GITHUB_BROKER_URL` before the fix, then passed with all three broker settings and the negative controls. - Passed before moving the regression: 615 Rust core tests, with 3 existing ignored tests. After moving it into the integration suite, all 20 sidecar integration tests and the 9 remaining sidecar unit controls pass. The moved regression was also run against the unfixed source and failed on the missing broker URL. - Passed: 85 TypeScript environment, launcher, sandbox, and configured-environment controls. These include operation identity changes, revocation, recovery, and exclusion of capabilities from persisted environment data. - Passed: Runner Rust typecheck and format check, plus the Runner build through `node scripts/agent-preflight.mjs --skip-rust-debug`. - Live before/after: real `claude-opus-5-5` through the actual Rust daemon and built ACPX sidecar reproduced the failure with the old binary. Neither shell command reached the broker. With the fixed binary, `gh` identified the connected account and `git` read private repository refs. A second turn retained a workspace file and authenticated again. After the controlled broker denied access, the next command failed without reusing the prior token. No GitHub writes were made. - Live release cutover: saved a session with the unfixed daemon, suspended it, and restored it with the repaired daemon on `aa9f5b30614c011ebc3fe4509e11173b766c1fea`. The same session and workspace marker survived. `gh` authenticated after restoration, then failed when the controlled broker revoked access. - Live acceptance used an Anthropic API credential and a loopback broker with an authorized operator GitHub credential. This isolates the repaired transport; it does not qualify the hosted subscription credential or production grant resolver. - The first PR review gate did not recognize the in-source Rust test. The regression now lives in the existing integration test file, and the same test-coverage gate passes locally. - `pnpm verify` returned its documented exit 2 and required broad CI coverage. The final head passed the full CI typecheck, build, and test lanes. - Greptile reviewed `64cca69b76073cc3185b28334465dee48fbadb29` at 5/5 with no actionable findings or open review threads. Its check passed on that head. The same head is mergeable with no conflicts. - Final head `64cca69b76073cc3185b28334465dee48fbadb29` passed [full PR CI](https://github.com/paperclipai/paperclip/actions/runs/42761401396), [Windows Rust](https://github.com/paperclipai/paperclip/actions/runs/42761401117), and [isolated Runner compilation](https://github.com/paperclipai/paperclip/actions/runs/42761400973). Typecheck, build, all test shards, Runner checks, browser checks, and aggregate verification passed. No checks are pending or failed; five conditional jobs were skipped. ## Risks - The Claude sidecar now receives the intended broker capabilities. The existing broker must continue to enforce active-run identity and current connection access. Raw GitHub tokens remain excluded from the provider launch environment. - Previous-release checkpoint and profile formats carry forward unchanged. A running provider must retire and restart normally to inherit the repaired environment. No profile reset or data migration is required. - Hosted recovery requires delivery of the updated Runner binary. This PR does not deploy or retry existing tasks. Live acceptance has not exercised Linux sandbox confinement or a GitHub write. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning. Used repository inspection, code editing, and shell execution. The context window size is not exposed. Anthropic `claude-opus-5-5` was exercised as the acceptance-test provider. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a5e01634f6 |
feat(ui): ask self-hosted users to star Paperclip on GitHub after 5 done tasks (#20453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Many teams run Paperclip self-hosted, so the project grows when those users tell others about it > - A GitHub star is the cheapest way for a user to do that, but the app never asks > - Asking too early is noise. A user who has finished a few tasks knows whether Paperclip helps them > - This pull request adds a one-time, dismissable modal that asks self-hosted users for a star after their company has 5 done tasks > - The benefit is more visibility for the project, and no repeated prompt for the user ## Linked Issues or Issue Description **Subsystem affected** Board UI (dialogs shown in the main app shell). **Problem or motivation** Self-hosted users have no prompt to star the repository. Users who like Paperclip often do not think of it. A prompt at the right moment helps other teams find the project. **Proposed solution** Show a small modal once the selected company has 5 tasks with status "done". The modal shows the Paperclip mark, the headline "You completed 5 tasks!", a short body, a "Star on GitHub" button and a "Not now" button. It shows only once per browser. Closing it in any way counts as seen. **Alternatives considered** A permanent banner or sidebar link. Both stay on screen after the user decides, so they cost attention every day. A one-time modal costs attention once. **Roadmap alignment** This is a small UI addition. It does not overlap with planned core work in ROADMAP.md. No related public issues or PRs found (searched for "star", "github star"). ## What Changed - Added `GitHubStarPrompt` (`ui/src/components/GitHubStarPrompt.tsx`). It reads the health endpoint and shows only on self-hosted instances (never on Cloud), never during onboarding, and only after 5 done tasks in the selected company. It waits for any other open dialog to close. It stores a "seen" flag in `localStorage` when it opens, so a reload does not show it again. - Mounted the prompt in `BoardDialogs`. - Exported the modal-detection helpers (`MODAL_SELECTOR`, `useModalOpen`) from `AnnouncementWell` so both components use the same rule. - Added `PaperclipMark`, the mark alone from the existing lockup geometry. The lockup and the mark now share one path constant. - Added the `issues.starPromptDone` query key. - Added tests for when the prompt opens and when it stays hidden. ## Verification - `pnpm --filter @paperclipai/ui exec vitest run src/components/GitHubStarPrompt.test.tsx src/components/AnnouncementWell.test.tsx` - `pnpm check:token-gates` (all gates clean) - `pnpm typecheck` - Manual: on a self-hosted dev instance, mark 5 tasks in one company as done. The modal opens. "Star on GitHub" opens github.com/paperclipai/paperclip in a new tab. Reload: the modal does not open again. To see it again, delete the `paperclip.github-star-prompt.seen` key from `localStorage`. ## Risks - Low risk. The change is UI only and additive. - The prompt runs one small issues query (limit 5) while it is still eligible. After it is seen, the query is disabled. - If `localStorage` is not available, the prompt treats itself as already seen and does not show. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model id `claude-opus-5-5`, run through Claude Code with tool use (shell, file edits). Standard context window; extended thinking at low effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: squadbot <squadbot@squadbots-Mac-mini.local> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.5 |
||
|
DottaandClaude
|
2c2186706e |
test(restart-chaos): human-in-the-loop journeys (Q2, Q3), journey triggers and a journey x fault matrix (#20448)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane while real agents work, and grades each task. > - Almost every harness workload is "the agent does a file task and finishes". Few workloads have a person in the loop. > - On 2026-11-09, production failures came from journeys with a person in the loop: a saved answer that could not reach a dependency-blocked task (#20363), and an app-action approval that a routine OAuth refresh or a one-hour deadline made invalid (#20340). > - No workload exercised these journeys, so no wave could catch them. > - This pull request adds journey workloads with deterministic oracles and journey-specific fault triggers. It also adds a journey × fault matrix and a way to run a wave against an older build. > - The benefit: one command crosses every journey with every fault kind, and the same wave proves red before a fix and green after it. ## Linked Issues or Issue Description Refs #20363, Refs #20340, Refs #20336, Refs #20432 ## What Changed - New workload **Q2** (`journeys.ts`, `journey-driver.ts`): the agent saves a question with its native `request_human_input` tool. The board adds an unfinished `backlog` prerequisite, marks the task `blocked` by it, and answers through the task-thread respond route. The oracle requires that the answer reaches the agent in bounded answer mode while the task stays blocked, that the prerequisite relation stays, that no run has `issue_dependencies_blocked`, and that no stale Retry notice stays at rest. Then the board finishes the prerequisite and the agent finishes the task. - New workload **Q3**: a harness-owned OAuth 2.1 + MCP app on loopback (`journey-oauth-app.ts`, discovery, dynamic client registration, PKCE, rotating refresh tokens, one write tool). The board connects it through the Apps routes and sets the tool to Ask first for a scripted `process` agent. When the review is pending, the harness moves the review's saved timestamps and the token expiry back 61 minutes in the wave's own database, runs Check connection (which refreshes the token through the app), and approves. The oracle requires one review, a refresh between request and decision, an accepted approval, exactly one `tools/call` with the reviewed arguments, an `executed` request, and an agent report that carries a receipt only the provider result contains. - New triggers `asked` and `answered` fire a restart fault 0–3 s after the saved request is first seen and after the board's response. They work for the existing **Q1** saved question (#20432) too. The board answers before an `answered` fault fires and after every other planned fault. - New `--matrix` option: one task per workload × adapter × (a `K0` no-fault baseline, each restart fault at each of the workload's own triggers, each provider message fault). `K0` is a new task fault kind with no fault. - New `--server-from <checkout>` option: every server start and restart of the wave runs another prepared checkout, for example the parent of a fix, with the same harness code. - Q3 setup allows unpriced usage on the wave's company budget policy, because a scripted process run records usage with no price and the default policy pauses the whole company. - README documents the journeys, the triggers, the matrix and the red-before-fix recipe. Unit tests cover the oracles (pass and the pre-fix failure signatures), the driver's ordering, the matrix plan, option validation and the OAuth app. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 295 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Live local waves, codex, one wave at a time (shared machine lock), `--concurrency 2`, at most 3 tasks: | Wave | Server build | Command | Result | | --- | --- | --- | --- | | `w-20261110T024308-s106` | this branch (master `5570debc5b` base) | `--workloads Q3 --faults K1 --matrix --seed 106` | 3/3 pass (K0, K1^asked, K1^answered); the app saw one refresh before each decision and one `tools/call` | | `w-20261110T024447-s106` | `--server-from` parent of #20340 (`0e325e8d52`) | same | 3/3 fail as expected: approve returns 409 `action_expired` ("Tool review has expired"), no `tools/call`, task stays `in_review` | | `w-20261110T024644-s107` | this branch | `--workloads Q2 --faults K1 --matrix --seed 107` | K1^asked and K1^answered pass. The K0 cell failed only because the oracle then required a comment; the agent had reported `ANSWER-…` in its bounded turn's run result 14 s after the answer, while blocked. The oracle now accepts either. | | `w-20261110T030341-s107` | `--server-from` parent of #20363 (`eebdfe71b1`) | `--workloads Q2 --faults K1 --matrix --seed 107` | 3/3 fail as expected: `answer_reached_agent_while_blocked`. No run starts between the answer and the finished prerequisite (6 minutes of ready server). | | `w-20261110T033426-s108` | this branch, rebased onto #20432 | `--workloads Q1,Q2 --faults K1 --triggers asked,answered --tasks 4 --seed 108` | Q1^asked and Q2^asked pass. Q1^answered and Q2^answered fail on a product gap (below). | | `w-20261110T020037-s102` | this branch, before the rebase | `--workloads Q1,Q2,Q3 --faults K1 --triggers asked,answered --tasks 3 --seed 102` | Q1 with `asked` and `answered` K1 restarts: both done with the answer. | Waves `…-s107` (pre-fix) and `…-s106` (pre-fix) used the same harness code as the green waves. Only the server build changed. **New product failure (not fixed here).** A K1 restart 0.3–3 s after the board's answer can lose the answer turn's native run: - `w-20261110T033426-s108/t04` (Q1, K1 2.9 s after the answer): run `a40cc8f7` resumes with `resume_dead_runner`, then fails `runner_state_identity_mismatch: authority_indeterminate` (`adapter_failed`). The task shows "Couldn't finish this attempt. Your message is saved." and needs Retry. Retry works. - `w-20261110T033426-s108/t03` (Q2, blocked task, K1 0.27 s after the answer): run `5b5355de` is `process_lost`. Its bounded retry `92a57f50` fails `native_session_recovery_failed` (`session.snapshot` failed). The Retry click starts `0e6aa9d1`, which fails `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The task then waits with `retryWait.reason=process_identity_missing` and stays blocked without the answer. - Suspected path: native runner restart reconciliation of the runner state identity (`resume_dead_runner`, journal command settlement) in `server/src/services/native-runtime/` and `packages/paperclip-runner/`. Host load was 8–14 during the wave. ## Risks - Test harness only. No product code changes. - Q2 depends on the model following its instructions. The oracle accepts the answer in either a comment or the bounded turn's run result, because both were seen. - Q3 shifts persisted timestamps only in the wave's isolated database. It does not shift unrelated tables, so it simulates elapsed time for the review and the token, not for the whole instance. - `K0` and the new triggers add values to shared registries (`types.ts`, `scheduler.ts`). Existing plans keep their meaning. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>canary/v2026.1110.0-canary.4 |
||
|
DottaandClaude Opus 5.5
|
120017e39d |
fix(workspaces): read saved workspace files in place instead of extracting the whole archive (#20442)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a cloud task finishes, its workspace is saved as an encrypted
snapshot and its sandbox is released. The board's Files view then reads
that saved copy
> - Saved Files access downloaded, decrypted and extracted the whole
archive for every request, and refused archives over 512 MiB with
`too_large`
> - A finished task in a Paperclip-sized repository saves a snapshot of
about 730 MiB, so its files could not be listed or opened at all
> - The archive format already records each file's size and SHA-256, and
the snapshot records the manifest digest, so single files can be read
and verified without the rest of the archive
> - This pull request reads the saved archive in place, one verified
file at a time
> - The benefit is that users can open any file of a large finished
workspace quickly, with bounded memory and no whole-archive transfer
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves on Daytona with a real Paperclip clone, the
committed workspace snapshots were 763,944,535 bytes. After the sandbox
was released, every saved Files read returned 422 `too_large` (506 reads
in total), because `createSavedWorkspaceLoader` refuses archives over
`maxArchiveBytes` (512 MiB). The board Files view uses the same loader,
so it fails the same way.
**Expected behavior**
A user can list and open any single file of a saved workspace of
Paperclip-repository size without the server extracting or buffering the
whole archive. Memory and time stay bounded per request, and integrity
is still verified.
**Steps to reproduce**
1. Run a cloud task whose workspace is a large repository, so the final
snapshot exceeds 512 MiB.
2. Let the task finish and its sandbox be released.
3. Open the task's Files view, or call `GET
/api/issues/:id/file-resources/content?workspace=execution&path=…`: it
returns 422 `too_large`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
5ab98b0063 |
test(restart-chaos): add database contention workload and universal checks U1-U4 (#20421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks on an isolated control plane and grades them. > - Two production failures on 2026-11-09 passed the harness. Its tasks made little output and little accounting, so database write contention never happened (#20385). Its grading was per workload, so a task stuck for a reason no scenario targeted was not caught (#20361). > - The harness needs a contention workload and checks that apply to every task. > - This pull request adds the L1 workload, a `--db-pressure` knob, and the universal checks U1–U4. > - The benefit is that the #20385 failure class and unexplained stuck or misleading tasks now fail a wave. ## Linked Issues or Issue Description Refs #20385, #20383, #20361. **What happened?** The harness ran real embedded PostgreSQL, but no task streamed enough run-log output while usage receipts committed. The #20385 receipt race (SQLSTATE 40001, retained outbox, `native_event_replay_conflict`) could not occur. Also, no rule checked every task for an owner or an actionable item, for retry loops, for diagnosable failures, or for stale notices. **Expected behavior** A chaos wave can create run-row contention between run-log writes and accounting writes and check that accounting stays exact. Every task of every wave gets scenario-independent liveness and notice checks. **Steps to reproduce** Run an L1 wave with `--db-pressure 4` against the parent of the #20385 merge (red), then against master (green). Commands are in Verification. ## What Changed - `contention.ts` (new): workload L1 (five 3,000-line shell outputs as five tool calls, then a brief), its accounting oracle, and `RunRowPressure`, the `--db-pressure <writers>` knob. The writers run `update heartbeat_runs set updated_at = updated_at` on the wave company's running runs, in the wave's own isolated database only. - L1 oracle: no `native_event_replay_conflict` or `runner_event_commit_stalled`; each producer event journaled once; a terminal receipt for each succeeded native run; finished runs accounted; per-run cost equals its cost events; the board's spend equals the cost-event sum with no runs pending accounting; and proof that the task produced contention. - `universal.ts` (new): U1 liveness, U2 no silent retry loop, U3 diagnosable failures, U4 no stale notices. `grader.ts` grades them on every task. U1, U2 and U4 fail a task. U3 is a warning. - `invariants.ts`: `readUniversalDbEvidence` reads failed-run error text and recovery-action causes from the wave database at wave end, for U3. - `universal-audit.ts` (new): runs U1–U4 over saved wave evidence to measure false positives and bound headroom. - `contention-db.test.ts` (new): on real PostgreSQL, under the pressure writers, the pre-#20385 repeatable-read receipt write exhausts its retries with 40001 and the current writer commits. - Small registry edits: `types.ts` (L1 id, `dbPressure`, optional `contention` and `universal` fields), `workloads.ts` (L1 entry), `config.ts` (`--db-pressure`, L1 not a default), `main.ts` (start and stop the writers, read database evidence before teardown), `ledger.ts` (universal verdicts and warnings in the ledger and `SUMMARY.md`). - README: "Database contention" and "Universal checks" sections, with the audit table and bound reasons. ## Verification - `contention-db.test.ts` (real embedded PostgreSQL): with `--db-pressure` writers (4), the pre-#20385 repeatable-read receipt write exhausted its retries with 40001 in 8 of 8 attempts; the current READ COMMITTED `persistUsageReceipt` committed 8 of 8. This is the harness-level red/green for #20385. - Live L1 waves (`--env local --adapters codex,claude --workloads L1 --faults K1 --concurrency 4 --tasks 4 --seed 20385 --db-pressure 4`), red on the #20385 parent `bf818280f2` with this harness overlaid and green on this branch: queued behind the shared one-wave-at-a-time lock and not yet run. The first red attempt failed at startup (the fresh checkout lacked a built plugin SDK; no tasks ran). Live results are not claimed here. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: all harness unit tests pass, including the new `universal.test.ts`, `contention.test.ts` and `contention-db.test.ts`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Universal-check audit over 74 saved waves (1,592 tasks): U1 7 flags (one C1 stall and six R1 runs queued 8 min behind a retry row, the #20361 shape), U2 1, U3 22, U4 0. None of U1, U2 or U4 flags a passing task or an all-pass wave. U3 flags 3 passing tasks (all `adapter_failed`), so it ships as a warning. ## Risks - U1, U2 and U4 can now fail a task that passed R1–R6 and T. The audit found no such case in 70 saved waves. The bounds are environment-configurable. - `--db-pressure` is opt-in and writes no values. It writes only to the wave's own temporary database. - Other agents edit the same harness. This change keeps registry edits small; `grader.ts` and `invariants.ts` changes are additive. - No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0c7a628fd1 |
test(chaos): read remote R1 files from the execution workspace and record why a read failed (#20440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - In replays w-20261110T010031-s9913 and w-20261110T012332-s9950 the tasks recovered (R1, R3, R6 pass), but all 506 content reads returned 422, even after the closing save left the workspace clean (`snapshotId` set, no writer, `location: null`) > - The harness discarded each error, so the waves could not show which 422 the server returned > - This pull request asks for the execution workspace explicitly and records the last error per file in the task record and the timeline > - The benefit is that the next wave names the exact cause instead of a blind `module_present` failure ## Linked Issues or Issue Description Follow-up to #20348. **What happened?** Every file read for a finished Daytona R1 task returned 422, and the response was discarded. **Expected behavior** Failed reads leave their reason in the evidence. **Actual behavior** Only the outcome, a `null` file, was kept. ## What Changed - File reads add `workspace=execution`, so the server reads the issue's execution workspace and not the `auto` choice. - `readRemoteWorkspaceFile` and `captureRemoteWorkspaceFiles` report each failure through a callback. The observer keeps the last message per file in `record.remoteFileErrors` and logs `remote_file_read_failed` once per distinct message. - The remote-workspace file reader stays in `remote-workspace-files.ts`; `observer.ts` is at 993 lines. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 263 passed. The test server now requires `workspace=execution`, so the existing remote tests were red before, and a new test records the 422 `workspace_unavailable` message. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - If a Daytona task's files only exist in a project workspace rather than its execution workspace, an explicit `execution` selector reports that clearly instead of falling back. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1110.0-canary.3 |
||
|
DottaandClaude Opus 5.5
|
d064bd21e4 |
perf(plugins): pick each lifecycle resource's head in one pass (#20406)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Plugins receive resource lifecycle events (agent, project and agent-runtime changes) through a durable inbox > - The inbox lists an event only when it is the first unacknowledged event of its resource, so each resource delivers in order > - The list query checked that rule with a `NOT EXISTS` for every unacknowledged event. Each check walked the resource's whole acknowledged history > - A resource with a long acknowledged history and a backlog behind a deferred head costs history × backlog probes. On the hosted staging instance three copies of this statement ran for more than 23 minutes each, on a database that was at 6–8 of 8 vCPU all day > - This pull request picks the deliverable head of each resource in one pass > - The benefit is a list call that is linear in the company's events, and database CPU that other requests can use ## Linked Issues or Issue Description **What existing behavior does this improve?** `pluginLifecycleInbox(...).list()` in `server/src/services/plugin-lifecycle-inbox.ts`. The delivery worker calls it in a loop and when it checks for idle work. **Subsystem affected** Plugin lifecycle delivery (server service, PostgreSQL query). **Current behavior** Neon "Active queries" for the staging database on 2026-11-09 showed three copies of this statement at 1,381–1,401 s each. In the same view, trivial primary-key reads waited 100–280 ms. Locally, one resource with 2,000 acknowledged events and a 2,000-event backlog takes 3.5 s for one `list()` call. The cost grows with acknowledged history × backlog. **Proposed behavior** Only the head of a resource is deliverable: its unacknowledged creation if it has one, otherwise its lowest unacknowledged event. This is exactly the set that the old `firstForResource` predicate admits. The query now selects these heads with `DISTINCT ON (resource_type, resource_id)` ordered by `(action <> 'create'), id`, and then applies the page cursor, order and limit. **Reason and benefit** The same fixture lists in 8–24 ms instead of 3.5 s. Long-running copies of this statement no longer hold database CPU that every request needs. **Breaking changes** None. `acknowledge()` keeps its per-event eligibility check. ## What Changed - `pluginLifecycleInbox.list()` uses a `lifecycle_heads` CTE with `DISTINCT ON` to pick one head per resource, then filters by the `afterId` cursor and orders by id with the limit. - New test in `server/src/__tests__/resource-lifecycle-events.test.ts`: one resource with 2,000 acknowledged events and a 2,000-event backlog, plus a second resource. It asserts the exact heads in id order and a list time below 1 s (3.5 s before, 8–24 ms after). ## Verification - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts -t "acknowledged backlog"` — red on `origin/master` (3,546 ms), green with the change (24 ms, 8 ms). - `cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-assignments.test.ts src/__tests__/agent-runtime-hooks.test.ts src/__tests__/project-instance-secrets.test.ts` — the existing ordering, paging, per-plugin isolation, backfilled-creation and late-commit tests pass. (One `agent-runtime-hooks` case is timing-sensitive on a loaded machine and also fails on `origin/master` there. It passed 22/22 on rerun.) - `pnpm verify` selection and CI. - After deploy: Neon "Active queries" no longer shows long-running `resource_lifecycle_events … NOT EXISTS` statements. ## Risks - Low risk. The result set is the same: per resource, the unacknowledged creation, otherwise the lowest unacknowledged id. The cursor still applies after the head is chosen, as before, so paging past failed heads does not skip later events. - Each call still reads the company's events of the requested types once. A company with millions of acknowledged events still pays one linear pass per call. A per-plugin acknowledgment watermark would remove that and is out of scope. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser, code execution). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0934b834c4 |
perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path > - Paperclip manages agent work and preserves its workspace files. > - Online migration captures local files and publishes remote workspace authority. > - Each capture verifies encrypted readback, restores the archive and compares the current source. > - A cohort kept every completed parent payload until the whole cohort published. > - A larger cohort could retain many complete copies while later members still worked. > - This pull request releases only finished parent payloads before a capture worker starts another member. > - Nested-child proof, final fresh checks and atomic publication keep their existing owners. ## Linked Issues or Issue Description Refs: #20418, #20437, #20441. **What happened?** A cohort retained each member's capture archive, verified archive, comparison archive and restored tree until atomic publication. The capture worker limit did not limit that retained parent data. **Expected behavior** Release completed parent payloads once their consumers finish. Keep files that a later publication check still needs. **Steps to reproduce** Run the added held-peer regression against the previous owner. It completes one parent, holds a later member and finds all four completed parent paths still present. ## What Changed - Release three named parent archives and the restored tree after a successful online cohort capture, before that worker takes another member. - Keep parent scratch roots and all nested-child verification areas. Keep every final fresh child and publication check. - Retain payloads for archived repository intents. Use the existing writable-tree cleanup for read-only restored directories. - Add real filesystem regressions for early release, retained child proof, late child mutation and the final archived-intent path check. Update the source-fence guide. This branch incorporates the final #20441 owner and both distinct saved-relative-path regressions. Its scratch-specific production change is nine added lines and one removed line. Single imports and saved journal/archive formats stay unchanged. ## Verification - Canonical Node24 preflight passed. The Runner build came from the content cache. After the current-master merge, the preflight checked 1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin SDK prerequisites passed. - Genuine RED on the unchanged #20441 owner: one held-peer case failed at four retained payload paths; the archived-intent control passed. The run took21.14s, with221 unrelated cases intentionally unselected. The first sandbox-restricted setup attempt hit DNS errors and was stopped; the normal network-qualified preflight then passed. - Initial treatment: four selected PostgreSQL/filesystem cases passed in38.61s, with219 intentionally unselected. The held peer observes released parent payloads and retained child areas. A later child mutation still refuses the whole cohort and publishes no authority or activity. - The archived-intent control observes all parent payloads at the actual final saved-relative-path check. The changed path refuses publication and keeps task/repository state intact. This exercises its actual singleton owner; archived-only sources do not enter the online cohort API. - After a normal merge of current master and the final #20441 test, the five affected cases and server TypeScript passed. No full local matrix was repeated. Initial CI failed the inherited recovered-upload assertion because the new retention test leaked its archive and filesystem spies. That case passed alone but failed in the adjacent three-case sequence (two pass, one fail); scoped `finally` cleanup made the same sequence pass three of three in17.15s, with191 unrelated cases intentionally unselected. No upstream assertion or production code changed. Server TypeScript passed again. All required [CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511) passed on final normal-merge head `604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and one neutral; no pending or failing checks). The previously failing server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with no open threads. The merge resolves the duplicate #20441 squash insertion; both tested owner and test files are byte-for-byte identical to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior head received [Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885); its archived-cohort coverage warning was [answered using the actual public admission boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017). - `git diff --check` passed. No hosted cleanup, migration, provider action or deployment was performed. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication' pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reduces retained parent data. It does not establish a total disk bound. Active capture/encryption/readback files and retained child areas still need measured capacity. - Cleanup failure uses the existing worker abort and peer-join path before publication. Original source files, uploaded objects and journals remain intact. - Archived repository intents still need their restored payload at publication. Those payloads stay retained. No capture limit, budget or integrity guard changes. - Saved operations and archive identities carry forward unchanged. Resume still uses the existing full source and authenticated object checks. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f00f2b83e |
fix(workspaces): recognize committed imports in linked source scope (#20444)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration fences every local Git source that a capture can reach. > - A linked worktree can remain on disk after its workspace is migrated to cloud storage. > - The source inspector lists local workspace rows, so it then treats that linked worktree as unregistered. > - This pull request uses the existing canonical import journal and snapshot proof to recognize its exact original physical root. > - The benefit is that a later linked workspace can migrate without weakening source ownership or stopping agents. ## Linked Issues or Issue Description **What happened?** After one linked Git workspace commits its canonical migration, its local files and Git registration can remain. The workspace row now uses `cloud_sandbox`. A later migration of a related local workspace fails with `online_workspace_source_scope_unproven` because the original linked root is no longer in the local-row list. **Expected behavior** The inspector can recognize the exact original physical root from the same-company successful migration journal and its committed or superseded snapshot. Unknown or changed provenance must still fail. **Steps to reproduce** 1. Create two local Git workspaces with a shared worktree registration. 2. Complete canonical migration for one workspace while retaining its original local files. 3. Inspect or migrate the other workspace. Before this fix, the inspector returns no source scope. **Paperclip version or commit** Reproduced on `f73ff4ace9` with real PostgreSQL, Git, filesystem, and canonical snapshot storage. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:online_workspace_source_scope_unproven` Refs #20418, #20436, #20441. This fix changes source registration only. ## What Changed - Let the existing imported-source owner match optional exact linked physical roots in addition to its existing nested-child paths. - Require each added linked root to have a finished successful same-company journal, an exact committed or superseded snapshot binding, and the same manifest digest. - Include those proven physical roots in the online scope and its source fingerprint checks. - Retain Git root, backlink, pointer, ownership, company, foreign writer, source lock, and publication reinspection guards. - Keep existing nested-child behavior and all persisted schemas unchanged. Previous-release canonical imports remain readable. No runtime authority comes from an arbitrary provider path or remote authority. ## Verification - Genuine baseline RED: both committed and superseded imported sibling cases return a null source scope after a real successful import. - GREEN: 17 focused real-PostgreSQL/FS/Git tests pass. They cover both imported states, the original issue source fence, changed journal/snapshot/manifest/unfinished provenance, a foreign local alias, unregistered links, backlink/pointer corruption, and a listing change during upload. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'imported linked Git source|foreign local writer|unproven linked|revalidates the linked Git listing|nested Git wrapper.*shared|nested Git wrapper blocked'`. - Server layered typecheck, module boundaries, and `git diff --check` pass. - Actual controlled staging canary: the canonical API committed both targeted workspace snapshots in 310.249 seconds. The owned job closed with exit 0 and all owned processes joined. A separate read-only inventory confirmed that both workspaces left the legacy provider set. This used a reviewed private administrative artifact; the serving app image and normal release path stayed unchanged. - Final head `1759ac6f40a1b35f79587b246de8b36e7a371537`: 56 successful checks, 4 intentional skips, and 1 neutral status. Greptile and Ponytail both report 5/5 with no findings or unresolved threads. - Initial CI had unchanged Runner-support and workspace-recovery timeouts, plus an unrelated heartbeat plugin-environment timing/fixture failure. The same owning inputs passed on the sibling PR. Each failed job then passed one targeted retry on unchanged product bytes. The original failures remain recorded in the CI history. ## Risks - This recognizes a retained original root only from the existing committed import ledger. It does not treat arbitrary cloud workspace paths as local ownership. - Each physical root still passes the existing Git identity and foreign-scope checks. Publication re-inspects the source scope under the existing source locks. - This adds no object reads, provider calls, recurring work, capture algorithm changes, or data exclusion. The existing import query keeps its current 10,240-row limit and runs during explicit migration inspection only. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1110.0-canary.2 |
||
|
DottaandClaude
|
1b52f359dd |
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
0991d9bda1 |
fix(workspaces): protect migration sources and add guarded recovery service (#20443)
## Thinking Path > - Paperclip manages agents and their task workspaces. > - Online migration saves files before it publishes remote workspace authority. > - A saved migration journal can survive the importing process. > - Git cache expiration did not count that journal as a consumer. > - Expiration could remove a source and change its fingerprint while its import was incomplete. > - This PR protects unfinished imports and exposes a guarded internal service transition for an already affected cohort. > - New imports can preserve the current sources while all old recovery data stays retained. ## Linked Issues or Issue Description **What happened?** Git cache expiration could archive and remove a workspace after an import process ended but before its saved cohort finished. The saved source fingerprint then differed from the current archived workspace. Exact cohort resume correctly refused that changed source. Reopening the workspace cannot recreate the old metadata or fingerprint. **Expected behavior** An unfinished migration remains a durable consumer after process loss. If an older server already expired some sources, one canonical transition can fail the exact unpublished cohort without losing its snapshots, objects, source history, or recovery obligations. New attempts use the current verified sources. **Steps to reproduce** 1. Start a cohort import of eight Git worktrees with local changes. 2. Interrupt it after four snapshot uploads. 3. Let an older server expire two idle source worktrees. 4. Resume the saved cohort. It refuses the changed archived sources. Related: #20429 preserves exact cohort identity when independent companion imports change only the scope proof. #20424 preserves prepared recovery state for archived sources. This PR handles completed source expiration, not a scope-only change. Public searches found related expiration work and no matching compensation owner. ## What Changed - Count saved migration markers and unfinished migration journals as Git expiration consumers. Keep failed migration retention holds as consumers too. - Expose `supersedeExpiredRemoteWorkspaceMigrationCohort` as an internal service API for a privileged operator with a real database handle and reviewed company/cohort inputs. It is called directly by the PostgreSQL/Git test and reviewed one-time administrative qualification. It adds no CLI mode or HTTP route. The existing CLI still refuses a damaged cohort; this PR does not claim automatic CLI recovery. - Require an exact unpublished cohort of up to eight members. It requires completed expiration and an unexpired restore recipe for each missing member. The other members must retain their saved source fingerprints. - Recheck current source and recovery state under source, lifecycle, finalizer, and row locks. One transaction fails all old journals, writes audit records, and removes only their exact matched active markers. - Keep old cohort proofs, operations, snapshots, and object references. Genuine failure receipts use the existing failed-save retention hold. This transition does not report migration success. - Reuse the existing bounded source-lock queue when `sourceLockWaitMs` is set. Keep fast refusal by default. Cancellation targets only this waiter, and current checks still run after admission. - Record predecessor IDs on new attempts and reserve a snapshot generation above the existing ledger maximum. New success does not retire the old hold. - Document the owning transition and retention rules. ## Verification - The two new journal-consumer cases failed under the old expiration predicate and passed under the new predicate. - The full new recovery case passed in 21.12 seconds with real PostgreSQL locks, Git repositories, encrypted local object storage, and archive restores. It covers four retained uploads, two real expirations, unchanged old resume refusal, authority change refusal, transaction rollback, a held native finalizer, idempotence, eight new imports, dirty/untracked file restores, and a retention pass 30 days later with no object deletion. - Server TypeScript and module-boundary checks passed. - On `7cec8c915acfad2d08874eaf26abaa852f560219`, the queued-admission regression failed before forwarding the option (24.07 seconds), then passed (22.25 seconds). It verifies a real held project lock, cancellation that preserves the holder and all journals, release into full compensation, retention, and eight restored fresh imports. Current server TypeScript and module/diff checks passed. The only production follow-up changes are the optional queue input and its existing-owner call. - The earlier `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c` CI run had one Runner support failure: the unchanged credential-free Python/Pi calibration exceeded its outer 25-second subprocess deadline. That job passed 6,926 other tests and skipped four. The timeout is retained as a nonpass. The changed logical head requires fresh CI and review; no test deadline was raised. - The affected cohort, Git expiration, and eligibility suites passed all 148 cases in 228.41 seconds on `aa589f3f447e36d0277c32f8b44cbe5e84a673be`. Merged current master `a9d0946e64`; the only conflict retained both independent added test blocks. The complete merged cohort suite passed all 30 cases in 70.00 seconds on `737a0df3a71b10fe48d7bbfe7ffd8ae296b4852c`, including upstream larger-cohort controls. Final-head server TypeScript, incremental preflight, module-boundary, and diff checks passed. Unchanged expiration/eligibility evidence remains valid. Fresh `809ab704e619812d8f1bf38cf5de9c65893861cc` CI and review are pending. - On `809ab704e619812d8f1bf38cf5de9c65893861cc`, the complete recovery fixture passed in 21.38 seconds after adding one valid archived recipe with no restore point. It refuses with `migration_supersession_expiration_unverified` and keeps all workspaces, journals, and snapshots unchanged. The same case then verifies queue cancellation/release, full compensation, retained history, fresh imports, and restored bytes. Server TypeScript passed. This follow-up changes only the test and service-scope documentation; product bytes remain identical to qualified `7cec8c915acfad2d08874eaf26abaa852f560219`. - Earlier fixture attempts did not pass. They exposed the selection of a held source and an incorrect expected refusal string. The fixture now selects eligible sources and asserts the canonical authority refusal. Production guards were not relaxed. - Live qualification of the exact owning product bytes completed in 230.419 seconds. All eight old operations became failed with auditable supersession lineage. All eight old snapshot rows remained byte-for-byte identical, and their object references remain retained. The existing source-lock queue waited 1.151 seconds for a normal finalizer handoff. The three complete source/recovery passes took about 218 seconds including their transaction, which explains why the earlier 180-second total budget was insufficient for that measured workload. No source proof or publication guard was skipped. Compensation made no provider or object-storage calls. - Earlier live qualifications remain nonpasses: the 180-second attempt did not finish; its instrumented successor completed source checks and refused a busy source lock before mutation. The successful bounded-queue attempt retained those failure records. Fresh archived/online imports are a separate subsequent step and are not claimed complete here. - The local checks do not claim browser acceptance, app deployment, fleet enrollment, or a completed zero-downtime cutover. ## Risks - This owner mutates migration bookkeeping. It preserves all-or-nothing row comparisons and current source/recovery locks. A changed journal, authority, writer, finalizer, expired recipe, or partial transaction refuses the transition. - Historical snapshots and objects remain pinned after new success. This intentionally retains storage. Release requires a separate reviewed action. - An older importer cannot use the successor generation contract. Use the containing canonical owner for recovery. Normal app delivery must also carry the expiration guard; a private maintenance graph does not patch a running server's cleanup worker. - The migration owner crosses 1,000 lines. The bounded compensation transition remains beside its private source inspection, fingerprint, fence, reservation, and publication contracts. Moving it now would require exporting or duplicating those safety owners. No unrelated refactor or new framework is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source editing, Git, and local execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
29a66d679e |
fix(workspaces): verify archived repository intents before publication (#20441)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace migration moves local files into durable, authenticated snapshots. > - Archived Git workspaces can retain an unmaterialized repository request after their local tree is removed. > - Admission cannot prove filesystem absence from the removed tree. > - This pull request checks strict request candidates against the private restored Git tree and the authenticated snapshot readback. > - Publication rechecks current repository rows without changing the request or task history. ## Linked Issues or Issue Description **What happened?** An archived workspace with an authenticated restoration recipe and an unresolved, unmaterialized repository request fails migration with `repository_preparation_unresolved`. Its original directory no longer exists. **Expected behavior** Migration can preserve the request when the actual private restored tree proves its target absent and every parent safe. It must refuse partial materialization, symlinks, or changed request state. **Steps to reproduce** 1. Archive a Git workspace with a valid saved recipe and a pending repository request. 2. Remove its original local workspace directory. 3. Run canonical workspace migration. The previous implementation refuses before it can inspect the restored tree. **Paperclip version or commit** Reproduced on `1ca3e8ea2a759e511ab2d2659339c6c770a7a2ba`. **Deployment mode** Self-hosted server. This is a core workspace migration bug. **Relevant logs or output** `remote_workspace_migration_blocked:repository_preparation_unresolved` Refs #20416, #20424, #20436. These fixes retain their source, recovery, and ownership guards. ## What Changed - Separate strict repository request metadata from filesystem absence proof. - Admit only strict candidates backed by a canonical archived Git recipe to private materialization. - Verify target absence and safe parents before a new upload and after authenticated archive restore. - Lock and recheck current repository rows before authority publication, including recovered uploaded archives. - Preserve previous-release recipes, request rows, task selection, history, provider references, and unresolved recovery obligations. No persisted schema or public option changes. ## Verification - Genuine baseline RED: both pending and needs-repair archived requests fail on unchanged master. - GREEN: 27 focused real-PostgreSQL/FS/Git tests pass. Includes partial/symlink refusal before upload, request mutation during upload, recovered-upload readback, and existing native F/source checks. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'archived repository intent|unmaterialized repository request|unresolved repository request|pending repository|unselected archived recipe|archived prepared source|competing native F owner'`. - After rebasing onto current master, all six new cases pass on `0b6854789acc067a44e697acebdc04e51b9afe88`. Server layered typecheck, module boundaries, and `git diff --check` pass. A review follow-up adds one pending-path mutation regression (1/1 real-PG pass) without changing product code. Fresh required CI runs on final head `2a30a4b444108488ac8b51d0c46c340ee15227b9`; fresh Greptile is 5/5. Server7 had an unchanged trace-parent assertion failure; that same shard passed on the previous product-identical head and the linked-source sibling PR. The original failure remains recorded. The single targeted server7 retry passed. Final head has 56 successful checks, 4 intentional skips, and 1 neutral review status. ## Risks - Metadata admission is only a candidate check. A candidate still fails unless the private authenticated filesystem proves absence. - Publication keeps repository rows locked while it checks their current paths against the owned restored tree. A real pending-relativePath mutation now tests this final check. Existing workspace/source/native locks remain in place. - Absence is proved only in SAVED data reconstructed from the retained recipe and snapshot. This does not prove that a request never materialized before archival. Existing Git restore points can exclude ignored `.paperclip-repositories` paths outside recorded checkouts. This PR does not change that preexisting coverage or recover bytes that were never saved. - The original recipe, restore point and checkout references remain unchanged. No files are recovered into or removed from the original directory. No request is marked ready or restored. Partial target files present in reconstructed saved data remain blocked and retain their state. ## Model Used OpenAI GPT-6.1 Codex, with reasoning, repository tools, and code execution. The host does not expose an exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6f4008f9b |
docs: explain individual agent interaction permissions (#20422)
Document all eleven interaction controls, sparse defaults, CLI administration, ownership rules, pending-delivery behavior, and access boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5bad074baa |
feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude
|
4158799d1b |
test(chaos): add provider message faults (PV1-PV3) and a saved-question scenario (#20432)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos`) tests that agent work survives faults. > - The harness has two fault families: control-plane restarts (K*) and storage faults (ST*). > - On 2026-11-09, two production failures came from a third family. The provider was alive, but a protocol message was late or missing (#20330, #20336). > - The harness could not produce that family, so it did not catch these failures. > - This pull request adds provider message faults (PV1 delay, PV2 drop, PV3 duplicate) on the real path between the runner and the Codex app-server. It also adds a saved-question scenario (Q1) that uses the faults. > - The benefit: the harness can now find this class of failure, and it fails against both pre-fix commits. ## Linked Issues or Issue Description Refs #20330 Refs #20336 **What happened?** A Codex interruption terminal that arrived after 18 s failed a governed wait (#20330). A dropped terminal after a saved handoff failed shutdown settlement with "Couldn't finish this attempt" (#20336). The chaos harness could not inject either fault. **Expected behavior** The harness can delay, drop or duplicate a chosen provider protocol message for one task. A scenario fails on the pre-fix product code and passes on master. **Steps to reproduce** Run the wave commands in Verification. Run them on master and in a worktree at the pre-fix commit with `tests/restart-chaos` copied from this branch. ## What Changed - `provider-faults.ts` and `provider-shim/`: the PV1–PV3 fault family. - A resolve hook goes into the wave's server through `NODE_OPTIONS`. It answers only the `@openai/codex/package.json` lookup that product code uses to find the local Codex. It returns a stand-in package in the instance temp root, and then removes itself from `NODE_OPTIONS`. - The stand-in `codex` bin is a stdio shim. It starts the real Codex with the same arguments and relays JSON lines unchanged. It changes only the message that the task's rule names: the `turn/completed` after a `turn/interrupt` (`terminal`) or the response to `turn/interrupt` (`interrupt_ack`). Each rule applies once per task. The task nonce binds the rule to the task. - Every decision goes to `<wave>/provider-faults.jsonl`. A PV task fails `provider_fault_applied` when its fault did not happen. - There is no product change. Thus the same harness runs against older product commits. - `question-handoff.ts`: Q1 workload. The agent saves one question with `request_human_input`. After the asking run settles, the board answers through the respond route. The oracle checks that the question was asked once, the asking run succeeded, no run failed, nothing ran before the answer, and the brief has the answer. - PV faults run only with `--env local --adapters codex`, only with workloads that interrupt a turn (Q1, S1, G1), and not in the same wave as restart faults. A PV1 delay passes only when the held message was released. - Registry edits: `--faults` accepts PV1–PV3. New options are `--provider-fault-message` and `--provider-fault-delays` (a sweep, cycled across tasks). The edits are in `types.ts`, `config.ts`, `scheduler.ts`, `workloads.ts`, `observer.ts` (Q1 board step) and `main.ts` (install the seam and add the check). - README section with commands, the seam, and what is not covered. ## Verification - Unit tests: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`. 23 files and 279 tests pass on the final head. The new tests drive the real shim against a stand-in provider for each kind (delay, drop, duplicate, interrupt ack, once per task, held message discarded on exit, a multibyte character split across writes). They also run the resolve hook in a child Node process. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (real Runner, real Codex 0.160.0, gpt-6.1-sol, macOS arm64): Pre-fix runs use a worktree at the merge parent with `tests/restart-chaos` copied from this branch. Common arguments: `--env local --adapters codex --workloads Q1`. | Scenario | Product code | Wave | Result | | --- | --- | --- | --- | | 1. PV1, terminal 18 s late (`--faults PV1 --provider-fault-delays 18 --tasks 1 --seed 101`) | master (`8004ad069e`) | `w-20261110T000333-s101` | pass: terminal held 18 s and released, asking run `succeeded`, one question, brief has the answer | | 1. same | `34c1e6469c` (parent of #20330 merge) | `w-20261110T014734-s101` | **fail**: Codex stopped at 14.9 s with the terminal still held, asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt. Your message is saved.", Retry needed (R3) | | 1. sweep 5/14/25/29 s (`--faults PV1 --provider-fault-delays 5,14,25,29 --concurrency 2 --tasks 4 --seed 102`) | master | `w-20261110T015426-s102` | 4/4 pass | | 1. same sweep | `34c1e6469c` | `w-20261110T015708-s102` | 5 s and 14 s pass; **25 s and 29 s fail** (`native_provider_terminal_failed`, Retry needed): the old 15 s drain | | 2. PV2, terminal dropped (`--faults PV2 --tasks 1 --seed 103`) | master | `w-20261110T015239-s103` | pass: terminal dropped, asking run `succeeded` through verified shutdown, one question, nothing ran before the answer | | 2. same | `a67b936885` (parent of #20336 merge) | `w-20261110T015010-s103` | **fail**: asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt", Retry needed (R3) | The first master wave exited 1 for one cold `agent_boot` speed sample (8.6 s against a 7 s p95 budget), before any fault, while a preflight build ran in parallel. Its task passed. No new product failure was found on master. ## Risks - Test-only. No product code changes. - The shim adds one Node process in front of Codex for PV waves only. Other waves do not change. - The nonce binds a rule to a Codex process. If one Codex process serves two tasks, the second task's traffic can bind it too. Q1 tasks use separate sessions. - Claude is not covered. Claude runs through the ACPX sidecar against a hash-verified provider pack, and its ACP protocol has no interrupt acknowledgement. A Claude shim needs a different seam. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use, code edits and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
490d97bf1e |
test(perf): guard Mine against waiting on the company-wide workspace list (#20419)
> Follows #20388 (merged as |
||
|
DottaandClaude Opus 5.5
|
9a4a5066fa |
perf(inbox): stop optional reads from holding Mine rows hidden (#20407)
> Follows #20388, which merged to `master` as `50df9a19b9`. This branch now includes `origin/master`, so its diff carries only this PR's change: `ui/src/pages/Inbox.tsx`, `ui/src/pages/Inbox.test.tsx`, `ui/src/api/approvals.ts` and `ui/src/api/access.ts`. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Mine inbox is the first page a board user opens to see the tasks that need them > - The page hides task rows until every inbox read finishes, including reads that do not change the rows > - A slow user directory, approvals or join-request read therefore delays every task row, and a stalled read keeps the skeleton up > - Loading is also a layout contract: content that arrives late must not push visible rows down or reset keyboard selection > - This pull request removes the user-directory gate where names only fill row text, and bounds the approvals and join-request gates so a failure or stall cannot hold rows hidden > - The benefit is that a slow optional read no longer hides task rows, and the rows still do not move when late data arrives ## Linked Issues or Issue Description Refs #20388 **What existing behavior does this improve?** The Mine inbox reveal gate waits for every source, including the user directory, approvals and join requests. **Subsystem affected** Board UI inbox (`ui/src/pages/Inbox.tsx`). **Current behavior** If the user directory is slow, every task row waits for it, although user names only fill text in the rows. If approvals or join requests stall, the skeleton stays up until the request ends. **Proposed behavior** Rows reveal on session, experimental settings, `inbox/mine`, dismissals, agents and projects. The user directory fills names in place. Approvals and join requests can delay rows for one attempt of at most 10 s. **Reason and benefit** A slow optional read no longer hides the task list. A failed read shows the existing "Some inbox items couldn’t load" banner with a retry instead of a skeleton. **Breaking changes** None. ## What Changed - User directory: it no longer gates task rows. It still gates rows when they are grouped by assignee, because user names name and order those groups. - Approvals and join requests: they still gate rows (see "Kept gates" below). The gate now covers only the first attempt (`isLoading && failureCount === 0`). That attempt uses a 10 s request timeout (`INBOX_OPTIONAL_READ_TIMEOUT_MS`). After a failure or timeout, rows reveal and the query retries in the background. - Timeout notice: a timed-out approvals or join-request read is a transient error, which the inbox hides. Once the app's retries end, the inbox now shows the "Some inbox items couldn’t load" notice and "Retry missing items" button for a `TIMEOUT` on those reads. Other transient errors stay hidden as before. - `approvalsApi.list`, `accessApi.listJoinRequests` and `accessApi.listUserDirectory` accept optional `RequestOptions` (signal and timeout). Other callers do not change. - Tests in `ui/src/pages/Inbox.test.tsx` ("optional reads and the reveal gate"): - rows reveal while the user directory is pending, and keep their order, their DOM nodes and the keyboard selection when it arrives; - assignee grouping still waits for the user directory; - a delayed approvals read holds rows (the kept layout guarantee); - a failed approvals or join-request read reveals rows with the retry banner and no skeleton; - after one failed attempt, rows reveal while the approvals read retries; - a stalled approvals read times out and reveals rows; - approvals or join requests that keep timing out under the app retry policy (`shouldRetryRequest`) show the missing-items notice and retry button. ### Kept gates Approvals and join requests stay in the gate. On Mine they sort in among the task rows by time (`getInboxWorkItems`). A pending approval is usually recent, so it sorts near the top. If it arrived after the rows were visible, it would push every task row down by one row. I could not reserve space for it, because the count is unknown until the read returns, and an empty reserved slot would collapse and shift the rows the other way. So I kept these two gates, and I bounded them so they cannot hold rows hidden on failure or stall. Keyboard selection already follows the selected row by key, so late data does not reset it. The user-directory test checks this. ## Verification Focused checks on this revision: - `cd ui && npx vitest run src/pages/Inbox.test.tsx src/api/access.test.ts`: all passed (91 Inbox tests). - Red/green: with the #20388 versions of `Inbox.tsx`, `approvals.ts` and `access.ts`, three new tests fail (user directory pending, retry after first failure, stalled approvals). They pass with this change. The other four pin guarantees that already held. - Review fixes: the timeout-notice tests fail without `eb5d55da86` and pass with it. `cd ui && npx vitest run src/pages/Inbox.test.tsx`: 94 passed. - After merging `origin/master` (`54d73aef19`): `cd ui && npx vitest run src/pages/Inbox.test.tsx src/api/execution-workspaces.test.ts`: 101 passed. `pnpm --filter @paperclipai/ui typecheck`: passed. - CI note: on head `e438e16c80`, `ci / e2e shard (3/8)` failed in `planning-mode-visual-verification.spec.ts` (the first task's `task-chat-composer-mode` did not appear within 30 s). That journey runs onboarding into the first task chat. It does not open the inbox or use the changed reads, so I treat it as unrelated. The other 33 tests in the shard passed. The new head reruns it. `pnpm check:token-gates`: clean. - `pnpm verify --base perf/inbox-referenced-workspaces` selected 2,219 Vitest files because `ui/src/api/access.ts` is imported widely. I did not run that broad set locally (the shared host was under heavy load). Required CI supplies that coverage. Performance (same local fixture as #20388: authenticated mode, non-admin member, 8,750 execution workspaces, 38,750 tasks, production UI bundle, headless Chromium, new context per run). The host load average was about 30–48 from other work, so I used alternating pairs. | Scenario | #20388 | This PR | Median paired saving | | --- | --- | --- | --- | | No injected delay, 6 pairs | median 870 ms (747–1,922) | median 1,046 ms (795–2,002) | mixed signs, no measurable change | | User directory delayed 3 s, 5 pairs | median 3,980 ms (3,852–6,048) | median 1,096 ms (698–1,263) | 2,814 ms | | Approvals stalled 20 s, 1 run each | 20,829 ms | 10,737 ms | — | On this fixture the optional reads finish before `inbox/mine`, so this PR does not change the normal cold load. It helps when an optional read is slower than `inbox/mine`, for example under server load. ## Risks - Low risk. The change only removes or bounds waits for reads that do not decide row order. - Assignee names can appear a moment after the rows on a slow user directory. The rows keep their height and position. - If approvals or join requests time out on every retry, the notice and retry button appear. If they time out and later succeed, they insert into the list after the rows are visible. This happens only in that failure path. - The 10 s timeout applies to these inbox reads only. Other pages that read the same data do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, headless browser). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
99a6daf5bc |
perf(auth): resolve a Cloud tenant actor in one database round trip (#20390)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Paperclip Cloud, each request resolves its board actor from trusted Cloud headers in `actorMiddleware` > - This runs for every request after the liveness probe, also for static files such as `favicon.ico` and `index.html` > - The resolution ran three database statements one after another: the stale `instance_admin` purge, the membership read and the owner-elevation read > - On a loaded database, each statement waits. Three sequential waits make every request slow, also a request that needs no data > - This pull request keeps the purge and the tenant sync in order and runs the two reads at the same time as them > - The benefit is one database wait per request instead of three ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud tenant actor resolution in `server/src/middleware/auth.ts` (`resolveCloudTenantActorOnce`). **Subsystem affected** Server authentication middleware (Cloud trusted-header path). **Current behavior** Measured on the hosted staging instance on 2026-11-09 from a signed-in browser, 45 paired samples in 40 s: - `/api/liveness` (mounted before actor resolution): client median 92 ms, p90 255 ms, server time ≤ 1 ms. - `/favicon.ico` with a session (after actor resolution): client median 413 ms, p90 1,751 ms, max 4.6 s. - `/api/cli-auth/me`: server median 346 ms, p90 1,528 ms. The same Node process answers liveness quickly, so the event loop is not blocked. The database is busy (6–8 of 8 vCPU for the whole day), and trivial primary-key reads waited 100–280 ms each. Three sequential reads explain the ~320 ms median gap between liveness and favicon. **Proposed behavior** The purge and the debounced tenant sync stay ordered. The membership read and the owner-elevation read run at the same time as them. The sync writes only the primary company's membership, and the membership read excludes that company, so the result does not change. **Reason and benefit** A synced request costs one database wait instead of three. With 150 ms of injected database latency, a synced resolution goes from 914 ms to 310 ms (3 runs each, same machine). **Breaking changes** None. ## What Changed - `resolveCloudTenantActorOnce` starts the purge-and-sync, the additional-membership read and `resolveOwnerInstanceAdmin` together and awaits them with `Promise.all`. A membership read failure still degrades to the primary company. The owner-elevation read still fails closed. - New regression test `server/src/__tests__/cloud-tenant-actor-round-trips.test.ts`. It connects through a TCP proxy that delays every client write by 150 ms and asserts that a synced resolution finishes in less than 4 × the delay. Before the change it takes about 6 × (914 ms); after the change about 2 × (310 ms). - `auth-session-route.test.ts`: the fake database binds each `returning()` to its own insert, and the test matches inserts by content instead of position, because the instance-settings insert can now happen in parallel with the tenant sync. ## Verification - `cd server && npx vitest run src/__tests__/cloud-tenant-actor-round-trips.test.ts` — red on `origin/master` (914 ms, 913 ms, 916 ms), green with the change (310 ms). - `cd server && npx vitest run src/__tests__/cloud-tenant-company-provisioning.test.ts src/__tests__/auth-session-route.test.ts src/__tests__/liveness.test.ts src/__tests__/cloud-warm-standby.test.ts src/middleware/cloud-tenant-actor.test.ts` - `pnpm verify` (selected tests), and CI. - After deploy: compare `/favicon.ico` and `/api/cli-auth/me` client time with `/api/liveness` on staging. Expect the median gap to drop from about 3 database waits to about 1. ## Risks - Low risk. Authorization does not change: the same statements run and the request waits for all of them before it continues. - Each request now uses up to three pooled connections at the same time instead of one. The staging pool limit is 500, so this is not a concern there. Smaller pools reach their limit sooner under a burst. - The diff re-indents the purge-and-sync block. The statements in it do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, browser, code execution). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
a9d0946e64 |
fix(ui): keep the persisted cache's final state when it cancels reads (#20431)
The persisted read cache cancels in-flight reads with revert: false when it clears account data or evicts a denied task, then writes their final state itself. A non-silent cancel settled later as a query error, which overwrote that state with CancelledError and reached QueryCache.onError. Cancel these reads silently so the written state stays. Refs #20430 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
f73ff4ace9 |
fix(native): reprove stale filesystem preservation receipts (#20438)
## Thinking Path > - Paperclip manages agent work and preserves saved native sessions across releases. > - Filesystem independence receipts bind database facts to preserved controller and skill evidence. > - A workspace migration can add run metadata without changing those preserved bytes. > - That change makes the saved facts digest stale. Full native preservation then refuses the existing receipt, even after fresh source checks. > - WEB preservation already supports a full current proof and guarded receipt refresh. > - This pull request applies that pattern to full native receipts, reuses the authenticated archive, and keeps the existing ownership and destruction checks. > - The benefit is a current receipt without a new archive or provider replay. ## Linked Issues or Issue Description Refs: #20322. **What happened?** Adding the canonical `legacyWorkspaceRecoveryMigration` marker to a settled native run makes its saved receipt stale. The full native preservation owner checks current evidence but then refuses the old facts digest. **Expected behavior** The owner can refresh the proof metadata after it proves all current facts and preserved bindings again. Changed evidence or authority must leave the prior receipt intact. **Steps to reproduce** Use the added PostgreSQL regression. It saves a valid receipt for a succeeded source with 453 checkpoint parts, adds the producer-shaped migration marker, and calls preservation again. The unchanged owner fails with `native_harness_filesystem_independence_unverified`. ## What Changed - Reuse the prior controller archive through the existing authenticated readback path. - Require the old receipt to match every current immutable binding before a facts refresh. Reconstruct its exact digest from unchanged bindings and the prior proof fields. - Recheck current database facts, assets, controller and local files under the existing finalizer, selection, task and run fences. - Publish with an exact prior-receipt comparison. Keep a valid concurrent winner and refuse changed receipts or new owners. - Audit the prior receipt digest and four proof fields. Add focused source, integrity, race and DB-only validation tests, and update the database guide. ## Verification - Local Node24 canonical preflight and plugin SDK prerequisites passed. The Runner build came from the content cache. After a normal merge of fresh master, preflight confirmed unchanged build inputs. - Genuine RED on `3040448da5`: the saved receipt is valid before the migration marker and stale afterward. Reproof fails at the old receipt guard (one failed, 102 intentionally not selected). Both retained RED attempts reached that same guard. - GREEN on final local merge `00f4b5d678083106afc88fb2f7e0588961b46d6d`: 19 PostgreSQL controls passed, 91 intentionally not selected (17.75s). The wrapper completed cleanup. - The positive preserves the same archive, controller bytes, assets, local proof and all 453 checkpoint parts. It performs authenticated object reads and zero archive PUTs. Runs, results, finalizers and leases remain unchanged except for the receipt. The audit reconstructs the exact prior receipt. - DB-only validation passes with the original home, state directory and volume absent, while storage reads throw. - Refusal and race controls cover changed asset bindings, raw controller bytes, missing destruction evidence, unknown receipt fields, a valid concurrent winner, changed receipts and a late active owner. Existing WEB refresh, physical finalization-lock and saved-binding controls also pass. - Server TypeScript passed, including the adjacent integration test. Fresh Greptile is 5/5 on exact head `00f4b5d678083106afc88fb2f7e0588961b46d6d`, with no review threads ([review](https://github.com/paperclipai/paperclip/pull/20438#issuecomment-6756548830)). Final-head CI is terminal: 56 success, 4 skipped and 1 neutral. The aggregate `ci / verify` passed ([run](https://github.com/paperclipai/paperclip/actions/runs/42737704110)). - The first Runner-support job had one unrelated process-inspection nonpass: `server-stop.test.ts` expected a successful wrapper close, but its bounded process inspector returned no table. Its cause remains unknown. The unchanged test, process helpers, workflow and dependency inputs match the passed master control. One authorized same-head job retry passed all 328 files / 6,927 tests (4 skipped); the exact previously failed case passed in 665ms ([retry](https://github.com/paperclipai/paperclip/actions/runs/42737704110/job/129576420893)). No timeout, assertion or source change was made for the retry. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts -t "reproves stale native facts|retains stale native proof|guards stale native reproof|invalidates a published receipt after changed|actual physical-finalization lock|refreshes stale WEB facts|refuses a WEB facts refresh|retains or refuses the existing receipt when an earlier capture finishes late" pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - Old receipt schemas and immutable bindings remain required. Full fresh proof carries compatible saved receipts forward; unknown fields or changed evidence refuse refresh and preserve the old receipt. - Full native preservation still requires canonical destroyed-provider receipts. The change retains the existing source capability, cancellation, finalizer and publication checks. - Only proof metadata changes. No provider operation, archive replacement, hosted recovery or deployment was performed for this PR. Live qualification remains with the operator after delivery. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting mergecanary/v2026.1110.0-canary.1 |
||
|
DottaandPaperclip
|
07f6e8ff75 |
perf(workspaces): group same-root cohorts with bounded workers (#20437)
## Thinking Path > - Paperclip preserves agent work when its control-plane server is replaced. > - Online migration imports local workspaces into authenticated snapshots. > - Several workspace records can point to one physical directory. > - Shared child verification avoids repeated work inside one held cohort fence. > - The eight-member limit splits 30 identical-root aliases into four cohorts and repeats the child passes. > - This pull request admits up to 32 members only for one canonical physical root, while keeping each work stage bounded to eight peers. > - The benefit is fewer repeated child checks while capture and publication guards stay in place. ## Linked Issues or Issue Description Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found shared verification and restart fixes, but no larger same-root cohort owner. The shared verifier and archived-source changes are merged into this branch from current master. **Current behavior** Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and 6. Each cohort performs an initial and final local child-validation pass. The same children are checked eight times in total. **Proposed behavior** Admit one cohort with nine to 32 members only when every canonical resolved physical root is exactly equal. Keep the existing complete source-scope equality and admission guards. Keep mixed-leaf cohorts limited to eight members. Bound source reads, reservations, recovery checks, final readers, replay and cleanup to eight joined peers. Capture concurrency stays at most eight. **Reason and benefit** For N distinct canonical covered children, the 30-alias case changes 8N local child checks to 2N, a 75% reduction in repeated child validation. If N is 68, that is 544 checks versus 136. Thirty parent captures, uploads and restores remain required. No total wall-clock improvement is claimed. **Breaking changes** Saved cohort fields and exact membership remain unchanged. Older tools that enforce the eight-member bound refuse larger cohorts. Use the newer tool to resume them; never split or regroup their saved membership. ## What Changed - Raise the canonical service, saved-cohort and maintenance-request member bounds to 32. Keep the capture-worker bound at eight and single-workspace modes at one. - Require one exact canonical physical root for more than eight members. Keep the existing full source-scope comparison, leaf rule and P/I/L/F guards. - Use one private stage mapper with batches of at most eight. Join every started peer before an error can leave the stage. Stop future batches after a failure or cancellation. - Preserve the existing cancellation and pending-snapshot ordering for stages of at most eight members. Check cancellation around extended batches. - Keep all saved cohort, member, operation and snapshot identities, proof refresh and atomic publication checks. - Bound cleanup without skipping later scratch cleanup after an earlier cleanup error. - Extend the existing PostgreSQL fixture with ten real aliases, partial-upload resume, cold restoration, replay and mixed-root refusal. Extend the request boundary test. - Document old-tool refusal, work counts and the retained-scratch capacity bound. ## Verification - Canonical Node24 worktree preflight and Plugin SDK prerequisites passed. After merging current master, preflight passed again with unchanged cached build inputs. - Genuine RED on the old bound: both real ten-alias import/resume cases and the 32-member request reject the eight-member limit (3 failed; 47 not selected; 18.23 seconds). - First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias case performs exactly four local child captures for two children: one initial and one final pass. Actual reader, transaction and upload maxima are eight. All ten archives cold-restore after the original source is parked; replay retains the same cohort. - The partial-upload case keeps all ten saved operation IDs and the existing snapshot IDs. More than eight members across different canonical leaf roots refuse before any journal or storage work. The request parser accepts 32 members and eight workers, and refuses 33 members or nine workers. - The first expanded corpus had 55 passes and one real compatibility failure: a new eager cancellation check skipped the old pending snapshot boundary. The mapper now retains existing cancellation ordering for at most eight peers. That failure remains recorded as a nonpass. - Final sources include current master at `18697b8d21`, with the verifier, archived-source and UI repairs. The selected four-file corpus passes 47 cases in 78.88 seconds, with 191 intentionally not selected. The full maintenance suite separately passes 23 cases in 10.21 seconds. Nine cases overlap; the final union is 61 distinct passing checks. - Final coverage includes all 27 cohort cases, all 23 maintenance cases, all six actual old-format compatibility cases and five archived prepared-source controls. Existing source mutation, finalizer contention, actual fence loss, cancellation peer joins, journal compare-and-swap and outer-commit rollback controls pass. - Final server TypeScript, module boundaries and whitespace checks pass. Only the resource-risk documentation changed after that test run; runtime and fixture bytes stayed unchanged. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/maintenance/online-workspace-migration.test.ts server/src/__tests__/online-workspace-migration-compatibility.test.ts`; archived controls use the existing `remote-workspace-migration.test.ts` cases named `unselected archived recipe` and `archived prepared source`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial source review was 4/5 and requested a control for the extended reader-batch stop boundary. The test-only follow-up `ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member failure and cancellation cases. They hold a first-batch peer, verify that the source fence stays held, and prove members nine and ten never start preflight. Both pass in 10.87 seconds with no new journal, snapshot or upload effects. Runtime source hashes are unchanged; the local union is now 63 distinct checks. - The first stop-test attempt observed the two controlled readers before all SQL-backed peers had entered the inspection callback. Its assertion was too early. The corrected test checks the held first-batch readers at that point, then requires all eight to join before completion. Both cases pass; the earlier attempt remains a nonpass. - Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664) and [aggregate verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765) passed. Full rollup: 56 successes, four intentional skips and one advisory neutral result. Classic Snyk passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693) both reviewed that exact head at 5/5. The one review finding is addressed and resolved; no open findings remain. Older partial CI and the initial 4/5 review remain nonfinal evidence. Hosted acceptance and measured wall-clock savings remain pending. ## Risks - Completed parent captures retain their scratch until the atomic publication completes. A 32-member cohort can retain up to four times the parent scratch of an eight-member cohort. Check actual parent capture, verification and restore sizes against current target filesystem capacity before selecting a larger cohort. Keep the existing resource guards. - Larger member arrays must not raise execution fanout. Every reader/reservation/final stage is bounded to eight and joins current peers before failure. Capture workers remain at most eight; child-verification workers remain two. - More than eight members require the exact same canonical physical root and complete source scope. A different root, new writer, changed source, recovery owner or journal still refuses through the existing owners. - Shared verification is limited to the held invocation fence. The separate fresh final pass remains required after all captures join. Different child sets and restart attempts do not share a local-byte proof. - The old saved membership and archive identities remain authoritative. Older tools fail closed on the larger bound. No schema, archive or provider format changes are included. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
1ca3e8ea2a |
fix(native): let a resume claimed after a restart write the run's existing log (#20427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run is interrupted, Paperclip can persist a same-run resume: a later attempt continues the same run from its exact checkpoint > - If the server restarts before that resume is dispatched, the new server process claims it. The resume then reopens the run's existing log, and only the current controller may write it > - The resume claim took the lease owner but left the dead process's controller identity on the lease. The log writer check compares the boot id, so it refused, and the exact checkpoint resume failed as `run_log_writer_authority_lost` > - This pull request stamps the claiming process's identity on the resume claim, as restart recovery's own claim already does > - The benefit is that a resume persisted before a restart continues after it, instead of failing and needing a manual Retry ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261110T003423-s9903` (local ST2, seed 9903), task t01 (codex C1, K1). The wave went 7/8; t01 was the one task that needed a Retry. - Run 279772b3 was SIGKILLed during its first turn. The restarted server resumed it, and the Codex snapshot failed ("cannot safely recover an ambiguous Codex turn start"). A same-run resume was persisted. - A second SIGKILL hit (00:35:37), and the next process dispatched the scheduled resume at 00:35:56. - The resume's run-log claim failed in `createRunLogWriterAuthority`: the coordinator lease still carried the killed process's `controllerBootId`. The exact checkpoint resume was refused (`run_log_writer_authority_lost`), and the run ended `adapter_failed`, needing a Retry. The wave had S3 latency injected, but the failure does not depend on it. The new test reproduces it on master with no storage faults. ## What Changed - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `dispatchNativeSessionResumptions` writes the claiming process's controller identity (boot id, pid, host hash, process start) along with its lease owner and expiry. This matches the restart-recovery claim, so ownership proofs and the run-log writer check see the process that actually owns the resume. - `server/src/__tests__/native-session-resumption.test.ts`: a regression test for this case. ## Verification - `native-session-resumption.test.ts`: - A coordinator persisted for resume by a previous process (stale boot id), on a run with an existing object-store log, is dispatched. - The run-log writer check accepts the new claim, and the lease carries this process's identity. - The test fails on master with `run_log_writer_authority_lost`. - Regression: the native resumption, binding, recovery-deadline, restart-recovery, scheduled-resume, owner-release, runner-ownership and recovery suites pass locally. `tsc --noEmit` passes for `server`. ## Risks - The resume claim now also refreshes the controller pid, host hash and process start on the coordinator row. These describe the process that owns the lease, exactly as after a restart-recovery claim. A claim by the same process writes the same values as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
232480d18f |
perf(runner): write preparation span rows to the run log in the background (#20426)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before a native run starts its provider, the executor copies the preparation spans (queue, prepare phases, environment) into the run trace and the run log > - It records about 25 spans and waits for one run-log insert per span, one after another > - On a hosted test deployment, that step (`native.controller.span_persistence`, added in #20396) took a median of 0.28 s, and every task waited for it before the Runner started > - The run trace already has a way to write diagnostics without blocking the caller: `defer`, which `finish` drains before the root span closes > - This pull request writes those run-log rows through that path, in order, while the spans themselves are still recorded at once > - The benefit is about 0.25 s less before the provider starts, on every native run, cold or warm ## Linked Issues or Issue Description **What existing behavior does this improve?** The time between the end of heartbeat preparation and `native.session.execute` for native Runner tasks. **Subsystem affected** Server: native runtime tracing (`server/src/services/native-runtime/native-run-trace.ts`) and the native session executor. **Current behavior** `executeNativeSession` calls `trace.record` for each preparation span and awaits it. `trace.end` awaits `emitCompletedSpanEvent`, which inserts one `run.performance.span` row through `onEvent`. On a hosted test deployment, measured with the #20396 spans across 10 runs, `native.controller.span_persistence` took a median of 0.28 s: - 0.30 to 0.36 s for runs right after a computer wake - 0.17 to 0.25 s for runs on an awake computer It is the second largest step between preparation and the provider start (the median of that whole gap is 1.0 s). **Proposed behavior** The spans are still started and ended at once, with the same parents. The speed-budget bookkeeping still runs at once too. Only the run-log row is written later: one row at a time, in the order the spans ended, before the trace finishes. **Reason and benefit** The rows are diagnostics, and they already carry their own start time (`wallStartedAtMs`, `startOffsetMs`). Readers do not depend on the row order: the only reader in the repository, `summarizeStartupReuse`, looks for `runner.process.launch`, which this change does not defer. The provider can start without waiting for these inserts. **Breaking changes** None. ## What Changed - `native-run-trace.ts`: `end` and `record` take an optional `deferRunLog`. With it, `end` queues the run-log row on a per-trace chain registered with `defer`, so rows are written one at a time in order and `finish` waits for them. A row requested after the trace finished is dropped, as for other deferred diagnostics. - `native-session-executor.ts`: the preparation spans and the `environment.startup` scope use `deferRunLog: true`. - `doc/run-log-events.md`: note that native preparation span rows are written late, in order. - `native-run-trace.test.ts`: two tests. - Spans are recorded under their parent while every row write is blocked, and the rows are written in order before `task.provider_session.measured`. - A deferred row requested after `finish` is not written. ## Verification - `npx vitest run src/services/native-runtime/native-run-trace.test.ts src/services/native-runtime/native-run-trace-context.test.ts` (34 passed). Both new tests fail without the source change: the first times out waiting on the blocked write, and the second sees the row written. - `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts src/services/native-runtime/native-session-executor.test.ts` (1321 passed) - `pnpm --filter @paperclipai/server typecheck` ## Risks - Low risk. The change is diagnostic only: OpenTelemetry spans and the speed-budget checks are unchanged. - The preparation rows can now land in the run log after the first Runner events. Readers already use the payload times, not the row order. - If the server process dies between the provider start and `finish`, the preparation rows of that run can be missing from the run log. The OpenTelemetry spans are not affected. - The deferred rows are written while the run continues, as budget-breach rows already are, so the sink sees concurrent writes from one run. This was already the case for deferred diagnostics. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, Honeycomb queries) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4608744977 |
fix(workspaces): preserve recovery for verified archived Git sources (#20436)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved Git files and pending native recovery. > - The archived-Git owner validates saved recipes, exact cleanup receipts, and surviving owned branches. > - Online recovery preservation currently accepts only the recipe variant. > - A valid cleanup receipt or owned branch therefore cannot migrate while an old native reference is still prepared. > - This pull request admits every verified archived-Git variant through the same recovery and finalization locks. > - Tasks stay done and keep their selected workspace, history, and pending provider obligations. ## Linked Issues or Issue Description Refs #20424, #20298, and #20418. The related cleanup work in #19812 does not cover this admission gap. **What happened?** Canonical inspection accepts the exact old commit of an archived Git workspace through a valid branch-deletion receipt or an owned surviving branch. Migration still reports `remote_copyback_unresolved` when a terminal native run retains a prepared reference. A read-only staging inventory reproduced this sole blocker for four such workspaces. Their original Git objects and canonical source proofs already exist. **Expected behavior** Preserve the verified Git baseline and every typed native/provider recovery obligation through the existing migration owner. Keep the task status, selection, history, archive state, and recovery records unchanged. Do not reopen the task or substitute the current default branch. **Steps to reproduce** 1. Save a clean archived Git workspace with an authentic cleanup receipt or a surviving owned branch, without a restore recipe. 2. Retain a terminal prepared native reference and its exact provider allocation and recovery action. 3. Remove the old task worktree. 4. Call `migrateRemoteWorkspace` online. **Paperclip version or commit** Reproduced on merged #20424, `915edcff49bb2cbacfbaee604b41e082e0397109`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Change three conditions to accept a non-null canonical archived-Git source instead of requiring its recipe field. - Use the same typed recovery inspector, sorted native finalization locks, fresh reinspection, and source-task publication locks for all accepted variants. - Reuse the existing Git source validator, private scratch materializer, encrypted snapshot restore, journal, and recovery markers. No source-proof schema or provider protocol changes. - Extend the existing real fixture for cleanup receipts and retained branches. Verify exact old Git commits and unchanged task/recovery state. Refuse receipt revocation and branch movement during upload. ## Verification - Genuine RED on merged #20424: both new source-kind cases fail with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - GREEN: nine focused PostgreSQL/Git/encrypted-storage cases pass. They include both new variants, two actual upload-time source mutations, the existing recipe interrupted-journal resume, reference/provider mutations, and a competing native finalization owner. - Command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'no-recipe archived|unselected archived recipe|archived prepared source|competing native F owner'`. - All nine focused cases, server typecheck layers, module boundaries, and whitespace checks pass on exact published head `03065f97b31846a405a8911c30640ef62374e4a4`, after the clean rebase onto merged #20418. Required CI is now clean on this exact head:56 SUCCESS and4 expected SKIPPED checks, with no pending or failed checks. Fresh Greptile is5/5 with no actionable findings or unresolved threads. - Previous-release state: existing cleanup receipts, owned branches, prepared v3 references, and journal shapes carry forward unchanged. Saved snapshots and source identities retain their existing owners. Unrecognized or changed sources still refuse by the existing named errors. Migration does not declare remote files recovered or provider allocations stopped. - Hosted work in this PR was read-only source inspection and inventory. No staging import, source mutation, lifecycle reopen, or provider call was performed by this PR author. ## Risks The existing archived-Git validator now admits its other verified variants to recovery preservation. Incorrect source proof could publish incomplete authority. This PR keeps its exact commit/receipt/branch checks and full authenticated capture/readback. It also keeps all native ownership, reference, allocation, and publication checks. Missing or changed sources remain blocked, and pending remote data retains its original recovery obligation. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
5570debc5b |
fix(ui): stop reporting cancelled queries to Sentry (#20430)
React Query's CancelledError keeps the default Error name, so the name-based abort check never matched it. A query cancelled with revert: false (for example by the persisted cache when an issue read is denied or the account changes) reached the query cache onError as an "unknown" error and was sent to Sentry. Recognize it by class. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
18697b8d21 |
perf(workspaces): share cohort child-source verification (#20418)
## Thinking Path > - Paperclip must preserve agent workspace bytes during online migration. > - A cohort can contain several aliases of the same physical source. > - The importer verifies imported children before capture and again before publication. > - The immutable-object cache avoids repeated downloads but still recaptures local child bytes for every alias. > - This pull request shares the initial proof within the held cohort fence and checks fresh bytes once more after every capture joins. > - The benefit is less repeated local archive work while child mutations still block the whole cohort. > - Required browser CI twice reproduced an independent stale composer-mode bug. A focused regression proves that refreshed persisted task mode was ignored after mount. The state repair follows task data when there is no local override. Explicit local mode choices retain their own generation so a completed send cannot clear the next draft’s choice. ## Linked Issues or Issue Description Refs: #20311, #20387, #20401, #20404. **Current behavior** Two aliases recapture the same imported child's local bytes four times, even with one immutable object read. A larger alias cohort repeats this multi-GB work for each member. **Proposed behavior** Share the successful initial verification for the same physical root, Git roots and exact recorded child set or discovery mode. After all captures join, run a separate fresh final verification for each root and recorded child set. Keep every member's database and publication checks. **Reason and benefit** The local PostgreSQL fixture proves that two aliases now perform two child recaptures instead of four. Hosted timing improvement has not been measured. ## What Changed - Add invocation-local initial and final proof maps to the existing cohort child verifier. - Retain the existing canonical source-owner check before and after shared results. - Defer cohort final nested checks until all captures finish, and join every final reader before publication or fence release. - Keep singleton verification, saved journal schemas, snapshot IDs, archive bytes and per-member publication checks unchanged. - Extend the existing real PostgreSQL alias fixture with a child mutation during the last parent upload. - Repair the required CI planning-mode race with a nullable local override: absent override follows task data; a user selection belongs to the draft. A successful send or confirmed resend/receipt clears it only if no newer choice exists. Failed saves preserve the mode and restored message. Queued-edit backup retains whether the mode was explicit. Leave the browser assertion and timeouts unchanged. ## Verification - Canonical Node24 worktree preflight and plugin SDK prerequisites passed. Server TypeScript passed. - Genuine unchanged-source PostgreSQL RED: two aliases performed four local child recaptures instead of two (one failed, 22 intentionally not selected). GREEN: five source-sharing, last-upload child mutation, capture/cancellation join and actual outer-commit rollback controls passed (18 intentionally not selected; 66.53s). The actual fence-loss control also passed (one passed, 22 intentionally not selected; 38.13s). Both wrappers completed cleanup. - Migration source and tree bytes remain identical to locally qualified `466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair the composer and its tests. - Required planning browser CI originally failed, then failed its one unchanged-head retry, because the planning chip remained absent after opening-question dismissal. Those failures remain recorded ([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985), [retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)). The final-head identical case passes in 30.5s, with all 34 shard cases passing ([final CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)). Assertions and timeouts were not relaxed. - Composer regressions reproduced stale loaded mode, optimistic-save rollback, overwritten next-draft choices, and stale mode after confirmed resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI TypeScript and token gates pass. The original planning browser case also passed locally (15.5s; 26.9s including setup). - The first final-head serialized-suite attempt was automatically cancelled at its unchanged 20-minute job limit. Grouped Turbo output did not expose an individual failing assertion; cause remains unknown ([retained attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)). One same-head diagnostic retry was a cache miss and executed all 419 heartbeat recovery tests successfully in 376.94s on the identical CI checkout ([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)). It is not classified as infrastructure or a known flake. - Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61 checks terminal (56 success, four skipped, one neutral); required CI is green ([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)). Fresh Greptile is 5/5 with every review thread resolved ([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)). Focused commands: ```sh node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished" pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx pnpm --filter @paperclipai/server exec tsc --noEmit pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates ``` ## Risks - Reuse is limited to one invocation under the same complete source and finalizer fences. Different physical roots, Git roots, recorded child sets and discovery mode have different keys. No local-byte proof is stored durably or reused after restart. - Initial and final maps are separate. A child change during any parent upload must fail the fresh final verification before the atomic publication transaction starts. - Final readers all finish before an error can release the source fence. Per-member database child rows, source fingerprints, journals, keys and publication locks retain their existing checks. - This reduces repeated verification for aliases. It does not change capture concurrency, upload policy or immutable checkpoint validation. Hosted speedup remains unverified. - The CI repair must distinguish task data from explicit draft intent. Only a successful send with the same choice generation clears its override; a later draft selection and failed-save rollback retain it. Regression tests cover these transitions. It adds no styling, request, or persistence changes. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
915edcff49 |
fix(workspaces): preserve archived native recovery during migration (#20424)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must retain saved files and pending recovery work. > - An archived Git workspace can have an authenticated restore recipe and a terminal native run with pending remote files. > - Its task can select a different workspace. Reopening the archived workspace would change that task. > - The current migration owner refuses this source because it has no live local directory. > - This pull request uses the existing online recovery preservation owner for this exact archived recipe case. > - Migration retains the old task, provider allocation, recovery action, and native sync reference. ## Linked Issues or Issue Description Refs #20298, #20394, and #20418. Related cleanup work in #19812 does not cover this migration case. **What happened?** A historical archived Git workspace with a saved restore recipe and a terminal prepared native sync reference cannot migrate online. Admission reports `remote_copyback_unresolved` and `workspace_recovery_unresolved`. The old task can already select another workspace, so reopening it is not a safe migration path. **Expected behavior** Authenticate the archived files into private scratch. Preserve all typed native and provider recovery obligations in the existing migration journal and markers. Keep the original task selection, history, archive state, and restore flags unchanged. **Steps to reproduce** 1. Archive a Git worktree with staged, unstaged, and untracked files in a saved recipe. 2. Retain a terminal failed native run, its prepared sync reference, and its matching remote allocation and recovery action. 3. Remove the old local worktree and select a different workspace on the task. 4. Call `migrateRemoteWorkspace` without `writersStopped`. **Paperclip version or commit** Reproduced on `b2a7d322ed0bafda04a0b78ce14e4f25c8d5d170`. **Deployment mode** Self-hosted server with PostgreSQL and configured object storage. ## What Changed - Permit the existing online recovery owner to inspect recipe-backed archived Git sources. - Acquire the existing native finalization locks for those sources. Reinspect their exact proofs after lock acquisition and before publication. - Lock related source-task ancestry before publication. - Reuse the existing private Git restore, authenticated archive, journal, marker, and recovery-source contracts. Add no schema or provider calls. - Leave active sources and archived sources without a valid recipe on their existing refusal paths. - Include the shared UI correction from #20418. A nullable draft override follows fresh task data when the user has no local choice. Explicit choices increment a generation. Normal and recovered successful sends clear only their own choice; failed sends and newer choices remain intact. ## Verification - Unchanged master RED: the new historical-source fixture fails with `remote_workspace_migration_blocked:remote_copyback_unresolved,workspace_recovery_unresolved`. - Five focused real PostgreSQL, filesystem, Git, and encrypted-storage tests pass on Node 24.22.2. The success case loses an upload reply, resumes the same operation and snapshot, restores the staged/unstaged/untracked bytes, and verifies unchanged task history, lease, recovery action, finalizer, and recipe flags. - Recipe, prepared reference, and provider allocation changes during the actual upload each refuse publication. A competing native finalization owner refuses before capture. - Migration command: `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'unselected archived recipe|archived prepared source|competing native F owner'`. - All server typecheck layers, module boundary checks, and whitespace checks pass for the migration revision. The migration source bytes remain unchanged through the shared UI follow-ups. - Initial CI run 42727051535 exposed an existing planning-mode browser failure: shard 3/8 passed 33 cases but the refreshed task mode did not appear. The same issue occurred twice on #20418. A real local regression proved that initial cached data could leave the composer in its old mode. This path does not invoke migration. - Review then exposed genuine defects in the initial shared UI repair: an optimistic save rollback could reset a failed draft, and an earlier send could replace a newer draft choice. Recovered delivery also left a consumed choice pinned. All three have real RED→GREEN controls. The final shared fix uses an explicit nullable override, a choice generation, and one attempt-bound settlement owner for normal and recovered success, with no value-equality inference or durable payload changes. - Exact shared UI source now passes all 145 composer tests (2.88s), the original desktop/mobile planning browser case (15.5s; assertions and timeouts unchanged), token gates, and the canonical UI typecheck. The test harness uses the full persisted `IssueWorkMode` type, including `skill_test`, to fix the observed CI typing failure. - These UI results are reused from the identical shared source in #20418 through `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`. The unchanged planning browser pass is on the explicit-state model before the later settlement-helper extraction. The final receipt/resend controls, full composer suite, typecheck, and token gates include that extraction. They do not claim a local browser run on this entire combined branch. Fresh required CI and review qualify this PR head. - Previous-release state: existing recipe, v3 prepared reference, and journal shapes carry forward unchanged. A saved partial operation resumes with the same IDs. Migration does not certify remote file recovery or provider termination. Live provider recovery and hosted migration remain separate acceptance steps. ## Risks The change admits one more source class into an existing preservation path. An incorrect proof could publish incomplete file authority. The canonical recipe validation, full authenticated restore, native finalization locks, source reinspection, and publication checks remain mandatory. Pending remote data stays pending and retains its original allocation and reference. The shared UI repair updates only an unchanged draft mode; its regression checks preserve a local user choice. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a deployment suffix or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |