mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
d5c1f48180d56c83b87f89bb74f73caeb4a7104d
690
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
8e332ccb5e |
fix(evals): name stale SDK pins and fix four chat-eval harness defects (#20579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature eval catalog (`tests/runner-e2e`) runs real chat journeys against a real server and grades the saved state, the run's tool events and the rendered reply > - The full-cohort run on master `f4337bb5fb` had 12 failures that come from the harness, not from the product or the model > - Six plugin and secrets cells stopped in 7–9 s, before the server started, because the SDK public source pins were stale > - Four more cells failed on harness timing, a first-visit prompt, or a read receipt that ignored a dedicated tool > - This pull request fixes those four harness defects and adds a cheap guard that names stale SDK pins > - The benefit is that these cells now report on the product and the model again ## Linked Issues or Issue Description Refs #19614, #20550, #20453, #20417, #20483 **What happened** On master `f4337bb5fb`, these chat eval cells failed for harness reasons: - `plugins.configure-plugin`, `plugins.disable-plugin`, `plugins.install-plugin`, `plugins.use-plugin-tool`, `routines.plugin-managed-routine` and `secrets.least-scope-delivery` stopped before the server started. The source SDK privacy scan refused, because `tests/runner-e2e/source-sdk-public-inputs.json` did not match `packages/shared/src`. #20550 and #20362 already resynced the pins on master. The pins are current at `ce1ee9e89f`. - `agent-chat.track-delegated-work` and `navigation-preferences.edit-profile`: the reply was correct and had the right link. The case read the rendered links immediately after the runs settled, before the reply rendered, so it recorded no links. - `inbox-search.archive-done` and `inbox-search.restore-archive`: setup completes five tasks, so the GitHub star prompt from #20453 opened over the chat and blocked the composer. - `status-cards.explain-card`: the follow-up run read the deleted source with `get_task` and received a 404. The reply said so. `apiReadStatuses` counted only `call_api` reads, so the grader recorded no unavailable read. Why the existing pin check did not stop the drift: `ci / Runner support suite` runs the source SDK tests and is part of the required `ci / verify`. On #19614 it passed at 03:43Z on a merge ref from before #20417 (06:12Z) and #20483 (08:14Z). #19614 merged at 14:02Z without a new CI run on current master. After that, the suite failed on every PR until #20550. So the check exists and fires; the gap was a stale green result, and the failure message was opaque (`SDK public source proof unavailable`). **Expected behavior** - A change to a pinned SDK input without a pin refresh fails with the changed files and the command that fixes it. - Cases read rendered links only after the saved reply renders. - The eval browser is a returning user's browser that has already seen the star prompt. The product does not change. - A dedicated task read tool (`get_task`, `list_task_documents`, `read_task_document`, also when namespaced) counts as a read of the record it names, with the status its receipt reports. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.<case>` for one of the cases above on master `f4337bb5fb`. 2. Read `result.json` and `snapshots/chat-catalog-observation.json` in the evidence bundle. 3. See the failures described above. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.mjs`: add `staleSdkPublicInputs(root)`. It compares the registry with `git ls-tree HEAD` for every pinned input (the roots and build inputs that the pins file lists), and returns each changed, unpinned or removed file. - `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: a test fails when the committed pins do not match the committed tree. The message names the files and gives the `--write` command. The Runner support suite runs it for every `packages/**` change. A synthetic test covers changed, unpinned and removed inputs. - `tests/runner-e2e/chat-catalog-domains/runtime-handoffs.ts` and `identity-profile.ts`: call the existing `waitForRenderedReply` before they read rendered links. - `tests/runner-e2e/chat-returning-browser.ts` (new) and `chat-catalog-flow.ts`: inside the flow's failure boundary, before setup, mark `paperclip.github-star-prompt.seen` in the eval browser. An init script covers later documents of the app origin, and the open page is marked directly. A test checks that the key equals `STAR_PROMPT_SEEN_KEY` in `ui/src/components/GitHubStarPrompt.tsx`. - `tests/runner-e2e/chat-catalog-domains/api-read-receipts.ts`: bind dedicated task read tools by the digest of the `call_api` input that the server builds for them (`taskBridgeApiRequest`). The result must carry the same `apiOperationId`. Other tools, other records, mismatched receipts and unpaired results do not count. - Bump `oracleVersion` in code and descriptor for every case that uses `api-read-receipts.ts` (shared grader change) and for `navigation-preferences.edit-profile`. Add `internal-comment-reference.ts` to the source files of the two cases that now import it. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` (whole config): 11301 passed; 1 local-only failure in `controller-replacement-smoke.test.ts` (server spawn, no changed imports). - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts --testTimeout=180000 tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-returning-browser.test.ts` - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk` - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Red checks: with the `f4337bb5fb` registry, the new pin test fails and names 15 inputs. Without the new waits, the two rendered-reply tests fail. A regrade of the retained `explain-card` run events now gives the follow-up run `unavailable: true, live: false` and the first run `live: true`. - Paid runs of the affected cells (`chat-features.runner-opencode.local.*`): - Run 42852698879 (head `d54c04d5f0`): `track-delegated-work`, `restore-archive`, `explain-card`, `configure-plugin` and `plugin-managed-routine` pass. I read each reply and saved state. `track-delegated-work` now records the rendered `/COM/issues/COM-2` link. `explain-card` records `laterUnavailable: true, laterLive: false` from the follow-up `get_task` 404. - `plugins.install-plugin` now starts its server (the pin refusal is gone). It fails a later grading check, `truthful-installation-handoff`: the reply sends the person to the plugin settings page but does not name the package. This is outside this PR. - `inbox-search.archive-done`: the star prompt no longer blocks the composer, and the query is sent. The agent then archives the 501 Done tasks one at a time (about 2 s each, 57 in the 240 s window), because only the per-task archive route exists. This is outside this PR. - `navigation-preferences.edit-profile` did not start in that run (the AWS credentials step timed out). Run 42857020103 (head `2f0b80aa84`): pass. The agent write gets 401, the reply links `/COM/company/settings/instance/profile`, and the person follows the rendered link and saves the name. ## Risks - Low risk. Test harness only; no product code changes. - The receipt change counts more reads as record reads. It only adds dedicated reads bound to the exact record id and operation, so a grader that needs a read can pass on a genuine `get_task` read. The affected cases have new oracle versions. - The star prompt is not tested by any chat catalog case. Product e2e tests for the prompt do not use this flow. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
84007892e9 |
refactor: move AGENTS.md section detail to docs and rules, add a bug-fixing section (#20585)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` is the instruction file that each human and AI contributor reads before a change > - Pull request #20557 shortened the file, but several sections still held detail that applies to only some changes > - The scoped-rules section told agents to read each matching rule in full, and most changes match five rules > - The core engineering rules mixed three product invariants with process conventions > - This pull request moves that detail to the docs and to scoped rules, and adds a section on how to fix a bug > - The benefit is a 211-line file that states what applies to every change and points to the rest ## Linked Issues or Issue Description Refs #20557 **Issue type** Outdated or unclear contributor documentation. **Where is the issue?** `AGENTS.md` at the repository root, and `rules/`. **What's wrong?** - The scoped-rules section made agents read the full body of each matching rule before a change. The rule bodies total about 47 KB. - Sections 4, 6, 8, 9 and 10 repeated detail that belongs in `doc/DEVELOPING.md`, `doc/DATABASE.md`, a route rule, `DESIGN.md` and `CONTRIBUTING.md`. - Section 4 had a database reset command for `data/pglite`. That directory does not exist. - Section 5 had seven rules. Four of them were process conventions or path-specific detail. - The file did not say how to fix a bug. **Suggested fix** Make the rule headers the index. Move section detail to the document or rule that owns it. Keep only the three invariants in section 5. Add a bug-fixing section. ## What Changed - The scoped-rules section now tells agents to list the rule headers with one `grep` command. Agents read a rule body only for a design decision in its area, when the description is not sufficient, or when a reviewer cites it. - `rules/README.md` says that each `description` must state the required behavior in one sentence. - `rules/pull-request-review.mdc` has a description that states its requirements. - Section 4 (dev setup) and section 11 (pull requests) now point to `doc/DEVELOPING.md` and `CONTRIBUTING.md`. - The schema change workflow moved from section 6 to `doc/DATABASE.md`. - The API and auth expectations moved from section 8 to the new `rules/api-route-expectations.mdc`. - Section 9 links to `DESIGN.md` for design rules. `AGENTS.md` no longer tells contributors to run `pnpm check:token-gates`. - Added section 10, "Fixing a Bug": reproduce the error, write a failing test, write the fix, see the test pass, and verify the fix. - Section 5 now has three rules: company scope, contract synchronization, and the control-plane invariants. - The company-scope exceptions moved to the new `rules/company-scope-exceptions.mdc`. - Removed the strategic-docs rule. The plan-doc convention moved to the repo map. The artifact rule moved to section 7. - Removed rule 7. Five code comments that cited `AGENTS.md §5.7` now cite `rules/data-paths.mdc`. - Regenerated the SDK public source pin, because one of those comments is in `packages/shared/src/crash-points.ts`. - Added both new rules to `.greptile/files.json`. - The API route rule applies only to routes that read or write company data. Its status code list is a list of common codes. - The company-scope rule lists the sidebar company order as an instance-wide exception. - The `honeycomb-tracing` skill now points to `rules/data-paths.mdc` for the Telemetry workflow. ## Verification - Open `AGENTS.md` and confirm that each link resolves. A local script checked this. - Run `grep -E '^(description|globs|alwaysApply):' rules/*.mdc` and confirm that each rule prints a description. - Run `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`. It passes locally. - `packages/shared/src/crash-points.test.ts` passed locally before the rebase onto `master`. - Five source files changed in comments only. No other test suite ran locally. ## Risks - An agent that reads only rule descriptions can miss a detail in a rule body. Reviewers still apply the full rules. - Pull requests and the definition of done are now sections 11 and 12. Sections 1 to 9 keep their numbers. - Dated plan documents under `doc/plans/` still cite `AGENTS.md §5.7`. They are historical records. - Pull request #20565 also changes `AGENTS.md`. The second one to merge needs a conflict fix in section 1 and section 2. ## Model Used - Claude (Anthropic), in Claude Code with tool use. - `claude-fable-5-2`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d0c30e69a9 |
fix(storage): abort a snapshot upload a restart cut off once its save attempt is superseded (#20542)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save their files as encrypted snapshots in object
storage. Snapshots over 16 MiB use S3 multipart uploads.
> - A server restart can cut off a snapshot upload in the middle of an
`UploadPart` request.
> - The server aborts a multipart upload only in-process, on an error or
an abort signal. A SIGKILL can never do that. On SIGTERM, the drain step
timed out while the store held the part, and the process exited without
`AbortMultipartUpload`.
> - The incomplete upload then stays in the bucket. Nothing records its
upload id and nothing finds it again, so incomplete uploads pile up and
the operator pays for their parts.
> - This pull request aborts the incomplete uploads of each save attempt
when that attempt can no longer win, with no polling and no schema
change.
> - The benefit is that no incomplete upload outlives its save attempt.
## Linked Issues or Issue Description
**What happened?**
A restart-chaos wave (Daytona, R1 workload, storage fault ST4,
`--storage-trigger snapshot-part`, faults K1 and K2, master
`fea4530a30`) passed all 3 tasks. All 3 committed snapshots read back
intact. But the storage audit found 3 incomplete multipart uploads. Each
one was the exact upload that a restart cut off in the middle of
`UploadPart`. The retried save then published a different attempt of the
same snapshot.
**Expected behavior**
When a save attempt can no longer become the snapshot's object, its
incomplete multipart uploads are aborted. The storage audit shows 0
incomplete uploads after the wave.
**Steps to reproduce**
1. Run a snapshot save larger than 16 MiB against S3.
2. Kill the server (SIGKILL), or send SIGTERM while the store holds an
`UploadPart` request.
3. Let the save retry and publish.
4. List the bucket's multipart uploads. The cut-off attempt's upload is
still there.
**Paperclip version or commit**
master `fea4530a30` and later.
**Relevant logs or output**
`shutdown step timed out; continuing shutdown without it
{"step":"delivery_work_stop","durationMs":20001}`, then `Paperclip did
not stop within 30000ms; sending SIGKILL`.
## What Changed
- The S3 storage provider and the storage service get
`abortIncompleteUploads(objectKey)`. It lists the multipart uploads
under that exact key (`ListMultipartUploads` with the key as the
prefix), skips other keys that only share the prefix, and aborts each
one. `NoSuchUpload` counts as already done. The local-disk provider has
no multipart uploads and does nothing.
- Save publication: the save journals every attempt's object key in
`metadata.uploads` before any remote I/O. When one attempt moves the
snapshot from `pending` to `uploaded`, no other attempt can win any
more. The winning process now aborts the incomplete uploads of every
other journaled attempt key. Each abort is best effort and bounded to 5
s. A snapshot with one attempt sends no extra request.
- A losing uploader that is still live: if its upload fails after
another attempt has published the snapshot, it now returns the published
snapshot (after it verifies it) instead of the error. So an abort from
the winner never fails a save that is still running in another process.
- Snapshot retention: before it deletes a retired snapshot's objects
(this includes saves that #20502 retires as abandoned), retention aborts
the incomplete uploads of each attempt key. A store that refuses the
listing (for example a missing `s3:ListBucketMultipartUploads`
permission) logs a warning and does not block the deletion.
- Abandoned saves: when the ownership reconciler retires an abandoned
save (`remote_snapshot_save` and `remote_snapshot_ledger` settlement,
from #20502), it now aborts the incomplete uploads of every journaled
attempt key after its transaction commits. The owner run is terminal and
the save work is settled, so no live uploader can own those uploads.
This is the durable path for SIGKILL: the reconciler pass after boot
settles the dead save and cleans its uploads.
- SIGTERM drain: the multipart upload code keeps an in-process set of
running uploads. Shutdown aborts them right away when
`delivery_work_stop` times out. It also aborts them after the heartbeat
run drain, before the finalizer and workspace-recovery waits, after a 5
s grace for them to finish. It waits at most 10 s for the
`AbortMultipartUpload` calls. Shutdown does nothing extra when no upload
runs.
- No polling and no periodic sweep. All triggers are events that already
exist: publication, abandonment settlement, retention of a retired
snapshot, and shutdown.
Previous-release story (`rules/upgrade-safe-run-state.mdc`): this adds
no persisted state and changes no saved shape. A snapshot that the
previous release saved with cut-off attempts in `metadata.uploads`
carries forward. Its incomplete uploads are aborted when the new release
publishes a retry of that snapshot, or when retention retires the
snapshot.
## Verification
- Red before, green after (all fail on master, pass with this change):
- `server/src/services/remote-workspace-snapshot-storage.test.ts`:
"aborts the incomplete upload of an attempt a restart cut off once a
retried save publishes" and "adopts the published save when another
uploader aborts this attempt's upload".
- `server/src/services/remote-workspace-snapshot-retention.test.ts`:
"aborts each attempt's incomplete multipart upload before deleting a
retired snapshot's objects". A second test proves that a refused listing
does not block retention.
- `server/src/__tests__/storage-s3-multipart-shutdown.test.ts`: file and
streamed uploads whose part the store holds are aborted at shutdown, and
the provider aborts only the exact key across listing pages.
- `server/src/__tests__/remote-snapshot-ledger-settlement.test.ts`:
"aborts the incomplete multipart uploads of every attempt when the
reconciler retires an abandoned save" (both settlement kinds; a key
outside the snapshot's prefix is skipped).
- `tests/restart-chaos/storage-orphan-uploads.test.ts` (MinIO): the real
provider aborts both cut-off uploads of one attempt key, and keeps a
sibling attempt and a longer key that shares the prefix.
- `cd server && npx vitest run src/__tests__/storage-s3 src/storage
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/services/remote-workspace-snapshot
src/services/remote-workspace-transport.test.ts
src/__tests__/remote-workspace-file-resources.test.ts`: 19 files, 517
tests pass; with `src/services/recovery`, `src/__tests__/ownership*` and
`heartbeat-workspace-busy.test.ts`, 32 files and 580 tests pass.
- `cd server && npx tsc --noEmit` and `node
scripts/typecheck-test-projects.mjs tests/restart-chaos` pass.
- Live proof, Daytona R1, ST4, `--storage-trigger snapshot-part`, K1/K2,
seed 1033, the evidence wave's flags, image
`paperclip-daytona-runner@sha256:05a4a092…` (controllerMatch: match):
- Before (master `fea4530a30`): `w-20261110T130541-s1033`: 3/3 pass, **3
incomplete multipart uploads**.
- `w-20261110T134623-s1033` on an earlier revision of this PR (publish +
retention hooks only): 2 incomplete uploads remained. Those saves were
abandoned, not retried, so neither hook ran. This led to the settlement
hook.
- After (head `8bc8fc66d0`): `w-20261110T141144-s1033`: 3/3 tasks pass,
exit 0, every fired restart proved it cut the held `UploadPart`, the
reconciler settled 3 abandoned saves, shutdown logged one in-flight
abort, and the storage audit shows **0 incomplete multipart uploads**
and 3 committed snapshots that read back intact.
## Risks
- A live uploader in another process can lose its incomplete upload when
a different attempt publishes first. That uploader now adopts the
published snapshot, so its save still succeeds.
- Narrow gap: an uploader that recorded its attempt but had not yet sent
`CreateMultipartUpload` when the winner published, and that then crashes
during its upload, leaves an incomplete upload until retention retires
the snapshot. An S3 lifecycle rule for incomplete uploads stays a good
backstop.
- Stores without `ListMultipartUploads` permission log a warning and
keep the previous behavior.
- Shutdown can take up to 10 s longer, and only when an upload is still
running after the drain.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
10bc6431da |
test(restart-chaos): measure TTFT in a real repository workspace, after an idle restart, and on Daytona (#20047)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos TTFT driver measures time to first token, but only with a tiny fixture workspace and only locally > - Dotta asked whether speed is good with a workspace like the paperclip repository itself, and that had never been measured > - This pull request lets the driver run in a real repository workspace, measure the first follow-up after an idle restart, run on Daytona (per-turn or warm reusable sandboxes), and report each run's workspace operations > - The benefit is a repeatable large-workspace speed measurement; its first run found a full workspace re-upload on every remote turn ## Linked Issues or Issue Description **What happened?** The TTFT driver seeded only a tiny fixture repository, created tasks outside any project, ran only locally, and could not show workspace operation durations. So large-workspace speed (worktree creation, provision, remote upload) was never measured. **Steps to reproduce** 1. Run `ttft-baseline.ts` with any options. 2. Every task runs without a project workspace and on the local host. 3. Nothing in the report shows worktree, provision or sync cost. **Expected behavior** The driver can measure a real repository workspace, locally and on Daytona, and names the workspace operations each run waited on. ## What Changed - `--workspace-repo <path>`: the seed uses that repository as the project's primary workspace (git_worktree isolation), and tasks are created in the project. - A `postfault` phase: K1 while idle, then a follow-up on each task; reported in its own table. - `--env daytona`, `--daytona-image`, `--daytona-cpu`, `--daytona-warm` (the reusable warm Daytona fixture). The Daytona controller preparation moved to `daytona-controller.ts`, and every sandbox is cleaned up at the end; incomplete cleanup fails the run. - Each run's workspace operations (`/api/heartbeat-runs/:id/workspace-operations`) are saved, and printed ranked by duration. ## Verification - `tsc -p tests/restart-chaos`: no errors in harness files. - Used for the large-workspace measurement, a fresh paperclip clone on local and Daytona (per-turn and warm). First turn / follow-up: local 31–34 s / 7 s; Daytona per-turn 58–63 s / 30 s; Daytona warm 45–51 s / 8–10 s. Every Daytona sandbox was cleaned up (environment deleted, none remaining). ## Risks - Test harness only; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution, Daytona measurements). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
ce1ee9e89f |
fix: stop prompts from encouraging subtasks (#20362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents read runtime prompts, tool help, skills, and role instructions. > - Several of these texts tell agents to create child tasks as a normal next step. > - One accepted-plan prompt even forbids work on the current task. > - Task parentage is useful, but it does not need a prompt that encourages task creation. > - This pull request removes those prompts and keeps the task APIs available. ## Linked Issues or Issue Description Refs #10642. This change adjusts prompt wording; it does not add delegation-cycle guards. **Current behavior** Prompts tell agents to create children after plan approval, use children for long work, and create a task just to pass context. Onboarding requires a child even for a single request. **Proposed behavior** Continue work on the current task. Keep explicit assignments, optional task creation, parentage, dependencies, and approval gates. Use the separate-task recipes when the user requests that structure or when continuing existing work. **Reason and benefit** Reduce needless task trees without removing subtask support. Keep the patch limited to wording, existing test expectations, and generated catalog data. ## What Changed - Align accepted-plan prompts and remove direct child-creation advice from runtime and error text. - Make task tool descriptions neutral. Keep their fields, defaults, and handlers. - Update the main skill, onboarding, CEO instructions, company templates, specialist workflows, and matching guides. - Keep historical release snapshots intact. Refresh the explicitly seeded schema fixture's catalog digest; the server still verifies historical release snapshot hashes. - Refresh the affected generated catalogs, capability inventory, semantic contracts, and fixture manifest. ## Verification - Final tested head: `c6e304d0c105cdbfbe0b2ab0a233307fcd91a04b`. - All 66 check runs are complete: 60 pass, one is neutral, and five are skipped by CI conditions. Snyk also passes. No checks fail or remain pending. - [PR CI](https://github.com/paperclipai/paperclip/actions/runs/42846226554) passes typecheck, build, server/general tests, all eight E2E shards, Runner support, Runner Rust, and the aggregate verify gate. [Windows](https://github.com/paperclipai/paperclip/actions/runs/42846226546) and [Docker Runner](https://github.com/paperclipai/paperclip/actions/runs/42846226108) also pass. - [Greptile](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751419835) and [Paperclip review](https://github.com/paperclipai/paperclip/pull/20362#issuecomment-6751454010) give this exact head 5/5. All six inline threads are resolved. Neither current summary has findings. - Merged master `bf0220d34a156c81853755c3d1365ddc527037de`. Resolved three generated-file conflicts with the canonical generators. - Local verification passes: 473 focused prompt, continuation, UI, catalog, runtime and native-execution tests; Runner TypeScript build; generated metadata; UI token gates; and diff checks. - The prior head `c22018cb0f` failed three SDK evidence tests because two source hashes were stale. Reproduced all three failures locally. The existing generator refreshed only the hashes for the denial-copy file and its test. All 53 affected evidence/source-pin tests then passed, including private-value rejection. No scanner logic, assertions, or runtime dependency pins changed. Current CI confirms the support suite passes. - A prior commit failed the Rust process reaping test. Its cause remains unproven. No Rust code or assertions changed. Later Rust runs pass; this does not establish the cause of the earlier failure. - Local preflight built shared/plugin SDK packages, then stopped because `cargo` is absent. Current-head CI supplies Rust and full build coverage. `pnpm verify --explain` selects broad required CI; full local suites were not repeated. ## Risks - Wording changes guide the model; they do not enforce a ban on subtasks. - Existing installed skill versions and saved custom instructions need their normal refresh process. This PR does not migrate company configuration. - Existing approved task assignments and approval gates remain in effect. ## Model Used - OpenAI GPT-6, through Codex, with reasoning, tool use, and code execution. The exact routed model ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
d0057ee384 |
test(restart-chaos): fire a planned restart at native start (#20538)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness (`tests/restart-chaos`) restarts the
server during real agent runs and grades the recovery.
> - Several native recovery fixes only apply when a restart lands after
the run has started its native runner. Examples: a restart claim whose
lease lapses in a storage outage (#20529), and a K2 seconds after native
start on a remote sandbox.
> - On Daytona R1, workspace preparation takes 100-250 s and varies a
lot under storage faults. A time-based fault and `--storage-trigger
run-log` both land while the run is still preparing. In three waves,
none reached native start.
> - This pull request adds `--native-trigger`, which fires the planned
fault a set delay after the run's own `native.process_start_requested`
event or first provider event.
> - The benefit is that a wave can reliably restart the server at native
start and prove those recovery paths live.
## Linked Issues or Issue Description
Refs #20529, Refs #20514
**What happened?**
Waves `w-20261110T113240-s11936`, `w-20261110T120901-s11937` and
`w-20261110T124555-s11936` (Daytona R1, K2) were meant to restart the
server after native start. Every K2 landed during remote workspace
preparation, before native runtime selection. The drain took the legacy
interrupt path, so the native recovery path under test never ran.
**Expected behavior**
A wave can fire its planned fault a known number of seconds after the
run starts its native runner.
**Steps to reproduce**
1. Run a Daytona R1 wave with `--faults K2 --fault-after-seconds 110`,
optionally with `--storage-faults ST1`.
2. Read `server.log`. The graceful drain lists the runs under
`interruptedRunIds`, not `restartSuspendedRunIds`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
8d2bf78323 |
test(runner-e2e): resync the SDK public source pins with master (#20550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite checks that the plugin SDK's public types come only from reviewed source inputs, pinned by git blob hash in `tests/runner-e2e/source-sdk-public-inputs.json` > - #19614 landed those pins, generated against an older master; since then `package.json` and ten files under `packages/shared/src` changed > - So `source-sdk-public-type.test.ts` (2 tests) and `source-sdk-evidence.test.ts` (1 test) fail the `ci / Runner support suite` check on every PR merge ref > - This pull request regenerates the source pins with the script made for that job > - The benefit is a green Runner support suite for every open PR again ## Linked Issues or Issue Description **What happened?** Since #19614 merged, `ci / Runner support suite` fails on every PR (for example #20047 and #20538) with `SDK public source proof unavailable` and `Source SDK evidence privacy check failed`. **Steps to reproduce** 1. On master at |
||
|
|
f4337bb5fb |
test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7329eaf353 |
perf(ui): open sidebar Recents tasks from cache (#20450)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI sidebar has a Recents list of the tasks a person opened
last
> - The Recents list renders from a local snapshot, but its links had no
warm-up. A click waited for the task-detail code and about nine
first-frame reads
> - Task lists (Inbox, Tasks) already warm a task on hover, press and
click, and #20355 added a batched `chat-bootstrap` read that had no UI
consumer
> - The hosted database is CPU-bound, so a fix must add at most one read
per page load and must not compete with the page's own reads
> - This pull request gives Recents the same open-intent prefetch as
task lists, warms the task-detail code, and preloads the top ten recent
tasks with one `chat-bootstrap` request after the page is idle
> - The benefit is that a click on a recent task shows its chat from
cache, without a wait for the API
## Linked Issues or Issue Description
Refs #20355 (server read this PR consumes). Builds on #20389, #20398 and
#20419.
**Is your feature request related to a problem?**
Opening a task from the sidebar's Recents was slower than opening it
from a task list. The Recents links had no prefetch. The task page
waited for its first-frame reads after the click.
**Describe the solution you'd like**
Recents links use the task lists' open-intent prefetch. When the page is
idle, one batched request loads the first frame of the most recent tasks
into the task page's caches.
**Describe alternatives you've considered**
Per-task prefetch of every recent task. That makes up to ten times nine
requests per page load, which the hosted database cannot afford.
**Additional context**
I searched open PRs and remote branches for "chat-bootstrap" and
"instant task opens". The only related work is the server-side branches
that merged as #20355. There is no other in-flight UI consumer of
`chat-bootstrap`.
## What Changed
- `ui/src/hooks/useIssueNavigationPrefetch.ts`: the open-intent handlers
from `IssueLinkQuicklook` move into one hook (light warm on pass-over;
`history: true` on a resting hover or focus, a mouse or pen press, and a
click). `IssueLinkQuicklook` now uses the hook. Its behavior does not
change.
- `ui/src/components/RecentTaskNavItem.tsx`: one Recents link wrapper
that applies the hook. The classic list (`SidebarRecentTasks`), the
Coordinator task rows and `DesignSidebar` use it. `SidebarNavItem`
accepts the handlers through a new optional `linkHandlers` prop.
- Classic Recents links now open `/issues/<identifier>` (Coordinator
already did). The old UUID link remounted the task page after a redirect
to the identifier route.
- `ui/src/hooks/useRecentTaskOpenWarmup.ts`: when Recents shows a task,
it fires the task-detail code warm-up (`useWarmTaskDetailCode`). After
two frames, a visible tab, no fetching queries and an idle callback, it
sends one `issues/chat-bootstrap?ids=` request for the ten most recent
tasks. It skips tasks whose first frame is cached and fresh. It sends no
request for an empty list or a hidden tab. It sends at most one request
per account per page load. Unmount or an account change aborts the
request.
- `ui/src/lib/issueChatBootstrapCache.ts`: `seedIssueChatBootstrap`
writes each field under the keys the task page reads, for the requested
ref, the id and the identifier: core issue, recent comments, comment
input facts (one batch, through the new `seedIssueIdBatch`), recent
activity, the recent run window (through the new shared
`buildRecentIssueRunWindow`), interactions, attachments, live runs and
active run. An explicit `null` active run is stored. A missing field is
not stored. It does not replace data that is as new or newer, data with
a read in flight, or data with a pending invalidation. Keys whose page
merges observed history (comments, activity, runs, input facts) are
filled only when empty. Items for another company are ignored.
- The seed fills the `?view=core` key, not the full detail key.
`seedIssueDetailCache` is not used: the core payload cannot stand in for
the detail read's related data, and seeding it there would suppress that
read.
- `issuesApi.chatBootstrap` client method.
- `tests/perf/collections/recent-task-open.spec.ts` and its README
section.
## Verification
Local disposable instance (`local_trusted`, production UI build,
embedded PostgreSQL, collection fixture plus 12 comments on each of 8
recent tasks, streamlined UI with the Coordinator sidebar). Journey:
open Projects with 8 Recents, wait for network idle plus 3 s, click a
recent task, measure until its latest comment is on screen. Each run
uses a fresh browser context (empty HTTP cache and storage apart from
Recents). Median of 5 runs, same server and fixture for both arms:
| Arm | Click to first chat frame | Idle page API requests |
| --- | --- | --- |
| `master` (
|
||
|
DottaandClaude Opus 5.5
|
15a734d068 |
fix(heartbeat): show a storage outage wait as live status, then record it once (#20460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In cloud mode a run reads and writes file storage on its critical path: it starts its run log (`s3_segments`), restores its saved workspace, and saves it again > - Since #20299 a run waits out a storage outage there for up to 5 minutes instead of failing > - That wait was silent. The board showed "running" with no sign of life, so a storage wait looked like a dead run (restart-chaos R5 `silent_dead_run`) > - The first version of this PR wrote a run event on every wait. That writes the run row (`next_event_seq`, `updated_at`) while the run-log claim is still in progress, and it covered only the run-log begin > - This PR shows the wait as live run status instead, writes nothing during the wait, and records one lifecycle event when the wait ends. It covers every critical-path wait in the run's storage scope > - The benefit is that a storage outage shows on the board as waiting, the run's history keeps one honest line about it, and the claim path is left alone ## Linked Issues or Issue Description Refs #20299 (wait out a store outage on a run's critical path). **What happened?** - `w-20261110T043533-s11936` (Daytona R1, ST1 194 s, K2): run `d49f58d8` waited 186 s in `heartbeat.prepare.run_output` with no event. The grader failed it as `R5:silent_dead_run`. - `w-20261110T105903-s11936`: the restore download (`storage.snapshot.download`) waited 182 s with no sign of life. **Expected:** the board shows the run waiting for file storage, and nothing about the notice can disturb run-log ownership. **On the failed `run_output` seen on builds of the first version** (`w-20261110T101517-s11936` t01 run `9d39550a`, `w-20261110T081647-s11936` t02 run `9da10424`): the evidence points to an expired restart lease, not to the notice's row writes. Neither the writer-authority check (`run-log-authority.ts`) nor the journal claim compares `next_event_seq` or `updated_at`. A remote `bootstrap_incomplete` restart claim gets a 60 s lease (`REMOTE_CONTROLLER_LEASE_MS`), and preparation renews it only for `reattach_remote_runner` claims. In t01 the claim was at 10:19:55, the outage ended at 10:22:18.15, and `run_output` failed at 10:22:18.17, on the first authority check after the store came back. That fix ships separately as its own PR. ## What Changed - `storage/storage-unavailable.ts`: - The run's storage scope can carry a wait observer (`reportStorageOutageWaitsTo`). - `retryThroughStorageOutage` tells it about each backoff (`waiting`). The operation that owns the wait reports one `ended` (`recovered` true or false), so nested waits do not double-report. - `storageOutageWaitFor(purpose, work)` names the step. - An observer that throws never changes the wait. `endStorageOutageWaitOn` keeps the observer. - `heartbeat/storage-wait-notice.ts` (new): - Each wait sets live runtime status (phase `waiting_for_storage`, "Waiting for file storage before starting the run log (45s so far)") and publishes `heartbeat.run.progress`. Nothing goes to the database. - When the wait ends, it writes one `lifecycle` warn event: "Waited Ns for file storage before …", with payload `reason: "storage_outage_wait"`, `purpose`, `waitedMs` and `recovered`. - `heartbeat.ts`: installs the notice next to `endStorageOutageWaitOn` (one line). - Purposes: - run-log begin: "starting the run log"; - `loadWorkspaceSnapshotArchive`: "restoring the workspace"; - `uploadWorkspaceSnapshot`: "saving the workspace". - `packages/shared`: adds `waiting_for_storage` to `HeartbeatRunStatusPhase`. - `tests/restart-chaos/live-status-evidence.ts` (new): R5 also counts the board's live run status (`/api/issues/:id/live-runs` `currentStatusUpdatedAt` / `lastEventAt`) as evidence that a running run is alive. Wiring it into `observer.ts` takes one import and one changed line. - `doc/run-log-events.md`: documents the live status and the one event. - Removed: the first version's per-wait run event. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-output.test.ts src/__tests__/storage-outage-wait.test.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/object-byte-journal.test.ts src/__tests__/storage-s3-credential-rotation.test.ts`: 103 pass. - Red/green: with the observer disabled (master behaviour), these new tests fail. With this change they pass: - run-output "shows the wait as live status and writes nothing to the run while its log claim waits": no run-row write and no event during the claim's waits; the live status is `waiting_for_storage`; the claim's writer-authority check still passes; one lifecycle event afterwards; - the helper's observer tests (one end per wait; gave-up reporting; a throwing observer; the Stop signal keeps the observer); - snapshot storage "tells the run it is waiting for storage while a save or a restore waits out an outage". - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/live-status-evidence.test.ts tests/restart-chaos/observer.test.ts tests/restart-chaos/triggers.test.ts`: 65 pass. `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. `cd server && npx tsc --noEmit`: pass. - Live restart-chaos waves: results in a comment below. ## Risks - Low. - The retry policy, its budget and Stop handling are unchanged. - During a wait, the only new effect is an in-memory status and a live event. - Afterwards, one event per wait, written without blocking the run. - Runtime status is per server process. A board on another instance sees the waiting line only after the closing event. - Shared journal work (`runSharedStorageWork`) keeps its own scope without the observer. Its waits surface through the enclosing run-scoped wait, as before. - Previous-release state: this persists no new run or session state. A run saved by the previous release resumes unchanged. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
c874cfa4da |
test(restart-chaos): anchor two-server speed samples on the primary's restart (#20535)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed, including how fast runs recover after a server restart > - Two-server faults (MS1–MS3, #20518) record their `readyAt` after the second server stops > - The speed grade used that time as a restart-recovery anchor, also for faults that never restart the primary > - Wave `w-20261110T115048-s302` passed every task but exited 1 on four such samples > - This pull request anchors two-server samples on the primary's own restart > - The benefit is that a two-server wave fails speed only for a real recovery delay ## Linked Issues or Issue Description Refs #20518 (the two-server faults). **What happened?** In wave `w-20261110T115048-s302` (Codex, B5, MS2/MS3, 4 tasks) every task passed, but the speed grade counted 4 single-run breaches on `restart_recovery_control_plane` (p50 10.9 s, p95 12.8 s, budget p95 4.0 s), one per fault. - MS2 never restarts the primary (it freezes and thaws it), yet it produced a restart-recovery sample. - MS3's sample started at the fault's `readyAt`, after the second server's 33–43 s tail. The primary's replacement had been ready since 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier. - So each sample measured the B5 run's next provider event after the second server stopped, while `bench.sh` ran quietly. **Expected behavior** MS1 and MS2 produce no restart-recovery sample. MS3's sample starts when the primary's replacement is ready. ## What Changed - `wave-speed.ts`: - New `primaryRestartReadyAt(fault)`: a non-MS fault keeps its `readyAt`; MS1 gives none; MS2 gives none, unless the primary exited after the thaw and was replaced (then the replacement's `healthyAt`); MS3 gives `multiServer.primaryRestart.readyAt`. - `faultRuns` (recovery samples and the retry "restart" phase) uses that anchor and skips faults without one. - `outageWindows` removes only the time the primary could not serve: none for MS1, the freeze for MS2, and for MS3 from the second server's readiness (when the primary is told to stop) to the replacement's readiness. Before, the second server's whole window was removed from run timings. - `collectWaveSpeed` sets a run's restart phase only from faults that restarted the primary. A freeze still subtracts its downtime, but a run it hit before its first activity stays in its cold or warm budget (review finding). - `wave-speed.test.ts`: six tests built from s302's saved receipts, including two `collectWaveSpeed` cases (a run frozen before its first activity; a retry after an MS2 freeze). - README: one note on how the speed grade treats MS faults. Test-only. No longer freeze and no lease change, as agreed. ## Verification - Red against master `wave-speed.ts`: all 6 new tests fail. Green with the change. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 480 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Offline re-grade of s302 with the fix, from its saved `faults.json` and run timings (`t0*.grade.json`): - Restart-recovery samples go from 4 to 2. The MS2 samples (f01, f03) are gone. - The two MS3 samples are anchored at 1794311949160 (f02) and 1794312728179 (f04), 43.4 s and 33.6 s earlier than before. - Outage removed from run timings: 110.0 s and 113.3 s for the MS2 freezes, 5.6 s and 6.5 s for the MS3 restarts. - No s302 run was hit before its first activity (each freeze came about 68 s into its run), so no run changes phase. - The MS3 latency itself needs the run's event list, which a wave does not save for passing tasks. The observer's saved samples show the run's first recorded activity 0.5 s (f02) and 1.6 s (f04) after the replacement was ready. The next provider event comes at or after that, so this is a lower bound, not the graded value. - `restart_recovery_control_plane` has n = 2 < 5, so its row reports "insufficient sample". The wave's speed verdict now depends only on whether each MS3 sample is under 4.0 s, which this offline data cannot confirm. - No new wave (as agreed). ## Risks - Low. Test-harness code only. - After MS1/MS2 a retry of a failed run is graded as an ordinary warm run, since the primary never restarted. - MS1 overlap and the MS2 thaw tail no longer count as outage, so run timings in those windows are graded in full. The primary served during them. - A primary replaced after an MS2 thaw is graded like any restart. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
be34a2d9d2 |
test(restart-chaos): two servers on one database (MS1-MS3) and a cross-server invariant (#20518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) tests how runs survive control-plane faults > - Every wave runs one server, so no wave can reach a race between two servers on one database > - Hosted Cloud deploys by rolling overlap: a new server's startup recovery and sweeps run beside the old server's live runs > - #20508 fixed such a race (another server's orphan sweep failing a run whose owner took its lease back) with a database test only > - This pull request adds three two-server faults (MS1–MS3) and a cross-server invariant > - The benefit is that cross-server ownership races can be reproduced and graded in a live wave ## Linked Issues or Issue Description Refs #20508 (the race this harness can now reach live). **What is missing?** The harness starts one control plane per wave. A fault replaces or freezes that one server. Nothing starts a second server on the same database while the first still owns runs, so cross-server ownership bugs are found only by database tests. **How production runs (checked before choosing the scenarios)** - Hosted Cloud runs on Railway with Neon Postgres. A deploy is a rolling A/B overlap: B starts on the same database, A serves until B is ready, then A drains. Defaults are a 5 s overlap and up to 90 s of drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`, lines 374–375). The chaos plan names the same shape as K12 (`doc/plans/2026-11-06-control-plane-restart-chaos.md`, line 103). It also records B's startup recovery hitting a lock on a run A still owned (lines 229–230). - This repository has no replica count or deploy strategy of its own. `docker/ecs-task-definition.json` describes one task, and the real Cloud deploy lives in another repository. The self-hosting docs do not describe several servers on one database. - Server-side ownership: per-boot controller lease of 90 s (`server/src/services/controller-instance-lease.ts`), legacy run lease of 60 s, native executor lease of 20 min, and the orphan sweep in `server/src/services/heartbeat/recovery.ts`. So MS1 models B's overlap beside a live A, MS2 models A paused (for example a host sleep) past its controller lease while B is up, and MS3 models the rolling upgrade itself. ## What Changed - New `tests/restart-chaos/multi-server.ts`: - MS1: B boots on the same database, instance directory, storage and secrets, on its own port. It stays up 60–150 s beside a live A, then gets SIGTERM. - MS2: B boots. A and its tree are frozen (TM2's freeze without aging) for 100–150 s, past the 90 s controller lease, then thawed. B stays up 30 s more, then stops. - MS3: B boots, then A gets K2 and a replacement on A's port. B stops 20–45 s after the replacement is ready. - B is always stopped once started, also when a step fails. - `checkCrossServer`, graded once per wave with an MS fault, over the wave database and the server process windows (pid, start, exit, freezes): - `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while its owner was alive and not frozen (#20508's shape included). The owner's own lease loss counts only without a prior freeze past the lease. - `MS_double_executor`: a run recorded a second runner identity before the first was shown stopped. - Unattributable ownership failures are listed, not charged. - `supervisor.ts`: records every server's process window and freezes, spawns and stops B (`<wave>/server-b.log`), and runs the MS steps. TM freezes are recorded too. - `types.ts`, `scheduler.ts`, `journey-matrix.ts`, `config.ts`: MS1–MS3 fault kinds, seeded holds, and validation (local only; no storm, crash point or K14). - `main.ts`: writes `<wave>/cross-server.json` and fails the wave on a violation. - README section "Two servers, one database (MS1–MS3)", with the production evidence and the limits. Test-only. No `package.json` or `.github` changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 473 tests pass. - New `multi-server.test.ts` (14 tests): option validation and seeded holds; MS1/MS2/MS3 step order and timing with fake hooks; B stopped when a step fails; the invariant for a live owner, a frozen or exited owner, #20508's thaw-then-write shape, lease loss with and without a freeze past the lease, unattributed owners, and double executors. - New `multi-server-db.test.ts` (embedded Postgres): a run still `running` with `native_execution_ownership_unverified` (#20508's marker) is read at its write time and charged as `MS_foreign_failure`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave `w-20261110T105448-s301` on head `a9ec5a4502` (master product code + this harness), `--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3`: 3 of 3 pass, all crossed their fault. B booted on the shared database in about 10 s each time. MS2 froze A for 104 s; A stayed up after the thaw. Cross-server invariant: 0 violations, 0 unattributed. ## Risks - Test-harness code only. - B runs on the same host and PID namespace as A, so it can check A's PIDs directly, which a Railway replica cannot. These faults are a lower bound on cross-server exposure. The README says so. - MS2 outlasts the 90 s controller lease, not the 20 min native executor lease. - MS faults refuse TM2/TM3 in the same wave, and fail when B exits before its planned stop. - `MS_foreign_failure` uses the run's `finished_at` (or `updated_at` for a still-running row) and the owner recorded on the run. A run whose owner changed after the failure could be misattributed; the evidence file keeps the owner and windows for review. - `MS_double_executor` reads runner identity events; it cannot see a second executor that never recorded one. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
16c30ac32b |
fix(runner): relaunch a provider whose startup only timed out without a reply (#20521)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through `paperclip-runnerd`. After a restart, a new runner relaunches the provider (for example the Codex app server). A durable startup fence stops a second runner from launching a provider while an unadmitted one may still run. > - #19784 lets a new runner retire a fence whose recorded provider process and group have provably exited. That rule covers a launch that a kill cut off. It deliberately keeps a failed initialization closed, because that can be a real provider failure. > - A relaunch whose reply never came within the runner's 30 s request deadline is saved as the same failed initialization. A stalled or frozen host causes this. The restart-chaos fault TM2 SIGSTOPs the runner and Codex, and their monotonic clocks keep running while they are stopped. > - Every later runner then refused to restore with "provider startup ownership remains unadmitted", and the user had to click Retry. The owner's rule is that a run interrupted by a restart or sleep must resume by itself. > - This pull request records whether initialization failed only because the provider gave no reply, and lets a later runner retire that fence once the provider is provably gone. > - The benefit is that a stall during a provider relaunch continues automatically, while a real provider rejection still fails as before. ## Linked Issues or Issue Description Refs #19784 **What happened?** Restart-chaos wave `w-20261110T091152-s426`, task t03 (Codex, B5), run `c0bbce18-836b-43af-8195-92ebe3e21cca`. A TM3 restart relaunched the Codex provider. A TM2 freeze stopped the tree 1 s later for 45 s, and a TM3 kill followed the thaw. The next three runners each failed `runner.drain` with "provider startup ownership remains unadmitted". Recovery stopped with `native_session_retry_exhausted`, and the task needed Retry. **What did you expect?** The new runner relaunches the provider and the run continues. **Steps to reproduce** The new test `restart_relaunches_a_provider_whose_relaunch_timed_out_without_a_reply` (runner-core `tests/codex_provider.rs`). The fake Codex stalls its first `thread/resume` reply past the 30 s deadline. On `origin/master`, the saved attempt is `initialization_failed` / `thread_open`, and the next runner fails with "provider startup ownership remains unadmitted". ## What Changed Design: - The deadline that passes is the runner's own 30 s request deadline. When a startup request (`initialize`, `thread/start`/`thread/resume`, `thread/read`) reaches that deadline with the provider still running and silent, the runner sets `noReply: true` on the durable `initialization_failed` startup record. - Not `noReply`: a provider error response, a provider that exited before it replied, or closed stdout. - `retire_orphaned_startup_attempt` retires an `initialization_failed` fence only when `noReply` is set and the recorded process and its process group are both gone (`ESRCH`). This is the same proof as `spawned` in #19784. A rejection, an exit, a live process or group, and spawn or admission failures stay fenced. - Validation accepts `noReply` only on `initialization_failed` at the `initialize`, `thread_open` or `thread_read` stage. - Bounded: a runner restores once (`restore_checked` / `restore_error`), so the loop is one relaunch per recovery attempt. Recovery's existing resume budget (`native_session_retry_exhausted`, seen after three attempts in the wave) stops a provider that never replies. - Upgrade: `noReply` is `#[serde(default, skip_serializing_if = "Not::not")]`. A new runner reads state saved by master as `noReply: false`, so that state keeps today's behaviour and stays fenced. - Rollback: an old runner (`deny_unknown_fields`) rejects state that holds a `noReply` fence and fails closed. That run needs Retry, which is today's outcome for the same fence. The field is written only while a timed-out fence is present, and admission clears it, so all other state is unchanged for old runners. - Normal path: nothing is added before `turn.start`. The cost is one in-memory flag per request; no extra I/O. Files: - `runner-core/src/codex_provider.rs`: `last_request_got_no_reply` and the `no_reply` field on `ProviderStartupObservation::Failed`. - `runner-core/src/provider_backend.rs`: the `noReply` field, its validation, and the retirement rule, with unit tests: - a no-reply fence is retired at `initialize`, `thread_open` and `thread_read`; - it is kept while the provider may still run; - a rejection or a state saved without the field stays fenced. - `runner-core/src/bin/fake-codex-app-server.rs`: the test-only switch `--stall-first-thread-resume-ms`. - `runner-core/tests/codex_provider.rs`: the end-to-end red/green test. - `tests/restart-chaos/README.md` and `time-faults.ts`: correct the TM2 claim. A stopped process's monotonic clock keeps running, so TM2 is also a long process stall. ## Verification - `cargo test -p paperclip-runner-core --test codex_provider restart_relaunches_a_provider_whose_relaunch_timed_out` is red on `origin/master` (the next runner refuses restore) and green here, in about 31 s. - `cargo test -p paperclip-runner-core`: 1080 passed, 0 failed. This includes `failed_autonomous_restore_and_rollover_keep_their_exact_startup_origin`, where a real "no rollout found" rejection still never restarts. - `cargo fmt -p paperclip-runner-core -- --check` is clean. - A restart-chaos proof wave on seed 426 will follow in a comment. ## Risks - A provider that hangs for good is relaunched once per recovery attempt until the existing resume budget stops it. Before, it failed on the first attempt. This costs at most a few extra 30 s attempts, and they are bounded. - Rollback to an older runner while a `noReply` fence exists fails closed for that one run, which needs Retry, as it does today. - No database, server or API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, cargo and test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
c2423e30e8 |
test(restart-chaos): provider faults across K2/K4 restarts, per-workload message, per-run speed split (#20515)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos`) proves that runs recover when the control plane restarts > - PR #20494 fixed a saved governed wait whose provider terminal was late or lost across a restart. The harness proved it only with a K1 (kill -9) restart > - The harness could not restart with K2 (graceful SIGTERM) or K4 (outage hold) after a provider fault, so the graceful shutdown drain was not tested > - S1 needs the `interrupt_ack` message and Q1 needs `terminal`, but one wave could set only one message > - G1 was listed as a provider-fault workload, but its agent ends its own turn. No rule ever applies, and each G1 task waits until its deadline > - Speed breaches could not be explained after a wave, because the instance database is deleted > - This pull request adds K2/K4 provider-fault restarts, a message for each workload, removes G1, and keeps a speed split for each run > - The benefit is that one wave can test #20494 across graceful and held restarts. Also, a cold-start breach now shows whether Paperclip or the provider was slow ## Linked Issues or Issue Description Refs #20494 **What happened?** The harness accepted only `--provider-fault-restart K1`. A wave with `--workloads G1` and a provider fault planned tasks that could not apply the fault. Waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211 show `applied=0` for G1 with Codex and Claude. The board then waited for a restart that did not come. **What did you expect?** A provider fault can cross every restart kind that the harness supports. The harness rejects a workload that does not interrupt a provider turn. **Steps to reproduce** Run `--adapters codex --workloads G1 --faults PV1 --provider-fault-restart K1 --concurrency 1`. The Codex shim logs `bound` and then `exited` with code 0. It logs no `interrupt_requested`. The task fails `R1:no_rest_within_budget`. ## What Changed - `--provider-fault-restart` accepts K1, K2 and K4. A comma list (for example `K4,K4,K2,K2`) cycles across PV1/PV2 tasks in plan order. K4 holds the control plane down for 120 s. - `--provider-fault-message` accepts `WORKLOAD=message` entries next to the default, for example `S1=interrupt_ack,terminal`. - G1 is removed from `PROVIDER_FAULT_WORKLOADS`. The config check that told G1 to add a restart is removed, because G1 is now rejected. - `speed.json` keeps a `runs` list. Each entry has `bootMs` (Paperclip share), `queueMs`, `providerFirstActivityMs` (turn accepted to first activity) and the pre-turn spans. - README documents the new flags, the G1 limit, and the speed split. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 434 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave w-20261110T092735-s213 (Codex, seed 213, concurrency 1). All 4 tasks pass R1–R6, T and `provider_fault_applied`: - t01 S1 PV2 drop `interrupt_ack` + K4 (120 s hold) - t02 Q1 PV1 delay `terminal` 18 s + K4 (120 s hold) - t03 S1 PV1 delay `interrupt_ack` 18 s + K2 - t04 Q1 PV2 drop `terminal` + K2 - That wave wrote the new `runs` split in `speed.json`. Cold boot was 1.3–1.7 s and provider first activity was 2.0–2.6 s. ## Risks - Low risk. Only the test harness changes. Product code does not change. - Old `plan.json` files keep `providerFaultRestart: "K1"`. That value is still valid, so replays of old waves do not change. - A wave that used G1 with provider faults now fails at parse time. Before, it ran and failed at the deadline. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
7964433cd0 |
test(restart-chaos): arm a provider-fault Stop on the first live provider turn (#20512)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) injects provider message faults (PV1–PV3) to test recovery > - A board Stop task (S1) with a PV `interrupt_ack` fault needs the Stop to reach a live provider turn > - The harness clicked Stop only after a seeded wall-clock window, so a fast agent finished first and nothing was injected > - That task then failed `provider_fault_applied`, which made a clean wave exit 1 with no product failure > - This pull request arms the Stop on the first live provider turn for PV tasks > - The benefit is that PV S1 tasks inject their fault every time, so a wave result means what it says ## Linked Issues or Issue Description Refs #20498 (the product fix whose live waves showed this harness gap). **What happened?** In waves `w-20261110T082702-s212` and `w-20261110T091057-s213` (Claude, `--workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack`), task t03 (S1) failed only `provider_fault_applied` with `applied=0`. Its run finished before the seeded Stop window, so the Stop was missed and the fault never fired. Every product rule (R1–R6, U1) passed. The wave still exited 1. **Expected behavior** A PV S1 task injects its fault on every run. A missed injection is a harness gap, not a product failure. **Steps to reproduce** Run `--adapters claude --workloads Q1,S1 --faults PV2 --provider-fault-message interrupt_ack --concurrency 1 --tasks 4`. The second S1 task often finishes before its Stop window. ## What Changed - `observer.ts`: the run-event cursor now tracks a live provider turn (`turn.started` with no `turn.completed/failed/cancelled/interrupted` yet). Run samples expose it as `providerTurnActive`. - New `stopArmed()`: a task with a provider fault arms its Stop on the first poll that shows a live provider turn. Other tasks keep the seeded wall-clock window, unchanged. - A runner replacement or stop clears the live-turn mark, as it already clears open tools. - A PV Stop still counts as missed when the run ends with no live turn seen and the task rests after the window. - The continue comment still waits for the seeded window and `STOP_CONTINUE_AFTER_MS`. - README note under the provider fault section. No product code changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. - New tests in `observer.test.ts`: - `S1 with a provider fault: stops on the first live provider turn, not the wall-clock window` - `S1 with a provider fault: does not stop after a turn terminal, and stops once when the next turn starts` - `S1 with a provider fault: records a missed Stop when the run ends before any live turn and the task rests after the window` - `noteLivenessEvent: a runner replacement or stop clears the live provider turn` - `stopArmed: a provider-fault Stop needs a live provider turn; others keep the window` - Red against master `observer.ts`: the first three new tests fail. Green with the change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - No live wave was run (the wave queue was full); the coordinator asked for a test-only change. ## Risks - Low. This is test-harness code only. - A PV S1 Stop can now land earlier in the turn than before. That is the intended test point for `interrupt_ack`. - A turn shorter than the poll interval (`--poll-ms`, 2–5 s) can still be missed. The README states this limit. - If a provider never records `turn.started`, the PV Stop waits until the task rests and is then recorded as missed, the same result as before. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
1a89fca1ce |
test(restart-chaos): add --rr-stop-on recovering so RR follow-up stops land inside recovery (#20511)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - #20480 added repeated restarts (RR1–RR3): a K1, then more restarts while the replacement recovers. > - In the first four live waves, no follow-up stop landed inside recovery. Every stop saw `/api/ready` report `ready`. > - The saved receipts show why. `/api/ready` reports `recovering` only 15–80 ms before `/api/health` answers, and `ready` 0.25–0.5 s after it. A gap measured from health, even 1 s, always lands after recovery. > - This pull request adds `--rr-stop-on recovering`. Each follow-up stop fires on the first `recovering` status, polled from the replacement's spawn. > - The benefit is that RR waves now test a restart that interrupts recovery itself. ## Linked Issues or Issue Description Refs #20480 **What is the problem?** RR follow-up stops wait a gap from the replacement's health. On a local host recovery ends about 0.3 s after health, so no stop in four waves (14 tasks, 18 follow-up stops) landed inside recovery. **What is the solution?** Add `--rr-stop-on recovering`. It replaces the gap: the harness polls `/api/ready` every 50 ms from spawn and stops the replacement on its first `recovering` status. This is the phase poll that K6 already uses. **Why is it needed?** The deploy loops of 2026-11-08 and 2026-11-09 restarted the server while it recovered interrupted runs. The harness must reach that window to test it. ## What Changed - `repeated-restarts.ts`: `--rr-stop-on recovering` is parsed by `parseRepeatedRestartOptions`. It needs an RR fault and does not combine with `--rr-gap-seconds` or `--replay`. `runRepeatedRestarts` takes `stopOn`: with it, no gap is slept, each stop waits on the new `waitForStopPhase` hook, and only the last replacement is waited on to health. - `supervisor.ts`: `waitForStopPhase` reuses the K6 phase poll (`waitForCrashLoopPhase`, 50 ms). The record carries `stopOn`. - `types.ts`, `config.ts`, `main.ts`: the `rrStopOn` option, its flag, and one line to pass it to the supervisor. - The evidence shows each stop's `/api/ready` status (`phaseAtStop`). With `stopOn`, `gapMs` is null, and the earlier entries have no `healthyAt`. - README: the option and the measured timings. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 438 tests pass. New tests cover option parsing, stops on the first `recovering` with no sleep, health waits only on the last replacement, a `ready` stop when recovery was too short to see, an exit before a status, and a missing watcher. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - A live wave from this branch (`--workloads A1,B5,C1 --faults RR1,RR3 --rr-stop-on recovering`, codex, 4 tasks) follows; its results go to a comment. ## Risks - Harness-only change; no product code changes. RR waves without the flag are unchanged. - A stop on `recovering` lands before the replacement answers health. This is the K6 behavior, and the server log shows it. A replacement that never reports a status fails the fault after the 5-minute ready timeout, as K6 does. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7eae4bc2ea |
test(restart-chaos): add ST4 partial multipart and ST7 read-after-write lag storage faults (#20504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` puts a fault proxy between the server and a loopback MinIO (ST1–ST3, ST5/ST5R) > - Remote workspace snapshots over 16 MiB upload as S3 multipart uploads, and saves are read back right after they are written > - No fault hit single upload parts or the completion, and no fault made a just-written object briefly invisible > - Nothing checked that a wave left no orphaned multipart uploads or that every committed snapshot reads back > - This pull request adds ST4 (partial multipart) and ST7 (read-after-write lag), and a teardown store audit > - The benefit is that waves can prove snapshot saves and restores survive part failures and read lag without data loss or leaked uploads ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults treat every request the same. They cannot target one multipart part or the completion, and they cannot lag reads behind writes. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** A wave cannot fail the Nth `UploadPart` of a snapshot upload, cut a part body mid-transfer, or fail `CompleteMultipartUpload`. A wave cannot make a `GET` or `HEAD` answer `NoSuchKey` right after a `PUT`. No wave checks for incomplete multipart uploads or unreadable committed snapshots. **Proposed behavior** `--storage-faults ST4` fails a seeded subset of snapshot upload parts (500/503, reset, or reset after 1–4 MiB of the body), and sometimes the completion. `--storage-faults ST7` answers 404 `NoSuchKey` to reads of an object for 5–30 s after its write succeeded. With either fault, the wave reads every committed workspace snapshot back from MinIO (size and SHA-256 against its row) and, after the server stops, lists incomplete multipart uploads under the wave prefix. Either problem fails the wave. **Reason and benefit** These are the storage failures that can lose a workspace or leak storage cost without any visible error. ## What Changed - `storage-s3-ops.ts` (new): names the S3 operation from the method and query, and reads the part number. - `storage-multipart.ts` (new): ST4 rule planning. - `storage-lag.ts` (new): ST7 rule planning, and a tracker of successful `PutObject` and `CompleteMultipartUpload` writes per key. - `storage-audit.ts` (new): teardown audit (committed snapshot read-back, incomplete multipart uploads), `storage-audit.json`, and a `Storage audit` section in `SUMMARY.md`. MinIO lists multipart uploads only for an exact key or no prefix, so the audit lists the wave-owned bucket and filters by the wave prefix. - `storage-proxy.ts`: rules can match `operations` and `partNumbers`. New faults `truncate` (reset after N body bytes) and `read_after_write_lag`. `storage-proxy.json` reports `lagged`. - `storage-chaos.ts`, `main.ts`: registry lines for ST4/ST7 and the audit calls. - README: ST4/ST7 rows and the audit. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts`: 35 files, 423 tests pass. The new `storage-multipart-lag.test.ts` fails before this change (the modules and kinds do not exist) and passes after. It includes real MinIO tests with the AWS SDK: part 2 fails once with 500, a part body is cut after 1 MiB, the completion fails once, the audit finds the incomplete upload and then finds none after completion; a just-written object answers `NoSuchKey` to `GET` and 404 to `HEAD` until the lag passes; the snapshot check reports `sha256_mismatch` and `object_missing`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves run from master after merge (local ST7, Daytona R1 ST4 and ST7). Results go in a follow-up comment. ## Risks - Test harness only. No product code changes. - The audit runs only on waves with ST4 or ST7, so other storage waves do not change. - The snapshot check verifies the stored ciphertext against its row. It does not decrypt, because the key lives in the instance. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
82abb3ea04 |
fix(native): classify a Codex API-key unknown model as a model rejection (#20488)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runner agents run on a provider (here Codex) with a model the
user saves on the agent
> - When the provider rejects that model, Paperclip should classify the
run as `native_provider_model_rejected`, ask the user to choose a model,
and let Retry finish the task
> - Paperclip recognized only the ChatGPT-account rejection (a 400 JSON
envelope). With API-key auth, Codex returns a different terminal error
for an unknown model
> - So the run failed as a generic `adapter_failed` ("Native session
failed"), and recovery opened a reconciliation incident ("Recovery
needed") for a turn that never did any work
> - This pull request recognizes the real API-key shape, bound to the
configured model, and names the rejected model on the task
> - The benefit is that the task asks for a model choice instead of a
reconciliation, and Retry completes the task after the user picks a
valid model
## Linked Issues or Issue Description
**What happened?**
The restart-chaos config fault CF3 `model_id` saves a Codex agent with
the model `chaos-missing-model-<nonce>`. The run stays "running" for
about 60 seconds, then fails with `errorCode: adapter_failed`, "Native
session failed". The task shows "Recovery needed", cause
`native_continuation_requires_reconciliation`. The grader fails as
`CF3/model_id:actionable_message` (waves `w-20261110T061914-s206-cf` t01
and `w-20261110T053831-s205-cf` t02).
Root cause: the wave runs Codex with an OpenAI API key. I drove the
bundled `codex app-server` (0.160.0) directly with an unknown model:
- With API-key auth, Codex retries the Responses API 404 ten times
(about 59 s), then fails the turn with `{"message":"unexpected status
404 Not Found: The model \`<id>\` does not exist or you do not have
access to it., url: https://api.openai.com/v1/responses, cf-ray: …,
request id:
req_…","codexErrorInfo":{"httpConnectionFailed":{"httpStatusCode":404}}}`.
- With ChatGPT auth, the turn fails in about 3 s with `message` =
`{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"The
'<id>' model is not supported when using Codex with a ChatGPT
account.","code":"model_not_found"}}`. Paperclip already classified this
one.
`createNativeProviderFailureObservation` parsed only the JSON envelope,
and the runner's provider-failure category for an HTTP 404 is `unknown`.
The terminal therefore stayed unclassified.
**Expected behavior**
The run is `native_provider_model_rejected`, the task shows `userAction:
"choose_model"` and names the model, no reconciliation incident opens,
and Retry completes the task after the user picks a valid model.
**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF3 --adapters codex
--references model_id --tasks 1 --concurrency 1 --restarts K2 --seed
206` on `master` (with `OPENAI_API_KEY` set).
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude
|
182a7cd4fe |
test(restart-chaos): keep process start identity out of TM2/TM3 aging (#20500)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) checks recovery under faults. TM2 (host sleep) and TM3 (aged outage) move every database time back by an age. > - The aging also moved process start times: `controller_process_started_at`, `process_started_at`, and `processStartedAt` copies in JSON. > - A process start is identity, not a deadline. The server reads its own start from the OS once and compares it with the stamp an owner saved. A real sleep never changes it. > - After a shift, every live owner looked like a different process and failed its identity check, whatever its lease said. This harness artifact hid whether a lease fix works (#20474). > - This pull request keeps process start identity unaged and makes the grader's unshift skip it too. > - The benefit is that TM2/TM3 failures now come only from deadlines that really lapse, as on a real laptop. ## Linked Issues or Issue Description Refs #20458 (adds TM1–TM3) and #20474 (the lease fix whose proof wave exposed this). **What happened?** A TM2 wave against #20474 still failed at the 300 s renewal. The fix takes back a lapsed lease only for the same owner (token, attempt, generation, boot, pid and process start). The aging had moved the saved `controller_process_started_at` back by one hour. The live process still reported its real start, so the identity comparison failed. **Expected behavior** Aging moves deadlines and event times. It never moves facts that a real sleep cannot change, such as when a process started. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304 --server-from <#20474 checkout>`. Before this change it fails 0/2 (w-20261110T072106-s304). ## What Changed - `time-faults.ts`: add `isProcessIdentityTime`. `agingStatements` skips columns named for a process start. `paperclip_shift_json` keeps JSON keys named for a process start (any spelling) unchanged. `unshiftJson` skips the same keys, so the grader does not move them forward. - `time-faults.test.ts`: a unit test for the statements and the unshift. - `time-faults-db.test.ts`: the embedded-Postgres aging test now checks that a `controller_process_started_at` column and `processStartedAt` / `processStartedAtMs` JSON keys keep their values. - `README.md`: one paragraph under **Aging** that says why. ## Verification - `npx vitest run -c tests/restart-chaos/vitest.config.ts tests/restart-chaos/time-faults-db.test.ts tests/restart-chaos/time-faults.test.ts`: 26/26 pass. Without the `time-faults.ts` change, 2 fail (the new unit test, and the DB aging test's identity check). - `npx tsc --noEmit -p tests/restart-chaos` is clean. - Test-only change. No product code. ## Risks - Low. Only the harness changes. A column or key whose name contains "process started at" is no longer aged. If such a field were really a deadline, it would no longer lapse, but none is. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2f59b9c67c |
test(chaos): fire a planned restart while the server is mid S3 transfer (#20451)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness restarts the control plane while agent runs are in flight, and `--storage s3` runs the server against a loopback MinIO behind a fault proxy > - Planned restarts fire by time or by run events. A time-based restart seldom lands while the server is moving bytes to or from the store > - The riskiest storage moments are a workspace snapshot upload, a snapshot restore, and a run-log journal write. A restart in the middle of one must not lose or corrupt the workspace or the log > - This pull request adds a storage-triggered restart: the planned fault fires the moment the proxy sees a matching S3 request, and the proxy holds that request so the restart interrupts it > - The benefit is that restart-during-transfer becomes a repeatable wave with receipts that name the interrupted request ## Linked Issues or Issue Description **Subsystem affected** Restart-chaos test harness (`tests/restart-chaos`), cloud storage mode. **Problem or motivation** A K1/K2/K4 restart fires on a timer or on a run trigger. With `--storage s3`, the timer almost never lands during a snapshot upload, a snapshot download, or a run-log write. So the harness does not prove that a restart in the middle of a transfer is safe. **Proposed solution** Add `--storage-trigger <target>`. At its planned time, each task's fault is armed. The next server request that matches the target fires it. The proxy holds the request open (up to 60 s) while the harness snapshots the tasks and restarts the server. **Alternatives considered** A new `--triggers storage` id. It needs edits to the observer (near its 1,000-line limit), the trigger registry and the delay table. The storage trigger is a wave-level mode in a new module instead. ## What Changed - New `tests/restart-chaos/storage-trigger.ts`: targets `snapshot-upload`, `snapshot-part` (multipart `UploadPart`), `snapshot-download` and `run-log`; the `StorageTrigger` that arms faults, fires on a match, holds the request and records receipts. - `storage-proxy.ts`: an `observe(listener)` hook. The listener sees every request and can hold it (the existing latency path). - `main.ts`, `config.ts`, `types.ts`: registry lines only. `--storage-trigger` (needs `--storage s3` and `--triggers time`); a due fault is armed instead of fired; the wave writes `storage-trigger.json`. - `storage-trigger.json` records the matching request (method, path with upload ids hidden, declared bytes, outcome `aborted` or `completed`, status), every other request in flight, and the fault id. A fault that never matched is recorded with `missedReason`. - README: "Restart during a transfer". ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (25 files, 304 tests pass). - Red/green: `storage-trigger.test.ts` against the master `storage-proxy.ts` fails 3 of 6 tests (no hook, so nothing is held or fired); it passes with this change. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local run-log, Daytona snapshot upload and download) are recorded in a PR comment as they finish. ## Risks - Low risk. Test harness only. Without `--storage-trigger`, the proxy has no listener and behaves as before. - A replay re-arms its faults at their observed offsets. It fires on the next matching request, not on the exact same request. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7fef6321fe |
test(restart-chaos): remove storage fault window time from speed budgets (#20501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness grades each wave's speed against local budgets (`wave-speed.ts`) > - The speed grade already removes restart outage time (fault start to server ready) from spans and restart recoveries > - It did not remove storage fault window time (ST1 outage, ST5 rejected credentials, and so on) > - A restart recovery that correctly waits for the store inside a storage fault window then counts as a speed breach > - This pull request removes the overlap with storage fault windows the same way, and shows the removed time on its own line > - The benefit is that storage waves fail only on real slowness, not on correct waiting ## Linked Issues or Issue Description **What existing behavior does this improve?** The wave speed grade treats time spent inside a storage fault window as product slowness. **Subsystem affected** tests/restart-chaos (speed grade). **Current behavior** The ST5 waves w-20261110T050044-s503 and w-20261110T062313-s506 exited 1 on speed breaches only. Their `restart_recovery_control_plane` p95 was 62–67 s against a 4 s budget. The recoveries were inside an ST5R window and correctly waited for a valid key. **Proposed behavior** Spans and restart recoveries do not count time inside storage fault windows (from the `storage-faults.json` receipts). Restart outage and storage windows that overlap are removed once. `SUMMARY.md` shows the storage time that was removed, and `speed.json` has `storageFaultWindows` and `storageFaultRemovedMs`. **Reason and benefit** Storage waves must fail on real product failures and real slowness, not on the harness's own fault windows. ## What Changed - `wave-speed.ts`: `storageFaultWindows()` turns receipts into windows (an open window ends now). `mergeWindows()` joins overlapping windows, so `outageFreeClock()` removes shared time once. `collectWaveSpeed()` takes the storage receipts and removes their time from spans; `gradeWaveSpeed()` removes it from restart recoveries. The storage share is reported apart from the restart outage share. - `main.ts`: one line passes `storageChaos.receipts` to `collectWaveSpeed()`. - README: one paragraph in the Cloud storage section. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/wave-speed.test.ts`: 20 pass. The 3 new tests fail before this change (`storageFaultWindows is not a function`, overlap removed twice, span not trimmed) and pass after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. ## Risks - Test harness only. No product code changes. - ST2 (slow store) windows are also removed from speed budgets. Slowness during a slow-store window is expected, so this is intended. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
a14ae9b5c9 |
fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Before an agent starts, Paperclip prepares the task workspace and runs the project's configured setup commands (for example the workspace provision command) > - When that command fails, the run fails as `setup_failed`, and the task stops with a Retry > - The task page shows only "Failed" and a generic recovery notice. The real error (the command, its exit status and output) is visible only on the run page > - The user cannot see from the task which setting is broken or what to change > - This pull request records a bounded, named cause for a failed setup command and shows it on the task page > - The benefit is that the task names the broken setting and what to change, and classification, retry and recovery stay the same ## Linked Issues or Issue Description **What happened?** The restart-chaos config fault wave CF1 sets a project `workspaceStrategy.provisionCommand` to `bash ./scripts/provision-worktree.sh`, a script that does not exist. The run fails as `setup_failed` with `Execution workspace provision command "bash ./scripts/provision-worktree.sh" failed: bash: ./scripts/provision-worktree.sh: No such file or directory`. The task page shows "Failed" and "Paperclip automatically retried continuation … still has no live execution path". The grader fails the task as `CF1/provision_command:message_only_in_run_detail`. **Expected behavior** The task page names the failed setup step, the configured command, its exit status and a short tail of its output, and says what to change. Retry behavior stays the same. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks 2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`. **Paperclip version or commit** `master` at |
||
|
DottaandClaude
|
1e13c58a4e |
fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat run settlement records why a run failed, and the board shows that record. > - Runs that fail without a named code settle as `adapter_failed` (or `setup_failed`) with no cause. > - A user, an operator, and the restart chaos harness cannot tell what went wrong from such a run. > - Some run errors also show raw SQL statement text to users. > - This pull request records a bounded `failureCause` code for generic failures, shows it on the board, and keeps SQL text out of run errors. > - The benefit is that a generic failure now names its cause, for example "adapter failed (40001)", without leaking messages, SQL, or secrets. ## Linked Issues or Issue Description Refs #20385 (the nested-cause extraction this change reuses). Related: #20421 (restart chaos universal checks U1-U4). **What happened?** The restart chaos audit of 79 saved waves found 30 failed runs with `errorCode: "adapter_failed"` and a null or generic error. Every U3 ("diagnosable") warning was `adapter_failed`. Nobody could tell what failed. Some board run errors also showed raw SQL statement text from wrapped database errors (Drizzle `Failed query: ... params: ...`). **Expected behavior** A run that fails with a generic code names a short, safe cause. The board shows the cause next to the stop reason. Run error text never shows SQL statements or their parameters. **Steps to reproduce** Make an adapter throw a wrapped database error (an `Error` whose nested `cause` has `code: "40001"` and whose message has a Drizzle `Failed query:` statement). Before this change, the run settles as `adapter_failed` with no cause, and `heartbeat_runs.error` contains the SQL statement and parameters. The new tests in `server/src/services/heartbeat/run-completion.test.ts` reproduce this. ## What Changed - Runner: move the #20385 event-commit cause extraction into `safeErrorCauseCode` (`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and export it. The Runner's own diagnostics use the same function, unchanged in behavior. - Shared: add the `failureCause` contract (`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]` token of at most 80 characters, `normalizeRunFailureCause`, `readRunFailureCause`, and `withoutSqlStatementText`. - Server: add `server/src/services/heartbeat/run-failure-cause.ts`. Settlement records `resultJson.failureCause` only when the code is generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same. - Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`, else `provider_<category>`, else the first safe code in the error and up to three nested causes (`40001`, `ECONNRESET`), else the error class name, else `unknown_error`. - Adapter results without a code (`completeRun`): the error family, `signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`, `adapter_reported_error`, or `unknown_error`. - The failure log lines (`heartbeat execution failed`, `heartbeat execution setup failed`, and a new `heartbeat run failed` warning for generic adapter results) include `failureCause` and `errorCode`. - Server: replace SQL statement text in run error text with "Database query failed (<cause>)" at all three settlement paths, including the cancelled branch. The execution blocker's `runError`, the attention inbox (failed-run excerpt and metadata), and continuation summaries apply the same replacement on read, so older rows also stop showing SQL there. - Server: Retry admission and failed-preparation replay accept only a closed set of `resultJson` keys. `failureCause` is added to each of those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`, `cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is a diagnostic code with no execution or replay authority, so a failed-preparation run stays retryable. - Server: `runsForIssue` returns a validated top-level `failureCause`. It is null for older rows, non-string JSON values, and malformed values. - UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed (40001)". Older runs keep "adapter failed". The Storybook run-ledger fixture has a failed run with a cause. - Restart chaos harness: run samples keep `failureCause`. U3 treats a generic code with a specific `failureCause` as diagnosable. A bare `Error`, `unknown_error`, or the no-code fallback `adapter_reported_error` still counts as vague. U3 stays a warning. - Docs: `doc/run-log-events.md` and the harness README describe the field and the SQL replacement. ## Verification - Red before the fix: the four new `run-completion.test.ts` cases failed against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL text in `error`). - Green: - `server/src/services/heartbeat/run-completion.test.ts`: all tests passed (real embedded PostgreSQL). New cases cover a nested `40001` three levels deep for execution and setup failures, SQL and a secret parameter absent from `error` and `resultJson`, the bound (an 81-character code and a SQL "code" fall back to the class name), `exit_code_137` for a code-less adapter result, a cancelled result with SQL text, no cause for a named code, and the log line. On this loaded host the suite's `afterAll` database teardown hit its 30 s hook limit after every test passed (the known local teardown limit, also seen in #20385). CI runs the suite. - `server/src/services/heartbeat/run-failure-cause.test.ts` (4), `server/src/__tests__/activity-service.test.ts` (recorded, legacy, malformed, overlong, boolean, and number cause projections), `server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL row). - `packages/shared/src/run-failure-cause.test.ts` (6), `ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos `universal.test.ts` and `observer.test.ts`. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`: 207 passed (the stall-diagnostic cases still read `40001` and `ECONNRESET`). - Typecheck: shared, runner, server (all layers), UI, restart chaos harness. `pnpm check:token-gates`: all gates clean. - `pnpm verify` returned exit 2 (global doc input changed); required CI supplies broad coverage. - CI on `6295f8d` found that the new key made failed-preparation runs ineligible for Retry (`chat-channels.integration.test.ts` saved board-message Retry got 409; `workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries). Both reproduced locally and pass after the allowlist fix, with the other 459 Retry cases in `chat-channels.integration.test.ts` and all 25 in `workspace-recovery-preparation-dispatch.test.ts`. - No chaos waves were run. Storybook was not opened in a browser; the fixture typechecks and the label is covered by the UI test. ## Risks - Low risk. `errorCode` values do not change, so retry, recovery, and comparison logic is unaffected. - No schema change and no migration. `failureCause` is a new optional `resultJson` key. Rows saved by the previous release have no key and render as before; a new row read by an old server is ignored. - The SQL replacement only acts on a Drizzle `Failed query:` marker, or on `$n` placeholder text with a statement shape. Ordinary error text does not change. When it acts, it also drops the text after the statement. - Code that checks `resultJson` against a closed key list must allow `failureCause`. I found and updated the four lists on the Retry path; a list I missed would refuse a Retry for a run with a generic failure. - This is the run log / instance database path. It adds no Telemetry event. - The server failure log line still logs the raw error message (unchanged behavior); only the board-facing text is changed. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file editing) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
5c9641f141 |
test(restart-chaos): add repeated restarts (RR1-RR3): a restart inside the previous restart's recovery (#20480)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Every existing fault restarts the server once per task. > - Real incidents on 2026-11-08 and 2026-11-09 were deploy loops. The server restarted again 5–60 s after it came back, while recovery of interrupted runs was still in progress. > - No wave puts a second restart inside the first restart's recovery window. > - This pull request adds a fault family, repeated restarts (RR1–RR3). It does not change product code. > - The benefit is that waves now test recovery that is itself interrupted. ## Linked Issues or Issue Description **What is the problem?** The chaos harness restarts the server only once per fault. Production deploy loops restart it again while it recovers, and no wave tests that. **What is the solution?** Add three repeated-restart faults. Each one starts as a K1 and then stops the replacement again after a seeded gap from its health. **Why is it needed?** Recovery of interrupted runs can itself be interrupted. A second restart in that window can leave a lease, a claim or a wake half done. Refs #20458 (the TM fault family this one follows). ## What Changed - Add `RR1`, `RR2` and `RR3` to `--faults` (`repeated-restarts.ts`): - **RR1 double crash**: K1, then one more `SIGKILL` 3–20 s after the replacement reports healthy. - **RR2 crash, then graceful**: K1, then one `SIGTERM` (as K2) 3–20 s after health. - **RR3 crash loop**: 3–4 `SIGKILL`s in all, 10–30 s after each health. The last replacement stays up. - Add `--rr-gap-seconds <n>` or `<min>-<max>` (1–300). It replaces every RR gap. RR faults do not combine with `--storm-minutes`. - The gaps are seeded per task (`restartSteps` in `plan.json`, also in `--matrix` plans). They are drawn only for RR tasks, so the plans of existing seeds do not change. `--replay` restarts again after the recorded gaps. - The supervisor runs the whole sequence as one fault receipt. The grader therefore sees one fault window, from the first kill to the last replacement's ready. Crossing, R1's outage allowance, R5, the T-oracle, task deadlines and the recovery speed budget treat it as one fault. No check is relaxed. - Each stop and replacement is recorded with its signal, gap, PID, the `/api/ready` status just before the stop, and its stopped, restarted and healthy times. The record goes to `<wave>/repeated-restarts/<faultId>.json` (also written when the sequence fails part way), to the receipt (`repeatedRestart` in `faults.json` and in each failure's `startup-phases.json`), and to the timeline (`repeated_restart_stop`, `repeated_restart_healthy`). - Small registry additions: `types.ts`, `config.ts`, `scheduler.ts`, `journey-matrix.ts`, `main.ts`, `evidence.ts`, `supervisor.ts`, and the README section next to TM. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 33 files, 402 tests pass. The new `repeated-restarts.test.ts` covers option parsing, seeded planning for each kind, unchanged plans for other faults, steps in the task schedule, the matrix and replay, the stop-and-replace loop on a fake clock (gaps measured from health, `SIGTERM` for RR2, an exit during a gap, shutdown), and one crossing window for the sequence. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - `pnpm verify` asks for broader coverage (the README is a global CI input), so required CI supplies it. - Live waves run from master after merge, as the merge rule for test-only harness PRs allows. Their results go to a follow-up comment. ## Risks - Harness-only change; no product code changes. Waves without RR faults are unchanged: the plan draws, the fault receipt and the grader paths are the same. - During an RR sequence the server stays "not ready" for the harness, so no other fault fires and the observer does not click Retry until the last replacement is ready. This is the same rule as for K6. - A replacement that exits on its own during a gap fails the fault as a harness error, with the stops so far recorded. The wave then recovers the server as after any failed fault. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
179694de7d |
test(chaos): provider message faults for Claude (ACPX) and across a K1 restart (#20454)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) proves that agent work survives control-plane restarts and provider faults > - PR #20432 added provider message faults (PV1 delay, PV2 drop, PV3 duplicate), but only for Codex, and only in waves without restarts > - Claude runs through the ACPX sidecar, whose provider pack and sidecar are digest-verified, so the Codex executable shim cannot reach it > - A late or missing terminal can also arrive across a control-plane restart, which no wave could exercise > - This pull request adds a Claude (ACPX) seam with no product change, and a K1 restart that fires after a provider fault applies > - The benefit is that governed waits, Stop and approvals are now tested under provider message faults for both main adapters, also across a kill -9 ## Linked Issues or Issue Description Refs #20432, #20330, #20336 **What is the problem?** Provider message faults covered only Codex. They could not run with restart faults. PV3, the `interrupt_ack` target, and the S1 and G1 workloads under PV1/PV2 never ran live. **What is the expected outcome?** The harness can delay, drop or duplicate one targeted provider message for Claude and Codex. A PV1/PV2 fault can straddle a K1 restart, and the grade states what a pass means. **Why is this the right layer?** This is test-harness work. The product is not changed. The seam replaces only a file the runner already hashes and verifies at launch, inside a per-wave stand-in package. ## What Changed - `acpx-provider-faults.ts`: builds a per-wave stand-in runner package under `<repo>/node_modules/.restart-chaos-<wave>/` (symlinks to the real package, except a generated `dist/cli/acpx-runtime-sidecar.cjs`). It also parses `--provider-fault-restart`. - `provider-shim/acpx-sidecar-hook.mjs`: a server-only resolve hook that loads `runnerd-codex-transport` (source `src/live/*.ts` or built `dist/live/*.js`) from the stand-in package. Product code then hashes the stand-in sidecar, and the runner verifies and launches it. The module's own imports resolve to the real package. - `provider-shim/acpx-fault-tap.cjs`: the stand-in installs this tap and then runs the real sidecar bundle in the same process, as the runner's verified CommonJS loader does. For Claude, `terminal` is the provider's ACP `session/prompt` response after a `session/cancel`. `interrupt_ack` is the sidecar's `turn.cancel` response, because ACP's cancel has no acknowledgement. The provider process is still spawned and admitted by product code. - `--provider-fault-restart K1 [--provider-fault-restart-after-seconds n]`: each PV1/PV2 task also kills -9 the control plane n seconds (default 3) after its fault applied, through a new internal `provider` trigger. PV3 and planned K cells stay rejected. - `withProviderFaultCheck`: with a restart, the check passes only if the fault applied once, the K1 started after it, and for PV1 no release came first. The ordinary R1–R6 grade and the workload T-oracle still decide whether the task recovered. - `config.ts`: provider faults accept `--adapters codex,claude`. G1 with a provider fault needs `--provider-fault-restart K1`, because its oracle grades the board decision after a restart. - README: the Claude seam, the target table per adapter, what a pass means across a restart, and the Claude PV1 delay window. - Unit tests: the ACP and runner-pipe taps, the generated stand-in with a fixture sidecar, the hook against a fake runner package, planning, config validation, and the restart check. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (all pass after rebase on master) - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` - Live waves (local, one at a time through the wave-slot protocol): | Wave | Scenario | Result | | --- | --- | --- | | w-20261110T040202-s202 | Claude Q1, PV1 terminal 18 s | Task passed its Q1 oracle; `provider_fault_applied` failed: the runner settled the cancelled turn ~3.6 s after the cancel and closed the sidecar before the release, so 18 s is a drop for Claude (documented) | | w-20261110T041224-s203 | Claude Q1, PV1 terminal 1 s and 3 s | 2/2 pass (held and released) | | w-20261110T041355-s204 | Claude Q1 and S1, PV2 terminal | 2/2 pass | | w-20261110T043354-s205 | Codex Q1, PV3 terminal (duplicate) | 1/1 pass | | w-20261110T050407-s206 | Codex S1 and G1, PV1/PV2 terminal | 0/4: S1 terminal never sent (Codex exits after Stop, so `terminal` cannot apply; documented, use `interrupt_ack`); G1 needs a restart by design (now rejected without `--provider-fault-restart K1`). No product failure. | | queued | Codex+Claude Q1/S1 PV1/PV2 `interrupt_ack`; Codex+Claude Q1/G1 PV1/PV2 + K1 | Runs from master after merge (queue is ~90 min deep) | ## Risks - Low product risk: no product code changes. - The seam depends on the runner computing the local sidecar path from the transport module URL and on the ACPX provider bootstrap marker string. If either changes, the tap logs nothing and every Claude PV task fails `provider_fault_applied`. It does not pass silently. - The stand-in package is written under the workspace `node_modules` (gitignored) and removed at wave end. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5` (Claude Code agent with tool use and code execution) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
bf1524c5f8 |
test(restart-chaos): pin each wave's API URL to its own loopback server (#20462)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness starts an isolated Paperclip server for
each wave and runs real agents against it
> - Agents find the server through `PAPERCLIP_API_URL`, which the server
derives at startup
> - On a developer machine with a public URL for the main instance (for
example a tailnet origin), the wave server derived that public URL
instead of its own loopback origin
> - So `process` agents sent the run key for the temporary instance to a
different host, which returned 502
> - This pull request pins each wave server's `PAPERCLIP_API_URL` to its
own loopback origin and removes the Q3 script workaround
> - The benefit is that a wave's agent credentials only go to the wave's
own server
## Linked Issues or Issue Description
**What happened?**
In isolated restart-chaos wave instances, `process` agents received
`PAPERCLIP_API_URL` set to the machine's public tailnet origin. That
origin serves the developer's main instance, not the wave. The agent's
run key for the temporary instance was sent to that host, which returned
502. The Q3 agent script worked around this with a hard-coded loopback
base URL.
**Expected behavior**
A wave's agents call the wave's own server.
**Steps to reproduce**
1. Configure a public URL (for example a tailnet origin) for the
developer's main Paperclip instance on the machine.
2. Run a restart-chaos wave with a `process` agent that reads
`PAPERCLIP_API_URL`.
3. The agent's API calls go to the public origin, not to the wave's
`127.0.0.1` port.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude
|
400bb1f0f9 |
test(restart-chaos): one-row db pressure, bound accepted-Retry waits in U1, live L1 results (#20472)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness grades real agent tasks on an isolated control plane. > - #20421 added the L1 contention workload, `--db-pressure`, and the universal checks U1–U4. > - The first live L1 waves showed two harness gaps and one missed product shape. > - The pressure writers locked several run rows per statement and deadlocked with product transactions. A real stream does not do that. > - U1 counted an accepted Retry that waits on the system as a live owner forever. Ten tasks sat 8 minutes in that state with no run. > - This pull request fixes both gaps, adds the owed direct test for `collectContentionEvidence`, and records the live wave results. ## Linked Issues or Issue Description Refs #20421, #20385. **What happened?** The live L1 waves (`w-20261110T040531-s20385` on the #20385 parent, `w-20261110T044338-s20385` and `w-20261110T050818-s20386` on master) produced harness deadlocks (40P01) from multi-row pressure statements. U1 passed tasks whose accepted Retry waited on `process_identity_missing` for 16 minutes without a run. **Expected behavior** Pressure writers behave like a run-log stream (one row per statement). U1 flags an accepted Retry that waits on the system past the queue bound. **Steps to reproduce** Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <results dirs>` over the saved waves, or the L1 wave command in the README. ## What Changed - `contention.ts`: each pressure statement updates one running run row. The old statement updated every running row at once and deadlocked with product transactions. - `universal.ts` (U1): an accepted Retry that waits on the system (`retryWait`) is an owner only for the queue bound (5 min of server-ready time). While it waits, the Retry button, failed notice and blocker do not count as actionable items, because the person already acted. A scheduled retry is still a timer. - `contention-db.test.ts`: a real PostgreSQL test for `collectContentionEvidence` (receipt count, duplicate producer events, terminal receipt, accounting fields, company scope). - README: live wave results table, updated audit numbers and the U1 rule. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 28 files, 345 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Audit over 79 saved waves (1,620 tasks, 1,483 passing): U1 17 flags, none on a passing task or an all-pass wave. The longest accepted-Retry wait on a passing task was 278 s. U2 1, U3 30 (3 on passing tasks, warning only), U4 0. - Live waves (codex and claude, `--workloads L1 --faults K1 --concurrency 4 --tasks 4 --db-pressure 4`): - `w-20261110T040531-s20385`, #20385 parent: 117 receipt-write serialization failures (40001) in the server log; 1 of 4 tasks passed. Three runs failed `adapter_failed` after 40001 exhausted in `createRunUsageRecorder`, `reserveRunBudget` and native finalization. - `w-20261110T044338-s20385`, master `8004ad069e` with the harness: 2 serialization failures; 1 of 4 passed. Failures included 40001 in `createRunUsageRecorder` and deadlocks caused by the old multi-row writers (fixed here). - `w-20261110T050818-s20386`, same build with one-row writers: 0 of 4 passed. Every first run failed in `reserveRunBudget` before spawn, and every accepted Retry then waited on `process_identity_missing` without starting a run. - No wave reached the runner's commit stall, so none showed `native_event_replay_conflict`. ## Risks - U1 can now fail a task whose accepted Retry waits more than 8 minutes. No passing task in 79 saved waves came close (longest 278 s). - One-row writers produce fewer statements per second than the multi-row statement did; `contention-db.test.ts` still reproduces the #20385 receipt race with them. - Test-only change; no product code. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2444658c6f |
test(restart-chaos): add ST5 credential rejection and ST5R key rotation storage faults (#20455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness runs real agent tasks while it injects infrastructure faults > - `--storage s3` already puts a fault proxy between the server and a loopback MinIO (ST1 outage, ST2 latency, ST3 errors) > - Cloud deployments use the default AWS credential chain, and keys can expire or rotate during a run > - The harness had no fault for a rejected or rotated key, so this failure class was untested > - This pull request adds ST5 (a window of 403 credential rejections from the proxy) and ST5R (a real key rotation at MinIO) > - The benefit is that waves can prove runs wait out credential problems and pick up rotated keys ## Linked Issues or Issue Description **What existing behavior does this improve?** The restart-chaos storage faults cover outages, latency and 5xx errors only. **Subsystem affected** tests/restart-chaos (storage chaos). **Current behavior** No wave can reject the server's S3 credentials or rotate the store's key. **Proposed behavior** `--storage-faults ST5` answers every request in a window with 403 `ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch`. `--storage-faults ST5R` restarts MinIO with a new root key (the old key stops working) and gives the server the new key through its credential source only when the window ends. **Reason and benefit** The first local ST5 wave (w-20261110T041556-s501) shows the gap: a 41 s `ExpiredToken` window failed 4 of 6 tasks with `adapter_failed`, and the automatic retries failed within one second. The product fix is a separate PR. ## What Changed - `tests/restart-chaos/storage-credentials.ts` (new): ST5 and ST5R rule planning, a wave-owned `AWS_CONFIG_FILE` profile whose `credential_process` prints the current key, MinIO restart with a new root key on the same loopback port, and a proxy wrapper that holds traffic during the restart and publishes the new key when the window ends. - `storage-proxy.ts`: the `expired_credentials` fault takes an S3 error code; a `credential_rotation` marker passes traffic through. - `storage-chaos.ts`, `types.ts`, `config.ts`, `main.ts`: registry lines for ST5/ST5R and the rotation wiring. - README: ST5/ST5R rows and the credential-source description. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts` (28 files, 328 tests pass). The new `storage-credentials.test.ts` fails before this change (the kinds do not exist) and passes after. It includes a real MinIO test: after a rotation the old key gets 403 `InvalidAccessKeyId`, a client with a cached key fails, and the same client succeeds after it re-resolves its profile. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave: `--env local --workloads A1,C1 --faults K1,K2 --tasks 6 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds 30-90` ran and injected 62 credential 403s (storage-proxy.json). ## Risks - Test harness only. No product code changes. - ST5R restarts MinIO. Requests are held for up to 5 s during the restart, so a slow restart can also show as a short 502 blip. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
95cb3c94c4 |
test(restart-chaos): cap store bandwidth during ST2 slow-store windows (#20452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness tests how runs survive restarts and cloud storage faults > - A Paperclip-sized remote workspace (R1 on Daytona with `--storage s3`) saves about 730 MiB per snapshot as a multipart upload and restores it as one large download > - The ST2 "slow store" fault only adds latency per request. On a healthy loopback store the whole multipart upload takes about 2 s, so a latency fault adds a fixed delay per part and never makes a large transfer slow > - A real degraded store is slow per byte too. We need that to test large saves and restores that take minutes > - This pull request gives each ST2 window a seeded bandwidth cap that all requests in the window share > - The benefit is that ST2 waves now slow large snapshot uploads and downloads by minutes, and the harness can find save and restore timeouts on big workspaces ## Linked Issues or Issue Description Refs #20299, #20313 (storage outage handling on small workspaces), #20442 (saved workspace Files over 512 MiB). **What existing behavior does this improve?** The ST2 slow-store fault in the restart-chaos storage proxy. **Subsystem affected** Test harness (`tests/restart-chaos`). **Current behavior** ST2 adds 1–10 s of latency before each request. The bytes then flow at loopback speed. A 730 MiB snapshot upload or download is still fast. **Proposed behavior** ST2 also caps bandwidth at a seeded 2–16 MiB/s. One rate per direction is shared by every request in the window, so parallel multipart parts share the cap. **Reason and benefit** Large saves and restores now take minutes under ST2. This shows the save and restore timing, the storage request counts, and any timeout on a slow store. ## What Changed - New `tests/restart-chaos/storage-bandwidth.ts`: a shared byte-rate pacer (`ByteRate`), a pacing stream (`paced`), and the seeded ST2 bandwidth draw (2–16 MiB/s). - `storage-proxy.ts`: the `latency` fault takes an optional `bytesPerSecond`. The proxy paces the request body and the response body through one rate per direction for each rule. - `storage-chaos.ts`: ST2 windows set `bytesPerSecond`. Saved receipts keep the rule, so a replay uses the same cap. - README: the ST2 row describes the cap. ## Verification - `PAPERCLIP_IN_WORKTREE=false npx vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/storage-bandwidth.test.ts tests/restart-chaos/storage-proxy.test.ts tests/restart-chaos/storage-chaos.test.ts`: 14 pass. - Red before: with the proxy and ST2 changes reverted, the proxy cap test and the ST2 draw test fail. Green after. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. ## Risks - Low risk. Test harness only. ST2 now draws one more seeded number, so a seed gives different later windows than before. Replays read the saved receipts and are not affected. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
30d99f4e32 |
test(restart-chaos): add time faults (TM1-TM3): freeze, sleep with aged rows, aged outage (#20458)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) tests if agent tasks survive control-plane restarts. > - Each wave runs for minutes on a fresh instance. Nothing time-bound ever expires during a wave. > - Production failures on 2026-11-09 were about time: an OAuth refresh after more than 1 h made pending approvals invalid (#20340), and a timeout used the wrong window (#20330). > - This pull request adds a fourth fault family, time faults (TM1–TM3). The faults make leases, deadlines and waits expire during a run or across a restart. They do not change product code. > - The benefit is that waves now find failures that occur only when time passes. The first waves found one: a run whose turn completes after a host sleep fails, and the task needs Retry. ## Linked Issues or Issue Description Refs #20340 Refs #20330 **What is the problem?** The chaos harness never moves the clock, so it cannot find failures where a lease, a deadline or a pending wait expires. **What is the solution?** Add three time faults. Each one makes stored deadlines lapse in a way that a real host sees: a process pause, a host sleep, or a long outage. **Why is it needed?** Two days of chaos testing did not find the failures of #20340 and #20330, because no wave lasted long enough for a deadline to pass. ## What Changed - Add `TM1`, `TM2` and `TM3` to `--faults` (`time-faults.ts`, `time-faults-config.ts`): - **TM1 freeze**: send `SIGSTOP` to the server process only, for 95–150 s, then send `SIGCONT`. The runner and Postgres continue. The pause is longer than the controller-instance lease (90 s) and the execution-control deadline (60 s). - **TM2 sleep**: send `SIGSTOP` to the server and all of its descendants. While they are stopped, move every database time back by the age. Then send `SIGCONT`. This is a stand-in for a laptop sleep. - **TM3 aged outage**: send `SIGKILL` to the server tree (as K1). While it is down, move every database time back by the age. Then start a replacement. With `--workloads G1 --triggers approval`, an approval stays pending for a day across a restart. - Add `--time-age-hours` (1–72, default 26). TM faults run only with `--env local`, and not with storms, crash points or K14. - Aging is one transaction. It moves every `timestamp` column, and every time inside a `jsonb` column (ISO strings in their own format, and epoch milliseconds in the wave's window), back by the same interval. Triggers are off for the shift. Copies of a time in JSON (for example the comment `updatedAt` in a continuation receipt) still match their columns. The first wave showed that a shift of the columns only causes false failures. - The identity, state and time columns of the time-bound tables are written before the shift to `<wave>/time-faults/<faultId>.json`. Per-table counts go to the fault receipt (`timeFault`) and the timeline (`time_fault_aged`). - The observer and grader read through `TimeShiftedApi`. It adds the age back to each time that a shift moved, so R4, R5, U1–U4 and the speed budgets use real time. - Make small additions to the registries: `types.ts` (fault kinds, receipt field, option), `config.ts`, `scheduler.ts` (a seeded freeze length only for TM tasks, so the plans of existing seeds do not change), `supervisor.ts` (the freeze and aging hooks), `main.ts` (one line for the API) and the README. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 30 files, 368 tests pass. The new tests are: - `time-faults.test.ts`: option parsing, freeze order, Postgres exclusion, statement generation, and reading times back at their real time. - `time-faults-db.test.ts`: aging runs on real embedded Postgres. The test checks that trigger rewrites are suppressed, that JSON copies match their columns, that text columns do not change, that the evidence file is written, and that an age the wave nearly outlived is refused. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (local, codex), each run from a separate worktree and one wave per slot: | Wave | Scenario | Result | | --- | --- | --- | | `w-20261110T031929-s301` (A1,C1 × TM1–TM3) | First run, before JSON aging | t01: TM1 missed, because A1 finished first. t02: pass. t03 (C1): its turn completed after TM2 and failed with `native_session_lease_lost`; a follow-up comment woke a new run. After TM3, that run failed with `continuation_user_authorization_missing` and needed Retry. That second failure was a harness artifact: the receipt kept the comment's `updatedAt` in JSON, which the first version did not shift. JSON aging fixes it. | | `w-20261110T041910-s303` (B5 × TM1–TM3) | Freeze, sleep, aged outage | t01 and t02 crossed TM1, then TM2. Both turns completed after the thaw and failed with `native_session_lease_lost`, so both needed Retry (R3). This is a **product failure**; see below. t03 (TM3, 26 h aged outage): **pass**, with no Retry. | | `w-20261110T050125-s302` (G1 × TM3, `--triggers approval`) | Aged wait: an approval pending for 26 h across a restart | **Pass** (2/2). Both approvals were still pending after the restart and the board decided after the fault: t01 rejected, so the task is cancelled; t02 approved, so it is done with exactly one subtask. No Retry; U1–U4 pass. | | `w-20261110T051535-s304` (B5, TM2 with `--time-age-hours 1`, then TM1; one task at a time) | Attribute the failure | t01 (TM2 sleep, only 1 h): its turn completed after the thaw, then failed with `native_session_lease_lost`, so the task needed Retry. This reproduces the product failure. t02 (TM1 alone, 113 s server-only freeze): **pass**. The failure follows the database clock jump, not the pause. | **Product failure found (not fixed in this PR).** A local native run whose host sleeps longer than the 20-minute execution lease loses its completed work. After the wake, the provider finishes the turn and the brief is saved. Then result acceptance (`native-session-executor.ts`, the transaction at about line 12031) requires `native_run_finalizations.lease_expires_at > clock_timestamp()`. The database clock has moved on, so the run fails with `native_session_lease_lost`, and the task shows "Couldn't finish this attempt … Retry". No other boot claimed the lease: the owner, attempt and controller identity are unchanged, and the server's monotonic deadline did not expire. The lease is fenced by wall-clock expiry alone. A safe fix changes lease fencing semantics, which is a heartbeat-boundary decision, so I report it here rather than include a fix in this harness PR. Reproduce with `--workloads B5 --faults TM2` (`--time-age-hours 1` is enough; see `w-20261110T051535-s304`). ## Risks - Harness-only change; no product code changes. Waves that do not use TM faults are unchanged. `TimeShiftedApi` returns responses unchanged when no shift was applied. - `SIGSTOP` goes only to the wave server's own process tree. The tree is found from the server PID, and Postgres is excluded. Every frozen process gets `SIGCONT` in a `finally` block and in a process `exit` hook. - Aging is a model: the database becomes what it would be if everything happened the age earlier. Times outside Postgres (in-memory timers, runner state files), times inside `text` columns, and HMACs over a time are not moved. The README lists these limits. Agent JWT expiry is not covered: native runs mint a token for each tool call. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
2c2186706e |
test(restart-chaos): human-in-the-loop journeys (Q2, Q3), journey triggers and a journey x fault matrix (#20448)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane while real agents work, and grades each task. > - Almost every harness workload is "the agent does a file task and finishes". Few workloads have a person in the loop. > - On 2026-11-09, production failures came from journeys with a person in the loop: a saved answer that could not reach a dependency-blocked task (#20363), and an app-action approval that a routine OAuth refresh or a one-hour deadline made invalid (#20340). > - No workload exercised these journeys, so no wave could catch them. > - This pull request adds journey workloads with deterministic oracles and journey-specific fault triggers. It also adds a journey × fault matrix and a way to run a wave against an older build. > - The benefit: one command crosses every journey with every fault kind, and the same wave proves red before a fix and green after it. ## Linked Issues or Issue Description Refs #20363, Refs #20340, Refs #20336, Refs #20432 ## What Changed - New workload **Q2** (`journeys.ts`, `journey-driver.ts`): the agent saves a question with its native `request_human_input` tool. The board adds an unfinished `backlog` prerequisite, marks the task `blocked` by it, and answers through the task-thread respond route. The oracle requires that the answer reaches the agent in bounded answer mode while the task stays blocked, that the prerequisite relation stays, that no run has `issue_dependencies_blocked`, and that no stale Retry notice stays at rest. Then the board finishes the prerequisite and the agent finishes the task. - New workload **Q3**: a harness-owned OAuth 2.1 + MCP app on loopback (`journey-oauth-app.ts`, discovery, dynamic client registration, PKCE, rotating refresh tokens, one write tool). The board connects it through the Apps routes and sets the tool to Ask first for a scripted `process` agent. When the review is pending, the harness moves the review's saved timestamps and the token expiry back 61 minutes in the wave's own database, runs Check connection (which refreshes the token through the app), and approves. The oracle requires one review, a refresh between request and decision, an accepted approval, exactly one `tools/call` with the reviewed arguments, an `executed` request, and an agent report that carries a receipt only the provider result contains. - New triggers `asked` and `answered` fire a restart fault 0–3 s after the saved request is first seen and after the board's response. They work for the existing **Q1** saved question (#20432) too. The board answers before an `answered` fault fires and after every other planned fault. - New `--matrix` option: one task per workload × adapter × (a `K0` no-fault baseline, each restart fault at each of the workload's own triggers, each provider message fault). `K0` is a new task fault kind with no fault. - New `--server-from <checkout>` option: every server start and restart of the wave runs another prepared checkout, for example the parent of a fix, with the same harness code. - Q3 setup allows unpriced usage on the wave's company budget policy, because a scripted process run records usage with no price and the default policy pauses the whole company. - README documents the journeys, the triggers, the matrix and the red-before-fix recipe. Unit tests cover the oracles (pass and the pre-fix failure signatures), the driver's ordering, the matrix plan, option validation and the OAuth app. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 295 passed. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passed. - Live local waves, codex, one wave at a time (shared machine lock), `--concurrency 2`, at most 3 tasks: | Wave | Server build | Command | Result | | --- | --- | --- | --- | | `w-20261110T024308-s106` | this branch (master `5570debc5b` base) | `--workloads Q3 --faults K1 --matrix --seed 106` | 3/3 pass (K0, K1^asked, K1^answered); the app saw one refresh before each decision and one `tools/call` | | `w-20261110T024447-s106` | `--server-from` parent of #20340 (`0e325e8d52`) | same | 3/3 fail as expected: approve returns 409 `action_expired` ("Tool review has expired"), no `tools/call`, task stays `in_review` | | `w-20261110T024644-s107` | this branch | `--workloads Q2 --faults K1 --matrix --seed 107` | K1^asked and K1^answered pass. The K0 cell failed only because the oracle then required a comment; the agent had reported `ANSWER-…` in its bounded turn's run result 14 s after the answer, while blocked. The oracle now accepts either. | | `w-20261110T030341-s107` | `--server-from` parent of #20363 (`eebdfe71b1`) | `--workloads Q2 --faults K1 --matrix --seed 107` | 3/3 fail as expected: `answer_reached_agent_while_blocked`. No run starts between the answer and the finished prerequisite (6 minutes of ready server). | | `w-20261110T033426-s108` | this branch, rebased onto #20432 | `--workloads Q1,Q2 --faults K1 --triggers asked,answered --tasks 4 --seed 108` | Q1^asked and Q2^asked pass. Q1^answered and Q2^answered fail on a product gap (below). | | `w-20261110T020037-s102` | this branch, before the rebase | `--workloads Q1,Q2,Q3 --faults K1 --triggers asked,answered --tasks 3 --seed 102` | Q1 with `asked` and `answered` K1 restarts: both done with the answer. | Waves `…-s107` (pre-fix) and `…-s106` (pre-fix) used the same harness code as the green waves. Only the server build changed. **New product failure (not fixed here).** A K1 restart 0.3–3 s after the board's answer can lose the answer turn's native run: - `w-20261110T033426-s108/t04` (Q1, K1 2.9 s after the answer): run `a40cc8f7` resumes with `resume_dead_runner`, then fails `runner_state_identity_mismatch: authority_indeterminate` (`adapter_failed`). The task shows "Couldn't finish this attempt. Your message is saved." and needs Retry. Retry works. - `w-20261110T033426-s108/t03` (Q2, blocked task, K1 0.27 s after the answer): run `5b5355de` is `process_lost`. Its bounded retry `92a57f50` fails `native_session_recovery_failed` (`session.snapshot` failed). The Retry click starts `0e6aa9d1`, which fails `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_commands_unsettled(turn.start.failed)`. The task then waits with `retryWait.reason=process_identity_missing` and stays blocked without the answer. - Suspected path: native runner restart reconciliation of the runner state identity (`resume_dead_runner`, journal command settlement) in `server/src/services/native-runtime/` and `packages/paperclip-runner/`. Host load was 8–14 during the wave. ## Risks - Test harness only. No product code changes. - Q2 depends on the model following its instructions. The oracle accepts the answer in either a comment or the bounded turn's run result, because both were seen. - Q3 shifts persisted timestamps only in the wave's isolated database. It does not shift unrelated tables, so it simulates elapsed time for the review and the token, not for the whole instance. - `K0` and the new triggers add values to shared registries (`types.ts`, `scheduler.ts`). Existing plans keep their meaning. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
5ab98b0063 |
test(restart-chaos): add database contention workload and universal checks U1-U4 (#20421)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks on an isolated control plane and grades them. > - Two production failures on 2026-11-09 passed the harness. Its tasks made little output and little accounting, so database write contention never happened (#20385). Its grading was per workload, so a task stuck for a reason no scenario targeted was not caught (#20361). > - The harness needs a contention workload and checks that apply to every task. > - This pull request adds the L1 workload, a `--db-pressure` knob, and the universal checks U1–U4. > - The benefit is that the #20385 failure class and unexplained stuck or misleading tasks now fail a wave. ## Linked Issues or Issue Description Refs #20385, #20383, #20361. **What happened?** The harness ran real embedded PostgreSQL, but no task streamed enough run-log output while usage receipts committed. The #20385 receipt race (SQLSTATE 40001, retained outbox, `native_event_replay_conflict`) could not occur. Also, no rule checked every task for an owner or an actionable item, for retry loops, for diagnosable failures, or for stale notices. **Expected behavior** A chaos wave can create run-row contention between run-log writes and accounting writes and check that accounting stays exact. Every task of every wave gets scenario-independent liveness and notice checks. **Steps to reproduce** Run an L1 wave with `--db-pressure 4` against the parent of the #20385 merge (red), then against master (green). Commands are in Verification. ## What Changed - `contention.ts` (new): workload L1 (five 3,000-line shell outputs as five tool calls, then a brief), its accounting oracle, and `RunRowPressure`, the `--db-pressure <writers>` knob. The writers run `update heartbeat_runs set updated_at = updated_at` on the wave company's running runs, in the wave's own isolated database only. - L1 oracle: no `native_event_replay_conflict` or `runner_event_commit_stalled`; each producer event journaled once; a terminal receipt for each succeeded native run; finished runs accounted; per-run cost equals its cost events; the board's spend equals the cost-event sum with no runs pending accounting; and proof that the task produced contention. - `universal.ts` (new): U1 liveness, U2 no silent retry loop, U3 diagnosable failures, U4 no stale notices. `grader.ts` grades them on every task. U1, U2 and U4 fail a task. U3 is a warning. - `invariants.ts`: `readUniversalDbEvidence` reads failed-run error text and recovery-action causes from the wave database at wave end, for U3. - `universal-audit.ts` (new): runs U1–U4 over saved wave evidence to measure false positives and bound headroom. - `contention-db.test.ts` (new): on real PostgreSQL, under the pressure writers, the pre-#20385 repeatable-read receipt write exhausts its retries with 40001 and the current writer commits. - Small registry edits: `types.ts` (L1 id, `dbPressure`, optional `contention` and `universal` fields), `workloads.ts` (L1 entry), `config.ts` (`--db-pressure`, L1 not a default), `main.ts` (start and stop the writers, read database evidence before teardown), `ledger.ts` (universal verdicts and warnings in the ledger and `SUMMARY.md`). - README: "Database contention" and "Universal checks" sections, with the audit table and bound reasons. ## Verification - `contention-db.test.ts` (real embedded PostgreSQL): with `--db-pressure` writers (4), the pre-#20385 repeatable-read receipt write exhausted its retries with 40001 in 8 of 8 attempts; the current READ COMMITTED `persistUsageReceipt` committed 8 of 8. This is the harness-level red/green for #20385. - Live L1 waves (`--env local --adapters codex,claude --workloads L1 --faults K1 --concurrency 4 --tasks 4 --seed 20385 --db-pressure 4`), red on the #20385 parent `bf818280f2` with this harness overlaid and green on this branch: queued behind the shared one-wave-at-a-time lock and not yet run. The first red attempt failed at startup (the fresh checkout lacked a built plugin SDK; no tasks ran). Live results are not claimed here. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: all harness unit tests pass, including the new `universal.test.ts`, `contention.test.ts` and `contention-db.test.ts`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Universal-check audit over 74 saved waves (1,592 tasks): U1 7 flags (one C1 stall and six R1 runs queued 8 min behind a retry row, the #20361 shape), U2 1, U3 22, U4 0. None of U1, U2 or U4 flags a passing task or an all-pass wave. U3 flags 3 passing tasks (all `adapter_failed`), so it ships as a warning. ## Risks - U1, U2 and U4 can now fail a task that passed R1–R6 and T. The audit found no such case in 70 saved waves. The bounds are environment-configurable. - `--db-pressure` is opt-in and writes no values. It writes only to the wave's own temporary database. - Other agents edit the same harness. This change keeps registry edits small; `grader.ts` and `invariants.ts` changes are additive. - No product code changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0c7a628fd1 |
test(chaos): read remote R1 files from the execution workspace and record why a read failed (#20440)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - In replays w-20261110T010031-s9913 and w-20261110T012332-s9950 the tasks recovered (R1, R3, R6 pass), but all 506 content reads returned 422, even after the closing save left the workspace clean (`snapshotId` set, no writer, `location: null`) > - The harness discarded each error, so the waves could not show which 422 the server returned > - This pull request asks for the execution workspace explicitly and records the last error per file in the task record and the timeline > - The benefit is that the next wave names the exact cause instead of a blind `module_present` failure ## Linked Issues or Issue Description Follow-up to #20348. **What happened?** Every file read for a finished Daytona R1 task returned 422, and the response was discarded. **Expected behavior** Failed reads leave their reason in the evidence. **Actual behavior** Only the outcome, a `null` file, was kept. ## What Changed - File reads add `workspace=execution`, so the server reads the issue's execution workspace and not the `auto` choice. - `readRemoteWorkspaceFile` and `captureRemoteWorkspaceFiles` report each failure through a callback. The observer keeps the last message per file in `record.remoteFileErrors` and logs `remote_file_read_failed` once per distinct message. - The remote-workspace file reader stays in `remote-workspace-files.ts`; `observer.ts` is at 993 lines. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 263 passed. The test server now requires `workspace=execution`, so the existing remote tests were red before, and a new test records the 422 `workspace_unavailable` message. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - If a Daytona task's files only exist in a project workspace rather than its execution workspace, an explicit `execution` selector reports that clearly instead of falling back. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
1b52f359dd |
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
4158799d1b |
test(chaos): add provider message faults (PV1-PV3) and a saved-question scenario (#20432)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart chaos harness (`tests/restart-chaos`) tests that agent work survives faults. > - The harness has two fault families: control-plane restarts (K*) and storage faults (ST*). > - On 2026-11-09, two production failures came from a third family. The provider was alive, but a protocol message was late or missing (#20330, #20336). > - The harness could not produce that family, so it did not catch these failures. > - This pull request adds provider message faults (PV1 delay, PV2 drop, PV3 duplicate) on the real path between the runner and the Codex app-server. It also adds a saved-question scenario (Q1) that uses the faults. > - The benefit: the harness can now find this class of failure, and it fails against both pre-fix commits. ## Linked Issues or Issue Description Refs #20330 Refs #20336 **What happened?** A Codex interruption terminal that arrived after 18 s failed a governed wait (#20330). A dropped terminal after a saved handoff failed shutdown settlement with "Couldn't finish this attempt" (#20336). The chaos harness could not inject either fault. **Expected behavior** The harness can delay, drop or duplicate a chosen provider protocol message for one task. A scenario fails on the pre-fix product code and passes on master. **Steps to reproduce** Run the wave commands in Verification. Run them on master and in a worktree at the pre-fix commit with `tests/restart-chaos` copied from this branch. ## What Changed - `provider-faults.ts` and `provider-shim/`: the PV1–PV3 fault family. - A resolve hook goes into the wave's server through `NODE_OPTIONS`. It answers only the `@openai/codex/package.json` lookup that product code uses to find the local Codex. It returns a stand-in package in the instance temp root, and then removes itself from `NODE_OPTIONS`. - The stand-in `codex` bin is a stdio shim. It starts the real Codex with the same arguments and relays JSON lines unchanged. It changes only the message that the task's rule names: the `turn/completed` after a `turn/interrupt` (`terminal`) or the response to `turn/interrupt` (`interrupt_ack`). Each rule applies once per task. The task nonce binds the rule to the task. - Every decision goes to `<wave>/provider-faults.jsonl`. A PV task fails `provider_fault_applied` when its fault did not happen. - There is no product change. Thus the same harness runs against older product commits. - `question-handoff.ts`: Q1 workload. The agent saves one question with `request_human_input`. After the asking run settles, the board answers through the respond route. The oracle checks that the question was asked once, the asking run succeeded, no run failed, nothing ran before the answer, and the brief has the answer. - PV faults run only with `--env local --adapters codex`, only with workloads that interrupt a turn (Q1, S1, G1), and not in the same wave as restart faults. A PV1 delay passes only when the held message was released. - Registry edits: `--faults` accepts PV1–PV3. New options are `--provider-fault-message` and `--provider-fault-delays` (a sweep, cycled across tasks). The edits are in `types.ts`, `config.ts`, `scheduler.ts`, `workloads.ts`, `observer.ts` (Q1 board step) and `main.ts` (install the seam and add the check). - README section with commands, the seam, and what is not covered. ## Verification - Unit tests: `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`. 23 files and 279 tests pass on the final head. The new tests drive the real shim against a stand-in provider for each kind (delay, drop, duplicate, interrupt ack, once per task, held message discarded on exit, a multibyte character split across writes). They also run the resolve hook in a child Node process. - Typecheck: `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live waves (real Runner, real Codex 0.160.0, gpt-6.1-sol, macOS arm64): Pre-fix runs use a worktree at the merge parent with `tests/restart-chaos` copied from this branch. Common arguments: `--env local --adapters codex --workloads Q1`. | Scenario | Product code | Wave | Result | | --- | --- | --- | --- | | 1. PV1, terminal 18 s late (`--faults PV1 --provider-fault-delays 18 --tasks 1 --seed 101`) | master (`8004ad069e`) | `w-20261110T000333-s101` | pass: terminal held 18 s and released, asking run `succeeded`, one question, brief has the answer | | 1. same | `34c1e6469c` (parent of #20330 merge) | `w-20261110T014734-s101` | **fail**: Codex stopped at 14.9 s with the terminal still held, asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt. Your message is saved.", Retry needed (R3) | | 1. sweep 5/14/25/29 s (`--faults PV1 --provider-fault-delays 5,14,25,29 --concurrency 2 --tasks 4 --seed 102`) | master | `w-20261110T015426-s102` | 4/4 pass | | 1. same sweep | `34c1e6469c` | `w-20261110T015708-s102` | 5 s and 14 s pass; **25 s and 29 s fail** (`native_provider_terminal_failed`, Retry needed): the old 15 s drain | | 2. PV2, terminal dropped (`--faults PV2 --tasks 1 --seed 103`) | master | `w-20261110T015239-s103` | pass: terminal dropped, asking run `succeeded` through verified shutdown, one question, nothing ran before the answer | | 2. same | `a67b936885` (parent of #20336 merge) | `w-20261110T015010-s103` | **fail**: asking run `failed native_provider_terminal_failed`, banner "Couldn’t finish this attempt", Retry needed (R3) | The first master wave exited 1 for one cold `agent_boot` speed sample (8.6 s against a 7 s p95 budget), before any fault, while a preflight build ran in parallel. Its task passed. No new product failure was found on master. ## Risks - Test-only. No product code changes. - The shim adds one Node process in front of Codex for PV waves only. Other waves do not change. - The nonce binds a rule to a Codex process. If one Codex process serves two tasks, the second task's traffic can bind it too. Q1 tasks use separate sessions. - Claude is not covered. Claude runs through the ACPX sidecar against a hash-verified provider pack, and its ACP protocol has no interrupt acknowledgement. A Claude shim needs a different seam. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use, code edits and local execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
490d97bf1e |
test(perf): guard Mine against waiting on the company-wide workspace list (#20419)
> Follows #20388 (merged as |
||
|
Nicky LeachandClaude Opus 5.5
|
ff4ba7d319 |
perf(runner): run launch artifacts in place from a trusted read-only root (#20428)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust Runner (`paperclip-runnerd`) starts provider sidecars and proxies from qualified launch artifacts: a Node runtime and the scripts it runs > - Since #12699, Runner copies each artifact into a sealed memfd and checks its SHA-256 on every verification. This defends against a path swap or an in-place rewrite between the check and the exec > - On a hosted agent computer (4 vCPU), the copy and digest of the 127 MB Node takes about 0.26–0.30 s warm and about 1.2 s right after a wake. This is on the path from `session.open` to the sidecar spawn, and again on every warm `run.attach`. Each Runner also holds about 125 MiB of shmem for the copy > - On that computer, the artifacts live on the Runner ROM: a read-only erofs image that the deploy pins by digest. The threat that the memfd defends against does not exist there. On local hosts the artifacts are writable, so the memfd must stay > - This pull request adds an explicit, path-scoped, default-off `--trusted-launch-root <dir>`. Runner runs an artifact in place only when its open descriptor proves that it is on an immutable mount under that root. Everything else keeps the sealed, hashed snapshot > - The benefit is a shorter `session.open` and `run.attach` on agent computers, and about 125 MiB less shmem per Runner, with no change where the flag is absent ## Linked Issues or Issue Description **What existing behavior does this improve?** Launch-artifact verification in `paperclip-runnerd` (`qualified_launch.rs`). Its callers are the ACPX `verified_transport` (every `session_config`, including the park rebind) and the OpenCode and Cursor Cloud proxy launches. **Subsystem affected** Rust Runner (`packages/paperclip-runner/runner/crates/runner-core`). **Current behavior** Every verification opens the artifact, copies all bytes into a sealed memfd, and hashes them. Node is 127 MB. The cost is described in the Thinking Path. **Proposed behavior** When the host passes `--trusted-launch-root /opt/paperclip/runner` and the root passes its checks, Runner holds each admitted artifact open in place and executes it through `/proc/self/fd/N`. It does not copy or re-hash it. Without the flag, nothing changes. **Reason and benefit** See the Thinking Path. Refs #12699 (introduced the sealed memfd). Refs #20423 (overlap, see below). ## What Changed - `qualified_launch.rs`: new `TrustedLaunchRoot` and `set_trusted_launch_root` (process-global, set once at startup, like `--task-memory-cgroup`). - **Root checks at startup.** The path must be absolute, already canonical (no symlink or `..`), a directory, and on a read-only, exec-permitted erofs or squashfs mount. If a check fails, runnerd logs a warning and keeps today's behavior. - **Per-artifact checks, on the open descriptor.** After today's checks (regular file, not a symlink, not group- or world-writable, open-time identity), the descriptor must be a regular file on the root's device. Its kernel path (`readlink /proc/self/fd/N`) must lie under the root by path components, so `/opt/paperclip/runner2` does not match. Its mount must pass `fstatvfs` (`ST_RDONLY` set, `ST_NOEXEC` clear) and `fstatfs` (erofs or squashfs magic). If any check fails, Runner logs the reason and uses the sealed snapshot. - **Admitted artifacts** become `VerifiedProcessArtifact::in_place_on_trusted_root` (crate-private). The held file goes through the existing `inherited_artifact` path, so the exec is `/proc/self/fd/N` and `PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE` still names a descriptor that nested launches can reopen. - New `verify_assigned_executable_launch_artifact` never uses the root. `stdio_mcp.rs` now calls it, so host-assigned stdio MCP binaries always keep the sealed snapshot. - `paperclip-runnerd.rs`: parse `--trusted-launch-root <dir>`. Older runnerd builds ignore unknown flags, so a host can pass it before every Runner understands it. - The trusted path is Linux-only. On macOS and Windows, `TrustedLaunchRoot::open` always refuses. - `tests/runner-e2e/SECURITY.md`: documents how the Kimi and Pi provider-death fixtures behave under a trusted root. ### Decision: the trusted path drops the digest The trusted path does not hash the artifact, not even once per process: - The ROM image is pinned by digest at deploy time. - On the agent computer, the Launcher computes the expected `--acpx-sidecar-command-sha256` and `--acpx-sidecar-script-sha256` from the same ROM files immediately before it starts Runner. A re-hash of an immutable file can only repeat that result. - One hash per process would not help `session.open`, because each task starts a new Runner process. - The mount check is the guard: erofs and squashfs have no write path, so the bytes cannot change while the descriptor is open. ### Fixtures that assert the sealed memfd - Untrusted-mode assertions are unchanged. The fixtures do not pass the flag. - Kimi (`kimi-provider-fault.py` `checked_parent_descriptor`): with a trusted root, the parent descriptor is the ROM's Node file (one link). The existing linked-descriptor branch handles it: same inode as `/proc/<pid>/exe`, and the complete bytes hashed against `nodeSha256`. That branch never accepts the file as a memfd. The existing test `test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim` covers it. - Pi provider-death admission binds `/proc/self/fd/3|7` to the native distribution snapshot's Node. The TypeScript ACPX closure copies that file whether or not a trusted root is set, so the flag does not affect it. ### Overlap with #20423 #20423 (draft) moves the same copy and hash to Runner boot and reuses the result. Where this flag is set (Cloud agent computers), there is nothing left to precompute, so #20423 is mostly redundant there. #20423 still helps hosts without a trusted root (local and sandbox Runners). This PR does not change #20423. ## Verification All runs are on Linux (aarch64, Rust 1.97.1, in a container): - `cargo fmt --manifest-path runner/Cargo.toml --all -- --check` - `cargo check --manifest-path runner/Cargo.toml --locked --workspace` - `cargo test -p paperclip-runner-core --lib -- qualified_launch trusted_root commonjs`: 12 passed. The new tests are: - `an_artifact_under_an_immutable_trusted_root_runs_in_place_without_a_digest`: trusted-root hit, with no memfd, the same inode as the source, and no re-hash. - `an_artifact_outside_the_trusted_root_keeps_the_sealed_snapshot`: a path outside the root is sealed, and a wrong digest still fails. - `a_sibling_with_the_root_as_a_name_prefix_is_outside_the_root`: `rom2/` does not match `rom`. - `a_root_that_is_not_read_only_falls_back_to_the_sealed_snapshot`: the real probe refuses a writable root at startup. A per-artifact mount failure falls back, and a wrong digest still fails. - `the_real_mount_probe_rejects_a_writable_filesystem` - `symlink_and_parent_escapes_from_the_root_fall_back`: a directory symlink out of the root, a `..` path out of the root, and a final-component symlink (still refused). - `a_root_must_be_absolute_canonical_and_a_directory` - `without_a_trusted_root_launch_artifacts_are_sealed_and_hashed`: flag absent, behavior unchanged. - `host_assigned_executables_ignore_the_trusted_root` - `trusted_root_artifacts_execute_the_held_file_and_reopen_it_for_nested_launches`: Runner execs from the held file, and a child re-execs `$PAPERCLIP_VERIFIED_RUNTIME_EXECUTABLE`. Both report the source file as `/proc/self/exe`. - `cargo test -p paperclip-runner-core --test stdio_mcp --test process_supervisor`: 4 and 8 passed. - Ad hoc, not committed: a privileged container with a real loop-mounted squashfs. The real `fstatvfs`/`fstatfs` probe admitted a `ro` mount and refused a `ro,noexec` mount. The unit tests replace only that mount probe, because CI cannot mount filesystems. - Not verified: a run on an agent computer with the real erofs ROM, and the latency and memory savings there. The Cloud Launcher change that passes the flag is a separate follow-up. ## Risks - **This change needs review from a security owner before merge.** It removes the sealed copy and the per-launch digest for admitted artifacts. That relies on the host's assertion that the root is a digest-pinned immutable image, plus the descriptor checks above. - **Misconfiguration fails closed.** A wrong, writable, `noexec`, non-canonical or missing root only loses the speedup, and runnerd logs a warning. There is no global "trust everything" switch, and Runner never selects a root from the filesystem it finds. - **Residual trust.** On an admitted file, Runner no longer detects a ROM image that differs from what the launch profile expected. On the agent computer, the Launcher derives the expected digests from the same ROM. Another host that passes the flag must pin its image the same way. An attacker with root in the guest (who could write the backing block device or remount) is out of scope; such an attacker can already control runnerd. - **Allowlist.** Only erofs and squashfs are accepted, because they cannot be written through any mount. A read-only bind mount of a writable filesystem is refused. A ROM that changes to another filesystem type will silently fall back to the sealed path (slower, still safe). - No migration. Behavior is unchanged when the flag is absent. Older runnerd builds ignore the flag. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, container builds). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b184e40108 |
perf(ui): serve every project list from one includeArchived read (#20395)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The board UI reads the company project list in many places: the
sidebar, starred projects, the command palette, the new-task dialog, and
archive-aware pages such as the Mine inbox
> - Active-only consumers read `GET /projects`, and archive-aware ones
read `GET /projects?includeArchived=true`, under two React Query keys
> - A Mine load therefore fetched the same large payload twice (580–590
KB each on our dev instance)
> - The two responses differ only by archived rows; the server's active
list is exactly the rows where `archivedAt` is null
> - This pull request makes every consumer share the
`includeArchived=true` query, and active-only consumers filter it with
`select`
> - The benefit is one project read per page instead of two
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Duplicate project list reads on page load.
**Subsystem affected**
Board UI project queries (`ui/src/lib/company-projects-query.ts` and its
consumers).
**Current behavior**
A cold Mine load sends both `/companies/:id/projects` (sidebar) and
`/companies/:id/projects?includeArchived=true` (inbox).
**Proposed behavior**
One `includeArchived=true` read serves both. Active-only consumers apply
`selectActiveProjects`.
**Reason and benefit**
The project list is one of the largest shell payloads. Reading it once
saves a large response on every page that has both kinds of consumer.
## What Changed
- New `companyProjectsQuery(companyId, { includeArchived? })` in
`ui/src/lib/company-projects-query.ts`. It always uses the
`includeArchived: true` key and fetch. For active-only use it adds a
stable, module-level `select` that keeps rows without `archivedAt`.
- 22 files that read the active list now spread
`companyProjectsQuery(...)`, and so do the plugin SDK project picker and
16 archive-aware pages that wrote the shared key and fetcher inline.
Their `enabled` and other options are unchanged. `OnboardingWizard`'s
one imperative read is unchanged. `pages/Inbox.tsx` already reads the
same key and fetcher, so it shares the entry without an edit.
- Invalidation: project mutations and live events invalidate
`queryKeys.projects.all(companyId)`, which matches the shared key. The
onboarding wizard invalidated the old active-only key; it now
invalidates `projects.all`.
- Tests: new `company-projects-query.test.tsx` (one request for both
consumers, correct filtering, and one refetch after `projects.all`
invalidation). Three existing tests (`CommandPalette`, `CompanyExport`,
`GitHubBotManagement`) now seed or expect the shared key.
- New browser check
`tests/perf/collections/shared-project-list.spec.ts`: after the Mine
inbox read completes, the page has sent no active-only project request.
## Verification
- `pnpm verify` passes.
- `ui`: the consumer suites (`CoordinatorSidebar`, `SidebarProjects`,
`SidebarStarredProjects`, `CommandPalette`, `NewIssueDialog`,
`AgentWorkPanels`, `CoordinatorNeedsYou`, `IssueDetail`,
`ProjectDetail`, `Projects`, `CompanyExport`, `ChatExecutionDefaults`,
`GatewaysList`, `Search`, `IssueProperties`, `persisted-query-cache`)
pass.
- Browser check:
`PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec
playwright test --config tests/perf/collections/playwright.config.ts
shared-project-list.spec.ts`. It passes on this branch and fails on
`master`.
## Risks
- A page that needs only active projects now downloads archived rows
too. For most companies that is a small difference; the duplicate read
it removes is a full copy.
- The server does not order either list, and consumers sort for display,
so filtering on the client keeps behavior.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser). 1M-token context window. Extended
thinking was off.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c7eb35809e |
perf(ui): read skills and routines only when an editor mounts (#20392)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI wraps every page in an `EditorAutocompleteProvider` that supplies slash commands to Markdown editors > - The provider read the full company skills and routines lists as soon as a company was selected, on every page > - The Mine inbox has no editor, but it still paid for both reads: 259–284 KB for skills and 225–333 KB for routines on our dev and staging instances, with skills taking 8.5–13.7 s on a saturated staging server > - Those reads compete with the inbox rows for the same server > - This pull request starts the two reads only when the first Markdown editor mounts > - The benefit is two fewer large reads on every page without an editor, and no change for pages that have one ## Linked Issues or Issue Description **What existing behavior does this improve?** The requests that every board UI page sends on load. **Subsystem affected** Board UI context (`ui/src/context/EditorAutocompleteContext.tsx`). **Current behavior** A cold load of `/:prefix/issues?view=mine` sends `GET /companies/:id/skills` and `GET /companies/:id/routines` although nothing on the page renders them. **Proposed behavior** The provider keeps both queries disabled until a component calls `useEditorAutocomplete()`. `MarkdownEditor` is the only caller, so the reads start when an editor mounts: a task composer, the new-task dialog when it opens, a comment box. **Reason and benefit** Each page without an editor saves two large reads. Pages with an editor read the same data at mount, before a user can type `/`. ## What Changed - `EditorAutocompleteProvider` holds a `requested` flag. Both queries are `enabled` only after it is set. - `useEditorAutocomplete()` sets the flag in an effect on first use. The flag stays set, so the lists stay current through the existing live-update invalidations for skills and routines. - New unit test `EditorAutocompleteContext.test.tsx`: no reads without an editor; one read of each list when two editors mount. - New browser check `tests/perf/collections/shell-editor-reads.spec.ts`: a Mine load sends no `skills` or `routines` request. ## Verification - `pnpm verify` passes. - `ui`: `npx vitest run src/context/EditorAutocompleteContext.test.tsx src/components/task-chat/TaskChatComposer.test.tsx` passes. - Browser check against a disposable local instance: `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts shell-editor-reads.spec.ts`. It passes on this branch and fails on `master`. - Manual: on a local instance the Mine load dropped from 34 to 32 API requests. Opening a task shows the composer; `/` lists skills and routines as before. ## Risks - Low risk. On the first editor mount the slash menu is empty until the two reads return. On `master` the same reads ran at app start, so a user who typed `/` within the first moments of a page with an editor saw the same gap. - `attention/summary` is not changed. The navigation's "Needs you" count renders it on the Mine page. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser). 1M-token context window. Extended thinking was off. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7e93ccb4b3 |
perf: fetch recent task summaries in one request (#20389)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI sidebar shows the person's recent tasks on every page, including the Mine inbox > - The sidebar refreshes those rows through `GET /companies/:id/issues/summaries`, which accepted at most five IDs > - `useRecentTasks` sent the chunks one after another, so a 25-task history cost five sequential requests > - On staging each request took 3–6 s, so the recent-task rows settled about 20 s after load and every request competed with the inbox rows > - This pull request raises the per-request cap to 50 and fetches any remaining chunks in parallel > - The benefit is one summaries request for a typical history instead of a serial chain ## Linked Issues or Issue Description **What existing behavior does this improve?** The refresh of the sidebar's recent tasks on page load. **Subsystem affected** Board UI (`ui/src/hooks/useRecentTasks.ts`) and the `/issues/summaries` route bound (`packages/shared`, `server/src/routes/issues.ts`). **Current behavior** A history of N recent tasks sends ceil(N/5) summaries requests in series. Measured on staging on 2026-11-09 for a cold Mine load: five requests of five IDs, started at 1.0 s, 4.3 s, 8.1 s, 12.3 s and 18.4 s, each taking 3.3–6.1 s. **Proposed behavior** Up to 50 recent tasks are one request. Longer histories send their chunks at the same time. **Reason and benefit** The staging server is saturated, so each extra serial round trip delays the sidebar and the inbox rows. One request removes four round trips from a typical load. ## What Changed - `ISSUE_SUMMARIES_MAX_IDS` is now 50 (was 5). The shared validator, the route's 400 message, the service guard and the OpenAPI description use the new bound. - `useRecentTasks` builds all chunks first and fetches them at most three at a time (`RECENT_TASK_SUMMARY_CONCURRENCY`). The recent list has no size cap, so this keeps a long history from becoming a burst. The cache key, omission handling and pruning are unchanged. A failed chunk still fails the batch, so nothing is pruned on a partial failure. - Tests: the route test accepts exactly `MAX` IDs and rejects `MAX + 1`; a hook test checks that at most three chunks are in flight; the API test uses the constant; `useRecentTasks` tests cover a 25-task history in one request and a 103-task history with three chunks in flight together, with and without a failed chunk. - New browser check `tests/perf/collections/recent-task-summaries.spec.ts`: a Mine load with a 12-task history sends summaries requests that each carry all 12 IDs, and the server answers 200 with all 12 tasks. ## Verification - `pnpm verify` (selected Vitest files) passes. - `server`: `npx vitest run src/__tests__/issue-summaries-routes.test.ts src/__tests__/issue-display-status.test.ts` passes (embedded PostgreSQL). - `ui`: `npx vitest run src/hooks/useRecentTasks.test.tsx src/api/issue-summaries.test.ts src/components/task-chat/TaskChatBlockerLinks.test.tsx src/components/CoordinatorSidebar.test.tsx src/components/SidebarRecentTasks.test.tsx` passes. - Browser check against a disposable local instance: `PAPERCLIP_COLLECTION_PERF_BASE_URL=http://127.0.0.1:<port> pnpm exec playwright test --config tests/perf/collections/playwright.config.ts recent-task-summaries.spec.ts`. It passes on this branch. With `master`'s hook and constant it fails: three requests of 5, 5 and 2 IDs. ## Risks - Low risk. The route still validates UUIDs, deduplicates, and applies the same read authorization per row. Blocker attention is computed with set-based queries, so 50 IDs is the same number of queries as 5. - During a rolling deploy, a new UI against an old server would get a 400 for a history above five tasks. The server serves the UI bundle, so both change together on a normal deploy. The sidebar keeps its stored snapshot when the refresh fails. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, browser). 1M-token context window. Extended thinking was off. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
eebdfe71b1 |
fix(ui): show submitted screenshots and Markdown in steering messages (#20367)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People send follow-up messages while an agent works. > - The queue above the composer holds those messages until delivery. > - The queue showed raw Markdown and had no screenshot previews. > - Submitted screenshots stayed in the composer until the save response arrived. > - This pull request shows formatted text and previews in the queue at submission. > - It keeps upload receipts so failed saves can restore the draft and its files. ## Linked Issues or Issue Description **What happened?** Send a message with Markdown and screenshots while an agent works. The queue shows Markdown source. It has no screenshot previews. The composer keeps the submitted screenshots until the save response arrives. **Expected behavior** Show formatted text and screenshot previews in the queue. Clear the submitted previews with the text. Restore them if the save fails. Keep files added for the next message. **Steps to reproduce** 1. Open the Screenshot Submission Storybook case under Steering and queued messages. 2. Add a screenshot and formatted text. Select Send. 3. Leave the simulated save pending. Inspect the queue and composer. 4. Reject the save. Confirm the draft and screenshots return. Retry and complete the save. Related work: #19321 bounds the queued-message list. This fix keeps that scrolling behavior. ## What Changed - Hide submitted attachment previews as soon as Send clears the text. Keep their upload receipts until the request settles. - Use the shared Markdown renderer in queued rows. Show up to three compact screenshots with a count for more images. Strip standalone image syntax from the text preview. - Keep paragraph boundaries. Wrap thumbnails and controls within narrow rows. - Load queue previews only from local attachment and asset content paths. Add an opt-in image omission mode to the shared Markdown renderer for compact queue text. - Correct the case lookup API summary so the existing search test passes with the full catalog. - Add regressions for successful, rejected, and uncertain saves. Check files added to the next draft. - Add a production-component Storybook case with controlled save responses and a desktop/mobile browser test. - Document the queue display and draft recovery behavior in DESIGN.md. ## Verification - Red: the original components fail five new unit cases. The desktop and mobile browser cases also fail on raw Markdown. Screenshots show the previews still in the composer. - Green: 256 composer, queue, and shared Markdown unit tests pass. - Related coverage: 466 thread, draft, and attachment tests pass. One existing test is skipped. - Three browser cases pass. Desktop and mobile cases check pending save, rejection, retry, successful save, and steering. A 320px case checks separate paragraphs with no margins, three loaded thumbnails, the additional-image count, and auto-send controls. It checks that all controls and images fit within the queue. - UI typecheck and design token gates pass after the review fixes. The initial implementation also passed the UI production build, Storybook catalog checks, and 30 map/catalog tests. The final PR workflow passed typecheck, build, all required test suites, and browser E2E checks. - Browser command: `pnpm exec playwright test queued-message-screenshots.spec.ts --config=tests/storybook-visual/playwright.config.ts --project=stories`. Use `STORYBOOK_TEST_BASE_URL` to target an owned Storybook server. - Review regressions first failed with two unit failures and an inline-paragraph browser failure. They pass after the fixes. - Initial CI failed the case lookup search test. It reproduced locally and now passes all three cases after the API summary correction. - Initial CI also failed process identity inspection during a server shutdown test. All 22 focused shutdown tests pass locally without source changes. The cause remains unconfirmed; the final CI Runner support suite also passed without changes to process cleanup. - The browser cases mount production components with simulated upload/save and steer responses. They do not prove backend persistence or live provider delivery. - `pnpm verify` returns exit 2 because DESIGN.md is a global CI input. The required PR workflow supplies the broad CI coverage. - Native preflight cannot finish because Cargo is absent in this environment. UI dependencies were installed. The missing plugin SDK build was prepared before the successful UI checks. - Final head: `2219de45721360d36341ea1ae46641b6bcb20360`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42708839687) passed. No checks are pending or failing. Greptile and Paperclip Review each gave 5/5. All five prior review threads are resolved. ## Risks - Narrow rows can grow to show two text lines and thumbnails. The existing queue height limit still applies. - Screenshot previews use the existing image-reference parser and a local attachment path filter. Image omission is opt-in; thread and decision media keep their existing modes. - Pending uploads remain in draft state while their previews are hidden. Rejection and uncertain-response tests check that the same receipts return without losing later attachments. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific serving model ID or context limit. Used reasoning, tools, code execution, and browser checks. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
61c169753e |
test(chaos): fire planned faults at a fixed time after launch with --fault-after-seconds (#20345)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness fires each task's fault at a seeded time inside the workload's window (R1: 20–90 s after launch) > - With a Paperclip-sized remote workspace on Daytona, preparation (sandbox clone and the initial snapshot save) takes longer than 90 s > - Every restart in a Daytona R1 wave therefore lands before the run resolves its runtime mode, so only the pre-resolution recovery path is ever tested > - This pull request adds `--fault-after-seconds`, which fires every planned fault at a fixed time after launch while keeping the rest of the seeded plan > - The benefit is that one seed can test a restart both during preparation and after native resolution ## Linked Issues or Issue Description **What happened?** Daytona R1 waves w-20261109T163514-s9913 and w-20261109T185009-s9950 restarted the controller at 43 s and 84 s. Both runs were still preparing their remote workspace, so the native recovery path for a real-size workspace was never exercised. **Expected behavior** An operator can choose when the fault fires relative to the task launch. **Actual behavior** The fault time could only come from the seeded workload window. ## What Changed - `--fault-after-seconds <n>` (5–3600) sets `faultAfterMs`. `planTasks` still draws the seeded fire time, so nonces, holds and triggers are unchanged, but it uses `faultAfterMs` for every planned fault except K13. - The option is rejected with storms and crash-point waves, which have no planned fault times. - README: one line under Options. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 259 passed. The new scheduler test fails before the change. It checks that every planned fault fires at 150 s, that the rest of the plan matches the same seed without the option, and that out-of-range or storm use is rejected. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A fixed fault time can land after a fast task finishes, in which case the fault crosses no work for that task. The trigger is recorded as missed, as for any late fault. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
97b89981d3 |
test(chaos): wait for the closing save before reading a remote R1 workspace's files (#20348)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades Daytona R1 tasks by reading their files through `GET /issues/:id/file-resources/content` > - The server serves a remote workspace's files either from its live lease or, once the closing save finishes, from the saved snapshot > - The closing save takes about a minute for a Paperclip-sized workspace, and during it the workspace is dirty with a writer, so reads return 422 `workspace_unavailable` > - The harness read once at `done` and once at rest, both inside that window, so every Daytona R1 task failed its file checks even when the agent did the work > - This pull request retries a 422 read for up to 4 minutes, until the saved snapshot is readable, and returns a 404 at once > - The benefit is that Daytona R1 grades the agent's actual files ## Linked Issues or Issue Description Follow-up to #20288. **What happened?** Daytona R1 wave w-20261109T191837-s9950 (K1 at +180 s, native runs) recovered both tasks without a Retry (R1, R3 and R6 pass), but both failed `T:module_present`. The server log shows every content read returning 422, from the moment the task was `done` (19:23:04) through rest (19:23:10). Track 3's R1 waves lost 10 of 16 tasks the same way. **Expected behavior** The oracle reads the files once the workspace's closing save makes them readable. **Actual behavior** It gave up on the first 422. ## What Changed - `remoteWorkspaceFile` retries a 422 every 2 s for up to 4 minutes (`REMOTE_FILE_WAIT_MS`). Any other failure, including 404 for a file the agent never wrote, returns `null` at once. - README: notes the wait. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 260 passed. New tests: a server that answers 422 twice and then the files (red before), and a missing file that returns within 10 s. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. ## Risks - A remote workspace whose closing save never completes now costs up to 4 minutes of grading per file read before failing, which is visible in the wave's duration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
908d2aae7a |
test(chaos): run waves against S3-compatible storage with a fault proxy, and click Retry in the real board UI (#20288)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud deployments keep run logs, traces, attachments, artifacts and
saved workspace files in S3-compatible object storage
> - The restart chaos harness ran only with local disk storage, so
storage failures and slowdowns were never tested
> - It also clicked Retry through the API, so a missing or broken Retry
button in the board UI went unnoticed (staging PAP-1683: "the
saved-files recovery Retry does nothing")
> - This pull request runs waves against a loopback MinIO behind a fault
proxy, adds store-outage, latency and throttling faults, and clicks
every Retry in the real board UI
> - The benefit is that storage outages and broken Retry paths now
surface in chaos waves. The first ST1 wave found that a run starting
during a store outage fails immediately and exhausts recovery
## Linked Issues or Issue Description
**What happened?**
Staging saves to object storage and has failures there that local chaos
waves cannot reproduce, because the harness used `local_disk`. The
harness also clicked Retry through the API, which skips the board's
actual button.
**Expected behavior**
Chaos waves can run the server in cloud storage mode, inject storage
faults, and press Retry the way a user does.
**Actual behavior**
They could not.
## What Changed
- `--storage s3`:
- starts a loopback MinIO (`RELEASE.2025-04-22T22-12-26Z`, from
`~/.cache/paperclip/tools/minio/minio` or `RESTART_CHAOS_MINIO_BIN`)
with its data under the instance temp root and random root credentials;
- creates the bucket;
- puts a fault proxy in front;
- points the server's `PAPERCLIP_STORAGE_*` settings at the proxy, with
path-style addressing and a per-wave prefix. The supervisor declares
these as explicit workspace storage for the isolation check.
- `storage-proxy.ts`: a transparent, non-retrying S3 fault proxy that
keeps the signed Host header. Rules match by method, path and remaining
count. Faults are outage (503 `ServiceUnavailable`), latency with
jitter, a 5xx or `SlowDown` error rate, connection reset, and 403
`ExpiredToken`. It counts requests by method and status, written to
`storage-proxy.json`.
- `--storage-faults ST1,ST2,ST3` (`storage-chaos.ts`): seeded 60–180 s
windows of outage, 1–10 s latency, or 30% 503 / `SlowDown`, applied
while the wave runs. They're logged in the timeline and in
`storage-faults.json`.
- `board-ui.ts`: every task and subtask Retry the grader needs is now
clicked in the board UI. The harness uses headless Chromium through
Playwright, opens `/issues/<identifier>`, waits for the visible Retry
button and captures the `POST /issues/:id/retry` the click sends.
- R3 fails a missing button as `ui_retry_button_missing (user saw: …)`
and a click that sends nothing as `ui_retry_no_request`.
- Crash-point waves, which serve no UI, keep the API path.
- README: a cloud storage section.
## Verification
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 244
passed. New tests:
- the proxy against a fake upstream: passthrough, bounded outage,
`SlowDown`, `ExpiredToken`, latency and reset;
- live MinIO through the proxy: `If-None-Match` returns 412, and an
injected 503 reaches the client;
- Playwright against local pages: a working button, a missing button,
and a dead button;
- the observer clicking through the board, and R3 signatures for UI
failures;
- window planning, apply and clear;
- `--storage` and `--storage-faults` parsing.
The observer and grader tests failed before the change.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- Live waves on master
|
||
|
DottaandClaude Opus 5.5
|
f7c85bca18 |
test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness boots a full temporary Paperclip instance per wave: embedded Postgres, workspaces and caches, often 5–13 GB > - At wave end it deleted that instance only when the wave passed with no harness error and no `--keep` > - Almost every chaos wave has at least one failing task, so almost every instance stayed on disk > - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the machine's disk (ENOSPC), which stopped a running storm wave > - This pull request deletes the instance at the end of every wave, smoke and baseline run unless it is explicitly kept or its processes could not be stopped, and expires kept instances after 6 hours > - The benefit is that chaos waves can run back to back without filling the disk ## Linked Issues or Issue Description **What happened?** `main.ts` kept the temporary root whenever `keep || anyFailure || harnessError`. A failing task is the normal output of a chaos wave, so instances accumulated until the disk was full. The codex K13 storm w-20261109T113925-s9403 died after about a minute with every task cut off. **Steps to reproduce** 1. Run any wave in which at least one task fails. 2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays behind, with no expiry. **Expected behavior** The instance is removed when the wave ends. Evidence is already in `results/`. Only an explicit `--keep`, or a stop failure that may leave processes running, keeps it, and kept instances expire. **Actual behavior** Every failing wave leaks its instance permanently. ## What Changed - New shared `tests/runner-e2e/instance-retention.ts`: - `releaseInstance` deletes the root unless `--keep` was passed or the processes could not be stopped. In those cases it writes a `.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets the directory to 0700. An optional `retain` list keeps small evidence files and removes everything else. - `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*` directories that carry the marker, are older than 6 hours, and have no live `postmaster.pid` anywhere inside them, and that no live process command line references. Each root is made removable first, and one failed delete does not stop the sweep. Unmarked directories are never touched, so existing directories from older runs stay until someone decides on them. `recovery_state` markers never expire. - `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets it. - Callers: - A wave whose Daytona cleanup is still pending (environment DELETE blocked, sandboxes still running after cleanup, or cleanup errors) keeps its instance as `recovery_state`, because its database owns that cleanup. - The wave releases its instance before writing the ledger and summary, so a failed report write cannot skip cleanup. - Restart chaos `main.ts` (crash-point suite included) releases the instance in its final cleanup, which also runs on failure and interrupts. The crash-point task-count check moved above supervisor creation so that error no longer leaks a fresh root. - `ttft-baseline.ts` releases after stopping the supervisor, including on SIGINT and SIGTERM. - The rolling-deployment and controller-replacement smokes now delete their roots, keeping only the evidence report, logs and screenshots. If cleanup is unconfirmed, the root is kept with a marker. - `launch.ts` marks its unconfirmed-remote-cleanup root as `recovery_state`, which never expires, and its `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing deletion path is unchanged, and the file gets one line shorter. - README: the instance lifecycle and `--keep`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328 files, 6915 passed. The new retention tests cover: - deleting a finished instance, and retaining only listed evidence; - keeping it with a marker and mode 0700; - sweeping only marked, expired instances whose Postgres isn't running, while sparing fresh, live, recovery-state, unmarked and unrelated directories; - a stale `postmaster.pid`; - sweeping on `createRunnerE2ETemporaryRoot`. The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran the sweep. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221 passed. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and `tests/restart-chaos`: clean. - No wave or smoke was run, because the disk is still nearly full. ## Risks - A failed wave's instance is no longer available for post-mortem DB inspection unless the wave ran with `--keep`. Evidence bundles in `results/` are unchanged. - The sweep relies on `postmaster.pid`. An instance whose Postgres was started outside the harness, without that file, could be removed after 6 hours. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ac0d0374f2 |
perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
610b54f551 |
test(chaos): close review-thread gaps in speed grading, S1 stops, readiness and stall reads (#20244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness grades how real tasks survive control-plane restarts, and its grades decide which product fixes come next > - Reviewers left 46 unresolved threads on ten merged harness PRs (#20071, #20094, #20106, #20111, #20119, #20147, #20158, #20161, #20178, #20194) > - Several are real harness defects: false speed breaches from one-sample rows, a codex-only wave that cannot start, event paging that can loop forever, an S1 task failed for a Stop the harness never sent, and a readiness wait that aborts on a transient lifecycle error > - Wave S1 t05 (w-20261109T091247-s9806) hit the S1 defect: the run finished before the stop window, so neither the Stop nor the continue comment was sent, and the task was graded `done_too_late Infinity` > - This pull request fixes the valid findings and adds the missing tests > - The benefit is that a failed task points at the product, not at the harness ## Linked Issues or Issue Description Refs #20071, #20094, #20111, #20147, #20158, #20161, #20178, #20194 (review threads). The security finding from #20111 was fixed separately in #20228. **What happened?** Harness defects found in review and in waves: - Aggregate p50/p95 rows breached at n = 1–3, which caused most marginal speed failures. - `--adapters codex` with the default workloads failed at parse because M1 needs two adapters. - `runEvents` paged forever if a full page had no `seq`. - A timed-out task whose last sample was `done` was graded `done_too_late Infinity`, and could pass when a fault had no ready time. - S1 waited forever for a running run to stop; it also clicked Stop while another task's fault had the server down or recovering. - The readiness wait aborted on any `lifecycleError`, including the transient one lifecycle `defer` sets before a 60 s retry, and hid read errors as "not ready". Daytona used the 240 s local timeout. - One failed parent read could hide an M1 stall for the whole 10 s check window. ## What Changed - Speed: aggregate rows breach only from 5 samples (`insufficient sample (n < 5)` otherwise). Restart recoveries over the control-plane p95 budget now count in the per-run check, so a single slow recovery still fails. - Config: the default workloads skip M1 when fewer than two adapters are given; an explicit `--workloads M1` with one adapter still fails. - `runEvents` throws when a full page has no later `seq`; the run's speed read fails instead of hanging. - `gradeR1` fails a timed-out task as `no_rest_within_budget` before checking its status. - S1: the Stop waits for a ready server with no fault in progress. If the run already finished when the window opens, the harness records a missed Stop, skips the stop-triggered fault, still posts the continue comment, and the oracle checks only the continue. - Readiness: only a terminal lifecycle state (rejected, terminating, cleaning up, terminated) aborts at once. Any other lifecycle error is logged when it changes (`seed_agent_lifecycle_error`) and the wait continues until ready or the timeout, which names each agent's last lifecycle or read error. The agent API exposes neither `lifecycleOperation.retryRequired` nor a retry-claim timestamp, so a deferred retry cannot be told apart from a parked failure earlier. Daytona waits up to 10 min. - M1: a failed parent read retries on the next poll. - Child speed collection moved into `childSpeedTasksInTurn` so its one-at-a-time order is tested. - Tests for each change, plus the gaps reviewers named: the stop trigger's arming, stall-wake parsing, and `seedChaosLab`'s readiness wiring. - README: S1 missed Stop, R1 timeout rule, speed sample minimum. ## Verification - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 230 passed. Each new test failed before its fix, such as the paging test running out of memory in the loop and the stop-trigger test firing a fault after a missed Stop. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean. - Wave evidence for the defects: S1 t05 in w-20261109T091247-s9806 (missed Stop); codex warm agent_boot p50 breach at n = 3 in w-20261109T104527-s9901. ## Risks - Aggregate rows with fewer than 5 samples no longer fail a wave. Per-run p95 checks still catch individual slow runs and recoveries, but a consistently slightly slow p50 in a small wave is now only reported. - A lifecycle error that persists without `retryRequired` now waits for the readiness timeout (4 min local, 10 min Daytona) instead of failing at once. The timeout message names the error. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
347d21fcbd |
test(runner-e2e): scope each agent's sandbox to its own run's saved state (#20253)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The runner E2E chat evals run real agents against a local Paperclip
instance on a CI host
> - PR #20044 hides the eval harness from local native agents with a
Linux Landlock domain around each runnerd launch
> - That domain is the same for every run in an attempt, so an eval
agent can still read another run's saved state
> - Recent chat-eval runs show agents that read other runs' OpenCode
session databases, another run's run log, the instance `pg_wal`,
`control-plane-state.json`, and `/proc/<pid>/cmdline` of harness
processes
> - This pull request scopes each runnerd launch to its own run and
moves the selected case id off the command line
> - The benefit is that an eval agent cannot learn about the case or
other runs from files outside its own run
## Linked Issues or Issue Description
Refs #20044
**What happened**
In local native OpenCode chat-eval cells, worker agents read files that
belong to other runs or to the harness:
- other runs' OpenCode sqlite session databases under
`runtime/paperclip-runner/durable-sessions/` (a status-cards run)
- `/proc/<pid>/cmdline` of harness processes, which shows harness paths
and the case id (`routines.pause-owned`)
- another run's run-log ndjson, and a listing of the instance
`db/pg_wal` (`routines.pause-owned`)
- a listing of `control-plane-state.json`
(`runs-adapters.list-model-options`)
The Landlock sandbox from #20044 hides only the harness (`tests/`, the
Git store, private attempt directories). All runnerd launches share one
hidden set, so the instance's per-run directories stay readable.
**Expected behavior**
An agent process sees only its own run's durable session, OpenCode
session and provider trace. Other runs' sessions, run logs and traces,
the instance database files, secrets and the shared control-plane state
in the attempt instance are denied. The case id is not on any harness
command line.
**Steps to reproduce**
1. Dispatch a local native OpenCode chat-features cell that has two
agents (for example a status-cards case).
2. In the worker's shell, run `ls
<instance>/runtime/paperclip-runner/durable-sessions/*/` and `ls
<instance>/db/pg_wal`.
3. Run `cat /proc/<launcher pid>/cmdline`.
4. Before this change, all three succeed and show other runs' files and
`--id chat-features...`.
## What Changed
- `agent-filesystem-sandbox.py`: new `--expose PATH` option. It grants a
path beneath a hidden directory. Landlock does not check traversal, so
the run uses its own entry while the parent stays unlistable. The helper
refuses a missing exposure, an exposure outside every hidden path, or
one that would reveal a hidden path. It fails the launch unless every
exposed path is readable.
- `agent-filesystem-sandbox.ts`: the launcher passes the instance root
in the sandbox configuration. `runnerdLaunchSandbox` adds per-launch
hidden paths: `durable-sessions/`, `runner/`, `control-plane/`,
`opencode/` under `runtime/paperclip-runner/`, and `data/run-logs/`,
`data/provider-traces/`, `data/backups/`, `db/`, `secrets/`. It exposes
only the launch's own durable-session directory (from `--state-dir`),
Codex runner state directory, OpenCode session directory (from
`--session-id`, named as the OpenCode driver names it) and provider
trace file.
- The wrapper creates the exposed entries, and the shared instance
directories that the product creates on first use, before runnerd
starts. It logs the run-scoped hidden and exposed paths for each launch.
- New `probeRunScopedSandbox`: at server start, a sandboxed shell runs
against a fixture instance. Another run's session, OpenCode database,
runner state, run log, trace, `pg_wal`, secrets, control-plane state and
the server's `/proc/<pid>/environ` must be denied. The run's own
session, OpenCode directory, ACPX directory and trace must be writable.
In CI any failure stops the attempt. The result goes to `server.log`.
- `selectors.ts` and `launch.ts`: the launcher reads the selected cell
from `PAPERCLIP_RUNNER_E2E_SELECTED_ID` and removes it from its
environment.
- `runner-full-stack-e2e.yml` (Run paid cell) and
`chat-catalog-testops-case.yml`: pass the cell in that variable instead
of `--id`. A target branch that does not read the variable still gets
`--id`.
- `SECURITY.md`: documents the run scope, the `/proc` limit and what
stays readable.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/workflow-security.test.ts
tests/runner-e2e/manual-private-workflow.test.ts
tests/runner-e2e/private-trust-workflow-security.test.ts` passes on
macOS. The Landlock host tests (including the new fixture-instance probe
test) run only on Linux CI.
- `npx tsc --noEmit -p tests/runner-e2e` shows no errors in the changed
files.
- Paid run
[42611358428](https://github.com/paperclipai/paperclip/actions/runs/42611358428):
`chat-features.runner-opencode.local.agents.change-manager` on the same
commit applied to `chat-eval/master-candidate` (
|
||
|
DottaandClaude Opus 5.5
|
0d80b74c21 |
test(chaos): treat a crash-restart EPERM as a warning once the server pid is dead (#20232)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness crashes the local Paperclip server with SIGKILL (K1, K13 storms) and grades how tasks survive > - `crashForRestart` sends SIGKILL to every process group the server owns > - On macOS, `killpg` can report EPERM for a group whose members are still exiting > - When that happens the fault is recorded as failed, and the wave exits 2, even though the server process itself is dead > - This pull request tolerates EPERM in the crash path once the server pid is confirmed dead, and logs it as a warning > - The benefit is that a wave grades the product, not a cleanup race in the harness ## Linked Issues or Issue Description **What happened?** A local Codex K13 storm at concurrency 6 (wave w-20261109T084023-s9403, 112 tasks) exited 2 because faults f04 and f09 failed with `kill EPERM`. The server pids of both faults had died, and the tasks recovered normally. **Steps to reproduce** 1. Run a long K13 storm on macOS with high concurrency. 2. Occasionally the SIGKILL of the server's process groups races with members that are exiting. **Expected behavior** If the server process is dead, the crash completes and the remaining cleanup continues; the EPERM is recorded as a warning. **Actual behavior** The fault fails with `kill EPERM` and the wave reports an error. ## What Changed - `crashForRestart` catches EPERM from the group SIGKILL. It then sends SIGKILL to the server pid directly if needed and waits for it to exit within the forced timeout. - If the server exited, it re-observes the tree and sends SIGKILL to each remaining group on its own, so one group's EPERM does not leave the others to drain gracefully. It logs a warning naming the server pid and any group that still refused SIGKILL, then continues with the normal post-kill cleanup. - If the server is still alive, or the error is not EPERM, the error is rethrown as before. - Four tests: EPERM with a dead server finishes with a warning; a detached descendant is still killed without a graceful drain when the first group kill reports EPERM; EPERM with a surviving server fails; a non-EPERM error fails. The failure tests stop their owners in `finally`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/server-stop.test.ts`: the dead-server test fails before the change (`kill EPERM`), the descendant test fails against the first revision (the descendant drains on SIGTERM), and all 22 pass after it. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e`: clean. ## Risks - A group that is still running and returns EPERM is no longer reported in the crash step itself. The cleanup that follows still checks every observed group, and it fails if any group survives. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9715b3d210 |
test(restart-chaos): do not grade a recovery that a back-to-back fault interrupted (#20243)
## Thinking Path
- #20183 split restart recovery so the speed budget judges Paperclip's
own share (`restart_recovery_control_plane`).
- In Phase 3's local mixed wave (`w-20261109T104527-s9901`, master
|