mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
74096c19f4e4be224d05abb585673f2dfa6691f6
7796
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
74096c19f4 |
fix(evals): admit the in-flight chat completion follow-up on a settled source (#21313)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The `source-settled-with-active-work` run policy checks that the CEO chat is settled while delegated work can continue > - After a delegated task finishes, the product starts its normal `chat_task_completed` follow-up on the source chat > - The policy required every source run to be `succeeded` and the source to be `waiting`, so a re-grade during that follow-up failed > - This pull request admits that one product follow-up while it is in flight, and keeps every other source run strict > - The benefit is that the eval grades the saved private-delegation outcome, not the timing of the reload ## Linked Issues or Issue Description **What happened?** The OpenCode cell of `chat-features.runner-opencode.local.tasks.private-task` failed with "Independent oracle survives browser reload: expected true, received false". The saved state was correct. The CEO answered, Morgan saved the private draft in the private project, and the unrelated member could not read it. The grade passed for its full 5-second stability window. Morgan's run then finished, and 73 ms later the product started the CEO `chat_task_completed` follow-up on the source chat. The harness reloaded the page and re-graded while that follow-up was running. The only failed check was `catalog-settled-source-separate-work`, because it required all source runs to be `succeeded`. **Expected behavior** The case already counts this follow-up in its run bound (`expectedRunCount: 3`, `minimumRunCount: 2`). A queued or running product follow-up on an answered source must not fail the case. **Steps to reproduce** Run the private-task case. Let the worker finish within about 5 seconds of the first passing grade. The post-reload re-grade sees the follow-up as `running` and fails. ## What Changed - Add `chatCatalogSourceCompletionFollowUp` to `tests/runner-e2e/chat-catalog-run-policy.ts`. It identifies the lead's `chat_task_completed` run on the source whose every completion update names separate work in the run ledger. - In the `source-settled-with-active-work` check, admit at most one such follow-up while it is `queued`, `running` or `succeeded`. Accept a source `conversationState` of `active` only while that follow-up is in flight. - Keep the other rules strict. The original answer must exist and must have succeeded. Any other active, failed or extra source run still fails. The declared run-count bound still applies. - Also admit the follow-up before its turn is prepared. The server claims the run first and saves the completion updates after that, so a queued run, or a running run in preparation, has only its delivery IDs. It qualifies only while queued or running, when automation woke it, and after a separate worker run in the ledger succeeded. - Add a red/green test that reproduces the observed race and covers the negative cases. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-run-policy.test.ts`: the new test fails before the change (1 failed, 13 passed) and passes after it (14 passed). The queued-snapshot cases were added after review. - The related runner-e2e tests (`chat-catalog-run-policy`, `chat-catalog-readiness`, `chat-catalog-failure-retirement`, `runtime-dependency`, `internal`) pass: 200 tests. - `pnpm test:e2e:runner:typecheck` passes. - Paid run [43374419951](https://github.com/paperclipai/paperclip/actions/runs/43374419951) on the first commit of this branch: `chat-features.runner-opencode.local.tasks.private-task` passed. ## Risks - Low risk. The change is in eval grading only. It does not change product code. - The check accepts a follow-up that is still running at grade time. A later failure of that follow-up is not observed by this case. Completion delivery is covered by its own completion cases. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
23c2343015 |
feat(server): read a retained provider pack from its Runner image (#21308)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner executes agent work in a sandbox. It needs a provider pack of agent tools, and the server uploads that pack when the sandbox does not have it > - A retained session keeps the pack it started with. When that pack is not the server's own, the server downloads the pack's transport archives from a registry (#19779) > - Today the server looks for them in the application image tagged `sha-<commit that built the pack>` > - `master` now publishes the Runner one time for each set of inputs, as `ghcr.io/paperclipai/paperclip-runner`. A pack from that image names the commit that published the image, and few commits get an application image, so that tag often does not exist > - This pull request reads such a pack from its Runner image, and keeps the old source for every other pack and for a Runner image that cannot be read > - The benefit is that a retained session can still get its pack when the pack came out of the published Runner image ## Linked Issues or Issue Description Refs #19779 (the retained pack source), #21228 (the pack field `runnerInputsDigest`), and #21204, #21229, #21230 (the Runner image). No issue exists for this change. **Problem or motivation** `retained-provider-pack-oci.ts` reads `ghcr.io/paperclipai/paperclip:sha-<runnerSourceRevision>`. It needs that image to exist, to carry the pack, and to have one layer with a known build-history line. For a pack from a published Runner image, `runnerSourceRevision` is the commit that published the image. `docker.yml` writes `sha-<commit>` only when its `master` run for that commit completes, and a newer push can replace a waiting run. The restart-chaos harness has the same assumption in another place: it requires the pack of a sandbox image to be built by the commit in the image's `org.opencontainers.image.revision` label. **Proposed solution** A pack that names its Runner image in `payload.runnerInputsDigest` is read from `ghcr.io/paperclipai/paperclip-runner:inputs-<digest>`. A sandbox image can name the commit that built its Runner in a label of its own. No application image carries a pack with that field yet, and no image carries that label, so behaviour today does not change. **Alternatives considered** - Keep the application image as the only source and require an application image for every Runner publication. Publication would then depend on `docker.yml`, which serves one ref at a time. - Take a fixed number of layers ("the image has four, take the last two"). A server in the field cannot be changed when the image gets another layer. The server now uses the rule that holds for any number of layers: the full image is the slim image plus transport layers. - Fall back to the application image for every failure of the Runner image. A Runner image that exists and does not match the pack is a fault, and a second source would hide it. **Roadmap alignment** This is build and runtime infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 3 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image (this pull request) 4. #21310: a registry build cache for the Runner image build This pull request uses no code from parts 1 and 2. It is stacked on them because all four parts edit `doc/runner-image-contract.md`. ## What Changed - `server/src/services/native-runtime/retained-provider-pack-oci.ts`, for a pack whose `payload.runnerInputsDigest` is 64 hexadecimal characters: - The server reads the full index `inputs-<digest>` of the Runner image. The index must carry `io.paperclip.runner.inputs-digest` equal to the pack's value, `org.opencontainers.image.revision` equal to the pack's `runnerSourceRevision`, and a valid digest in `io.paperclip.runner.slim-digest`. - It reads the slim index by that digest, never by the slim tag, and checks the bytes it gets against the digest. - For the pack's platform, the layer digests of the slim image must be the first layer digests of the full image, in the same order. The layers after them are the transport layers. There must be at least one and at most eight, and their total size has the existing limit. - The server downloads only those layers. It checks each against its descriptor and refuses every entry that is not a transport archive, the receipt or the `provider-pack.archives` directory. A file may not be in two layers. - When the Runner image cannot be read, the server reads the application image as before. That covers a registry error status for the token, for either index or for either manifest: for example 404 for a tag or a slim index that does not exist. Nothing is written before that point. - A Runner image that can be read and does not match fails with `runner_remote_provider_source_invalid`. The server does not try the application image then. - The same file, for every other pack: the old path, unchanged in what it checks. Its code moved into functions that both paths use (`request`, `pullToken`, `platformImage`, `receiveTransportLayer`). - The downloaded layer file is removed as soon as it is unpacked. Before, the importer removed it after its checks, so `retained-provider-pack-source.ts` loses that one line. - The cache record gets an optional `runnerImage` field with the inputs digest and the digests of the transport layers. `transportLayerDigest` is the last of them and `transportLayerBytes` their total. An older server ignores the new field. - `tests/restart-chaos/daytona-image.ts`: when the image has a non-empty label `io.paperclip.runner.source-revision`, the pack must be built by that commit. Otherwise the pack must be built by the commit in `org.opencontainers.image.revision`, as before. The receipt field `providerPackRunnerSourceRevision` records the pack's commit. - `doc/runner-image-contract.md` has a new section, "Reading the transport archives alone". `tests/restart-chaos/README.md` describes the label. One choice needs a reviewer's eye. When the full index can be read and the slim index cannot (a registry error status), the server falls back to the application image; it does not fail. Without the slim manifest the server cannot tell which layers are transport layers, so it has not accepted any byte of the Runner image, and the old source has its own complete checks. A slim index that is read and has other bytes than its digest still fails. ## Verification Linux (`node:24.22.3-bookworm`, GNU tar 1.34, zstd 1.5.4, the worktree mounted read-only): - In `server/`: `retained-provider-pack-source.test.ts` (34 tests, 15 of them new), `provider-pack-archive.test.ts`, `remote-provider-pack-verification.test.ts`, `remote-provider-pack.test.ts`: 127 tests pass in 4 files. - `node node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/daytona-image.test.ts`: 25 tests pass, 5 of them new. macOS (Node 26.12.0): - `daytona-image.test.ts`: 25 tests pass. - `retained-provider-pack-source.test.ts`: the 15 new tests pass. Six older tests fail, with and without this change. They run the real uploader, which needs GNU tar, and this host has bsdtar. - `tsc` on the two changed server files and the test, and on the two restart-chaos files, with the settings of their test projects: no error. The new cases: - Read from the Runner image with 6 registry requests (two indexes, two manifests, two layers), none to the application image and none for the slim tag. - Take every layer after the slim image's: one transport layer, and three. - Fall back when the tag does not exist, when the token is refused, and when the slim index cannot be reached. - Refuse, with no request to the application image: an index for other inputs, an index for another revision, no slim annotation, a slim annotation that is a tag, a slim index with other bytes than its digest, a slim image that is not a prefix of the full image, a slim image equal to the full image, nine transport layers, and a transport layer that holds a pack file. - Four deliberate faults in the reader each make a test fail: the slim index bytes are not checked, no prefix check, the slim annotation is not validated, and no bounds on the number of transport layers. A real run, in the same Linux container (`linux/arm64`), with no credential: - The script took `provider-pack/provider-pack.json` of `linux/arm64` from the published image `ghcr.io/paperclipai/paperclip-runner:inputs-0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. Pack digest `sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f`, `runnerSourceRevision` `25e9a9c2e9277f3be8649be8c809cc81bde6f0c5`. - `resolveProviderPackUploadSource`, with another pack as the server's own, returned `{ archiveOnly: true }` after 23 seconds. That function calls `receiveRetainedProviderPack` and then the importer's checks: the manifest inside the overlay archive is the selected manifest, every archive matches the receipt, and the content of every file archive matches the digest of its file. - The server code made 9 requests: one anonymous token request, the full index, the `arm64` manifest, the slim index `sha256:f63ea436…4f3c` by digest, its `arm64` manifest, and two layers, each with one redirect. No request went to the application image. - The cache record: ```json { "schema": "paperclip.retained-provider-pack-source.v1", "sourceRevision": "25e9a9c2e9277f3be8649be8c809cc81bde6f0c5", "providerPackDigest": "sha256:604c29cb32a181843cc4d425871f88c0e65c8185b0a158c801fda4eb1bfce52f", "imageIndexDigest": "sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162", "imageManifestDigest": "sha256:73dbe864fbab6208616f0aa6f1610a7ece4ef22f68a73d30901cbaea60139589", "imageConfigDigest": "sha256:740e48463d5caf0bfead4b923cdbd8b220acb93d058419c3db9d00c7f1831d65", "transportLayerDigest": "sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf", "transportLayerBytes": 426602696, "runnerImage": { "inputsDigest": "0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca", "transportLayerDigests": [ "sha256:6a340088ca2cfa85f10f0385d9c8f5cdccb22c3a2bbb2062620b1d90616ba5b5", "sha256:35433b9f788641666b470f41c37fef9fc9b872262b7edd5bd79f0e288f75bcdf" ] }, "receiptSha256": "2afdd26645845369d9e851b1e6e2884eb6c3884ea7c4866a1eeb2efb9fb230d7", "verifiedTarPaths": true } ``` - `uploadVerifiedProviderPack` then installed the pack from that source through a local stand-in for the sandbox, with no further registry request, in 4 seconds. The installed tree is 1,249,060,841 bytes. Its `provider-pack.json` is the selected manifest, all 7 artifacts of the manifest have the digest the manifest names, and the pack's own Node prints `v24.22.3`. Never run: - The same real run for `linux/amd64`, whose transport layers are about 1 GB. - A session in a real sandbox. The real run used a local stand-in for the sandbox. - The fallback against the real registry. Fixtures cover it. - The full test suite and the full typecheck. CI must supply them. ## Risks - The old path was moved into shared functions. Its checks are the same, and its 19 existing tests pass on Linux, but a review must read that code as changed. - The layer file of the old path is now removed before the importer's checks, not after them. The importer does not read it. - The reader depends on two rules of the image contract: the full index names its slim index, and the full image starts with the slim image's layers. A Runner image that breaks either rule is refused, with no fallback. - The fallback covers a registry error status only. A network error ends the import, as it does today. - The label `io.paperclip.runner.source-revision` is new and no image carries it yet. The change to the sandbox image that sets it must use this name. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0477f622d6 |
fix(recovery): keep the sandbox that holds abandoned unsaved files out of replacement and cleanup (#21304)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes, and the workspace files live in those sandboxes until a save > - When a run cannot resume exactly, the fresh continuation fallback (#21135) abandons the old workspace epoch and keeps its unsaved files "recorded and unapplied" on the old sandbox > - The abandonment clears the dirty flag, the writer and the location, so the workspace looks clean. No code reads the abandonment receipt > - A later failed resume can then replace the sandbox, and cleanup can destroy it. That sandbox holds the only copy of the unsaved files > - This pull request adds one shared hold. The task continues on a new sandbox, and the old sandbox is parked until its files are saved, discarded, or kept for 7 days > - The benefit is that the fallback keeps its promise. Abandoned files are not lost ## Linked Issues or Issue Description Refs #21135 Refs #21297 **What happened?** After the fresh continuation fallback abandoned a workspace epoch, the receipt in `remoteWorkspaceAbandonedEpochs` was write-only. `reusableSandboxReplacementIsSafe` saw a clean workspace with a committed save and allowed replacement. `retireReplacedReusableSandboxLease` then moved the old lease to `pending_cleanup`, and the cleanup sweep destroyed the sandbox with the unsaved files. `nativeWorkspaceExportPendingForLease` (the check that the sweep, the retained orphan settlement and the scoped destroy paths use) also answered "nothing pending". **What did you expect?** A sandbox that holds abandoned, unsaved files stays until the files are saved, a person discards them, or a stated retention period ends. **Steps to reproduce** 1. Abandon a remote workspace epoch whose location names sandbox L. 2. Let the next run fail to resume L. 3. The replacement decision says safe, L goes to `pending_cleanup`, and the sweep destroys it. ## What Changed - New `server/src/services/remote-workspace-abandoned-files-hold.ts` holds the single predicate for "this lease holds abandoned files". It is one SQL fragment (`abandonedWorkspaceFilesHeldSql`) plus a TypeScript wrapper (`abandonedWorkspaceFilesHeldOnLease`). A lease is held while a workspace in its company has an abandoned-epoch receipt whose `location.providerLeaseId` equals the lease's provider lease id, and that receipt has no `released` object. A receipt written by the current release (no new fields) counts as unreleased. - Replace and park. A failed resume of a held sandbox still lets the run replace it, so the task continues at once on a new sandbox restored from the last committed save. `retireReplacedReusableSandboxLease` parks the old lease as `retained` with the `abandonedWorkspaceFilesParked` marker (failure reason `abandoned_workspace_files_held`) instead of `pending_cleanup`. `reusableLeaseCanBeResumed` and `reusableLeaseCanBeCleanedUp` skip a parked lease, so later runs never select it again. The check is a metadata read with no query, so the run path does no extra work. - Every path that can destroy a sandbox now honours the hold: - `nativeWorkspaceExportPendingForLease` answers pending for a held lease. This covers the pending cleanup sweep (defer), the orphaned active lease sweep, the retained orphan settlement `heldBy`, `destroyClaimedSandboxLease` (terminal workspace reaper `destroyRemoteWorkspaceLease`, `destroyReusableSandboxLeases` and the environment delete path, and run lease release with `destroy`), the `destroyRunLease` dispatcher, and the plugin sandbox teardown retry. - The run lease settlement (`environment_run_lease_active`) is the one caller that ignores the hold (`{ destructive: false }`). It never destroys a sandbox. It stops the sandbox and keeps it, or it moves a lease without a sandbox to `pending_cleanup`, where the sweep parks a held lease. If it waited for the hold, a dead run's lease would stay `active` and block the retry for the whole retention period. - `destroyReusableSandboxLease` parks the lease and returns without a provider call. This covers obsolete reusable lease cleanup, the destroy after an expired or mismatched resume, release with `expired`, and the sandbox driver's `destroyRunLease`. - The `retryPendingSandboxTeardown` dispatcher refuses a destructive teardown of a held lease. Stop-only cleanup still runs, because a stop keeps the filesystem. - The retained orphan listing (`alwaysHeldSql`) uses the same SQL fragment, so held orphans are not polled. - Release. `releaseAbandonedWorkspaceFiles` marks a receipt `released` with a reason (`recovered`, `discarded`, `retention_expired`) and an actor. It moves the parked leases for that sandbox to `pending_cleanup` when no other unreleased receipt holds them, and it writes a `workspace.abandoned_files_released` activity entry on the task in plain language. No export path for abandoned epochs exists today. The `recovered` reason is the hook for one; this pull request adds no new export mechanism. - Retention expiry. New receipts carry `retainUntil` (7 days). An old receipt uses `abandonedAt` plus 7 days. `nextEnvironmentCleanupAt` includes the earliest retention end of any unreleased receipt. The shared cleanup worker therefore wakes at that real deadline, and a restart rebuilds it from the database. At the deadline the sweep runs `expireAbandonedWorkspaceFileHolds`. This releases each due receipt as an attributed system decision (actor `workspace-retention`) and logs the held receipts and parked leases for each company. Each expiry always writes a task entry: "Unsaved files from an interrupted run were kept for N days, and their retention period has ended." The entry adds "They are scheduled for removal." only when a parked lease was queued for cleanup. A separate entry, "... kept for N days and have now been removed.", is written only after the provider confirms the delete. Each receipt is released in its own transaction. A failed receipt is logged and skipped, and it is retried after a backoff (5 minutes, doubling, at most 6 hours). The other receipts are still processed. The entries reference a run or a task only if it still exists. If the task was deleted, the decision goes to the server log only. - Upgrade. A `pending_cleanup` lease (not in flight, not stop-only) that an unreleased receipt holds goes back to `retained` and is parked by the sweep. A sandbox that a release before this change already destroyed cannot be recovered. Its receipt ends at its retention time like any other receipt. - Receipt trim (review P1). `abandonUnverifiedRemoteWorkspaceEpoch` trims only released receipts and keeps the last 20 of them. It never drops an unreleased receipt. A new abandonment of the same sandbox and root replaces the unreleased receipt for that root. So a workspace holds at most one unreleased receipt for each distinct sandbox and root abandoned in the last 7 days. ## Verification - `server/src/__tests__/abandoned-workspace-files-hold.test.ts` (19 tests). It follows the lease through the real acquisition path with a failed resume, the real `abandonUnverifiedRemoteWorkspaceEpoch`, the real `sweepPendingCleanupLeases`, and `settleRetainedOrphanSandbox`: - `gives the next run a new sandbox at once and parks the held one, which cleanup and later runs leave alone` - `pulls a held lease already queued for cleanup back to retained, and leaves it on the provider` - `treats a receipt written before release fields existed as unreleased` - `records a retention expiry on the receipt` - `hands the parked lease to normal cleanup once the files are recovered` / `... discarded` - `removes the files only after the retention period, and says removed only after the provider confirms it` - `does not report removal when the provider delete fails` - `records the end of retention without scheduling removal when another receipt still holds the sandbox` - `records the end of retention when it expires before the lease was ever parked` - `wakes the cleanup worker at the receipt's retention end, also after a restart, and then goes quiet` - `releases a receipt whose run was deleted and records it without the run reference` - `releases a receipt whose task was deleted without writing a task entry` - `keeps expiring later receipts when one fails, and retries the failed one after a backoff` - `keeps a held sandbox when the retained orphan settlement reviews it` - `lists the sandbox for the retained orphan settlement again once the receipt is released` - `keeps holding the first sandbox after many later abandonments on another sandbox` - `replaces and cleans the sandbox as before when the receipt names a different sandbox` - `replaces and cleans the sandbox as before when no epoch was abandoned` - Red on the base revision ( |
||
|
|
e363b0e52d |
fix(docker): keep local build output out of the Runner image's build context (#21312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. `master` publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The name of a published image is a digest of every file the Runner build receives from the build context > - `.dockerignore` keeps the Runner's own build output (`packages/paperclip-runner/dist`) out of the context, but not the eval kernel's (`packages/paperclip-eval-kernel/dist`), which the Runner's build also writes > - A tree where the Runner was built therefore has another inputs digest than a clean checkout of the same commit > - The Runner package's Vite builds write five more directories with the same problem > - This pull request adds the missing ignore rules, and a test that finds the next such directory > - The benefit is that a developer's tree and a clean checkout of the same commit name the same Runner image ## Linked Issues or Issue Description Refs #21204, which added the image identity. No issue exists for this change. **Problem or motivation** The inputs digest covers every file the build receives under `packages/paperclip-eval-kernel`. The Runner's build compiles the eval kernel and writes `packages/paperclip-eval-kernel/dist`. After a local Runner build, that directory is in the build context, so the digest of the tree differs from the digest of a clean checkout. A lookup of the published Runner image for such a tree finds nothing. CI checkouts are clean, so no published image is affected. **Proposed solution** Add `packages/paperclip-eval-kernel/dist` and `packages/paperclip-runner/dist-*` to `.dockerignore`, next to the rule for the Runner's own `dist`. Add a test that derives every build-output directory inside the image's inputs and requires a rule for each. **Alternatives considered** - Exclude the directory in the identity script only. Then the digest would not describe what the build receives, which is the property the identity is built on. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This pull request stands alone. It is based on `master` and needs no other open pull request. ## What Changed - `.dockerignore`: two rules. - `packages/paperclip-eval-kernel/dist`: the Runner's own build writes it. - `packages/paperclip-runner/dist-*`: the output of the Runner package's Vite builds (`dist-browser`, `dist-issue-thread`, `dist-scenarios`, `dist-sdk`, `dist-standalone`). The Runner's `build` script does not write them, but a tree where one of those builds ran had the same digest drift. No tracked path matches the rule, and no later `!` rule re-includes those directories. - `server/src/__tests__/runner-image-identity.test.ts`: one new test. For each package among the identity's inputs it collects the `outDir` of every tsconfig a script compiles with `tsc -p`, the `target` directory beside every Cargo manifest a script names, and the `outDir` of every Vite configuration. It requires that the last `.dockerignore` rule matching each directory is an exclusion. Today that is eight directories. A new output directory inside the inputs fails the test until it has a rule. - No image build reads that directory from the host. The Runner stages, the `build` stage of the application image and `docker/daytona-runner` all compile the kernel themselves (`tsc -p`, not incremental), and the eval kernel is a development dependency of the Runner only. ## Verification The inputs digest, as the `runner-artifacts-identity` build target reports it, on a `linux/arm64` Docker host: | Tree | Without the rule | With the rule | | --- | --- | --- | | clean export at `62d673cc3f` | `0e6662ad…` | `0e6662ad…` | | the same, with an eval-kernel `dist` | `e4cf3095…` | `0e6662ad…` | | clean export at `bc71c23e0a` | `88fe2929…` | `88fe2929…` | | the same, with an eval-kernel `dist` | `ac331275…` | `88fe2929…` | | clean export at `731b52453c` | `88fe2929…` | `88fe2929…` | | the same, with two files in `dist-browser` | `44fe867d…` | `88fe2929…` | | the same, with one file in each of the five Vite directories | not measured | `88fe2929…` | - `0e6662ad…` is the digest of the image that `master` published at `25e9a9c2e9`, so the rule does not change the digest of a clean checkout. - A worktree after a real `pnpm --filter @paperclipai/paperclip-eval-kernel build` also gave the clean digest with the rule. - `runner-image-identity.test.ts` (42 tests) and `tests/runner-e2e/daytona-image.test.ts` (19 tests), which reads `.dockerignore`, pass on macOS and in `node:24.22.3-bookworm`. - The new test fails on the `.dockerignore` of `master` and names `packages/paperclip-eval-kernel/dist`. It fails without the `dist-*` rule and names `packages/paperclip-runner/dist-browser`. It fails without either of the two older Runner rules, and with a trailing `!` rule that re-includes one of the directories. Never run: - Any image build with the new rules beyond the identity target, including `.github/docker-context-checks.Dockerfile`. On a CI checkout the directory does not exist, so the rules remove nothing there. - `linux/amd64`. - The full test suite and the full typecheck. CI must supply them. ## Risks - **The sandbox image's content id changes one time.** `.dockerignore` is an input of that id (`tests/runner-e2e/daytona-image-content.ts`). At `62d673cc3f` the id goes from `be208823…` to `1e13a151…`. The next end-to-end run builds a new sandbox image, and the restart-chaos harness needs an image built from this tree. The image's contents do not change. - Low otherwise: the rules remove directories that no image build reads. The `Dockerfile`, `docker/daytona-runner/Dockerfile` and `.github/docker-context-checks.Dockerfile` name none of them and run no Vite build. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
bbafa78fe2 |
fix(retry): offer Retry only when admission can take it, and start held saved-work Retries (#21315)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run fails and its saved work is held for recovery, the task
shows a Retry button. Admission then decides if the Retry can start a
new turn.
> - The task view, the task's execution blocker, and admission each
computed their own answer. The task view said `canRetry: true`. The
blocker said `canRetry: false`, and its own `workspaceRecovery` part
said `canRetry: true`.
> - Admission refused these Retries with no reason. The receipt kept the
generic `execution_recovery` wait ("Retry is waiting for execution
checks"), and each later click joined the same held receipt. Nothing
ran.
> - The refusals had real causes. A preparation proof rejected
host-written workspace notices in the run log. A repair proof did not
accept a snapshot that the failed preparation only restored and never
changed.
> - This pull request fixes both proofs, names every admission refusal
of a held Retry, and makes the task view and the blocker read one shared
Retry answer.
> - The benefit is that Retry starts work in these cases, and Retry is
offered only when admission can take it.
## Linked Issues or Issue Description
**What happened?**
On hosted staging, several blocked tasks showed a Retry button. Pressing
Retry was accepted but nothing ran. The task said "Retry is waiting for
execution checks. Your message is saved." The task view had
`interruption.canRetry: true` and `retryWait.reason:
execution_recovery`. The task's `executionBlocker` had `canRetry:
false`.
- Saved-work recovery failed with `source_unavailable`. The legacy
preparation had failed before the provider started
(`executionRecovery.providerWorkStarted: false`), its sandbox epoch was
abandoned, and its sandbox was destroyed. The blocker contradicted
itself: `workspaceRecovery.canRetry: true`, top-level `canRetry: false`.
- Saved work showed "recovered", but `workspaceRepairRequired` stayed
`true`. The failed preparation had restored a snapshot that an earlier
run produced. The repair proof accepted only a snapshot that the run
itself saved.
**Expected behavior**
The task view, the execution blocker, and admission give the same Retry
answer. A Retry that admission can take starts work. When saved work
cannot be recovered, Retry continues in a fresh session from the
recorded work. Every admission refusal of a held Retry has a named
reason. No Retry is offered for a refusal that another click cannot
change.
**Steps to reproduce**
1. Let a legacy `paperclip_runner` preparation fail before the provider
starts, after the host writes its workspace notices ("Execution
workspace reuse freshness action ...", "Skipping saved session resume
...") to the run log.
2. Let saved-work recovery end with `source_unavailable` and
`use_alternative_strategy`.
3. Press Retry.
4. Observe that the Retry receipt is held with `executionWait.reason =
execution_recovery` and no run starts.
Or:
1. Let a legacy preparation restore the task's saved snapshot and fail
with `setup_failed` before the provider starts.
2. Let saved-work recovery complete.
3. Observe that the blocker keeps `workspaceRepairRequired: true`, and
that Retry is held with `execution_recovery`.
**Paperclip version or commit**
Staging on `origin/master` `62d673cc3f` (after #21267 and #21295).
**Deployment mode**
Hosted cloud (staging), remote Daytona sandbox runner.
**Agent adapter(s) involved**
`paperclip_runner` (legacy runtime runs, native agent).
Refs #21267, #21295.
## What Changed
- `hasVerifiedPreparationLog` (`verified-preparation-log.ts`) now
accepts a log made only of host notices that the host writes before it
hands the turn to an adapter: the mentioned-skills line (first record
only, checked against the saved skill selection) and the workspace
freshness and session-reset notices. Every record must have the same
attempt, sequential `seq`, `stdout`, and the excerpt and watermark must
match. Any other line still keeps the run held.
- `verifiedUnchangedRestoredSnapshot`
(`remote-workspace-snapshot-reuse-receipt.ts`) proves that a failed
preparation's saved state is the snapshot it restored. It needs
`executionStage: preparing`, `providerWorkStarted: false`, exactly one
succeeded restore of that snapshot at that epoch and lease, and no
snapshot save by the run. `verifiedWorkspaceRepairReceipt` uses it next
to the existing reuse proof.
- `savedWorkOffersRetry` (`workspace-recovery-receipt.ts`) is the one
rule for Retry on a saved-work hold. Retry is offered when the saved
work is recovered and verified, or when recovery is exhausted in a state
that only a Retry restarts (a new recovery generation, or the
fresh-session fallback after `source_unavailable`). Retry is not offered
while recovery re-arms by itself (a stop recheck or a newer capture
policy), or while a copyback owner's stop is unconfirmed. The blocker
uses this rule for the top-level `canRetry` and for
`workspaceRecovery.canRetry`, so the two parts cannot contradict.
- `task-retry-eligibility.ts` lists the held-Retry refusals that another
click cannot change. The task view and the blocker both use it.
- Admission (`explicit-native-continuation.ts`) names each refusal of a
held request that was silent before: `previous_execution_unverified`,
`execution_settling`, `cleanup_pending`, and `remote_cleanup`. It names
a refusal only when an execution blocker holds the task, so a request
without a hold still takes the normal path.
- The task view now reads the blocker's answer for a task with a
saved-work hold. It shows no Retry when the blocker offers none, and it
names the recovery wait.
## Verification
- Red/green: `cd server && npx vitest run
src/services/saved-work-retry-eligibility.test.ts`. The tests fail on
`origin/master` sources and pass with this change. They are built from
the staging shapes:
- offers the same Retry in the task view and the blocker when saved work
is no longer available
- offers no Retry in either view while saved-work recovery waits for an
unconfirmed file copy to stop
- continues in a fresh session from the recorded work when saved work is
no longer available
- verifies the repair of a restored snapshot that the failed preparation
never changed
- keeps the repair unverified when the failed preparation may have
changed the restored snapshot
- names a refusal that another click cannot change and offers no Retry
for it in either view
- names the admission refusal of a held Retry and offers Retry only when
admission would take it
- `npx vitest run src/services/verified-preparation-log.test.ts`: new
cases accept the host notices and keep a provider line, a different
attempt, or a late skills line held.
- Related suites: deferred execution wait, execution blocker read,
explicit native continuation, governed wait runner loss, legacy
preparation retry hold, remote workspace snapshot reuse, task
interruption presentation, task retry, and all `workspace-recovery-*`
suites.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Medium-low. The preparation-log proof accepts more host lines. Each
accepted line has a fixed host format, and the run must still pass every
other preparation proof (events, leases, no provider evidence).
- The restored-snapshot proof applies only when the run recorded that
the provider never started and it never saved its own snapshot.
- Admission now records a named wait instead of the generic one for some
held requests. It changes no admission decision.
- The task view calls `getExecutionBlocker` for tasks that offer Retry
and have a saved-work hold. These tasks are few, so the extra reads are
small.
- Not covered here: a copyback owner on a replaced deployment still
needs its stop confirmed before saved work can be copied out of a full
sandbox. The task view now shows that wait instead of a Retry that
admission refuses.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b555087f0f |
fix(native): continue again when a deploy interrupts an automatic continuation (#21314)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner (Daytona) that the server controls; a deploy restarts the server while runs are in flight > - When the restart's reattachment proves the Runner process gone mid-turn, #21039 makes the server request one automatic Retry that continues in a fresh session from the saved work, without replaying the turn > - Staging deploys every 20 to 45 minutes, and Codex runs often take longer, so that automatic continuation is often interrupted by the next deploy too > - The hook treated that second loss as "automatic Retry already used" and left the task blocked for a person (`native_continuation_requires_reconciliation`) > - This pull request gives an automatic continuation one more fresh-session continuation each time a server restart or deploy took its controller, and a deploy spends no automatic budget > - The benefit is that a task interrupted by repeated deploys continues by itself, and no provider turn that may have run is sent again ## Linked Issues or Issue Description Refs #21039, Refs #21052, Refs #21272, Refs #21023, Refs #21120, Refs #21161 **What happened?** On staging, the largest failure class for a Codex agent on Daytona was `runner_remote_process_identity_unavailable` with error code `native_continuation_requires_reconciliation`: 18 runs in 3 hours. The error name is misleading. It is the shared code for every refused remote reattachment (`REMOTE_RUNNER_RECOVERY_ERROR_CODE`). In every sampled run the refusal was `process_verification` / `process_not_running`: the Runner process was gone, its stderr log was empty, and the last turn event was `turn.submitted` or `turn.accepted` (unsettled). It was not a missing identity file. Sampled 12 such runs. In 9 the #21039 hook worked: it recorded `native.interrupted_turn_outcome_unknown` and requested the fresh-session Retry. The tasks that stayed blocked had one shape: 1. A run lost its Runner after a deploy. The hook requested the automatic Retry (key `task-retry:restart-interrupted-turn:<run>`). 2. The Retry run started a fresh session. The next deploy came during its turn (a graceful `SIGTERM` shutdown transition, then "Recovering the existing native runner process after server restart"). 3. The same-run reattachment found that Runner gone too (`process_not_running`). 4. The hook returned `automatic_retry_used`, because the run's Retry intent came from the hook. The task stayed blocked. Examples: a `task_retry` run created 40 s after its predecessor failed (the hook's Retry), interrupted by the 05:17 deploy, failed at 05:18, task blocked. A second chain (assignment, then the hook's Retry) failed the same way at 05:12. **Expected behavior** A deploy is not the task's failure. When a server restart or deploy took the controller of an automatic continuation and its Runner is proven gone, the server continues the task again in a fresh session from the saved work. It never replays the turn. A loss without restart evidence keeps the existing one-time rule. **Steps to reproduce** 1. Start a long Codex run in a remote Daytona Runner. 2. Deploy the server during its provider turn so that the reattachment finds the Runner gone. The server requests the automatic Retry. 3. Deploy again during the Retry run's turn so that its Runner is also lost. 4. See the Retry run fail with `runner_remote_process_identity_unavailable` and the task stay blocked with no new run. **Paperclip version or commit** Staging Core `25e9a9c2e9`. Fixed on `master` after `731b52453c`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`, `codex_app_server`). ## What Changed - `restart-interrupted-turn.ts`: `lostControllerToRestart` finds restart evidence on the run: a `native.recovery.transition` with `restartKind: "graceful"` (shutdown), or a restart claim (`reattach_remote_runner`, `reattach_existing_runner`, `resume_dead_runner`, `bootstrap_incomplete`) without the `same_run_retry` trigger. - Both `automatic_retry_used` refusals (a person's Retry that the hook requested, and a system retry of owed work) now continue when the run has restart evidence. A deploy interruption spends no budget, so a task that only deploys interrupt never runs out. Each continuation needs a new deploy during its run, so deploys bound it. A loss without restart evidence keeps the one-time rule. All other gates are unchanged: proven Runner exit, unknown-turn record, the issue still assigned and open, idempotency per run, and the Retry's admission gates. - `doc/execution-semantics.md` describes the rule. ## Verification - Red before the fix, green after: `server/src/services/runner-lost-mid-turn-continuation.test.ts` › "an automatic continuation that the next deploy interrupted". The test seeds the staging sequence: a lost run, the hook's Retry run with a `turn.submitted` turn, the shutdown and restart transitions with staging payloads, then a same-run reattachment that proves the Runner gone. On the old code the two continuation tests return `automatic_retry_used`. With the fix they request one fresh-session Retry, record the unknown turn outcome once, recover the saved work and queue the successor with `forceFreshSession: true`. - A chain of 12 deploy continuations, each with a system retry that inherits its Retry receipt, still continues (red on the first revision of this PR, which had a budget of 8). Losses without deploy evidence stay refused (with and without earlier deploy continuations). - `npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` 23/23 pass. `governed-wait-runner-loss.test.ts` and `workspace-recovery-retry-copyback.test.ts` 103/103 pass. - `pnpm --filter @paperclipai/server typecheck` passes. - `pnpm verify` exits 2 (a global doc input changed), so required CI supplies the broad coverage. ## Risks - Low to medium. A task whose Runner is lost on every deploy now continues after each deploy instead of once. Each continuation is a full run, so it costs tokens. The deploy count bounds it, because each continuation needs a new restart during its run. - Not fixed here: why the reattached Runner is gone after a deploy. In the sampled runs the reattachment after the restart hung in `native session provider recovery timed out after 900000ms`, and the next attempt found the process gone with empty stderr. That needs its own investigation. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, browser read-only API GETs on staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
03f9b83e52 |
feat(ci): report the Runner image on master and build the app image's Runner path on pull requests (#21307)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work, and `master` now publishes it alone as the image `ghcr.io/paperclipai/paperclip-runner` > - The application image will soon copy that Runner instead of compiling its own. Part 1 of this stack adds the resolver that picks the Runner for a build > - Two gaps make that next change risky. The resolver has never run in CI on `master`, so its lookup time and how often a commit has a published Runner are unknown. And `docker.yml` has no pull-request trigger, so a change to how the application image gets its Runner first runs after it merged > - This pull request adds a report-only job that runs the resolver on every `master` push, and a pull-request check that builds the Runner path of the application image > - The benefit is real data from `master` before any build depends on the resolver, and a failed check on the pull request instead of a failed image build on `master` ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** The resolver decides whether a build copies the published Runner. On `master`, a commit that changes the Runner's inputs is ahead of its publication for some minutes, and nobody knows yet how often an application build will meet that. The application image itself is built only on `master` and on tags. The change that removes the Runner's compile stages from that image will edit the `Dockerfile`, and today nothing builds those stages on a pull request. **Proposed solution** Add one job to `docker.yml` that only reports, and one new workflow that only checks. Neither changes what an image contains, and no existing job reads either of them. **Alternatives considered** - Run the resolver inside the build legs. A failure there could fail the image build. A job of its own, with `continue-on-error`, cannot. - Give the lookup the job token. The Runner package is public, so an anonymous lookup works and the job needs no credential. - Build `production` in the pull-request check. It adds three `COPY` instructions and a readability check to the two stages that this check builds, at the cost of the tool layer and two more copies of several gigabytes. - Trigger the check on every Runner source change. `runner-image.yml` already builds the Runner on those pull requests, and CI builds the server and the UI. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 2 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path (this pull request) 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `.github/workflows/docker.yml`: new job `runner_image_report`. - It runs only for a `master` push in this repository. - It runs `runner-image-resolve.mjs inputs`, then `resolve`, and the result goes to the job summary with the line "Report only: no build read this result." - It has `continue-on-error: true`, declares no `outputs`, and no job needs it. Its failure does not fail the run. - It holds no credential. Its only permission is `contents: read`, the checkout does not keep its token, and the lookup is anonymous. It does not log in to a registry and writes to none. - Time limits: 8 minutes for the job, 5 for the step that builds, 2 for the lookup. - It is the first job in the file. Existing tests cut the other jobs out of the file by name, up to the next job or to the end of the file, so no other place leaves those tests unchanged. - `.github/workflows/docker-app-check.yml` (new): on a pull request, builds the `build` and `runner-provider-pack` targets with `PAPERCLIP_BUILD_COMMIT` set, for `linux/amd64` only. - `permissions: {}` at the top, `contents: read` for the job, no credential, no `pull_request_target`. - It exports nothing (`--output type=cacheonly`) and pushes nothing. It reads the public build cache that `docker.yml` writes and never writes it. - It refreshes the lockfile in the build context first, as `docker.yml` does. - Time limits: 60 minutes for the job, 45 for the two builds. - Trigger paths, and why: | Path | Reason | | --- | --- | | `.github/workflows/docker-app-check.yml` | the check itself | | `Dockerfile`, `.dockerignore` | the recipe, and the files it receives | | `server/scripts/prepare-runner-vendor.mjs` | the step of `build` that reuses or rebuilds the Runner | | `packages/paperclip-runner/scripts/build-inputs.mjs` | the build stamp that step compares with the files the image build received | - `server/src/__tests__/docker-runner-workflows.test.ts` (new, 90 lines): only security properties. The report job has no credential and one read permission, nothing can read it, and its two steps have time limits. The check has no credential, cannot publish, and exports nothing. In both, every action is pinned by commit and no shell script holds an expression. - `doc/runner-image-contract.md` and `doc/DOCKER.md` name the job and the check. ## Verification macOS (Node 26.12.0): - In `server/`: `docker-runner-workflows`, `runner-image-workflow`, `docker-build-stamp`, `container-init-reaping`, `cloud-image-bundled-plugins`, `docker-image-layer-reuse` and `cloud-image-sentry` tests: 52 tests pass in 7 files. - `tests/runner-e2e` workflow tests (`workflow-security`, `private-trust-workflow-security`, `workflow-image-only`, `paid-runner-routing`): 27 tests pass in 4 files. - `node --test .github/scripts/tests/*.test.mjs scripts/merge-group-image.test.mjs scripts/reuse-canary-image.test.mjs scripts/cloud-source-verification.test.mjs scripts/standard-image-contract.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: 962 of 962 pass. - Both workflow files parse as YAML. - `docker buildx build --target runner-artifacts-context --cache-from type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-amd64 --output type=cacheonly .` succeeds, so both flags are accepted and the cache is readable. Linux (`node:24.22.3-bookworm`, with `zstd` and `jq` added): the same three sets pass: 52, 27 and 962 of 962. The two commands the report job runs were run by hand, with no credential, on clean exports of the tree. For the inputs `0e6662ad…d1ca` the answer was `published`, with the two index digests that `master` published. For a commit whose image was not yet published the answer was `inline`, `absent`. Part 1 has the output. With the job at the end of `docker.yml`, two existing tests failed. They are the reason for its place: `docker-canary-promotion.test.mjs` reads the last job to the end of the file, and `release-verify-workflow.test.mjs` allows no BuildKit configuration after `merge-and-push`. Never run: - Either workflow. Only GitHub can run them. This pull request starts the check by its own path; the report job first runs on `master`. - The two builds of the check (`--target build`, `--target runner-provider-pack`) on a local host. Their time and disk use on a hosted runner are not known. - The full test suite and the full typecheck. CI must supply them. ## Risks - The check builds the whole `build` stage, with the UI and server builds. Its duration is not measured. It runs only when one of five paths changes, and its first real run is on this pull request. - `docker.yml` holds one concurrency slot for each ref until all its jobs end. The report job normally ends long before the build legs. A stalled job can hold the slot for at most 8 minutes when the legs end sooner. - `scripts/watch-release-source.mjs` warns about every job of the image run that waited more than 60 seconds for a runner. The report job is one more job that it can name. - `scripts/merge-group-image.mjs` refuses to promote a merge-queue image for a push that changes `docker.yml`. The push that lands this change is such a push. The merge queue is not enabled. - The lookup has 2 minutes. The registry client retries a failing request for longer than that, so a registry outage ends the step by its time limit, and the job then reports nothing. The run stays green. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ff91bfe2dc |
fix(runner): withhold Claude's self-scheduling tools from Paperclip ACPX sessions (#21276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip owns scheduling. An agent uses Paperclip tools such as task monitors and routines to continue work later. > - Paperclip Runner runs Claude Code through the ACPX sidecar. > - Claude Code offers its own scheduling tools to the model: `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. With these tools the provider can wake itself. Paperclip does not see or govern that work. > - Today the Runner stops the Claude process when a turn ends, so these tools have little effect. A later change keeps the Claude process alive between turns. > - This pull request removes the four tools from Claude sessions that Paperclip starts through ACPX. > - The benefit is that a Claude agent schedules work only through Paperclip. ## Linked Issues or Issue Description No public issue exists. The issue is described here. Refs #19912 (the design for turn-scoped ACPX MCP authority). **What existing behavior does this improve?** The set of tools that Claude Code offers the model in a Paperclip ACPX session. **Subsystem affected** Paperclip Runner, ACPX provider (Claude). **Current behavior** A Paperclip ACPX Claude session offers the model 26 built-in tools. Four of them are `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. **Proposed behavior** - The Claude settings that Paperclip writes into the session home deny the four tools. Claude Code then does not offer them. - Each Claude launch sets `CLAUDE_CODE_DISABLE_CRON=1`. This stops the cron scheduler of Claude Code and removes the three cron tools. - All other tools stay. `SendMessage`, `ListAgents`, `Workflow`, `Agent` and `TaskStop` do not change. **Reason and benefit** The provider cannot schedule its own wake-ups. Paperclip tools stay the only way to schedule work. ## What Changed - `packages/paperclip-runner/src/drivers/acpx/permission-policy.ts`: new constant `CLAUDE_SELF_SCHEDULING_TOOLS` with the four tool names. - `packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts`: the Claude `settings.json` now always has `permissions.deny` with the four tools, in each permission mode. The `allow` rules do not change. - `packages/paperclip-runner/src/drivers/acpx/environment.ts`: `createSanitizedAcpxSpawnInput` sets `CLAUDE_CODE_DISABLE_CRON=1` for each Claude launch. The host cannot override it. - New test `claude-scheduling-tools.claude-acp.test.ts`. It prepares the session home with the production `prepareAcpxRuntimeSandbox`. It then runs the pinned `claude-agent-acp` and the pinned Claude Code client against a local fake Anthropic API. It reads the tools that the client offers the model. - Unit tests in `environment.test.ts` and `runtime-sandbox.test.ts`. One assertion in `runtime-host.test.ts` now expects the `deny` list. - No change to the `claude-agent-acp` patch, the lockfile or a pinned digest. No change to Codex. ## State saved by the previous release - A session that the previous release saved continues with no change. The new restriction is not part of a saved identity or digest. - Each time Paperclip starts Claude for a session, it writes its own `settings.json` again with the deny list and sets `CLAUDE_CODE_DISABLE_CRON`. This includes a resumed session and a start after an upgrade. - A resumed session keeps its identity, its home and its transcript. It loses the four tools from its next start. - A Claude process that the previous release started keeps its tools until it stops. Paperclip does not refuse it and does not classify it differently. Its next start gets the restriction. - No refusal code is necessary, because no saved state becomes incompatible. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts` - Without the product change: `2 failed` (`expected [ Array(26) ] to not include 'CronCreate'`). - With the product change: `2 passed`. - `npx vitest run src/drivers/acpx/claude-scheduling-tools.claude-acp.test.ts src/drivers/acpx/environment.test.ts src/drivers/acpx/runtime-sandbox.test.ts src/drivers/acpx/runtime-host.test.ts`: `4 passed (4)` files, `147 passed` tests. - `npx tsc --noEmit -p tsconfig.json` in `packages/paperclip-runner`: no errors. - Tools offered after the change, isolated home: the same 26 tools without `CronCreate`, `CronDelete`, `CronList` and `ScheduleWakeup`. - `npx vitest run src/drivers/acpx/runtime-host.test.ts -t "previous release"`: the new test "resumes a Claude session saved by the previous release and withholds self-scheduling tools on relaunch" fails without the product change and passes with it. The test saves a session, puts its home into the shape of the previous release, and resumes it through `AcpxRuntimeHost.open`. - `npx tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`: no errors. - The fixture in `claude-scheduling-tools.claude-acp.test.ts` now stops the process group of the provider and waits for it before it removes the fixture directory. - Not run: a live run with a real Claude account, and a remote Runner. Not run: `pnpm verify` and the full suite. CI supplies that coverage. ## Risks - **A preconfigured home still offers `ScheduleWakeup`.** In that mode Paperclip does not write the Claude settings of the user, so the deny list does not apply. The environment variable removes only the three cron tools. To remove `ScheduleWakeup` there, the Runner must pass `disallowedTools` to the session. ACPX 0.22 does not forward that option. This is a follow-up. - **The new test runs the real pinned Claude Code client.** It needs the platform binary, as `runner-tool-bridge.claude-schema.test.ts` does today. It uses no credentials and no network. - **A prompt or skill that tells Claude to use these tools no longer works.** No Paperclip instruction refers to them. - The prepared harness home and remote Runners use the same code path. They are not tested separately. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change, reviewed the diff and ran the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), investigated the options and wrote the change and the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
bcadbde449 |
feat(runner): record the OpenRouter host that served each OpenCode model call (#21298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - OpenCode agents can use models through OpenRouter, and the Runner proxies those model calls > - OpenRouter routes one model across many hosts, and these hosts use different quantizations (fp4, mxfp4, fp8) > - Chat evaluations with identical prompts behave very differently between runs, but the run log does not record which host served a call > - The run usage shows `provider: "deepseek"` and `biller: "openrouter"`, and `providerRequestId` is always `null` > - This pull request records the serving host and the OpenRouter generation ID for each model call in the run log > - The benefit is that we can compare evaluation cohorts by host, and the cost event names a real OpenRouter generation ## Linked Issues or Issue Description **Subsystem affected** Runner OpenCode provider proxy, runnerd usage normalization, and native run usage accounting. **Problem or motivation** OpenRouter's default routing sends a request to one of many hosts. Each host can use a different quantization. Chat evaluation results change a lot between runs with identical prompts. Paperclip does not record which host served a call, so we cannot compare results by host. `usageJson.providerRequestId` is always `null` for OpenCode runs. **Proposed solution** Every OpenRouter completion response has a top-level `provider` field and an `id` (the generation ID). Each streamed chunk has them too. The Runner proxy reads them from the first chunk after it forwards the bytes. The Runner attaches them to the usage of that call. The values go to the run log, and the run's first generation ID goes to `usageJson.providerRequestId`. **Alternatives considered** A new PRP event type such as `provider.upstream.served` would need a new event enum value. An older control plane rejects an unknown event type. A `harness.diagnostic` event shows in the task chat as a diagnostic. The usage payload already flows to every consumer, and older consumers ignore extra fields. Thus this change uses the usage payload. **Additional context** Run-log change only (AGENTS.md section 7). The data stays in `heartbeat_run_events` and `heartbeat_runs.usageJson`. It does not touch `packages/shared/src/telemetry/` or OpenTelemetry. ## What Changed - New `createUpstreamServedTap` (`packages/paperclip-runner/src/drivers/opencode/upstream-served.ts`). It is a pass-through stream. It pushes each chunk downstream first, then it reads `provider` and `id` from the first SSE `data:` event (or from a JSON body at its end). It scans at most 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies and unknown content types. Any parse problem or recorder error stops the scan and leaves the response unchanged. - `startOpenCodeProviderProxy` uses the tap only for 2xx responses. The OpenCode session keeps the most recent served call. Each `item.completed` usage payload and `session.usage()` now carry `upstreamProvider` and `providerRequestId`. The usage dedupe fingerprint does not include these fields, so the number of usage events does not change. - The OpenCode app-server proxy adds the two fields to `thread/tokenUsage/updated`. Rust `usage.reported` now fills its existing `providerRequestId` field and adds an optional `upstreamProvider`. The TypeScript runnerd transport keeps both fields when it rehydrates the usage notification. `usage.schema.json` documents `upstreamProvider` (the protocol manifest and schema bundle are regenerated). - Server native turn accounting keeps the first generation ID the run reports. The usage checkpoint and the native adapter result carry it as `providerRequestId`, so `usageJson.providerRequestId` and the final cost event name it. - `doc/run-log-events.md` documents the fields in a new "OpenRouter serving host" section. ## Verification - Red/green: `opencode-server-driver.test.ts` "passes OpenRouter's stream through unchanged and records the host that served it on the turn's usage". The test sends a streamed fixture with a comment line, a host chunk, a malformed chunk, and `[DONE]` through the proxy. It checks that the bytes are identical and that the turn's usage names `Novita` and the generation ID. The test fails without the proxy wiring. - `upstream-served.test.ts`: streamed fixture, chunk split inside a multibyte character, non-streamed JSON, malformed chunks, a truncated JSON body, an encoded body, the 64 KiB limit, and a recorder that throws. - `opencode-proxy-usage.test.ts`, `runnerd-codex-transport.test.ts` (rehydration), Rust `provider_events::tests::carries_the_serving_host_and_generation_on_proxy_usage`, and server `native-usage-accounting.test.ts` (first generation ID stays stable across turns). - Local: OpenCode driver, OpenCode proxy, and runnerd transport suites (31 files, 596 tests) pass. `cargo test -p paperclip-runner-core` passes (one timing test failed once under load and passed alone). Runner and server `tsc --noEmit` pass. Protocol manifest, type, and coverage checks pass. - Latency: a local benchmark proxied a 400-chunk SSE stream 200 times with the tap and 200 times without it. The median time to first byte was 292 µs with the tap and 259 µs without it (+33 µs, which is the extra stream stage). The total stream time changed from 491.4 ms to 492.5 ms. The tap spends about 5.5 µs of CPU to inspect the first chunk, after it has forwarded that chunk. OpenRouter time to first token is hundreds of milliseconds. - Paid evidence: one single-case run of `chat-features.runner-opencode.local.skills.fork-read-only` on this branch. The result is in a comment on this pull request. ## Risks - Low risk. All new fields are optional and additive. An older control plane ignores the extra usage fields. `usage.schema.json` is not bound by the event schema, and no validator enforces it on `usage.reported`. - Saved state from the previous release: this change adds no persisted run, session, checkpoint, or launch state. A saved session resumes unchanged. A run that is in flight during a deploy reports usage without the new fields until its Runner is new. `usageJson.providerRequestId` then stays `null`, as before. - `providerRequestId` on the run and its cost event is the run's first generation ID, not each call's ID. The value is stable, so a late usage receipt still matches the posted charge. Per-call IDs and hosts are in the run log. - If OpenCode starts a new model call before it reports the usage of the previous call, a usage event can name the newer call. OpenCode reports a message's usage when its step ends, so this is not expected. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as an agent with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c730872974 |
fix(recovery): a chat turn waiting on the person is not stranded work (#21302)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In a chat, the agent's turn ends and the conversation waits for the person's next message. The stranded-work sweep must leave that wait alone > - The sweep recognizes a finished chat turn by `finalizationReasonCode: "conversation_turn_finished"` on the run record > - Run completion re-reads the run through the 64 KiB read projection. When the stored result is larger, that projection drops the finalization fields, and completion wrote the reduced copy back > - The chat turn then lost its finalization reason. Its reply was not presented, and the sweep queued `issue.productive_terminal_continuation_recovery` 30 s later. That is an extra run while the agent waits on a person > - This pull request keeps the full stored result through run completion, and makes the sweep read the committed decision from its durable rows > - The benefit is that a turn waiting on a person is never treated as stranded work ## Linked Issues or Issue Description **What happened?** In the paid chat eval `app-permissions.new-quarantined-action`, the CEO's chat turn yielded with a `response_wake` continuation and waited for a person to allow an app action. The stored run result was 113,937 bytes. Its saved record had the read projection's `truncated`, `truncationReason: oversized_result_json` and `originalSizeBytes` fields and no `finalizationReasonCode` or `decisionId`. The run's presentation decision was `interaction_owns_conversation_response`, so no reply comment was written. 30 s later the sweep queued `issue.productive_terminal_continuation_recovery`, and the case failed its run limit. **Steps to reproduce** 1. Finish a native chat turn that yields `response_wake`, so finalization records `conversation_turn_finished`. 2. Let the stored run result be larger than 64 KiB when run completion reads it. 3. Run completion writes back the reduced copy without `finalizationReasonCode`. 4. Run the stranded-work sweep. It queues a productive continuation run. **Expected behavior** The run record keeps its finalization fields, and the sweep leaves the waiting chat turn alone. No new run starts until the person replies. ## What Changed - `server/src/services/heartbeat/run-completion.ts`: run completion reads the run with `unsafeFullResultJson` before it merges the native result. It no longer writes the read projection's reduced copy back. - `server/src/services/native-runtime/native-response-wake-wait.ts`: `hasCommittedNativeConversationTurnWait` checks the run's committed finalization decision (`conversation_turn_finished`) through `native_run_finalizations` and `status_decisions` for the conversation's own agent. - `server/src/services/recovery/service.ts`: the sweep's finished-chat-turn skip also uses that durable check, so runs already saved with the reduced copy are recognized. - Tests: `run-completion.test.ts` keeps the finalization record when the stored result is larger than the read projection. `heartbeat-process-recovery.test.ts` finalizes a real chat turn with the eval's yielded result, saves the reduced run record, and expects two sweeps to start no run. ## Verification - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts -t "larger than the read projection"` — fails on master (the saved result is the truncated copy) and passes here. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "chat turn that yielded"` — fails on master (`continuationRequeued: 1`) and passes here. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts src/__tests__/heartbeat-process-recovery.test.ts` (614 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Run completion now reads the full stored result for one run. This is one row; the read projection still serves every API read. - The sweep check adds one indexed query for chat issues whose latest run succeeded and whose run record lacks the reason. - Previous-release saved state: no persisted shape changes. Runs already saved with the reduced copy are covered by the durable decision check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>nightly/v2026.1113.0-nightly.0 canary/v2026.1113.0-canary.3 |
||
|
|
f79e2c5d84 |
feat(ci): add the Runner image resolver and a no-rebuild mode for the Runner vendor step (#21306)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - `master` now publishes the Runner alone as the image `ghcr.io/paperclipai/paperclip-runner`, one time for each set of Runner inputs (#21204, #21229, #21230) > - The application image still compiles its own Runner on every build. The next steps make it copy the published Runner instead > - Before a build can do that, it must answer one question the same way as every other build: "does a published Runner exist for exactly this tree?" It must also fail clearly when the Runner it was given does not match the tree > - This pull request adds the script that answers the question, and a mode of the Runner vendor step that fails instead of rebuilding > - The benefit is that the later change to the image build is small, and that a wrong or unreachable answer can only mean "build the Runner yourself" ## Linked Issues or Issue Description Refs #21204, #21229 and #21230 (the Runner image). No issue exists for this change. **Problem or motivation** Three builders make images that contain the Runner: `docker.yml`, the merge-queue build and the release preview build. Each one would have to repeat three steps to find the published Runner of its tree: check the lockfile, compute the inputs digest in the build, and look up `inputs-<digest>`. Three copies can decide differently. A build that copies a Runner built elsewhere also has no Rust toolchain. If the copied Runner does not match the tree, `prepare-runner-vendor.mjs` starts a rebuild, and the build fails late at `cargo` with a message that hides the cause. **Proposed solution** Add `scripts/runner-image-resolve.mjs`, the one place that decides. Add `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` to `prepare-runner-vendor.mjs`. Nothing calls the resolver and nothing sets the variable in this pull request, so no workflow job changes. **Alternatives considered** - One command that builds and looks up. Two commands keep a registry credential, when a caller has one, out of the step that builds. - A wait for the publication of the commit. The decision whether the application build waits is open, so the resolver has no wait mode. It reports `absent` and stops. - A build argument that names the expected inputs. No `Dockerfile` declares one yet. The change that makes the application image copy the Runner will add both ends together. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 1 of 4 of a stack ([stack #21309](https://github.com/paperclipai/paperclip/stack/21309)). Merge the parts in order: 1. #21306: the resolver and the no-rebuild mode (this pull request) 2. #21307: the report-only job in `docker.yml` and the pull-request check of the application image's Runner path 3. #21308: reading a retained provider pack from its Runner image 4. #21310: a registry build cache for the Runner image build ## What Changed - `scripts/runner-image-resolve.mjs` (new, 185 lines), with two commands that share a directory: - `inputs --dir <dir>` exports the `runner-artifacts-context` target, runs the lockfile check on that export, and builds the `runner-artifacts-identity` target when the lockfile is current. It reads no credential. A lockfile that is behind stops there: no image is published for such a tree. - `resolve --dir <dir>` looks up `inputs-<digest>` and prints `mode`, `reason`, `inputs_digest`, `image`, `slim_image`, `build_contexts` and `labels`. It also writes them to `$GITHUB_OUTPUT` and a short report to `$GITHUB_STEP_SUMMARY`. - `mode` is `published` only for a complete publication whose full index names these inputs and both platforms. Both images are pinned by index digest. The slim digest is the one the full index names in `io.paperclip.runner.slim-digest`; the slim tag is never read on its own. - Every other answer is `inline`, with the last four values empty: | `reason` | Meaning | | --- | --- | | `lock-behind` | `pnpm-lock.yaml` is behind the Runner's manifests | | `absent` | no tag exists | | `incomplete`, `broken` | a publication stopped, or its two tags do not belong together | | `mismatch` | the full index names other inputs, or lacks a platform | | `unknown` | the registry refused, failed or could not be reached, or the lookup threw | - The resolver reuses `runnerLockState`, `readRunnerImageManifest`, `runnerPublicationState` and `runnerImageTags` from `scripts/runner-image-publish.mjs`. That file now exports `environmentCredentials`; it has no other change. - `server/scripts/prepare-runner-vendor.mjs`: with `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1`, the step reuses a current build as before. When the build is not current, it prints the reason, starts no rebuild, leaves the stamp in place and exits with status 1. It also fails when `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` is set at the same time. Without the variable the step is the same as before. - `doc/runner-image-contract.md` has a new section, "Choosing the Runner for an image build". `doc/DEVELOPING.md` describes the new variable. - The resolver is not an input of the Runner image, so the inputs digest does not change. ## Verification macOS (Node 26.12.0), in `server/`: - `node node_modules/vitest/vitest.mjs run scripts/prepare-runner-vendor.test.mjs src/__tests__/runner-image-resolve.test.ts src/__tests__/runner-image-publish.test.ts src/__tests__/runner-image-identity.test.ts src/__tests__/runner-image-workflow.test.ts src/__tests__/server-package-build-script.test.ts`: 179 tests pass in 6 files. 27 of them are new resolver tests and 3 are new tests of the vendor step, whose end-to-end test also has a new case. - `tsc` on the new test file, with the settings of `server/src/__tests__/tsconfig.json`: no error. Linux (`node:24.22.3-bookworm`, GNU tar 1.34, the worktree mounted read-only): the same files, 179 tests pass. Real runs, on clean exports of the tree (`git archive`), with no credential, on a `linux/arm64` Docker host: - On `master` at `62d673cc3f`: `inputs` gave `lock=current` and `inputs_digest=0e6662ad9c8c6451556c76bf326723645d4afaab2decc73cd5e0025f7dd8d1ca`. `resolve` gave `mode=published`, `image=ghcr.io/paperclipai/paperclip-runner@sha256:84f2c051aa4ed16c5cc138c08e3533fa417e92177f7d15af5f2aa3e0ba467162` and `slim_image=ghcr.io/paperclipai/paperclip-runner@sha256:f63ea43666ed708f1b0e5d2ade8bdc002885bf04252b9ed6cee21f7920e54f3c`. These are the two indexes that `master` published for those inputs. - The same tree with one line added to a Runner source file: another digest, `mode=inline`, `reason=absent`. - On `master` at `bc71c23e0a`, which changed the Runner's inputs, before its image was published: digest `88fe2929321b4706c77f11a098572ce990fec4449d505544c62ef3a84c28308a`, `mode=inline`, `reason=absent`. - Four deliberate faults in the resolver each make a test fail: no index check, only `absent` is inline, the identity is built for a lockfile that is behind, and the slim image is taken by tag. Never run: - A lookup with a registry credential. The package is public, so no caller needs one now. - Any workflow. No workflow calls the resolver in this pull request. - `linux/amd64`. - The full test suite, the full typecheck and `pnpm build`. `pnpm verify --explain` reports exit 2, "broader verification required", so CI must supply them. ## Risks - Low for existing behaviour: nothing calls the resolver, and the vendor step is the same unless the new variable is set. - The resolver does not verify the attestation of the image. The workflow that consumes the image must do that from the digest the resolver reports. - The resolver reports both build contexts, `runner-artifacts` and `runner-artifacts-slim`. A build that reads only one stage gets one context it does not use. - A `broken` publication is `inline`, not a failure. The Runner image workflow fails for that state. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
731b52453c |
fix(environments): verify workspaces on run leases and release settled tasks (#21289)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns run leases and delegates remote execution to environment plugins. > - A retry must verify its old checkout on the placement that will run the task. > - A completed task must still release its provider resources. > - This change uses the existing lease for workspace verification and always calls the release hook after settlement. > - Saved settlement proof lets cleanup retry without repeating execution work. ## Linked Issues or Issue Description Refs #21237. That PR records stop proof before lease release. This PR keeps provider cleanup after settlement and changes workspace verification sequencing. Both touch task completion; preserve both changes if the merge order requires conflict resolution. **What happened?** Workspace verification ran before a portable task had a lease, so a provider could not verify the checkout on its selected placement. Core also skipped the provider release hook after successful task completion. This could leave resources allocated after the run ended. **Expected behavior** Verify the old checkout through the selected run lease. A confirmed refusal can use the authorized fresh-workspace fallback. A transport failure must remain retryable. Call the provider release hook after settlement and retry failed cleanup without changing the run result. **Steps to reproduce** 1. Retry a task whose prior checkout uses plugin-managed storage and portable task placement. 2. Verify that the plugin receives the same task lease for verification and preparation. 3. Complete a task, fail the provider release once, and reconstruct the cleanup controller. 4. Confirm that cleanup calls only the release hook and retains the completed run outcome. ## What Changed - Acquire the logical run lease before provider workspace verification and reuse it for preparation and execution. - Preserve the existing session and workspace choices. Recompute workspace-dependent inputs only after a definite refusal; acquire credentials and managed runtime inputs once. - Save the exact refusal and lease before cleanup. Use a workspace-scoped creation key only for the resulting fresh attempt. - Replay interrupted verification on the same unlaunched lease. Keep transport errors retryable and reject replay after execution starts. - Always call the provider release hook after task settlement. Pass refreshed settlement proof and route failed releases through the existing cleanup reaper. - Add database regressions for call order, restart, retained writers, refusal, transport errors, and settlement-only cleanup replay. `plugin-workspaces.ts` grows from 997 to 1,001 lines. The four added lines extend the existing workspace transaction and lease callbacks. Keeping the checks there preserves the ownership assertion, provider result validation, and binding update in one operation. Extracting only the verification branch would split that transaction without removing a separate responsibility. ## Verification - Server TypeScript check passed: `tsc -p server/tsconfig.json --noEmit`. The complete `server/scripts/typecheck-layers.mjs` check also passed, including all test-tree layers. - The first CI typecheck caught a readonly array in a test fixture that the narrower source check did not include. Commit |
||
|
|
61ab829d5d |
fix(queue): another agent's comment no longer hides a pending approval (#21303)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person approves a new plan while the agent is busy, the approval waits in the task's message queue. Its Interrupt control stops the current attempt and starts a fresh one with the approved plan > - The task view shows one queue at a time. A saved card response (an approval) gives way to any later queued message, so it never hides the person's own later messages > - Another agent's comment (here, the CEO's direction with `resume: true`) is also a queued message. It took over the view, and the approval and its Interrupt control disappeared > - The person could then not interrupt for the plan they approved > - This pull request lets only the person's own later messages hide a pending approval. A queue that holds only agent comments no longer hides it > - The benefit is that the Interrupt control for an approved plan stays available until the person uses it or writes a newer message ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.interrupt-with-direction`, the person approved a saved simpler plan while Morgan was busy. The approval queued with `requiresFreshSession`, and its Interrupt control showed. The CEO then posted its own direction comment on the task at 04:03:21. That comment queued as a separate deferred wake. `GET /queued-comments` then returned only the CEO comment's queue. The approval's Interrupt control (`task-chat-queued-interrupt-<approval id>`) never rendered, the click timed out, and no interrupt request was sent. The approval was not lost: its deferred wake was still saved. Only the view and its control were hidden. **Steps to reproduce** 1. Approve a plan on a busy task, so the approval waits for a fresh session. 2. Let another agent comment on the task, so its comment queues after the approval. 3. Read `GET /api/issues/:id/queued-comments`. The approval is gone from the queue. **Expected behavior** The approval stays the visible queue with its Interrupt control. A newer message from the person still takes the view, as before. ## What Changed - `server/src/routes/issues.ts`: `findQueuedCommentWake` returns a later message queue only when it holds a message from a person. A queue with only agent comments yields to a pending saved response. With no saved response it is still returned, as before. - `server/src/__tests__/issue-queued-comments-routes.test.ts`: the eval ordering (a fresh-session approval, then another agent's comment) keeps the approval as the queue, and a later person message in that queue still takes the view. ## Verification - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "another agent's comment queues after it"` — fails on master (the CEO comment's queue is returned) and passes here. - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts` (all tests passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - While a saved response waits, queued agent-only comments are not shown in the composer queue. They stay saved and are delivered by the normal deferred-wake path, as they were before the person approved anything. The person's own messages are never hidden. - One indexed comment lookup per queued message wake during the queue read. - No persisted shape or API contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
99d9e608f4 |
fix(heartbeat): hold the new owner's run while the reassigned owner's run settles (#21301)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person reassigns a task, the old owner's native run is cancelled and the new owner's run is queued > - The old run's executor keeps closing its provider session for several seconds after the run is marked cancelled > - #21031 holds a queued run while the task's previous session close is pending. That hold reads the runner's cleanup gate, which exists only after the close begins > - The stale-lock sweep cleared the task lock as soon as the old run was terminal. The new owner's run was then claimed before the close began, met the close as it escalated, and failed with `native_session_cleanup_quarantined` > - This pull request keeps the new owner's run queued while the old owner's stopped run still settles in this process, and wakes it when that executor finishes > - The benefit is that a reassignment never fails the new owner's run and needs no Retry ## Linked Issues or Issue Description Refs #21031 **What happened?** In the paid chat eval `tasks.edit-properties`, the old owner's run `6cf2ee38` was cancelled for reassignment at 04:03:47.611. Its process stopped only at 04:03:57. The stale-lock sweep cleared the task lock at 04:03:47. The new owner's run was claimed at 04:03:48.626. The runner admission waited for the old close, which escalated to operator-required at 04:03:56, and the new run failed with `native_session_cleanup_quarantined`. Why #21031 did not apply: its hold reads the runner's in-memory cleanup gate. At claim time the old executor had not started its session close, so no gate existed. The existing `predecessor_settling_in_process` hold covers only the same agent, and a reassignment changes the agent. #21260 does not touch this path: its new checkpoint runs only before a turn in a recovered session, and both sessions here were new. **Steps to reproduce** 1. Run a native task for agent A. 2. Reassign it to agent B. A's run is cancelled, but its executor is still settling. 3. The stale-lock sweep clears the task lock. 4. B's queued run is claimed before A's session close begins. **Expected behavior** B's run stays queued until A's executor has finished. It starts by itself after that, then #21031's gate covers any close that is still settling. ## What Changed - `server/src/services/heartbeat/queue.ts`: the `predecessor_settling_in_process` hold in `claimQueuedRun` also holds for another agent's native run on the same task when that run is terminal but its executor is still active in this process. A live run of another agent (for example a reviewer) does not hold it. - `server/src/services/heartbeat/queue.ts`: `notifyOtherAgentsQueuedOnSettledIssue` sends a queued-run dispatch signal when a native executor settles and another agent has a queued run on the same task. It does nothing when no run waits. - `server/src/services/heartbeat.ts`: the executor calls it after its cleanup, next to the existing same-agent `startNextQueuedRunForAgent`. - `server/src/services/heartbeat/queue.test.ts`: the exact ordering (cancelled old-owner run still in the executor set, lock cleared, new owner's run queued), a control for a live run of another agent, and the dispatch wake. ## Verification - `cd server && npx vitest run src/services/heartbeat/queue.test.ts -t "reassigned owner|another agent's live run|next owner once"` — the hold and the wake tests fail on master and pass here. - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts src/services/queued-run-dispatch-liveness.test.ts src/__tests__/heartbeat-workspace-busy.test.ts` (all passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - A queued run on a task waits while another agent's terminal native run on that task is still settling in this process. That wait ends when the executor finishes, and the new wake re-drives dispatch. The queued-run liveness redrive still covers a lost signal. - The wake is event-driven: one indexed query when a native executor settles, and one notification only when another agent waits on that task. It adds no polling. - Previous-release saved state: no persisted shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f82ec65800 |
fix(runs): a person's Stop wins over lease-release teardown (#21300)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can stop an agent run with Stop (cancel). The run must then end as cancelled > - A run that a person stops just as it starts tears down its environment lease at the same moment > - The lease-release teardown wrote `interrupted` (`lease_released_before_terminal`). It recognized only an acknowledged Stop, and it decided from a run snapshot that it read before the Stop request landed > - An interrupted run looks like an infrastructure failure, and it is a recovery candidate. That is against the person's decision > - This pull request makes a recorded person Stop win over the teardown in both orderings > - The benefit is that a person's Stop always ends the run as cancelled ## Linked Issues or Issue Description **What happened?** In the paid chat eval `steering.stop-versus-pause`, the fixture cancelled a resumed run as it reached native running. The Stop request was recorded at 04:02:11.049. The teardown wrote `interrupted` with `lease_released_before_terminal` at 04:02:11.083. The native cancellation intent was still `pending`, so the teardown did not see it as a Stop. The eval needs `cancelled`. **Steps to reproduce** 1. Start a native run. 2. Record a person Stop request on the run (the cancel route does this first). 3. Let the lease-release teardown run before the cancel finalizer writes `cancelled`. 4. The run ends `interrupted`. **Expected behavior** The run ends `cancelled`. This is true when the Stop is recorded before the teardown reads the run, and when it is recorded between that read and the write. ## What Changed - `server/src/services/acknowledged-native-stop.ts`: `hasRecordedUserStopRequest` and `recordedUserStopRequestSql` recognize a person Stop that the cancel route recorded (`cancelledByActorType: "user"`, a user id, and a requested or acknowledged execution cancellation or a startup cancellation fence). - `server/src/services/heartbeat/run-lifecycle.ts`: `terminalizeRunOnLeaseRelease` re-reads the run before it decides. A recorded person Stop now gives `cancelled`. The `interrupted` write is fenced on "no person Stop recorded", so a Stop that lands between the read and the write makes the write fail. The function then reads again and writes `cancelled`. - `server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts`: two orderings (Stop recorded before the teardown read; Stop recorded after the read, with the stale snapshot) and a control (an agent-only request still reports `interrupted`). ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts` — both new ordering tests fail on master (`interrupted`, `lease_released_before_terminal`) and pass here. - `cd server && npx vitest run src/__tests__/heartbeat-run-lease-release-terminalization.test.ts src/__tests__/heartbeat-run-terminalize-before-release.test.ts src/services/heartbeat/run-lifecycle.test.ts` (67 passed). - `cd server && npx tsc --noEmit` passes. - Paid E2E: see the combined-branch run in the comments. ## Risks - Low. Only the lease-release teardown changes, and only when a person Stop is recorded on the run. The cancel path already wrote `cancelled` for the same run when it won the race. - The teardown now makes one more read of the run row. This is one indexed primary-key read per teardown. - Previous-release saved state: no persisted shape changes. Runs saved by the previous release carry the same Stop fields. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bc71c23e0a |
fix(test): remove the Pi fault calibration file race and halve its runtime (#21286)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite checks the Pi provider fault helper. The
helper kills the exact Pi child process on Linux with a pidfd.
> - `tests/runner-e2e/pi-provider-fault.test.ts` failed in 3 unrelated
PRs on 2026-11-12. One failure passed on a rerun of the same commit.
> - The failures had two causes. A fixture file race gave a JSON parse
error. A CPU-heavy calibration ran past the 25 s outer timeout on the
slower shared fleet.
> - This pull request removes the file race and removes about half of
the calibration work. It also states the outer timeout margin.
> - The benefit is a deterministic test that runs in about 3.4 s instead
of about 5.9 s on Linux.
## Linked Issues or Issue Description
**What happened?**
The test "calibrates the exact Pi fault helper with metadata negatives
and native Linux pidfd" failed intermittently in the "Runner support
suite" lane. There were two errors:
- Job 131195539083: `test_real_descriptor_launch_then_exact_child_pidfd
(descriptor=7)` failed at `json.loads(ready.read_text())` with
`JSONDecodeError: Expecting value: line 1 column 1 (char 0)`.
- Jobs 131420813114 and 131420614853: `spawnSync /usr/bin/python3
ETIMEDOUT` after 25 s. Both jobs ran on the `paperclip-public-pr-x64`
fleet (m5n.xlarge, 4 cores, load average about 4). In those jobs, every
file in the lane ran about 1.8x slower than on the hosted `ubuntu-24.04`
image.
**Expected behavior**
The test passes on every run of the same commit.
**Steps to reproduce**
1. Error A: add a delay between file creation and write in the fixture
child (for example `openSync`, then `Atomics.wait` for 500 ms, then
`writeSync`). Run `/usr/bin/python3 -B
tests/runner-e2e/pi-provider-fault.test.py` on Linux. All four real
calibrations fail with the same `JSONDecodeError`.
2. Error B: run the suite on a loaded 4-core runner. The Python suite
needs more than 25 s.
**Paperclip version or commit**
`master` at
|
||
|
|
e65a3095a7 |
fix(runner): quiesce the Core-dialed route once the Runner commits its suspend (#21245)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Core runs agent sessions through the Paperclip Runner. With provider
ingress, Core dials the Runner over a PRP WebSocket.
> - When Core closes a session, it asks the Runner to suspend. The
Runner suspends, reports the result, and exits.
> - Core's ingress connection did not know that this exit was expected.
It treated the drop as a transport loss and redialed the Runner. Each
redial got a 404 until the close released the route.
> - The release (park) path already quiesces the connection before the
Runner goes away. The suspend path did not.
> - This pull request makes the suspend close quiesce the route in the
same way, once the Runner commits its suspend.
> - The benefit is that a clean suspend no longer looks like a lost
Runner. A drop before the suspend is committed still redials and
recovers as before.
## Linked Issues or Issue Description
**What happened?**
After a provider-ingress session closed cleanly, Core redialed the
Runner's ingress endpoint several times and got 404 each time. The
Runner state was `suspended`, `runner.drain` and `runner.suspend` had
both completed, and the Runner had exited with code 0. The redials
followed the outbound reconnect backoff (about 250, 500, 1000 and 2000
ms) and stopped when the registration release closed the outbound.
**Expected behavior**
After Core commits to `runner.suspend` and the Runner reports it
completed, the Runner's exit is expected. Core must not redial or
classify the drop as a transport loss. The release path already works
this way: it calls `outbound.quiesce()` before it releases the task.
**Steps to reproduce**
1. Open a session through a Core-dialed route (plugin provider ingress
or the `listen_ws` driver).
2. Close the session. Core drains, queues `runner.suspend`, and the
Runner suspends and exits.
3. See that the outbound reconnect loop redials the gone Runner until
the registration release closes it.
**Paperclip version or commit**
`master` at
|
||
|
|
62d673cc3f |
fix(retry): start a held Retry once a stopped preparation is proven stopped (#21295)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run fails, a person can press Retry. Admission starts
a new turn from the recorded work only after it proves that the previous
execution stopped.
> - For a legacy runner run that failed while it prepared its sandbox,
the proof reads the run's events and its environment lease.
> - After the automatic retry budget is spent, execution recovery
settles a no-replay hold and appends its own lifecycle event to the run.
The proof accepted only the error and measured events, so this platform
event made the proof fail.
> - The proof also asked for a workspace repair receipt for every run
with a lease, even when the run never recorded a remote workspace and
its sandbox was destroyed. No repair could ever exist, so the proof
could never pass.
> - Admission then refused the Retry without a named reason. The receipt
kept the generic `execution_recovery` wait ("Retry is waiting for
execution checks"), and every later Retry joined the same held receipt.
> - This pull request accepts the recovery hold's own settlement event
when it is bound to a settled hold for the same run, and asks for a
workspace repair only when the run holds prepared files.
> - The benefit is that Retry starts work at once when the previous
preparation is proven stopped, and it continues from the recorded work.
## Linked Issues or Issue Description
**What happened?**
A board user pressed Retry on a blocked task. The Retry was accepted,
but no run started for more than 13 hours. The task said "Retry is
waiting for execution checks. Your message is saved." A second Retry the
next day attached to the same held request and also did nothing. The
execution blocker said `legacy_execution_requires_reconciliation` with
`canRetry: true` and "A person can use Retry on this task ... to
continue once the previous execution has stopped." The previous run was
a legacy `paperclip_runner` run that failed with `setup_failed` while
preparing (`executionRecovery.kind = bootstrap`, `providerWorkStarted =
false`). It had no process identity, no controller lease, and its
Daytona sandbox lease had a `destroyed` termination receipt. Execution
recovery had settled a no-replay hold on it and appended the "Automatic
recovery stopped" lifecycle event.
**Expected behavior**
When the previous execution is proven stopped, a person's Retry starts a
new turn promptly and continues from the recorded work. The task view,
the execution blocker and admission agree that Retry works.
**Steps to reproduce**
1. Let a legacy `paperclip_runner` run fail while it prepares a remote
sandbox (`setup_failed`, bootstrap evidence with `providerWorkStarted =
false`) on its third automatic attempt.
2. Let the sandbox lease be released with a `destroyed` termination
receipt.
3. Let execution recovery settle the
`legacy_execution_requires_reconciliation` hold. It appends the
"Automatic recovery stopped" lifecycle event to the run.
4. Press Retry on the task.
5. Observe that the Retry receipt keeps `executionWait.reason =
execution_recovery` and no run starts. Press Retry again and observe
that it attaches to the same held receipt.
**Paperclip version or commit**
Staging Core `53608bd1d6`. Fixed on top of `origin/master` `c4f639ed1b`.
**Deployment mode**
Hosted cloud (staging), Daytona remote sandbox.
**Agent adapter(s) involved**
`paperclip_runner` (legacy runtime).
Related: #21267 (a Retry held on a resolved decision,
`decision_pending`). This pull request does not change its files.
## What Changed
- `readFailedPreparationSettlement` (`failed-preparation-settlement.ts`)
accepts one trailing host lifecycle event: the no-replay settlement
event that execution recovery appends. The event must have the exact
payload shape and must name a resolved
`legacy_execution_requires_reconciliation` hold whose evidence binds the
same run. Any other event still fails the proof.
- `failedNativePreparation` (`cancelled-native-startup.ts`) asks for a
verified workspace repair only when the run holds prepared files: a
recorded remote workspace, a workspace restore failure, or required
workspace recovery. A run with no recorded remote workspace and a
stopped sandbox has nothing to repair. The lease termination receipt,
the bootstrap evidence and the settled-event proof are still required.
- New regression test `legacy-preparation-retry-hold.test.ts` uses the
exact shape of the staging run.
## Verification
- Red/green: `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts`.
- "starts the person's Retry once the stopped preparation's sandbox is
destroyed" fails on `origin/master` (the receipt keeps
`executionWait.reason = execution_recovery`, no run) and passes with
this change (one queued successor with `previousRunId` set to the failed
run and the Retry request id).
- The Retry stays held when the proof does not hold: a settlement event
bound to another run's hold, a provider execution event, a lease without
a termination receipt, and a recorded remote workspace without a repair
receipt.
- Related suites pass (299 tests): `task-retry`,
`saved-bootstrap-continuation`, `interrupted-preparation-hold-settle`,
`workspace-recovery-preparation`,
`workspace-recovery-preparation-gates`, `heartbeat-session-preparation`.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low. The change only widens two proofs for a run that never started
provider work: the platform's own settlement event is accepted only when
it binds a settled hold for the same run, and the repair receipt is
skipped only when the run recorded no files to repair.
- Retry still starts a new turn. It never replays the stopped run.
- Not in scope: two other tasks show the same generic wait with a
different cause on the saved-work side. Their blocker says `canRetry:
false` while the task view offers Retry. One has saved-work recovery
that failed with `source_unavailable`. The other shows recovered work
without a verified repair receipt. They need a separate fix.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, browser for read-only staging API
reads).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
db3d7223e0 |
fix(verify): report stale SDK public input pins locally, before CI (#21283)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite proves the plugin SDK public types against
pinned source. The pins are in
`tests/runner-e2e/source-sdk-public-inputs.json`.
> - The pins cover every file in `packages/shared/src` and
`packages/plugins/sdk/src`, and the SDK build inputs (for example
`pnpm-lock.yaml`).
> - A PR that edits a pinned file and does not refresh the registry
fails three files in the "Runner support suite" lane.
> - Between 2026-11-12 10:27Z and 2026-11-13 01:14Z, this failure
occurred in 14 CI runs on 12 PRs.
> - The local check, `pnpm verify`, returned exit 2 ("let CI cover it")
for these changes. It did not mention the pins. Authors pushed, and CI
failed.
> - This pull request makes `pnpm verify` find stale pins before the
push. It also makes the opaque fixture error name the file and the fix.
> - The benefit is that authors find and fix the problem in seconds,
locally, instead of after a 6-minute CI lane.
## Linked Issues or Issue Description
**What happened?**
Three files in the "Runner support suite" lane failed together on many
unrelated PRs: `source-sdk-evidence.test.ts`,
`source-sdk-public-inputs.test.mjs` and
`source-sdk-public-type.test.ts`. Two of the three failed with the
opaque error `Reviewed SDK reference source changed`.
**Root cause**
The pins on master are not stale. The tests pass on master (58/58). Each
failing PR edited a pinned input and did not refresh the registry. I
compared the PR heads with their merge base:
- All 14 failing heads (12 PRs) changed a pinned input. None of them
changed `source-sdk-public-inputs.json`.
- All 19 heads in the same window that changed a pinned input and passed
this lane also refreshed the registry. This includes later heads of
three PRs that failed first (#21119, #21121, #21156).
- Two of the 12 PRs are Dependabot PRs that change only `pnpm-lock.yaml`
(#19593, #19594). The lockfile refresh bot already resyncs the pins for
lock changes on master.
**Steps to reproduce**
1. Start from `origin/master`.
2. Add one line to `packages/shared/src/types/heartbeat.ts` and commit
it.
3. Run `node node_modules/vitest/vitest.mjs run --config
tests/runner-e2e/vitest.config.ts source-sdk-evidence.test.ts
source-sdk-public-inputs.test.mjs source-sdk-public-type.test.ts`.
4. The result is the same as CI: 3 files fail, with the same 4 failures.
5. Run `pnpm verify --explain`. It returns exit 2 and does not mention
the pins.
**Expected behavior**
`pnpm verify` reports the stale pins and the refresh command before the
author pushes.
## What Changed
- `tests/runner-e2e/source-sdk-public-inputs.mjs`: add
`staleSdkPublicInputsInWorkingTree(root)`. It applies the existing drift
guard to the working tree (tracked and untracked files that git does not
ignore). Add `isSdkPublicInput(file)` and the shared failure message
`staleSdkPublicInputsMessage(stale)`. The committed-tree guard keeps its
behavior and now uses a shared comparison.
- `scripts/local-verify.mjs`: when the diff touches a pinned input, the
registry or `report-lock.json`, `pnpm verify` checks the pins first. If
they are stale, it exits 1 and prints the files and the refresh command.
The check reads files only. It does not write the registry.
- `tests/runner-e2e/source-sdk-reviewed-fixture.ts`: the error `Reviewed
SDK reference source changed` now names the file and gives the refresh
command.
- `tests/runner-e2e/source-sdk-public-inputs.test.mjs`: new test for the
working-tree guard (uncommitted edit, untracked file, ignored file). The
CI drift test uses the shared message.
- `doc/DEVELOPING.md`: document the new check under "Incremental local
verification".
No assertion is weakened. No pin is changed. No file under `.github/` is
changed.
## Verification
- Master, without this change: the three files pass (58/58).
- Red: on master plus a one-line edit to
`packages/shared/src/types/heartbeat.ts`, the three files fail with the
CI signature. `pnpm verify --explain` returns exit 2 and does not
mention the pins.
- With this change on the same red commit: `pnpm verify` exits 1 in
about 15 s and prints `tests/runner-e2e/source-sdk-public-inputs.json is
stale for 1 pinned input(s): packages/shared/src/types/heartbeat.ts
(changed). Commit your change, run ... --write, and commit the updated
registry.`
- Green: I followed the printed command (`node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`, then
commit). The three files then pass (59/59, with the new test).
- `node --test scripts/__tests__/local-verify.test.mjs`: 14/14 pass.
- `pnpm verify` on this branch returns exit 2 because
`scripts/local-verify.mjs` is a global CI input. Required CI supplies
the broad coverage.
## Risks
- Low risk. The new check only reads files. It runs only when the diff
touches a pinned input.
- `pnpm verify` now fails (exit 1) for a change that also fails in CI.
Before, it returned exit 2.
- The check needs a working `git ls-files` in the repository. `pnpm
verify` already needs git.
- Dependabot lock PRs still fail this lane until the lockfile bot
refreshes the pins. This PR does not change that path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic), Claude Code agent with tool use (shell, git,
GitHub CLI). Commits are attributed to Claude Fable 5.2.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
0c63f0fdb3 |
test(skills): budget list read bytes for fixture rows, not the bundled catalog (#21285)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The skill library list reconciles the bundled skill catalog, folders, and local sources on each read > - #18023 added a test that limits the `company_skills` bytes one warm `list()` call reads, so the folder pass and the list query keep their narrow projections > - The limit counts every `company_skills` read, so it also counts the bundled catalog rows that every company gets > - Each new bundled skill adds about 9.6 KB to that total, so pull requests that only add skills under `skills/` fail a test about projections > - This pull request measures a catalog-only company next to the fixture company and applies the same limit to the difference > - The benefit is that the test fails only when the fixture rows are read too widely, not when the catalog grows ## Linked Issues or Issue Description Refs #18023 (added the test). Refs #20760, #20764, #20767, #20768, #20769 (failed on this test). **What happened?** `companySkillService.list > keeps folder classification and stored content while narrowing inventory refresh reads` failed on five open pull requests that only add bundled skills under `skills/`. The error was `expected 1163219 to be less than 950000`. Other runs had 983607, 1035956, 982321, and 985561. The test passes on master. **Expected behavior** The test must fail when `list()` reads the unused metadata or the Markdown of company skill rows. It must not fail when the bundled catalog gets larger. **Steps to reproduce** 1. Check out master. 2. Run `git checkout <head of #20760> -- skills` to add its 34 bundled skills. 3. Run `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts -t "narrowing inventory refresh"`. 4. The test fails with `expected 1163504 to be less than 950000`. **Paperclip version or commit** `9b613562b70b7bea9e7562ddc1087d4c1b1788b1` (master). ## What Changed - The test creates a second company that has only the bundled catalog. - The test lists both companies warm and captures the `company_skills` response-value bytes of each call. - The test applies the existing 950,000-byte limit to the difference. The difference is about 663 KB and comes only from the ten fixture rows. - The test also checks that the fixture company lists the catalog rows plus the ten fixture rows. - No service code changes. ## Verification Where the bytes come from (master, one warm `list()` call): | Read | Master catalog | Catalog of #20760 | | --- | --- | --- | | Main list query (`company-skills.ts:3324`) | 671,003 B, 20 rows | 691,694 B, 52 rows | | Bundled reconciliation `getByKey` reads (one per bundled skill) | 165,423 B, 13 calls | 471,771 B, 45 calls | | Total (old assertion) | 836,465 B, pass | 1,163,504 B, fail | | Fixture company minus catalog-only company (new assertion) | 663,153 B, pass | 663,166 B, pass | Red and green: - Old test, master `skills/` plus the `skills/` tree of #20760: fails, `expected 1163504 to be less than 950000`. - New test, same enlarged catalog: passes, difference 663,166. - New test, master: passes, difference 663,153. The new test still catches the regression it exists for. I widened each projection on purpose with the enlarged catalog, ran the test, and reverted: - Folder pass reads the full `metadata` column instead of `sourceKind` only: fails, `expected 1323122 to be less than 950000`. - Main list query also reads `markdown`: fails, `expected 1183162 to be less than 950000`. Commands: - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/company-skills-service.test.ts`: 81 passed. - `pnpm verify`: not completed. It selected 2,615 of 3,435 test files (conservative selection), and I stopped it to keep load low on a shared machine. CI ran the full required checks on the final commit: 54 passed, 4 skipped. ## Risks - Low risk. The change is in one test only. - The new test runs one more warm `list()` call, about 0.5 s more. - The bound is unchanged. A widened projection on the fixture rows still adds at least 540 KB and fails. - This test no longer limits the bytes the bundled catalog adds to each list call. Each bundled skill adds about 9.6 KB per warm call, because `ensureBundledSkills` calls `getByKey` with the full row for each bundled skill. That cost is real but separate. A fix should narrow that read, not let this test limit the catalog size. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: test-only change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (Superagent Security Scan is neutral: "Scan inconclusive") - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c6c7a61e2c |
fix(retry): release a Retry held on a resolved decision (#21267)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails, the task shows a Retry button. Admission can hold an accepted Retry while a person must answer a pending approval or question (`decision_pending`). > - The hold is saved on the deferred Retry receipt. Nothing cleared that saved reason when the person answered, so the task kept saying "Retry is waiting for a pending approval or question" when nothing was left to answer. > - The deferred-receipt worker drops a refused receipt from its schedule. No producer hinted it when the decision resolved, so the held Retry did not start until some unrelated event rescanned the company. > - A Retry was also held by a confirmation that was already open when the failed turn was admitted. That turn had already run past the confirmation, so the Retry added a gate that the original turn did not have. > - A person's comment on the task started a continuation, but the held Retry receipt stayed deferred next to it. > - This pull request makes a resolved decision release the held Retry at once, keeps the projection from showing a resolved decision as the wait, stops holding a Retry on a decision that predates the failed turn, and settles a held Retry when a comment or answer continues the same failed run. > - The benefit is that Retry always works: a person never waits on a decision that is already resolved. ## Linked Issues or Issue Description **What happened?** A board user pressed Retry on blocked tasks. No run started. The task said "Retry is waiting for a pending approval or question to be resolved. Your message is saved." In one task, the only interaction was a `request_confirmation` that the previous successful run created. The user's later comment started the turn that failed. The confirmation stayed pending, so every Retry attached to the same held receipt. After the user rejected the confirmation and sent a new comment, the task still showed the same wait. No run started until a server restart rescanned the deferred receipts about three minutes later. The task projection said `canRetry: true` with the `decision_pending` wait. **Expected behavior** An accepted Retry is never held on a decision that is already resolved. Accepting or rejecting the decision releases a held Retry right away. A person's comment on the blocked task starts work, and the held Retry is settled. The task projection never shows a resolved decision as the Retry wait. **Steps to reproduce** 1. Let an agent run on a remote runner create a `request_confirmation` for the task. 2. Let the runner be lost so the run fails with `native_continuation_requires_reconciliation`, and let saved work recovery complete. 3. Press Retry. The Retry receipt is saved with `executionWait.reason = decision_pending`. 4. Reject or accept the confirmation. 5. Observe that the receipt keeps `decision_pending`, the task keeps showing the wait, and the Retry does not start until an unrelated rescan. **Paperclip version or commit** Staging Core `57aeda190b`. Fixed on top of `origin/master` `53608bd1d6`. **Deployment mode** Hosted cloud (staging), remote sandbox runner. **Agent adapter(s) involved** `paperclip_runner` (native runtime). ## What Changed - `hasPendingContinuationDecision` (`explicit-native-continuation.ts`) is now the single check for a decision that holds a person's continuation. Admission and the task projection both use it. - For a Retry, a pending interaction that was created before the failed source run was created does not hold the Retry. That turn was already admitted while the decision was open. Decisions that the failed run created (for example a plan review) still hold the Retry. - `releaseDecisionHeldExecution` (`deferred-execution-wait.ts`) removes the saved `decision_pending` wait from deferred receipts on the task and hints each receipt to the deferred-receipt worker. It runs in the resolution transaction of every interaction resolution path (through `notifyChatConfirmationResolution`) and in question answers. - The task projection reads the live decision state. A saved `decision_pending` wait whose decision is resolved is not shown. This also repairs receipts saved before this change. - When a comment or an interaction answer continues a failed run, `retireTaskRetriesContinuedBy` (`task-retry.ts`) settles held Retry receipts for the same source run as `task_retry_superseded`. ## Verification - Red/green: `cd server && npx vitest run src/services/governed-wait-runner-loss.test.ts`. The five new tests fail on `origin/master` sources and pass with this change (17/17 pass): - releases the held Retry as soon as the person chooses accept - releases the held Retry as soon as the person chooses reject - never shows a resolved confirmation as the Retry's wait, and both projections offer the same Retry - a person's comment starts work and settles the held Retry once nothing is left to answer - starts a Retry at once when the confirmation was already open before the failed turn started - The existing tests that keep a Retry or a message on hold while a plan review, governed tool action, secret proposal, or connection authorization is pending still pass. - Related suites: deferred execution wait, explicit native continuation, interaction resolution and service, task interruption presentation and query, task retry, deferred receipt work, interaction routes. ## Risks - Medium-low. A Retry now starts when the only pending interaction is older than the failed run. That turn already ran with the interaction open, so the Retry gets no new authority. Governed tool actions still need their own approval before the tool runs. - Interaction resolution now also updates deferred receipts on the same task. The update is limited to receipts with a saved `decision_pending` wait. A receipt that is still blocked by another pending decision is held again on its next admission. - Issue-linked approvals (`approvals` table) do not yet hint the held receipt when they resolve. The projection no longer shows them as the wait after they resolve. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0b9a9593d3 |
fix(tool-gateway): advertise only the MCP capabilities the gateway token may list (#19279)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A named MCP gateway gives a client its assigned tools. Each gateway token has a list of allowed actions > - A native run token allows only `tools/list` and `tools/call` > - `initialize` always advertises the `tools`, `resources` and `prompts` capabilities, whatever the token allows > - A client that follows the advertised capabilities calls `resources/list` and `prompts/list`, and the gateway refuses both with 403 `gateway_token_action_denied` > - This pull request makes `initialize` advertise only the capabilities whose list action the token allows > - The benefit is that a client is not told to call methods that its token cannot use ## Linked Issues or Issue Description This continues #17688, which made the same change and was closed unmerged by its author on 2026-11-02 ("we are no longer contributing upstream"). That pull request reports that Claude Code requests every advertised capability at once, drops the HTTP connection after the first 403, and then has no gateway tools for the run. I did not reproduce the Claude Code part. The gateway part is below. I searched pull requests for "allowedActions tools/list", "gateway token allowed actions context tools", "paperclip_list_resources", "gatewayTokenAllowedActions", "gateway_token_action_denied", "MCP gateway capabilities resources prompts", "initialize capabilities token" and "named gateway token actions discovery", and issues for "paperclip_list_resources" and "gateway token allowedActions". Results: - #17688 (closed, not merged): this change. - #14427 (open): filters the four `paperclip_*` context tools in `tools/list` by the token's allowed actions. It does not change `initialize`, so the two pull requests do not overlap. They touch different handlers in server/src/routes/tool-gateway.ts. No issue matched. **What happened?** On master, the `initialize` handler in server/src/routes/tool-gateway.ts returns `capabilities: { tools: {}, resources: {}, prompts: {} }` for every token. For a token that allows only `tools/list` and `tools/call`, a `resources/list` call returns 403 with reason code `gateway_token_action_denied`. **Expected behavior** `initialize` advertises `tools`, `resources` or `prompts` only when the token allows `tools/list`, `resources/list` or `prompts/list`. A session with no token action list keeps all three. **Steps to reproduce** 1. Create a named gateway and a token with `allowedActions: ["tools/list", "tools/call"]`. 2. Send `initialize` with the token. On master the result advertises `resources` and `prompts`. 3. Send `resources/list` with the same token. The gateway returns 403 `gateway_token_action_denied`. **Paperclip version or commit** master at |
||
|
|
8a0f95849d |
perf(native-runtime): start the first turn without re-copying launch files, and never fail a run on a slow copy (#21287)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run
## Linked Issues or Issue Description
Refs #19622
**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.
**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.
**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.
**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).
## What Changed
- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.
## Verification
Local (this branch, on master
|
||
|
|
5d298b1b48 |
test(server): judge the Linux real-process boundary cases only by the processes they start (#21284)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In `local_trusted` mode, the operator's local processes get implicit board access. The local board boundary refuses that access to processes of an agent run > - #21209 added a Linux rule for a connection whose owner the server cannot inspect. The server keeps board access only if every same-user process it cannot inspect belongs to the operator > - The real-process test "keeps board access for an operator process the server cannot inspect" used that rule on the whole CI host. Other processes of the same user on the host can be uninspectable and not the operator's > - Since #21209 merged, this test failed in about one in four `PR` runs, on PRs that did not touch it. 22 failed server-test jobs between 2026-11-12 23:05Z and 2026-11-13 03:06Z > - This pull request makes the real-process cases judge only the processes that each case starts. It adds a case that pins the host-wide rule with a known uninspectable agent-run process > - The benefit is that the test is stable on shared CI hosts, and the product rule stays tested ## Linked Issues or Issue Description Refs #21209 **What happened?** `server/src/__tests__/local-agent-process-boundary.test.ts` failed in the "General tests (server)" lane of unrelated PRs. Both cases "keeps board access for an operator process the server cannot inspect" (127.0.0.1 and ::1) received `{ type: "none", source: "none" }`. They expected `{ type: "board", source: "local_implicit" }`. The failures started with the first runs that included #21209 (merged 22:52Z). They continued until at least 03:06Z. Two runner image versions failed in the same way. **Expected behavior** The test passes on a CI host that runs other processes. **Actual behavior** The boundary looks at every same-user process on the host that it cannot inspect. When the owner of a connection is hidden, one uninspectable process that is not clearly the operator's makes the boundary refuse. Examples are a process of another agent run, a process whose parent is PID 1, or a process that exits before the server reads its parent chain. The test did not control which processes were on the host. **Steps to reproduce** On Linux, as a non-root user, run `python3 -c 'import ctypes,time; ctypes.CDLL(None).prctl(4,0,0,0,0); time.sleep(60)' &`. Then make the operator request from the test. The boundary refuses it. ## What Changed - The Linux real-process cases use an inspector that wraps the real Linux inspector. For a hidden owner, it keeps only the candidate processes that descend from the process that the case started. All other parts are the real Linux inspector: the TCP table reads, the fd scan, the recheck, and the ancestry reads. - Each Linux block starts an idle, uninspectable process with `PAPERCLIP_RUN_ID`. This process stands for another agent run on the host. - A new case, "refuses board access to an operator process the server cannot inspect while another agent run's process it cannot inspect is running", uses the unwrapped inspector. It pins the host-wide rule in a real-process test. - If the test runner can inspect every process (for example, as root or with `CAP_SYS_PTRACE`), the idle process is not uninspectable. The host-wide case then skips. The other cases run as before. - No product code changed. ## Verification - Red, Linux (Docker `node:24-bookworm`, non-root user): a script runs the product module `server/src/middleware/local-agent-process-boundary.ts` from master with the same operator request as the test. With no other process, the operator gets board access 10/10. With one idle uninspectable agent-run process, it gets 0/10. With one uninspectable process whose parent is PID 1, it gets 0/10. With short-lived uninspectable processes, it gets 8/10. Agent-run requests are refused 5/5 in all cases. - Red, CI Linux: in this PR's run, the new host-wide case passes. It shows that the earlier assertion fails when one uninspectable agent-run process is on the host. - Green, CI Linux: in the same run, with that process on the host, both "keeps board access for an operator process the server cannot inspect" cases pass. Result: 27 tests passed in the shard that ran the file. - macOS: `pnpm --dir server exec vitest run src/__tests__/local-agent-process-boundary.test.ts` gives 15 passed, 12 skipped (Linux-only). - Not verified: the specific process that was on the CI host in each failed run. One fresh CI run with a temporary diagnostic found no other uninspectable process. That run passed. The diagnostic is not in the final commit. ## Risks - Low risk. The change is test-only. - Product note for a maintainer, not changed here: on Linux, an operator process that the server cannot inspect loses implicit board access when any other uninspectable same-user process on the machine is not clearly the operator's. A process of another agent run is an example. This is the rule from #21209, and the new case pins it. If this is too strict for operators, that needs a product decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Docker on Linux, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: test-only) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
0c04ff76eb |
fix(native-runtime): a follow-up that arrives as a warm session closes continues the task (#21279)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent session on a remote sandbox stays warm after a run, and an idle timeout closes it after about 5 minutes > - When a person's follow-up comment arrived just as that idle close began, the new run failed with `native_session_supervisor_busy` > - The idle close of the finished owner could also fail its final checkpoint with `native_runner_checkpoint_owner_lost`, because the task already pointed at the queued follow-up run > - Then the retry was held for workspace repair and the task stayed blocked; the environment could not be deleted (409) because its warm runner could not be stopped > - This pull request makes the follow-up wait for the idle close that is in flight, and lets the finished owner save its final checkpoint while its successor still waits in the queue > - The benefit is that a follow-up that lands at the warm session's idle close continues the task with no person involved ## Linked Issues or Issue Description Refs #18011 (the checkpoint ownership fence this change extends). **What happened?** A follow-up comment arrived 295–340 s after the previous run ended, while the warm session's idle close was in progress. The new run failed with `native_session_supervisor_busy`. In another case, the idle close failed its final checkpoint with `native_runner_checkpoint_owner_lost`, the next run failed `native_instruction_materialization_handoff_unavailable`, and the retry was cancelled with `execution_reconciliation_required`. The task stayed blocked. A hosted instance showed the same class as "previous run's session on this sandbox could not be retired". **Expected behavior** The follow-up run waits for the idle close (inside the existing bounded release wait) and then starts. The finished owner saves its final checkpoint and closes. The task completes hands-off. **Steps to reproduce** 1. Run an opencode agent on a Daytona sandbox with a warm session (idle timeout 300 s). 2. Let a run finish. 3. Post a comment on the task 298–306 s after the run ended. **Paperclip version or commit** master `444dc18d98`. **Deployment mode** Local server with Daytona remote sandboxes (restart-chaos harness, light lane). The same class was seen on a hosted staging instance. ## What Changed - `claimWarmNativeInstructionCopy` now also waits when an idle close is in flight for the session scope (`closingWarmNativeSessions`). The wait uses the same deadline as the existing release wait (`warmSessionReleaseWait`), so it adds no new wait and nothing before `turn.start` on the normal path. - `validateNativeRunnerCheckpointOwner` accepts the finished owner's final checkpoint (accepted result, no new work) when `issues.executionRunId` points at a successor run that is `queued` or `scheduled_retry` for the same company and task and has no `native_run_finalizations` row. - Tests: a server test for a follow-up that arrives while the idle timeout closes the warm session; four new scenarios (`queued_successor`, `scheduled_retry_successor` × `runner.suspend`/`turn.stop`) and four new negatives (`queued_wrong_issue`, `queued_new_work` × both commands) in the early-retirement checkpoint matrix. - Restart-chaos harness: `RESTART_CHAOS_MIX_FOLLOW_UP_AFTER_END_MS=<min>-<max>` and `RESTART_CHAOS_MIX_FOLLOW_UP_COUNT` post the staging-mix follow-up a set delay after the task's last run ended, so a cell can aim the follow-up at the warm idle close. The restart-chaos README describes both variables. A failed post is retried with the same `clientRequestId`, the grade check `after_end_follow_ups_served` needs an agent reply after each post and the ledger marker, and the task budget includes the planned waits. - Harness: the wave database cleanup (`tests/runner-e2e/controller-database.ts`) no longer waits forever when a group signal (for example `timeout` or Ctrl-C) already stopped the embedded Postgres. Before, `embedded-postgres` `stop()` waited for an exit event that had already fired, and an interrupted wave hung after the Daytona R6 sweep. ## Verification - Fix 1, red without the product hunk: `native-session-executor.test.ts -t "idle timeout is closing"` fails with `promise rejected "Error: native_session_supervisor_busy"`. Green with the fix: 1 passed. - Fix 2, red without the product hunk: `native-runner-restart-recovery.integration.test.ts` fails 4 of 208 (`queued_successor` and `scheduled_retry_successor` × `runner.suspend`/`turn.stop`) with `native_runner_checkpoint_owner_lost`. Green with the fix: 208 passed, 14 skipped. The two new negative cases pass in both states. - Harness hang fix, red without the fix: the new `controller-database.test.ts` case ("finishes cleanup when the database already stopped because a group signal reached it") times out on cleanup (`expected 'hung' to be 'cleaned'`). Green with the fix: 5 passed, 1 skipped. - After the rebase onto master `7154030ef6`, both server test files together: 1639 passed, 21 skipped. - Restart-chaos harness tests (`staging-mix`, `staging-mix-retry`, `observer`): 75 passed, including a test for the after-end retry, the `after_end_follow_ups_served` grade check and the added budget. `tsc --noEmit` passes for `server`, `tests/runner-e2e` and `tests/restart-chaos`. - `pnpm verify` exits 2 because `tests/restart-chaos/observer.ts` is a global CI input. Required CI supplies the broad coverage. - Daytona cell on this branch (4 opencode tasks, warm sandboxes, 1 vCPU, follow-up comment posted 300.8–308.2 s after each task's last run ended, image built for master `444dc18d98`): 4/4 tasks done hands-off, 8/8 runs succeeded, 0 Retry banners, 0 `owner_lost` / `supervisor_busy` / "could not be retired" / `execution_reconciliation_required` lines in the server log, environment delete clean, warm-shutdown grade pass, 0 sandboxes left. The harness exited by itself after 1076 s. - Same cell before the fix (master, 8 tasks): 1/8 tasks blocked (`native_session_supervisor_busy` → `native_runner_checkpoint_owner_lost` → `execution_reconciliation_required`). With fix 1 only: 1/8 still blocked (`owner_lost` on the idle close). With both fixes: 8/8 done and 0 failed runs (cut by the 20-minute cap before cleanup), then 4/4 above. - The cell grade `T:completion_once` fails on all 4 tasks by design: the timed follow-up comment arrives after the task is done, so the agent posts a second completion comment. ## Risks - Fix 1 can delay a follow-up by up to the existing warm release wait (30 s) while an idle close finishes. Before this change, that run failed. If the close fails or the wait expires, the claim returns null and the run uses a fresh instruction copy, as before. - Fix 2 widens the checkpoint ownership fence from #18011. It does not reopen what #18011 closed: - The exemption applies only to the finished owner's final save: the checkpoint must hold an accepted result and no new work, so no provider turn can be granted through it. - The successor must still be unclaimed (`queued` or `scheduled_retry`) and must have no native coordinator (`native_run_finalizations`) row. No controller has started it, so nothing can have consumed or depended on the predecessor's checkpoint. Once the successor is claimed, the existing preparing/claimed rules from #18011 apply unchanged. - The successor must belong to the same company and the same task. - Cases that must still reject, with tests in the same matrix: - `queued_wrong_issue`: a queued successor for a different task is rejected with `native_runner_checkpoint_owner_lost` and the stored checkpoint does not change. - `queued_new_work`: a `turn.start` (new work) while the successor is queued is rejected with `native_runner_checkpoint_owner_lost` and the stored checkpoint does not change. - All earlier rejections stay: unclaimed native coordinator, dispatching, expired or foreign preparation, foreign native preparation, wrong task, new work, unaccepted result, replaced controller. - Not changed: an owner whose close fails still blocks environment delete (409). That is a separate product decision. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, test runs). Earlier commits on this branch were written by the same model in an earlier session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c4f639ed1b |
fix(ci): re-run cancelled runs that lost a runner, and let runner-loss evidence win (#21291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - PR CI runs about 50 jobs per run on AWS spot instances. AWS can reclaim a machine mid-job, and the job dies. > - `ci-runner-loss-rerun.yml` re-runs such a run once, automatically, when every failed job lost its runner. > - In a 17-hour sample, 42 runs lost a runner and were still the PR's current head. The automation rescued only 14. > - Two gaps caused most of the misses. A spot reclaim on RunsOn v3.2.3 usually ends the job, and the run, as `cancelled`; the workflow and script only look at `failure`. Also, the Runner support suite always carries `gateway_indexing_incomplete` error annotations from a passing test, so a lost runner there reads as a real failure. > - On-demand fallback for the fleet was turned off on 2026-11-13, so reclaims will happen more often. About 20% of runs are expected to lose at least one job. > - This pull request makes the re-run act on cancelled runs and jobs, but only with positive runner-loss evidence, and lets that evidence win over other error annotations. > - The benefit is fewer manual "Re-run failed jobs" clicks: an estimated 34 of 42 instead of 14 of 42 in the sample. ## Linked Issues or Issue Description No public issue exists. **What happened?** Run [43285064435](https://github.com/paperclipai/paperclip/actions/runs/43285064435) lost the runner of `ci / General tests (server (8/12))`. The job concluded `cancelled` (its log has `##[error]The runner has received a shutdown signal`). The run concluded `cancelled`, so the re-run workflow's `if:` skipped it. Run [43325855069](https://github.com/paperclipai/paperclip/actions/runs/43325855069) lost the runners of `ci / Runner support suite` and `ci / General tests (server (2/12))`. Both logs have the shutdown signal. The re-run bot ([decision log](https://github.com/paperclipai/paperclip/actions/runs/43327873358)) said "1 of 1 failed jobs did not lose their runner", with evidence `gateway_indexing_incomplete`. It ignored the cancelled job. It read the annotation as a real failure. **Expected behavior** Both runs get one automatic re-run of their failed jobs. **Steps to reproduce** The new tests replay trimmed snapshots of both runs (`.github/scripts/tests/fixtures/runner-loss-rerun/run-43285064435.json` and `run-43325855069.json`) through the script. In a 17-hour sample on 2026-11-12, 42 runs lost a runner and were still the PR's current head. Only 14 were re-run automatically. 16 stayed red because the run concluded `cancelled`. 4 stayed red because an error annotation in the lost job was read as a test failure. **Additional context** The `gateway_indexing_incomplete` annotation does not come from the shutdown. `scheduled-gateway-indexing.test.ts` prints `::error` lines while it passes, 84 seconds before the shutdown signal in this run. Attempt 2 of the same run passed with the same two failure annotations. So, before this change, a lost runner in the Runner support suite could never be re-run. ## What Changed - `ci-runner-loss-rerun.yml`: the job also runs when the PR run concluded `cancelled`. The trigger type, `permissions`, the repository, path, event and attempt-1 conditions do not change. - `runner-loss-rerun.mjs`, `eligibleRun`: accepts `completed/cancelled` as well as `completed/failure`. - `runner-loss-rerun.mjs`, `decide()`: classifies jobs that concluded `cancelled` as well as `failure`. It still skips the `ci / verify` and `ci / e2e` aggregates, and a failed or cancelled routing gate still blocks the re-run. - `decide()` now runs `staleReason` (open PR, current head, no newer run) before it reads any annotation or log, and again just before the re-run, as on master. A run cancelled by a new push has about 40 cancelled jobs; it now costs two API listings and no log reads. The final check closes the window where a push lands during the log reads: a re-run of the old run would join the PR's `cancel-in-progress` concurrency group (`pr-<number>` in `pr-trusted.yml`) and cancel the new commit's CI. - Following CONTRIBUTING.md ("Do not submit a PR that includes code comments"), this PR adds no new code comments. The reasons for each rule are in this description and in `doc/DEVELOPING.md`. - `classifyJob`: runner-loss evidence wins. A lost-communication annotation means lost. A cancellation annotation plus a shutdown signal in the log means lost, even if the job also has other error annotations. A cancelled job without this evidence (for example, a person cancelled the run) counts as a real failure and blocks the re-run. - Fixtures: two new trimmed snapshots from the example runs (jobs with `id`, `name`, `status`, `conclusion`; failure annotations; `##[error]` log lines only). The two older fixtures get the real annotations and log error lines of their cancelled jobs. Those jobs were spot-reclaimed too (each log has the shutdown signal), but the old code ignored them. - `doc/DEVELOPING.md`: the runner-loss paragraph describes the new rules. ## Verification - Red first. Against the unchanged script, the new tests fail for the reasons above: 43285064435 gives `run is completed/cancelled, not completed/failure`; 43325855069 gives `1 of 1 failed jobs did not lose their runner` with evidence `gateway_indexing_incomplete`, the same decision the bot logged. - Green: `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 25 of 25 pass. - Push during log reads: the PR head moves after the first log read. With the final freshness check removed, this test fails (the script re-runs); with it, there is no re-run and the reason is `no open pull request still has this run's head commit`. - Guard cases, all with no re-run and no POST: - Superseded run: 52 jobs cancelled, the PR head moved. The test asserts zero annotation and log reads. - Person cancelled the run: cancelled jobs with only "The operation was canceled." in the annotation and log, and some never-started jobs with no annotation. - Mixed: one cancelled lost job and one real test failure (`1 of 2 failed jobs did not lose their runner`). - Once only: a cancelled run on attempt 2 is refused after one API read. The workflow condition test still requires `run_attempt == 1`. - Cancelled routing gate: left for a person. - Policy job command: `node --test '.github/scripts/tests/*.test.mjs'`: 806 of 806 pass. - `pnpm verify`: exit 2 ("Global CI input changed: .github/scripts/runner-loss-rerun.mjs"). It selects no local Vitest files and asks for required CI coverage. I did not run `pnpm verify --full`; required CI on this PR supplies that coverage. - This workflow runs on `workflow_run` from the default branch. This PR's own CI cannot exercise the new behavior. The fixture tests are the evidence. I did not dry-run the script against live runs. ## Risks - A test that really fails, and then a spot reclaim of the same job a minute later, now gets one wasted re-run. Attempt 2 then fails for real. It cannot loop: the workflow `if:` and `eligibleRun` both accept attempt 1 only. - A run a person cancels on the current head now starts the re-run job. The job reads the annotations, and the log of each job with a cancellation annotation, and then does not re-run. This costs some API calls on a GitHub-hosted runner. - A concurrency cancel by a label event that does not request CI (the newer run is `skipped`) passes the head check. Its cancelled jobs have no runner-loss evidence, so it is not re-run. - Trust: no change to `permissions`, the trigger, the accepted repository, event, workflow path, attempt, or the routing-gate rule. A cancelled run reuses its attempt-1 gate output, the same as a failed run does today. - Estimated effect, from a 17-hour sample: 14 of 42 rescued before; 16 cancelled runs and 4 runs with error-annotation noise become eligible (about 34 of 42). The other 8 stay skipped on purpose (a real test failure in the same run, a person re-ran first, or the PR closed). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent. Model ID claude-opus-5-5, with tool use (shell, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
af1c685f5f |
fix(secrets): return 422 for secret proposals missing a required field (#15348)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can propose a company secret or a secret binding through the API. > - The proposal route reads the request body and passes it to the service. > - The service trimmed required string fields before it checked them. > - A request that omitted name or justification threw a TypeError. > - The route then returned 500 instead of a validation error. > - This pull request checks the required fields first and returns 422. > - The benefit is that a client can tell a bad request from a server fault. ## Linked Issues or Issue Description Closes #15334 ## What Changed - Check `name` and `justification` in `createSecret` before `trim`, and return 422 when either is missing. - Check `justification` in `createBinding` the same way. - Keep the existing empty-value and malformed-value checks and the optional `key` field unchanged. - Add a regression test that covers the missing-field cases for both proposal kinds and keeps the complete-body case at 201. No documentation change is needed. The OpenAPI document already lists 422 for this route, so the server now matches the documented contract. ## Verification - Run the route suite. It passes. `./node_modules/.bin/vitest run server/src/__tests__/secret-proposals-routes.test.ts` gives 26 passed. - Revert only the service change and run the new test. It fails with `expected 500 to be 422`, which proves the test catches the bug. Restore the change and it passes again. - Run the server typecheck. It passes. `pnpm --filter @paperclipai/server run typecheck` exits 0 with no errors. - A `POST /api/agents/me/secret-proposals` that omits `justification` returns 422 after the fix and returned 500 before it. A complete body still returns 201. ## Risks - Low risk. The change adds two field checks before an existing `trim` call. Valid requests still return 201. A missing required field now returns 422, which the route already documents. ## Model Used Claude Opus 4.8 (`claude-opus-4-8`), with extended thinking and tool use. I chose the issue, set the evidence bar, directed each step, and reviewed the code, the tests, and the results before this PR. The model traced the route and the service, wrote the fix and the regression test, and ran the checks under that direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> |
||
|
|
25e9a9c2e9 |
feat(ci): publish both variants of the Runner image from master (#21230)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Parts 2 and 3 of this stack build the Runner alone as an image with two variants, and check it on pull requests. Nothing publishes it > - Downstream sandbox and VM images can copy the Runner only from a registry, and they must be able to verify where it came from > - A published image must be exactly the bytes that were checked, and its name must never point at something else later > - This pull request publishes both variants from `master` as `ghcr.io/paperclipai/paperclip-runner`, one time for each distinct set of inputs, and attests them > - Nothing consumes the image in this pull request. The application image build does not change > - The benefit is one verified, attested Runner that later changes can copy by digest ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21096 (the merge-queue image build). No issue exists for this change. **Problem or motivation** The Runner image exists only inside a pull-request check. A consumer cannot pull it. There is no rule for which commit publishes an image, no protection for its tags, and no way to verify a copied Runner. **Proposed solution** Publish from a `master` push in this repository, and from nothing else. Build only when no image has the digest of the commit's inputs. Let the build legs hold no write access. Push the bytes that the legs checked by digest, attest each multi-platform index, and write the tags last. Write the full tag after the slim tag, so that the full tag means "both exist". **Alternatives considered** - Publish from `docker.yml`. That workflow holds one concurrency slot for each ref, and the Runner image needs its own signer identity, so that its attestation cannot stand in for the application image's. - Resolve the lockfile in the workflow before a `master` build, as `docker.yml` does. Then the published Runner depends on the state of the npm registry at run time, and the digest cannot be computed again from a commit. - Hand each variant to the publish job as its own OCI archive. The two archives hold the two shared layers two times: 1.27 GB for each architecture in place of 0.85 GB. - Write the full tag first. Then a consumer can see a full image whose slim image does not exist yet. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 4 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant 4. #21230: publish both variants from `master` (this pull request) ## What Changed - `.github/workflows/runner-image.yml`: adds the trigger `push` to `master` and the job `publish`. - `identity` checks the committed lockfile against the Runner's manifests, computes the digest in the build context, and asks the registry whether these inputs are published. - `build` is the job of part 3. It also records what it built, on every event. On a canonical `master` push it uploads that. It cannot write to the registry on any event. The only token in the job can read packages, and only the lookup step receives it. - `publish` runs only for a `master` push in this repository. It builds nothing and runs no dependency code. - Lockfile rule. A `master` push builds from the committed `pnpm-lock.yaml` and resolves nothing, so the digest is a function of the commit. When the committed lockfile is behind the manifests that the Runner build reads, the run writes a notice, publishes nothing, and stays green. The commit that merges the lockfile refresh publishes the image. Pull requests and manual runs resolve the lockfile first, always build, and never publish. - Hand-off. Each leg joins its two OCI archives into one OCI layout that names the images `slim` and `full`. The layout stores the two shared layers one time. `record` then reads the layout back, each image by its name, and checks every blob against its digest. The leg's record binds the layout to both image digests, the architecture, the inputs digest, and the commit. Those digests are the digests of the archives that part 3 unpacks and checks, so the tree that is checked and the image that is pushed are the same bytes. The publish job accepts only records for its own inputs and commit, and only a layout that still has the recorded bytes. - `record` runs on pull requests too. Only the upload is for a `master` push. A pull request therefore runs the hand-off on real build output. - Publication order. The publish job pushes the four images by digest (`skopeo copy --preserve-digests`). It makes a candidate index for the slim variant, then one for the full variant that names the slim index in the annotation `io.paperclip.runner.slim-digest`. It verifies that each index names exactly the recorded images, attests both, and then writes `inputs-<digest>-slim` and, last, `inputs-<digest>`. - The full tag is the completion marker. Every "is it published" lookup reads the full tag. `publication-state` reports one of: | State | Registry | What a run does | | --- | --- | --- | | `absent` | Neither tag | Builds and publishes | | `incomplete` | The slim tag alone | Builds and publishes. It replaces the slim tag with its own image, then writes the full tag | | `published` | The full tag, and the slim tag at the digest that the full index names | Nothing | | broken | The full tag, and a slim tag that is missing or points at another image | Fails, with the steps for an operator | | `unknown` | No clear answer | Builds. The publish job fails before it writes a final tag | - Recovery rule. A run that stops between the two tags leaves the state `incomplete`. It does not block those inputs. The next run never keeps the earlier slim image: it writes its own slim tag immediately before its full tag, verifies both against its own index digests, and then reads the pair back as every later run will see it. The broken state cannot be repaired by a run, because no run moves a full tag. An operator must delete the two package versions and re-run the workflow run of a `master` push with those inputs. - Pairing rule for consumers, in the contract: resolve the full tag first, take the slim digest from the annotation of the full index, and pull the slim image by that digest. - `verify-index` accepts `linux/arm64` with the variant `v8`, which some BuildKit versions record, and no other variant. - `scripts/runner-image-publish.mjs` (533 lines): the commands `lock-state`, `record`, `verify-legs`, `publication-state` and `verify-index`. It reads archives with the reader of `scripts/runner-image-oci.mjs` (607 lines here), which can now read one named image of a layout. `lock-state` refuses a checkout, because pnpm can rewrite the lockfile of the tree that it checks. The workflow gives it an export that no build reads. - The workflow test pins the build arguments of the image build. Build arguments are not part of the image identity. - `doc/runner-image-contract.md` (197 lines): the section "Publication": the lockfile rule, the order, the pairing rule, the recovery rule, the create-only tags, the internal `candidate-` tags, the command that verifies provenance, and the step to make the package public. `doc/DOCKER.md` says where the image is published. ## Verification Tests, and the build jobs of the workflow on GitHub. Nothing in this pull request ran against a registry, and no publish job has run. - Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd): `runner-image-publish.test.ts` (75), `runner-image-workflow.test.ts` (12), the three other Runner image test files, the provider pack archive tests and the existing Dockerfile tests: 372 tests pass in 10 files. The same files pass on macOS. - The tests of the leg record failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does, and the publish script uses the one reader of part 3. - Leg records: both variants are accepted for this run's inputs and commit; other inputs, another commit, a changed layout, a record without a variant, an archive that lacks a blob, a full image that does not start with the slim image's layers, and a build that is not a publication build are refused. - Publication state: all five states, and the half-finished publication end to end. A slim tag alone lets the next run publish. A full image that is left with another run's slim tag is reported as broken. - Index: exactly the recorded images of one variant; the full index must name its slim index, and the slim index must name none; `v8` is accepted on arm64 only. - Registry lookups: a missing tag only when the registry says so; every refused, failed, or unreachable request is `unknown`. - Workflow: job permissions; the publish job and the upload are behind the canonical `master` push; the build steps receive no credential; the full tag is written last, after a strict lookup and both attestations; one export and a time limit for each build command; the two build arguments; actions pinned by commit; no expression inside a shell script. - `record` on the two OCI archives of a real `linux/arm64` build from part 3: the joined layout is 850,149,376 bytes, and both images read back by name with every blob under its digest. The two archives together are 1,273,646,592 bytes. The image digests in the record are the digests that `unpack` reports for the trees that were checked. - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `docker buildx build --check` reports nothing. The image identity is the same as in part 3 (`a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`), because this pull request changes no input of the image. GitHub, at this head: - The `Runner image` workflow passes with these changes: `Build and check (amd64)` in 18m30s and `Build and check (arm64)` in 7m48s. Each leg builds both variants, checks them, and writes the leg record. `Publish and attest` is skipped, as it must be on a pull request. Never run: - The publish job. It runs only for a `master` push. - The publish path: the registry lookups with a workflow token, `skopeo copy` from `oci-archive:<layout>:<name>` by digest, `docker buildx imagetools create` with index annotations, and the tag writes. - The attestations, and the `gh attestation verify` command in the document. - `actionlint`. It is not installed on the local host. - The full test suite and the full typecheck. CI must supply them. ## Risks - **Manual step.** GitHub creates the `paperclip-runner` package private on the first publication. A maintainer must make it public one time. Until then, consumers cannot pull the image without credentials. - The first `master` runs of the workflow are the first test of the publish path. A failure before the tags publishes nothing. A failure between the two tags leaves the `incomplete` state, and the next run with those inputs repairs it. - A slim tag can move while its publication is incomplete. A consumer must resolve the full tag first and take the slim digest from the annotation of the full index. It must never resolve `inputs-<digest>-slim` by tag on its own. The contract says so. - `candidate-` tags are written before the attestations and can name an index that is not attested. They are internal. - The broken state needs an operator, and it fails every `master` run with those inputs until then. It can arise only when someone changes or deletes a tag outside the workflow. - **Known residual.** Two `master` runs that both start before a new digest is published can each build the image. Each build leg and the publish job look again before they work, and the publish job runs one at a time for each digest. The cost is runner time, not correctness. - A commit whose lockfile is behind the Runner's manifests has no published Runner image until the lockfile refresh merges. A lockfile that the refresh cannot fix leaves every `master` run green with only a notice, for as long as it lasts. - The tags are create-only because the workflow never moves a tag of a complete publication. The registry does not enforce that. Consumers must pin the digest and verify the attestation. - `payload.runnerSourceRevision` in a pack from this image is the commit that published the image. Code that reads it as the application's commit must change before it consumes the image. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
717aae060a |
fix(environments): let same-run recovery create a replacement once its sandbox is proven stopped (#21223)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs execute on environment leases, and `createOwnedSandbox` records each sandbox creation so that a lost reply never creates a second sandbox > - The record key is company, environment, run and request key. The plugin request key is the environment fingerprint, so it is the same for every attempt of one run > - A same-run recovery can need a new environment after the first lease is closed with successful cleanup and a termination receipt > - The old record is still `attached` to the closed lease, so the recovery fails with "Sandbox creation has already transferred ownership" before the provider is called > - This pull request lets the creation fence accept the same stop proof that recovery already requires. A creation record whose sandbox is proven stopped becomes resolved, and a resolved record no longer holds its key > - The benefit is that same-run recovery gets one replacement sandbox, while unconfirmed cleanup still blocks a second sandbox ## Linked Issues or Issue Description Fixes #21214 Refs #20755. That pull request fixes the same failure with a different rule. It retries under a new key generation (`<key>#2`, `#3`, ...) when the recorded lease has ended, and it accepts any ended status, including `failed`, without a termination receipt. A lease whose cleanup is not confirmed can then produce a second sandbox while the first one can still run. This pull request keeps the original key and requires the same stop proof that Core uses elsewhere (`hasRemoteTerminationReceipt`). It does not add a key generation. ## What Changed - `server/src/services/sandbox-creation-operations.ts`: when `createOwnedSandbox` finds the key held by an `attached` operation whose lease is not active, it checks the same stop proof that same-run recovery requires before it acquires: `hasRemoteTerminationReceipt(resolveRemoteTerminationLease(lease))`, for the sandbox that this operation created (`creationProviderLeaseId`, or the lease's `providerLeaseId`). - With that proof, it runs the same admission transaction one more time. That transaction first marks the old operation `resolved` (`WHERE id AND state = 'attached' AND lease_id`), and then inserts the new operation. - Without that proof, it throws exactly as before. - The check covers every source of a stop receipt: `releaseLease`, the Runner `complete` operation (which writes receipts with direct SQL for reusable leases), stop-and-retain settlement, and the receipt that `resolveRemoteTerminationLease` derives for an expired reusable lease. - A task replacement can leave an earlier operation attached to a lease that now holds another sandbox. That operation is not proven and keeps its key. - Concurrent recoveries serialize on the row update. The second one updates nothing, finds the first one's unresolved operation through the partial index, and replays or refuses as before. There is one provider call. - Migration `0475_sandbox_creation_owner_unresolved.sql` recreates `sandbox_creation_operations_owner_idx` under the same name as a partial unique index (`state <> 'resolved'`). The drop and the create are in one migration transaction. The Drizzle schema matches. - The conflict lookup in `createOwnedSandbox` ignores resolved rows. An active attached lease is still replayed. Pending and `needs_reconciliation` rows still refuse. `releaseLease` does not change. - `doc/DATABASE.md` and `doc/idle-environment-cleanup.md` describe the operation lifecycle. ## Verification - `pnpm --filter @paperclipai/db generate` - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/server typecheck` - `vitest run src/migration-snapshot-drift.test.ts` (db) - `vitest run src/__tests__/sandbox-creation-operations.test.ts`: 48/48. With the service change reverted, the 5 replacement tests fail. - The 9 other server test files that read `sandbox_creation_operations`, plus `environment-runtime.test.ts`: 564/564 - `pnpm check:migrations` passes New tests in `sandbox-creation-operations.test.ts`: - A stop receipt from `releaseLease` (sandbox driver, and the plugin driver through `acquireRunLease` in the #21214 sequence), from the `complete` operation's direct write on a retained reusable lease, and derived only from the destroyed successor of an expired reusable lease each let the next creation resolve the old operation and create exactly one replacement. A retry replays that replacement. These tests fail before the fix. - Failed cleanup, success without a receipt, a receipt for another lease or another provider task, a retained lease, and pending cleanup each still throw "transferred ownership" with no provider call. - An earlier task's operation that a task replacement left attached keeps its key. - Two concurrent recoveries make one provider call. This test fails before the fix. - Reapplying migration 0418 keeps the partial index. I did not run `pnpm verify` or the full suite locally. CI runs them. ## Risks - Migration: the drop and the rebuild of the index take an exclusive lock on `sandbox_creation_operations`. The table has one small row for each sandbox creation, so the lock is short. - Rolling deploy: the migration is marked `online_compatible`. The previous release uses an untargeted `ON CONFLICT DO NOTHING`, which also matches a partial unique index. Its conflict lookup has no state filter, so it can find a resolved row and throw. That fails closed and never creates a second sandbox. - A legacy runner-task operation without `creationProviderLeaseId` matches any lease it is attached to. Runner-task operations record this field before they attach, so this is only a theoretical gap. - An operation stays `attached` until the next creation with the same key checks it. This is the same as before when no recovery comes. - Stop-and-retain settlement is not tested directly. The shared receipt check covers it. - The fence is unchanged for every case without a stop receipt. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4dc5492c2d |
fix(recovery): retry a failed Retry run's sandbox resume failure when the sweep reaches it first (#21124)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task that runs on a reusable cloud sandbox resumes the same sandbox on its next run > - When the resume fails before provider work (`reusable_sandbox_resume_failed`), the run gets a bounded automatic retry > - On a hosted staging server, 8 of 14 such failures in 10 hours did not get that retry. Each task ended `blocked` and needed a person to press Retry > - In all 8, the stranded-task sweep blocked the task between the failed write and the run's own retry scheduling. The block cleared the claim that the retry needed > - This pull request makes the sweep give such a run to the same bounded retry scheduler, not escalate it > - The benefit is that a transient sandbox resume failure no longer costs the task, also when settlement is slow or the server stops during an upgrade ## Linked Issues or Issue Description Refs #21108, #21113, #21078 **What happened?** A person pressed Retry on a task. The run tried to resume the task's saved Daytona sandbox before provider work. The resume failed (`cause=transport_timeout`, or `provider_unavailable status=503`), and the run was written `failed` with `reusable_sandbox_resume_failed`. Then the stranded-task sweep moved the task to `blocked` and cleared its claim. This happened 1 to 43 seconds after the failed write. The run's executor scheduled its bounded retry 2 to 81 seconds after the failed write, and the gate refused it ("The automatic retry could not revalidate the original user continuation."). Read-only audit of 14 failures (all legacy runtime, `executionRecovery: bootstrap, providerWorkStarted: false`): 6 got their retry, and the 5 most recent ones recovered. 8 were refused this way and wait for a person. In one example the run was written failed at 14:23:31, the task was blocked at 14:24:14, and the retry was refused at 14:24:52. The server was being replaced at that time. **Expected behavior** A transient preparation failure before provider work continues with its bounded retry. The sweep must not take the claim away from that retry. When the retry budget is spent, or the continuation is refused, the task escalates once, as before. **Steps to reproduce** 1. Press Retry on a task whose reusable sandbox resume fails before provider work. 2. Let the run's executor be slow to settle the failure (more than the remaining controller lease, at most 60 seconds), or stop its server after the failed write. 3. Let the stranded-task sweep run on another server. The task becomes `blocked`, and no retry is scheduled. **Root cause** `classifyContinuationFailure` lists `reusable_sandbox_resume_failed` and the other transient preparation codes in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The sweep assumes that the executor already scheduled the dedicated bounded retry, so it escalates at once and never asks the scheduler. #21108 added a skip while the executor still settles the run. That skip uses a live local execution or an unexpired controller lease. The lease stops renewing when the run is written terminal (renewal requires `status = 'running'`). So the skip covers at most about 60 seconds after the last renewal, and it covers nothing when the executor's server stops during settlement. **Paperclip version or commit** Staging at `a65644d1e1`. Current `master` has the same code path. **Deployment mode** Hosted (cloud), two servers alive during each upgrade. **Agent adapter(s) involved** `codex_local` on a Daytona sandbox environment. The change is adapter-independent. ## What Changed - `server/src/services/heartbeat/retries.ts`: add `failedPreparationOwnsBoundedRetry(run)`. It is the persisted form of the run-completion decision: a failed legacy run with no native session, bootstrap evidence that provider work did not start, and a transient preparation code. A restore failure counts only when its owner marked it retryable. Codes that an accepted interaction retries itself are excluded. - `server/src/services/recovery/service.ts`: when the latest run is such a run, the sweep does not escalate it as non-retryable. It goes to `scheduleRecoveryRetry`, which is the run's own bounded scheduler. The scheduler reuses a successor that already exists, so the executor and the sweep create one successor. The task escalates when the scheduler reports exhaustion or a refused continuation. When the scheduler declines the retry for another reason (for example a permanent refusal), the task escalates once as a non-retryable failure, as before. The #21108 settling skip still runs first. - `server/src/services/heartbeat.ts`: pass the predicate to the recovery service. - `doc/execution-semantics.md`: document the rule. ## Verification - New tests in `server/src/services/task-retry.test.ts`, in the staging shape: a Retry-admitted run failed with `reusable_sandbox_resume_failed`, bootstrap evidence, and an expired controller lease from another server. - "retries a failed Retry run's transient preparation failure when the sweep reaches it first" (`slow_executor`, `executor_gone`). Red before the fix: the task became `blocked`, as on staging. Green after the fix: the task stays `in_progress` with one `scheduled_retry` successor (`transient_failure`, attempt 1), and dispatch accepts it. For `slow_executor`, the executor's own scheduling then reuses the same successor, and there is still one successor. - "still escalates a failed Retry run's transient preparation failure once its bounded retries are spent": the task becomes `blocked` with no successor. - "escalates a failed Retry run's transient preparation failure once when the scheduler declines it permanently" (review finding): red before the follow-up commit (the task stayed `in_progress` with no run), green after (the task is `blocked` once, and a second sweep adds no notice). - `pnpm exec vitest run src/services/task-retry.test.ts`: 114/114 pass. - `pnpm exec vitest run src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-process-recovery.test.ts` (with task-retry): 631 of 633 passed. The 2 failures were 15-second timeouts under parallel load in native Board-wait tests that this change does not touch. They passed when run alone. - `tsc -p server/tsconfig.json --noEmit` passes. - Not run: a live staging reproduction. The staging evidence is read-only (run events, task activity, run records). ## Risks - The sweep can now schedule the bounded retry before the executor does. The scheduler locks the task and the run and reuses an existing successor, so there is never a second successor. All continuation gate checks still apply. - The sweep acts only after the #21108 settling skip lapses, so a healthy run is not affected. No wait is added before a normal turn start. - No schema change. Rows from older releases follow the same rule because the predicate reads only persisted fields. - Out of scope: the provider-side causes (Daytona 503 at 14:21Z, resume calls that time out). #21113 handles the in-run wait for idle transport timeouts. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, read-only API reads, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
afe86e9ded |
fix(runtime): record a plugin task's stop receipt when its stop is proven (#21237)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A plugin environment runs each Paperclip Runner turn as a provider task, and a follow-up run can continue the same Claude/ACPX conversation from the saved checkpoint > - Core authorizes that continuation only when the predecessor run's remote execution has stopped (`remoteExecutionHasStopped`), so two Runners never write one checkpoint > - The predecessor's own settlement already proves the stop: the provider reports `executionStopped`, and Core then completes the task > - On a per-turn (ephemeral) lease, Core records the stop receipt only when it releases the lease, and that release can be held for a long time (for example by a pending workspace export) > - In that window, an ordinary follow-up fails with `adapter_failed: Plugin Runner retained session is not authorized`, even though the stop is already proven > - This pull request records the stop receipt when the stop is proven, and the stop check accepts it before the release > - The benefit is that follow-ups continue as soon as the predecessor is proven stopped, and the release gates keep their meaning ## Linked Issues or Issue Description Fixes #21211 Refs #21214 and #21223. Those cover the creation fence for same-run recovery, which still requires a released lease. Production evidence for this change: in both observed failures, the predecessor was a per-turn plugin run. Its `task.settle` completed with all provider calls successful. Core then did not release its lease for 30 minutes or more, and no release call was made in that time. The follow-up failed in checkpoint recovery (`recoverPluginRunnerCheckpoint` → `resolvePluginTaskResume`) within seconds of admission. ## What Changed - `server/src/services/environment-task-runtime.ts` - After a recorded provider stop, `complete` writes the bound stop receipt (`remoteTerminationReceipt(..., { state: "stopped" })`) on the current lease for every lease policy. Before, it did this only for `reuse_by_environment` leases. Reusable tasks still confirm every owner and still move retained owners to released. - New work on a lease (`submit`, `copilot_metadata`, first portable resume) removes the stop receipt in the same write that clears `runnerTaskExecutionStopped`. A replacement physical task does not copy the old receipt. - `server/src/services/remote-execution-termination.ts` - New `hasBoundRemoteStopReceipt` checks only the receipt binding (company, run, lease, provider and physical task). `hasRemoteTerminationReceipt` uses it and keeps its released-only semantics. Resource cleanup gates do not change. - `remoteExecutionHasStopped` and `stoppedRemoteCleanupScopes` accept a lease that is released with its receipt, or that is still `active` (not released) with a bound stop receipt. A lease whose release started and did not settle does not count. After release starts, the release owns the receipt. - New `remoteExecutionHasReleased` gives the release-based answer. - `server/src/services/environment-run-orchestrator.ts`: same-run plugin recovery uses `remoteExecutionHasReleased`. The same-run resume check after it requires released receipts, so its behavior does not change. - `resolvePluginTaskResume` keeps its checks and their order. Its refusal now names the first failing check: `Plugin Runner retained session is not authorized (reason: <code>)`, for example `predecessor_not_stopped`, `scope_mismatch` or `identity_mismatch`. The old message stays as the prefix, and no ids go into it. The stale-availability recheck in `environment-task-runtime.ts` names its refusal the same way. Production telemetry for #21211 could not tell which check fired. The next occurrence will show it. - `doc/execution-semantics.md` and `doc/testing-native-remote-continuity.md` describe the stop receipt and when it is recorded. Caller audit for the changed meaning of `remoteExecutionHasStopped`: | Caller | Needs | Decision | | --- | --- | --- | | `agent-instruction-working-copies.ts` | stop | Accepts. Directory cleanup still requires a released "destroyed" receipt. | | `environment-run-orchestrator.ts` | release | Moved to `remoteExecutionHasReleased`. | | `plugin-task-resume.ts` | stop | Accepts. This is the fix. | | `native-runtime/native-runner-ownership.ts` | stop | Accepts. `hasPendingRunCleanup` still counts the active lease as pending. | | `native-runtime/preserved-native-result-finalization.ts` | stop | Accepts. Workspace recovery has its own gate. | | `heartbeat/case-handoff-lifecycle.ts` | stop | Accepts. `hasPendingRunCleanup` still holds active leases. | | `heartbeat/run-control.ts` (3 sites) | stop | Accepts. Stop acknowledgement and quarantined ownership retirement. Holds and blockers still apply on the normal path. | | `heartbeat/runtime-execution.ts` | stop | Accepts. Release errors still fail it. | ## Verification From `server/`, after `node scripts/agent-preflight.mjs`: - `npx vitest run src/__tests__/environment-task-authority.integration.test.ts`: 10/10 pass. The 2 new tests fail without the fix. - The plugin task and environment runtime suites (`environment-task-authority.integration`, `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `environment-task-runtime`, `plugin-task-source-recovery.integration`, `environment-runtime`): 6 files, 366 pass. - The audited caller suites (`agent-instruction-working-copies`, case handoff legacy and native, `environment-run-orchestrator`, `heartbeat-case-handoff`, `explicit-native-continuation`, `run-control`, `native-remote-retry-admission`, `preserved-native-response-wait`): 9 files, 806 pass. - `pnpm --filter @paperclipai/server typecheck`: pass. The new integration tests cover: - `complete` writes a bound stop receipt on a still-active ephemeral lease. `remoteExecutionHasStopped` is true and `hasRemoteTerminationReceipt` is still false. - `resolvePluginTaskResume` (inspect and launch) authorizes a follow-up's new task, and the real host `read_state` checkpoint read succeeds, while the predecessor's release is held. - Refusals: no provider stop, a receipt for another provider task, an unstopped sibling lease, and a release in progress. - New work on the lease removes the receipt. Release overwrites the receipt with its own receipt. Warm owners keep their behavior. I did not add an executor-level test. `native-session-executor.test.ts` mocks `resolvePluginTaskResume`, so the integration test runs the real checkpoint read path instead. I did not run the full suite locally. CI runs it. ## Risks - `remoteExecutionHasStopped` now returns true earlier for a run whose leases are still held. Each caller is audited above. Callers that need released resources use the release-based predicates. - The stop receipt can now be on an `active` lease. The SQL readers of `remoteExecutionTermination` (cost accounting, execution blockers, idle-work proof, orphan settlement) also require a released status. `pinFailedObservedWorkspace` treats an active ephemeral lease with a receipt as not pinnable. That path applies only to the remote-workspace flow and falls back to `preserveNativeFailedWorkspace`. - Leases that settled before this change have no early receipt. They continue to wait for their release. - Nothing compares the refusal message exactly. Matchers use substrings, and run failure classification does not use the message. - A follow-up still fails closed if the predecessor's stop was never proven. This change does not add a recoverable wait. - The reason the release was held in production is not addressed here. - No migration and no API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, sub-agents, Honeycomb trace queries) and extended thinking. One sub-agent wrote the change. A second session reviewed it before this pull request. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b9308f9828 |
fix(ui): keep long agent names inside task columns (#21155)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Task views show the assignee and creator in fixed-width columns. > - Long names can extend past those columns and cover adjacent values. > - The identity labels already support text truncation but need a width limit. > - This pull request limits each identity to its task column. > - Names stay on one line, avatars keep their size, and adjacent values stay readable. ## Linked Issues or Issue Description **What happened?** Long agent and human names extend outside the Responsible and Kicked off by columns in task views. **Expected behavior** Names fit inside each column. Long names show an ellipsis. Avatars remain visible. The title attribute retains the full name. **Steps to reproduce** 1. Open a task view with Responsible and Kicked off by columns visible. 2. Use a long agent name for the assignee and creator. 3. Observe names covering adjacent columns. ## What Changed - Limit agent and human identity width to the containing task column. - Add a production-component Storybook example for long names, names without spaces, and short names. - Add browser layout checks for both identity columns, avatar size, full text, adjacent values, and row height in both themes at 640px and 1200px. ## Verification - The browser regression failed before the fix: an identity reached x=424 while its column ended at x=144. - The focused Chromium suite passes all four cases at 640px and 1200px in light and dark modes. - `pnpm --dir ui exec vitest run src/components/IssueColumns.test.tsx src/components/IssuesList.test.tsx`: 83 tests pass. - `pnpm --filter @paperclipai/ui typecheck`: passes. - `node scripts/check-token-gates.mjs` and `git diff --check`: pass. - `pnpm verify` returned exit 2 because the browser test requires broader CI coverage. The complete PR CI run supplies that coverage. - All tests, typecheck, build, and e2e gates pass on final commit `f73568822add60b4659f2cbdad0196fdfe010d3b`: https://github.com/paperclipai/paperclip/actions/runs/43271777908. - The separate commitperclip review passed: https://github.com/paperclipai/paperclip/actions/runs/43271777344. - Greptile gives this commit 5/5. There are no open findings or review threads. - All 61 checks are complete: 57 success, 1 neutral, and 3 skipped. The Storybook workflow was skipped; the focused browser suite passed locally. ## Risks Low risk. The width limit applies only to identity labels in shared task columns. The full name remains in the DOM and the existing title attribute. No API or stored-data changes. ## Model Used OpenAI Codex, based on GPT-6, with code tools and Chromium browser checks. The runtime does not expose a more specific model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
497ccba8f1 |
fix(native): replace a reattached Runner that left its recovery read unanswered (#21272)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs execute in a remote Runner that the server controls over the durable PRP channel > - When a run is interrupted, a same-run retry reattaches the live Runner and reads its session (`session.snapshot`) before it continues > - On staging, the reattached Runner authenticated and restored its session, and then did not answer the read > - The recovery kept that Runner, so each later attempt adopted the same Runner and failed the same way until the attempt budget ended the run > - This pull request makes the next attempt replace that Runner with a fresh Runner over its saved journal, and makes a failed Codex history read say what failed > - The benefit is that an interrupted Codex run continues without a manual Retry, and its provider turn is not sent again ## Linked Issues or Issue Description Refs #21161, Refs #21053, Refs #20853 **What happened?** A Codex agent on Daytona ran recovery runs (wake `issue_recovery_action_restored`). In one hour, 11 of about 35 finished runs failed. All failed the same way: - `native_session_recovery_failed: HarnessCapabilityUnavailableError: read is unavailable: Error: runnerd exited while waiting for session.snapshot` (8 runs) - `native_session_recovery_failed: HarnessReconciliationError: codex_history_read_failed: thread/turns/list requires supported paginated history; Error: provider_transport_failed: PRP command session.snapshot timed out (runner connected; pending ahead 0; last runner event seq 338 76s ago)` (3 runs) The run log of each run shows one sequence: 1. Attempt 1 fails with `runnerd did not report the provider turn identity`. The close leaves the Runner alive and unsettled for the same-run reattachment. 2. Attempt 2 adopts that Runner (same PID). The Runner authenticates. The log shows `runnerd restored its durable PRP session and provider thread`. Then the driver's recovery read (`thread/read` or `thread/turns/list`, which both send `session.snapshot`) gets no answer. The Runner still answers WebSocket pings, but it sends no command result and almost no events (`pending ahead 0`, last event 76 s to 391 s earlier). The recovery treats this as transient and detaches. The Runner stays alive. 3. Attempt 3 adopts the same Runner again and fails the same way. In some runs a protocol fault (`thread_binding_mismatch`) closes the transport during the read, so the message is `runnerd exited while waiting for session.snapshot`. 4. The provider attempt budget ends the run. #21161 did not help. Its wait extension applies only while a queued answer is being committed. Here the Runner sent no answer at all. The `paginated history` text was also wrong. `codex-history.ts` called every failed history read unsupported, also a transport timeout. **Expected behavior** A Runner that left a recovery read unanswered is not adopted again. The next attempt starts a fresh Runner from the saved journal, resumes the provider session from the saved checkpoint, and does not send the provider turn again. A failed history read reports the real failure. A Codex build that refuses paginated history falls back to one complete thread read. **Steps to reproduce** 1. Start a Codex run in a remote (Daytona) Runner. 2. Let the first attempt fail at turn start, so that the close keeps the Runner alive. 3. Let the reattached Runner stop answering commands after it authenticates (for example, stop the Runner process after its first answer). 4. See `PRP command session.snapshot timed out (runner connected; pending ahead 0; ...)` on attempt 2 and the same failure on attempt 3, with the same Runner PID. **Paperclip version or commit** Staging Core `57aeda190b` (contains #21161 and #21120). Fixed on `master` after `53608bd1d6`. **Deployment mode** Hosted (staging), remote Daytona sandbox Runner. **Agent adapter(s) involved** Codex (native Runner, `paperclip-runnerd`, `codex_app_server`). ## What Changed - `nativeRecoveryCommandUnanswered` (executor) finds a `native_session_recovery_failed` whose Runner did not answer a recovery command (`PRP command <type> timed out` or `runnerd exited while waiting for <type>`). Identity, authority, integrity and incompatibility refusals do not match. - When such a failure ends a `reattach_remote_runner` attempt on a sandbox target, the failure detail records `adoptedRunnerUnanswered: true`. - `claimNativeRestartRecoveries` copies it to the next claim as `replaceUnansweredRunner: true`, in the same way as `exitedBeforeSessionOpen`. - `verifyRemoteRunnerOrSettledExit` with `replaceUnansweredRunner` keeps all identity checks. When the old Runner is still alive, it returns no adopted process, and the attempt launches a fresh Runner from the saved journal. The existing launch fence stops the old `paperclip-runnerd` on that state folder and its whole lineage, and fails the launch closed when it cannot prove them gone. A Runner that already exited mid-turn keeps the existing fail-closed path, because its provider or tool processes can survive where the fence cannot find them. The task then continues in a fresh session from the saved work. The attempt budget is unchanged. - `codex-history.ts`: a history read that fails without a provider answer is now `codex_history_read_failed: <method> read failed; <error>`. A provider refusal of the method (`-32601` or `-32600`) on the first page falls back to one `thread/read` with `includeTurns: true`. That fallback fails closed when the thread differs, or when turns, items or turn identities are missing. A refusal on a later page still fails closed. - `doc/execution-semantics.md` describes both rules. The launch-fence code comment now names the replacement case. ### Changed existing test `codex-history.test.ts` "does not turn an interrupted page or unsupported method into idle history" used a plain `Error("method not found")` on page 2 and expected the `supported paginated history` text. A plain error is now a failed read, not an unsupported method. The test now uses a real provider refusal (`CodexRpcError` `-32601`) on page 2 and still expects a fail-closed rejection. A new test covers the failed-read message. ## Verification - Red before the fix, green after: - `server/src/services/native-runtime/native-session-executor.test.ts` › "a same-run reattachment whose adopted Runner never answered its recovery read": the replacement and staging-message tests failed on the old code (the Runner was adopted again, the three staging messages were not recognized). Two further tests ("never resumes beside an exited unanswered Runner's lineage") failed on the first revision of this PR, which relaunched beside an exited mid-turn Runner, and pass now. All 15 pass. - `native-runner-restart-recovery.integration.test.ts` › "asks the same-run reattachment to replace a Runner that left its recovery read unanswered" (embedded Postgres): the claim had no `replaceUnansweredRunner` on the old code. Now passes. - `codex-history.test.ts`: 7 new tests failed on the old code. All 16 pass. - Real `paperclip-runnerd` and fake Codex: `runnerd-codex-transport-remote-retention.test.ts` › "continues the same provider thread from a fresh Runner over the journal of a retained Runner that stopped answering, without replaying its turn". It runs the staging sequence: turn started, close keeps the stopped Runner, a second controller adopts it, the Runner stops answering and the controller detaches, the Runner and provider are killed (the launch fence), and a fresh Runner over the same journal returns the same thread. The fake Codex call log shows one `thread/start` and one `turn/start`. - Focused suites: `packages/paperclip-runner` `src/drivers/codex/` plus the remote-retention, adoption and cold-restore transport suites (442 tests). Two suites timed out under parallel load and pass alone. Server: `native-session-executor`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `runner-lost-mid-turn-continuation` (1714 passed, 21 skipped). - `pnpm typecheck` in `server` (all layers) and `tsc` for `paperclip-runner` source and tests pass. - Not run locally: the container-gated remote Runner suites (`PAPERCLIP_TEST_RUNNER_CONTAINER`) and the full suite. CI covers the full suite. ## Risks - Medium. The replacement relaunches a Runner over a journal that is not settled, but only while the old Runner is alive. Safety comes from three facts: the launch fence stops the old Runner and its lineage with proof or fails the launch; the Runner restores its own durable journal (as local `resume_dead_runner` does); and the real-runnerd test shows that the turn is not sent again. An exited Runner is never relaunched beside its possible survivors. - If the previous close left a `runner.suspend` pending, a fresh Runner can execute it and exit. The attempt then fails as it does today. The usual detach path after an unanswered read sends no suspend. - The replacement uses the existing attempt budget. It does not add attempts. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run events and logs), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
9b613562b7 |
feat(docker): split the Runner image into stable layers and add a slim variant (#21229)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work: a daemon (`paperclip-runnerd`) and a provider pack of agent tools > - Part 2 of this stack builds the Runner alone as the image `runner-artifacts`, in one layer of about 850 MB > - Most of those bytes are provider binaries and Node. They change only with a provider version or the Node pin, but one layer means that every Runner source change makes all 850 MB new > - About half of the image is the pack's transport archives. A consumer that copies the pack as files does not need them > - This pull request splits the image into layers by how often their content changes, and adds a slim variant without the transport archives > - The benefit is that a new image shares about 93% of its bytes with the one before it, and a consumer that needs no transport archives pulls half as much ## Linked Issues or Issue Description Refs #21204 (part 2 of this stack) and #21167 (provider pack layers). No issue exists for this change. **Problem or motivation** The image from part 2 is one layer. A registry and a builder store and move layers, not files. A change to one Runner source file therefore makes a new 850 MB layer, although 790 MB of its content is the same as before. The image also holds the provider pack two times: as files, and as transport archives of those files. Some consumers need only the files. **Proposed solution** Write the image as four layers: the pack's large dependency files, everything else, the transport archives of those large files, and the other transport content. Make the two large layers byte-identical from build to build. Add the target `runner-artifacts-slim`, which is the first two layers. The full image is the slim image plus the two transport layers, as the same layer blobs. Check all of this on the built images. **Alternatives considered** - Two unrelated images. Then the registry stores the shared content two times unless the layers happen to match, and nothing ties the two together. - Split by path only, with all files of 1 MiB or more in the first layer. Six of those files are the Runner's own bundles under `dist`. They change with the Runner's source, so the first layer would change with it too. - Leave the transport archives as they were written. Their bytes then depended on timing (see "What Changed"), and the archive layer never repeated. **Roadmap alignment** This is build infrastructure. It adds no product feature and does not overlap an item in `ROADMAP.md`. This is part 3 of 4. Each part is on top of the one before it: 1. #21228 (merged): an optional field in the provider pack manifest 2. #21204: build and check the Runner image, with no publication 3. #21229: layers by change rate, and the slim variant (this pull request) 4. #21230: publish both variants from `master` ## What Changed - `Dockerfile`: two final stages. `runner-artifacts-slim` is two `COPY --link` layers. `runner-artifacts` starts from it and adds two more. The build stage sorts its output into the four layers. Every path is where it was in part 2: consumers copy by path and see no difference. | Layer | Content | Changes with | | --- | --- | --- | | 1 | The pack's dependency files of 1 MiB or more: provider binaries and Node | A provider version or the Node pin | | 2 | Everything else, with `runner-image.json` | Every Runner source change | | 3 (full only) | The transport archive of each file in layer 1 | The same as layer 1 | | 4 (full only) | The other transport archives, the overlay archive and the receipt | The same as layer 2 | - Stable bytes. The files of layers 1 and 3 get timestamp 0, and the large dependency files get it before their transport archives are written. A file under the pack's `dist` is the Runner's own output, so it and its archive are in layers 2 and 4. - `scripts/create-provider-pack-archive.mjs`, two separate commits. **This script also writes the application image's archives.** - The same file now gives the same archive every time. The script piped tar into Node's zstd compressor. When the pipe ended while the compressor still had its last chunk queued, Node finished the zstd stream two times, and the archive got an empty frame of nine bytes at its end. That depended on timing. In the pinned Linux base, 37 of 60 runs of the old code had the frame. The compressor is now ended only after it has taken every chunk. - Archive N holds file N of the receipt, and the receipt listed the files in path order. `dist` sorts before `node_modules`, so one more large file in the Runner's own output gave every dependency file another archive name, and layer 3 became a new blob with the same contents. The receipt now lists the files outside `dist` first. The readers (`provider-pack-layers.ts`, `retained-provider-pack-source.ts`) pair file N with `N.tar.zst` and do not use the path order, so they need no change. For the application image, the archive numbers change one time, and the bytes of each archive do not change. - `runner-image.json` is the same file in both variants. Its new field `fullImageOnly` names the two paths that only the full image has. `paths` did not change. - `scripts/runner-image-layout.mjs check` takes `--variant slim|full`. It is required and has no default. A slim tree that holds a transport path fails. A full tree without them fails. - `scripts/runner-image-oci.mjs` (new, 585 lines). It is not an input of the image, because the image build does not run it. - `unpack <oci archive> --out <tree>` writes the tree of an image from the layers of its archive, in order. It checks every blob against its digest. It refuses a whiteout, an absolute path, a path with `..`, a path that is written two times, an entry behind a link, and every entry that is not a file, a directory or a link. - `check-layers --slim <oci archive> --full <oci archive>` checks the table above on the built images. It reads the receipt to decide which archive belongs in which layer. It fails when the full image does not start with the slim image's layer blobs in the same order, when a path that is not a directory is in two layers, and when a shared layer holds transport content under any name or directory. - The script reads the archive and its layers itself. It does not use the system's `tar`, so every platform reads them the same way. It finds a member only under the name that buildx gives it. - `.github/workflows/runner-image.yml`: each leg builds each variant one time, as an OCI archive and nothing else. It unpacks both archives, runs `check` on each tree with its variant, runs `check-layers`, and runs the smoke matrix on the slim tree. The tree that is checked is therefore the bytes of the image, not a second export of the same build. Every step that builds has a time limit. The workflow still has no registry access. - Why one export: in the first runs of this pull request, the slim build stalled on all four legs. Each build command had two exports, a tree and an archive. The slim command, with every stage from the cache, wrote its archive and then made no progress on the tree for 50 minutes. A test now holds the rule of one export and a time limit for each build command. - The smoke matrix runs on the slim tree. The daemon is in a layer that both variants share, so one run covers both. - `server/src/__tests__/runner-image-identity.test.ts`: the test that holds the rules for the Runner stages now finds a stage wherever BuildKit starts one. It fails for `FROM x as name`, a flag on `FROM`, a lowercase or indented `COPY`, an `ADD` or `ONBUILD`, a `COPY` with another flag, and a `COPY` that continues on the next line. Nine samples show each case. - Two comments are corrected. The comment on `PAPERCLIP_RUNNER_INPUTS_DIGEST` in `build-provider-pack.mjs` said that no check reads the field. The `Dockerfile` comment on the publication argument named a workflow "that builds the image for publication". - `doc/runner-image-contract.md` (159 lines): the two variants, the section "Layers", how the checks get their tree, and the new checks. `doc/DOCKER.md` names the slim target. Script sizes: `runner-image-layout.mjs` 969 lines, `runner-image-oci.mjs` 585, `runner-image-identity.mjs` 166, `create-provider-pack-archive.mjs` 114. ## Verification Image identity at this commit: `a544fc9dde780963419ede0f34252f3a57de43a760d41d3eb8fcd4f577504f6a`. Linux (the pinned `node:24.22.3-bookworm` base with GNU tar 1.34 and zstd, the worktree mounted read-only): - `runner-image-identity.test.ts` (41), `runner-image-layout.test.ts` (100), `runner-image-oci.test.ts` (50), `runner-image-workflow.test.ts` (8), `provider-pack-archive.test.ts` (47, none skipped), `retained-provider-pack-source.test.ts` (19), `remote-provider-pack-verification.test.ts`, `docker-image-layer-reuse.test.ts`, `docker-build-stamp.test.ts`: 293 tests pass in 9 files. - The same files pass on macOS. The tests of `check-layers` failed on Linux in the first runs of this pull request: their fixtures named the archive members `./index.json`, which bsdtar finds and GNU tar does not. The fixtures now name the members as buildx does. `linux/arm64`, on a local Docker host, with the new export path: - Each variant builds as an OCI archive only: the full target in 156 seconds without a cache, the slim target in 5 seconds from the cache. - `unpack` writes the full image (10,519 entries that are not directories) in 4 seconds, and the slim image in 4 seconds. - `check <tree> --variant <variant> --expect-inputs-digest <digest> --expect-source-revision <sha> --expect-architecture arm64` passes on each unpacked tree. `runner-image.json` is the same file in both. - `check-layers --slim <slim> --full <full>` passes. - The smoke matrix passes on all five base images, from the unpacked slim tree. | Layer | Digest | Bytes | In | | --- | --- | --- | --- | | 1 | `sha256:3f21b15b3a99…70ac1c` | 379,609,589 | slim and full | | 2 | `sha256:d5567acc977f…b4f12a` | 43,906,956 | slim and full | | 3 | `sha256:6a340088ca2c…6ba5b5` | 410,938,259 | full | | 4 | `sha256:789e97f5db88…3eab44` | 15,656,249 | full | - The slim image is 423,516,545 bytes in two layers. The full image is 850,111,053 bytes in four. Stability, with three builds that share no cache for the build step: | Build | Source revision | Runner build output | Layer 1 | Layer 2 | Layer 3 | Layer 4 | | --- | --- | --- | --- | --- | --- | --- | | A | `aaaa…` | this commit: 6 files of 1 MiB or more under `dist` | `3f21b15b…` | `d5567acc…` | `6a340088…` | `789e97f5…` | | B | `bbbb…` | this commit | `3f21b15b…` | `a2f8a4fe…` | `6a340088…` | `4e6c492d…` | | D | `dddd…` | one new source file that adds two large files under `dist` | `3f21b15b…` | `33b0bdaf…` | `6a340088…` | `0ea322bf…` | - Layers 1 and 3 are byte-identical in all three: 790,547,848 of 850,111,053 bytes, 93%. Build D has two more large files under `dist`, eight in place of six, and another image identity. Its dependency files keep the archive numbers 0 to 20. With the earlier numbering, those numbers moved by two, and layer 3 was a new blob with the same contents. - Layer 1 has the same digest as before this round. Layer 3 has a new digest one time, because the archives have new numbers. Other checks: - `tests/runner-e2e` workflow tests: 39 pass. `node --test .github/scripts/tests/*.test.mjs`: 799 pass. `node --test` on the image contract scripts: 33 pass. `check-docker-deps-stage.mjs`, `check-node-version-policy.mjs` and `docker buildx build --check` report nothing. GitHub, at this head: - The `Runner image` workflow passes on both architectures with the new export path: `Build and check (amd64)` in 14m41s and `Build and check (arm64)` in 8m41s. Each leg builds both variants, checks both layouts and the layer rule, and runs the smoke matrix. The first runs of this pull request stalled in the slim export before any check. Never run: - `native-session-executor.test.ts`. It does not load on this host, with or without this change, because `@paperclipai/plugin-sdk` is not built here. It imports the reader of the retained pack and no archive number. - The full test suite and the full typecheck. CI must supply them. ## Risks - The two changes to `create-provider-pack-archive.mjs` change the application image's transport archives: about half of them lose nine bytes, and all of them get new numbers one time. An archive's content does not change. A reader pairs file N of a receipt with archive N of the same build. - The large dependency files in the pack, and in their transport archives, now have the modification time 0 (1970). Before, they had the build time. Nothing in this repository reads those times. - `unpack` is a reader of tar and gzip that this repository owns. It reads the layers that BuildKit, GNU tar and bsdtar write (tested with each), and it fails for a header it does not know. - The check on the built images, not the recipe, is what proves the layer rule; it runs in each leg of the workflow. - `check` now needs `--variant`. A caller that does not pass it fails with a clear message. No caller exists outside this repository yet. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (shell, file edits, local Docker builds). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1113.0-canary.2 |
||
|
|
3b4c3265ed |
fix(recovery): offer Try again on a model-unavailable failure and recheck the model first (#21244)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run fails, the task thread shows a failure marker, and some markers offer Try again > - A `native_provider_model_rejected` failure blocks the task for the person and shows "Model unavailable", but it has no Try again control > - Recovery must just work: after the person fixes the model, they need one obvious control that continues the task, and that control must not start a run that is sure to fail > - This pull request shows Try again on the "Model unavailable" marker and makes the retry recheck the selected model before it starts a run > - The benefit is that the person fixes the model, presses Try again, and the task completes, and a too-early press explains which setting to change instead of failing again ## Linked Issue or Issue Description Refs #21210 **What happened?** Paid run 43321773136, `recovery.retry-exhausted` (OpenCode, `openrouter/deepseek/deepseek-v4.1-flash-1017`). The run failed with `native_provider_model_rejected`. The task stayed blocked and the thread showed "Model unavailable", but no Try again control appeared (`task-chat-run-failed-try-again` not found in 30 s). Only the "Run failed" and "Usage limit reached" markers offer Try again. **Expected behavior** The "Model unavailable" marker offers Try again. Pressing it checks the selected model first. If the provider-rejected model is still selected, no run starts, and the person sees which setting to change with a link to it. After the person changes the model, Try again starts one retry of the original run. **Steps to reproduce** 1. Set an agent's model to an id the provider rejects. 2. Assign it a task. The run fails with `native_provider_model_rejected`. 3. Open the task. Before this change, the marker has no Try again. ## What Changed - `server/src/services/native-runtime/model-rejected-retry.ts`: `rejectedModelRetryRefusal` reads the model the failed run sent (`runnerProfileJson.nativeExecutionInput.provider.model`) and the model a new run would use (the task's model override for its assignee, else the agent's model). If they are the same, it returns a 409 refusal with code `native_provider_model_still_unavailable`, the setting to change (`agent_model` or `task_model_override`) and the rejected model. - `server/src/routes/agents.ts`: the exact failed-run retry (`POST /agents/:id/wakeup`, `reason: retry_failed_run`) applies this check before it queues anything. - `ui/src/components/task-chat/adjacent-errors.ts`, `TaskChatThreadView.tsx`: one `isRetryableFailureMarker` rule, now including "Model unavailable". - `ui/src/lib/model-retry-refusal.ts`, `ui/src/pages/IssueDetail.tsx`: a refused retry shows "Model still unavailable" with the server's message and an "Open <agent> model settings" link to `/agents/<agent>/runtime`. A task override refusal names the override in the message. - Eval `recovery.retry-exhausted` (oracle `runtime.retry-exhausted.v7`): the rejected model stays selected after the failure. The person presses Try again (409, toast names the model, its link opens the agent's model settings, no run starts in 20 s, task stays blocked), restores the working model, presses Try again (202), and the one retry of the original run uses the working model, saves the brief and completes the task. 3 runs: failed 1, succeeded 2. The budget-hold and two follow-up chat phases are removed; the case now tests the model recheck. `recovery.yaml` context and expected outcome updated, recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in its test, descriptors and inventory digest updated. ## Verification - RED on master: `TaskChatThread.test.tsx` "offers Try again on a model-unavailable failure and retries that run" fails (no control). `agent-live-run-routes.test.ts` "refuses a retry while the rejected model is still selected and starts no run" fails (the route queues the run). - GREEN: both pass. `agent-live-run-routes.test.ts` 101/101. - `server/src/services/native-runtime/model-rejected-retry.test.ts` (embedded PostgreSQL), 6/6: refuses while the agent still selects the rejected model (names the agent setting); names the task override; allows the retry when the agent or the override now selects another model, or the run failed for another reason; ignores another company's run. - `ui/src/lib/model-retry-refusal.test.ts`: names the setting and links to `/agents/morgan/runtime`; no link for a task override; other errors fall through to the generic toast. - Server `tsc --noEmit` clean. UI `tsc` clean apart from the pre-existing generated storybook catalog. `node scripts/check-token-gates.mjs`: clean. - Eval: `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog tests/runner-e2e/chat-catalog-domains tests/runner-e2e/chat-*.test.ts`: 339 files pass. New calibration rejects an accepted early retry, a run started while the model is rejected, a refusal without the model, setting link or blocked task, a retry that used the old model, and duplicate or missing retries. `pnpm test:e2e:runner:typecheck`: pass. Manifest test 10/10. `source-sdk-public-inputs --check`: `changed: false`. - Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch `chat-eval/qa-mr-paid`. Result to follow in a comment. ## Risks - The recheck compares the selected model with the model the provider rejected. It does not call the provider. A model the provider rejected stays refused until the person selects another one; a different but also unsupported model starts a run that fails with the same marker again. - The refusal message is a toast (15 s). The marker detail still says to choose a supported model. - No persisted shape changes; runs saved by older builds carry the same `runnerProfileJson` field. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7154030ef6 |
fix(heartbeat): draining/yielding servers claim no queued run, and closed tasks get no system continuation (#21258)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server drains. It must not start new runs, because its own exit would cut them off. > - The drain gate (run admission closed at SIGTERM) and the hand-over gate (#21127, yield to a replacement that already dispatches) were checked only by the queued-run dispatch passes. > - Other start paths (executeRun of a queued run, delivered continuations, immediate wake starts) went straight to `claimQueuedRun`, so an outgoing server could still start runs after traffic moved. > - Also, a system continuation queued while its task was open started after another run had finished the task, and took an extra turn on a done task. > - This pull request makes `claimQueuedRun`, which every start path uses, honour the drain and hand-over gate, and cancel a system continuation whose task closed while it was queued. > - The benefit is fewer runs cut off by a swap, and no extra turns on finished tasks. ## Linked Issues or Issue Description Refs #21127, #21135 **What happened?** 1. An outgoing server kept starting runs minutes after traffic moved to its replacement. Each such run was cut off when the old server exited. The run start went through `executeRun` of a queued run, which called `claimQueuedRun` without the drain or hand-over check. 2. In release-swap waves, a continuation (execution reconciliation delivery, invariant or recovery continuation, blocker-stall wake) was queued while its task was open. Another run then finished the task. The continuation still started and took an extra turn on a done task. **Expected behavior** A draining or yielding server leaves a queued run queued for the serving server, on every start path. A system continuation does not start on a task that was closed while it was queued. A person's message on a done task still runs. **Steps to reproduce** 1. Close run admission (drain) or set the hand-over yield on a server. 2. Start a queued run through `executeRun` (for example a delivered continuation). 3. See the run claimed and started on the draining server. ## What Changed - `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`) refreshes and checks the run-admission gate first. When admission is closed (draining or yielding), it leaves the run queued and returns. The dispatch passes already did this check; now every start path does. - `claimQueuedRun` re-reads the task for a system continuation (`issue_recovery_action_restored`, `issue_continuation_needed`, `issue_assignment_recovery`, blocker-stall wake). If the task is `done` or `cancelled`, it cancels the run as `issue_closed_before_start`, skips its wake, and releases the task. Other wake reasons, including a person's message, are not affected. - That cancel never consumes a person's message. Admission can attach deferred comments to a queued system continuation without changing its wake reason. The cancel locks the run's wake and the run (the order admission's coalescing uses), reads the run's current comment ids, and if any is a person's comment that no run of this agent has received yet (read-only use of `deliveredQueuedCommentIds`, which includes the history-read rule), the run is not cancelled: it is claimed as that person's follow-up, which may reopen the task through the normal path. A comment attached concurrently is either seen under the lock or, after the cancel commits, saved behind the cancelled run by admission. - Admission is checked again inside each claim transaction (ordinary claim, queued-comment claim, native review claim), right after the write that moves the run to `running`. If admission closed during the awaited checks, the transaction rolls back: the run stays queued with no `startedAt`, its wake and the task's execution lock are untouched, and nothing is charged. - Every cancellation `claimQueuedRun` makes from the caller's copy of the run is now a compare-and-set from `queued` (`setRunStatusFromLive(..., ["queued"])` through a new `cancelRunIfStillQueued` helper): the closed-task cancel, the refused-interaction cancel, the missing-@mention cancel, the daily-cap cancel and the blocked-dependency cancel. A run another dispatcher already started or finished keeps its status and result, and no wake or release follows. The queue module no longer receives the unconditional `setRunStatus`. - The closed-task cancel takes its decision under the wake and run locks, then cancels through `setRunStatusFromLive` fenced on the row version (`xmin`) the decision read. That saves the same never-started proof (`executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`) as every other queued-run cancel, so task release promotes a person's message saved behind it. A message attached after the decision changes the run, the cancel matches nothing and the decision is taken again. `HeartbeatRecoveryDependencies.setRunStatusFromLive` (`recovery.ts`) now types its existing `cancellationCondition` parameter as `SQL`. - The dispatch ranking in `startNextQueuedRunForAgent` is not changed. ### What I checked in `claimQueuedRun` for writes from the caller's copy - Cancellations through `setRunStatus`: all five now conditional on `queued` (above). - The final claims (ordinary, queued-comment, native review) already lock and re-read the run and update `where status = 'queued'`. - The case-handoff cancel and the chat-control stop already lock the run and update `where status = 'queued'`. - The budget-stamp update already writes `where status = run.status` (`queued`). - `cancelRunInternal` (agent removed, agent not invokable, board-owned budget stop, pause hold, budget cancel) re-reads the run and only cancels a non-terminal status, so it cannot overwrite a finished result. It can still stop a run another server has just started; that is a policy stop (agent removed, budget, hold) that would apply to that run anyway, and it is unchanged here. - `cancelStaleQueuedRun` already takes `expectedStatus: "queued"`. ## Verification Red-to-green tests in `server/src/services/heartbeat/queue.test.ts`: - "claims no queued run once run admission is closed, whatever path asks (draining or yielding server)": red before the gate was added to `claimQueuedRun`, green after. - "does not start a %s run whose task another run already finished": `issue_continuation_needed` and `issue_recovery_action_restored` are cancelled as `issue_closed_before_start`; `issue_commented` (a person's message, carrying its `wakeCommentId`) is claimed and the stored run is `running`. - "answers a person's message attached to a system continuation on a closed task once": still undelivered, the continuation is claimed and is the only unfinished run (red when the cancel ignores comment ids, green after); already received by the closing run, the continuation is cancelled as `issue_closed_before_start` and no other run exists. - "keeps the result of a continuation another dispatcher already started and finished": the stale queued copy is passed after the run succeeded; the stored run stays `succeeded` with its result, and no wake or release follows. Red before the compare-and-set (the run was overwritten to `cancelled`), green after. - "returns the claim to the queue when run admission closes while the claim is being checked": admission closes inside an awaited check; the run stays `queued` with no `startedAt`, the task's execution lock and the wake are unchanged. Red before the in-transaction re-check, green after. Red-to-green test through the real release path, in `server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`: - "delivers a person's message saved behind a system continuation cancelled at claim because its task closed": a never-started legacy continuation on a done task, with a person's deferred message behind it. Red before the fix: the cancelled run had `resultJson: null`, with no never-started proof. Green after: the proof is present, the wake is skipped, and the message is promoted and runs once. Test expectations changed because they asserted the old behaviour: - `server/src/services/agent-interaction-delivery.test.ts` "cancels refused queued runs through the public queue claim without dispatch": now expects the cancel through `setRunStatusFromLive(runId, "cancelled", ["queued"], …)` instead of `setRunStatus`. - `server/src/services/heartbeat/queue.test.ts` (refused-assignment held case): asserts `setRunStatusFromLive` is not called, instead of `setRunStatus`. - The `issue_commented` case of "does not start a %s run…" now asserts the run is claimed and `running` (it previously asserted only that the error code was not `issue_closed_before_start`). Local command, on this branch based on current master (includes #21234; the dispatch ranking it changed is untouched here): - `cd server && pnpm exec vitest run src/services/heartbeat/queue.test.ts src/__tests__/recovery-wake-herd.test.ts`: 102 tests passed. After the proof fix, `queue.test.ts` again passed 97/97, and the new stale-queue test passed. (The file's `afterAll` database cleanup hook times out on this loaded machine on master too; it is not related to this change.) Why the local release-swap wave does not show the yield fix: - The "yield to the replacement" signal exists only when the run-start handover is created (`server/src/index.ts`, about line 1320). That happens only when a direct deployment listen URL exists, that is, with `PAPERCLIP_EPHEMERAL_SERVER=true` and `PAPERCLIP_DATABASE_LISTEN_URL` (the volume-free configuration). - The local short form of the wave leaves `--ephemeral-server` out, so the handover is disabled and `isRunAdmissionClosed()` becomes true only at SIGTERM. The wave therefore cannot exercise the yield. - The evidence for this fix is the unit test above and the reasoning that every start path now reaches the same gate in `claimQueuedRun`. Hosted confirmation must come from a volume-free instance. - The closed-task fix was found in release-swap waves (an extra turn on a done task in two tasks). ## Risks - `claimQueuedRun` can now await an admission refresh. The refresh returns at once when the cached state is fresh, so the normal path adds no wait before `turn.start`. - A system continuation for a closed task is cancelled. This matches admission, which already refuses such wakes for a closed task. - No migration. No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d169dd5a90 |
fix(presentation): offer Retry only when no automatic continuation or scheduled retry is pending (#21264, includes #21200)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a run fails with a transient error, Paperclip schedules the next attempt by itself and shows "retrying automatically" with a countdown. > - The task presentation also offered the Retry button next to that countdown. > - The system owns the next attempt while the run is `scheduled_retry`, so the button is misleading: it suggests that a person must act. > - This pull request stops offering Retry while a scheduled retry is pending. Retry returns when the scheduled retry is past due with nothing pending, or when the retries are exhausted. > - The benefit is one clear message: Paperclip retries by itself, and Retry appears only when it is the way forward. ## Linked Issues or Issue Description **What happened?** A legacy run in status `scheduled_retry` (for example reason `transient_failure`, which is not a shared-resource wait and has no system recovery) projected `state: "retry_scheduled"` with `canRetry: true`. The UI then showed the Retry button beside "retrying automatically". The cause is in `projectTaskInterruptionState` in `server/src/services/task-interruption-presentation.ts`: the `retryAt > now` branch set `canRetry: !preparationPending && !nativeResumeAt`, which ignores that the run itself is the scheduled attempt. **Expected behavior** While a run is `scheduled_retry` and its retry time is in the future, the task shows the countdown and offers no Retry. When the scheduled retry is past due with nothing pending, or the retries are exhausted, Retry is offered. **Steps to reproduce** 1. Let a task's run fail with a transient failure that schedules an automatic retry. 2. Open the task before the retry time. 3. The task shows "retrying automatically" and the Retry button. **What a Retry click does today in this state** `requestTaskRetry` (`server/src/services/task-retry.ts`) treats a latest run in `scheduled_retry` as busy. It accepts the click (202), records the request as `coalesced` onto the scheduled run and returns `scheduledRetryRunId`. It does not start a second attempt next to the scheduled one. The button is therefore misleading but not unsafe, and this pull request changes only what the task shows. ## What Changed - `server/src/services/task-interruption-presentation.ts`: in the `retry_scheduled` branch, `canRetry` is now also false when `run.status === "scheduled_retry"`. - `server/src/services/task-interruption-presentation.test.ts`: three tests for a scheduled transient-failure retry. - This branch is based on current `master` and includes #21200, which has not merged yet and changes the same file. The diff of this pull request against #21200 is one line and the tests. ## Verification - `cd server && npx vitest run src/services/task-interruption-presentation.test.ts -t "automatic scheduled retry"` - "shows a pending transient-failure retry as scheduled without a competing Retry": red on `master` (`canRetry: true`), green with this change. - "offers Retry once the scheduled retry is past due with nothing else pending": green before and after. - "offers Retry once the scheduled retries are exhausted": green before and after. - Locally: `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts` and `task-interruption-automatic-path.test.ts` pass (107 tests). Server typecheck: 0 errors. The wider suites that read the presentation run in CI; they were not repeated locally because the machine was under memory pressure. ## Risks - Low risk. The change affects presentation only. The Retry route still accepts a click and coalesces it onto the scheduled run, as before. - If a scheduled retry never dispatches, the task offers Retry again once the retry time passes, as before this change. - No schema, API or UI change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, test execution). Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6cb0dd0c6b |
fix(recovery): continue twice-interrupted tasks and waiting follow-ups after a release swap (#21250)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off by a restart or a release swap, the fresh-continuation fallback continues the task in a new run. > - The continuation goes through normal admission, or through a person's saved follow-up when one is waiting. > - In release-swap waves, three gaps left tasks stopped with no run: admission refused an isolated continuation of a task with no project, a person's saved follow-up behind the replaced run was never promoted, and the fallback relied on a saved follow-up that the cut-off run had already read. > - This pull request closes those three gaps. > - The benefit is that a twice-interrupted task, and a task with a waiting follow-up, continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. A task with no project was interrupted twice. The second continuation used a fresh isolated workspace, which clears the task's workspace binding. The admission preflight still required that binding for an isolated task with no project, so it refused the continuation as `workspace_worktree_requires_project` about every 15 s. Each refusal re-blocked the task and added a comment (329 to 428 skipped wakes per wave). The runtime itself supports this case: it prepares a new workspace (`create_new`). 2. The fallback continued a task with delivery `saved_input` (a person's deferred follow-up is the next turn). The stranded-queue pass refused to promote that follow-up, because the replaced native run's finalization was `terminal_failure` (restart recovery ended it), not `committed`. The release path would also have re-blocked the task with a reconciliation incident. The follow-up never ran. 3. The fallback chose `saved_input` because a deferred user wake existed, but the cut-off run had already taken that message into its turn. Promotion settled the message as delivered, and no run followed. The task showed "working". **Expected behavior** An interrupted task continues by itself in each case, with no person action and no repeated comments. **Steps to reproduce** 1. Run a release-swap wave with projectless tasks in isolated mode and person follow-ups during runs. 2. Interrupt the same task twice, or interrupt a run that has a deferred follow-up behind it. 3. See the task stay blocked or "working" with no run. ## What Changed - Admission (`server/src/services/execution-workspace-policy.ts`, `server/src/services/heartbeat/queue.ts`, `server/src/services/fresh-continuation-fallback.ts`): `isUnrunnableWorktreeCombo` accepts `freshIsolatedContinuation`. The admission preflight sets it from the new `isFreshIsolatedContinuationWake`, which returns at once (no query) unless the wake context carries `freshContinuationIsolation`. This does not touch the dispatch ranking. - Follow-up promotion (`server/src/modules/wake-queue/adapters/postgres.ts`): the stranded-queue pass and the release path skip the native finalization gate and the reconciliation incident for a run that the fallback superseded (`agentRunSupersededByFreshContinuation`). - Saved-input choice (`server/src/services/fresh-continuation-fallback.ts`): a deferred user wake counts as saved input only when it has no comment ids, or still has a message that the agent has not received (`undeliveredLegacyUserCommentIds`, called, not changed). Otherwise the fallback records its own continuation. - Review round 1: (a) whenever the old writer is not proven stopped (`fresh_isolated_workspace`), the fallback records its own isolated continuation first, also when the task had a binding; saved input follows it on the new workspace. (b) Release skips the manual-admission hold and the pending-cleanup check for a run the fallback replaced with an isolated continuation (`fresh_isolated_workspace` or `remote_epoch`, via `freshContinuationIsolatesRun`); other replacements still wait. - Review round 2 (the isolated continuation must run before the waiting follow-up): while the fallback's `fresh_isolated_workspace` continuation is still pending delivery, release leaves the person's follow-up deferred. Otherwise the follow-up would become the task's next run without the isolation marker, and delivery would count it as the continuation and skip the isolated run. The continuation's own release promotes the follow-up behind it. `freshContinuationIsolatesRun` now also reports `continuationPending`. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/execution-workspace-policy.test.ts`: admission keeps a fresh isolated continuation of a projectless task runnable. - `server/src/modules/wake-queue/adapters/postgres.test.ts` "restart-ended native run": saved input behind a run the fallback replaced is promoted. - `server/src/services/fresh-continuation-fallback.test.ts` "relies on saved input only while it still holds an undelivered message" (taken: true gives `recorded`, taken: false gives `saved_input`). - `postgres.test.ts` "promotes saved input behind an exhausted restart hold ... (released, continuation pending)": the follow-up stays deferred while the isolated continuation is pending, then is promoted when the continuation's run is released. It was red before the fix (`expected 'promoted' to be 'released'`). - Local, round 2: `cd server && npx vitest run src/modules/wake-queue/adapters/postgres.test.ts src/services/fresh-continuation-fallback.test.ts`: 2 files, 177 tests passed. Local command, on this branch based on current master (includes #21222 and #21234): - `cd server && npx vitest run src/services/fresh-continuation-fallback.test.ts src/__tests__/execution-workspace-policy.test.ts src/modules/wake-queue/adapters/postgres.test.ts`: 3 files, 221 tests passed (re-run after rebasing on #21234). Local release-swap cells: - L1 (6 tasks, 3 swaps): 5 fresh isolated continuations delivered, including a second one on the same task; 0 `workspace_worktree_requires_project` rows (before: 329 to 428 skipped wakes per wave). - R2 (6 tasks, 3 swaps): a task interrupted three times had its second and third continuations delivered. The same cell found gap 3 (a follow-up already read by the cut-off run), which the last commit fixes. - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds these fixes and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lost-continuation and follow-up-promotion fixes were visibly working in it. Review-round tests: "moves the task off the old writer's workspace and runs the isolated continuation before saved input" (changed expectation: `recorded` instead of `saved_input`; red on the previous head); "promotes saved input behind an exhausted restart hold once a %s continuation replaced the run" (isolated: promoted, red before; `writer_ended`: held). 3 files, 224 tests passed. Conflict note: the last commit was resolved onto current master. It no longer imports `taskHistoryReadsForRun`, which belongs to a separate opening-note change. ## Risks - An isolated projectless continuation now passes admission. It runs in a new workspace that the run prepares, so it never shares a workspace with the cut-off run. - A superseded run's follow-up is promoted without its native finalization committed. The superseded run is fenced (its writes are refused), so there is still one writer. - No migration. No schema change. The normal admission path adds no query. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a298ca7fbc |
fix(runner): prove ownership right before starting a turn in a recovered session (#21260)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner runs native agent sessions (Codex, Claude, and others) and keeps them alive across server swaps. A new server generation recovers the session and continues it. > - Only the generation that owns the run may submit a turn into the provider session. The Runner proves ownership with a fenced checkpoint (controller boot, generation and lease, judged by the database clock). > - On a recovered session, the startup turn and the restart continuation took that checkpoint right after recovery, then prepared the turn's input (attachments, queued input, steered messages), then submitted. > - A generation that lost the run during input preparation still submitted a turn into the shared provider session. > - This pull request repeats the fenced checkpoint right before `turn.start` on those two recovered-session paths. > - The benefit is that a server generation that no longer owns a run cannot add a turn to its provider session. New sessions are unchanged, so nothing is added before their first turn. ## Linked Issues or Issue Description **What happened?** On staging, a Codex run crossed three server swaps. After its turn finished without a result, a turn that no generation could account for appeared in the same provider thread. Every path that submits a turn into an existing session is preceded by the fenced checkpoint, but the two startup paths took it before input preparation, not right before the submit. A generation that lost the run in that window still submitted. **Expected behavior** A generation submits a turn into a recovered session only if it still owns the run at the moment of the submit. **Steps to reproduce** 1. Recover a native session on a new server generation. 2. While it prepares the startup turn's input, fence the generation (another generation takes the lease). 3. Before this change, the turn is still submitted. After it, the checkpoint fails and no turn starts. ## What Changed - `packages/paperclip-runner/src/native-session-runtime.ts`: - The restart continuation calls the fenced `checkpoint` right before `session.startTurn`, after attachments and steered messages are prepared. - The startup turn on a recovered session (`recovered === true`) calls `checkpoint` right before `session.startTurn`. A new session's normal path is unchanged. - `packages/paperclip-runner/src/native-session-runtime.test.ts`: new tests "proves it still owns a recovered session right before starting a turn in it" (startup turn) and "proves it still owns a recovered session right before a restart continuation" at startup and when the lost turn's terminal arrives (`resumeInterruptedTurn`). Turn starts into an existing session, and how each proves ownership right before the submit: - Startup turn on a recovered session: new checkpoint, covered by the first test. - Restart continuation (`startRestartContinuation`), at startup and via `resumeInterruptedTurn`: new checkpoint, covered by the second test (both cases). - Continuation disposition (`requestContinuationDisposition`): already persists the fenced checkpoint immediately before `startTurn`, with only a snapshot read between them and no input preparation. No change. - Compaction turn: started right after the fenced recovery checkpoint with an empty message; there is no input preparation between them (the only possible await is the disposition-replay read, which applies only when a disposition turn was submitted, and then no compaction turn starts). No change. - Runner-only change. No change to `protocol/`, `generated/`, Rust shapes or the ACPX patch, so the provider-pack `contractDigest` and interface revisions are unchanged and older image packs stay compatible. The Runner build and image content for this source change are produced by CI (`runner-artifacts`); nothing is deployed by this PR. ## Verification - Red-to-green: "proves it still owns a recovered session right before starting a turn in it" fails without the startup-turn checkpoint (the turn is submitted after the generation is fenced) and passes with it. Both restart-continuation cases fail with the checkpoint removed from `startRestartContinuation` and pass with it. - Local: `cd packages/paperclip-runner && pnpm exec vitest run src/native-session-runtime.test.ts`: 290 passed. - Local: `node scripts/typecheck-test-programs.mjs packages/paperclip-runner/tsconfig.test.json`: passed (the first CI run failed on the test's attachment prompt returning `undefined`; it now returns a string). - **Not run locally:** one test case that needs the Rust `paperclip-runnerd` binary, and `src/live/runnerd-restart-continuation.test.ts` (10 cases). The Rust binary was not built on this machine. CI must cover them. ## Risks - One extra checkpoint round trip before `turn.start` on a recovered session and on a restart continuation. These are recovery paths; a new session's first turn gets no added wait, so boot and time-to-first-token for new runs are unchanged. - If the checkpoint fails (the generation lost the run), the turn is not submitted. That is the intended behaviour; the owning generation continues the run. - No migration. No schema change. No pack contract change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
db802d1230 |
fix(heartbeat): deliver a queued comment again unless the reading run cites it as handled (#21243)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place
## Linked Issue or Issue Description
Refs #21175
Refs #21203
Refs #21210
**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.
**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.
**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.
## What Changed
- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.
## Verification
- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
- read and cited: no continuation (`...cited: true`).
- never read: one continuation, as before.
- Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
|
||
|
|
04b50e6007 |
fix(native-runtime): a timed-out step records its real cause instead of leaving the run unrecoverable (#21262)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a Runner and a provider session, then send the first turn > - When a startup step fails, the native session executor redacts the error, classifies it and records the failure > - A timed-out step fails with a `DOMException` (`TimeoutError` or `AbortError`). Its `message` property has only a getter > - The redaction assigned to `error.message`. The assignment threw a `TypeError`, so the classifier did not run and the real cause was lost > - The run then fell into a restart-recovery chain that ended in "requires reconciliation" wording for an operator > - This pull request makes the redaction safe for errors with a read-only message > - The benefit is that a timed-out step records its real cause and settles through the normal failure path ## Linked Issues or Issue Description **What happened?** In a local chaos cell with 8 opencode tasks on Daytona sandboxes, four first runs failed before their first turn. A step before the first turn ran under a five-minute `AbortSignal.timeout`. The step took longer than five minutes. The run log shows `TypeError: Cannot set property message of which has only a getter` at the startup failure handler, then `native_finalization_missing`. The run never recorded the timeout. Recovery then restarted the bootstrap, saw "runnerd exited while waiting for run.prepare", and ended with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete` and `native_continuation_requires_reconciliation`. **Expected behavior** A failed startup step records its own cause. The failure handler classifies it and settles the run. It does not throw a second error that hides the first one. **Steps to reproduce** Make any step before the first turn reject with `AbortSignal.timeout(n).reason`. Before this change, `executePaperclipNativeSession` rejects with the `TypeError` above and the coordinator row keeps no failure detail. The new integration test does this with the embedded database. ## What Changed - Add `redactNativeFailureError` in `server/src/services/native-runtime/native-failure-redaction.ts`. It redacts the message and stack in place when it can. When the message is read-only, it returns a new `Error` with the redacted message, the same `name` and the redacted stack. - Use it in the startup failure handler of `executePaperclipNativeSession` in place of the direct assignment. - Add unit tests for a real `TimeoutError`, a read-only `AbortError` that holds a secret, and an ordinary typed error. - Add an integration test: a step before the first turn rejects with a real `TimeoutError`. The run settles through the classifier. `failureDetail.message` is "The operation was aborted due to timeout". The run error code is not `native_continuation_requires_reconciliation`. ## Verification - Red on master: `native-failure-redaction.test.ts` failed 2 of 3 with `TypeError: Cannot set property message of which has only a getter`. The new integration test in `native-startup-retry.integration.test.ts` failed: the call rejected with that `TypeError` and not with the `TimeoutError`. - Green with this change: - `pnpm --filter @paperclipai/server exec vitest run --config vitest.config.ts src/services/native-runtime/native-failure-redaction.test.ts src/services/native-runtime/native-startup-retry.integration.test.ts` - `src/services/native-runtime/native-session-executor.test.ts`: 1431 passed, 7 skipped. - `pnpm verify` for the affected files. ## Risks - Low risk. The change only alters behavior when the error message cannot be assigned. Before, that case threw a `TypeError` and skipped the failure handler. - The replacement error keeps the `name` but is a plain `Error`, so `instanceof DOMException` is false after redaction. No code in the handler checks for `DOMException`. - After this PR, a capture timeout before the first turn still ends blocked. The run records the real cause, but it shows generic wording and waits for a person to press Retry. The reason: a generic timeout has no typed cause, so the classifier records it as `native_session_interrupted` and treats it as ambiguous state. Follow-up PRs give the capture step a typed cause and per-step bounds, and retry a failure that provably happened before any prompt automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
53608bd1d6 |
fix: expose reconciliation wake reason in diagnostics (#18236)
## Thinking Path > - Paperclip projects wake records into safe task diagnostics. > - The projection keeps a small allowlist of stable, non-sensitive reasons. > - `execution_reconciliation_required` is a stable wake reason, but the projection currently converts it to `other`. > - This hides the actionable reason for a skipped wake. > - This pull request preserves that reason in diagnostics. > - The benefit is that operators can diagnose this wake state without exposing raw error or payload data. ## Linked Issues or Issue Description Refs #13532 Related: #16966 handles the separate future wake recovery path. This draft does not add recovery or retry behavior. ## What Changed - Add `execution_reconciliation_required` to the safe wake-diagnostic reason allowlist. - Add focused projection coverage for the known reason and unknown-reason redaction. - Extend the existing diagnostics-route regression matrix. ## Verification - Passed: `pnpm exec vitest run server/src/routes/issues-wake-diagnostics.test.ts --reporter=verbose` — 1 file, 2 tests. - Added endpoint-level embedded-Postgres coverage in the existing route suite. - The full embedded-Postgres route suite did not complete locally: its tests timed out in this isolated checkout. A direct embedded-Postgres start, query, and cleanup probe succeeded. This PR remains draft until hosted CI provides the broader result. ## Risks Low. The change exposes one existing stable reason. Unknown reasons still project to `other`. It does not expose payloads or raw error text, and it does not change wake admission, recovery, or scheduling. ## Model Used OpenAI Codex — `gpt-5.6-terra`; tool-assisted coding and reasoning. Context-window size was not disclosed by the provider runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR is a targeted bug fix, not a new roadmap feature - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked the existing public issue - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change and contains no instance-derived details - [x] I have run the focused local test and it passes - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (not applicable: no public contract text changed) - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: DeadlySilent <3342508+DeadlySilent@users.noreply.github.com> |
||
|
|
8d3dfab52c |
fix(claude-local): fail Opus 5.5 ACP runs early when the bundled Claude Code is too old (#13927)
Thanks @spaceshipmike for the clear report in #13889 and the `CLAUDE_CODE_EXECUTABLE` workaround, and thanks @chen97 for the follow-up. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `claude_local` defaults to the ACP lane, which runs the ACP bridge's bundled Claude Code > - npm installs get Claude Code 2.1.257, because the root pnpm override does not reach them > - Opus 5.5 needs 2.1.280, so each run fails with only "terminal service failure" > - This pull request adds the CLI lane version check to the ACP lane > - The error then names both versions and the fix ## Linked Issues or Issue Description Refs #13889. This PR adds the clear failure the issue asks for. The bridge bump stays with maintainers (#13838 moved the workspace override to 0.3.280). Bridge 0.81.0 is the first stable release on SDK 0.3.280. ## What Changed - `acp.ts`: before launch, compare the bundled `claudeCodeVersion` with `minimumClaudeCliVersionForModel()`. Return `claude_cli_version_incompatible` with the CLI lane fields. - `cli-capabilities.ts`: `readBundledClaudeCodeVersion()` resolves the SDK through the bridge, like `build-provider-pack.mjs`. - Match the `jp.` and `au.` Bedrock prefixes. Opus 5.5 has both profiles. ## Verification Default ACP lane on Bedrock, published `2026.924.0-canary.0`. After = that install with this branch's build. | Case | Before | After | |---|---|---| | `us.anthropic.claude-opus-5-5` | ❌ `acpx_turn_failed`, "ACP agent reported a terminal service failure." (6.9 s) | ✅ clear `claude_cli_version_incompatible` before launch (0 s) | | `jp.anthropic.claude-opus-5-5` (ap-northeast-1) | ❌ same generic failure | ✅ same clear error | | `us.anthropic.claude-opus-5` (no minimum) | ✅ runs | ✅ runs, unchanged | | Opus 5.5 + `CLAUDE_CODE_EXECUTABLE` in agent env (2.1.281) | ✅ runs | ✅ runs, check skipped | | `engine=cli` + Claude Code 2.1.281, `jp.` profile | ✅ runs | ✅ runs, version probe passes | | Source checkout (override, bundled 2.1.280), `us.` Opus 5.5 | ✅ runs | ✅ runs, unchanged | ```text us.anthropic.claude-opus-5-5 requires Claude Code 2.1.280 or newer; the ACP lane runs the bundled Claude Code 2.1.257. Set CLAUDE_CODE_EXECUTABLE in the agent env to a newer Claude Code, or set engine=cli, before retrying. ``` ```text # source reverted, new tests kept Tests 7 failed | 4 passed | 40 skipped (51) # this branch (includes npm-style trees: SDK hoisted, and SDK nested under the bridge) Tests 11 passed | 40 skipped (51) # claude-local project (was 314 passed) Tests 322 passed | 1 skipped (323) # also: claude-local typecheck, check-forbidden-tokens, check-module-boundaries, UI effort tests (97), server adapter tests (34) ``` ## Risks Low risk. The check reads one `package.json`, only for models with a minimum. It skips remote targets, custom `agentCommand`, `CLAUDE_CODE_EXECUTABLE`, and an unreadable SDK. ## Model Used Claude via Claude Code (tool use, code execution). <details><summary>Root cause, raw runs, checklist</summary> - `claude-agent-acp@0.73.0` pins SDK `0.3.257` (`claudeCodeVersion: "2.1.257"`). The override at `pnpm-workspace.yaml:36` changes only the workspace lockfile (see `installation-integrity.ts:96`). - The CLI lane checks at `execute.ts:1263`. The ACP engine keeps provider text out of results (`acpx-engine/execute.ts:4757`). - A server-level `CLAUDE_CODE_EXECUTABLE` is not in the host env allowlists (`acpx-engine/execute.ts:556`, `:588`). The agent env works. This can explain what @chen97 saw. - The `CLAUDE_CODE_EXECUTABLE` test also passes on master. It guards the workaround. ```text claude 2.1.257 --model us.anthropic.claude-opus-5-5 (us-east-1): api_error_status=400 API Error: 400 Claude Code 2.1.257 does not support this model; version 2.1.280 or newer is required. claude 2.1.257 --model jp./au.anthropic.claude-opus-5-5 (ap-northeast-1 / ap-southeast-2): same 400 claude 2.1.281 --model jp.anthropic.claude-opus-5-5 (ap-northeast-1): Hello! before, server env CLAUDE_CODE_EXECUTABLE | us.anthropic.claude-opus-5-5: errorCode=acpx_turn_failed before, agent env CLAUDE_CODE_EXECUTABLE | us.anthropic.claude-opus-5-5: exitCode=0 hello ``` - [x] I used Claude Code to search GitHub for duplicate or related PRs (none references #13889) - [x] Thinking path, `Refs #13889`, tests, and risks included; public references only; descriptive branch name - [x] I used Claude Code to run the tests locally, and they pass - [ ] Model version listed, ROADMAP.md checked, docs updated - [ ] CI green, Greptile 5/5, reviewer comments addressed </details> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
2e795628eb |
fix: strip systemd service-manager env vars from spawned agent processes (#11584)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server runs under systemd on many self-hosted installs. Systemd
gives the server process a readiness socket (`NOTIFY_SOCKET`) and
socket-activation variables (`LISTEN_FDS`, `LISTEN_PID`,
`LISTEN_FDNAMES`).
> - Paperclip spawns agent child processes with `runChildProcess`. That
function merges `process.env` with caller-supplied `opts.env`.
> - The systemd variables leak into the merged environment and reach the
spawned agent process. The agent process can then write to the server's
own readiness socket.
> - Every service start then logs `Got notification message from PID N,
but reception only permitted for main PID M`, and a spawned process gets
access to a channel it has no reason to touch.
> - This pull request strips the four systemd variables from the merged
environment, in the same place that already strips other unwanted
inherited variables.
> - The benefit is a clean systemd log and one less inherited channel
between the server process and any process it spawns.
## Linked Issues or Issue Description
No public GitHub issue exists yet for this specific leak. Following the
bug report template inline:
**What happened?**
`runChildProcess` (`packages/adapter-utils/src/server-utils.ts`) merges
`process.env` with `opts.env` to build the child process environment.
The merge does not strip `NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, or
`LISTEN_FDNAMES`. When the server runs under systemd, those four
variables end up in every spawned agent child process.
**Expected behavior**
A spawned agent child process should not receive the server's own
systemd readiness socket or socket-activation variables. Only the
server's main process should be able to notify systemd of readiness.
**Steps to reproduce**
1. Run the Paperclip server under systemd with `NOTIFY_SOCKET` (and, if
socket-activated, `LISTEN_FDS`/`LISTEN_PID`/`LISTEN_FDNAMES`) set in the
server's environment.
2. Let the server spawn an agent child process through any local adapter
(pi-local, claude-local, codex-local, cursor-local, gemini-local,
opencode-local, copilot-local, acpx-local).
3. Inspect the child process environment (for example by having it print
`process.env`).
4. Observe that the four systemd variables are present in the child, and
that systemd logs a stray notification warning once the child process
happens to write to that socket path.
**Paperclip version or commit**
Current `master` at the time of this change.
**Deployment mode**
Self-hosted server, systemd-managed.
This change is scoped to the environment leak only. A related, larger
piece of work about sandboxing spawned agent processes more generally
(cgroup-level containment) is tracked separately and is out of scope
here.
## What Changed
- In `runChildProcess` (`packages/adapter-utils/src/server-utils.ts`),
strip `NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, and `LISTEN_FDNAMES`
from the environment.
- The strip happens after the `process.env` + `opts.env` merge, in the
same block that already strips the Claude Code nesting-guard variables,
so it also covers callers whose `opts.env` is built as `{
...process.env, ...env }` and would otherwise re-introduce the leak.
- This is a single chokepoint fix. It covers every local adapter that
goes through `runChildProcess`. Remote/sandbox execution targets do not
use `runChildProcess` and are unaffected.
## Verification
New unit test in `packages/adapter-utils/src/server-utils.test.ts`: sets
`NOTIFY_SOCKET`, `LISTEN_FDS`, `LISTEN_PID`, and `LISTEN_FDNAMES` in
both `process.env` and the caller's `opts.env`, spawns a child that
echoes its own `process.env` as JSON, and asserts none of the four
variables are present.
```
$ npx vitest run --project @paperclipai/adapter-utils src/server-utils.test.ts
✓ |@paperclipai/adapter-utils| src/server-utils.test.ts (93 tests)
```
Confirmed the new test fails on unmodified `master` (red) and passes
with this fix (green).
`pnpm --filter @paperclipai/adapter-utils typecheck` is clean.
(No per-package `test` script exists in
`packages/adapter-utils/package.json` or any sibling adapter package.
Tests run through the root vitest `projects` config, so `--project
@paperclipai/adapter-utils` is equivalent to `pnpm --filter
@paperclipai/adapter-utils test`.)
## Risks
Low risk. This removes four specific environment variables from a
spawned child's environment. No other variables are touched. A child
process that legitimately needed one of these four variables for its own
purpose would be the only regression case, and no local adapter has such
a need today.
## Model Used
Claude (Anthropic), model `claude-sonnet-5`, used within a coding-agent
harness with tool use (file edit, test execution, local verification).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes — not
applicable; internal environment-hygiene fix with no user-facing or
operator-facing documentation surface.
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green — `review`
(template/commitperclip gate) passes. Two unrelated jobs (`General tests
(server (2/5))`, `verify`) are currently red; not yet triaged. Not a
description issue; leaving this box unchecked until those two jobs are
green.
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
confirmed via the Greptile Review comment: "Confidence Score: 5/5", no
open P2s or follow-ups listed.
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: dzianisv <dzianisv@gmail.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: commitperclip-bot <305974387+commitperclip-bot@users.noreply.github.com>
|
||
|
|
1c33275ada |
fix(heartbeat): start the new owner after reassignment away from a queued creator run (#21256)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A lead agent often creates a task for itself and then reassigns it
to another agent
> - The creator is busy in its own chat run, so the new task's
assignment run stays queued and does not take the task's execution lock
> - #21030 releases the outgoing assignee's never-started run on
reassignment, but only when that run holds the execution lock
> - Here the run did not hold the lock, so it stayed queued. The new
owner's wake then adopted it as the active run and was deferred behind
it
> - When the queue later withheld the stale run as
`issue_assignee_changed`, nothing promoted the deferred wake, and the
stranded-wake sweep ignores assignment wakes. The new owner never ran
> - This pull request closes all three gaps
> - The benefit is that a reassigned task starts its new owner at once,
and recovery still starts it if a release is missed
## Linked Issues or Issue Description
Refs #21030
**What happened?**
In the paid chat evals `onboarding.first-work` and
`questions-and-approvals.duplicate-answer` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master
|
||
|
|
57840ac6b2 |
fix(projects): say where a person restores an archived project (#21254)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can archive a project for a person, and the person may
later want it back
> - The project update route did not say how archiving works or where a
person restores a project
> - The agent's API search returned the status-card route, which says to
restore from the company Status page with Show archived
> - The agent then told the person to restore the project from the
Status page, which is wrong
> - This pull request describes project archive and restore on the
project update route, with the real page
> - The benefit is that agents send people to the project's own
Configuration page, where Unarchive is
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `projects.archive-project`, the agent archived the
project correctly. Its reply said the person could view or restore it
"from the company Status page using Show archived". That is the
status-card flow. Projects are unarchived on the project's Configuration
page.
**Steps to reproduce**
1. Ask an agent to archive a project and keep its history.
2. The agent searches the API with `archive project update project` and
sets `archivedAt`.
3. Read where the reply says to restore the project.
**Expected behavior**
The reply links `/<prefix>/projects/<projectId>/configuration` and says
to use Unarchive there.
## What Changed
- `server/src/routes/openapi.ts`: `PATCH /api/projects/{id}` now says
that archiving sets `archivedAt`, hides the project from the sidebar and
selectors, and keeps the project, its tasks and their history. Restoring
sets `archivedAt` to null. A person unarchives on
`/<prefix>/projects/<projectId>/configuration` with the Unarchive
button, not on the company Status page.
- `server/src/__tests__/project-archive-restore-guidance.test.ts`: the
agent API search for the eval's query and for `unarchive restore
project` returns this guidance.
## Verification
- `cd server && npx vitest run
src/__tests__/project-archive-restore-guidance.test.ts` passes (2
tests). The tests fail on master.
## Risks
- Low risk. Only an agent-facing route description changes.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
444dc18d98 |
fix(routines): keep webhook secrets out of chat and link the reveal page (#21253)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can add a webhook trigger to a routine, and the API
response includes the webhook URL and its secret
> - Paperclip redacts secrets from chat output, so a secret that an
agent writes in a reply reaches the person as `[REDACTED]`
> - An agent pasted the bearer token in chat, the person saw
`Authorization: Bearer [REDACTED]`, and had no way to get the key
> - The person can reveal a key once on the routine's Triggers page, but
the agent did not know that page
> - This pull request tells the agent never to write the secret in chat
and to link the Triggers page instead
> - The benefit is that secrets stay out of chat and the person still
gets a working key
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `routines.webhook-trigger`, the agent created a
bearer webhook trigger. Its reasoning said the token "is sensitive; but
user needs it. I'll provide it." Output redaction removed the token, so
the reply showed `Authorization: Bearer [REDACTED]` and gave no secure
way to get the key.
**Steps to reproduce**
1. Ask an agent to add a webhook to a routine so a test service can
trigger it.
2. The agent calls `POST /api/routines/{id}/triggers` and reads the
secret in the response.
3. Read the agent's reply.
**Expected behavior**
The agent shares the webhook URL and signing mode, does not write the
secret, and links the routine's Triggers page, where the person reveals
the key once.
## What Changed
- `server/src/routes/openapi.ts`: `POST /api/routines/{id}/triggers` and
`POST /api/routine-triggers/{id}/rotate-secret` now describe the secret
handling. For a signed or bearer trigger, the URL and signing mode can
be shared. With `signingMode: none` the URL alone starts the routine, so
it is treated as a credential too. The secret value, or a header that
contains it, is never written in a chat reply, comment, document or
task. The agent links `/<prefix>/routines/<routineId>/triggers`. For a
trigger still awaiting setup, the link is
`/<prefix>/routines/<routineId>/triggers?triggerSetup=<triggerId>` and
the person chooses Generate new key. Otherwise the person chooses Edit
webhook, then Replace key. These are the same controls the human webhook
setup uses.
- `server/src/__tests__/routine-webhook-secret-guidance.test.ts`: the
agent API search for both operations returns this guidance.
## Verification
- `cd server && npx vitest run
src/__tests__/routine-webhook-secret-guidance.test.ts` passes (2 tests).
The tests fail on master because the operations have no description. The
tests also check the unsigned-URL rule.
## Risks
- Low risk. Only agent-facing route descriptions change. Server-side
redaction is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
500bc694b5 |
fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
52817dff55 |
fix(runner): name missing tool fields and what the call included (#21252)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools, and the runner validates each
tool call against its input schema
> - When a call is invalid, the runner returns an "Invalid tool input"
error to the model
> - For a call that left out required fields, that error was a list of
JSON Schema messages such as `/ must have required property 'kickoff'`
> - A model sent `paperclip_create_case` with only
`workflow_document_text` for about 5 minutes, believed it had sent every
field, and never replied to the person
> - This pull request makes the error name the missing fields, the
fields the call did include, and a short example of the missing fields
> - The benefit is that a model can see its own mistake and correct the
call on the next attempt
## Linked Issues or Issue Description
**What happened?**
In the paid chat eval `cases.create-case` (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`), the agent called
`paperclip_create_case` about 20 times with only
`workflow_document_text`. Each call returned `Invalid tool input: / must
have required property 'idempotencyKey'; / must have required property
'caseType'; ...`. The model's reasoning said that the other fields were
being stripped. It never created the case and never replied.
**Steps to reproduce**
1. Validate `{ "workflow_document_text": "..." }` against the
`create_case` input schema.
2. Read `toolInputError(validator)`.
**Expected behavior**
The error says which required fields are missing, says that the call
included only `workflow_document_text`, and shows a short example of the
missing fields.
## What Changed
- `packages/paperclip-runner/src/drivers/tool-input-error.ts`: for
missing top-level required fields, the error now says `missing required
fields: idempotencyKey, caseType, title, kickoff.`, `This call included
only: workflow_document_text.`, and a placeholder example built from the
schema, such as
`{"idempotencyKey":"<idempotencyKey>",...,"kickoff":{"toAgentId":"<toAgentId>","phase":"<phase>","instructions":"<instructions>"}}`.
Other validation errors follow after `Also:`. Field-description guidance
is unchanged.
- `packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts` and
`runner-tool-bridge.ts`: pass the call arguments to the error builder.
Only argument names are read; values are never echoed.
- `packages/paperclip-runner/src/drivers/tool-input-error.test.ts`: a
test on the real `create_case` schema and a test that other errors still
follow the missing fields.
- The `create_case` schema is not flattened. The model also left out the
flat fields (`idempotencyKey`, `caseType`, `title`), so nesting was not
the cause, and `kickoff` is shared with `handoff_case`.
## Verification
- `cd packages/paperclip-runner && npx vitest run
src/drivers/tool-input-error.test.ts
src/drivers/opencode/mcp-bridge.test.ts
src/drivers/opencode/opencode-server-driver.test.ts` passes (115 tests).
The two new tests fail on master.
## Risks
- Low risk. Only the error text for invalid calls changes. Validation
and dispatch are unchanged. The example is bounded to 600 characters and
the field lists to 12 names.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
bc2cdeaa7c |
fix(evals): grade full cohort 9 chat cases on saved state and served links (#21249)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks that agents in chat do the right thing. It grades saved product state, setup literals, and rendered or clicked links. > - Full cohort 9 ran OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017` on master |