mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
ec2c2dc2a62f286df719b7e96c6802513d2a5ef0
3180
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Nicky LeachandClaude Opus 5.5
|
026dc6d135 |
fix(native): release a session-resume owner when the resumed run fails before its provider claim (#19371)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row. Its lease owner tells the
control plane that an execution may still be live.
> - Pausing an agent waits for `agentExecutionsHaveStopped`. That check
refuses to finish while any run of the agent has a lease owner, unless
the run was cancelled and has stop proof.
> - A same-run session resume claims that lease before it dispatches the
run. If the run then fails before the native executor takes its own
provider claim, nothing releases the resume owner.
> - The leaked owner blocks pause and resume for good, and the task
keeps looking like it has a live recovery path.
> - This pull request settles those rows at the source, and it adds a
stop-proof-gated release for rows that already leaked.
> - The benefit is that agents can pause and resume after an ordinary
resume failure, without database edits, and the cancelled-run stop-proof
rule stays as it is.
## Linked Issues or Issue Description
**What happened?**
An agent stayed in `pausing` with `hostComplete=false` and climbing
attempts. Its runs had `native_run_finalizations` rows with `phase:
observed`, `recovery_state: resuming_session`, a `lease_owner` like
`default:resume:<uuid>`, and no `result_id`. Each heartbeat run had
ended `failed`: either `setup_failed` (`Failed to acquire lease for
environment … Sandbox creation has failed`,
`native_remote_recovery_lease_mismatch`) or `adapter_failed`
(`runner_state_identity_mismatch: …`). The lease expired about 20
minutes after the run finished. Nothing settled the row. The only fix
was to set `lease_owner` to null by hand.
**Expected behavior**
When a resumed run fails before any provider turn starts, its
finalization row goes to `terminal_failure` and its lease owner is
released, as the native executor does for failures it sees. A row that
leaked before this fix is released in the background once its lease
expires and the run has stop proof. Pause then completes.
**Steps to reproduce**
1. Leave a native run in `retryable_failure` with a persisted
checkpoint, on a task that already has an active recovery incident.
2. Let session resume claim it (`claimNativeSessionResumptions` sets
`phase: observed` and a `…:resume:…` owner).
3. Make the resumed run fail before the executor claims it, for example
when the environment driver cannot create the sandbox.
4. The heartbeat failure path marks the run `failed`. The issue release
path does not settle the row, because it settles only when it opens a
new incident. The owner stays set, and `agentExecutionsHaveStopped`
returns false indefinitely.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
06c0566a08 |
fix: make task Retry work after a refused automatic attempt (#19466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry stores a durable request before it starts a new run. > - An automatic attempt can stop at the execution gate before a provider starts. > - That cancelled attempt becomes the source of the next Retry click. > - Admission rejected that source, or chose an older run that did not match the stored Retry authority. > - This pull request verifies the refused attempt and keeps it as the exact authority source. > - The user can start one new turn after the older execution holds pass their existing checks. ## Linked Issues or Issue Description **What happened?** Retry can remain deferred when the latest run was cancelled with `execution_reconciliation_required` before dispatch. An older failed preparation has an execution recovery hold. The Retry receipt names the cancelled child, but admission only accepts a failed preparation as an extra source. It can also choose the older run as `previousRunId`, which fails the exact source check at dispatch. **Expected behavior** Retry starts one new turn when the latest attempt has a verified pre-dispatch refusal and every older execution hold is ready. Provider activity, retained execution resources, missing receipts, or a live earlier process keep the hold. **Steps to reproduce** 1. Seed a failed native preparation and its recovery hold. 2. Let a later automatic attempt be cancelled by the stale queued execution gate before dispatch. 3. Click Retry against that latest cancelled attempt. 4. Before this change, continuation admission returns no receipt. After this change, one successor starts with authority bound to the clicked attempt. **Paperclip version or commit** Reproduced on `2f35eb1080`. Rebased onto `6cafcaf89b` for submission. **Deployment mode** Local source checkout with embedded PostgreSQL and the native Runner on macOS arm64. Refs #15383 and #16939. Those changes cover legacy conversation cancellation and queue promotion. This change covers the exact source of a task Retry receipt after a gate refused its automatic child. ## What Changed - Reuse the existing database-backed pre-dispatch refusal proof when admitting the exact Retry source. - Keep the clicked source as `previousRunId`. Continue to check every older execution hold. - Add database regressions for safe refusal, unsafe evidence, concurrent dispatch, and duplicate clicks. - Add an opt-in acceptance test that drives the authenticated Retry route through a real Codex turn and task finalization. - Document the admission contract. ## Verification - TDD: the new database regression returned `null` on the original code. It passes with the repair. - Final head `c43a84b1bf`: 487 tests pass across task Retry, explicit native continuation, and execution continuation. Server TypeScript and module boundary checks pass. - Live acceptance: `PAPERCLIP_LIVE_REFUSED_RETRY=true PAPERCLIP_RUNNER_BINARY=<current-source-runnerd> pnpm --dir server exec vitest run --config vitest.config.ts src/services/native-runtime/refused-task-retry.live.test.ts`. - The real HTTP request returned 202 on the final rebased head `c43a84b1bf`. Codex `gpt-6.1-sol` with high effort produced the requested file. The task became done, both task locks cleared, and finalization committed. A repeated click reused the same receipt and did not create another run. The source file stayed unchanged. - Server TypeScript check and module boundary check. - `pnpm verify --explain` returned 2 because the execution contract doc requires broad CI coverage. The required CI checks supply that coverage. ## Risks This changes execution admission. The existing proof requires the stored refusal receipt and no provider work or retained execution resources. Earlier recovery holds, process stop checks, decisions, actor authority, and task ownership still apply. No schema change or migration. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The host does not expose an exact author model ID or context window. The live acceptance provider was `gpt-6.1-sol` with high reasoning effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — all GitHub Actions checks pass on `c43a84b1bf`; Paperclip Review completed at 5/5 with no findings (neutral check conclusion). - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — reviewed `c43a84b1bf` with no actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
6ec4180ea1 |
fix: preserve ordinary diagnostic text during environment secret redaction (#19461)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run diagnostics help operators find the cause of a failed agent run. > - The failure reporter treated unknown environment settings as credentials. > - It replaced every use of those values in diagnostic text. > - A flag set to `1`, `2`, or `safe` removed parts of dates, model names, versions, and normal words. > - This pull request preserves explicit ordinary settings and keeps unknown values private. > - Operators can read the failure details while real credentials stay masked. ## Linked Issues or Issue Description **What happened?** Run-failure contexts contained redaction markers inside timestamps, model names, driver versions, and the word "safely". Provider failure diagnostics used the same broad environment-value rule. **Expected behavior** Ordinary flags, counts, paths, regions, and mode labels leave diagnostic text readable. API keys, URL passwords, and declared secret bindings remain masked. **Steps to reproduce** 1. Set ordinary environment settings to `1`, `2`, and `safe`. 2. Report a failed native run with a timestamp, `gpt-6.1-sol`, `phase6-v2`, and "safely" in its details. 3. Inspect the run failure report. Before this fix, unrelated fragments are replaced. Related work: #13668 addresses a separate model-ID false positive. #11311 and #15524 address credential capture at other boundaries. This change keeps the existing command scanner and registered-secret replacement. ## What Changed - Add a shared environment selector in adapter-utils. Keep unknown values private. Exempt explicit ordinary process settings, typed flags and counts, mode labels, and key selectors. - Use the same selector for run-failure reports and inherited provider failure environments. Keep one policy for both consumers. - Keep configured provider bindings private because that API does not carry separate secret-reference metadata. - Preserve short declared secrets, unknown opaque values, encoded credentials, and inherited Bedrock tokens. Add regressions for ordinary `SORT_KEY` values and both Bedrock consumers. - Add screenshot-shaped regressions, real database reporter coverage, and credential tests. - Document the policy in `doc/observability.md`. The optional monitoring gate stays in place. ## Verification Linux, Node v25.9.0, pnpm 9.15.4. The published tree matches the tested local tree `cb236f072cca195e4c2cfb4034c7cc06ea7adbc5`. - Before the fix, four diagnostic regressions failed. The output reproduced the damaged timestamp, model, version, and word. - Review regressions: six tests failed on the first revision. They proved the missed Bedrock token, opaque inherited-value exposure, and key-selector false positive. All pass after the correction. - 555 focused tests pass on the final source tree. - `cd server && pnpm exec vitest run src/services/__tests__/run-failure-diagnostics.test.ts src/__tests__/redaction.test.ts src/__tests__/redact-sensitive.test.ts src/__tests__/run-secret-redaction.test.ts`: 206 passed. - `cd server && pnpm exec vitest run src/services/__tests__/run-failure-report.test.ts`: 289 passed, with real embedded PostgreSQL. - `pnpm exec vitest run packages/adapter-utils/src/command-redaction.test.ts packages/adapter-utils/src/acpx-engine/terminal-session-failure.test.ts`: 60 passed. - `pnpm --filter @paperclipai/adapter-utils typecheck`: passed. - `git diff --check`: passed. - The first reporter attempt could not load the unbuilt plugin SDK. It passed after `pnpm --filter @paperclipai/plugin-sdk ensure-build-deps`. - The filtered install omitted the CLI tsx loader, so `pnpm verify --explain` could not start. The equivalent verifier entry points ran with the installed server loader and returned exit 2: broader CI coverage is required for the observability document. - Full local build/typecheck and browser suites were not duplicated. [Final-revision CI](https://github.com/paperclipai/paperclip/actions/runs/42231412547) passed on `013960a4c6fa6279e1f28b0f6af5f036d7cad5a7`, attempt 1. This includes typecheck, build, general/server/chat tests, serialized suites, Runner checks, eight E2E shards, and the canary dry run. - [Greptile review](https://github.com/paperclipai/paperclip/pull/19461#issuecomment-6686592783) scored this head 5/5 with zero new findings. All four inline threads are resolved. The security rescan and Snyk status passed. The PR has no merge conflicts. ## Risks - Ordinary-setting exemptions need a known name or a name/value type rule. Unknown opaque values remain private. Declared secret bindings and recognizable credentials override these exemptions. - A real short secret still masks matching fragments. Removing that protection would expose the secret. - Existing stored redaction markers cannot restore their original text. This fix applies when new diagnostics are produced. - No migration, dependency, workflow, authorization, or default monitoring changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. This change is a focused bug fix. ## Model Used OpenAI GPT-6, through Codex, with shell execution and file tools. The host does not expose the exact runtime variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e680fa6814 |
fix(native): retry startup interrupted by a closed runtime API listener (#19462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs prepare a remote workspace, then start a runner and a provider session > - A graceful server shutdown closes the HTTP listener before it drains runs > - A run that is still preparing then asks for the local runtime API origin, and the lookup fails with `local_runtime_api_listener_unavailable` > - Recovery did not know this code. It treated the failure as an ambiguous provider state and blocked the task behind a board-owned action, although no runner or provider process existed > - This pull request classifies the closed listener as a definite pre-session failure > - The benefit is that a deploy no longer strands a preparing task. The same run gets a bounded bootstrap retry that the next server process claims ## Linked Issues or Issue Description **What happened** A native Codex task on a remote sandbox spent about ten minutes in workspace preparation. A deploy started a graceful shutdown during that time. When preparation finished, the executor called `resolveLocalRuntimeApiUrl()` to build the tool authority. The listener was already closed, so it threw `local_runtime_api_listener_unavailable`. No runner backend or provider session had started. The failure classifier returned the generic `native_session_interrupted` code. `nativeProviderRecoveryEvidence` then returned `ambiguous_state`. The run became a terminal failure, the task became blocked, and the recovery action told the board to resolve an ambiguous session and to not open a replacement session. The same signature occurred on three other runs during an earlier deploy. **Expected behavior** A failure that occurs before any runner or provider process exists is safe to retry on the same run. The task must stay in progress with a scheduled, bounded bootstrap retry. If a provider event or checkpoint exists, recovery must stay operator-owned. **Steps to reproduce** Start a native run. Close the server's HTTP listener before the executor creates the runner backend. Observe a `terminal_failure` with `recoveryMode: "ambiguous_state"` and a blocked task. The new integration test reproduces this with the real `resolveLocalRuntimeApiUrl()` and a real Postgres database. **Environment** Server native runtime, remote sandbox target, `codex_app_server` driver. Related recovery work: #19249, #19304, #19305. ## What Changed - `nativeSessionFailureSourceCode` returns a new `local_runtime_api_listener_unavailable` source code for this error. - `nativeProviderRecoveryEvidence` treats that code as definitely pre-session. The existing guard stays: the run must have no checkpoint and no durable provider event. - The chat failed-run retry accepts the code in its list of ordinary pre-session failures, so an exhausted startup retry can still be retried from chat. - New integration tests in `native-startup-retry.integration.test.ts` cover the retry path and the operator-owned path when a provider event exists. ## Verification - Red before the fix: the new test failed with `phase: "terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable: false` and a board-owned recovery. This matches the failed run on the deployed instance. - Green after the fix: `cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` passes 26 of 26 tests. The retry test asserts `retryable_failure`, `bootstrap_retry`, backoff timing, an agent-owned `resume_native_run` wake, and a claim through `claimNativeSessionResumptions`. - Safety case: when a durable `turn.started` event exists, the same error stays `terminal_failure` with `ambiguous_state` and a blocked task. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "listener|classif|pre-session"` passes. - Not run: a live deploy that interrupts a preparing sandbox run. The test uses the real listener lookup and the real recovery tables, but not a real sandbox. ## Risks - Low risk. The change only affects runs that fail with this exact error and have no checkpoint and no provider event. - The retry uses the existing bootstrap budget of three attempts with backoff. Repeated failures still stop and hand the task to the board. - The GitHub callback bridge that started in the sandbox before the failure is a per-attempt helper. The existing cleanup path releases it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use, code execution and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
6cafcaf89b |
fix(workspaces): recover unpublished archived Git uploads (#19459)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace imports preserve existing files before the server changes workspace authority. > - Archived Git imports reconstruct saved files and Git state in a private repository. > - A failed upload can leave a pending snapshot journal with no object in storage. > - A later reconstruction can have different Git administrative bytes for the same saved logical source. > - This pull request lets that import record a new unpublished capture after it proves that prior objects are absent. > - The benefit is safe retry through the original import journal while source checks and atomic publication remain in place. ## Linked Issues or Issue Description Refs #19396. Refs #19409. A snapshot upload can fail before object persistence. The caller then removes its private archive. A retry reconstructs the same saved Git workspace. If its administrative bytes differ, the snapshot owner rejects the retry with `remote_workspace_snapshot_operation_content_changed`. The pending journal remains blocked even though no snapshot object or remote authority was published. ## What Changed - Add an optional unpublished recapture capability to snapshot storage. Bind it to the original source operation. Existing callers keep the exact manifest check. - Enable this capability only for archived Git reconstruction. The migration owner keeps the source recipe, task binding, ownership, and publication checks. - Verify every prior object before new capture admission. Unknown or corrupt objects stop the retry. Reuse an available prior object, including an object that appears while the replacement upload runs. - Compare the observed upload journal under the snapshot row lock before appending an attempt. Keep network I/O outside that lock. Preserve all prior attempt metadata. - Keep the first atomic publication as the winner. A losing uploader verifies and returns that saved reference. - Add real PostgreSQL, filesystem, and Git regression tests. Document the retry behavior. ## Verification - Baseline `70d23612dd9176cae8ee6b2b6dcf2f6eade9a42a`: the new real Git fixture fails with `remote_workspace_snapshot_operation_content_changed`. It fails the first upload before persistence, removes the old source cache, and advances unrelated remote refs. The saved task commit and restore point stay fixed. - Final frozen source: 20 tests pass in a fresh process. This includes the real Git regression and all snapshot storage cases. Tests cover prior object recovery, corrupt and unknown objects, normal caller mismatch rejection, operation binding, stale journal admission, and late upload publication in both orders. - Reproduce the final focused check from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/remote-workspace-snapshot-storage.test.ts src/services/remote-workspace-migration.test.ts -t 'failed-upload-retry|snapshot storage' --reporter=dot ``` - An earlier six-suite run had 201 passing cases and one failing late-object case. That run started before the final prior-object recheck was added. The failing case passes in the fresh final run above. This earlier run is not a clean full-family result for the final source. - That run skipped 16 lifecycle cases because `PAPERCLIP_REMOTE_WORKSPACE_LIFECYCLE_TEST_DATABASE_URL` was unset. Migration, storage, reservation, and concurrency tests used real disposable embedded PostgreSQL. - `git diff --check` passes. Full repository typecheck passed before rebasing. Full test, build, and current-head CI results are pending. ## Risks - The optional capability permits different physical captures. The migration owner must prove that the logical source and current ownership are unchanged. Ordinary mutable workspace uploads keep the previous mismatch guard. - An old upload can complete after the last object check. The first authenticated atomic publication wins. A late upload cannot replace its object or snapshot reference. Every candidate remains in the retention journal. - Unknown storage results and journal changes fail closed. They can require another canonical retry. This change adds no retry loop and changes no database schema. ## Model Used OpenAI Codex, GPT-6 family. The worker used reasoning, tool use, code editing, and local test execution. The worker runtime did not expose an exact model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2a6aae3963 |
perf(workspaces): overlap bounded snapshot file hashing (#19458)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server saves portable workspace archives before it moves file authority to object storage. > - Each capture checks and hashes every eligible file before and after archive creation. > - Those stable reads run in series and leave file I/O idle on workspaces with many files. > - This pull request overlaps at most four stable reads across the whole directory walk. > - It preserves entry order, source checks, Git checks, and archive bytes. > - The benefit is shorter archive capture with bounded extra I/O concurrency. ## Linked Issues or Issue Description **What existing behavior does this improve?** Portable remote workspace archive capture in the server. Related: #19396 preserves saved-upload recovery. #17132 applies a similar technique to the separate Runner file-diff snapshot path; this PR changes the server archive owner only. **Current behavior** The archive walker waits for each file's stable read and hash before visiting the next file. A local packed-Git fixture with 6,000 files and 52.4 MB of contents took a median 4.640 seconds in the final two serial samples. **Proposed behavior** Keep up to four stable reads active across all nested directories. Reserve each entry's position before reading it. Await all reads before returning success, failure, or cancellation. The final two treatment samples took a median 3.439 seconds, a 25.9% reduction in capture time. **Reason and benefit** Captures run twice during a fresh workspace migration. Bounded I/O overlap reduces time spent in this owner without changing the archive contract. This measurement does not establish an end-to-end migration speedup. **Breaking changes** None intended. The file format, manifest order, source validation, Git validation, API, and schema stay the same. ## What Changed - Share one four-read limit across the recursive walk. Preserve deterministic entry positions when reads finish out of order. - Drain active reads on every exit before callers can remove scratch files or return control to a source writer. - Check cancellation before creating a file read stream. - Exercise real file handles across nested directories, out-of-order completion, source replacement, cancellation, and serialized archive capture. ## Verification - Focused archive, archive-environment, and snapshot-storage suites: 113 tests passed. The first sandbox run could not bind local test sockets; the same suites passed with local listener access. - Full workspace typecheck passed before rebasing. The rebase changed neither touched file. - Full local `pnpm test:run` stopped at an unchanged chat integration test: `resolves ambiguous provider effects explicitly and never lets terminal failures starve reconciliation`, with `socket hang up`. The same case passed once in isolation on the rebased head. Its original cause remains unknown; this is not a full-suite pass. - Full `pnpm build` passed on the rebased head. Current-head CI is pending. - Benchmark: actual owning sources, Node 26.4 on macOS arm64, one importer, the same packed-Git fixture. All 12 trial archives had identical manifest digest, SHA-256, and 72,908,826 bytes. Final paired medians were 4.640 seconds serial and 3.439 seconds with four reads. Total process plus Git CPU increased about 12%. - An initial pair was slower with concurrency (23.352 seconds versus 11.228 seconds); the cause remains unknown. Later paired reductions ranged from 24% to 37%. The benchmark host was shared. Linux Node 24 and eight-importer throughput are not yet qualified. - No user commands, configuration, or document contracts changed. Code comments explain ordering and cleanup. ## Risks - Four simultaneous reads can increase I/O and CPU contention when several captures run together. The bound is per capture. Hosted throughput needs measurement after the ordinary release. - A shared-host local benchmark has limited predictive value. This PR claims a measured capture improvement on that fixture only. - Rollback is a code revert. No migration or stored-data rewrite is required. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated review. The runtime did not expose an exact deployment variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4696ff4990 |
fix: reserve agent In Progress display for active runs (#19453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task status must tell operators whether an agent is working or waiting. > - A task can retain its in-progress lifecycle status after its run stops. > - The shared read projection only handled scheduled monitoring waits. > - The UI also ignored other projected statuses for in-progress tasks. > - This pull request requires an active task run for the In Progress display. > - Operators can see waits without changing execution or recovery authority. ## Linked Issues or Issue Description **What happened?** An agent task still showed In Progress after its run ended with an access failure and left a pending connection request. Stale execution locks and saved lifecycle state could show the same false activity. **Expected behavior** An agent task shows In Progress only with a task-linked queued or running run. Otherwise, it shows Blocked for recovery, In Review for a pending response or review, Monitoring for a scheduled check, or Todo for remaining work. Human ownership keeps its selected status. **Steps to reproduce** 1. Create an agent-owned task with an in-progress lifecycle status. 2. End its run and retain a pending human request. 3. Read the task, its summary, and search results. Before this change, they still show In Progress. ## What Changed - Derive visible activity from company-scoped, task-linked runs in the existing shared SQL projection. - Project inactive tasks to existing waiting or todo states. Preserve lifecycle status, locks, admission, and recovery. - Make the shared UI resolver use the projection while preserving explicit and optimistic status changes. - Cover stopped runs, stale locks, human waits, company isolation, monitoring, filters, counts, search, and UI labels. - Document the presentation rule in the product and execution contracts. ## Verification - Red before the server fix: 7 new regression cases failed because they returned in_progress. The 10 existing cases passed. - Green after the server fix: 20 cases passed in issue-display-status.test.ts, using embedded Postgres and real task APIs/search. Seven nearby active-question cases also passed. - Review regression: dismissed requests, historical questions, and returned review work each failed before the follow-up fix. All three now pass using the existing active-interaction rule and pending review state. - Red before the UI fix: 4 label cases failed. Five additional safety cases caught stale projections hiding explicit transitions. - Green after the UI fix: 43 tests passed across task-status-presentation, issue-filters, and issue-detail-subissues. - pnpm -r typecheck passed. All UI token gates and synchronization checks passed. - pnpm build passed. Server typecheck and build passed again after the review fixes. - Full CI passed on |
||
|
DottaandPaperclip
|
70d23612dd |
fix: offer Interrupt when queued-message steering is unavailable (#19450)
## Thinking Path > - Paperclip lets people manage work performed by AI agents. > - Task chat saves follow-up messages while an agent runs. > - A native runner can accept Steer only after an active provider turn exists. > - During preparation, or with an unsupported provider, the queue showed a disabled Steer button and no delivery action. > - The server also rejected Interrupt for ordinary native queues. > - This change offers an explicit Interrupt fallback and uses the existing cancellation and admission path. > - People can send their saved messages while Stop confirmation, cleanup, and execution gates remain enforced. ## Linked Issues or Issue Description Related: #15865 covers mixed legacy queues. #15030 covers steering acknowledgements. This change covers native queues without available steering. **What happened?** A queued message had a disabled Steer button while the native runner was still preparing. Clicking it did nothing. The server returned HTTP 409 for an explicit Interrupt request against the same native queue. **Expected behavior** Offer Steer when the provider can accept same-turn input. Otherwise offer Interrupt, labeled with its actual effect. Preserve saved messages until the previous execution stops and normal admission accepts the successor. **Steps to reproduce** 1. Start a task with the Paperclip Runner adapter. 2. Submit a follow-up while the run is preparing, before its provider turn opens. 3. Observe the disabled Steer button above the composer. 4. With this change, click Interrupt. The existing Stop path cancels the previous execution and submits the saved queue through normal admission. **Paperclip version or commit** Reproduced locally on master at |
||
|
DottaandPaperclip
|
275b334fc5 |
fix(workspaces): skip redundant repository admission snapshots (#19444)
## Thinking Path
> - Paperclip manages AI agents and their execution state.
> - Remote workspace admission must preserve files before execution
starts.
> - Admission saved the entire workspace once per registered repository
on every run.
> - Existing snapshot receipts can prove that validated repositories are
already saved.
> - This change uses those receipts and saves new repositories once per
admission batch.
> - It also preserves fresh configuration metadata and reports actual
workspace reuse.
## Linked Issues or Issue Description
**What happened**
Remote follow-up runs uploaded an unchanged workspace for each
registered repository before starting the provider. Configuration
persistence also overwrote new evaluation metadata, and freshness logs
could claim replacement when the same workspace was reused.
**Expected behavior**
Validate existing repositories without uploads when a committed snapshot
covers their descriptors. Save acquired repositories before publishing
ready receipts. Retain current evaluation metadata and report the actual
workspace outcome.
**Steps to reproduce**
Use a remote workspace with two registered repositories. Prepare it,
then admit a follow-up run. Observe repository snapshot operations
despite unchanged repositories. The regression tests exercise this with
real Postgres, Git, and separate sandbox directories.
**Environment**
Server remote workspace admission at commit
|
||
|
DottaandPaperclip
|
c0214826b3 |
fix(workspaces): preserve pending repository requests during import (#19439)
## Thinking Path > - Paperclip manages agent work and its saved files. > - An ephemeral web server needs remote workspace authority before the old volume can detach. > - A task can have a repository request that has never prepared a checkout. > - The offline importer currently treats that pending request as unfinished file publication. > - This blocks migration even when the target does not exist and there is no saved commit, branch or failure. > - This pull request preserves that request in the database and imports the existing files separately. > - The next authorized sandbox preparation can acquire the repository through its existing owner. ## Linked Issues or Issue Description **What happened?** On Core `253bf071f0a496554b61fff897ad3f11afaf8766`, offline migration of a volume-backed workspace fails with `repository_preparation_unresolved` for an untouched pending repository request. The requested directory has never existed. The request has no pinned commit, branch or failure. **Expected behavior** Preserve the pending request and all existing files. Do not label the absent checkout as part of a saved archive. Keep partial publication and active preparation blocked. **Steps to reproduce** Create a local task workspace. Add an unmaterialized pending repository request. Run the canonical offline importer after stopping writers. The unchanged importer rejects the request before preserving the workspace. Related migration work: #19429. A GitHub search found no duplicate pending-repository import fix or matching public issue. ## What Changed - Admit an untouched pending request only at the existing offline writer fence and after proving its target is absent. - Keep the database request unchanged and exclude it from materialized snapshot descriptors. - Recheck file absence and source identity before publication. Reject a request that becomes ready during capture. - Cover cold filesystem loss, real sandbox-side Git acquisition, and eleven blocked states. Document the preservation contract. ## Verification - Red: the new real PostgreSQL/filesystem regression fails on the unchanged importer with `repository_preparation_unresolved`. - Green: both affected migration/lifecycle suites pass, **236 tests**. The cold test deletes the predecessor directory, restores saved bytes, and acquires the pending repository through the full admission owner, checking ready state and the pinned commit. Real Git runs against a local HTTP origin through an exact-URL redirect at the sandbox transport boundary. It does not claim live Daytona or GitHub qualification. - Full workspace typecheck and build pass at the final head. Both affected suites pass **236 tests**, including the full admission owner and eleven blocked states. All final-head CI checks pass: **54 success, 2 skipped, 1 neutral**. Greptile is **5/5**, with no unresolved review threads. - The full local test command failed in `recovers lost inline and check responses without duplicate GitHub publications` in the chat integration suite. That exact case passes in a fresh isolated invocation. Its cause remains unclassified; this is not a full-local-suite pass claim. The original failure is retained alongside the isolated result. - No candidate code has been applied to the serving instance. ## Risks - The exception applies only to offline migration and a pending request with no target, commit, branch or failure. Unreadable paths, symlinks, partial directories and active/repair states retain their blockers. - Requests retain normal repository authorization and acquisition behavior. This change neither fetches repositories during migration nor marks requests ready. - A changed materialized source aborts publication and leaves original files intact. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool use. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
21d2a577bb |
feat: show and change routine responsible users (#19128)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Routines create tasks for repeated work. > - Each routine has a responsible person. > - The Overview did not show this person or permit a change. > - People need to see who will be responsible for automatic tasks. > - This pull request adds a small name picker beside Default agent. > - Authorized people can adopt a routine or transfer it to an active company member. ## Linked Issues or Issue Description **What existing behavior does this improve?** The routine Overview and the routine update API. **Subsystem affected** UI, server, and shared contracts. **Current behavior** The routine stores a responsible user. The Overview does not show that person. The update API cannot change that person. **Proposed behavior** Show Responsible user beside Default agent. Click the name to search active company members. Save only the new person and the current revision ID. **Reason and benefit** People can see who owns future automatic tasks. An authorized person can adopt a routine without making a new routine. **Breaking changes** The detail response adds an optional actor-specific field. The update API adds an optional responsible user ID. Existing requests keep their current behavior. Related: #17368 covers local Board ownership migration. This change adds the routine control and explicit transfer. ## What Changed - Show a quiet, searchable person picker beside Default agent. - Explain that automatic tasks use this person. Manual runs use the person who starts them. - Permit human operators with `tasks:assign`, instance administrators, and local operators to transfer responsibility. - Check active company membership inside the update transaction. - Save a revision and record the old and new person in the activity log. - Keep existing task responsibility and ownership during revision restore. - Show errors for failed saves and stale revisions. - Add tests for permissions, company boundaries, transfers, inheritance, and picker states. - Keep the picker disabled until saved data is refreshed. Refresh the activity list after a transfer. - Capture the actual previous person under the update row lock for the activity record. - Use shared query error handling and retain cached people during a connection failure. ## Verification - Focused checks cover 145 tests in 5 files. The first post-fix run passed 144 tests. One existing Fireflies route test timed out during the first import while the UI typecheck also ran. The same test passed in the pre-fix run. The isolated route suite then passed all 21 tests with no source change. - The new tests reproduce slow detail refreshes, activity refresh, and two concurrent transfers without a base revision. The concurrency test uses PostgreSQL row locks. - UI and server TypeScript checks passed at commit `1df9587eb`. - The UI production build passed for the updated head. - The branch includes master through `26391707f`. - Design token gates, query-error policy, and Git whitespace checks passed. The query-error policy reports zero findings. - Local browser checks passed on desktop and mobile for the initial feature commit. Search, selection, reload persistence, and revision creation passed. No page errors or horizontal overflow were found. The follow-up does not change the normal layout. - Earlier full local checks could not finish: Cargo was absent, and concurrent full test/build checks exceeded the 8 GiB memory limit. These attempts are not passes. - The first CI run failed on two error-rendering policy findings. That failure is fixed locally. The three findings from the first Greptile review are fixed and their threads are resolved. - Greptile reviewed `1df9587eb2d3e034c2769b1f6955ac52323c95bb` and scored it 5/5. It confirms all three prior findings are addressed. There are no open review threads. - [CI attempt 1](https://github.com/paperclipai/paperclip/actions/runs/42046369415/attempts/1) passed 53 checks. The server 8/12 shard failed only because the existing `history-http` runtime fixture did not become ready within 10 seconds (99 refused connections). The fixture and runtime source are unchanged from the merged master. That case passed locally on the same PR head with no source change. Its full 75-test suite also passed locally. The cause of the CI failure remains unknown. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42046369415/attempts/2) passed on `1df9587eb`: 54 successful checks, 3 skipped, and 1 neutral. No failed or pending checks remained. The original fixture-readiness failure remains recorded above. - A later review found that viewers with an explicit `tasks:assign` grant could see an editable picker even though writes were denied. Commit `8b95ee707` now mirrors active-membership and viewer restrictions in the GET capability flag. Instance-admin and local-operator behavior is preserved. - Three membership regressions failed on the incorrect editable flag before the fix. After the fix, all 43 routine-route and company-access cases passed (`--testTimeout 60000` accommodated this sandbox's cold route imports; no repository timeout was changed). The first pre-fix run timed out during the cold import in one case; the second pre-fix run reproduced all three assertion failures. - Layered server typechecking and whitespace checks passed after the permission fix. - [Fresh CI](https://github.com/paperclipai/paperclip/actions/runs/42057752313) passed on `8b95ee7075417527687b4a86cbaf63ba87408364` without a retry. Greptile reviewed this exact head, scored it 5/5, and added no new findings. All four review threads are resolved. The security status is also successful. - The separate Paperclip Review check is still queued because its assigned review worker requires runtime recovery. This is the remaining check before completion; the CI workflow itself passed. ## Risks - A transfer changes the person used for future automatic tasks. Existing tasks keep their current person. - Only people can transfer responsibility. The server checks the permission again on every write. - A stale revision rejects the save. The user must reload before retrying. - No database migration is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact runtime model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (CI passed; separate Paperclip Review check pending) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
253bf071f0 |
fix(workspaces): preserve resolved pending merges in snapshots (#19429)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task workspaces must survive replacement of the web server. > - Workspace snapshots therefore preserve both files and portable Git state. > - A resolved merge can still await its merge commit. > - The archive rejected this state, so an existing workspace could not migrate. > - This pull request saves and checks the complete supported merge state. > - The benefit is that migration preserves the user's choice to continue or abort the merge. ## Linked Issues or Issue Description **What happened?** A legacy workspace contained a resolved merge with `MERGE_HEAD`, `MERGE_MODE`, `MERGE_MSG`, `ORIG_HEAD`, and `AUTO_MERGE`. Its index had no unresolved entries. The snapshot owner refused the operation. This kept the original safe but blocked migration. **Expected behavior** Save this supported state without changing the source. Restore it without the original repository, borrowed object store, or linked worktree. Retain the merge parents, selected index, later edits, and Git's normal safety checks. **Steps to reproduce** 1. Merge two branches that conflict. 2. Resolve and stage the conflict. Leave the merge uncommitted. 3. Capture the workspace. The base implementation rejects the merge marker. Related prior work: #19310 preserves an inactive rebase bookmark. No duplicate open merge-preservation PR was found. This is a workspace preservation fix, not a new roadmap feature. ## What Changed - Admit only resolved pending merges with valid, bounded metadata and available objects. - Retain and verify the merge mode, message, parents, original head, automatic merge tree, and index. - Verify copied metadata and object closure before publication. Keep the before/after source checks. - Keep unresolved conflicts, autostash, rerere, mixed operations, malformed state, and unsafe links blocked. - Test real ordinary, linked, and borrowed repositories through local and serialized capture. Delete all original sources before continuing or aborting the restored merge. - Document the supported state and the remaining refusal cases. ## Verification - Six new continuation cases fail against the unchanged base at the pending-operation guard. - The first candidate restored and continued the merges, but six abort assertions failed. The fixture rewrote the worktree without updating Git's index cache. It now first verifies that abort protects later edits, uses normal `git restore` to discard those fixture edits, then aborts. No archive assertion was removed. - The final archive and environment suites pass all 98 tests at `3c87d13d1b`, including linked-metadata protections and pending merges in ordinary, linked, borrowed, shallow, partial, and SHA-256 repositories. All 12 local/serialized continuation cases pass after source removal. - Full repository build and typecheck pass at `1544cc4a90`. The build first found a nullability error; an explicit rejection return fixed it. - The first full local test attempt stopped before tests ran because omitted package postinstall steps left PostgreSQL library links absent. Direct `initdb --version` reproduced the missing-library failure. Restoring the package links made that probe pass; the full test rerun is active. - The initial typecheck also found missing excluded-plugin SDK links after the same install. Running the repository postinstall link helper corrected this, and the full typecheck now passes. - All current-head CI gates pass at `3c87d13d1b`: 54 success, 2 skipped, 1 neutral. Fresh Greptile is 5/5 and no review threads remain open. - Review requested shallow and SHA-256 coverage. The current commit adds both plus partial-clone coverage; all 98 archive tests and the full server typecheck pass. - All six local chat shards pass. The monolithic local general-suite run reports one viewer-permission test failure and remains active. That exact test passes alone, and its full 505-test file also passes alone. The original failure cause is unresolved; this is not a full local-suite PASS. The complete current-head CI suite is green. ## Risks - This widens capture eligibility only for the checked resolved-merge state. No migration commits or aborts the source operation. - A missing object or changed metadata still blocks publication and retains the source. - Unresolved merges and other operation formats remain unsupported. Their preservation guard is unchanged. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, and code execution. The runtime does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass; full-suite limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented the risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
8c578668b8 |
fix(heartbeat): start new queued runs promptly after restart and name the wait (#19431)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued agent turns. Restarts and deploys
happen often.
> - After a restart, new queued turns waited minutes before they
started. The task showed only "Queued", with no reason.
> - The cause is startup order. The queued-run dispatch worker registers
only after many unrelated first passes finish one after another: tool
reviews, delivery workers, the terminal workspace sweep, native restart
recovery, and residual and issue recovery.
> - The server binds its listener before this recovery, and
`/api/health` answers during it. A deployment can route traffic to the
new server for minutes before queued work starts.
> - This pull request starts queued turns that never executed as soon as
plugin workers and adapters are loaded. It also names the wait while the
server starts, recovers, or shuts down.
> - The benefit is that tasks resume quickly after a restart and show a
visible reason while they wait.
## Linked Issues or Issue Description
**What happened?**
On a hosted instance, the server restarted twice for deploys. Both
times, queued runs on idle agents started about 3.5 to 4 minutes after
the restart. Before the restart, the runs also waited about 5 minutes
while the deploy drained the old server. During the wait after the
restart, the execution presentation showed `phase: queued`, `label:
"Queued"`, `cause: null`.
**Expected behavior**
Queued work resumes automatically and quickly after a restart. While it
waits, the task shows a reason.
**Steps to reproduce**
1. Start a task drain (`POST /api/instance/task-drain`). Assign issues
to idle agents. The runs stay queued.
2. Stop the server (SIGTERM) and start it again.
3. Make any startup first pass slow (a large instance does this
naturally). Queued runs start only after every earlier startup pass
finishes.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
140fcf98ed |
fix(retry): name a declined task Retry and let a new click replace it (#19426)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task interrupted by a restart or deploy must resume automatically, or offer a Retry that works on first use > - A board Retry saves an intent first. Dispatch then rechecks the requester's authority and saved source before it starts a run > - When that recheck refused the intent, dispatch returned a bare "deferred". The intent stayed saved with no reason, and the task still showed Retry > - Each later click with a new key joined that same intent, so Retry returned 202 and never started a run > - This pull request names the refusal on the intent and lets a new click replace an intent that can never start > - The benefit is that a Retry button that is shown starts one run, or explains why it cannot ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox lost its runner twice. Saved-work recovery recovered the files. The task showed "Saved work recovered. Retry or send a message to continue." and `canRetry: true`. A Retry click returned 202 and logged `issue.retry_requested`, but no run started. A second click with a new idempotency key returned the same request ID and `state: failed`. No run started. **Expected behavior** If Retry is offered, a click starts exactly one run. If the Retry cannot start, the task names the reason, and a later click is a fresh attempt. A Retry accepted during the pre-deploy drain starts after the restart. **Steps to reproduce** 1. Recover the files of a failed native run so the hold offers Retry. 2. Save a Retry intent whose dispatch-time authority recheck fails (for example, the requester lost access). 3. Dispatch it: no run starts, and the intent has no wait reason. 4. Click Retry again with a new key: the response returns the first request ID, and no run starts. **Paperclip version or commit** Reproduced on `b400914345` (staging served `4d49a31286`; the affected code is the same). Related: #19399 (automatic Retry after a lost runner), #19422. ## What Changed - `heartbeat/queue.ts`: when dispatch declines a task Retry because the requester can no longer change the task, or its saved board message changed, record a named wait (`retry_authority_denied`, `retry_source_changed`) and log it. Before, it returned "deferred" with no record. - `task-retry.ts`: a new click with a new key replaces an unadmitted intent that dispatch declined for its requester's access (`retry_authority_denied`). The old intent becomes `skipped` / `task_retry_replaced`, and activity records `replacedRequestId`. Waits on shared state (cleanup, recovery, an active run, preservation) still coalesce. The restart hook never replaces a user's click. Chat-owned intents keep their own owner. - `task-interruption-presentation.ts`: fixed public messages for the two new wait reasons. `retry_authority_denied` keeps Retry, because a new click is a fresh attempt. `retry_source_changed` is terminal: an edited saved message cannot be replayed, so Retry is hidden and the user sends a new message. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres. The fixture runs the automatic Retry from a lost runner, then that Retry's own run loses its runner, and recovery saves its files: - `starts exactly one run when Retry is pressed after the automatic Retry's own run lost its runner` (also checks that a replayed dispatch and a second click start no second run). - `does not let a Retry that could not start swallow a later Retry`: red before the fix in two steps. With no fix, the declined intent has no wait reason. With only the wait fix, the second click returned the first request ID. Green after the fix: one run, bound to the second click. - `dispatches a Retry accepted during the pre-deploy drain once the restarted server resumes`: green before and after. The drain does not lose a saved Retry. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/quarantined-task-retry.test.ts src/services/explicit-native-continuation.test.ts`: 892 passed. - Not verified on staging. After deploy, an intent that dispatch refuses shows a named `retryWait` reason. ## Risks - Low risk. Only intents that dispatch declined for their requester's access are replaced. All other coalescing stays. - A replaced intent's late dispatch finds it `skipped` and starts nothing, so a double click still starts at most one run. - The exact gate that held the staging intent is not confirmed from records. If it was a shared-state wait, the new code names it only if it is one of the two reasons above. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7e3caa7bac |
fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.
## Linked Issues or Issue Description
Refs #19394
**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.
**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.
**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.
**Paperclip version or commit**
master at
|
||
|
Devin FoleyandClaude Opus 5.5
|
2c7ee84221 |
fix(server): stop reporting work cancelled by a disconnected client (#19037)
A route that aborts its storage read when the response closes rejects with AbortError after the client has gone. The error handler reported that cancellation as a server crash. Skip the error sinks for an AbortError when the response was closed before it finished; report it otherwise. Fixes PAPERCLIP-BACKEND-5A Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
bf53060904 |
fix(recovery): continue saved input after verified workspace copy-back (#19422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution preserves accepted results and task files after a failure. > - A later user message can request a new turn after the old execution stops. > - File recovery can finish by copying files back or by retaining a verified archive. > - The retained-result gate accepted the archive proof but missed a verified copy-back receipt. > - This change accepts that existing receipt for terminal failures and preserves execution authority and prior results. ## Linked Issues or Issue Description **What happened?** A task keeps a user message queued with `finalizing_saved_result` after its old native coordinator reached `terminal_failure`. The accepted result and a successful, exact `workspace_recovery` copy-back receipt are saved. The gate only checks the alternate `workspace_recovery_snapshot` receipt. An older failed result can also keep a newer failed turn's saved input blocked. **Expected behavior** A new user message or deliberate task Retry can continue from verified recovered files after the old execution stops. Preserve the old result and unknown action outcomes. Keep an active finalizer or missing stop evidence blocked. **Steps to reproduce** 1. Retain an accepted result when a native run reaches terminal failure. 2. Complete host copy-back with the exact run, allocation, descriptor and final file digest. 3. Save a new user message or request task Retry. 4. Observe that admission still reports `finalizing_saved_result`. **Paperclip version or commit** Reproduced against `4d49a3128613b282fb344272ed7ca15f98dea2ac`. **Deployment mode** Authenticated native execution with retained workspace recovery. Local regressions use disposable PostgreSQL and the real heartbeat queue. They start no model provider or sandbox. Related work: #19355 covers saved input after Stop. #18267 concerns manual reconciliation of duplicate legacy holds; it excludes workspace and system recovery and does not fix this receipt gate. I searched open PRs and issues before starting this change. ## What Changed - Reuse the existing host-verified repair receipt when an accepted result belongs to a coordinator in `terminal_failure`. - Retain the separate checks for active controllers, environment termination, live executions, questions, approvals and current user authority. - Add message, saved-message and task Retry regressions, plus concurrent queue delivery and invalid-receipt cases. - Document this continuation path. Add no schema, API or background worker. ## Verification - All 483 tests pass across explicit continuation, task Retry, quarantined Retry and finalized recovery migration suites. - The final focused fixture has 17 cases. All pass with the change. With the original production gate restored, all four positive cases fail and all 13 safety cases pass. - Negative cases cover missing, unfinished and mismatched receipts, another task or source, unrecovered files, a live finalizer, an active controller, missing remote stop and active execution. - The real heartbeat path delivers one saved message under concurrent recovery calls. Later recovery calls create no second turn. The old accepted result and recovery budget stay unchanged; unknown outcomes stay unknown. - The first delivery fixture omitted the newer failed source needed by the recovery scanner. I corrected the fixture to preserve that history and repeated the unchanged-code comparison. Earlier failure logs remain in the verification record. - `pnpm -r typecheck`: passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - `pnpm build`: passed. - Full local `pnpm test:run`: running with a 20-minute bound; no full-suite pass is claimed. All hosted CI checks passed. Greptile scored the exact head `d04fb98892fd77853ad03e6b44cd612e805abc13` 5/5 with no findings or review threads. - Hosted delivery of the already saved input still requires a qualified staging build containing this change. ## Risks The new path requires the existing exact repair receipt and a terminal failure. A coordinator still accepting or committing a result keeps the previous hold. Stop, owner and live-run checks still run before admission. The old result is neither applied nor replayed. Receipt verification adds bounded indexed reads only on this retained-result failure path. No migration or provider configuration changes are required. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex in T3 Code, with xhigh reasoning. The context-window size is not exposed. Used code editing, tool calls and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the available runtime details - [x] I have checked ROADMAP.md and confirmed this is a scoped bug fix - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR using the bug-report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the affected tests locally and they pass - [x] I have added regression tests and updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review findings before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
11bc490cda |
fix(recovery): keep a stopped copy-back's files instead of holding the workspace (#19385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can have its result accepted before the copy-back of its sandbox files finishes. If the copy-back fails, it retries. > - A board Stop during those retries ends the run. It sets the run's finalization to `terminal_failure` / `native_retry_cancelled`, and the result stays accepted. > - After that, nothing exports the files. The finalization worker skips terminal rows, and cancelled-workspace preservation skipped any run with an accepted result. > - The task workspace stays dirty and bound to the stopped run. Every later run on the task waits for the workspace, then fails setup with the unfinished-writer error. > - This pull request lets cancelled-workspace preservation also admit an accepted result whose copy-back a Stop ended. > - The benefit is that stopping a stuck copy-back saves its files and frees the task, like stopping any other run. ## Linked Issues or Issue Description Refs #16700, #17677, #18820, #19304 and #19305 (cancelled-workspace preservation). This pull request adds the accepted-result case to that path and keeps its other checks unchanged. **What happened?** On a hosted instance, a native run's result was accepted, but its copy-back kept failing and retrying. A board operator pressed Stop to unblock the task. The run became `cancelled`, its sandbox was stopped and kept, and its finalization moved to `terminal_failure` / `native_retry_cancelled` with the result still accepted. The next run on the task was cancelled with "Shared execution workspace is busy: run … ended, but its workspace save has not settled". About 24 automatic retries followed over an hour. The last one failed with "The workspace still has an unfinished writer or save operation." After that nothing ran: the task stayed `in_progress` with no run and no recovery action. A Retry or a new comment would hit the same block. The workspace stayed `dirty: true` with the stopped run as its writer, and no `workspaceRestoreRecovery` was ever recorded. The reason: `preserveCancelledNativeWorkspaces` requires `resultId IS NULL`, and `registerCancelledWorkspacePreservationWork` and `preserveNativeFailedWorkspace` both return early when `resultId` is set. **Expected behavior** Stopping a run during its copy-back retries keeps its files, as stopping any other run does. Paperclip's workspace recovery saves them and releases the workspace. Nothing commits or replays the stopped result. **Steps to reproduce** 1. Accept a native result for a run whose copy-back is in `retryable_failure`. 2. Stop the run with the board Stop. The finalization becomes `terminal_failure` / `native_retry_cancelled`. 3. Start another run on the task. It waits for the workspace and then fails setup. No save of the stopped run's files is ever recorded. The new database tests run steps 1 and 2 against PostgreSQL and check that the save is recorded. **Paperclip version or commit** `master` at `9ceaec6f97e`. **Deployment mode** Any deployment that uses remote (sandbox) task workspaces. Seen on a cloud-managed instance. ## What Changed - `server/src/services/native-runtime/native-failed-workspace-recovery.ts`: cancelled-workspace preservation now also admits an accepted result when the finalization is `terminal_failure` with `native_retry_cancelled`. That is the state a Stop leaves after ending copy-back retries. All three entry points use the same rule: - the startup scan `preserveCancelledNativeWorkspaces`; - the worker `registerCancelledWorkspacePreservationWork`; - `preserveNativeFailedWorkspace`, only when it is called with `requireCancelledStop`. Preservation still requires everything it required before: a cancelled run, an acknowledged Stop, no live finalization owner or controller lease, a prepared workspace reference, and the exact retained source. - An accepted result with any other failure code is still skipped; those runs keep their own export repair path. Failed (not cancelled) runs are unchanged. - The accepted result stays recorded and uncommitted. Preservation does not change the task's status or queue a run. - `doc/execution-semantics.md`: describe the stopped copy-back case in the paragraph on writers stopped mid-run. - `native-cancelled-workspace-preservation.test.ts`: move the accepted-result fixture into an `acceptResult` helper, and add a test with three cases. ## Verification - Failing first. The new test "preserves files of an accepted result whose copy-back was stopped" has three cases: worker, startup scan, and a sandbox that the Stop already stopped and kept, through the real `prepareSandboxStopAndRetain` and `settleStopOnlyCleanup`. All three fail on `master`, with no `workspaceRestoreFailure` recorded, and pass with this change. - Each case checks that: - the save obligation and recovery action are recorded for the exact lease; - `inspectWorkspaceRecoverySource` finds one source; - the finalization keeps its `resultId`, `terminal_failure` and `native_retry_cancelled`; - the workspace stays held by the stopped writer until the save publishes; - no run or wake is created, and the task status is unchanged. - The existing "does not admit preservation for accepted-result" case still passes. It uses an accepted result with a different failure code. - `cd server && npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts`: 44 passed. - Related suites, 10 files and 854 passed: - `native-failed-workspace-recovery`, `workspace-recovery`, `heartbeat/run-control` - `native-cancellation-request` (unit and integration) - `heartbeat-remote-workspace-busy`, `native-workspace-export-retry`, `native-workspace-reconstruction` - `native-stop-accounting`, `task-workspace-binding` - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not run end to end on a live instance. Diagnosed from that instance's run, finalization, lease, workspace and recovery-action rows. ## Risks - On deploy, the startup scan and the worker will pick up existing cancelled runs in this state and record their saves. That is the intended repair, and it uses the same retention checks as other stopped writers. - A copy-back interrupted by a server restart can leave a `remote_workspace_snapshot` operation marked `running` after its process is gone. This change does not settle such rows. If one blocks the later save, the save follows its existing retry and board hand-off. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
39096a55a4 |
fix(runner): answer lease renewal outside the PRP frame chain and name lease loss (#19419)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server over one authenticated PRP connection. The runner (`paperclip-runnerd`) renews its connection lease on that connection. > - The server serializes every inbound PRP frame through one promise chain per connection. Before this change, `lease_renew` waited in that chain behind earlier events. An event whose business commit stalls (for example the commit of a tool event) therefore also stalls the lease reply. The runner then stops polling its provider (ACK backpressure) and exits with "active connection lease expired" when the lease runs out. > - When that happens, the server sees only a socket close. The run log and the terminal error say "transport loss", not "lease expired". > - A follow-up from #19407 asked for renewal to be handled outside the chain and for lease loss to be named. > - This pull request answers `lease_renew` (and `pong`) as soon as the frame is decrypted. It keeps in-order decryption and the order of state-changing frames. It also records why a lease was lost and reports it as `runner_connection_lease_expired`. > - The benefit is that a slow event commit can no longer starve lease renewal, and a lease loss is visible in the run log and error code. ## Linked Issues or Issue Description Refs #19407 **What happened?** On staging, native runs with ACPX providers (Grok, Kimi, and Pi with OpenRouter/DeepSeek) ran a long foreground shell command that printed nothing (a 15–20 minute `sleep 60` loop). Each run went silent soon after `tool.execution.started`. The command finished in the sandbox, but the run never got the completion and stayed `running` until someone cancelled it. It happened in 3 of 3 attempts. The same task on Codex kept reporting activity and succeeded. **What did you expect to happen?** A quiet tool call keeps its runner connection and its lease. If the connection or lease is lost, the run log and the error code say why. **Steps to reproduce** 1. Assign a native ACPX agent a task that runs a foreground shell loop with no output for longer than 2 minutes. 2. Observe that the run goes silent and does not finish after the command ends. **Root cause analysis** - The ACPX path emits no PRP frames during a silent tool call. The runner has no timer, and the agent sends no `tool_call_update`. Codex keeps the connection busy, because its model polls the running command. Silence is the only adapter-specific factor that I found. I found no ACPX code path in runnerd that blocks the main loop: polls use a 1 ms timeout, and synchronous sidecar requests run only for server commands. - The staging build (`de1143448b`) did not include #19407. That build had no server pings. With a 60-minute connection lease (the default for `runner-prp-coordinator` and `DurablePrpCodexTransport`), the runner renews only every 30 minutes. A fully silent connection that goes through the sandbox preview proxy therefore carried no traffic at all. A route that the proxy drops without a FIN then becomes half-open forever. #19407 added 20-second pings and a 90-second termination, which closes that window. - The lease hypothesis from #19407 is real but secondary. The 60-second lease applies only when the TTL is not set. Renewal waited behind any slow event commit, and the server could not report lease loss. This pull request fixes both. ## What Changed - `DurablePrpControlPlane.attachWireConnection` decrypts each secure frame when it arrives, in counter order. Frames that arrive before the channel is secure, or behind such a frame, still decrypt in the chain. - `lease_renew` and `pong` are handled at once, outside the per-connection chain. Admission against persisted authority is unchanged. Events, command results, and handoff frames keep their serialized order. - `#renewLease` persists the renewed expiry with `save()`, so an event write that is already staged is absorbed into the same write. Before, the renewal used `commit()`, which would invalidate that staged event and close a healthy connection when renewal runs concurrently. - New `connectionLeaseFailure` getter (exported type `DurablePrpConnectionLeaseFailure`). It records `connection_lease_expired` when an authenticated connection is refused or closes with an expired lease. A new authenticated connection clears it. A credential revoked by a warm handoff is not reported, because a replacement runner owns the run. - `connectRunnerPrpIngress` reads that reason after a close. It logs `state=reconnecting failure=runner_connection_lease_expired` in the run log. If reconnect grace runs out, it fails with that code instead of `runner_ingress_unavailable` or `runner_ingress_auth_failed`, and keeps the reconnect error as the cause. - `packages/paperclip-runner/docs/durable-recovery.md` documents the new behavior. ## Verification - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane` (220 passed). - New: "answers lease renewal while an earlier event commit is still in flight". It was red before the fix (no `lease_renewed` while `onCommittedEvent` waits), and it is green now. It also checks that the event later commits and ACKs, and that the renewed expiry survives. - New: "keeps a staged tool-progress write when a lease renewal lands mid-write". It holds a file-backed `tool.execution.progressed` write at `fsync`. It is red with the old `commit()` renewal (the event is dropped and the connection closes), and green with `save()`. - New: "reports an expired connection lease instead of closing silently". It was red before the fix and is green now. - `pnpm --filter @paperclipai/server exec vitest run src/realtime/runner-prp-outbound.test.ts src/realtime/runner-prp-ws.test.ts src/services/native-runtime/runner-prp-coordinator.test.ts` (all passed). - New: "names an expired runner connection lease in the run log and terminal error when reconnect is refused/unauthenticated". It was red before the fix and is green now. - `src/live/runnerd-codex-transport.test.ts` has the same result with and without this change with a locally built `paperclip-runnerd`: the same 48 tests fail locally on both, for environment reasons, and the other 207 pass. ## Risks - Lease renewal now runs concurrently with an in-flight event handler. The handler builds its state candidate at commit time, and `save()` absorbs a staged event write. A new regression test covers the file-backed race. External checkpoint persistence commits events synchronously, and the renewal reply still waits for the checkpoint flush. - Renewal still stays silent while a warm handoff or a terminal lifecycle command is pending. The runner treats any other control frame inside those fences as fatal, so this pull request does not change that behavior. - No protocol, schema, or contract-digest change. The runner is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs) and a read-only exploration subagent on the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c1d567ff6a |
fix(workspaces): isolate archived nested Git reconstruction (#19409)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace import must preserve archived task files and Git history. > - A saved nested checkout can be a linked worktree whose branch belongs to its old absolute path. > - Archived import reconstructs that checkout in a new scratch directory. > - Ordinary linked recovery rejects the old branch at that new path. It can also write into a cache used by other import workers. > - This pull request reconstructs archived nested checkouts as private standalone repositories. > - Each import preserves saved files and extra history without changing shared worktree registrations or branches. ## Linked Issues or Issue Description **What happened?** Archived workspace import fails when a saved nested linked checkout has a branch owned by the old path. The new scratch path fails the linked-worktree ownership check. The existing cache lock is process-local, so separate import processes also share mutable Git state without a common lock. **Expected behavior** Archived import must create an independent portable copy of the saved checkout. It must retain the saved commit, branch name, staged edits, working files, ignored recovery files, settings, and extra saved commits. It must keep the source cache unchanged and enforce source authorization. **Steps to reproduce** 1. Create a managed nested linked checkout with an unpushed commit and staged and unstaged edits. 2. Save an encrypted restore point, including ignored files and an extra history tip. 3. Restore its saved contents under a different absolute workspace path. 4. Before this repair, the new path cannot adopt the branch owned by the old path. **Paperclip version or commit** The real Git regression fails before the repair at `fc1f5f3052`. The final branch includes the merged interrupted-upload and nested-source authorization fix from #19396. **Deployment mode** Archived remote workspace migration with real Git repositories, disposable PostgreSQL, and encrypted object storage. Local integration tests use the filesystem storage provider. Builds on merged #19402 and #19396. Related #19374 concerns interrupted clone staging; it does not implement private archived reconstruction. No duplicate open repair was found. ## What Changed - Enable private repository reconstruction only for archived migration. Ordinary linked recovery keeps its path ownership guard. - Add an object-only source to the managed checkout owner. A verified cache supplies only the saved commit and authenticated bundle prerequisites through scoped fetches. The private copy includes no unrelated task refs or objects, hard links, object alternates, or current cache files. - Fetch missing authenticated bundle prerequisites before importing a saved bundle in both private reconstruction and ordinary linked recovery. A pruned cache can use the approved remote; unavailable history still blocks restore. - Keep every authenticated restore point's extra commit under a private ref in root and nested private copies so portable archive capture and garbage collection retain it. Ordinary recovery keeps its existing history and expiration rules. - Share the verified, bounded ignore-file copy between root and nested reconstruction. Sources remain unchanged. - Retain source authorization before and after reconstruction and the final source-row locks from #19396. - Test concurrent private copies, source rejection, foreign task refs and objects, ordinary linked prerequisite recovery, a real migration, and cold restore after removing all original repositories. Test hard links, oversized excludes, unsafe info paths, and replacement during a descriptor read. ## Verification - Final affected run: 292 tests pass across ten suites. These cover real Git repositories, encrypted restore points, disposable PostgreSQL migration, portable archives, the restore-point ledger, source rejection, and concurrent copies. - Full repository typecheck and build pass after the scoped-object repair. Final server typecheck and full build also pass after restricting preserved refs to archive mode. - RED then GREEN: a linked checkout at a different path; a pruned bundle prerequisite still available from the approved remote; foreign task refs and objects entering the archive; root extra history disappearing after cold restore; and archive-only refs blocking ordinary standalone expiration. - Two concurrent private copies retain the saved head, branch, staged and unstaged edits, ignored recovery bytes, settings, and modes. Source refs, index, worktree registrations, and working files remain unchanged. - Delete the original checkout, cache, remote, and private copies after portable capture. Cold restore retains the saved files and extra history after garbage collection. No alternates or unrelated task objects remain. - Ordinary linked recovery retains its path ownership check and recovers pruned prerequisites. New and existing ordinary standalone destinations have no archive-only refs and remain eligible for expiration after normal object cleanup. - Unsafe exclude tests reject hard links, oversized files, nonregular paths, info-directory substitution, and replacement during the descriptor read. They preserve the target and replacement source bytes. - Preserve the failed attempts: an initial migration fixture changed its home after constructing the source path. Later expiration fixture attempts used a filtered current-head tip, omitted the local-source callback, or used cleanup that generated Git server-info files or left packed unreachable objects. Correct those setup assumptions before the actual regressions fail. Final affected suites pass. - Two earlier local full-suite attempts were interrupted before editing valid review findings. Neither is reported as a full-suite PASS. Final-head full CI passes; review is 5/5 with no open findings. ## Risks Archived imports use additional private disk space for independent Git objects. Missing, changed, or unsafe sources still block import. The repair does not weaken the default linked-worktree ownership check, change stored formats, or update live provider state. The shared checkout addition is optional and has no effect for existing callers that do not supply an object source. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
90b8190c33 |
fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
92452524f4 |
fix(server): recover authenticated empty retention checkpoints (#19417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral server enrollment must preserve existing task files. > - Archived task directories can depend on encrypted retention checkpoints. > - A missing object blocks migration even when its authenticated contents were exactly empty. > - Metadata counters alone cannot prove that no saved files existed. > - This change verifies the exact candidate against the original encrypted key envelope before normal migration publication. > - The benefit is lossless migration of this bounded case without resetting task history or weakening ordinary restore. ## Linked Issues or Issue Description **What happened?** Migration stops at an archived empty directory when its retained encrypted object is missing. The original encrypted key envelope still authenticates the payload hash, byte length, company, workspace, checkpoint ID, and generation. **Expected behavior** Administrative migration can reconstruct those exact original bytes when storage confirms absence and every authenticated check passes. It must keep missing nonempty, expired, corrupt, or ambiguous sources blocked. **Steps to reproduce** 1. Capture and encrypt an empty task directory. 2. Retain its reference and encrypted key envelope. 3. Remove the old directory and storage object. 4. Run the normal remote workspace migration. **Paperclip version or commit** The regression fails before this change at `205ffcb606`. Related preservation work is in #19396, #19402, and #19409. No duplicate open fix was found. **Deployment mode** Administrative migration to external workspace storage. Tests use real encryption, real filesystem storage, and disposable PostgreSQL. ## What Changed - Let the encryption owner verify a candidate against the existing scoped encrypted key envelope. Normal ciphertext validation keeps its previous order and checks. - Share the task-directory encoder with an exact empty payload constructor. - Add a default-off loader option for confirmed missing empty checkpoints. Only the administrative migration caller enables it. - Keep the original checkpoint reference and object unchanged. Publish the recovered bytes through the existing snapshot and source-fence owners. - Reject forged counters, wrong identities, missing nonempty data, present corruption, unreadable present objects, unknown absence, and expiry. ## Verification - Final affected run: all 238 tests pass across five crypto, snapshot, retention, and migration suites. Full repository typecheck and build pass; server typecheck and full build also pass after the cancellation repair. - RED then GREEN: exact missing-empty recovery and cancellation of a stalled HEAD through the real storage-service boundary. Cancellation retains the caller's original abort reason. - Disposable PostgreSQL migration publishes a cold-restorable empty archive. Archived status, task status, task binding, original reference, and the absent original source stay unchanged. Replay keeps one canonical snapshot. - Negative cases reject forged empty counters, an authenticated directory or file, every foreign identity field, expiry, present corruption, unreadable present objects, unknown absence, and missing nonempty data. Ordinary restore keeps its previous failure behavior. - Initial verification retained one new fixture-path failure after 145 passes, and a typecheck rejected the fixture's widened schema literal. Correct the fixture to use the returned staging path and its explicit reference type. The final complete affected run passes. - The new HEAD request runs only after a failed read with the administrative option enabled. It uses the existing cancellation contract and adds no polling or background work. - Final-head full CI passes; review is 5/5 with no open findings. This PR does not claim deployed enrollment or a live migration result. ## Risks Recovery requires the original valid encrypted key envelope and an exact format match. A format change or unavailable key continues to block migration. Existing ciphertext hash, size, authentication, and ordinary restore behavior remain enforced. This change has no schema, provider-session, API, or UI contract changes. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
a4bb059f59 |
fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task runs in an execution environment. A managed sandbox
environment gets a lease from a sandbox provider (for example Daytona)
before the agent starts.
> - When the provider answered sandbox create with a temporary error
(HTTP 502), the run failed as the generic `setup_failed`.
> - `setup_failed` is the fail-closed code for unknown setup errors. It
gets no automatic retry, so the task went to `blocked` at once. The task
showed only "Couldn't finish this attempt".
> - A user had to find and press Retry. The Retry then worked, which
shows that the failure was temporary.
> - This pull request classifies temporary provider refusals during
lease acquisition from structured fields. It retries them within the
existing bounded bootstrap budget, and it explains the block when the
retries are used up.
> - The benefit is that a short provider outage no longer stops a task.
When the outage continues, the task shows the cause and a Retry that
works.
## Linked Issues or Issue Description
**What happened?**
A task was assigned to an agent that uses a managed Daytona sandbox
environment. Sandbox creation returned `HTTP 502`, and the cleanup of
the provisional sandbox could not be confirmed. The run failed with
`setup_failed` and the error `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP
502); cleanup could not be confirmed for paperclip-create-…`. Three
seconds later the task was `blocked`. No automatic retry was scheduled.
The task showed no specific reason. A manual Retry started a new run
that completed normally.
**Expected behavior**
A temporary provider error (HTTP 5xx, 429, timeout) before the agent
starts must retry automatically with bounded backoff. The task must show
why it waits. If the retries are used up, the task must show the cause,
a next action, and a Retry that works on the first click.
**Steps to reproduce**
1. Configure an agent with a sandbox environment.
2. Make the provider return HTTP 502 on sandbox create.
3. Assign a task to the agent.
4. See that the run fails with `setup_failed` and the task goes to
`blocked` with no retry.
**Paperclip version or commit**
`master` at `612da1e8e0`.
## What Changed
- Plugin SDK: an `environmentAcquireLease` error now carries a closed
diagnostic envelope (HTTP status or a closed errno code) across the
worker RPC. It takes the status of the create call only, not of the
compensating cleanup lookup. The existing creation-cleanup ownership
envelope is unchanged. New module: `environment-acquire-error.ts`. The
resume diagnostic helpers are shared.
- Server: new `EnvironmentProviderUnavailableError` and code
`environment_provider_unavailable`. The environment run orchestrator
sets it as the cause of `lease_acquire_failed` only when structured
evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or
a closed connection errno). It never parses message text. Saved-sandbox
resume and stopped-worker refusals keep their own codes.
- Setup failure (`failRunSetup`): a fresh, pre-dispatch run records
`environment_provider_unavailable` with a `providerFailure` (phase
`startup`, retryability `transient`). The run uses the bounded bootstrap
retry (30 s, then 60 s). An accepted interaction continuation keeps its
own continuation retry, as for `plugin_worker_unavailable`.
- Recovery: the new code is registered with the other bootstrap codes.
It is not sent to the generic continuation retry. It does not create an
execution-reconciliation hold, because no provider work started. On
exhaustion the stranded notice has the title "Sandbox provider was
temporarily unavailable" and a specific next action.
- UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox
provider". The task notice shows the provider failure, "Paperclip will
retry automatically" while a retry is scheduled, and Retry after
exhaustion.
- Docs: `doc/execution-semantics.md` describes the policy and the
sandbox ownership rule.
## Verification
Red before the fix, green after the fix:
- `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient
sandbox provider refusal during lease acquisition":
- "retries automatically and runs exactly one successful successor
without user action": the real heartbeat. The lease runtime returns a
Daytona 502 once, then delegates. Before the fix: one `failed` run and
no successor. After the fix: one `scheduled_retry` successor with no
hold and a `retry_scheduled` presentation. After promotion, exactly one
successor succeeds.
- "explains the block with a working Retry once automatic retries are
used up": three refusals. The task is `blocked` with a "Sandbox provider
was temporarily unavailable" notice. The presentation is `{ state:
"failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and
no execution-reconciliation hold exists. One `requestTaskRetry` is
accepted, and its run succeeds.
- `server/src/services/heartbeat/run-completion.test.ts` › "routes a …
sandbox provider lease refusal through bounded bootstrap recovery": the
real orchestrator wrapping. Only the fresh, structured 502 is retried. A
message-only 502, a 400, a run after provider dispatch, and a retained
native run stay `setup_failed`.
- `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the
real worker RPC. The test checks the create status (not the 404 cleanup
lookup), the errno, no provider text, the unchanged cleanup envelope,
and that forged envelopes are rejected.
Commands:
```sh
pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts
cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts
cd packages/plugins/sandbox-providers/daytona && npx vitest run
pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck
```
I did not run a live Daytona 502 on staging. Re-check on staging after
deploy (see Risks).
## Risks
- Sandbox leaks or double creation: each retry is a new run with a new
durable creation record and a new provider name
(`paperclip-create-<operationId>`). The failed attempt stays owned by
its own record, and the existing creation reconciler settles it. A retry
never reuses or abandons it. While a provisional sandbox waits for
reconciliation, up to two more sandboxes can be created for the same
task. This is the same as a manual Retry today.
- Plugins built with an older SDK do not send the envelope. Their
refusals stay `setup_failed`, which is the current behavior.
- An in-process error with `status`/`statusCode` 500–504 or a connection
errno during lease acquisition is now retried twice. The budget is
bounded and shared with the other bootstrap codes.
- An RPC timeout on the host side is not classified by this change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code as an agent with tool use (shell, file edits, test runs).
Extended reasoning was on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Devin FoleyandClaude Opus 5.5
|
558a5143ed |
fix(workspaces): reject damaged Git entries before workspace capture (#19160)
Workspace capture now checks that each .git entry is the root of its own repository before it reads any Git state. A structurally damaged entry (an empty .git directory, a gitfile for a removed worktree, an invalid HEAD, or missing objects/refs) stops capture with the closed reason remote_workspace_archive_invalid_git_repository instead of a raw "Command failed: git ..." error, and capture no longer saves an enclosing repository's state as the workspace's own. Other Git failures (dubious ownership, a full disk, a timeout, a failing Git program) keep their existing diagnostics. Metadata reads are bounded and never block on named pipes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
66d3fa97a9 |
fix(workspaces): preserve ignored files in archived Git imports (#19402)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspace migration must preserve archived files and saved recovery data. > - An archived Git workspace can contain ignored recovery receipts and bundles in its encrypted restore point. > - The importer reconstructs a private repository from a verified source before it applies that restore point. > - Reconstruction omitted the source repository's local ignore rules. The restored files then changed classification, and the exact restore-point check failed. > - This pull request copies the verified local ignore rules into the private repository before restoration. > - The source stays unchanged, and the original restore-point integrity checks still apply. ## Linked Issues or Issue Description **What happened?** An archived workspace with saved files under an ignored recovery directory failed migration with `restored_contents_mismatch`. The task commit and staged index matched. The ignored files were restored, but the private repository treated them as untracked work because its `info/exclude` rules were missing. **Expected behavior** Reconstruction from an existing verified source must preserve its local ignore policy. Saved ignored files must survive migration and a cold object-store restore with their original classification. Import must not follow a symlink to an unrelated ignore file or edit the source repository. **Steps to reproduce** 1. Exclude a recovery directory in the managed source repository's `info/exclude` file. 2. Save staged edits, unstaged edits, an ignored recovery receipt, and an ignored binary bundle in an encrypted restore point. 3. Archive and remove the task worktree. 4. Import the archived workspace. Before this change, restore-point validation fails because the reconstructed repository lacks the local exclusion rule. **Paperclip version or commit** Reproduced on master at `879140cbbd`. **Deployment mode** Cloud workspace migration with a retained verified Git source, PostgreSQL, and encrypted durable object storage. The tests use real disposable PostgreSQL, Git repositories, and encrypted filesystem-backed object storage. Related migration work: #18826, #19138, and #19396. No duplicate open fix was found. ## What Changed - Copy the verified source's `info/exclude` file into the private reconstructed repository before applying its restore point. - Bound the read to one MiB. Reject symlinked, non-regular, hard-linked, or changing source files. - Keep the original source bytes unchanged. - Add regressions that check staged and unstaged edits, ignored recovery files, cold restore, and rejection of a symlinked ignore file. - Document the reconstruction behavior. ## Verification - Before the repair, the saved-ignored-files regression fails with `restored_contents_mismatch`. - After the repair, both focused cases pass. - At rebased head `81a0b1c041`, all 206 tests across migration, snapshot storage, archive, and migration concurrency suites pass. The merged interrupted-upload and nested-source approval regressions remain intact. - Server package typecheck passes. - Full repository typecheck and build pass at the rebased head. The full local test suite is running. The prior local full-test run was interrupted for rebase, and is not a passing result. - Initial CI hit a 15-second timeout in unchanged `activity-routes.test.ts`; the exact case passed in a focused local run. The browser shard lost its self-hosted runner, as confirmed by its check annotation; GitHub did not retain its full log. Those failures remain recorded. An unchanged retry was superseded by the required conflict rebase. At exact head `81a0b1c041`, all 53 CI checks and both required aggregate gates pass; Greptile is 5/5. The full local suite remains in progress, so no local full-suite PASS is claimed. A second automated review noted the existing nested-checkout counterpart. This PR fixes root reconstruction. The nested linked-checkout repair and its saved-ignored-files/cold-restore regression are implemented in #19409, which is required before the migration rollout uses nested imports. This PR leaves that sibling recovery path unchanged. The two identical scope warnings are addressed through that linked repair. ## Risks The copy applies only when an archived Git restore-point source is still present and verified. It changes only the private reconstruction directory. Missing sources retain the existing restoration path. Unsafe or unstable local ignore files block import and retain the original migration obligation. No database migration or saved format changes are included. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
612da1e8e0 |
test(server): drain throttled follow-up in busy recovered-source execution-control test (#19403)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs durable execution-control workers, which include the workspace-recovery sweep > - `execution-control-work.test.ts` has a test for a Retry that commits during a busy recovered-source pass > - This test fails intermittently in PR runs that do not touch it, so it blocks unrelated PRs > - The test assumed that one `settle()` pass always ends idle, but the worker can correctly owe a throttled follow-up pass > - This pull request makes the test drive fake time through that throttle window while it waits for the sweep to finish > - The benefit is a deterministic test that still proves every source is visited and the worker becomes quiet ## Linked Issues or Issue Description **What happened?** `execution-control durable work > visits a newly committed earlier Retry during a busy recovered-source pass, then becomes quiet` failed in 2 of the last 250 failed `PR` workflow runs (jobs 127272079294 and 127206639630). Both failures were identical: `AssertionError: expected 1 to be +0` at `vi.waitFor` in `settle()` (`execution-control-work.test.ts:146`). The test took about 6s, which is the 5s `waitFor` limit plus overhead. **Expected behavior** The test passes on every run. Its result does not depend on runner speed. **Steps to reproduce** 1. Add a 300ms real-time delay in `registerExecutionControlWork` after `await duty.run()` and before `await duty.next()`. This simulates slow database I/O on a loaded runner. 2. Run `pnpm exec vitest run src/__tests__/execution-control-work.test.ts -t "busy recovered-source pass"` in `server/`. 3. The test fails 3 of 3 times with the same CI error. **Root cause** - `registerExecutionControlWork` lets a duty make one full pass per `EXECUTION_RECONCILIATION_INTERVAL_MS` (15s), measured from the start of the previous full pass. - The released recovery jobs call `worker.wake()` when they finish. If a wake arrives while the full pass is still reading its next deadline, `hasPending()` stays true. The coordinator then keeps its idle-tracked intent until the next permitted pass, up to 15s of fake time later. - `settle()` advances fake time by 15s and then waits 5s in real time for `idleWorkSnapshot().active === 0`. The full pass starts inside that 15s advance, so the follow-up is due after the window ends. The wait then expires. - On a fast machine the jobs usually finish after the deadline read, so the test passes. On a slow runner the order changes, so the test fails. - This is correct product behavior: the worker owes the follow-up pass. The defect is in the test's timing assumption. ## What Changed - In the busy recovered-source test, replace the bounded `settle()` loop with a `vi.waitFor` that advances fake time by `EXECUTION_RECONCILIATION_INTERVAL_MS` on each attempt. It waits until every source was visited and no idle-tracked work is active. - The test still calls `quiet(c)` after this, so it still asserts no further scheduling and no database calls during the next 90s. - The shared `settle()` helper is not changed, because `reschedules an earlier deadline…` asserts an exact `nextWakeAt()` after `settle()`. ## Verification - Red: with the 300ms delay injected (not committed), the test failed 3/3 times with the CI assertion. - Green: with the same injection at 300ms (3 runs) and at 1000ms (1 run), all four recovered-source Retry tests pass. - Without injection, the full `execution-control-work.test.ts` passed 5/5 times with 12 CPU-burner processes running. - `pnpm exec vitest run src/__tests__/execution-control-work.test.ts` in `server/`. ## Risks - Low risk. This is a test-only change. The test still requires every source to be visited, the worker to become idle, and then 90s of quiet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, GitHub CLI). Default context window. No extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
223687bbb9 |
test(server): size heavy chat-channel tests to their workload to stop timeout cascades (#19404)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat channels connect GitHub, Slack, and other providers to tasks,
and `chat-channels.integration.test.ts` covers them against a real
embedded Postgres
> - Two parameterized tests do much more database work than the other
tests in the file, but they use the global 15s `testTimeout`
> - On a slow CI runner they time out. Vitest then keeps the timed-out
test body running, and that body breaks later, unrelated tests in the
same file
> - This failure blocks PRs that do not touch chat code
> - This pull request gives the two heavy tests a time budget that fits
their workload
> - The benefit is that a slow runner no longer turns one heavy test
into a cascade of 2–7 failures
## Linked Issues or Issue Description
**What happened?**
In run 42098299218 (job 127495756270, shard `General tests (chat
(3/3))`), 7 chat tests failed in two Vitest processes on a PR that
changed only unrelated test files:
- Process 1: `reactivates denied retry notices when authority is
restored during bounded pass` timed out at 15000ms. Then `… during
requester merge` failed with `A function to advance timers was called
but the timers APIs are not mocked`.
- Process 2: `preserves one current GitHub Retry across backlog_51` and
`… consumed_backlog_52` timed out at 15000ms. About 110 tests later,
three pre-run FIFO tests failed: `advances pre-run FIFO selection past a
suppressed page with microsecond timestamps (manual)` (`expected +0 to
be 1`), `retires an already-visible pre-run FIFO notice once without
publishing scheduler failures` (`expected [] to have a length of 1`),
and `retries a contended pre-run FIFO receipt before I/O and observes
cancellation after the lock releases` (`expected undefined to match
object { state: 'retry' }`).
**Expected behavior**
These tests pass on a slower runner. A timeout in one test does not
change the result of a different test.
**Steps to reproduce**
1. On `master`, in `server/`, run:
`pnpm exec vitest run src/__tests__/chat-channels.integration.test.ts -t
"GitHub Retry across (backlog_51|consumed_backlog_52)|advances pre-run
FIFO selection past a suppressed page|retires an already-visible pre-run
FIFO notice once|retries a contended pre-run FIFO receipt before I/O"
--testTimeout=1200`
2. The lower timeout simulates a slow runner. The two backlog tests time
out, and then the same three FIFO tests fail with the same assertions as
in CI. The `scoped` FIFO case passes, as it did in CI.
3. Run the same command without `--testTimeout`. All 6 tests pass.
**Root cause**
- The workload is large. `backlog_51` and `consumed_backlog_52` deliver
51–52 real inbound GitHub messages through the full delivery path.
`bounded pass` inserts and authorizes 100 rejected retry notices.
- CI timing data supports this. On passing runs these tests take
5.4–7.7s (backlog) and 3.6–5.3s (bounded pass), and lighter tests in the
same file take 0.5–1s. On the failing runner, the lighter tests were
2–3x slower (for example, `idle` took 1856ms instead of 500–680ms). This
put the heavy tests over 15s.
- When Vitest times out a test, it does not stop the test body. The body
keeps running while the next tests start:
- The unfinished backlog tests left 51+ `deferred_issue_execution`
inbound wakeups. The pre-run FIFO tests use unscoped, bounded passes
(`enqueueInboundWakeupPublications`, `processPendingPublications`). The
leftover rows filled those pages, so the new test's row was not reached.
- The unfinished `bounded pass` body later ran its `finally` block,
which calls `vi.useRealTimers()`. This disabled the fake timers that
`requester merge` had just installed.
## What Changed
- `preserves one current GitHub Retry across %s`: set an explicit 60s
timeout.
- `reactivates denied retry notices when authority is restored during
%s`: set an explicit 60s timeout. Increase the `bounded pass` wait for
its held 100-row reconciliation from 10s to 30s, so that the inner wait
does not become the new limit.
- The same file already uses explicit 20s, 25s, 30s, 40s, 60s, and 90s
budgets for other heavy tests. No product code changed.
## Verification
- Red: the reproduction command above fails on `master` with the CI
cascade. A tighter `--testTimeout=1600` on `reactivates denied retry
notices` also shows that later modes fail after `bounded pass` times
out.
- Green: the same reproduction command with this change (still
`--testTimeout=1200` for all other tests) passes 10/10 selected tests,
including the three FIFO tests.
- `pnpm exec vitest run src/__tests__/chat-channels.integration.test.ts
-t "GitHub Retry across|reactivates denied retry notices|pre-run FIFO"`:
46 passed.
## Risks
- Low risk. This is a test-only change. A real hang in these two tests
now fails after 60s instead of 15s. This is well inside the job limit.
- The cascade can still occur if one of these tests needs more than 60s.
That would mean a runner about 10x slower than a normal one.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, GitHub CLI). Default context
window. No extended thinking mode.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
ec94e43c4a |
test(server): budget the 130 MB isolated re-home test for its data volume (#19400)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Isolated repository re-home moves a task workspace onto the shared project clone, so the private clone can be deleted > - One re-home test moves two 65 MiB files on purpose, to prove that large changes still move > - On CI this test times out at the 15000 ms default in pull requests that do not touch re-home > - The test does a fixed, large amount of CPU work, and a loaded runner cannot finish it in 15 seconds > - This pull request gives that one test a 60000 ms budget that matches its data volume > - The benefit is a stable server shard, with no change to what the test proves ## Linked Issues or Issue Description **What happened?** A survey of failed `PR` workflow runs found intermittent failures of one test in pull requests that do not touch re-home: ``` FAIL src/__tests__/isolated-repository-rehome.test.ts > isolated repository source re-home > re-homes a workspace whose changes exceed the default restore point limits Error: Test timed out in 15000ms. ``` Examples: jobs 127465954304, 127499682176, and 127538360101 (shard `general-server-without-chat` 7/12). The survey counted 4 such failures. The other 27 tests in the file passed in each of these runs. **Expected behavior** The test passes on every run, and it still proves that a 65 MiB untracked file and a 65 MiB saved ignored file move with an exact snapshot match. **Steps to reproduce** Run the unchanged test on master while the CPU is oversubscribed: ```sh cd server node load.mjs 72 pnpm exec vitest run src/__tests__/isolated-repository-rehome.test.ts -t 'exceed the default' --reporter=verbose ``` `load.mjs` starts 72 busy-loop Node processes (4 per core on an 18-core machine) and runs the command. The test fails with `Test timed out in 15000ms`. **Root cause** The file sizes are part of the test contract. #17410 added the test to prove two fixes that only show above 64 MiB: `applyRestorePoint` streams `git diff --binary` into `git apply`, and the ignored-file read sizes its buffer from the tree. So the files cannot get smaller. Each full pass over the workspace writes both files into a Git object store, so it hashes and deflates 130 MB. One test run makes about eight passes: the test's `before` snapshot, the re-home `before` snapshot, the capture, the import into the shared clone, the apply, the two verification snapshots, and the test's final snapshot. That is about 1 GB of SHA-1 and zlib work, plus the byte comparison. I timed the phases with the re-home hooks on an idle machine: snapshot 0.6 s each, capture and import 1.9 s, worktree add and apply 0.9 s, verification 1.0 s. With 2 busy processes per core the same phases took 1.3 s, 3.9 s, 1.3 s, and 2.1 s. The work is CPU-bound and scales with load. No phase waits on a timer, a lock, or a background task. The 15000 ms default is the cause. ## What Changed - Give `re-homes a workspace whose changes exceed the default restore point limits` an explicit 60000 ms test budget. - No other test changes. No production changes. No retries, no skips. ## Verification Red, on master, 72 busy-loop processes: ``` × ... re-homes a workspace whose changes exceed the default restore point limits → Test timed out in 15000ms. ``` Green, with this change, same load: ``` ✓ ... re-homes a workspace whose changes exceed the default restore point limits 30414ms ``` So the test needs about 30 s at 4 processes per core, which is half of the new budget. A hung re-home still fails inside 60 s. Full file, idle: 28/28 tests pass. ```sh cd server pnpm exec vitest run src/__tests__/isolated-repository-rehome.test.ts ``` ## Risks - Low risk. One test budget changes. - A real slowdown in the re-home path of up to 4 times would no longer fail this test. Other tests in the file keep the 15000 ms default. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9668e73b4b |
fix(native): continue a turn automatically when reattachment loses its runner (#19399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs in managed sandboxes recover through reattachment: after a server restart, and for a same-run retry when a turn ends without a semantic result > - Reattachment fails closed when the original runner or sandbox is gone (`native_remote_recovery_lease_mismatch`, `runner_remote_process_identity_unavailable`) > - Saved-work recovery then saves the files, but the task stays blocked with "Saved work recovered. Send a message to continue." The recovered hold offers no Retry > - A restart, deploy or maintenance window must not need a user message to continue work > - This pull request requests the task Retry automatically, once, when it is safe, and offers Retry on the recovered hold when it is not > - The benefit is that interrupted work continues after infrastructure events, and a user who must act gets a Retry that works on first use ## Linked Issues or Issue Description **What happened?** A native task in a managed sandbox was interrupted. In one case a server restart claimed a queued user-comment turn for reattachment. In another case a turn completed without a semantic result, and the same-run retry claimed the run for reattachment with no restart. In both cases the original runner or sandbox lease was gone. The run failed with `native_remote_recovery_lease_mismatch` or `runner_remote_process_identity_unavailable`. Recovery saved the work. The task stayed `blocked` with "Saved work recovered. Send a message to continue." The execution blocker had `canRetry: false`. A manual `POST /api/issues/:id/retry` started a run that succeeded. **Expected behavior** After an infrastructure interruption, the task continues automatically when the interrupted turn's outcome is known. Otherwise the task offers a Retry that works on first use. **Steps to reproduce** 1. Start a native task in a managed sandbox from a user comment. 2. Lose the sandbox lease or the remote runner process (for example, restart the server during a maintenance window). 3. Let reattachment fail and saved-work recovery complete. 4. See that the task waits for a new message and offers no Retry on the execution blocker. **Root cause** Reattachment (`reattach_remote_runner`) correctly refuses to replace a missing runner or allocation. Nothing then continued the user's request. The automatic continuation rule only allows a new user turn after reconciliation. `getExecutionBlocker` hid Retry whenever a workspace restore hold existed, also after the files were verifiably recovered. The same-run retry path also re-enters the restart-recovery owner, so its event said "after server restart" with no restart. ## What Changed - New `server/src/services/heartbeat/restart-interrupted-turn.ts`: after a reattachment claim executes, if the run failed because its runner or allocation was lost, request the same task Retry that `POST /issues/:id/retry` creates. It runs once per run (idempotency key `restart-interrupted-turn:<runId>`), under the authority of the user who sent the message, assigned the task, or clicked Retry. It requires that the run recorded no provider turn or that its last provider turn is settled. System wakes, in-flight turns, and a Retry that this hook already requested do not continue automatically. - `heartbeat/recovery.ts`: call the hook after each reattachment claim. Label same-run retry reattachment as "Reattaching to the existing remote runner to resume this run", and add `trigger: "same_run_retry"` to the event payload. - `execution-blocker.ts`: a failed run whose saved work is verifiably recovered now offers Retry (`canRetry: true`) with "Saved work recovered. Retry or send a message to continue." An intentional Stop keeps "send a message" only. - `task-retry.ts`: optional `initiatedBy: "restart_recovery"` records the activity as actor `system`/`restart-recovery` with `onBehalfOfUserId`. - `doc/execution-semantics.md`: document the rule. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` use the real recovery, Retry and dispatch services on embedded Postgres: - `continues a user turn once ... after $title and recovery saved the work` (3 cases: restart lease loss before the turn, same-run retry with the runner gone after a settled turn, same-run retry with the lease gone after a settled turn). Each asserts one queued successor bound to the Retry and the user's authority after recovery, idempotent hook calls, system activity attribution, and no second automatic attempt for the automatic Retry's own failure. - `does not continue automatically when $title, and offers a Retry that works on first use` (in-flight turn, system wake, unrelated error). Each asserts no successor, `canRetry: true` on the blocker and the interruption presentation, and one successor after one Retry click. - Red before the fix: all 6 tests fail (no successor, blocker `canRetry: false`). Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts src/services/task-retry.test.ts src/services/task-interruption-presentation.test.ts src/services/execution-blocker-read.test.ts src/services/heartbeat/recovery.test.ts src/services/explicit-native-continuation.test.ts src/services/quarantined-task-retry.test.ts src/__tests__/workspace-restore-repair-route.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts`: 1098 passed. - Not verified on staging. After the next deploy, check a task interrupted by a restart: it should continue without a message, or show Retry. ## Risks - The automatic Retry uses the original user's authority. Admission rechecks that user's current permission, stop proof, saved-work recovery, approvals, pauses, budget and ownership. A refused request leaves the visible Retry. - A settled turn that ended without a result is continued with a fresh Retry turn from saved work. A turn whose outcome is unknown (in flight) is not. - The hook runs after the claim's execution returns. If the server stops between the failure and the request, the task shows Retry instead. - Not in scope: same-run disposition recovery still cannot start a new runner in the existing sandbox after the original remote runner exits. This change continues the task with one fresh Retry turn instead. If that Retry turn also ends without a result and loses its runner, the task shows Retry. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19355 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e7fb554647 |
test(server): load setup-token route modules during suite setup (#19398)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The setup-token login route lets an owner sign in to Claude for a
sandbox environment
> - The route suite `server/src/routes/setup-token-route.test.ts` fails
on CI in pull requests that do not touch it
> - The first test in the suite times out at 15000 ms, and the other 29
tests pass
> - The first test pays the one-time import cost of the agent route
graph inside its own test budget
> - This pull request moves that import to suite setup, which has a
30000 ms hook budget
> - The benefit is a stable server shard with no change to what the
tests prove
## Linked Issues or Issue Description
**What happened?**
A survey of failed `PR` workflow runs found intermittent failures of one
test in pull requests that do not touch this area:
```
FAIL src/routes/setup-token-route.test.ts > company-and-environment setup-token route — full path > drives start, status, prompt, code, and completion with no agent id
Error: Test timed out in 15000ms.
```
Examples: jobs 127423956746 and 127457298585 (shard
`general-server-without-chat` 12/12). The survey counted 5 such
failures.
**Expected behavior**
The full-path test passes on every run. The test budget measures the
route behavior, not the module import.
**Steps to reproduce**
Run the unchanged suite on master while all CPU cores are busy:
```sh
cd server
node load.mjs 24 pnpm exec vitest run src/routes/setup-token-route.test.ts --reporter=verbose
```
`load.mjs` starts 24 busy-loop Node processes and runs the command. The
first test fails at 15005 ms.
**Root cause**
`createApp()` called `vi.importActual("../routes/agents.js")` and four
other modules on each call. The first call loads and transforms the full
agent route graph. I timed it: `createApp()` took 11670 ms of the 11698
ms test on an idle machine. The HTTP steps took less than 30 ms. On a
loaded CI runner the import alone crossed the 15000 ms test budget.
Later tests reused the module cache, so only the first test failed.
## What Changed
- Add `loadRouteModules()`, which imports the route and logger modules.
- Call it once in the suite `beforeAll`, after `vi.resetModules()` and
the mock registration. The hook budget is 30000 ms in
`server/vitest.config.ts`.
- `createApp()` now reuses the loaded modules. Each test still builds a
fresh Express app, a fresh transport, and a fresh log capture.
- This follows the pattern in
`server/src/__tests__/agent-skills-routes.test.ts` (#16396).
- No timeout changes, no retries, no skips.
## Verification
Red, on master, with 24 busy-loop processes on an 18-core machine:
```
× ... drives start, status, prompt, code, and completion with no agent id 15005ms
```
Also red on an idle machine on 1 of 3 runs (15006 ms; the other runs
took 11790 ms and 13748 ms).
Green, with this change:
- Idle, 3 runs: 30/30 tests pass. The first test takes 42 to 46 ms.
- With 24 busy-loop processes: 30/30 tests pass. The first test takes 40
ms. The file takes 19.4 s in total.
```sh
cd server
pnpm exec vitest run src/routes/setup-token-route.test.ts --reporter=verbose
```
`tsc --noEmit -p server` reports no errors in this file.
## Risks
- Low risk. The change touches one test file only.
- The import cost now counts against the 30000 ms hook budget. Locally
it is 12 to 15 s idle. The mocks register before the import, as before,
so each test sees the same mocked services.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
da34bcd48d |
fix(workspaces): resume archived imports from their saved upload (#19396)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Durable workspaces let a replacement web server restore saved task files. > - Legacy workspace imports keep a journal and an encrypted upload receipt across interruption. > - An archived Git import can rebuild a missing source from its saved task commit and edits. > - A fresh clone can also contain newer remote refs. Those refs change the archive manifest even when the task files are unchanged. > - This pull request recovers the exact authenticated upload for an interrupted archived Git import. > - The original journal, source binding, task binding, and execution checks still control publication. ## Linked Issues or Issue Description **What happened?** An archived Git workspace import stopped after its upload persisted. A retry rebuilt the source from the remote repository. New remote refs changed the archive manifest, so the original upload receipt rejected the retry with `remote_workspace_snapshot_operation_content_changed`. **Expected behavior** The retry must recover its original saved upload and complete the same import. It must keep the saved task commit, staged edits, unstaged edits, lifecycle state, and task binding. A changed source or task binding must still block publication. **Steps to reproduce** 1. Archive a Git task with staged and unstaged edits, save its restoration recipe, and remove its local repository. 2. Start the workspace import and lose the upload reply after object persistence, or interrupt its independent restore after the upload is acknowledged. 3. Advance the remote default branch and add an unrelated branch. 4. Retry the original import. Before this change, both interruption cases fail with the same content-change error. **Paperclip version or commit** Reproduced on master at `d8b0eae68772a3bd2bb4a8163b5ce95764b97cff`. **Deployment mode** Cloud workspace migration with external PostgreSQL and durable encrypted object storage. The regression tests use real disposable PostgreSQL, an HTTP Git remote, and encrypted filesystem-backed object storage. Related merged work: #18826 and #19138. No duplicate open fix was found. ## What Changed - Add scoped recovery of an acknowledged snapshot or a pending upload whose object persisted. Authenticate the archive before returning it. Recovery alone does not publish the snapshot. - Resume archived Git imports from those exact saved bytes. Keep the original operation and all publication checks. - Lock and recheck each saved nested source’s company, project, and repository identity in the final publication transaction. - Keep mutable local sources on the existing capture and content-comparison path. - Add regressions for both interruption points, unrelated remote changes, source revocation during recovery, and local edits after interruption. - Document the retry behavior. ## Verification - Before the repair: both new archived retry regressions fail with `remote_workspace_snapshot_operation_content_changed`; the mutable-source control passes. - After the repair: all 204 tests in the migration, snapshot storage, archive, and archive environment suites pass. The suites use real PostgreSQL and encrypted storage. - Server package typecheck and full repository typecheck pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. - An added source-revocation test initially expected a later expiry error. The existing publication fingerprint rejected the change first. The test now requires that earlier `remote_workspace_migration_publication_changed` error and still checks that authority was not published. - The review finding was reproduced before the safety repair: changing a saved nested repository URL after upload still allowed publication. The new regression requires rejection, and unchanged approved nested sources still restore. - Full repository typecheck and build pass on the final head. The full local test command stopped at one Slack manager authorization fixture with `socket hang up`; the exact case then passed in isolation on the same head, and all three CI chat shards passed. The original local failure is retained. - The first CI server shard run had 4,724 passes and one runtime-process readiness failure in an unchanged fixture. The exact fixture passed in isolation. One unchanged rerun of that shard passed. The first failure remains recorded in the workflow history. - The earlier candidate’s full local test run was stopped after the confirmed review finding; that interrupted run is not reported as a pass. - All CI gates, including `ci / verify` and `ci / e2e`, pass on `bffebbab60edbdf50b42fc37a9e58beb01ceb94d`. Greptile is 5/5 on that head, with no remaining findings or unresolved review threads. ## Risks Recovery applies to archived Git sources, whose saved recipe and lifecycle checks still run before publication. A source change during recovery must keep the snapshot unpublished. Missing or corrupt saved objects still fail authentication or retain the existing import obligation. Mutable local sources still reject changed contents. This change adds no database migration and changes no saved archive format. ## Model Used OpenAI GPT-6 in Codex, with code editing, tool use, and test execution. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
5c185db77c |
fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.
## Linked Issues or Issue Description
**What happened?**
A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.
**Expected behavior**
A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.
**Steps to reproduce**
1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.
**Root cause**
- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.
## What Changed
- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.
## Verification
- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
- A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.
## Risks
- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
240d46e2ec |
fix(native-runtime): join the deferred observer seen when a duplicate append arrives (#19397)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime writes runner events through
`PaperclipControlPlanePort.appendEvent`.
> - A terminal event can defer its post-commit observer. A duplicate
append of the same event must join that observer. It must not run
`onDuplicateEvent`.
> - The duplicate looked up the observer only after its own event
transaction finished. If the original observer settled during that
transaction, the observer left the map. Then the duplicate ran
`onDuplicateEvent`.
> - This made the test "joins the exact deferred observer on duplicate
append..." fail on slow CI (job 127633060467, and once on Windows).
> - This pull request captures the pending observer when the duplicate
arrives. The join decision no longer depends on database latency.
> - The benefit is a deterministic join and a stable test.
## Linked Issues or Issue Description
**What happened?**
The server test `paperclip-control-plane-port.test.ts > joins the exact
deferred observer on duplicate append and replays it after a new port
opens` failed intermittently with `Error: must join the original owned
observation`. The test gives the duplicate append 30 ms and then
releases the original observer. On a slow runner, the duplicate's event
transaction took longer than 30 ms. The original observer settled and
its `.then` handler removed the map entry. The duplicate then found no
observer after its transaction and ran `onDuplicateEvent`. (The CI
failure took 90 ms.)
**Expected behavior**
A duplicate append that arrives while the original deferred observer is
pending joins that observer. Database latency does not change this
result.
**Steps to reproduce**
1. Add a 60 ms sleep inside the `appendEvent` transaction when the
receipt disposition is `duplicate`.
2. Run `pnpm vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts -t
"joins the exact deferred"` from `server/`.
3. The test fails with `must join the original owned observation`. This
is the same error and stack as CI.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
958f9aa9f3 |
feat(issues): limit agent-created task trees to parent and child (#19312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
4033f7f17f |
fix(recovery): let a task Retry run keep its bounded transient retry (#19384)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can click Retry on a blocked task. Paperclip then admits one new run that continues in the saved workspace > - That run can fail before the provider starts, for example when the download of the saved workspace snapshot gets a transient `ECONNRESET` > - The scheduler classifies this failure as transient and tries to schedule the normal bounded retry > - The automatic-retry admission accepts only continuations that a user comment started. It refuses a Retry continuation (`commentId: null`, `taskRetryRequestId` set) with `continuation_user_authorization_missing` > - So the task does not retry. Paperclip escalates the restore failure, and the task becomes blocked again after one network reset > - This pull request lets a plain task Retry renew its authority for the bounded transient retry, with its own receipt for each new run > - The benefit is that a transient setup failure after Retry uses the normal retry budget and does not block the task ## Linked Issues or Issue Description **What happened?** A run started by the task Retry button failed during setup: `remote_workspace_restore_failed: step=snapshot_download; storage=transport_closed code=ECONNRESET`. The run event said "The automatic retry could not revalidate the original user continuation." (retry reason `transient_failure`, attempt 1 of 2). No retry was scheduled, and the task became blocked. **Expected behavior** A transient setup failure of a Retry-admitted run gets the same bounded automatic retry as a run that a user comment started. **Root cause** `admitExplicitContinuationRetry` in `server/src/services/explicit-native-continuation.ts` required `explicitUserContinuation.commentId` to be a GUID. A Retry continuation has `commentId: null`. Also, dispatch in `server/src/services/execution-continuation.ts` bound a Retry receipt to exactly the run that it admitted, so a successor run could not prove its authority. Refs #19207 (it named remote restore failures and left a board recovery action) Refs #19340 (it retries quarantined tasks in their saved workspace) Refs #19326 (same bounded retry path for an out-of-date Runner) ## What Changed - `admitExplicitContinuationRetry` now accepts two kinds of authority. The comment path is unchanged and moved into `readCommentRetryAuthority`. The new `readTaskRetryRetryAuthority` accepts a plain task Retry only. The Retry request must be coalesced, from a user, with the same generation and actor, strategy `continue`, and no chat, queued-message, canonical-intent or fresh-workspace readiness. Interrupt and failed-run retries stay one-run. - The successor run gets its own `explicit_user_continuation_retry` receipt with the same `taskRetryRequestId` and actor and `automaticRetry.sourceRunId` set to the parent. The Retry request stays bound to the run that it admitted. - Dispatch (`buildExecutionContinuation`) accepts a Retry successor only from its own receipt, its exact `retryOfRunId` parent, the current `executionRunId` and the unchanged Retry intent. The direct path for the admitted run is unchanged. - Tests in `server/src/services/task-retry.test.ts`: one regression for the full retry chain, and eight refusal cases. ## Verification - Red: before the fix, the new test failed with `{ outcome: "not_scheduled", errorCode: "continuation_user_authorization_missing" }`. This is the same refusal as the incident. - Green: `pnpm --filter @paperclipai/server exec vitest run src/services/task-retry.test.ts` gives 58/58. - The regression test proves these points: the first retry is scheduled with `{ previousRunId: parent, commentId: null, taskRetryRequestId }`; the Retry request stays bound to the admitted run; dispatch accepts the successor; a repeated schedule call adopts the same successor without a second receipt; a second transient failure retries from its immediate parent; the budget then stops with `retry_exhausted`; dispatch refuses the successor when its own receipt is removed. - The refusal cases prove that no successor and no receipt are created after: wrong actor, unadmitted request, wrong generation, request bound to another run, chat Retry, fresh-workspace readiness, reassignment, superseding work. - Nearby suites pass (601/601): `execution-continuation`, `legacy-explicit-continuation-recovery`, `explicit-native-continuation`, `execution-continuation-provenance`, `quarantined-task-retry`, `issue-queued-comments-routes`, `task-retry`. - I did not run the layered server typecheck locally because of a disk limit in the sandbox. CI must confirm it. ## Risks - Behavior change: a plain task Retry run that fails transiently now retries automatically within the existing bounded budget (2 attempts). The scheduler still decides which failures are transient. The same rule already applies to runs that a user comment started. - Chat, queued-message and fresh-workspace Retries, failed-run retries and Interrupt keep their one-run contracts. - No schema or migration change. - Documentation gap: `doc/execution-semantics.md` (line 1210) says "One-run Retry and Interrupt intents retain their separate admission contracts; they are not renewable user-message authority." That sentence must change to describe this rule. The author's publication path cannot carry the full 320 KB file, so that one-sentence edit is still pending. The proposed text: "A plain task Retry that continues in the saved workspace renews the same way: the successor's receipt names the same Retry request and actor, while the Retry request itself stays bound to the run it admitted. Dispatch accepts that successor only from its own receipt, its exact retry parent, and the unchanged admitted Retry request, so a transient failure before or during that run, such as a reset saved-workspace download, uses the normal bounded retry budget instead of blocking the task. Chat, queued-message and fresh-workspace Retries, failed-run retries, and Interrupt intents retain their separate one-run admission contracts; they are not renewable authority." > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code as a Paperclip agent, with tool use and code execution (shell, Vitest on embedded PostgreSQL). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (the `doc/execution-semantics.md` sentence above is pending) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
8f79a32898 |
fix(connections): keep agent access consistent through setup and permission changes (#19352)
Task-hosted tool setup kept requester-only defaults, and audience edits could leave runtime installations restricted after permission bindings were widened. Large provider catalogs could also fail before creating an access card. Use the normal agent audience defaults for new tool setup, preserve explicit selections through verified OAuth and draft resumption, and synchronize permission bindings and runtime installations atomically. Enforce agent-edit authority and preserve existing access during additive account reuse. Validation: 915 focused regression tests, typechecks, and builds passed. All 54 CI checks passed and Greptile Apex scored the reviewed head 5/5 with no unresolved threads. Broad local test failures, recoveries, and incomplete heartbeat coverage are documented in the PR. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
36643d9378 |
fix(native): retire finished owners whose continued turn rotated authority (#19373)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner runs per-turn agent sessions in managed sandboxes.
A controller keeps a durable session root for each session.
> - When a per-turn owner finishes without a saved runner checkpoint,
the next turn retires that owner. Retirement first checks that the
session root holds only known entries.
> - The runner transport writes two entries that the check does not
know: `authority-epochs/` and `runtime-context.json`. A continued turn
writes `authority-epochs/` when it rotates authority from the turn
before it.
> - So the third turn of an ephemeral-lease task cannot retire the
second turn's owner. The run fails with
`native_continuation_requires_reconciliation` and the task waits for a
user message.
> - This pull request classifies every session-root entry in one
contract module, and adds a test that scans the writers so the list
cannot drift again.
> - The benefit is that follow-up turns continue without manual help,
and any future refusal names the unexpected entry.
## Linked Issues or Issue Description
Refs #18846, Refs #19061, Refs #19355
**What happened?**
On staging, the third turn of every task for one agent (gpt-5.6-sol,
native runner, ephemeral sandbox lease) failed. The error was
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=unexpected_root_entry`. The first two turns
succeeded. The task then went to `blocked` with "Saved work recovered.
Send a message to continue."
**Expected behavior**
The third turn retires the finished second-turn owner and continues the
conversation.
**Steps to reproduce**
1. Assign a task to a native agent that uses an ephemeral sandbox lease.
2. Let the first turn finish. Reopen the task with a comment so a second
turn runs and finishes.
3. Reopen the task again. The third turn fails with the error above.
**Paperclip version or commit**
Staging near 9a79384958; reproduced locally on master
|
||
|
DottaandClaude Opus 5.5
|
de1143448b |
fix(queue): clear interaction receipts a run already delivered (#19376)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask people for input through thread cards: questions, confirmations, plan approvals and connection requests > - When a person resolves a card, Paperclip queues the response for the agent. The task chat shows it above the composer as a read-only receipt, for example "Answered: Questions" > - A second wake can carry the same receipt. One example is a recovery continuation that raced the response's own wake. The agent receives the response from the first run, but the second wake stays deferred > - The composer then shows a receipt that the agent already has. The user cannot edit, discard or clear it. The same receipt also blocks the agent's handback with `interaction_response_queued` > - This pull request treats a receipt as pending only until a run that carries the interaction has started > - The benefit is that an answered card leaves the queue once the agent has it, for every card kind ## Linked Issues or Issue Description No public issue exists. A hands-on onboarding test on a staging cloud workspace (Claude subscription path) found this bug. **What happened?** During onboarding, a queued item stayed above the task-chat composer. It said "Answered: Questions", but the user had already answered the questions. Later items also stayed after the interview answers and after the user connected GitHub. The user could not trash, steer or edit them. Evidence from the staging workspace (interactions, wake diagnostics and run records for the onboarding task): - The first questions card (no title, so its label is "Answered: Questions") was answered at 17:37:48. - Run `ba3bc7d6` (`issue.interaction.respond`) delivered that answer from 17:37:48 to 17:38:27. - A recovery wake (`issue.interaction_continuation_recovery`) for the same card was requested at 17:37:48 too. It stayed deferred behind `ba3bc7d6` and then ran again as `cacb095d` (17:38:33 to 17:39:17). During that time the composer showed the stale "Answered: Questions" receipt. - The plan approval had the same pattern from 17:42:06 to 18:20 (fixed at its source by #19332). The declined Slack connection receipt sat behind that chain. **Expected behavior** A receipt leaves the composer queue as soon as a run has received it. The agent can hand back the task after it has received the response. **Steps to reproduce** 1. An agent asks a question card on a task. 2. A user answers it. The response wake starts a run that carries the answer. 3. A second wake for the same card is deferred behind that run or a later run. 4. `GET /api/issues/:id/queued-comments` returns the second wake's receipt as an immutable entry until it runs. ## What Changed - `server/src/services/queued-interaction-response.ts`: `readQueuedInteractionResponse` returns no receipt when a run for the same issue has started with this interaction in its context and is `running` or `succeeded`. A cancelled or failed run does not count, so that response is still owed. `hasQueuedInteractionResponse` (the handback guard) uses the same rule. - `server/src/routes/issues.ts`: `findQueuedCommentWake` skips a delivered receipt, so a later undelivered receipt can still show. - `server/src/__tests__/issue-queued-comments-routes.test.ts`: red/green tests for every card kind and outcome: answered questions, accepted and rejected confirmation, accepted plan (checkbox) approval, connected and declined connection. Each runs with a `succeeded` and with a `running` delivering run. Controls check that an undelivered receipt still shows, that a later undelivered receipt shows instead of a delivered one, and that the handback guard ignores a delivered receipt. - N-02 (hover shows raw JSON): `master` already shows only the one-line summary in the label and hover (`queuedMessageLabel`). This PR does not change the UI. ## Verification Red, on `master` code with only the new tests added: ``` $ cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "receipt" × clears a ask_user_questions answered receipt once a succeeded run already delivered it 1309ms × clears a ask_user_questions answered receipt once a running run already delivered it 1041ms × clears a request_confirmation accepted receipt once a succeeded run already delivered it 1436ms × clears a request_confirmation accepted receipt once a running run already delivered it 829ms × clears a request_confirmation rejected receipt once a succeeded run already delivered it 666ms × clears a request_confirmation rejected receipt once a running run already delivered it 776ms × clears a request_checkbox_confirmation accepted receipt once a succeeded run already delivered it 1179ms × clears a request_checkbox_confirmation accepted receipt once a running run already delivered it 4227ms × clears a connection_intent accepted receipt once a succeeded run already delivered it 1954ms × clears a connection_intent accepted receipt once a running run already delivered it 1634ms × clears a connection_intent rejected receipt once a succeeded run already delivered it 1505ms × clears a connection_intent rejected receipt once a running run already delivered it 1064ms × shows a later undelivered receipt instead of an earlier delivered one 480ms × does not hold agent handback for a receipt the agent already received 1272ms ⎯⎯⎯⎯⎯⎯ Failed Tests 14 ⎯⎯⎯⎯⎯⎯⎯ AssertionError: expected [ 'Answered: Questions' ] to deeply equal [] AssertionError: expected [ 'Accepted: Approve hiring 2 agents' ] to deeply equal [] AssertionError: expected [ 'Rejected: Approve hiring 2 agents' ] to deeply equal [] AssertionError: expected [ Array(1) ] to deeply equal [] AssertionError: expected [ 'Connected: GitHub connection' ] to deeply equal [] AssertionError: expected [ 'Declined: Slack connection' ] to deeply equal [] AssertionError: expected [ Array(1) ] to deeply equal [ 'Declined: Slack connection' ] AssertionError: expected true to be false // Object.is equality Tests 14 failed | 11 passed | 85 skipped (110) ``` Green, with the fix: ``` $ cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts -t "receipt" Tests 25 passed | 85 skipped (110) ``` Neighbouring suites, with the fix: - `issue-queued-comments-routes.test.ts` (full file, on current `master`): 110 passed. - `question-response-delivery`, `queued-comment-consumption`, `issue-thread-interaction-routes`, `satisfied-connection-intents`, `issue-comment-reopen-routes`, `issue-execution-policy-routes` together with the queued-comments file: 494 tests passed. One run hit an embedded-Postgres `afterAll` cleanup timeout under 7-file parallel load. The same file passed alone twice. - `heartbeat-process-recovery.test.ts -t "plan approval|accepted interaction|answered question|answered interaction|onboarding first task"`: 14 passed. - `pnpm --filter @paperclipai/server typecheck`: passes. ## Risks - Low risk. The change only hides a receipt that a started run already carries. An undelivered receipt keeps its current behavior. - The redundant wake itself is not cancelled here. It still runs later as a normal continuation, but it no longer holds the composer queue or blocks handback. #19332 removes the main source of these duplicate wakes. - Each receipt read adds one indexed-by-company lookup on `heartbeat_runs` for the issue. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, browser inspection of the staging workspace API). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9ceaec6f97 |
fix(issues): stop showing resource waits as failed attempts (#19369)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task page shows one interruption banner. The server builds it
from the task's latest run.
> - An auto-retry that waits for a busy workspace, AI connection or
runner update is not a failure. The dispatcher re-checks the resource,
so the run can stay `scheduled_retry` after its retry time.
> - The projection did not know about these waits. After the retry time
passed, it fell through to "Couldn't finish this attempt" with a Retry
button.
> - Retry cannot free the resource. The banner cleared for the 10-second
grace and then came back. Users had no way to fix it.
> - This pull request projects a scheduled resource wait as `working`,
with no Retry.
> - The benefit is that the failed-attempt banner shows only for real
failures.
## Linked Issues or Issue Description
**What happened?**
During onboarding on Paperclip Cloud staging (Claude subscription and
Codex subscription paths), the onboarding task showed "Couldn't finish
this attempt. Your message is saved." with a Retry button. The banner
did not name an attempt. Retry cleared it briefly, then it came back.
**Steps to reproduce**
1. Start a task run while another run holds the same workspace.
2. The run is cancelled with `workspace_busy`, and a `scheduled_retry`
run is created with `scheduledRetryReason: "workspace_busy"`.
3. Keep the workspace busy after `scheduledRetryAt`. On the staging
company, one retry stayed scheduled for 16 minutes past its retry time.
4. Open the task. The failed-attempt banner shows. Click Retry. The
banner comes back.
**Expected behavior**
No failed-attempt banner while the system waits for a shared resource.
The run status already says "Waiting for workspace"
(`execution-projection.ts`).
## What Changed
- `server/src/services/task-interruption-presentation.ts`: a
`scheduled_retry` run whose `scheduledRetryReason` is `workspace_busy`,
`ai_connection_busy`, `ai_connection_pool_wait` or
`runner_revision_wait` (and that has no system recovery) projects as
`state: "working"`, `canRetry: false`. User gates and failed Retry
preparations stay authoritative because they are checked first.
- The batched query now selects `scheduledRetryReason`.
- Tests: a Postgres integration test (`task-interruption-query.test.ts`)
that seeds the staging run chain for each reason, and unit tests for the
guard cases (failure-lane retries, paused agent, failed Retry
preparation).
## Verification
Red on `master` (fix not applied), `cd server && npx vitest run
src/__tests__/task-interruption-query.test.ts -t "overdue"`:
```
× does not present an overdue workspace_busy wait as a failed attempt 213ms
× does not present an overdue ai_connection_busy wait as a failed attempt 25ms
× does not present an overdue ai_connection_pool_wait wait as a failed attempt 13ms
× does not present an overdue runner_revision_wait wait as a failed attempt 11ms
- Expected
+ Received
- "canRetry": false,
+ "canRetry": true,
- "state": "working",
+ "state": "failed",
Tests 4 failed | 11 skipped (15)
```
Green with the fix, same tests plus the new unit tests:
```
✓ task-interruption-query.test.ts > does not present an overdue workspace_busy wait as a failed attempt
✓ task-interruption-query.test.ts > does not present an overdue ai_connection_busy wait as a failed attempt
✓ task-interruption-query.test.ts > does not present an overdue ai_connection_pool_wait wait as a failed attempt
✓ task-interruption-query.test.ts > does not present an overdue runner_revision_wait wait as a failed attempt
✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps an overdue workspace wait out of the failed-attempt UI
✓ task-interruption-presentation.test.ts > scheduled resource wait presentation > keeps failure retries, user gates and failed Retry preparation authoritative
```
Neighbouring suites: `task-interruption-presentation`,
`task-interruption-query`, `workspace-recovery`, `agent-project-access`,
`heartbeat-dependency-scheduling`: 5 files, 458 tests passed.
Server `tsc --noEmit` reports no errors in changed files. The only
errors locally come from the unbuilt `@paperclipai/paperclip-runner`
vendor package, which this change does not touch. No UI change, so `pnpm
check:token-gates` is not applicable.
Evidence: the staging onboarding task's run history shows
`workspace_busy` cancellations at 17:46, 17:52 and 18:10, each followed
by a `scheduled_retry`. One had `scheduledRetryAt` 17:53:57 and started
at 18:10:20. The next run succeeded at 18:20.
## Risks
- Low risk. Presentation only; dispatch and admission do not change.
- If a resource wait never clears, the task shows no failed-attempt
banner. The run status still shows "Waiting for workspace", and the
existing retry-attempt limits still end the wait.
- Not in scope: genuine failures without a provider or workspace
explanation still show the generic text, and an overdue failure-lane
retry still shows as failed.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser for read-only staging evidence).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
7440cfc9b1 |
fix(issues): accept queued Send while the previous run has not started (#19370)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task chat queues a message while the agent is busy. The
queued-message bar shows a Send button when no turn is running.
> - Send calls `POST /issues/:id/queued-comments/interrupt` with no
target run.
> - The issue can hold an execution run that has not started yet
(`queued`) or that waits for a retry (`scheduled_retry`). The GET
response then shows a deferred queue with no target run, so the UI shows
Send.
> - The interrupt route refused that click with 409. The UI then showed
"Couldn't send. Message is still queued."
> - This pull request saves the click as interrupt intent in that state.
The queue is delivered when the previous run stops.
> - The benefit is that the button the UI offers now works, and the
message goes next.
## Linked Issues or Issue Description
**What happened?**
Cloud onboarding test, Codex subscription path (N-06): the tester
clicked Send in the onboarding task. The UI showed "Couldn't send.
Message is still queued."
**Expected behavior**
Send is accepted. The message stays in the queue and is delivered next,
when the previous run stops.
**Steps to reproduce**
1. Have an issue whose execution run is `queued` (not started) or
`scheduled_retry`.
2. Post a message. It is deferred behind that run.
3. Click Send on the queued message.
4. The route returns 409: "This queue does not support interruption"
(paperclip_runner) or "The previous run has not stopped" (legacy
adapters).
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandClaude Opus 5.5
|
33269e1fec |
fix(observability): report directory-merge lock holder diagnostics (#17755)
A directory-merge lock timeout (ERR_WORKSPACE_RESTORE_LOCK_TIMEOUT) already carries allowlisted holder evidence, but server Sentry capture dropped it. - Add directoryMergeLockTimeoutDiagnosticsFor in adapter-utils: closed labels, booleans and bounded durations; no owner PID or paths. - server captureException: lock timeouts get an event-local directory_merge_lock context and error_code=workspace_restore_lock_timeout tag; default grouping and message unchanged. - run-failure-diagnostics restoreLock* fields use the shared reader (same allowlist and order). - Document the record in doc/workspace-restore-locks.md. Diagnostics only; does not change lock acquisition, timeouts or retries. Refs PAPERCLIP-BACKEND-4Q Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
e6c74982fe |
fix(workspaces): report why a workspace provider operation failed (#19362)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can store its task workspaces with a workspace storage
provider plugin. Core calls the plugin's `workspaceOperation` method to
prepare, inspect, finalize and remove a workspace.
> - `callWorkspaceProvider` catches every rejection of that call and
throws the same conflict: "Workspace storage operation is incomplete;
retry the same workspace".
> - A provider error, an RPC timeout, a crashed worker and a worker that
never got the call all look the same in the run, the log and the API.
The message tells the user to retry, also when the failure is a fixed
configuration problem that a retry cannot fix.
> - In one diagnosis, three different configuration failures in a row
showed this same message. Finding each cause took trace span counts and
database reads.
> - This pull request lets a provider send a reason from a closed set,
and makes Core tell transport failures apart from provider failures.
> - The benefit is that a failed run says what failed and whether a
retry can help. Provider text and values still never cross into Core.
## Linked Issues or Issue Description
**What happened?**
A workspace provider's `workspaceOperation` failed. The run failed with
"Workspace storage operation is incomplete; retry the same workspace",
`errorCode: setup_failed`, and no reason in the run record or the
`heartbeat execution setup failed` log. Retrying did not help, because
the cause was a provider configuration problem.
**Expected behavior**
Core tells these cases apart: the worker did not receive the call, the
call timed out, the worker stopped during the call, or the provider
reported a failure. For a provider failure, Core shows a bounded reason.
Core tells the user to retry only when a retry can help.
**Steps to reproduce**
1. Install a workspace storage provider plugin whose
`onWorkspaceOperation` throws during `prepare` (for example, because of
missing provider settings).
2. Run an agent task in a project that uses that provider.
3. See the failed run: the error is the generic retry message, and the
run record and logs do not show the cause.
**Paperclip version / commit**
`master` at
|
||
|
|
f2593b7905 |
feat: make Coordinator and Agent Chat standard and speed up navigation (#19360)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People enter that work through the workspace, agent conversations, and tasks. > - Coordinator and Agent Chat still require experimental enrollment. > - Task link prefetches do not warm the compact queries used by the default task view. Chat also waits for a request when its navigation list already holds the conversation. > - This pull request makes Coordinator and Agent Chat standard and reuses the reads needed for navigation. > - People can choose Classic workspace in Experimental settings. Cached work can appear while fresh data loads. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace selection, Agent Chat availability, and navigation between Chat, Tasks, and task details. **Subsystem affected** Cross-cutting: UI, workspace preferences, instance settings, and the database migration. **Current behavior** People must enable Coordinator and Agent Chat in Experimental settings. Opening a task can repeat requests because prefetch warms the full detail and comment queries, while the default view reads compact task data and a recent comment window. Opening Chat waits for a detail request even when the current person's conversation list contains it. **Proposed behavior** Coordinator is the standard workspace for everyone. Experimental settings offer Classic workspace, off by default. Old Coordinator experiment values do not select Classic. Agent Chat is available to everyone, including instances with an old saved or managed off value. Task link intent warms the queries used by the mounted view. Chat can use the current person's company-scoped conversation list during refresh. **Reason and benefit** People can use the standard workspace without setup. Repeat navigation can show cached content while the server refreshes it. **Breaking changes** Existing Coordinator experiment and Agent Chat off values no longer disable these surfaces. Classic requires an explicit new opt-in. The workspace API keeps its existing response shape and audit checks. The migration retains legacy values and adds a separate Classic preference. Execution permissions, company access, and chat privacy checks still apply. **Additional context** Roadmap check: this graduates existing surfaces and improves their navigation. It does not add a new agent execution capability. Duplicate searches found no open Coordinator workspace PR. Related server cache work, Refs: #17828, addresses compact issue-list invalidation; this PR addresses client navigation queries. ## What Changed - Add a default-off Classic workspace preference. Derive the existing workspace API response from that choice. Keep personal persistence, account isolation, and audited writes in the workspace preference service. - Replace the Coordinator experiment control with Classic workspace. Use Coordinator while preferences load. Update the existing settings examples and navigation recipe. - Retire the Agent Chat experiment control. Ignore old stored and managed off values. Retain the compatibility key and the existing execution and access checks. - Reuse account- and company-scoped chat list data with its original freshness timestamp. - Share compact task and recent-comment query options between prefetch and the task view. Keep full-history prefetch for Classic tasks and expanded history. - Add upgrade, replay, cache, account isolation, and browser switching regressions. Update product, database, development, and roadmap documentation. ## Verification - Targeted preference, settings, migration, cache, and UI regression tests passed during implementation. The cache tests hold refresh requests pending and check that content still renders. They also check account isolation and stale refresh behavior. - A real browser test verified fresh Coordinator defaults, Classic opt-in persistence, and return to Coordinator. It used an isolated source checkout with fresh data on macOS. - Token, migration numbering and safety, map, and Storybook catalog checks pass after the rebase. - Repository typecheck and build pass. Typecheck and token gates also pass after review repairs. The broad local run passed all six chat integration shards (1,931 tests). It then found the host-CLI-dependent recovery fixture described below. Its remaining duplicate work was stopped after the full current-head CI matrix passed; this is not a claim that the broad local command completed green. - Review repairs passed 419 UI/migration fixture tests, 19 Coordinator navigation tests, and all 214 tests in the five affected server suites. The 24 Chat/workspace browser tests pass, including standard Coordinator switching and explicit Classic persistence. - The broader browser repair run passed 85 tests. It retained two failed assertions and skipped five later serial cases. The affected files then passed all 11 tests after the fixes: reopen the desktop panel after reload and open More for Audit and Connectors. This includes the five serial cases that had not run. - The broad local suite exposed a recovery fixture that executed the host Codex CLI during a simulated remote probe. On a host with Codex 0.154.0, model fallback broke its fixed-model assertion. The fake sandbox now reports Codex 0.160.0. All 20 selected recovery/provider compatibility cases pass after the repair. - A CLI company import/export suite timed out during server startup in the second CI run. The port fallback is intentional in that fixture. No root cause was established. That suite passes on the final commit in CI. This records recovery, not a claimed product fix for the earlier timeout. - The first CI run exposed remaining assumptions about the retired flags and Classic UI. The repair commit updates those fixtures and navigation steps. It also restores the missing enabled Cases destination in Coordinator. Earlier local recovery attempts used the wrong fixture notification and advanced fake time during live execution; the corrected tests use the subscribed recovery queue and wall time during execution. - Greptile reviewed commit `98972b030d4505d838427a2fca6b196b8cad6d47` at **5/5** with no new comments. The original finding is resolved. - Current-head CI is green: **56 successful checks**, with three checks skipped by workflow filters. This includes typecheck, build, the complete Vitest matrix, all eight browser shards, Runner verification, release checks, and security checks. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42132651414). - Typecheck caught missing `pid` and `startedAt` fields in the simulated command result. The fixture now returns the complete process contract, and the server test-layer typecheck passes. - Storybook catalog alignment was checked. Storybook play tests were not run. Hosted latency and cold bundle loading were not measured. ## Risks - The new default changes the workspace for people who previously left the Coordinator experiment off. They can opt into Classic in Experimental settings. - The additive migration must run before the new server code. It is safe to replay and preserves explicit Classic choices. - Cached content can be stale during background refresh. Existing query invalidation and source timestamps still control freshness. - These changes reduce avoidable loading waits. They do not establish a measured improvement in hosted request latency or initial bundle download time. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and browser automation. The exact serving revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
1a36195868 |
fix(storage): name the S3 read when a response body is cut off (#19284)
Record read_object_body with HTTP status, S3 request id and attempts when a GetObject body stream errors after send() resolved, and add the storage fields to run failure execution diagnostics. Diagnostics only; no retry behavior changes. |
||
|
Devin FoleyandPaperclip
|
8080620fc9 |
fix(workspaces): size archive deadlines and preserve resume budgets (#19364)
Give large workspace archives a bounded per-transfer deadline without changing provider settings. Preserve configured post-upload command limits and the saved provider budget when resuming an exact retained sandbox. Verification: 341 focused Core/SDK/adapter tests and 588 Daytona tests passed. Workspace and provider typecheck/build passed. Hosted CI: 54 successful checks, two intended Storybook skips. Greptile 5/5 with no unresolved review threads. The bounded full local run passed all six chat shards and stopped at twenty minutes during general server suites; that full local run remains incomplete. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a20dfbc695 |
fix(recovery): continue a message sent during Stop settlement after recovery (#19355)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
7c0922a7b6 |
fix(native): preserve provider backups during slow workspace saves (#19359)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions save provider state so another web server can continue the same conversation. > - A provider can suspend while its accepted result is still saving a large workspace. > - The run has released its live lease at that point. A new backup-owner lookup rejects this uncommitted state. > - This pull request captures the real owner during backend creation and retains it with the current session authority. > - The existing database checks still reject a changed boot, generation, result, or successor before inspection and during publication. > - The benefit is that a slow workspace save no longer prevents the provider backup. ## Linked Issues or Issue Description **What happened?** A remote session can fail to save its provider backup with `native_harness_storage_owner_missing`. The provider has suspended, but its accepted result is still in `workspace_finalizing`. The close callback tries to acquire a new owner after the run released its lease. A later allocation cannot use a backup that was never published. **Expected behavior** Save the suspended provider with the authority acquired during execution. Recheck that authority before provider inspection and before and after upload. Keep result finalization and provider-work permissions unchanged. **Steps to reproduce** 1. Start a remote native session with durable provider storage. 2. Accept its result and release the live lease while its workspace save remains pending. 3. Close the provider transport. The old owner lookup rejects the save before provider inspection. **Paperclip version or commit** Reproduced on master `bf10bc695abaef6b20e85b287f456b5ce2edb266`. **Deployment mode** A web server with object storage and remote managed Runner sessions. Related: #19225 retains failed-turn ownership through close. #18436 restores an external controller checkpoint before a changed warm process is replaced. This change fixes late backup-owner acquisition while an accepted result saves its workspace. ## What Changed - Capture the provider backup owner when the remote backend still owns its execution lease. - Bind that owner to the current session authority. An older close callback selects the current warm attachment's owner. Revalidate the captured fence before inspecting or skipping provider state. - Add a regression for the close callback during workspace finalization. - Restore encrypted provider files with a captured owner while workspace finalization remains pending. Reject boot, generation, result, and successor changes during upload. - Document the ownership contract. ## Verification - Observed the callback regression fail on the old code with `native_harness_storage_owner_missing`. The same test passes with this fix. - `cd server && node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 59 passed, using real PostgreSQL and encrypted object write/readback. - `node scripts/check-module-boundaries.mjs`: passed. - Full workspace typecheck: passed with the complete production fix. - Native execution: 1,151 passed. Managed warm lifecycle: 9 passed. These tests require a lost-owner error during shutdown, including before an incomplete provider checkpoint is skipped. The final change to the lifecycle test also requires rejection after lease renewal is lost. - Full local `pnpm build`: passed on the final head. - The full local `pnpm test:run` stopped in chat shard 6 on five existing deadline assertions. Earlier chat shards passed; the PR CI chat shards passed. This is not a full local-suite pass. - The first CI server shard 4 attempt failed the unchanged legacy environment-inheritance fixture at its five-second completion wait. The same failure reproduced locally on the repair head and on an earlier checkout without this repair. One unchanged CI retry passed. No fixture assertion or timeout change is shipped. - Latest-head CI has passed. Greptile reviewed the final head at 5/5 with no open threads. The separate Paperclip Review is paused. This PR does not claim staging acceptance. ## Risks The backup owner now lives with its session authority. A wrong authority selection must fail rather than publish provider state. Database publication still checks the original company, task, run, boot, generation, and accepted result before and after upload. Tests change the owner during upload and verify that the selected backup stays intact. No schema, data migration, or new provider-work permission changes. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning and code execution tools. The host does not expose the exact runtime model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
0f2c01b63a |
fix(server): retry one Cloud portfolio connection reset (#19033)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On Cloud instances, `GET /api/cloud/stacks` gets the user's stack portfolio from the Cloud control plane. > - The route sends one `fetch` to `/v1/tenant/portfolio`. It has a 10 s deadline. > - The Cloud edge can reset a pooled keep-alive connection when the request is written. The request then fails in a few milliseconds and the user gets a 502. > - The diagnostics from #15340 now show this failure mode on staging. > - This pull request retries the idempotent GET one time for a pre-response connection reset, inside the same deadline. > - The benefit is that a transient connection reset no longer shows an error to the user. Real Cloud failures and deadlines are still reported. ## Linked Issues or Issue Description Refs #15340 (bounded portfolio diagnostics). Refs #10850 (portfolio proxy). I searched the open and closed PRs for "portfolio". No other PR fixes this failure. **What happened?** On the hosted staging pool stack, one `GET /api/cloud/stacks` returned 502. The diagnostics were `phase: fetch`, `networkCode: ECONNRESET`, `upstreamStatus: null`, `elapsedMs: 19`. The Core process had started about 7 minutes before. CPU and memory were normal. The Railway HTTP logs for the Cloud harness show no failed or missing portfolio request at that time. All portfolio requests in a 12-minute window returned 200 in 15–91 ms. The next Core portfolio fetch, 0.4 s later, reached Cloud and returned 200. Thus the failed request did not get to the Cloud application. The connection was reset between Core and the Cloud edge. This failure is different from the older stalls on the same stack. Those failures had `DEADLINE_EXCEEDED` after more than 60 s, during whole-process freezes. This change does not retry those failures. **Steps to reproduce** 1. Start an HTTP server that resets (or closes) the socket for the first request and returns JSON for the second request. 2. Set `PAPERCLIP_CLOUD_API_ORIGIN` to that server and call `GET /api/cloud/stacks` with the native `fetch`. 3. Before this change, the route returns 502 with `networkCode: ECONNRESET` (or `UND_ERR_SOCKET`). Native `fetch` does not retry. **Expected behavior** The route retries the idempotent GET one time when the connection is reset before a response. The retry uses the same 10 s signal. Other failures stay unchanged. ## What Changed - `server/src/routes/cloud.ts`: If the fetch phase fails with `ECONNRESET` or `UND_ERR_SOCKET`, the route sends the same GET one more time. The retry uses the same `AbortSignal.timeout(10_000)`. The total time stays at or below 10 s. - The route classifies the first error with the existing bounded `CloudPortfolioError` diagnostics. Thus a reset after the deadline stays `DEADLINE_EXCEEDED` and gets no retry. - The first reset gets a warning log with the same safe fields (`Paperclip Cloud portfolio connection reset; retrying once`). It does not send a Sentry event. Thus the logs still show how frequently this occurs. - A second reset, a deadline, HTTP error statuses, body failures, `ECONNREFUSED`, `ENOTFOUND`, connect timeouts and unknown errors behave as before: one report and no retry. Failures are still not cached. - `server/src/__tests__/cloud-routes.test.ts`: The old test "classifies a fetch reset without retrying" now expects one retry. New tests cover a second reset, the codes that get no retry, a reset after the deadline, and two real-socket cases. `doc/observability.md` describes the diagnostic context. Its fields and semantics do not change. Per `CONTRIBUTING.md`, the change adds no code comments; the retry rationale is in this description. ## Verification - Red before the fix: with the old route and the new tests, `pnpm exec vitest run server/src/__tests__/cloud-routes.test.ts` gave 5 failed, 19 passed. The failures were the two mocked retry cases, the second-reset case and the two real-socket cases. Thus native `fetch` on Node 25.9 does not retry these resets itself. - Green after the fix: `pnpm exec vitest run server/src/__tests__/cloud-routes.test.ts server/src/__tests__/cloud-portfolio-error.test.ts` gave 31 passed. - Real dependency: the real-socket tests use `node:http` and the native global `fetch`. `socket.resetAndDestroy()` gives `ECONNRESET`. `socket.destroy()` gives `UND_ERR_SOCKET`. Each test makes sure that the second request reaches the server with the same authorization, the route returns 200, and no Sentry event occurs. - The second commit only removes the four-line comment that Greptile flagged. It has no runtime effect; a syntax check passed locally and CI reruns the suite. - Not run locally: `server/src/__tests__/sentry.test.ts` fails to load in my sandbox on the unchanged base and with this change. The error is the same: `Failed to resolve entry for package "@paperclipai/plugin-sdk"`. The workspace packages were not built. The full `tsc -p server` was stopped by the 8 GB sandbox memory limit. A typecheck of only the changed files gave the same output on the base and with this change, apart from a 5-line offset. CI runs the full typecheck and test suite. - Remaining limit: the staging evidence shows that the request did not reach the Cloud application. It does not show if the reset socket was a reused keep-alive connection or a new connection. The retry is correct in both cases. ## Risks - Low risk. The retry applies only to an idempotent GET that has no side effects on Cloud. It occurs only for a fetch-phase reset before a response. - The worst case is one more portfolio request to Cloud for each reset. The deadline is the same as before. - If the Cloud edge resets all connections, the user sees the same 502 as before, after two fast attempts. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use (shell, code execution, Sentry, Railway and GitHub connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
db95539d46 |
fix(agents): refuse agent purge when cost history references it (#19022)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can permanently delete (purge) a terminated agent > - The purge deletes the agent's heartbeat runs, but cost and finance events still point to those runs > - Postgres then rejects the delete with a foreign key error, and the API returns a 500 > - Cost and finance events are accounting history. The purge must not remove them > - This pull request makes the purge check for this history first and return a clear 409 > - The benefit is a clear, expected response instead of a 500, and accounting history stays intact ## Linked Issues or Issue Description **What happened?** Permanent agent deletion failed with `Failed query: delete from "heartbeat_runs" where "heartbeat_runs"."agent_id" = $1`. The cause is `update or delete on table "heartbeat_runs" violates foreign key constraint "cost_events_heartbeat_run_id_heartbeat_runs_id_fk" on table "cost_events"`. The server reported this as an unexpected error. Sentry: PAPERCLIP-BACKEND-4Z. **Steps to reproduce** 1. Run an agent so that a heartbeat run records a cost event. 2. Terminate the agent. 3. Permanently delete (purge) the agent. **Expected behavior** The purge stops before it deletes anything. It returns a clear conflict, and the agent stays terminated. This matches the existing `agent_case_history_retained` behavior. **Actual behavior** The transaction fails on the `heartbeat_runs` delete with a raw foreign key error (500). Related, older approaches: #9130 and #1987 tried to delete cost and finance rows during agent removal. This PR keeps the ledger instead. ## What Changed - `deleteAgentDependencies` now checks for `cost_events` and `finance_events` rows that reference the agent or one of its heartbeat runs. It does this before any destructive cleanup. - If such rows exist, it throws `409 agent_cost_history_retained`. Termination stays available. - The activity-log cleanup reuses the same agent-runs subquery. - New embedded-Postgres regression test `server/src/services/agent-cost-history-deletion.test.ts`. It covers a cost event on a run, a cost event attributed to the agent, a finance event on a run, and a normal purge with no cost history. ## Verification - Without the fix, 3 of the 4 new tests fail with the exact Sentry error. Examples: `cost_events_heartbeat_run_id_heartbeat_runs_id_fk`, `cost_events_agent_id_agents_id_fk`, `finance_events_heartbeat_run_id_heartbeat_runs_id_fk`. - With the fix, all 4 pass. - Related suites pass (8 files, 249 tests): `lifecycle.test.ts`, `execution-admission/adapters/postgres.test.ts`, `decision-models.test.ts`, `prompt-injection-prevention.test.ts`, `fast-responses.test.ts`, `agent-timer-deadlines.test.ts`, `case-deletion.test.ts`, and the new test. - `tsc --noEmit` in `server/` reports 0 errors. ## Risks - Behavior change: agents with recorded cost or finance history can no longer be purged. Before this change, these purges already failed with a 500, so no purge that used to succeed is now blocked. - No schema change and no migration. - Docs follow-up: `doc/SPEC-implementation.md`, `doc/DEVELOPING.md` and `maps/agent-access-map/agents.md` describe `agent_case_history_retained`. They do not yet mention the new `agent_cost_history_retained` code. I can add that in this PR if reviewers want it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |