mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
c6996799c427ba3e550a2f5eecbfb151ab46ba07
1832
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
c6996799c4 |
feat(task-chat): show an inline diff for every file edit (#19269)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat shows what an agent did during a run, including each tool call > - Codex and Claude both report the patch for every file edit, but Paperclip dropped it. Claude's ACP `diff` content was never read. The Codex `fileChange` item became a generic item with no patch. Codex `turn/diff/updated` had no native-runner mapping > - A reviewer could not see what an agent changed without opening the workspace > - This pull request carries a bounded patch on each edit call and shows it in the edit row. Each turn that changed files ends with a "Changed in this turn" card > - The benefit is that a reviewer can check every change in the conversation, with the same view for Codex and Claude ## Linked Issues or Issue Description Refs #19252 (precedent: canonical `plan.updated` events as a docked plan card). **Subsystem affected** Paperclip runner (Rust and TypeScript canonical event mapping), PRP provider-event schema, and the task-chat UI. **Problem or motivation** Edit rows say "Edited a file" with no content. Claude Edit/Write `diff` content is lost after the ACPX sidecar. The Codex native runner drops `fileChange.changes` and `turn/diff/updated`. The existing workspace-changes card showed a raw text patch in a dialog, and only on the in-process Codex path. **Proposed solution** Add an optional, bounded `fileChanges` list to `paperclip.tool.execution.v1`. Fill it from Codex `fileChange` items and Claude ACP `diff` content. Map Codex `turn/diff/updated` to `workspace.change.updated` on the native runner. In the UI, show `+N −M` on each edit row, open a compact colored diff, and end the turn with the change set. **Alternatives considered** A runner-side git diff at turn end. It needs workspace access and extra cost on every turn. The harness patches are already available. For Claude, the UI builds the turn card from the edit rows instead of new per-turn runner state. ## What Changed - Protocol: optional `fileChanges[]` on `toolExecution` in `provider-event.schema.json`, with the regenerated schema bundle, validators and manifest. Paths are workspace-relative. A patch is at most 32 KiB, and a call has at most 32 files. - Runner (Rust): new `tool_file_changes.rs`. A Codex `fileChange` item now also emits `tool.execution.*` with `fileChanges`; the native item event stays for the recovery and progress projections. `turn/diff/updated` becomes `workspace.change.updated`. `normalize_acpx_tool_call` re-validates the sidecar's `fileChanges`. `normalize_codex_notification_in` resolves absolute paths against the session cwd. - Runner (Rust): `has_bounded_rich_display_shape` accepts schema-valid workspace diffs. Without this change, durable storage cut each patch string to a 4 KiB preview. Patches use the uncapped secret redactor. - Runner (TS): new `tool-file-changes.ts`. The ACPX sidecar computes `fileChanges` from ACP `diff` content with the working directory. The in-process Codex and ACPX mappers do the same, with an optional `context` argument. - UI: new `TaskChatDiffView` (colored rows, hunk labels, a height cap, "Show N more lines"). Edit rows show counts and the diff. Collapsed activity groups show edit totals. The workspace card is now "Changed in this turn", with in-place per-file diffs, and it sits at the end of the turn. When a harness sends no whole-turn diff (Claude), the card is built from the edit rows. A change set is marked complete when its run ends. - UI: `native-run-events.ts` maps `workspace.change.updated` and attaches each edit's latest patch to its row (the `diff` transcript entries now carry `toolUseId`). - Design tokens: `--diff-add-*`, `--diff-remove-*` and `--sz-edit-diff-max-h` come from the status icon hues. - Storybook: `Tasks/Edit diffs` story. Docs: `doc/run-log-events.md`. ## Verification - Real runs on a local instance (`test-drive`, native runner): - Codex (`gpt-5.6-sol`): the "Applied a patch" rows show `+9` and `+35 −2` with their diffs. The turn ends with "Changed in this turn", 2 files, `+42`, from `turn/diff/updated`. - Claude (`claude-opus-5-5` via ACPX): the Write and Edit rows show `+5`, `+14 −1` and `+9` with correct hunks. The turn card is built from the edit rows. - Codex model note: with `gpt-5.4`, pinned Codex 0.160 reports "model metadata not found". The model then wrote files through the shell, so no `fileChange` items occurred. - `pnpm check:token-gates`: all gates clean. - `pnpm --filter @paperclipai/ui exec vitest run src/components/task-chat src/components/transcript src/adapters`: 1073 passed. - `packages/paperclip-runner`: `vitest run src/provider-events.test.ts src/tool-file-changes.test.ts src/cli/acpx-runtime-sidecar.test.ts` passed. `check:protocol-types` and `check:protocol-manifest` match. - `cargo test -p paperclip-runner-core --lib provider_events tool_file_changes`: passed. `cargo fmt --check`: clean. - Storybook: open `Tasks and conversations/…/Tasks/Edit diffs`. ## Risks - Larger payloads: an edit call can carry up to 96 KiB of patch in Rust, and the sidecar keeps 48 KiB under its 128 KiB frame. Above the budget, a file keeps its counts and drops its patch (`diffTruncated`). - Redaction: each patch goes through the shared secret redactor. Before this change, a 4 KiB preview cap also applied. - Claude turn card: it joins each edit's patch in order. It is not a net diff. For a file the turn created, the counts are net and exact. For an existing file, the counts are summed over its edits. - The workspace card moves to the end of the turn. One adapter test changed to match. - Local only: four Linux cgroup `task_memory` Rust tests fail on macOS. They do not touch these modules. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, browser and file editing) and a long context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
6293e581f6 |
fix(recovery): resume cancelled workspace retries from verified waits (#19276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A shared workspace can make a run wait before it calls a provider. > - The scheduler saves a retry for that wait. > - The control plane can cancel the retry while it is still queued. > - Recovery then asks for a process stop receipt for a run that never claimed execution. > - This pull request verifies the retry and its exact workspace-wait parent before it accepts a later user instruction. > - The benefit is that Retry and saved messages can continue through normal admission without replaying old work. ## Linked Issues or Issue Description Refs #13640 and #15483. Related fixes: #18838 covers dispatcher cancellation codes; this change covers a system-cancelled scheduled workspace retry. Related open PR #15383 covers claimed conversation-adapter preparation. This change requires an unclaimed retry with no run profile or owner. **What happened?** A scheduled workspace retry was cancelled before runtime selection. The run retained an expected system cancellation and a `startupCancellation.beforeNativeSelection: false` marker. It had no start timestamp, execution stage, run profile, controller, process, lease or provider events. Recovery recorded an unknown-action hold. A later user message and Retry could not pass that hold. **Expected behavior** A later authorized instruction can start a fresh turn when the saved retry and its exact parent prove that no provider work started. Missing or conflicting evidence must retain the hold. **Steps to reproduce** 1. Record a terminal workspace wait before provider dispatch, with `executionRecovery.kind: workspace_wait` and `providerWorkStarted: false`. 2. Save a scheduled retry for that same company, task and agent. 3. Cancel the retry through the control plane before it claims execution. 4. Send a new user message or request Retry after the cancellation. 5. On the base commit, admission returns no continuation. With this change, normal admission creates one fresh successor. **Paperclip version or commit** Reproduced in source based on master at `499712c68b3ca95801b6114d5f2f0f0ea6759318`. **Deployment mode** Source checkout with isolated embedded PostgreSQL. Hosted rollout and a real provider run remain to be verified after merge. ## What Changed - Extend the existing cancelled-startup proof with one source-bound workspace-retry case. Require the canonical retry parent and reason to match the saved context. - Verify the parent's company, task, agent, time order, immutable Runner claim and typed workspace-wait record. Reject runtime, process, lease, coordinator, result and output evidence on either run. - Keep validation of the full saved result in the database predicate. The bounded execution projection can add empty receipt fields without changing eligibility. - Reuse the existing admission, Retry and deferred-message paths. Preserve cancelled history and unknown action outcomes. Keep permission, Stop, budget and recovery-limit gates. - Add 23 regression cases, including heartbeat message and Retry admission, concurrent saved-message recovery, and 19 conflicting-evidence cases. Document the contract. The cancelled-startup service owns this proof. No provider protocol, schema, UI or shared contract changes. Reads use exact run IDs and return only narrow metadata. The new reads occur only for a cancelled, unstarted Runner retry candidate. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/explicit-native-continuation.test.ts src/services/legacy-execution-recovery.test.ts src/services/execution-blocker-read.test.ts src/services/task-retry.test.ts` — 452 passed. - `pnpm -r typecheck` — passed. - `pnpm build` — passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — passed. - Local diff review and secret/private-context scan — passed. Test identities are synthetic. - Regression history: the original positive case failed on unchanged source while 11 negative guards passed. Initial test fixtures then exposed ownership/contract constraints and a missing saved-wake envelope; the fixtures now use valid persisted shapes. The heartbeat Retry test exposed extra keys in the bounded evidence projection. Full-record validation now runs in SQL, and both assembled heartbeat paths pass. The unknown action outcome stays unchanged. - Platform: macOS arm64, Node 25.6.1, pnpm 9.15.4. Tests use real isolated PostgreSQL. They stop at admission and do not call a live provider. - Exact-head GitHub CI: 54 successful checks, 3 skipped, no unfinished or failed checks. Greptile scored `cfbf55cea68e3c69f65f50afcba98f775fa6fa3d` 5/5, with no actionable findings and no review threads. - The full repository `pnpm test:run` was not run locally. A previous full run in this repair effort stalled and was stopped after 2h45. The four complete affected suites pass; full PR CI must pass before merge. ## Risks - False admission could allow work before a predecessor is safe. The proof fails closed on missing parent records, different ownership, a changed retry binding, later parent completion, restore obligations, any execution resource, or any provider/output evidence. - This deliberately covers only system-cancelled, unclaimed workspace retries. Other cancellations retain their current reconciliation path. It does not grant automatic replay, certify unknown external writes, or reset exhausted recovery limits. - No migration. Verify the serving version and an existing saved-message continuation after deployment. ## Model Used - OpenAI `gpt-6.1-sol`, xhigh reasoning, through the Codex harness in T3 Code. Used code editing, shell tools and test execution. The harness does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
499712c68b |
fix(runner): restore cold state before inspecting completed handoffs (#19254)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner saves provider sessions and controller state across turns. > - A completed warm handoff leaves a receipt in the controller journal. > - A later cold continuation must restore external Runner state before it can inspect that receipt. > - The transport read the missing physical file before it called the restore hook. > - This pull request prepares admitted external state before that inspection for completed handoffs. > - Live Runner adoption and pending transitions retain their existing ownership checks. The provider thread can continue after a cold restore. ## Linked Issues or Issue Description Related: #19225 and #19157. This fix changes the shared transport read order. It does not depend on the open source-reconciliation extension in #19157. **What happened?** A cold continuation after a completed warm handoff could fail with `runner_remote_state_unavailable`. The saved controller journal was valid. The physical Runner state was absent because its restore hook had not run yet. A later automatic reattachment could then fail an allocation identity check. **Expected behavior** Restore admitted external state before inspecting a completed handoff. Retain the original provider thread. Refuse a denied restore or foreign session without starting provider work. **Steps to reproduce** 1. Open a Codex session and complete a warm handoff to a second run. 2. Suspend the session and retain the controller journal and provider backup. 3. Start a third run with no physical Runner state until the external preparation hook restores it. 4. Read the retained thread. Before this fix, the read fails before preparation. ## What Changed - Prepare external Runner state before inspecting a completed warm-handoff receipt. - Keep live Runner adoption, pending attachments, and active warm transitions on the existing exact-state path. - Add a real Runner and controlled Codex regression for successful restore, denied restoration, and a foreign restored session. - Document the cold-continuation order in the execution contract. ## Verification - Red: the new real-process regression fails on unpatched source because resume reads the absent `runner-state.json` before preparation. - Green: all three new cases pass. The restored thread keeps its identity. The controlled provider starts one thread and resumes it once. No turn is replayed. - Adjacent safety suites: 60 cases passed across warm attachment, interrupted attachment recovery, suspension, and cold restore. The new foreign-session case also passes separately. - Runner TypeScript and generated protocol checks passed. - Full workspace typecheck and build passed. Runner TypeScript checks passed again after the live-adoption guard. The full local test command remains in progress; all required CI test jobs passed on the current head. - The first CI run found two live-adoption regressions: an already completed staged handoff must not restore a backup while its Runner is live. The guard now excludes live adoption. All six cases across cold restore and staged handoff pass locally. The original CI failure remains recorded in run 42047027346. - Greptile reviewed the corrected head `f33a041c8902c39ca52b995d53128fc23b0b5499` at 5/5. There are no review threads or findings. All current-head CI gates passed. - Live staging verification remains pending normal deployment. No pinned serving image or bulk migration is part of this PR. ## Risks The change calls an existing external preparation hook earlier for completed handoffs. That hook must remain idempotent and must enforce restore authority. Live adoption, pending attachments, and active transitions do not gain this earlier call. The negative regressions assert that a refused restore and a foreign session start no provider work and do not change the authority journal. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e37ab23dbc |
fix: hold an agent's new work while its provider usage limit is active (#19246)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routine-driven agents run on the native Runner through ACP harnesses such as Claude > - When the agent's provider account runs out of allowance, every new wake for that agent fails the same way > - The ACP harness reported the exhausted allowance as a generic "terminal limit failure" with an `unknown` category, so Paperclip could not tell it from a context or turn limit > - Paperclip recorded a reset time only for the failing run's own startup retry; other wakes of the same agent kept dispatching, each spent about two minutes on workspace setup, and then failed > - This pull request classifies the limit, keeps its reset time, and holds the agent's new work until that reset > - The benefit is that a limited agent stops burning runs, shows a clear "waiting for usage limit reset" state with a time, and resumes its queued work after the reset ## Linked Issues or Issue Description Refs #13148 Refs #13379 **What happened?** A routine-driven Claude agent on the native Runner failed 29 times in about 15 hours. 20 runs failed with `native_provider_terminal_failed` and "ACP agent reported a terminal limit failure." The recorded provider failure had category `unknown`. 6 runs failed with `provider_session_open_rejected`. Their `session.open` result is `command_execution_failed` with classification `provider_rate_limit`, and the provider failure is HTTP 429 `rate_limit` ("would exceed your account's rate limit"). The startup retry of one such run recorded a cooldown until one hour later. A new wake for a different task of the same agent dispatched 16 minutes later, spent two minutes on workspace setup, and failed with the same 429. Separately, 4 runs failed with "Project authentication conflicts with this agent's managed AI connection". That message did not say which file or setting conflicted. The same message was also shown when the check could not finish. **What did you expect?** A provider usage limit is classified distinctly. Paperclip records the reset time when the provider reports one. New work for that agent waits for the reset with a visible reason. Queued work resumes after the reset, and no doomed run starts. A configuration conflict is reported only when one is found, and the message names the file to fix. **Steps to reproduce** 1. Use a native Runner agent with a Claude ACP harness whose account is rate limited or out of usage. 2. Let a routine or assignment wake the agent on several tasks. 3. See each run fail with a generic terminal limit failure or a `session.open` rejection, and see new runs keep starting. ## What Changed - Runner: The ACPX turn now receives the typed AIR session failure (`onTerminalSessionFailure`) through the runtime host and the ACPX port. The sidecar adds the provider's title to the provider failure message. Only a rate-limit or quota title becomes `rate_limit` or `quota_exhausted`. Context, turn and configured budget limits stay `unknown`. The failure details stay private. - Server: A native `turn.failed` with a `rate_limit` or `quota_exhausted` provider failure is now `native_provider_usage_limit`, the same code as the Codex usage-limit terminal. - Server: `agentProviderLimitCooldown` (extracted from the startup retry) also reads a reset clock in the provider's wording, such as "resets 2am (UTC)". The clock parser moved from the recovery service to `provider-quota-reset.ts` without changes. - Server: New `heartbeat/provider-limit-wait.ts`. Before any workspace or provider work, a queued run for an agent with an active provider limit is cancelled as `provider_limit_wait`. The run records the reset time, whether the provider reported it, the category and the run that reported the limit. Paperclip schedules one retry of the same wake at the reset through the existing AI connection wait lane. That lane does not consume the failure retry budget. Manual runs, recovering runs, conversation turns and pool-routed connections are not held. - Server: The execution projection shows "Waiting for usage limit reset" with the reset time and the next step. - Server: Only a found project authentication conflict is a configuration failure. Its message names the project file and the settings to remove. A check that cannot finish (timeout or another exit code) is now an ordinary failure, not a configuration conflict. - Docs: `doc/execution-semantics.md` describes the agent-level limit wait. ## Verification Red before the fix, green after: - `server/src/__tests__/heartbeat-provider-limit-wait.test.ts`: An automation wake after a 429 `session.open` rejection dispatched the adapter before the fix. Now the run is cancelled as `provider_limit_wait` and a scheduled retry exists at the reset. After the reset passes, the retry is promoted and the work succeeds. A manual run, a later success, expired evidence and an unclassified terminal still dispatch. - `native-startup-retry.integration.test.ts`: "reads the reset clock from an ACP terminal limit failure's provider wording" (it fell back to one hour before). - `native-session-executor.test.ts`: An ACP `turn.failed` with a typed `quota_exhausted` or `rate_limit` failure is a usage-limit terminal; `unknown` is not. - `packages/paperclip-runner/src/drivers/provider-failure.test.ts`: A typed AIR limit title is kept and classified; context, turn and budget limits are not; details do not leak. - `execution-projection.test.ts`, `ai-connections.test.ts`: The wait label, and conflict versus unfinished check. - Safety cases: `runtime-host.test.ts` and `codex-runtime-adapter.test.ts` forward the callback. The existing startup retry, provider capacity, recovery classification, legacy reconciliation, retry scheduling and execution configuration suites pass. Commands: - `cd server && npx vitest run src/__tests__/heartbeat-provider-limit-wait.test.ts src/__tests__/ai-connections.test.ts src/services/native-runtime/ src/services/recovery/ src/services/execution-projection.test.ts src/services/legacy-execution-recovery.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/heartbeat-execution-configuration.test.ts` - `cd packages/paperclip-runner && npx vitest run src/drivers/provider-failure.test.ts src/drivers/acpx/runtime-host.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts src/cli/acpx-runtime-sidecar.test.ts` - `tsc --noEmit` for server and paperclip-runner; the ACPX sidecar and provider-failure contract checks pass. I did not run a live provider limit on staging. The live evidence is the staging run records for the affected agent, which I read only. ## Risks - The limit evidence is per agent, not per account. If the board changes the agent's AI connection, automatic wakes can still wait until the old reset (at most one hour when no reset was reported, and at most one day). A manual run is not held, and one successful run clears the evidence. - A new wake that arrives during a limit is now cancelled and retried at the reset, instead of failing. The task stays in its status, and the run shows the reset time. - The Runner change takes effect when the provider pack is rebuilt and deployed. Until then, the server still holds new work using the existing `rate_limit` evidence from `session.open` rejections. - A turn terminal with a typed rate limit or quota is now `native_provider_usage_limit` instead of `native_provider_terminal_failed`. Both are terminal and board-owned. The recovery text now asks the board to restore provider usage capacity. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d0afeacdd5 |
fix(remote-workspaces): keep retained runner state when replacing a retired root (#19251)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote workspaces save a task's files as snapshots. Runtime state under `.paperclip-runtime` is never in a snapshot, because it belongs to the sandbox > - #19231 lets a parent task on a retained sandbox restore newer child work over its own retired workspace root > - That replace removed the whole retired root, including the parent's retained runner session state under `.paperclip-runtime/paperclip-runner/` > - The parent's native continuation then reads that runner state on the resumed sandbox, cannot find it, and fails with `runner_remote_state_unavailable` > - This pull request moves the retired root's runtime state into the restored root before it removes the retired root > - The benefit is that delegating parents on retained sandboxes continue their session with the children's saved work ## Linked Issues or Issue Description Refs #19231 **What happened?** After #19231, a parent task on an agent with a retained sandbox delegates two subtasks. Both subtasks finish. The parent's wake run now restores the newer saved work successfully, but the run fails right after with `runner_remote_state_unavailable: exit=1 timedOut=false`. The run error code is `native_continuation_requires_reconciliation` and the task becomes `blocked`. In one staging wave, 4 of 4 such parents failed. Root cause: the native continuation on a resumed sandbox reads `<remoteCwd>/.paperclip-runtime/paperclip-runner/sessions/<session>/runner/runner-state.json`. Snapshots exclude `.paperclip-runtime`, so the restored root has only the new restore receipt. #19231 parked the retired root and then removed it, so the runner state was removed too. **Expected behavior** The restored root has the newest saved files and the sandbox's retained runtime state. The parent's continuation reads its runner state and continues. **Steps to reproduce** 1. Prepare a remote workspace on sandbox A. Write files and runner state under `.paperclip-runtime/paperclip-runner/`. Save. 2. Make a second run the writer on sandbox B. It restores, writes new files, and saves. 3. Make a third run the writer on sandbox A again and prepare. The restore replaces the retired root. 4. Read the runner state with `readRemoteRunnerState`. Before this change, it fails with `runner_remote_state_unavailable: exit=1 timedOut=false`. **Paperclip version or commit** `master` at `26391707fb`, which includes #19231. ## What Changed - `server/src/services/remote-workspace-transport.ts`: before it removes the retired root, the restore script moves each entry of the retired `.paperclip-runtime` into the restored root's `.paperclip-runtime`. It does not move `workspace-ready.json`, `workspace-restored.json`, or `workspace-snapshots`, and it does not replace an entry that already exists. The move is a rename, so it keeps sockets and large session state. The script also finishes an interrupted carry when the same restore repeats. Only an authorized replace with that restore identity creates the `<root>.retired-<identity>` path. - `server/src/services/remote-workspaces.integration.test.ts`: the retained-sandbox regression now writes parent runner state, restores newer child work, and reads the state with the same `readRemoteRunnerState` call that the continuation uses. It also covers an interrupted carry on a repeated restore. - `doc/execution-semantics.md`: document that runtime state moves with the restored root. ## Verification - Red before the fix: `npx vitest run src/services/remote-workspaces.integration.test.ts -t "retired root"` failed in the `own retired root`, `own restored root`, and `read-only folder` cases with `Error: runner_remote_state_unavailable: exit=1 timedOut=false`. This is the same error as the staging failure. - Green after the fix: all nine cases pass. The continuation read returns the parent's runner state. The old `workspace-ready.json` is gone and the new restore receipt is present. A repeated restore finishes an interrupted carry and removes the retired root and its stale files. - `npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts`: 250 passed. - Staging evidence (read-only): the failed parent wake run recorded a succeeded `remote_workspace_restore` at the parent's own provider lease, and then failed in the runner-state read. - I did not run a live staging run. This PR does not deploy. ## Risks - Runtime state from the retired root now persists into the restored root. The state is the sandbox's own state for this workspace path, which is the same state that a same-allocation reuse keeps. - If the restored root already has an entry with the same name, the old entry is not moved and is removed with the retired root. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. It used tool use for code search, shell, test runs, and read-only staging inspection in the browser. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
21640b6d51 |
feat(connections): gate the built-in Google Workspace connector behind an experimental toggle (#19247)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps catalog lets operators connect external tools, including Google services. > - Paperclip has a built-in Google Workspace connector, but Google has not yet verified Paperclip's Google app. > - The built-in connector also hides the Composio and Arcade Google apps, because native Google definitions claim those app names. > - Users therefore see an unverified connector and cannot find the working aggregator Google apps. > - This pull request puts the built-in connector behind a default-off experimental setting. When the setting is off, the aggregator Google apps appear. > - The benefit is that users get working Google connectors now, and the team can still test the built-in connector on demand. ## Linked Issues or Issue Description **Problem** Google has not verified Paperclip's Google app yet. The Apps page still offers the built-in Google Workspace connector. The built-in Google definitions also suppress the Composio and Arcade Google apps (Gmail, Google Calendar, Google Drive, Docs, Sheets, Slides, Chat), so users cannot find them. **Proposed solution** Add the `enableGoogleWorkspaceConnector` experimental setting. The default is off. When it is off, hide and block the built-in connector, and show the aggregator Google apps. When it is on, keep the current behavior. **Alternatives considered** Delete the built-in connector until verification is complete. We rejected this option because the team must continue to test the connector. **Additional context** Composio documents Gmail (63 tools), Google Calendar (50 tools), and Google Drive toolkits that use Composio-managed OAuth. These toolkits do not need Paperclip's Google app verification. ## What Changed - Added the default-off `enableGoogleWorkspaceConnector` experimental setting to the shared types, validator, feature catalog, server defaults, and the managed-key list for Paperclip Cloud staging. - Added `experimentalFeatureKey: "enableGoogleWorkspaceConnector"` to the `google-workspace` app definition. The existing generic connector gate then hides the connector in the gallery and blocks its setup, discovery, execution, and agent suggestions when the setting is off. - Added a "Google Workspace connector (Paperclip)" toggle card to **Settings → Experimental**. - Apps page (`Browse.tsx`): when the setting is off, built-in Google rows without saved accounts no longer claim aggregator app identities. The Composio and Arcade Google apps then show as aggregator rows. The legacy Google slug filter now applies only to native rows, so the Composio `gmail` row is not dropped. - Saved per-service Google connections (Gmail, Drive, and others) are not gated. They stay visible and usable. - Updated `doc/connections/GOOGLE-WORKSPACE.md`. - Tests: added tests for the hidden and visible Apps page states, the settings toggle, and the definition gate. Existing built-in Google Workspace tests now enable the setting. ## Verification - `cd packages/shared && npx vitest run` passes (1055 tests). - `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/tool-access-service.test.ts src/__tests__/tool-gateway-service.test.ts src/services/gmail-tool-governance.test.ts src/services/google-chat-tool-policy.test.ts src/__tests__/instance-settings-service.test.ts src/__tests__/health.test.ts src/__tests__/instance-settings-managed-overlay.test.ts` passes (864 tests). - `cd ui && npx vitest run src/pages/apps src/pages/InstanceExperimentalSettings.test.tsx src/features/connections` passes. - `tsc --noEmit` passes for `packages/shared`, `server`, and `ui`. The `ui` run shows errors only in the untouched `storybook/support/feature-map` files. - `pnpm check:token-gates` passes. - Manual check: open **Apps** with the setting off. Search for "Gmail". The Composio Gmail row appears, and the built-in Google Workspace card does not. Turn on the setting in **Settings → Experimental**. The built-in card appears again. ## Risks - Existing built-in `google-workspace` connections stop running while the setting is off, because the generic gate blocks execution. This is intentional, so that the unverified Google app is not used. Turn on the setting on instances that must keep these connections. - Saved per-service Google connections are not affected. - When the setting is on, the behavior does not change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in Claude Code with tool use (shell, file edits, web fetch). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
03abe1f1a1 |
fix(native): keep the transport cause on a closed notification stream failure (#19240)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs read provider events from a shared session notification stream (Codex, OpenCode and ACPX drivers use it) > - When that stream closes, the driver fails the session with `notification_transport_failed` > - The run error was only "Provider notification transport failed closed." The real cause was only in a diagnostic event > - In a staging soak, many runs on OpenRouter and Grok agents showed this error. Operators read it as a provider fault. The diagnostic events showed a host-side cause: a stopped sandbox plugin worker ended remote runner supervision > - This pull request keeps the redacted, bounded cause on the terminal failure message, and does not name the provider > - The benefit is that a failed run shows why it failed, so operators do not chase the wrong provider ## Linked Issues or Issue Description **What happened?** Native runs failed with `native_provider_terminal_failed` and the error "Provider notification transport failed closed." The run did not show a cause. All failures in one cluster happened inside one minute, on several agents and models (OpenRouter DeepSeek, Grok). Each run's `harness.diagnostic` event had the real cause, for example `runner_process_termination_unconfirmed: runnerd process supervision failed: Cannot call "environmentExecute" — worker for "<plugin>" is stopped`, or `native_adopted_runner_identity_unverifiable: runner liveness could not be revalidated`. The sandbox duplex transport reported a `provider_exit` loss about 30 seconds before (the read-only process probe budget). **Expected behavior** The terminal run error keeps the redacted cause. It does not blame the model provider when the cause is host-side runner supervision. **Steps to reproduce** 1. Open a native Codex-protocol session and start a turn. 2. Make the notification stream throw (for example, a supervision error). 3. Read the `session.failed` and `turn.failed` messages. Before this change they are the bare generic text. ## What Changed - `codex-session-notifications.ts`: the `notification_transport_failed` terminal message is now `Notification transport failed closed: <cause>`. The cause is redacted first, then whitespace is collapsed and it is bounded to 1,000 characters. The `Error: ` prefix is removed. The code and `recoverable: false` do not change. - New regression in `codex-protocol-integrity.test.ts`: one `session.failed`, the cause and code are present, the secret is redacted, the length is bounded, and `turn.failed` has the same message. - `doc/execution-semantics.md`: an additive note that this failure can come from host-side runner supervision, and that the message keeps the redacted cause. ## Verification - Red: before the fix, the new test failed: `expected 'Provider notification transport faile…' to contain 'runner_process_termination_unconfirmed'`. - Green: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/ src/conformance/local-provider-fault-matrix.test.ts src/native-session-runtime.test.ts` (565 passed). - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json`: no new errors. The existing `src/eval/workflow-harness.ts` errors come from an unbuilt workspace package. - Staging evidence was read with read-only GETs only (run, events). ## Risks - Low risk. Only the message text changes. The failure code, `recoverable: false`, and the safe-replacement rules do not change. The server matches on the code, not on the message. - This PR does not add a retry in the same turn. Runner termination is unconfirmed when supervision is lost. A retry or replacement could start a second executor. The remote process probe already retries read-only checks inside a 30-second budget. Recovery stays with the existing ownership and recovery checks. - The cause text can include a plugin id. It goes through the existing provider-diagnostic redaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b3da1d886a |
fix(ui): retain Coordinator label filters across reloads (#19233)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use task filters to keep a useful view of their work.
> - Coordinator saves status and assignee filters in the task-list URL.
> - A selected label stayed only in React state, so reload removed it
and showed unrelated rows.
> - This change saves and restores selected labels through the same
page-owned URL path.
> - Operators can reload and share that filtered collection without
selecting its labels again.
## Linked Issues or Issue Description
**What happened?**
In Coordinator All tasks, select a status, assignee, and label. Sort by
Task ID and reload. Status, assignee, and sort remain selected, but the
label is cleared and rows without that label appear.
**Expected behavior**
Reload and history navigation retain the selected labels with the other
URL filters. Clear filters removes the selected labels.
**Steps to reproduce**
1. Open Coordinator All tasks with labeled and unlabeled tasks for the
same assignee and status.
2. Select that status, assignee, and a label.
3. Change Task ID sorting and reload the page.
4. Check both the selected label and the displayed tasks.
**Paperclip version or commit**
Source regression reproduced at master
|
||
|
DottaandClaude Opus 5.5
|
e8163fc860 |
fix(remote-workspaces): restore newer saved work over a retained sandbox's own retired root (#19231)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote workspaces save a task's files as snapshots and restore them on the sandbox that runs the next turn > - A parent task can delegate subtasks. The subtasks inherit the parent's workspace, run on their own sandboxes, and save newer snapshots > - When the parent wakes on its retained (warm, reused) sandbox, its earlier workspace root is still on that sandbox at the same published path > - The restore refused that root with `reason=occupied_root`, so the parent run failed at setup and the task became blocked with a board recovery action > - This pull request lets the restore replace that root only when the server proves it is the workspace's own earlier, fully saved allocation on this sandbox > - The benefit is that a delegating parent on a retained sandbox continues with the children's saved work, and unknown occupants are still refused ## Linked Issues or Issue Description **What happened?** A parent task on an agent with a retained sandbox delegates two subtasks to other agents. Both subtasks finish. The parent's wake run fails before the provider starts with `remote_workspace_restore_failed: step=sandbox_restore; ... stage=inspect_root reason=occupied_root`. The task becomes `blocked` with a stranded-work recovery action. Parents on agents with ephemeral sandboxes complete in the same wave. Root cause: the subtasks inherit the parent's execution workspace. Each subtask becomes the writer at a new epoch on its own sandbox, restores at the same published directory name, and saves a newer snapshot. The workspace location then points at the last subtask's sandbox. The parent's wake run gets a lease on its retained sandbox, so preparation is not "same allocation" and must restore the newer snapshot. The target path on the retained sandbox still holds the parent's epoch-1 root, which has a preparation receipt (`workspace-ready.json`) and no restore receipt. The restore script refuses any occupied root that is not the same restore, so it fails with `occupied_root`. A warm reuse cannot just keep the old root, because the snapshot is newer. **Expected behavior** The parent restores the newest saved work (its own files plus the children's results) into its published root and continues. A foreign or unknown occupant is still refused. **Steps to reproduce** 1. Prepare a remote workspace on sandbox A, write files, and save (final boundary). 2. Make a second run the writer on sandbox B. It restores, writes new files, and saves. 3. Make a third run the writer on sandbox A again (same provider lease id, new lease row) and prepare. 4. Before this change, step 3 fails with `stage=inspect_root reason=occupied_root`. **Paperclip version or commit** `master` at `47027e7ba5`. ## What Changed - `server/src/services/remote-workspaces.ts`: add `isRetiredOwnAllocation`. It reads this workspace's snapshot and restore operations recorded at the same environment, provider lease id, and root path. It returns true only when every such use is from an earlier epoch than the current writer, the last use ended in a succeeded `final` save with quiesced save work (`unknown=false`, no active invocations), no other save in that epoch is newer or unsettled, and the saved snapshot is committed or later. Preparation passes the result to the restore as `replaceRetiredRoot`, and only when there is no migration and the authority is clean. - `server/src/services/remote-workspace-transport.ts`: the restore script accepts `replaceRetiredRoot`. In that mode it replaces an occupied root only when the root carries a well-formed Paperclip receipt (`workspace-ready.json` or `workspace-restored.json` with a 64-hex identity). It parks the old root at `<root>.retired-<identity>`, publishes the restored folder, rolls the old root back if the publish rename fails, and then removes the parked copy. A repeated restore with the same identity adopts the published folder and removes any parked copy left by an interrupted attempt. Without a receipt, the script still fails with `reason=occupied_root`. `replaceRetiredRoot` and `allowEmptyRoot` cannot be combined. - `doc/execution-semantics.md`: document the rule. - `server/src/services/remote-workspaces.integration.test.ts`: add a red/green regression and the safety cases. ## Verification - Red before the fix: `npx vitest run src/services/remote-workspaces.integration.test.ts -t "retired root"` failed in the `own retired root` case with `remote_workspace_restore_failed: step=sandbox_restore; remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=inspect_root reason=occupied_root`. This is the same cause text as the staging failure. - Green after the fix: all six cases pass: - `own retired root`: the parent restores on its retained sandbox. The root has the parent's file and the child's file, the parked and staging folders are removed, and the authority records the parent's lease. A repeated preparation (lost reply) keeps the parent's new unsaved edits. - `own restored root`: an occupant with a restore receipt from a different preparation is also replaced. - `receipt missing` and `forged receipt`: the occupant is refused with `occupied_root`, and its files stay. - `later allocation`: a later use of that root on that sandbox without a final save is refused. - `never saved here`: a different sandbox with an unknown folder at that path is refused. - `npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts`: 247 passed. This includes the existing `occupied root` cold restore case, which is still refused. - `tsc --noEmit` shows no errors in the changed files. The local tree has unrelated errors from unbuilt workspace packages. - I did not run a live staging run. Staging is read-only for this work, and this PR does not deploy. ## Risks - The replaced root is removed after the publish. The server proof states that its content is in the saved lineage. If a process wrote to that root after its final save and outside a Paperclip run, those writes are lost. The same assumption applies when authority moves to another allocation today. - The proof uses operation records that have `exportCleanup` location metadata. Older records without that metadata do not qualify, and those cases still fail closed with `occupied_root`. - A warm runner process that kept an open working directory in the old root sees a removed folder. This is not verified live. The new turn is given the published path, which holds the restored files. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. It used tool use for code search, shell, test runs, and read-only staging inspection in the browser. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5d8d9bf83e |
fix: share native lifecycle policy with instruction checkpoints (#19230)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Warm native Codex turns save registered agent files at each turn boundary. > - Native execution can inherit warm mode from the agent when the environment has no explicit policy. > - File preparation checked only the environment policy and could miss that inheritance. > - The new copy then had no warm checkpoint callback, even though the native session stayed warm. > - This pull request resolves one admitted policy before both owners use it. > - The benefit is a terminal file checkpoint under the selected lifecycle, with the existing ownership and permission fences. ## Linked Issues or Issue Description Refs #14735 and #19225. The first defines warm file checkpoints. The second retains coordinator ownership while cleanup joins. This change fixes the earlier lifecycle selection mismatch. **What happened?** With no explicit environment policy and an agent warm default, native selection chose warm mode with a 900,000 ms idle window. Actual file preparation registered an agent_files copy without its warm flag or checkpoint callback. An unchanged turn could have no save marker. Later retirement of a retained lease could reach the stopped-file retrieval guard and preserve pending collection debt. **Expected behavior** File preparation and native execution use the same admitted lifecycle policy. Warm native Codex copies receive their terminal checkpoint callback. Explicit per-turn overrides and pinned admissions keep their existing precedence. A pending historical copy does not become saved because policy resolution changed. **Steps to reproduce** 1. Configure a native Codex agent with a warm default and a reusable sandbox without an explicit runner lifecycle policy. 2. Prepare its managed agent-file copy through the normal instruction owner. 3. Select native execution through the normal runtime owner. 4. Compare the selected warm policy with the copy's missing warm flag and checkpoint callback. **Paperclip version or commit** Base: `3a090b815b`. The owning policy code also exists on its parent revision. **Deployment mode** Native Codex with registered agent files and a reusable sandbox that inherits its lifecycle policy. ## What Changed - Move the existing lifecycle precedence and backend gates into a narrow shared resolver. - Resolve one value before instruction preparation and pass it into runtime selection. - Select warm file ownership from that value and the existing file capability gate. - Add real-Postgres unchanged and changed terminal checkpoint cases, policy precedence cases and updated preparation coverage. - Exercise the same resolve, timed preparation and runtime-selection orchestration used by heartbeat; preserve preparation error identity and skip selection on preparation failure. - Document new-admission overrides and pinned-policy behavior. ## Verification - Red: actual preparation and runtime-selection producers disagree. The selected native mode is warm, but the real persisted copy has no warm flag. The fixture uses real PostgreSQL, managed-root authorization and a synthetic loopback transport. - Green: 85 owner tests pass. They include actual unchanged and changed terminal checkpoints plus 19 lifecycle hierarchy/capability cases. Changed synthetic bytes reach canonical storage with a saved receipt; unchanged bytes record unchanged. - Safety: 25 existing warm retirement and stopped/destroyed/foreign-owner cases pass. Six authorization, stale-adoption and adopted-owner preservation cases pass. The lease guard and native executor are unchanged. - Workspace typecheck, build and map checks pass. The final candidate's code and test blobs equal the tested source; its exact build stamp is recorded locally. - No new repository-wide local run was repeated. A separate earlier broad command on an older head failed a reproducible GitHub enrollment fixture; it is not a full local pass, established flake or proven caused by this change. Fresh full hosted CI is green on the reviewed final head; both current-head Greptile and native substantive reviews are 5/5 with no actionable findings. - Review regression: removing actual policy propagation to preparation in the shared production orchestration fails both real-Postgres cases while runtime selection still chooses warm mode. Restoring it passes. These cases invoke the production helper used by heartbeat, not the entire outer heartbeat service. - The provider-facing transport is a local loopback fixture. No model turn, actual remote platform, physical stop receipt or deployed recovery is proved by these tests. ## Risks - New Codex copies that inherit agent warm mode now use the existing terminal checkpoint path. This can add the expected file checkpoint where only an unchanged probe ran before. - The resolver preserves explicit environment per-turn overrides, pinned admissions and backend/credential gates. It does not create a timer, provider retry or new permission. - Existing pending copies remain pending. This change does not authorize retrieval from stopped or replaced leases and does not establish recovery of historical files. - No database migration, public API field, agent setting, provider change or manual rollout is included. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection and local code execution. Context window and reasoning effort are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
47027e7ba5 |
fix(routines): retain authenticated agent responsibility on creation (#19222)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines save work for later execution under a responsible person's authority. > - An agent can create a routine through its normal REST or native API tools. > - The create route kept the agent creator but dropped the authenticated responsible user. > - The service then selected a parent task's user or a company default, even when the source run had a different user. > - This pull request passes the authenticated responsible user separately from creator attribution. > - Future automatic runs then use the authenticated responsible user saved in the routine revision. ## Linked Issues or Issue Description **What happened?** An authenticated native agent created a routine without a parent task. The stored responsible user became a different company owner. An unrelated parent task could also replace the authenticated user. The defect changes the identity saved for future automatic work. **Expected behavior** Creation must use the server-resolved responsible user from the authenticated run or API key. The agent remains the creator. A person who clicks Run now still supplies that manual run's identity. **Steps to reproduce** 1. Create an active company member and a different company owner. 2. Initialize a native agent run with the member's identity. 3. Use `call_api` to create a paused, self-assigned routine without a parent. 4. Read the routine and its first revision. Before this fix, both use the company owner. **Paperclip version or commit** Reproduced on master `56517dda6c5b32bc901b7dbf1605eaaa041c5fba`. **Deployment mode** Local source integration with real HTTP routes, agent authentication, native tool authority, and embedded PostgreSQL. No provider turn is needed for the reproduction. Related work: [#19128](https://github.com/paperclipai/paperclip/pull/19128) adds board controls for saved responsibility. [#15207](https://github.com/paperclipai/paperclip/pull/15207) repairs synthetic identities from built-in installers. Neither fixes this authenticated create path. ## What Changed - Pass the authenticated agent's responsible user as a separate internal actor field. - Select the board user first, then the agent’s server-resolved responsible user, before the existing parent and company fallback. - Keep creator attribution, initial revision attribution, manual dispatch identity, and existing records intact. - Add real native API, HTTP, and PostgreSQL regressions for identity selection, replay, client spoofing, stale claims, revoked membership, cancellation, cross-company calls, API keys, and fallback behavior. - Document the creation and manual dispatch identity rules in the execution contract. Correct the obsolete responsibility sentence in the agent access map. - Make the existing live-retry route fixture wait for its committed comment’s durable wake before draining. Keep all process and saved-owner assertions. ## Verification - Captured a failing native `call_api` test before changing production code. The persisted responsible user was the unrelated company owner. - The same real route and database boundary passes after the fix. - Initial affected routine and identity checks: 131 passed. The existing pending-AI-card fence also passed. - Follow-up routine checks: 118 passed. The owning scheduled-retry suite passed all 31 tests. - Initial CI serialized shard 7/9 failed with a `23503` wake-to-agent foreign key error during test cleanup. A controlled real-route/PostgreSQL pause reproduced that exact error on unchanged base `56517dda6c`. The drain returned before the route registered its deferred comment wake. The same pause passed after the fixture waited for the actual commentId’s durable wake receipt. The original failure is preserved. - Affected server typecheck after the fixture query change: passed. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - `pnpm test:run`: still running as supporting coverage. The worktree changed during the run: map cleanup, then the six-line fixture wait. Production and routine-test files stayed unchanged. This is mixed-working-tree coverage, not an immutable full-suite pass for a commit. Exact next-head CI remains required. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Exact head `a12e3f6c42b8ad5826fd58faa3cd9553ea251e09`: all CI gates passed. Greptile and the native substantive review both gave 5/5 with no open findings. The native reviewer did not run tests or exercise shadow-mode authorization. - Live provider execution remains unverified. This change does not grant access to another person's AI account. ## Risks New agent-created routines now use their authenticated responsible user before a parent or company default. This is the intended behavior change. The service keeps its fallback when no bound identity exists. Missing agent-key identity still fails authentication. Missing personal AI configuration still blocks execution. Existing routines, revisions, tasks, grants, and manual dispatch behavior do not change. The fixture waits for a persisted wake record before draining; its original live-process and saved-owner assertions remain. ## Model Used OpenAI `gpt-6.1-sol` with reasoning, tool use, and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a090b815b |
fix: retain native checkpoint ownership through warm close (#19225)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native warm session must save its checkpoint under its exact coordinator owner. > - A failed turn can return an accepted result before that checkpoint is saved. > - The executor stopped renewal and released the owner before the managed session closed. > - The backup owner guard then refused the uncommitted, released owner. A failed close could remain in the warm session map. > - This pull request keeps the claim and its existing renewal through checkpoint, close and instruction collection. > - The benefit is a valid checkpoint owner and one fenced release after the work joins. ## Linked Issues or Issue Description Refs #19208, #19206 and #14695. Those fixes cover a destroyed backup source, Stop before a provider session opens, and retained agent-file ownership. They do not change this checkpoint ownership order. **What happened?** A failed warm turn accepted its result. Its coordinator reached `workspace_finalizing`, with a result ID and no lease owner. Managed close then called the durable checkpoint writer. The writer correctly refused this state with `native_harness_storage_owner_missing`. The failed close could remain for the next fresh session to retire. **Expected behavior** The current claim and renewal remain valid until the required checkpoint, close and instruction collection join. Cleanup stops and joins renewal, then releases only that exact claim. A changed owner is not cleared. An earlier error keeps its identity. **Steps to reproduce** 1. Run the new real-Postgres regression against the base commit. 2. Complete a structured failed warm turn through the normal control-plane result producer. 3. Let the actual managed-session close call its production checkpoint callback. 4. Observe the missing-owner refusal after coordinator release. **Paperclip version or commit** Base: `56517dda6c`. The native ownership order also exists on the earlier failing revision. **Deployment mode** Native warm execution with durable harness storage and a managed remote workspace. ## What Changed - Join warm checkpoint, managed close and instruction collection before coordinator release. - Keep the existing lease renewal active through that work. Stop and join it before release. - Run the same fenced release after a checkpoint or collection error. Preserve the initiating error unless restart detachment has won. Keep the typed restart outcome and changed-owner fences. Attach only closed cleanup-failure flags. - Add real-Postgres ownership, delayed-renewal, error, retry and denial cases. Route the managed-session regression through the existing artifact-equipped native test lane. Update the continuity test for the retained claim. Document the active renewal lifetime. ## Verification - Red: the producing coordinator had an accepted result, `phase=workspace_finalizing` and `leaseOwner=null` at managed close. The production checkpoint callback refused it with `native_harness_storage_owner_missing`. - Green: all 9 real-Postgres regression cases pass. They cover a real 20-second renewal tick, a failed checkpoint, one force-fresh successor, and wrong-boot and changed-owner denial. Actual restart detachment wins even when checkpoint and joined renewal-stop both reject. The replacement claim remains unchanged. No new provider turn or cancellation receipt is created. A private canary does not enter the closed diagnostic flags or an error cause. - The executor, restart-recovery and heartbeat compatibility suites pass: 1,259 tests, with 20 gated skips. The container-enabled continuity case was not run locally. Its 21-second timer check remains. It now requires the exact claim during `hasChanges()` and checks release after the joined work. - All 39 shard-routing and combined-coverage tests pass. The new regression runs in the existing native lane with the required Runner artifact. - Final local head `0edcc37f35` passes `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs` and the 9-case regression. The earlier broader compatibility run used identical source bytes. A separate earlier repo-wide local suite on `6cdd68e915` ended with exit 1. Its general-server stage had one GitHub enrollment acknowledgement failure, 31,469 passing tests and 215 skipped tests. That acknowledgement failure reproduced in one narrow run on the same old tree. It is not an established flake or proven caused by this fix. Later repository stages did not run. No duplicate broad suite or full local pass is claimed. - The first published head failed review for detach-error precedence and the old continuity assertion. Its server shard also failed because the new test lacked a Runner daemon in the general lane. Those failures remain recorded. This candidate fixes their causes. Both exact-head reviews are now 5/5. The hosted native-server lane ran all 9 new ownership tests successfully. One unchanged tool-gateway rate-rule test failed in the first current-head CI run. It passed previously and passed locally, both narrowly and in its full 183-test file. The single same-head failed-job rerun passed. All current-head hosted gates are clean. The initial failure remains recorded; no test assertion was weakened. - The provider transport is an in-memory fixture. The managed wrapper, production checkpoint callback, control-plane result and Postgres ownership are real. No paid provider or live remote process is exercised. This does not prove a deployed recovery or a remote stop receipt. ## Risks - A slow required checkpoint extends the lifetime of the existing active renewal. Its cadence stays at 20 seconds for remote claims and 5 minutes for local claims. It runs only while the admitted checkpoint, close and collection remain active, then stops and joins on completion or unwinding. The tests observe a real remote renewal tick. No idle timer, new retry budget or new worker is added. - The release still checks company, task, run, owner, attempt, boot, PID, process start and expiry. Restart detachment retains its recovery owner. - The storage owner guard is unchanged. No migration, permission, provider setting or deployment change is included. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, local code execution and read-only staging metadata. Context window and reasoning effort are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
b8e8ab4641 |
fix(recovery): continue saved input after a recovered sandbox stops, not after its destroy (#19224)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A board user can stop or interrupt a native run and send a message for the next turn > - When the stop leaves sandbox files unsaved, automatic recovery saves them, re-stops the sandbox, and the saved message continues > - For agents whose sandbox is destroyed after each turn (an ephemeral lease), recovery then queues the destroy. That step removed the lease's stop receipt > - The continuation gate waits for a stop receipt, so the message waited for the destroy and its cleanup retry, about 5 to 8 minutes > - Agents with retained sandboxes continued within seconds, so the delay looked agent specific > - This pull request keeps the verified stop on the cleanup lease and lets the continuation and the next remote writer accept it > - The benefit is that a redirect, Retry or interrupt continues right after its files are saved, for every lease policy ## Linked Issues or Issue Description Refs #19206 **What happened?** On hosted staging, a Stop or `interrupt: true` comment on a native run left a follow-up turn that started 5 to 8 minutes later for some agents, and 25 to 40 seconds later for others. The slow agents use ephemeral sandbox leases. In each slow case, the stopped run's lease was released by `pending_cleanup_retry` about 5 minutes after recovery, and the saved message started one second later. The fast agents use retained leases, which recovery releases at once. **Expected behavior** The saved message continues right after recovery saves the files and the sandbox is stopped. The destroy of a stopped sandbox is resource cleanup and must not hold the next turn. **Steps to reproduce** 1. Use an agent whose sandbox lease is ephemeral (destroy after each turn). 2. Interrupt or stop its native run so that recovery must save the sandbox files. 3. Post a message. See the follow-up start only after the sandbox destroy finishes. **Paperclip version or commit** Hosted, at `56517dda`. ## What Changed - `server/src/services/workspace-recovery.ts`: `scheduleRecoveredWorkspaceCleanup` keeps the verified stop receipt as `workspaceRecoveryCleanup.stopReceipt` when it queues the destroy. - `server/src/services/remote-execution-termination.ts`: new `hasRecoveredCleanupStopReceipt(lease)`. It is true only when the lease has recovery cleanup for its own run, was released, and the carried receipt matches the lease, run, provider and provider lease id. - `server/src/services/explicit-native-continuation.ts`: the `remote_cleanup` gate accepts that receipt. - `server/src/services/execution-blocker.ts`: `hasPendingRunCleanup` ignores a lease with that valid receipt, so a saved native interrupt send (and Retry or fresh-workspace checks) does not wait for the destroy. Other cleanup holds are unchanged. - `server/src/services/remote-workspaces.ts`: the next remote writer accepts that receipt for the ended writer's lease. - `server/src/services/execution-blocker.ts` (from #19206 review, not in the merged head): pre-turn evidence uses the exact startup lifecycle event list, so turn-scoped events such as `workspace.diff.recorded` count as provider work. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red before the fix, green after: `cd server && npx vitest run src/services/workspace-recovery.test.ts -t "admits the interrupt redirect"`. An interrupted native run with recovered files and a queued destroy blocked the user message with `remote_cleanup` before the fix. After the fix the message is admitted. A lease without the stop receipt, or with a receipt for another run, still blocks with `remote_cleanup`. The test runs the same input as a plain message and as a saved native interrupt send (`queuedCommentInterruptId`); `hasPendingRunCleanup` was true before the fix and blocked the interrupt send with `cleanup_pending`. - `cd server && npx vitest run src/services/remote-workspace-reservation.test.ts -t "ended writer"`: the next writer reserves the workspace from an ended writer whose recovered lease awaits destroy; a missing or foreign receipt still blocks. Removing the new branch fails the positive case. - Updated `saves the authoritative remote workspace after native failed ...`: Retry now continues before the destroy, and the cleanup lease carries the stop receipt for its run and lease. - `workspace-recovery.test.ts -t "provider work cannot be excluded"` includes `workspace.diff.recorded` as provider work. - Related suites pass: `workspace-recovery.test.ts`, `explicit-native-continuation.test.ts`, `remote-workspace-reservation.test.ts`, `remote-workspaces.integration.test.ts`, `remote-workspace-lifecycle.test.ts`, `task-retry*.test.ts`, `native-cancelled-workspace-preservation.test.ts`, `workspace-recovery-source.test.ts`. `cd server && npx tsc --noEmit` passes. - Hosted evidence (read only): slow cases have ephemeral leases released by `pending_cleanup_retry` about 5 minutes after `issue.workspace_recovered`, with settlement one second later. Fast cases have retained leases released by recovery. ## Risks - Low to medium. The next turn can start while the old stopped sandbox is still being destroyed. The receipt proves that the sandbox was stopped after the files were saved, and an ephemeral successor uses a new sandbox. A destroy failure stays with the existing cleanup retry. - Leases without the carried receipt behave as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
beb73381ca |
fix(native): let a Stop before the provider session opens continue once (#19206)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f5891564ec |
fix(recovery): continue from a retired sandbox's backup and hand spent export repairs to the board (#19208)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native tasks keep a warm sandbox and a verified per-turn backup of the provider session. > - Recovery holds stop a task when continuation is not safe. They must have a next action and an owner. > - In a staging soak, two kinds of held tasks could not move forward. Users followed the product's own instruction, and nothing changed. > - First, a reply to a task whose warm sandbox Paperclip had already destroyed failed every time. The label then said "Send a message to continue" again. > - Second, an exhausted system workspace repair stayed active and system-owned for hours. The board saw no owner, and a reply started nothing. > - This pull request lets a reply continue from the backup when the source sandbox is provably gone, and hands a spent system repair to the board with a named Retry. > - The benefit is that productive work always has a next action that can succeed, and nothing replays or runs twice. ## Linked Issues or Issue Description **What happened?** (a) A native warm task ran one successful turn. A later legacy follow-up retired the warm sandbox lease. The next user reply got a new sandbox (`created` lease). The executor found a verified backup but no reusable lease, so it threw `runner_harness_state_mismatch: backup_without_reusable_lease`. The run retried on the restart-recovery path, failed `native_remote_recovery_lease_mismatch`, and ended as `native_continuation_requires_reconciliation`. Workspace recovery then said "Saved work recovered. Send a message to continue." Each new reply repeated the same failure. (b) A run failed `native_workspace_sync_out_retry_exhausted`. System workspace recovery used its 6 bounded attempts and became `exhausted`. The recovery action stayed `active` with `ownerType: system`. The UI hides system-owned recovery actions, so the board saw no action. Nothing ran for more than two hours. **Expected behavior** (a) When Paperclip destroyed the source sandbox and no provider work followed the backup, the reply continues once from the backup. Ambiguous states still fail closed. (b) When the bounded budget is spent, the action names a board owner and a Retry action that can progress it. **Steps to reproduce** (a) Finish a warm native turn. Retire its reusable lease through obsolete-lease cleanup. Send a reply. The new run gets a `created` lease and fails `backup_without_reusable_lease`. (b) Let system workspace recovery reach its deadline or retry limit. Read the recovery action. It is still `ownerType: system`. Related merged work checked first: #19145, #19184, #19187, #19189, #19190, #18943 (added the guard for migrated sessions). Open recovery PRs (`gh pr list --search recovery --state open`) do not touch these paths. ## What Changed - `native-retired-backup-source.ts` (new): `retiredSourceAuthorizesNativeBackupRestore` decides whether a created lease may restore the backup. All must be true: the backup's source lease is `expired` or `failed`; it has a provider termination receipt with state `destroyed`; its `nativeHarnessBackup` stamp names this exact manifest SHA-256, session and runner instance; exactly one host lease maps to the provider allocation; and no provider durable event for the session was recorded after the backup completed. The provider event type set moved here and the executor imports it. - `native-session-executor.ts`: `shouldRestoreNativeHarnessBackupIntoSandbox` accepts `retiredSourceContinuation`. A created lease with a configured reusable lease evaluates it, beside the existing migrated-session path. The restore uses the existing verified restore and stamps the new lease. - `system-recovery.ts`: `updateSystemRecovery` moves an `exhausted` action to `ownerType: board` and writes a next action and workspace recovery message that name the attempt count, the last failure code, and **Retry recovery**. Retry (`requestWorkspaceRecovery`) already starts a new bounded generation and returns ownership to the system. - `doc/execution-semantics.md`: documents both rules. ## Verification - Red before the fix, green after: - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "restores a warm backup into a created lease only when"`. Before: the destroyed-source case rejected with `runner_harness_state_mismatch: backup_without_reusable_lease`. After: it restores the runner state and provider home into the new sandbox and stamps the new lease. - `npx vitest run src/services/workspace-recovery.test.ts -t "hands an exhausted system export repair"`. Before: `exhausted` with `ownerType: "system"`. After: `ownerType: "board"`, the blocker says Retry recovery and never says "send a message", no provider run or lease resume happens, and Retry starts one new system generation. - Safety cases (fail closed): a stopped (not destroyed) source, a stamp for another checkpoint, provider work after the backup, a still-reusable `released` lease, a missing receipt, and two host leases for one allocation. These run in the executor test and in `native-retired-backup-source.integration.test.ts` against embedded Postgres. - Related suites pass: `workspace-recovery.test.ts`, `system-recovery-state.test.ts`, `native-session-executor.test.ts`, `task-retry.test.ts`, `explicit-native-continuation.test.ts`, `workspace-recovery-source.test.ts`, `native-workspace-export-retry.test.ts`, `native-remote-retry-admission.test.ts`, `native-workspace-warm-reuse.test.ts`, `native-workspace-sync-retention.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `execution-control-work.test.ts` (2,224 tests). - `npx tsc --noEmit -p server/tsconfig.json` passes. - Not done: a live run against staging. Staging access was read-only. The staging evidence is the run event timeline described above. ## Risks - (a) A backup restore after a lost sandbox could rewind a session if newer provider state existed. The predicate requires a provider `destroyed` receipt and no provider event after the checkpoint. A stopped sandbox, a pending cleanup, or a still-reusable lease still fails closed with the old error. - (a) The event-time check can fail closed if a late event from the backup turn is written after the checkpoint time. That keeps the old behavior; it never authorizes a restore. - (b) The action's next action text stays as the exhausted message while a later Retry generation runs. The task blocker shows the live workspace recovery message first, so users see "Recovering saved work…" during that time. - Not changed: when the predicate fails closed, the run still follows the existing reconciliation path. That path can still end with the "Send a message to continue" receipt. That residual is narrower now and needs a separate change to the workspace recovery receipt. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with tool use (shell, file edits, read-only browser inspection of staging). Default context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2ce5f03cd8 |
fix(workspaces): name remote restore failures and leave a board recovery action (#19207)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task restores its saved workspace into a sandbox before the agent starts. > - When that restore fails, the run records only `remote_workspace_operation_failed: operation=restore exit=1`. > - The run then fails as `setup_failed`. No retry starts, and a `blocked` task gets no recovery action. > - An operator cannot tell which step failed, and nobody owns the next step. > - This pull request makes the sandbox scripts name their failed step in a closed grammar, and it gives a restore failure a bounded retry or a named board recovery action. > - The benefit is a visible cause and a clear owner. Saved work stays protected. ## Linked Issues or Issue Description Refs #19152 and #19187. Those changes name copy-back (save) failures. Refs #18852 and #19195. Those changes added the bounded bootstrap retry that this change reuses. **What happened?** A board reply woke an agent on a `blocked` parent task. The run failed during setup 13 seconds later. The only run event was `remote_workspace_operation_failed: operation=restore exit=1`. The run had no workspace operation row. The task stayed `blocked`, and its recovery actions showed `active: null`. The same opaque family also appeared for workspace preparation (`exit=1` only). **Expected behavior** The run names which restore step failed, without command output or paths. A failure before provider work uses a bounded retry when a retry can help. Otherwise, and when the retry budget is spent, the board gets a named recovery action. The saved version is never changed or overwritten. **Steps to reproduce** 1. Save a remote workspace, then start a new run on a different allocation. 2. Put a folder at the published workspace path in the sandbox before restore. 3. Observe that the base code records only `operation=restore exit=1`, then `setup_failed`, with no recovery action for a `blocked` task. **Paperclip version or commit** Base: `87dfba6818`. **Deployment mode** Legacy runtime with sandbox (Daytona) remote workspace storage. ## What Changed - The prepare, repository, and restore sandbox scripts track their current stage. On failure they print `remote_workspace_<op>_failed: stage=<stage>` plus closed fields: `reason=` (the script's own snake_case refusal), `archive=` (the archive helper's fixed reason), `code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git output), and the existing `exit`/`signal`/`program` fields. - The server accepts that text only when it matches a strict grammar for the same operation. Other output is dropped, as before. - Provider refusals during the restore upload and restore command are recorded by closed labels, like copy-back. - A failed saved-workspace restore raises `RemoteWorkspaceRestoreError` with the server step (`snapshot_lookup`, `snapshot_download`, `sandbox_restore`). It is retryable only when the owner re-reads the same writer, epoch, and saved version with no unsaved work, and the sandbox did not name a refusal reason. - Setup failure handling classifies a pre-dispatch legacy restore failure as `remote_workspace_restore_failed`. A retryable failure uses the existing bounded bootstrap retry. A non-retryable or exhausted failure records a board-owned recovery action with the redacted cause, when stranded-work recovery does not scan the task status (for example `blocked`). - Stranded-work recovery gets a named notice for this code. Generic continuation recovery does not grant it a second budget, and legacy reconciliation does not invent provider actions for it. - `doc/execution-semantics.md` documents the contract. ## Verification - Red: the new transport tests failed on the base code with `remote_workspace_operation_failed: operation=restore exit=1` (and the same for prepare and repository). The new completion tests failed because the run became `setup_failed` with no retry and no recovery action. - Green: `npx vitest run src/services/remote-workspace-transport.test.ts src/services/heartbeat/run-completion.test.ts src/services/remote-workspaces.integration.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/services/legacy-execution-recovery.test.ts src/services/recovery src/services/remote-workspace-save-concurrency.test.ts src/services/native-runtime/native-workspace-finalizer-failure.test.ts src/__tests__/native-workspace-finalization-work.test.ts` passes (563 tests). - The transport tests use real sandbox commands. They prove an occupied folder reports `stage=inspect_root reason=occupied_root` and keeps its files. They prove a wrong manifest reports `stage=extract archive=manifest_mismatch`. They prove a missing branch reports `stage=checkout_branch`, and a changed origin reports `stage=verify_origin`. Forged output with a path is dropped. - The real-PostgreSQL cold-restore test proves an occupied root gives a non-retryable `sandbox_restore` failure and keeps the unknown files. - The completion tests cover retried, exhausted, refused, post-dispatch, and `in_progress` cases. - `tsc --noEmit` for the server reports no errors in the changed files. ## Risks - A retry keeps the same restore path. If the cause is deterministic but unnamed, the run uses its two retries before the board action. Named refusals skip the retries. - The board action is recorded only for statuses that stranded-work recovery does not scan, and only when no other action is active. This prevents a second recovery identity for one failure. - Script stderr grammar is new. Old sandboxes run the script that the server sends, so no runner update is necessary. - No migration, deployment, or credential change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use, code execution, and browser tools for read-only staging evidence. Context window and reasoning mode are host-managed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
TonioandClaude Opus 5.5
|
9e9a628a57 |
feat(ui): turn Artifacts V1 on for every instance and hide its toggle (#19164)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Artifacts show the output of agent work on the Artifacts page, in the panel Artifacts tab, and in the task thread > - #17228 added the Artifacts V1 presentation behind an experimental setting that is off by default > - The presentation is now ready for all users, so an opt-in toggle is no longer necessary > - A new default alone does not reach all instances, because each experimental write persists every flag > - This pull request removes the toggle and turns V1 on for every instance > - The benefit is one artifact presentation for all users, with an off switch for hosting operators ## Linked Issues or Issue Description Refs #17228 Refs #17263 **Is your feature request related to a problem? Please describe.** Artifacts V1 shows only on instances where an admin turned on an experimental setting. Other users see the previous cards. **Describe the solution you would like** Show Artifacts V1 on all instances. Remove the toggle from Experimental settings. **Describe alternatives you have considered** A new default without a change to the hook. This does not work, because many instances have a stored false that nobody chose, and no toggle would remain to change it. ## What Changed - Removed the Artifacts V1 card from Instance Settings, Experimental. - Changed `enableArtifactsV1` to default on in the schema, the feature catalog, and the settings service. - Changed `resolveArtifactsV1Enabled` to ignore a stored false. It honours false only when the value comes from managed config. - Kept the flag and the previous cards, so a hosting operator can hold V1 off. - Kept the key in the Cloud staging list, so a managed false does not hold V1 off on staging. - Updated unit tests, the Artifacts V1 plan, and the directions-lab Today reference. - Moved the attachment and upload e2e specs and the runner download helper to the V1 rows. - Added V1 tests for a Markdown review: expansion, deep link, and open, raw and download actions. ## Verification - `cd ui && npx vitest run`: one test file failed before its fix. That file tests the previous cards, and it now mocks the hook to off. I reran it with the other changed UI tests, and they pass. I did not repeat the full UI run after the fix. - `npx playwright test --config tests/e2e/playwright.config.ts tests/e2e/text-attachment-tabs.spec.ts tests/e2e/artifact-uploads.spec.ts tests/e2e/artifact-tab-arrival.spec.ts`: 10 tests pass on V1. - `tests/e2e/cloud-auth.spec.ts` failed once in CI on a task heading. It does not touch artifacts, and 10 local runs pass. - `cd server && npx vitest run src/__tests__/instance-settings`: 183 tests pass. - `cd packages/shared && npx vitest run src/feature-catalog.test.ts src/validators/instance.test.ts`: 35 tests pass. - `pnpm run check:token-gates` and the `ui` and `server` typechecks pass. I did not run the full repository check. - In the artifact directions lab, with no stored settings, the Today right panel renders the V1 rows. - I did not test this on a live instance. After merge, confirm on staging that the Experimental page has no Artifacts V1 card and that all three surfaces show V1. ## Risks - All users see V1 after the next release, not only staging users. To revert, revert this pull request. - Users cannot turn V1 off. Only a hosting operator can, through managed config, and not on Cloud staging. - An instance with a managed `enableArtifactsV1: false` stays on the previous cards. The cloud harness source has no reference to this key, but I cannot read the live fleet configuration. - V1 surfaces render before settings load. An operator-held instance can briefly show V1 before the previous cards. - Admins who use the API can still write the flag. The write has no effect on unmanaged instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ea42a2e417 |
fix(recovery): coalesce duplicate legacy no-replay holds (#17055)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery records keep uncertain execution outcomes from being replayed. > - A settled record can still block replay while it waits for a verified decision. > - Repeated scans can create more settled records for the same failed execution. > - Resolving only one duplicate leaves the task behind another hold. > - This pull request retains the settled hold and applies one verified decision to its exact duplicates. > - The task can continue through the existing delivery path without creating another delivery owner for each duplicate. ## Linked Issues or Issue Description Refs #14600. This change addresses duplicate settled holds and their same-source resolution. The visibility and other deferred-wake paths remain outside this change. Related: #10011 handles unchanged active incidents. This patch handles settled no-replay records. #15285 covers missing reconciliation input. #17031 covers a handoff to a new task owner. External-adapter continuation is covered separately by #15982. ## What Changed - Retain a resolved legacy no-replay decision when a later scan finds the same company, task, cause, fingerprint, and source run. - After a verified reconciliation, resolve duplicate records that also match the selected action's kind. Keep the selected action as the delivery owner for this group. Record the merged action IDs in the activity log. - Preserve independent workspace and system recovery records, existing reconciliations, and records for other sources. - Add PostgreSQL regression tests for repeated scans, historical duplicates, delivery count, repeated resolution, scope boundaries, simultaneous resolutions, and rejected reconciliation rollback. - Update the execution semantics and recovery maps. This patch has no schema migration or adapter-specific code. ## Verification Current head: `3c84bc89325d40bfdda5d28f24ed2fb67d48756c`. It merges `master` at `043f2514e67c50320433a0013fb228a75a51325c` without rewriting the existing commits. The only conflict was in `markExecutionReconciliation`. The resolution retains the upstream transaction and all three queue notifications. Source, sibling, and audit writes use the same transaction. The final merged tree passed: ```sh pnpm exec vitest run server/src/__tests__/issue-recovery-actions.test.ts server/src/services/legacy-execution-recovery.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts server/src/__tests__/execution-control-work.test.ts --maxWorkers=1 # 549 passed; zero skipped pnpm --filter @paperclipai/server exec tsc --noEmit # exit 0 node scripts/check-maps.mjs # passed: 35 areas, 191 capability IDs git diff origin/master --check # passed ``` Three new cases add real PostgreSQL and HTTP-route evidence. Two resolution requests target different duplicate holds while both wait on database locks. They leave one continuation owner, one coalescing audit record, and one queued successor. A live-provider rejection leaves all records unchanged. A checkout rejection after reconciliation writes rolls back the source, siblings, and audit row. The fixture occupies the dispatch slot, so no external model provider runs. The new contention case was also run with only `execution-recovery-resolution.ts` replaced by its parent revision at `65f59e339a9fb6e0635bcbd8c36791bd610326a6`. It failed with two continuation owners instead of one. The source was restored byte-for-byte, and all three new cases passed again before the merge. The final four-suite run above validates the merge with the upstream notification changes. The original positive regressions also failed on that parent revision: three scan-and-settle cycles left four records instead of one, and resolving one historical hold left its two duplicates unreconciled. The scope matrix covers company, task, kind, cause, fingerprint, source run, replay state, protected recovery records, and prior reconciliation. Fresh-checkout collection initially failed because plugin SDK build outputs were absent. After building them, PostgreSQL startup exposed native library symlinks missing from an installation with scripts disabled. The package's symlink setup resolved that failure. The standard server typecheck attempt started a Rust vendor prerequisite build and was stopped; direct TypeScript checking passed. These failed or incomplete attempts are not counted as passes. Full repository build, recursive typecheck, and CI were successful on the earlier `08e5c0285` revision ([CI run](https://github.com/paperclipai/paperclip/actions/runs/39541949702)). They are historical evidence, not results for this merge. The current local proof is the four suites and checks listed above. Full upstream CI for the merge is pending. Greptile reviewed `3c84bc89325d40bfdda5d28f24ed2fb67d48756c` with 5/5 confidence and no actionable findings; that source review is separate from test execution. ## Risks - A match that is too broad could clear an independent hold. Tests cover company, task, kind, cause, fingerprint, source run, replay state, workspace recovery, system recovery, and prior reconciliation boundaries. - The existing issue-lock transaction covers the selected action and its duplicates. The patch does not create a new lock or an automatic retry path. - The patch does not infer that a network or provider failure means no action occurred. Existing reconciliation validation still checks the source, owner, stopped processes, leases, and observed action outcome. - Previously reconciled records and their delivery state remain unchanged. This patch prevents new duplicate delivery owners when it resolves unreconciled duplicates. It does not migrate every historical delivery record. ## Model Used OpenAI Codex, based on GPT-6, assisted with the implementation, tests, and review. It used repository tools and code execution. The exact runtime model ID, context window size, and reasoning setting are not exposed in this session. An independent AGY review also ran; its provider model ID was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with the available version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked the existing issue with `Refs #14600` - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task IDs or instance details - [x] I have run the focused tests locally and they pass; full upstream CI is pending for the final merge - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Rockiez <7697772+Rockiez@users.noreply.github.com> Co-authored-by: Devin Foley <devin@paperclip.ing> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
87dfba6818 |
fix(native): retry an unpublished accepted task reply durably instead of dropping it (#19200)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run commits its status decision and its final reply comment
in separate steps
> - Background presentation recovery publishes an accepted reply when
the live path does not record one
> - That worker dropped the run after one refused attempt and only threw
on an error, with no record on the run
> - A task could then show done while the user never saw the answer,
until an unrelated wake reloaded the queue
> - This pull request makes the publication of an accepted ordinary task
reply durable, bounded, and recorded
> - The benefit is that a task cannot close silently without its answer
## Linked Issues or Issue Description
Refs #19130 (verify requested publication at finalization, for
documents), #19183 (deliver messages saved while a native run
finalizes), #19186 (stale owner done). This change does not duplicate
them. It covers the final reply comment.
**What happened?**
A native run accepted a result with disposition `done`. The status
decision set the task to done. The accepted final reply was not
published as a task comment. The run had no publication record (no
`run.presentation.resolved` event and no `presentationDecision`). The
reply was published by background recovery 58 minutes later, after an
unrelated wake reloaded the presentation queue.
**Expected behavior**
The accepted reply is published promptly. A refused or failed attempt is
retried with backoff and never posts twice. If publication fails
permanently, the run and the task show a named failure with an owner.
**Steps to reproduce**
1. Commit an accepted native task result with disposition `done`, with
the final reply not yet visible to presentation recovery.
2. Run the native response presentation worker check once. The attempt
is refused.
3. On `master`, the worker returns `null` and drops the run. Nothing
records the attempt. Nothing retries it until another notification for
that scope.
**Paperclip version or commit**
`master` at
|
||
|
Devin FoleyandPaperclip
|
07a2941f0c |
fix(recovery): reclaim verified inactive provider caches on full disks (#19192)
Recover from a full-disk workspace save by reclaiming only a verified inactive provider installation under the existing source and maintenance authority. Commit the preparation inventory in Core before changing cache contents. Bind interrupted cleanup to stable filesystem identities and keep small staged generation markers when directory birth times are unavailable. Skip read-only or permission-denied assets and stop freeing blocks once marker space is available. Preserve task files, Git, session history and uncertain action receipts. Only the normal saved-workspace and provider-stop receipts permit continuation. Validated with 446 complete recovery regressions on the repair revision, subsequent focused recovery and transport suites, typecheck/build, and all exact-head hosted CI. Greptile reviewed the final head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
037c58bb22 |
fix(daytona): keep the provider auto-stop strictly after the warm runner idle window (#19199)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm remote agent keeps its sandbox running after a turn. The host keeps a warm runner for an idle window (`runnerIdleTimeoutMs`, or the agent's `idleTimeoutMs`) > - The Daytona provider also stops an idle sandbox after `autoStopInterval` minutes. The default is 15 minutes > - On staging, the warm idle window was 900,000 ms (15 minutes). The two windows were equal > - A follow-up turn at the end of the warm window resumed a sandbox that Daytona was stopping. Daytona refused the resume in about 4 seconds, and the run failed at setup > - #19195 added a bounded retry for that refusal. It did not remove the cause > - This pull request makes the two windows non-overlapping by construction. Daytona derives its auto-stop for a warm reusable sandbox from the same warm idle value that the host uses > - The benefit is that a follow-up inside the warm window always finds a running sandbox, and idle sandboxes still stop ## Linked Issues or Issue Description Refs #19195 (bounded retry for a refused resume; this change removes the common cause). **What happened?** On a staging soak, three follow-up runs failed at setup with a refused sandbox resume. Each resume began 15 minutes 8 to 25 seconds after the previous turn released the sandbox. The warm idle timeout was 900,000 ms. Daytona's default `autoStopInterval` is 15 minutes. Nothing in the code kept these two values apart. **Expected behavior** A reusable sandbox that keeps a warm runner must not auto-stop inside the warm idle window or at its edge. The provider backstop must fire after the host's own idle handling, with a margin. Idle sandboxes must still stop. **Steps to reproduce** 1. Configure a Daytona sandbox environment with `reuseLease` and a warm runner with `runnerIdleTimeoutMs: 900000` (or an agent with `lifecycleMode: warm`, `idleTimeoutMs: 900000`). 2. Run a turn. Do not set `autoStopInterval`. 3. Before this change: the sandbox is created with `autoStopInterval: 15`, equal to the warm window. A follow-up near minute 15 races the provider stop. **Paperclip version or commit** `c656b95b6b` (current `master`). ## What Changed - Daytona plugin (`plugin.ts`): add `effectiveAutoStopIntervalMinutes`. For a reusable lease that keeps a warm runner, the effective auto-stop is at least `ceil(warm idle / 1 min) + 5` minutes. The plugin applies the floor only when the host forwards `warmRunnerIdleTimeoutMs`, or when the environment itself selects `runnerLifecycleMode: warm`. An explicit `0` (disabled), a longer configured value, `inherit` with a per-turn agent, per-turn, and non-reusable sandboxes keep the configured value. - Daytona plugin: use the effective value at sandbox creation. On resume, raise a shorter auto-stop on an existing sandbox. This update is best effort and bounded to 5 seconds; the #19195 retry still covers a refusal. The stale-handle refresh threshold keeps the configured interval, so a cached handle is not trusted for longer before the provider confirms the new interval. The warm window is not written to the saved normalized config. - Host (`runner-environment-lifecycle.ts`): add `resolveWarmRunnerIdleTimeoutMsForRun`. It returns the run's effective warm idle window (environment `runnerIdleTimeoutMs`, else the agent `idleTimeoutMs`) for a warm `paperclip_runner` sandbox run. It shares the lifecycle-mode resolution with `resolveRunnerEnvironmentForRun`. - Host (`environment-run-orchestrator.ts`, `environment-runtime.ts`): pass that value as `warmRunnerIdleTimeoutMs` to the sandbox-provider driver on acquisition and on remote recovery resume. The driver adds it to the provider worker config only. Lease identity, reuse matching, and lease metadata are unchanged. - Daytona manifest: describe the warm floor on `autoStopInterval`. - `doc/execution-semantics.md`: document the contract. ## Verification - Red (before the fix, same tests, with the floor disabled): 5 failures in `plugin.test.ts` "warm idle window versus Daytona auto-stop". The staging case failed with `expected 900000 to be greater than or equal to 1200000`. The resume case failed because `setAutostopInterval` was not called. - Red (host): the new `environment-runtime.test.ts` case first failed because the provider worker config had no `runnerIdleTimeoutMs`. The service wrapper dropped the field. The fix forwards it. - Green: `npx vitest run --config vitest.config.ts` in `packages/plugins/sandbox-providers/daytona`: 563 passed. This includes a setter that never settles (resume completes after the 5-second bound) and `inherit` with no forwarded window (explicit `autoStopInterval: 2` stays 2). - Green: `npx vitest run src/services/runner-environment-lifecycle.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/environment-run-orchestrator.test.ts` in `server/`: 242 passed. This includes recovery resume forwarding the admitted warm window. - The regression tests assert the relationship `autoStop >= warm idle + margin` for several warm windows (125 s, 300 s default, 900 s, 1,800 s) and for an explicit `autoStopInterval: 15`. They are not constant checks. - `node scripts/typecheck-low-memory.mjs` passes. `pnpm typecheck` in the Daytona package passes. ## Risks - A warm reusable Daytona sandbox now stays running up to the warm window plus 5 minutes after its last activity, instead of 15 minutes. With a 15-minute window this is 20 minutes. Cost is bounded and idle sandboxes still stop. - The resume-time `setAutostopInterval` call is best effort. If it fails, the old interval stays and the #19195 retry still applies. - Other providers are unchanged. They receive one extra host-owned config key (`warmRunnerIdleTimeoutMs`) only for warm `paperclip_runner` runs, and they ignore unknown keys. - No lease identity change, so existing retained sandboxes are still reused. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c656b95b6b |
fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action
## Linked Issues or Issue Description
Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.
**What happened?**
On a staging soak, two follow-up turns failed at setup after a
successful first turn:
- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.
Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.
**Expected behavior**
A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.
**Steps to reproduce**
1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.
**Paperclip version or commit**
Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).
## What Changed
- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.
## Verification
- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.
## Risks
- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
245b4d3b79 |
Preserve cancelled workspaces after controllerless Stop (#18820)
Admit cancelled workspace preservation after an audited Stop acknowledgement when no live controller received an interrupt. Retain the strict dispatched receipt for provider termination and operator handoff. Existing source, lease, coordinator and result ownership checks remain enforced. Validated with complete Stop/preservation suites (134 tests), real-file recovery, workspace typecheck and build, and green hosted CI. Greptile and Paperclip both reviewed the exact head at 5/5 with no unresolved findings. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
3b17b5ce15 |
fix(native): keep a reassigned-away run from closing the new owner's task (#19186)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native agent run reports "done" in an accepted result. A finalizer then decides the task status. > - The board can reassign a task while the run is in flight. The old run is cancelled, but its accepted result still goes to the finalizer. > - The finalizer did not check who owns the task now. The old owner's "done" closed the task of the new owner. > - The new owner never did the work, and the task shows done with no reply from the current owner. > - This pull request makes a done report apply only when the reporting run's agent still owns the task. > - The benefit is that reassignment keeps its meaning: the new owner keeps the open task, and the old result stays as history. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, agent A started a native run on a task. Agent A finished the work and submitted a "done" result. Then the board reassigned the task to agent B, and Paperclip cancelled agent A's run with `issue_reassigned`. About 45 seconds later, the native finalizer committed agent A's done decision with reason `completion_claim_policy_accepted`. The task became done. Agent B's runs were then cancelled because the task was terminal. Agent B never worked on the task. **Expected behavior** A done report from a run that no longer owns the task must not close the task. The accepted result stays as history. The task status stays as it is for the current owner. **Steps to reproduce** 1. Assign a low-risk native task to agent A and start a run. 2. Submit a done result for the run. 3. Reassign the task to agent B before finalization. 4. Finalize the run. Before this change, the task becomes done. **Paperclip version or commit** Observed on staging at `b9fbbf8d30`. Reproduced on `master` at `06826915d7`. Related: #19130 adds a publication check for requested documents at finalization. It does not check task ownership. ## What Changed - `status-arbiter.ts`: add `ownsNativeTaskCompletion` and `supersedeStaleOwnerCompletion`. A done decision from an agent that is not the current assignee (another agent, a user, or nobody) becomes `preserve` with reason `completion_owner_superseded` and no effects. Other decisions do not change. This is the same ownership rule that the provider-failure path already uses (`ownsProviderFailureDecision`). - `native-run-finalizer.ts`: apply the rule after arbitration and before the deliverable check. Conversation tasks keep their existing path. - `native-status-reassessment.ts`: apply the same rule, so new evidence from a reassigned-away run cannot close the task later. - The status committer does not change. A reassignment during finalization already increments `statusVersion` (migration 0358). The committer then replans, and the retry applies the new rule. - `doc/execution-semantics.md`: describe the rule next to the existing "a late finalizer cannot change a reassigned task" statement. ## Verification New file `server/src/services/native-runtime/native-stale-owner-completion.test.ts`. It uses embedded PostgreSQL, the real coordinator store and the real finalizer. - Control: the owning run's done result closes the task with `completion_claim_policy_accepted`. - Reassigned before finalization: the task stays `in_progress` with agent B. The decision is `completion_owner_superseded`. The accepted result row is unchanged. - Reassigned after arbitration and before the status commit: the commit replans and the task stays `in_progress`. - Reassessment control: later approved evidence closes the owner's task with `decision_superseded_by_new_evidence`. - Reassessment after reassignment: later approved evidence from the old run does not close the task. The decision is `completion_owner_superseded`. This case fails when the reassessment rule is disabled. - Pure policy cases: another agent, a board user and no assignee all preserve status. A non-done decision passes through. Red before the fix (same tests, fix not applied): ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 2 failed | 1 passed — both reassignment cases: expected "in_progress", received "done" ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-stale-owner-completion.test.ts # Tests 10 passed cd server && npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts \ src/services/native-runtime/native-completion-feedback.test.ts src/services/native-runtime/native-deliverable-feedback.test.ts \ src/services/native-runtime/native-skill-test-status.test.ts src/services/native-runtime/native-routine-status.test.ts \ src/__tests__/native-status-arbiter-corpus.test.ts src/__tests__/native-status-reassessment-work.test.ts \ src/services/native-runtime/agent-chat-response-recovery.test.ts src/__tests__/native-run-projection-work.test.ts \ src/services/native-runtime/task-reassignment.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts # 11 files, 614 tests passed ``` I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build` locally. CI runs them. ## Risks - A native run that reassigns its own task to a user or to another agent, and then reports done, no longer closes the task through arbitration. The status that the agent set through the API stays. This matches the existing provider-failure ownership rule. - A native run on a task with no assignee cannot close it through arbitration. Native task runs start from an assignment, so this is not expected in normal use. - Review runs and conversation tasks keep their existing paths. Review outcomes already preserve status before this rule. - No schema change and no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness. - Capabilities used: tool use, shell and code execution, repository search, browser reads of staging, and extended reasoning. - The model wrote the change, the tests and this description. A human requested the fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
46fb933b01 |
fix(recovery): keep the stop receipt when a file-preserving restart fails (#19181)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can execute in a remote sandbox. When a run fails, Paperclip stops the sandbox and records the provider's stop receipt on the environment lease. > - System workspace recovery can restart that stopped sandbox to save its files. Before the restart, it removed the stop receipt from the lease. > - If the restart failed and the provider could not stop the sandbox after that, the lease had no receipt. Continuation admission requires a stop receipt on every remote lease of the failed run. So every Retry and every comment waited for an environment that never stopped. > - This pull request keeps the original receipt until the provider confirms the restart. It restores the receipt when the restart fails and the sandbox cannot stop. A restart that fails because the sandbox disk is full now ends recovery with a fresh-workspace outcome. > - The benefit is that a failed file-preserving restart no longer blocks the task permanently. Saved input and Retry continue in a fresh isolated workspace, and the original sandbox stays retained. ## Linked Issues or Issue Description No public issue exists. This is a bug report in the format of `bug_report.yml`. **What happened?** 1. A native run failed because its Daytona sandbox filled its 10 GB disk. 2. Paperclip stopped the sandbox normally and recorded the stop receipt on the environment lease. 3. Some minutes later, system workspace recovery tried to restart the sandbox to preserve its files. Before the restart, `prepareStoppedSandboxForRecovery` deleted the stop receipt. 4. The restart failed on the full disk. The sandbox went into the provider's `error` state. 5. The stop-only teardown after the failed restart also failed, because the provider cannot stop a sandbox in that state. The lease stayed in `pending_cleanup` with no receipt. The cleanup sweep retried the same failing stop with backoff. 6. `admitExplicitNativeContinuation` requires a provider stop receipt on every remote lease of the failed run. Every Retry and every comment was deferred with "Waiting for the previous environment to stop. Your message will start automatically." The message never started. Also, after recovery was exhausted, a Retry replaced the exhausted generation with a new one. That generation repeated the same restart, and the Retry still reported success. **Expected behavior** A lease must not lose its stop receipt while the real sandbox state is unknown or not running. When the file-preserving restart cannot succeed, recovery must end in a state that lets the task continue, for example in a fresh isolated workspace. **Steps to reproduce** 1. Use a native run with a retained remote sandbox lease that has a `stopped` receipt and a pending workspace recovery. 2. Make the provider resume fail (for example with `no space left on device`), and make the provider stop fail. 3. Run `recoverRetainedWorkspace` until recovery is exhausted. 4. Save a user comment on the task and run the deferred-comment admission. 5. Before this change, the lease is `pending_cleanup` without a receipt. The comment stays deferred with reason `remote_cleanup`, and no successor run starts. **Paperclip version or commit** Observed on a hosted staging build. Reproduced on `master` at `6452e964a36` with the new tests in this PR. **Deployment mode** Hosted, with the Daytona sandbox provider plugin. The defect is in core server recovery code, so it is not specific to one adapter. ## What Changed - `server/src/services/sandbox-stop-and-retain.ts` - `prepareStoppedSandboxForRecovery` still writes the stop-only intent and moves the lease to `pending_cleanup` before the provider call. It now keeps the prior `remoteExecutionTermination` (and `sandboxStopAndRetainReceipt`, if present) as `priorStop` inside the `sandboxStopAndRetain` intent. - New `confirmSandboxRecoveryRestart` removes `priorStop` when the provider resume returns. From that point, the sandbox may be running, and only a fresh stop can settle the lease. - New `restoreSandboxStopAfterFailedRestart` restores the prior receipts and the `released` / `success` lease state. It works only while `priorStop` is present, so only when no restart was confirmed. It checks the restored receipt with `hasRemoteTerminationReceipt` first. It uses the same attempt fence as `settleStopOnlyCleanup`, and it writes through `writeEnvironmentCleanup` so dispatch and cleanup wake up. - `settleStopOnlyCleanup` now shares that attempt fence (`stopOnlyAttempt`). Its behavior does not change. - `server/src/services/workspace-recovery.ts` - After `resumeWorkspaceRecoveryLease` returns, recovery confirms the restart before any other check. - If the restart is not confirmed, recovery still tries the stop-only teardown first. A successful stop records a fresh receipt, as before. If the stop also fails, recovery restores the prior receipt instead of leaving the lease without one. - A later attempt that finds a pending stop-only lease (for example after a process restart during the reopen) uses the same rule when its stop fails. - A resume error that reports a full sandbox disk (`No space left on device` or `Disk quota exceeded`) becomes `source_unavailable` / `use_alternative_strategy`. The generation is exhausted at once, and the existing fresh-workspace continuation applies. Other resume errors keep the existing bounded retry policy (`provider_unavailable`, at most six attempts in 30 minutes). Because the receipt stays, that policy also ends in a state that lets the task continue. - `server/src/services/task-retry.ts`: Retry binds to an exhausted generation with a `use_alternative_strategy` failure and no fresh attempt, the same way it binds to a complete or running generation. It does not create a new generation that repeats a step the policy already rejected. The Retry continues in a fresh isolated workspace. An explicit workspace recovery request (`requestWorkspaceRecovery`) still starts a new generation. - `doc/task-recovery.md`: describes the receipt handling around the reopen and the Retry binding. - `server/src/services/workspace-recovery.test.ts`: four new tests (five cases) with the real database and mocked provider calls. ### Design choice I kept the existing write-ahead fence. The lease still leaves `released` and shows no receipt while the provider reopen is in progress. So no admission, reuse or destructive cleanup can act in that window, and a crash still leaves a stop-only obligation. The original receipt is not deleted. It stays in the stop-only intent and is discarded only when the provider confirms the restart. This makes one simple rule: the suspended receipt exists only while no restart has been confirmed. I did not use "restore only when the provider confirms that the sandbox is not running". The Daytona plugin reports a sandbox in the `error` state as `active` in `inspectLease`, and its `stopLease` cannot stop that state. So with the current plugin, no provider confirmation exists for this exact failure, and the hold would stay. A separate change to the Daytona plugin's `stopLease` can recover an errored sandbox before it stops it. This PR does not depend on that change. If that change lands and its recovery succeeds, the existing stop attempt records a fresh receipt, and the restore path is not used. I also did not change continuation admission to accept a lease in `pending_cleanup`. Many gates treat pending cleanup as unfinished work (`hasPendingRunCleanup`, the remote stop check and others). A change there would be wider and more sensitive. The restored receipt is the provider's original receipt, bound to the same company, run, lease, provider, allocation and plugin. No process ID or run status takes its place. ## Verification All commands ran from `server/` in a clean worktree of `master` on macOS (Apple Silicon), with the embedded PostgreSQL test database. Vitest ran from the real path. - Baseline on `master` (`6452e964a36`), before any change: `npx vitest run src/services/workspace-recovery.test.ts` → exit 0, 331 passed. - The new tests fail on `master`. I reverted only the three source files and kept the new tests: - `npx vitest run src/services/workspace-recovery.test.ts -t "file-preserving restart fails|without repeating a restart|keeps the stop obligation once|interrupted restart that the provider"` → exit 1, 4 failed, 1 passed. - In the comment cases, no successor run starts (`expected [] to have a length of 1`). A temporary log (not committed) showed the lease in `pending_cleanup` in both cases. In the `unknown_failure` case, the comment was deferred with `remote_cleanup` / "Waiting for the previous environment to stop. Your message will start automatically." This is the same hold as in the incident. - The guard test "keeps the stop obligation once the provider has confirmed a restart" passes on `master`, as expected. It makes sure that the fix does not restore a receipt after a confirmed restart. - With this change, the same command → exit 0, 5 passed. - `npx vitest run src/services/workspace-recovery.test.ts` on the committed head → exit 0, 336 passed (331 existing and 5 new). - Neighboring suites (`task-retry`, `explicit-native-continuation`, `heartbeat/recovery`, `environment-runtime`, and the other server test files that import the changed modules: `live-recovery-work`, `retained-native-session-cleanup-work`, `execution-control-work`, `legacy-workspace-restore-recovery`, `deferred-execution-receipt-work`, `issue-agent-mutation-ownership-routes`, `native-workspace-export-retry`, `native-cancelled-workspace-preservation`, `native-workspace-warm-reuse`, `native-workspace-reconstruction`, `native-failed-workspace-recovery`, `heartbeat/workspace-preparation`) → exit 0, 16 files, 1456 passed. - `npx vitest run src/__tests__/chat-channels.integration.test.ts src/__tests__/slack-conversation-lifecycle.test.ts src/services/native-runtime/preserved-native-response-wait.integration.test.ts` (other importers of `task-retry`) → exit 0, 3 files, 2074 passed. - `pnpm --filter @paperclipai/server typecheck` → exit 0. - Not run: the full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` repo checks. CI runs them. No live Daytona probe ran. New tests: - `lets saved input continue after a file-preserving restart fails on a sandbox that cannot stop` (`disk_full`, `unknown_failure`): resume fails and stop fails. The lease keeps its original receipt and is `released` / `success`. Recovery is exhausted (after one attempt for a full disk, and after six attempts for an unknown failure). The saved comment starts one successor run with `fresh_isolated_workspace` readiness. The source files stay in the retained sandbox. - `continues a Retry in a fresh workspace without repeating a restart that cannot succeed`: after a full-disk exhaustion, Retry keeps the same generation and binds its intent. The restart is not called again, and the Retry starts one run in a fresh isolated workspace. - `keeps the stop obligation once the provider has confirmed a restart`: resume returns, a later check fails, and the stop fails. The lease stays `pending_cleanup` without a receipt, and the intent has no `priorStop`. - `restores the stop receipt after an interrupted restart that the provider never confirmed`: the lease starts in `pending_cleanup` with a suspended receipt, as after a process restart during the reopen. The stop fails, the receipt is restored, and the attempt continues to the normal reopen. Assumption and limits (runtime dependency rule): the provider calls are mocked, and the database is real. The tests do not exercise the real Daytona error text. The full-disk check matches the patterns that Paperclip already uses for sandbox disk-full errors. If the provider error text does not match, recovery uses the bounded retry path, which now also ends in a fresh-workspace continuation. The change adds one database write after each confirmed reopen and one after a failed reopen and stop. It adds no polling. ## Risks - A resume can fail after the provider started the sandbox, for example when the reply is lost. If the stop-only teardown then also fails, recovery restores a `stopped` receipt for a sandbox that can be running. The old run's processes do not survive the earlier stop, so the receipt is still correct as proof that the old execution ended. But the retained sandbox can run until the provider stops it or a later recovery or cleanup stops it. Before this change, the same case caused a permanent hold. - A full-disk restart failure now exhausts recovery at once. The task continues in a fresh workspace without the failed run's files. The files stay in the retained sandbox. An operator can still request workspace recovery explicitly after the disk is freed. - Retry no longer renews an exhausted generation that has a `use_alternative_strategy` failure and no fresh attempt. For destroyed or missing sources, the result is the same as before but without the extra attempt. - No schema or migration change. A suspended receipt is one extra field in the existing `sandboxStopAndRetain` metadata. Older servers ignore the field. The intent equality checks compare the stored value, so a mixed-version deploy keeps working. - #18820 also changes `server/src/services/workspace-recovery.test.ts`. The two PRs touch different tests. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`), running as a Claude Code agent with tool use (shell, file reads and edits, local test and typecheck runs) and extended reasoning. The runtime does not expose the context window size to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full repo runs are left to CI, as noted above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
895cee28b3 |
fix(daytona): recover an errored sandbox before stop-only cleanup (#19178)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs use sandbox provider plugins. Daytona is one of
them.
> - After a failed run, Paperclip stops the sandbox and keeps its files
("stop-only cleanup"). A task can continue only when every remote lease
of the failed run has a provider stop receipt.
> - Daytona rejects Stop for a sandbox in its `error` state. A sandbox
that filled its disk went to that state. The plugin called Stop on it,
so every cleanup sweep failed with the same generic error.
> - The lease stayed in pending cleanup without a stop receipt. Each
Retry or comment waited forever with reason `remote_cleanup`. A manual
Daytona Recover was the only fix.
> - This pull request makes the Daytona stop-only hook recover a
recoverable sandbox once, with a bound, and then stop it. When it cannot
stop the sandbox, it returns a typed failure. The server escalates a
terminal failure at once instead of retrying it like a transient error.
> - The benefit is that a full-disk sandbox no longer blocks its task,
and a sandbox that cannot recover is visible to operators on the first
attempt.
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A task sandbox filled its 10 GB disk. Daytona then reported the sandbox
as `state: error`, `desiredState: started` and `recoverable: true`. Its
`errorReason` was a Docker `no space left on device` error. The Daytona
`stopLease` hook calls Stop on any sandbox that is not `stopped` or
`archived`. Daytona rejects Stop in the `error` state. The pending
cleanup sweep ran about every 30 minutes, and it failed on every pass.
The lease stayed `pending_cleanup` with `cleanup_status=failed` and no
stop receipt. Continuation admission
(`server/src/services/explicit-native-continuation.ts`) requires a stop
receipt on every remote lease of the failed run. So each Retry and
comment was deferred with "Waiting for the previous environment to stop.
Your message will start automatically." It never started.
A manual Daytona `POST /api/sandbox/{id}/recover` fixed it. The
sandboxes reached `started` in about one minute. The next sweep stopped
them, recorded the receipt, and the tasks continued.
**Expected behavior**
Stop-only cleanup recovers a recoverable errored sandbox and then stops
it. When the sandbox cannot be recovered, cleanup returns a clear, typed
failure that the server can tell apart from a transient error. Cleanup
never deletes the retained sandbox, and it never loops on Recover.
**Steps to reproduce**
1. Run a task in a Daytona sandbox until the sandbox fills its disk and
Daytona shows `state: error`, `recoverable: true`.
2. Let the run fail, so the server records a `sandboxStopAndRetain`
intent and calls `environmentStopLease`.
3. Before this change, Stop fails with `Sandbox failed to stop with
status: error` on every sweep, and the task cannot continue.
4. The new tests in `plugin.test.ts` reproduce this with a mock sandbox
whose Stop fails from `error`.
**Paperclip version or commit**
Observed on a hosted deployment. Reproduced in unit tests on `master` at
`6452e964a36`.
**Deployment mode**
Managed Daytona sandbox execution with plugin-backed stop-only cleanup.
Related: #18172 (Daytona exports under disk pressure) and #19050
(session reset on every Daytona restart path). This PR does not change
the resume path in `ensureSandboxStarted`, which already recovers a
recoverable sandbox.
## What Changed
- **Daytona plugin
(`packages/plugins/sandbox-providers/daytona/src/plugin.ts`)**: after
`stopLease` confirms the retention policy, a sandbox in `error` goes
through `recoverSandboxForStop`.
- Recoverable: the hook calls `sandbox.recover()` once. The SDK waits
for `started`. The plugin bounds the wait to 20 s, plus the existing
liveness margin for a hung connection. Then the hook stops the sandbox
normally and returns the usual `stopped` receipt.
- Not recoverable: the hook makes no Recover or Stop call. It throws
`PluginEnvironmentStopError("resource_unrecoverable")`.
- Recover fails: the hook reads the provider state once. A sandbox back
in `error` gives `recovery_failed`, or `resource_unrecoverable` if
Daytona now marks it so. A sandbox that is still starting, or a failed
read, gives `recovery_incomplete`.
- The recovered sandbox fails again before Stop finishes (for example,
the disk fills again): a fresh read that shows `error` gives
`recovery_failed`. The hook never calls Recover a second time in the
same call.
- The hook never deletes or archives the sandbox. The non-error stop
path is unchanged.
- **Plugin SDK (`packages/plugins/sdk/src/environment-stop-error.ts`,
new)**: `PluginEnvironmentStopError`, `environmentStopErrorData` and
`readEnvironmentStopFailure`. This follows the existing
`PluginEnvironmentRunnerIngressError` pattern. The envelope holds only
one closed reason. The reader validates the reason and derives
`terminal` from it. `resource_unrecoverable` and `recovery_failed` are
terminal. `recovery_incomplete` is not.
- **Worker RPC host (`packages/plugins/sdk/src/worker-rpc-host.ts`)**:
forwards that envelope as JSON-RPC error `data` for
`environmentStopLease` only.
- **Pending cleanup sweep
(`server/src/services/heartbeat/recovery.ts`)**: the catch reads the
validated envelope.
- On a terminal failure, it escalates on the first attempt. It writes
the one-time "needs operator attention" warning and uses the existing 30
minute retry interval. Before, this happened only at the fifth attempt.
- The failure log uses `errorKind: "provider_stop_terminal"` and
includes the closed `providerStopFailure` reason. The sweep still never
logs exception fields.
- The lease stays `pending_cleanup`. Automatic cleanup continues, and an
explicit Retry still bypasses the cooldown. Untyped and transient
failures keep their current behavior.
- **Docs**: the SDK README ("Stop failures"), the Daytona README,
`doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md` and
`doc/native-workspace-finalization-recovery.md` describe the new
behavior.
Shared contract note (module boundaries): the SDK envelope is the
smallest addition that lets the host tell a terminal provider stop
failure from a transient one. The worker host drops all other error data
across the RPC boundary, so the existing contracts cannot carry the
signal. The only consumer is the pending cleanup sweep. When a provider
does not throw the new error, or an older host ignores it, behavior is
unchanged. This PR does not touch
`server/src/services/workspace-recovery.ts` or
`sandbox-stop-and-retain.ts`.
## Verification
All commands ran in a clean worktree from `master`. Vitest ran from the
real path.
- Red on `master` (only the new tests added):
- Daytona, with `plugin.ts` from `master`: `npx vitest run --config
vitest.config.ts src/plugin.test.ts -t "provider error state"` exits 1.
All 7 new tests fail. The main case fails with `Sandbox failed to stop
with status: error`, which is the production error.
- SDK, with `worker-rpc-host.ts` from `master`: `npx vitest run
tests/worker-rpc-host.test.ts -t "typed stop failure"` exits 1 (the
typed case fails).
- Server, with `recovery.ts` from `master`: `npx vitest run
src/__tests__/heartbeat-pending-cleanup-sweep.test.ts -t "terminal
provider stop failure"` exits 1. All 3 cases fail (`errorKind:
"destroy_failed"`, no escalation).
- Green on this branch:
- `packages/plugins/sandbox-providers/daytona`: `npx vitest run --config
vitest.config.ts` exits 0 (534 passed, 26 skipped live tests). `pnpm
typecheck` exits 0. `pnpm build` exits 0.
- `packages/plugins/sdk`: `npx vitest run` exits 0 (222 passed). `pnpm
--filter @paperclipai/plugin-sdk typecheck` exits 0.
- `server`: 15 related suites exit 0 (1498 passed): pending cleanup
sweep, `heartbeat/recovery`, environment runtime, environment cleanup
work, native startup cleanup, native remote failure retention, observed
workspace preservation, retained native session cleanup, idle sleep
safety, explicit native continuation, task retry, workspace recovery,
native failed and cancelled workspace preservation, and Daytona close in
the plugin worker manager.
- `pnpm --filter @paperclipai/server typecheck` exits 0. `pnpm
check:module-boundaries` exits 0.
- New tests:
- Daytona: a recoverable errored sandbox is recovered once
(`recover(20)`) after the retention check and before Stop, then stopped.
An unrecoverable one returns `resource_unrecoverable` with no Recover,
Stop, start, delete or archive call. A failed recovery makes exactly one
Recover call for each outcome (`recovery_failed`,
`resource_unrecoverable`, `recovery_incomplete`). A Recover call that
never settles is cut off by the bound. A recovered sandbox that fails
again before Stop gives `recovery_failed` with no second Recover.
- SDK: the envelope holds only the reason. A forged or non-SDK error
gives no data. Unknown reasons, accessor fields and a sender-supplied
`terminal` flag are not trusted. The worker host forwards the typed
failure and drops a plain error's detail.
- Server: terminal reasons escalate on attempt 1 with the long retry and
one operator warning. `recovery_incomplete` keeps the short retry. The
lease stays `pending_cleanup`, and the provider message never reaches
the logs.
- Not run: the full `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`
and `pnpm run typecheck:build-gaps` repo checks. CI runs them. No live
Daytona probe ran. The mocks model the Daytona SDK `recover()` contract:
it waits for `started` and rejects when the sandbox returns to `error`.
## Risks
- Time budget: the host gives `environmentStopLease` at most 60 s. A
recovered stop uses up to 20 s for Recover and up to 30 s for Stop, plus
short reads. A slower recovery returns `recovery_incomplete`. The SDK
call can continue at Daytona after that, so the next sweep finds the
sandbox started and stops it normally. If the host deadline fires while
the worker still stops the sandbox, the next sweep sees `stopped` and
records the receipt.
- Classification: after a failed Recover, the plugin cannot tell a
rejected request from a sandbox that started and failed again. Both give
`recovery_failed` (terminal) when Daytona still shows `error`. A hung
Recover call with a sandbox still in `error` is also terminal. Terminal
failures still retry every 30 minutes, and each retry makes at most one
Recover call. An explicit Retry also makes one attempt. There is no
tight loop. A sandbox that fails Recover every time gets about two
Recover calls an hour until an operator acts.
- Recover restarts the sandbox container before Stop. The plugin runs no
command in it, and the retention policy is confirmed before Recover.
- The continuation message for a terminally failed lease still says the
message "will start automatically". This PR does not change admission. A
follow-up can use the typed reason there.
- Compatibility: a host without this change ignores the new error data
and keeps the old retry behavior. A plugin without this change sends no
typed data.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, code execution for tests and typechecks).
The runtime does not expose the context window size to the agent.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full repo
runs are left to CI, as noted above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandClaude Opus 5.5
|
f1516e8009 |
fix(native): stop a session.open model rejection instead of a doomed bootstrap retry (#19190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Runner runs start a provider session with the `session.open` command before any prompt > - A definite `session.open` rejection is treated as a pre-session failure and gets a bounded same-run bootstrap retry > - The runner also reports a closed classification. Two classifications say that the provider cannot run the selected model: `requested_model_unsupported` and `grok_model_mismatch` > - Paperclip ignored that classification. It scheduled a retry that must fail the same way. On a remote sandbox with a retained runner PID, the due retry entered restart recovery. Restart recovery has no checkpoint for a bootstrap retry, so it failed closed as `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence` > - The task then stayed `blocked`. The recovery actions settled as no-replay holds, so no active action told anyone to change the model, and the parent task stalled too > - This pull request classifies those two definite rejections as the existing `native_provider_model_rejected` failure > - The benefit is a terminal failure on the first attempt, an accurate error, and an active board recovery action that names the next step: choose a supported model ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a Grok agent run failed at `session.open`. The runner log said `ACPX sidecar command session.open was rejected (retryable=false, classification=grok_model_mismatch)`. Paperclip recorded `recoveryMode: "bootstrap_retry"` and scheduled a same-run retry 30 seconds later. The run had a remote sandbox and a recorded sandbox PID, so the due retry went to restart recovery. Restart recovery found no checkpoint and no provider events and failed closed with `ambiguous_remote_recovery_evidence`. The run ended as `native_restart_recovery_blocked`. The task stayed `blocked` with no active recovery action for more than one hour. Its parent task stayed blocked on it. **What did you expect to happen?** The selected model cannot start, so a retry cannot succeed. The run should fail once with the model-rejection error. The task should show the existing "choose a model" action, owned by the board, until someone changes the model and retries. **Steps to reproduce** 1. Start a native run whose `session.open` command fails with `classification=grok_model_mismatch` (or `requested_model_unsupported`). 2. Read the `native_run_finalizations` row. Before this change it is `retryable_failure` with `recoveryMode: "bootstrap_retry"` and a `nextAttemptAt`. 3. On a remote sandbox run with a recorded PID, the due retry enters restart recovery and fails as `native_restart_recovery_blocked`. **Paperclip version or commit** `master` at `f5f5184f23`. Related: #18910 routes due remote retries with a retained PID to restart recovery. #18908 continues verified failed native preparation. Open PRs #19177, #19180, #19182 and #19184 change no-replay hold settlement and workspace hold release. They do not change this classification. This PR does not duplicate them. ## What Changed - `nativeSessionFailureSourceCode` (`server/src/services/native-runtime/native-session-executor.ts`) returns `native_provider_model_rejected` for a definite `session.open` failure or rejection whose runner classification is `requested_model_unsupported` or `grok_model_mismatch`. Other definite `session.open` rejections still return `provider_session_open_rejected` and keep the bounded startup retry. An indeterminate `session.open` outcome stays `native_session_interrupted`. - `native_provider_model_rejected` is already a permanent failure. The run stops on the first attempt with no `nextAttemptAt`, so it never enters the restart-recovery retry path. - When no provider session exists, the coordinator `nextAction` and the board recovery action `nextAction` say: choose a supported model for this agent, then retry the task. - The existing product paths for `native_provider_model_rejected` apply: the task presentation shows `choose_model`, and the recovery service treats it as `configuration_incomplete`. - `doc/execution-semantics.md` describes the rule under "Native provider model capacity". ## Verification Red before the fix (new test, unchanged production code): ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 2 failed | 21 passed # failureCode: native_session_interrupted, phase: retryable_failure, recoveryMode: bootstrap_retry, # nextAction "Retry provider bootstrap on this same run ..." - the same record seen on staging ``` Green after the fix: ```sh cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts # Tests 23 passed cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts \ src/services/native-runtime/native-startup-retry.integration.test.ts src/services/task-interruption-presentation.test.ts # Test Files 3 passed, Tests 1186 passed pnpm --filter @paperclipai/server typecheck # exit 0 ``` The new integration cases use embedded PostgreSQL and the real executor failure path. For both classifications they check: - coordinator `terminal_failure`, `failureCode: native_provider_model_rejected`, `nextAttemptAt: null`, attempt 1 - run `failed` with `errorCode: native_provider_model_rejected` - task `blocked`, execution lock released - an `active` board recovery action with cause `native_provider_model_rejected`, return owner set to the agent, and the choose-model next action - `settleUnrecoverableExecutions` leaves the action `active`, so it does not become a silent no-replay hold - `claimNativeSessionResumptions` claims nothing, so there is no second executor The unit test also checks that `session_ensure_failed` keeps its bounded retry and that an indeterminate model rejection stays unclassified. A live Grok run with a mismatched model was not run. The failure record in the test matches the staging record field by field. ## Risks - Low risk. The change adds two closed runner classifications to an existing permanent failure code. Other `session.open` rejections keep their current retry. - A model mismatch that a retry could fix would now need an explicit retry. The runner already marks these rejections `retryable=false`. - This PR does not change restart recovery. A remote bootstrap retry with a retained sandbox PID for other `session.open` causes can still fail closed as `ambiguous_remote_recovery_evidence`. That needs a separate change with remote stop evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in Claude Code with tool use (shell, file edits, browser reads of the staging API). It did the investigation, code change, tests and PR text. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
46a50a3bdd |
fix(recovery): release a recovered workspace hold for the new assignee after reassignment (#19184)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a board user reassigns a task in the middle of a run, Paperclip
stops the old run and saves its sandbox files (copy-back).
> - If copy-back fails at the stop, Paperclip records a
file-preservation hold for the stopped run. The hold blocks all
execution on the task until the files are recovered.
> - The new assignee's wakes were already queued. The final dispatch
gate saw the hold and cancelled them.
> - Automatic recovery then saved the files. But the hold stayed
effective for the new assignee: explicit continuation accepts only the
stopped run's own agent, so no message, Retry or stranded-work sweep
could start the new assignee.
> - The task stayed `in_progress` with no run, no wake and no active
recovery.
> - This pull request releases a recovered file-preservation hold for a
different current owner.
> - The benefit is that a reassigned task continues with its new owner
after its files are safe, with no manual action.
## Linked Issues or Issue Description
**What happened?**
A board user reassigned a running native task from agent A to agent B
and posted a comment. A's run ended `cancelled` / `issue_reassigned`.
Copy-back was still in progress, so B's first run was cancelled
`workspace_busy`. Paperclip then recorded a resolved no-replay hold
(`native_continuation_requires_reconciliation`, fingerprint
`native-workspace:<A run>`). The final dispatch gate cancelled B's two
queued runs with `execution_reconciliation_required` ("Recovering saved
work…"). Automatic recovery then saved the files
(`workspaceRecovery.status = recovered`, `workspaceRepairRequired:
false`). The blocker still said "Saved work recovered. Send a message to
continue." with `canContinue: false`. The task stayed `in_progress` with
no run for hours. Two hosted reproductions had the same shape. A sibling
reassignment without a copy-back failure recovered normally.
**Expected behavior**
After the saved-file receipt verifies, the new assignee continues. The
hold must still block every executor while the files are not recovered.
The stopped run's own agent keeps its no-replay hold.
**Steps to reproduce**
1. Start a native run on a task with a retained sandbox.
2. Reassign the task to another agent while the run executes, so that
copy-back records a `workspaceRestoreRecovery` hold.
3. Let automatic workspace recovery finish.
4. Before this change, `getExecutionBlocker` still returns the hold for
the new assignee, and the stranded-work sweep queues nothing.
**Paperclip version or commit**
`b9fbbf8d30` (also seen on `818d2bb6`).
**Deployment mode**
Hosted, PostgreSQL. The test uses embedded PostgreSQL.
Related open pull requests (checked; none covers this path): #19177
(board reconciliation of sibling holds), #19169 (disposition repairs
behind a cancelled blocker), #19143 (park a workspace-busy wait), #19149
(interrupt stop identity).
## What Changed
- New `restoreHoldReleasedForCurrentOwnerPredicate`
(`server/src/services/execution-blocker.ts`) is a SQL predicate. It
matches a file-preservation hold (`workspaceRestoreRecovery`, not
`restore_unsafe_archive`) when all of these are true:
- the stopped run is terminal and finished;
- the task has an agent assignee, and that agent is not the stopped
run's agent;
- the stopped run's agent is not the current review participant;
- `completedWorkspaceRecoveryCondition` is true for the hold: completed
system recovery, `workspaceRecovery.status = recovered`, succeeded
preservation operations bound to this hold, and the `workspace_finalize`
receipt;
- every saved operation is a copyback into the task workspace
(`workspace_recovery`). A snapshot-only save keeps the hold, because the
new owner's ordinary turn does not attach snapshots.
- The predicate is wrapped in `coalesce(..., false)`. Its negation
therefore cannot hide an unrelated hold through NULL JSON reads.
- The proof is the durable completed-recovery receipt. It does not read
live workspace authority (epoch, writer, snapshot), so reuse of the
workspace by the new owner cannot re-arm the hold.
- `getExecutionBlocker` excludes matching holds from both the
file-preservation query and the general blocker query. There is no scan
limit and no extra query per hold.
- `admitExplicitNativeContinuation` excludes the same holds from its
source checks. A released hold therefore cannot reject a new owner's
message with "owner changed".
- `completedWorkspaceRecoveryCondition` also accepts a SQL expression
for the run id. Its behavior is unchanged.
- The hold row, its `automaticRecovery.replay = "blocked"` evidence and
its status are not changed. If the task returns to the stopped run's
agent, the hold applies again.
- `doc/execution-semantics.md` describes the rule next to the existing
"Reassignment does not discard the stopped run's files" sentence.
## Verification
New test in `server/src/services/workspace-recovery.test.ts`: `hands a
reassigned task to its new assignee once the previous owner's files are
recovered`. It uses the real retained-workspace recovery, embedded
PostgreSQL and the real stranded-work sweep. It builds the incident
shape: a cancelled `issue_reassigned` source run, a resolved no-replay
hold, a new assignee, and that assignee's gate-cancelled run.
- Before recovery, the blocker holds the new owner
(`workspaceRepairRequired: true`).
- After recovery, the blocker is `null`. The sweep creates exactly one
pending turn for the new owner. That turn is not a retry of the stopped
run. A second sweep adds no turn.
- The hold row keeps `status: resolved`, `outcome: blocked` and `replay:
blocked`.
- Live repair evidence on the stopped run is then changed, so
`hasVerifiedWorkspaceRepair` returns false (as after a remote reuse).
The blocker stays `null`.
- After a reassignment back to the stopped run's agent, the hold applies
again.
- An unrelated reconciliation hold for the same stopped run, without
file evidence, still blocks the new owner.
- New test `keeps a snapshot-only recovery held for a new assignee`: the
snapshot-only save does not release the hold. This test fails against
the first revision of this pull request.
Red before the fix (test kept, `execution-blocker.ts` restored from
`master`):
```
cd server && npx vitest run src/services/workspace-recovery.test.ts -t "hands a reassigned task"
AssertionError: expected { …(13) } to be null (blocker: "Saved work recovered. Send a message to continue.", canContinue: false)
```
With only the blocker assertion removed, the sweep also failed:
`expected [] to have a length of 1 but got +0`.
Green after the fix: 1 passed.
Related suites, all pass (991 tests): `workspace-recovery`,
`execution-blocker-read`, `legacy-explicit-continuation-recovery`,
`workspace-restore-reconciliation`, `native-failed-workspace-recovery`,
`workspace-restore-repair-route`, `execution-projection-recovery`,
`heartbeat-undispatched-execution-wait`,
`issue-update-comment-wakeup-routes`,
`heartbeat-observed-workspace-preservation`,
`native-workspace-reconstruction`, `explicit-native-continuation`.
Not run: a live run on a deployed instance. The hosted incident tasks
are not recovered by this pull request until it is deployed.
## Risks
- A new owner can start after the files are recovered, while the stopped
run's action outcomes stay unknown. This matches the existing rule for
holds without a workspace: Paperclip does not create those holds for a
reassigned task. The new owner gets a fresh turn, not a replay.
- The hold still blocks every executor until the exact receipt verifies,
so there is no second executor on the workspace during copy-back.
- Other execution holds (non-workspace holds recorded before a
reassignment) keep their current effect. This change does not alter
them.
- A reassignment after a snapshot-only save still needs a board action,
as before. That path is not changed here.
- The release is one correlated `exists` in the existing blocker
queries. Its subqueries use primary keys and the run and issue indexes.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
f7a4c5678f |
feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.
## Linked Issues or Issue Description
Refs #19108, #19038, #18943.
**What happened?**
A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.
**Expected behavior**
Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.
**Steps to reproduce**
1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.
## What Changed
- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.
## Verification
- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at
|
||
|
DottaandPaperclip
|
dbbde744d2 |
fix(workspaces): preserve finalized recovery holds through offline import (#19180)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Offline workspace import must save files before local volume
removal.
> - A task recovery hold can remain after native file finalization
succeeds.
> - The old importer rejects that hold even when its file source is
exact and no controller owns the run.
> - Clearing the hold would discard the task's recovery obligation.
> - This pull request saves files and carries each exact recovery source
forward without resolving the hold.
> - Accepted-result finalizers keep their existing retry path.
## Linked Issues or Issue Description
Refs #19144, #18943, #19038. This PR builds on merged #19144.
**What happened?**
Offline import reports `workspace_recovery_unresolved` for terminal
native runs with finalized workspace references and active task recovery
holds. Some holds also name an exact historical file allocation.
**Expected behavior**
Save the workspace durably. Keep task warnings, recovery rows, results,
coordinator phases and task pointers unchanged. Retain exact future file
recovery and accepted-result finalization retries.
**Steps to reproduce**
1. Retain a local task workspace and an active recovery hold for a
finalized native run.
2. Stop all application writers through offline maintenance.
3. Attempt workspace import.
4. Observe that the old importer rejects the hold.
## What Changed
- Add an owner module that checks the exact terminal native run,
finalized reference, released ownership and task relationship.
- Require the normal finalizer's successful company/run/task receipt for
an accepted-result retry. Support historical receipts without a
workspace foreign key.
- Validate recorded file allocations with the existing recovery-source
owner.
- Journal and recheck hold evidence. Lock recovery, coordinator, run and
allocation rows before publication.
- Give finalized file sources a separate migration marker. Keep legacy
execution-replay suppression out of accepted-result retry paths.
- Use the owning parser for older finalized references that omit
optional fields.
- Redirect future file recovery to immutable snapshot preservation. Keep
provider-stop and source checks intact.
- Verify native copyback or snapshot receipts for each exact allocation.
Require proof for every source, including completed mixed repairs.
- Retain existing successful copyback receipts. Keep a historical
folder's obligation on its real source workspace.
- Retain the finalized migration marker through the execution blocker
projection. Discover finalized child-task sources from their exact
ancestor workspace reference.
- Document the receipt and lock contracts.
## Verification
- Fresh local run at head
|
||
|
Nicky Leach
|
8c062fa481 | fix(native): deliver plugin VM Runner usage to Core (#19162) | ||
|
Nicky Leach
|
53937be283 | fix(agents): select environment plugins only for agents on that environment (#19173) | ||
|
DottaandPaperclip
|
f5f5184f23 |
fix(workspaces): preserve retained branches and private commit sources (#19147)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must preserve existing task files before
volume removal.
> - An old private checkout can be absent while its recorded branch
still exists.
> - A deleted branch can also retain its exact commit in the original
private object source.
> - The importer currently rejects these surviving sources.
> - This pull request saves the verified committed tree and records its
exact source proof.
> - It keeps task state, branch ownership and recovery obligations
unchanged.
## Linked Issues or Issue Description
Refs #19138, #19144, #18826.
**What happened?**
Existing Git workspace records can have missing checkout folders and
retained branch objects. The importer returns
`local_source_unsafe_or_missing`. A successful deletion receipt can also
have a valid private object source that the importer overlooks in favor
of a missing or changed project cache.
**Expected behavior**
Save the surviving committed tree from the exact recorded branch or
deletion receipt. Reject competing owners, live worktree registrations,
changed branches and invalid source identities. Do not use the current
default branch as a substitute.
**Steps to reproduce**
1. Retain an isolated workspace branch in its managed project cache.
2. Remove its clean checkout but keep the branch and original workspace
record.
3. Attempt workspace migration.
4. Alternatively, retain a deleted commit in its recorded private source
while its shared source is unavailable.
## What Changed
- Pin a retained branch in its verified company and project Git cache.
- Require the recorded private path, no competing workspace owner and no
surviving Git worktree registration or index authority.
- For a user-owned branch, require its exact successful unforced removal
receipt after archival. Preserve the branch ownership metadata.
- Prefer a verified recorded private object source for successful
deletion receipts when that source still exists.
- Recheck branch and receipt proof before publication. Preserve
compatibility with existing migration journals.
- Allow missing active records only through the existing offline writer
fence. Keep their status unchanged.
- Add byte-restoration and rejected-authority tests. Document the source
limits.
- Revalidate and honor interrupted journals' original Git source path.
Reject detached registrations and retained orphan index data. Reuse the
cleanup producer command formatter for paths containing spaces.
## Verification
- 172 focused migration, concurrency and isolated-source tests pass with
real PostgreSQL and Git repositories.
- The previous importer fails the retained-branch and offline-active
restoration cases. The previous source selection also fails the
private-cache and clean user-owned receipt cases.
- Tests remove the old home and restore the pinned commit and task file
bytes. New shared default-branch work is excluded.
- Tests reject competing owners, registered worktrees, missing branch
ownership, deleted branches, changed branch tips, forced removal and
revoked receipts.
- Read-only re-inventory of 79 actual failed source records makes 78
eligible with committed owner overlays. One older local test repository
remains blocked. This is source-overlay qualification, not full
released-image acceptance.
- A separate bounded archive replay completed six imports before its
25-minute deadline. That timing failure remains open; it does not
establish complete migration.
- Local server typecheck has no changed-file diagnostics. Stale linked
declarations prevent a clean local whole-server result. Full clean CI
remains required.
- Serving enrollment and empty-home recovery remain separate acceptance
gates.
- Latest rebased head
|
||
|
DottaandClaude Opus 5.5
|
512fbea0bb |
fix(native): deliver messages saved while a native run finalizes (#19183)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can reply on a task while the agent's run is still finishing > - A remote native run first commits its result and its status decision (for example `done`). Then its sandbox environment settles (harness backup, warm retention). > - A message that arrives after the final turn took its input is saved as a deferred receipt behind that run > - The saved-message recovery path holds the receipt while the source run still owns its environment lease or its workspace copy-back owner > - Nothing notified that path when the lease or the owner settled, so the messages waited for an unrelated company event > - This pull request notifies the existing deferred-receipt work when a run's environment lease settles or its workspace finalization owner is released > - The benefit is that messages sent while a run finalizes are delivered once, soon after the run settles, and a task closed in that window is reopened for them ## Linked Issues or Issue Description **What happened?** A user posted three plain comments on a task while the agent's remote native run was finalizing. The final turn had already called its finish tool. The comments were saved in one deferred receipt. The run then committed `done` and its status became `succeeded`. No follow-up run started. The task stayed `done` with three unanswered requests. About 20 minutes later, an unrelated event in the same company woke the deferred-receipt worker, and the messages were delivered. **Expected behavior** A message that arrives after the last turn took its input, but before the run releases, gets one follow-up turn as soon as the run settles. If finalization marked the task `done` in that window, the task is reopened for the message. Messages that the finished turn already consumed are not replayed. **Steps to reproduce** 1. Start a native run with a remote (sandbox) environment. 2. After the agent calls its finish tool, post a user comment on the task. 3. Let finalization commit `done` and project the run as `succeeded`. 4. Observe that the comment's deferred receipt stays `deferred_issue_execution` after the environment lease settles. **Root cause** A remote native run is finished by native finalization, which projects the terminal status. The in-process run completion (`releaseIssueExecutionAndPromote`) does not run for it. Its saved messages depend on the stranded-queue sweep in `resumeDeferredExecutionReceipts`, which calls the release with `resumeStrandedQueue`. That release correctly waits while `hasPendingRunCleanup` is true (an unsettled environment lease or a workspace finalization owner). Finalization notifies the deferred-receipt worker when it projects the run, but the cleanup is still pending at that time. The worker then drops the entry. Lease settlement and owner release did not notify the worker again. Refs #19122 (saves a redirect sent while a stopped run settles; this change reuses the same deferred-receipt delivery path for normally completing runs). ## What Changed - `environmentService.releaseLease` notifies deferred-execution work for the company when a run's lease leaves `active` or `pending_cleanup` (released, retained, or failed). - `environmentService.releaseLeasesForRun` notifies deferred-execution work in the same way. - `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution work when it releases its owner. - Delivery uses the existing path: `resumeDeferredExecutionReceipts` → stranded-queue release → deferred wake promotion (with the existing reopen rule for user messages). No new delivery mechanism. - New regression in `deferred-execution-receipt-work.test.ts` with the real deferred-receipt worker, the real release adapter, and the real lease service. - `doc/execution-semantics.md`: one additive paragraph for messages sent while a run finalizes. ## Verification - Staging evidence (read-only): the task's saved queue held the three comments as one `deferred_issue_execution` receipt after the run succeeded. The run had no `run succeeded` lifecycle event (it was finished by native finalization, not run completion). The receipt was promoted about 20 minutes later with `reopenedFrom: done` by the same recovery path. - Red before the fix: the new test failed with `expected null to be truthy` (the receipt stayed deferred with no run after the lease settled). - Green after the fix: `cd server && npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing native run"`. - Safety cases in the new test: - The receipt stays deferred while the environment lease is active (cleanup holds still apply). - After settlement, exactly one run carries both messages in order, and the done task is reopened once. - A second settlement signal does not deliver the messages again. - Related suites: `deferred-execution-receipt-work`, `environment-service`, `environment-cleanup-work`, `native-environment-settlement-work`, `idle-sleep-safety`. - Not done: a live staging reproduction at the fixed revision. Deployment is out of scope. ## Risks - The deferred-receipt worker reloads the company's deferred receipts after each run lease settlement. Each reload is bounded and runs only when deferred receipts exist. The notification is transactional, so a rolled-back release does not wake the worker. - Low risk: no schema change, no API change, and no change to admission or reopen rules. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser for read-only staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
06826915d7 |
fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Child tasks can reuse their parent's execution workspace. Only one run at a time may write in a shared sandbox workspace. > - A run that finds the workspace held by another live run waits with a `workspace_busy` scheduled retry. > - That wait cancelled the run and created a new retry run every 60–120 seconds, until the holder ended. Each check made one cancelled run record. > - Three sibling tasks that waited for a parent made 15 cancelled runs in 13 minutes. They also got the workspace in random order, not in arrival order. > - This pull request keeps the waiting run as one parked retry while its holder is live, and wakes it when the holder ends. > - The benefit is a clean run history, a prompt handoff, and fair arrival-order access to a shared workspace. ## Linked Issues or Issue Description Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a stopped writer with a pending save into the same wait). Refs #12587 (an earlier proposal about fairness for this retry). **What happened?** A parent task delegated three subtasks that inherit its execution workspace. Each subtask run was created, then cancelled with `workspace_busy` ("Shared execution workspace is busy: run … is still running"), then created again about every 1.5–2 minutes. One subtask got 8 cancelled runs, one got 5, and one got 2. Every cancellation named a holder that was really running: first the parent run, then the sibling that got the workspace. So serialization was correct. The cost was the churn of run records, and the subtask that waited first ran last. **Expected behavior** A run that needs a busy shared workspace waits as one durable record. It continues when the holder finishes. Siblings that wait for one workspace get it one at a time, in arrival order. Two writers never share the workspace, and the user can still stop a waiting run. **Steps to reproduce** 1. Bind two or more issues to one shared execution workspace in a sandbox environment. 2. Keep a run live on one issue (the holder). 3. Wake the other issues. Before this change, each due check promotes the retry, cancels it with `workspace_busy`, and creates a new retry run. After this change, the retry stays parked until the holder ends. **Paperclip version or commit** Observed on staging at `53735db7`. The code is unchanged on `master`. ## What Changed - `finalizeWorkspaceBusyDeferral` records `contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince }`. The key is `execution:<id>` or `project:<id>`. `waitingSince` carries over when the run defers again for the same workspace. - New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`: - `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion transaction, after the existing gate allows the retry. If the recorded holder is still running and is not stale, or an earlier waiter for the same workspace is still scheduled, queued, or running, the retry stays `scheduled_retry`. Only its deadline moves. It does not create a run, an event, or a wake. The fallback deadline is 5 minutes, and never later than the holder's one-hour stale point. - `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a running run (`writeRunStatus`). It makes due the retries that wait for that run, and the retries that wait for the same workspace as that run. It registers the scheduled-retry dispatch notification before the write. - `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch domain, so the deferral gate and the parked check use one value. `heartbeat/retries.ts` exports it again. - A retry without a recorded wait (for example, a retry from before this change) keeps the old behavior. - `doc/execution-semantics.md` describes the parked wait. ## Verification - Red before the fix, green after (`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared execution workspace in a sandbox"): - "waits on a live holder in one parked run, then runs once when the holder ends": before, the retry was promoted while the holder was live. After, it stays parked through due checks at +1 s, +10 min, and +30 min. Ending the holder makes it due at once. It runs exactly once, and there is only one `workspace_busy` cancellation. - "hands the workspace to three waiting siblings in arrival order, one writer at a time": before, nothing woke the waiters when the holder ended. After, exactly one waiter is admitted per handoff, in arrival order. Each runs once, and there are 3 cancellations in total (one first deferral for each sibling). - "still lets the user stop a parked waiter": before, the waiter was promoted while the holder was live. After, Stop cancels the parked retry, and nothing brings it back. - Safety case (green before and after): "lets a stale holder reach the existing quiet-holder failure instead of parking forever". - `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35 passed. - `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`, `src/modules/run-dispatch`, `heartbeat/retries.test.ts`, `execution-projection.test.ts`: 266 of 267 passed. The one failure, "records pre-provider quota rejection…", is a timing flake on a loaded machine and passes when it runs alone. It does not use this path. - `tsc --noEmit` reports no errors in the changed files. - Staging evidence was read only. This change is not deployed. ## Risks - One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when a running run ends. The update and the notification happen only when waiters exist. - If an earlier waiter stays queued for a long time (for example, its agent is paused), later waiters for the same workspace wait behind it. Each check is a deadline update, not a run. The fallback check runs every 5 minutes. - A remote writer that has ended but has not finished its save (#19075) is not parked, because its run is no longer `running`. Such a wait can still make a small number of deferrals until the save settles. - Low migration risk: no schema change. Old retries without the new context key keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, browser read of staging run records) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c8013d06ad |
fix(workspaces): preserve task-only warnings during offline migration (#19144)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers must save existing workspaces before volume
removal.
> - Offline migration checks file ownership and recovery authority.
> - The same gate also rejects task assignment and configuration
warnings.
> - Those warnings do not require the old filesystem.
> - This pull request preserves known task-only warnings while it saves
files.
> - File recovery and execution obligations keep their existing
blockers.
## Linked Issues or Issue Description
Refs #18943, #19038, #19138.
**What happened?**
Offline workspace migration refuses tasks with assignment,
configuration, disposition or deliberate-wait warnings. It returns
`workspace_recovery_unresolved` even when the warning has no file or
execution recovery authority.
**Expected behavior**
Save local workspace bytes while retaining the task warning unchanged.
Continue to block unknown recovery kinds, file sources, export failures
and execution recovery work.
**Steps to reproduce**
1. Retain a local workspace on a task with an active assignment or
configuration warning.
2. Stop all app writers through the offline maintenance entry.
3. Attempt workspace migration.
## What Changed
- Admit four known task-only warning kinds during offline file
migration.
- Keep file recovery markers, system recovery, native failure markers,
export-failure causes and unknown kinds blocked unless their exact
legacy source is already preserved.
- Keep the existing live-migration recovery exclusion.
- Recheck warning evidence before publication. Do not resolve warnings
or change their owners.
- Add restoration, rejection and publication-race tests. Update the
migration plan.
## Verification
- 112 focused migration and concurrency tests pass with real PostgreSQL
and filesystem archives.
- The four task-only restoration cases fail against the previous
importer with `workspace_recovery_unresolved`.
- Tests delete the old home, restore its saved bytes and compare each
warning row unchanged.
- File and execution markers, unknown kinds and export failures remain
blocked.
- A task warning that gains file recovery during upload prevents
publication.
- Local server typecheck has no changed-file diagnostics. Stale linked
dependency declarations prevent a clean local whole-server result. Clean
full CI remains required before merge.
- The PR was rebased onto the merged Git preservation fix after
resolving a plan-document conflict. The rebased head passed full PR CI
(41943153422). Current head
|
||
|
DottaandPaperclip
|
7c66b6b1f6 |
test(onboarding): cover plan, hiring, and execution approval history (#19069)
## Thinking Path > - Paperclip manages agent work and human decisions. > - Onboarding can approve a plan, a hire, and an execution step in separate turns. > - Tests covered single approvals more closely than that full history. > - The repaired gate must retain valid plan approval through later decisions. > - Every consumer must recognize the saved confirmation kind and selected scope. > - This change tests that contract and registers a complete interview-to-execution journey. ## Linked Issues or Issue Description Refs #19043, #17914, #18283. The earlier tests admitted one accepted proposal at a time. Decomposition fixtures used standard confirmations. The direct-task E2E path forbids hiring. Those cases did not cover an accepted checkbox plan followed by hiring and a separate untargeted execution approval. ## What Changed - Add eight PostgreSQL contract cases for standard and checkbox plan approvals. Native tools create the cards. An authenticated human accepts them through HTTP. A later native run reads the approved revision and actual selected scope. - Exercise native task creation, generic REST creation, child REST creation, and accepted-plan decomposition. Check the assigned child and one creation audit event. Check durable replay on the three entry points that support it. A pending execution confirmation still prevents creation. - Register the explicit-only `onboarding-approval-history` suite for local native Codex and ACPX Claude. The browser chooses the interview, selects only drafting in the saved plan, approves one writer hire, and approves a separate Start drafting confirmation. The model must hire the writer and use that writer's own child execution to save the welcome document. - Grade each human decision, the unchanged plan revision and selection, approval timing, one assigned child and worker execution, the exact writer-authored saved revision, no pending requests, and persistence after reload. Calibrate the oracle against missing, forged, premature, duplicate, and changed outcomes. - Preserve synthetic option choices, document/run bindings, and saved revision authorship in the existing public first-task evidence projection. Keep private provider/account data excluded. Document the workflow, evidence, billing, and cleanup bounds. - Align the approval-history fixture with its completion oracle: the onboarding parent must be Done. Test both native profiles against graded completion evidence and reject a completed child whose parent remains In review. Count parent heartbeats separately from the writer: accept one to seven parent records while retaining the twelve-company-run cap and the independent three-approval/one-writer oracle. - Retain the existing 52-cell first-task suite. The new suite has one attempt, seven expected work turns, a twelve-run maximum, a fifteen-minute deadline, and the existing $5 company task budget. It is excluded from `--all`. - Update two stale cleanup assertions to include the existing thirty-second deletion timeout. ## Verification - Red control: temporarily restore the pre-fix admission module and run the new checkbox/native history case. It fails with the expected `First-task proposal requires human acceptance` error after plan, hiring, and execution acceptance. Restore the repaired module after the probe. - The final post-rebase history matrix and existing onboarding safety suite passed together: 74 tests, no skips. - Final focused Runner E2E support passed: 413 Vitest tests and all 370 Node support checks. This covers catalog discovery, journey controls, wrong-outcome calibration, public evidence, and existing first-task behavior. - Review controls: four authorship cases fail before the revision check. A delayed browser-response control fails before the approval-time fix. Both pass after repair. The final oracle requires the exact current document revision, writer ID, native child run ID, matching body, and readback after reload. - Credential-free discovery lists both new native cells. No provider call was made. `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` are absent, so the real provider journey remains unqualified. - The earlier broad credential-free E2E support run reported 6,728 passing cases, three skipped cases, and nine failures in three unrelated files. All nine reproduce with the changed E2E source restored to the unchanged baseline: three expired overhead-fixture cases, one stale Pi profile-version assertion, and five workflow security assertions. This change does not alter those files, paid workflow policy, or provider pins. - Previous full workspace typecheck/build and [Linux CI](https://github.com/paperclipai/paperclip/actions/runs/41921754638) passed on `4f431f5aae99d072cc75027e90d7b847b4c342ca`, including the new eight-case PostgreSQL file. Ponytail subsequently found a fixture/oracle terminal-status mismatch. Both native-cell controls fail on that fixture (`done` observed versus `in_review` expected), while the parent-in-review negative control passes. A second comparison uses the actual `matchesRunCount` predicate: graded evidence with six parent runs and one writer run fails the old exact-seven-parent requirement. Both fixes pass these controls, all 413 focused support tests, all 370 Node checks, and the owning E2E project typecheck. [Fresh CI](https://github.com/paperclipai/paperclip/actions/runs/41936699686) passed on `91c980b110252da451b92eda597c168517de1276`: 48 successful jobs and one expected skip. Greptile reviewed this head at 5/5 with no unresolved findings. Ponytail's terminal-status finding is fixed and its thread is resolved; its current-head re-review remains queued. Live provider cells remain unqualified. ## Risks The production API and approval gate are unchanged. The new paid journey can expose model or product failures at an intermediate approval boundary. It retains those failures and uses the existing diagnostics and cleanup. Registration and deterministic tests do not establish a live provider pass. The unrelated baseline support failures remain explicit. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused tests; unrelated baseline failures are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
5fdfd3f17a |
fix(recovery): bound repeated saved-record response bodies (#19168)
## Thinking Path > - Paperclip manages agent tasks and their saved execution history. > - A continuation carries completed API receipts to prevent duplicate work. > - Repeated saves can copy large record bodies into each new provider prompt. > - The resulting command can exceed the durable PRP frame limit before useful work starts. > - This change projects large successful saved-record responses as compact references. > - The original receipts and authenticated user direction remain available. ## Linked Issues or Issue Description **What happened?** A task with many saved document and comment responses can fail with `command_payload_too_large`. Same-run recovery can then show a runner identity failure that hides this first cause. Completed response bodies repeat content already stored in task records. **Expected behavior** Continue from saved records with completed-action evidence that fits the runner command. Keep user messages, answers and approvals intact. Keep the full original receipts. **Steps to reproduce** 1. Save 39 successful document, comment and task-write responses in a run's API receipt ledger. 2. Give each response a large record body. 3. Build and render a continuation, then queue it through the real durable PRP journal. 4. Before this fix, the regression throws `command_payload_too_large` with a 4.2 MB prompt. Related work: #16980 pages oversized assigned tool results. This change addresses earlier response bodies repeated in a later continuation. It does not change tool-result paging or protocol limits. ## What Changed - Project response data over 4 KiB for three known successful record-write operations. Keep record, revision and status metadata, HTTP outcome, receipt identity and a data digest. - Keep small, failed, unknown, mismatched and foreign-company results in their existing form. - Explain the reference format in shared prompt guidance and the execution contract. - Add database and real-journal regression coverage. Verify the original receipt ledger remains unchanged. The continuation service owns the projection. The shared adapter helper only explains the reference to native and legacy prompt consumers. There are no new dependencies, provider calls, timers or background scans. ## Verification - The unfixed database regression fails at the real PRP command limit with `command_payload_too_large`. - Continuation builder, native execution input and native continuation suites: 118 tests pass. - Shared prompt consumers: 142 tests pass. - Module boundaries and diff whitespace checks pass. - Greptile: 5/5 on `8109e7361c5b5e38f9aad35af9bf4ad7bdc5f338`, with no unresolved review threads. All CI checks pass. - Full workspace `pnpm -r typecheck` and `pnpm build`: passed. - Hosted verification: pending deployment. The source fix does not itself resume the affected hosted task. Local setup failures remain accounted for: the fresh checkout lacked an ignored root SDK link and embedded PostgreSQL library symlinks after an offline install that skipped lifecycle scripts. The maintained workspace-link helper, SDK build and platform hydration script repaired those prerequisites. A later test expectation assumed JSON insertion order across PostgreSQL JSONB; it now checks the digest against the unchanged stored receipt. These failures did not change runtime limits or recovery guards. ## Risks Large responses from these three successful operations now use explicit saved-record references in continuation context. Agents must read the record or source receipt when full content is needed. This is not a universal prompt-size limit: user history, unknown operations and other inputs retain their existing representation. No migration, permission, credential, provider-stop or replay policy changes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, through Codex in T3 Code. Used for investigation, implementation and local verification. No subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task id - [x] I have run the targeted tests locally and they pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address review comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
737db73275 | fix(native): measure resumed run usage from the prior run's total (#19161) | ||
|
Devin FoleyandPaperclip
|
94fccc3180 |
fix(workspaces): preserve snapshots beside runtime sockets (#19159)
## Thinking Path > - A workspace can contain saved source files alongside transient runtime IPC endpoints. > - The portable archive supports files, directories and confined symlinks; treating a Unix socket as a fatal entry prevents every file beside it from being saved. > - Keep the existing archive owner and validation. Record socket paths as exclusions, leaving the source endpoint and adjacent work intact. ## Linked Issues or Issue Description **What happened?** Retained workspace recovery repeatedly fails with `remote_workspace_archive_unsupported_entry` when the source tree contains a Unix socket, preventing adjacent source files from being saved. A real socket reproduces the error in local capture and the serialized sandbox helper. **Expected behavior** Save regular workspace files, record runtime socket paths as exclusions, and leave the source endpoints and new file edits intact. **Steps to reproduce** 1. Create a temporary workspace with an uncommitted text file. 2. Open a Unix domain socket in that directory with a Node server. 3. Capture the workspace through the local archive function or generated sandbox helper. Before this fix both captures fail; afterward the saved file restores and the socket exclusion is recorded. **Paperclip version or commit** Reproduced on `6fa54d55779295d90c455389cebdd3da96607d20` before this change. The socket fixture establishes this failure class; it does not establish the entry type behind every historical unsupported-entry occurrence. ## What Changed - Exclude Unix domain socket endpoints from portable workspace captures and include their paths in the existing exclusion manifest. - Keep source sockets in place. Preserve normal files and the existing rejection of named pipes, device entries and unsafe links. - Exercise local and serialized capture, an unchanged source endpoint, restoration of adjacent uncommitted work, FIFO rejection, and a socket becoming a real file during capture. A changed source still fails safely and retains that new file. - Document the portable snapshot behavior. ## Verification - Red regression: both local and serialized captures failed with `remote_workspace_archive_unsupported_entry` before the change. - `pnpm exec vitest run server/src/services/remote-workspace-snapshot-archive.test.ts`: **43 passed** using real filesystem sockets, pipes, Git repositories and archives. - `pnpm -r typecheck` and `pnpm build`: passed. - Module boundaries and `git diff --check`: passed. Added source and PR text were scanned for credentials, private incident identifiers and personal data before push. - Snapshot storage and save-concurrency consumer suites pass **23 tests**. Their first local runs exposed missing workspace SDK build links and unhydrated macOS embedded-Postgres library symlinks in the new checkout. Repairing those prerequisites made the unchanged consumer suites pass; the earlier failures remain recorded and no assertion was weakened. - Greptile reviewed `e313565a5088abce448f11b29982462aa338a692` at **5/5** with no unresolved threads. All current-head CI checks are green and the branch is conflict-free. Hosted recovery needs a qualified deployment and original-task verification. ## Risks Socket endpoints represent runtime IPC and are not recreated by archive restore. Their paths are visible in the existing exclusion manifest. This adds no archive entry type, subprocess, recurring work, provider call or dependency. It does not alter source authorization, stop receipts, ordinary file validation, restore safety or recovery budgets. Unix socket/FIFO fixtures are skipped on Windows; existing portable archive coverage still runs there. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, xhigh reasoning. Used repository tools, code execution and real local filesystem/process fixtures. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] Added focused behavioral and preservation regression coverage - [x] Updated the execution contract and documented limits - [x] Typecheck and build pass locally - [x] Scanned the diff and description before push - [x] Current-head CI is fully green - [x] Greptile is 5/5 with no unresolved comments Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c2cc69d1e4 |
fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. **What happened?** A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. **Expected behavior** Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. **Steps to reproduce** 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass **220 tests**: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence **5/5**; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
9e5039f6c8 |
fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description **What happened?** A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. **Expected behavior** A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. **Steps to reproduce** 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. **Paperclip version or commit** The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. **Deployment mode** Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
11f2a5293d |
fix(native): name the proof that holds a recovered per-turn owner (#19061)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). **What happened?** On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. **Expected behavior** When admission holds a follow-up, the error names the proof that failed. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. **Paperclip version or commit** Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - **Failure before the change:** I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - **After the change:** `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - **Not verified live:** no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5723fd82ca |
fix(native): release a fresh claim when task preparation fails (#19065)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. **What happened?** A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. **Expected behavior** When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. **Steps to reproduce** 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9cdbcbeddc |
fix(workspaces): restore archived commits from verified project objects (#19138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server must preserve historical task files outside its local disk. > - Archived isolated Git workspaces can have a successful branch-deletion receipt. > - Their private checkout can be gone while the project cache still owns the exact commit. > - The importer rejects that cache because it applies private-checkout validation. > - This pull request validates the managed object source for receipt-based restoration. > - The importer restores the recorded commit without using the current shared working tree. ## Linked Issues or Issue Description Refs #18826, #18943, #19038. **What happened?** An archived isolated workspace has a valid successful cleanup receipt and a retained commit. Offline migration rejects it with `archived_git_source_unavailable,local_source_unsafe_or_missing`. The private checkout was removed. The company and project cache still contains the exact commit. **Expected behavior** Restore the immutable commit from the verified project object cache. Keep strict validation for saved private edit recipes. Reject missing receipts, a foreign cache, a different origin and revoked cleanup authority. **Steps to reproduce** 1. Create an isolated Git workspace from a managed project cache. 2. Archive it and retain its successful branch-deletion receipt. 3. Remove its private checkout while the verified project cache retains that commit. 4. Attempt offline workspace migration. ## What Changed - Extract the existing managed Git object-source validator. - Use that validator only for exact successful deletion receipts that refer to a canonical company and project cache. - Keep private restore-point source validation unchanged. - Fetch immutable objects at the recorded commit into private scratch. Recheck receipt authority before publication. - Add real Git and database tests for restoration and rejected authority changes. Document the preservation path. ## Verification - 129 focused migration, concurrency and isolated-source tests pass across three files. - The new restoration case fails against the previous importer with the reported source-unavailable error. - Tests remove the old home and restore the retained commit. A newer shared working tree and unrelated files are excluded. - Tests reject a foreign company cache, wrong origin, missing receipt and receipt revocation during upload. - Local server typecheck has no diagnostics in changed files. Stale linked dependency declarations prevent a clean local whole-server result. Current-head full CI passes (54 passing checks, three skipped). Greptile is 5/5 with no unresolved review threads. - A replay of the 32 affected records is running in an isolated staging copy. Two committed owner overlays are under test. This is not released-image or serving-instance acceptance. ## Risks - A deletion receipt must identify an exact commit that still exists. Missing objects continue to block migration. - Shared working-tree contents must never substitute for archived private files. Only the recorded commit is fetched through this path. - The change does not recreate private edit authority, retire recovery records or alter serving configuration. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7d46532c6d |
fix(runtime): bound parallel offline harness backup transfers (#19108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server must retain provider history outside its
local disk.
> - Offline enrollment moves verified local harness backups to object
storage.
> - A cold rehearsal reached 699 current backup rows while processing
1,953 sources before its 25-minute deadline.
> - The serial transfer loop prevents this existing dataset from
enrolling.
> - This pull request overlaps at most four transfers within the
existing total scratch limit.
> - The benefit is faster preservation with the same ownership and
publication checks.
## Linked Issues or Issue Description
Refs #19038, #18943, #18932.
**What happened?**
Offline enrollment processed 1,953 local provider backups one at a time.
The normal Cloud image reached 699 current backup rows (including nine
pre-existing rows) before maintenance expired. The serving instance
remained unchanged during the isolated test.
**Expected behavior**
Enrollment must preserve every source within bounded time and disk use.
It must stop all in-flight transfers before it reports failure.
**Steps to reproduce**
1. Prepare a settled instance with multiple verified native harness
backups.
2. Run offline enrollment against shared object storage.
3. Observe that the serial owner waits for each archive, upload,
readback and publication before starting the next source.
## What Changed
- Process groups of up to four verified offline harness backups.
- Reserve space for the archive and encrypted copy before each group
starts. Keep the total scratch limit equal to one maximum-size serial
transfer.
- Derive a smaller archive limit from a closed manifest directory set.
Preserve unlisted files with an exclusive maximum-size reservation.
- Enforce the reservation while streaming. Keep the existing maximum for
ordinary runtime backups.
- Cancel and await peer transfers after failure. Keep completed
publications valid for retry.
- Preserve the original archive-stream failure when the tar writer also
fails after destruction.
- Document the migration bound and add integration regressions for
overlap, restoration, cancellation, retry, extra files and archive
limits.
## Verification
- Serial baseline: the new overlap regression fails with one active
transfer when four are required.
- Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem
archives and restored provider bytes. The bundled-entry test is skipped
locally because this source worktree has no runtime bundle. The full
normal image rehearsal remains required.
- Follow-up: 48 archive/storage checks pass after correcting the
PAX-header allowance. The real encoder produces a 972,801,024-byte
full-entry projection, above the old 805,306,368-byte allowance and
below the new 1,024,001,024-byte bound. The existing global object limit
remains in force. The old formula fails the new regression. Exact-size
restoration and one-byte-over-limit rejection pass.
- The previous revision passed full CI after one setup-route timeout
retry. That failure remains recorded; the corrected current revision
|
||
|
|
fb96537e28 |
test(ui): verify disconnect recovery through browser outages (#18832)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - Deploys can interrupt reads and writes while people use the board. > - Component tests cannot prove the browser keeps task content and saved drafts through a reload. > - The read and cache changes need a regression suite against the built app. > - This pull request drives those journeys through temporary API failures. > - CI discovers the new suite through the existing e2e shard plan. ## Linked Issues or Issue Description **What happened?** Outage recovery lacked one browser suite covering retained content, write recovery, cached reloads, access errors and sign-out clearing together. **Expected behavior** Keep loaded task content visible through a 20-second API outage. Show one connection banner and its recovery state. Save comments once and retain document edits. Restore approved reads after a reload. Keep real error responses readable and clear query records on sign-out. **Steps to reproduce** Open a task or settings. Return 503 with `tenant_app_unavailable` from browser API requests for 20 seconds. Type a comment, edit a document, or reload. Restore the API and compare the visible result with the saved server state. Repeat with real 403/404 statuses and sign-out. Stacked on Refs #18825, which depends on Refs #18822. Related durable-write foundation: Refs #17960. ## What Changed - Add `disconnect-resilience.spec.ts` with ten browser cases for comments, a queued comment across reload, document autosave, cached settings/sidebar/task reads, legacy Inbox reloads, 403 and 404 responses, sign-out clearing, restored-draft conflict detection, and first-socket outage control. - Assert one connection banner, readable recovery, no transient error boxes or raw codes, retained task text, one saved comment, and the saved document after reload. - Use the real local server and isolated embedded PostgreSQL. Intercept browser API traffic for outages. Control the logout/session response for the sign-out test. Create no agents or model runs. - Register realtime interception before navigation. Track both ends of current sockets, close them at outage start, and keep reconnects unavailable through the HTTP outage deadline. Preserve normal close forwarding and close tracked pairs during teardown. The regression captures the first browser socket without waiting for history or open, then checks that the outage closes it. - Prepare complete server-derived experimental settings and workspace preferences before the legacy Inbox navigation. Its route callbacks serve per-test snapshots across reloads, avoiding reads from disposed route.fetch responses. - Document the direct test command. The existing CI e2e shards already discover this file exactly once; no workflow or package script is added. ## Verification - Current head `228c1b68d30` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both required gates pass. The single failed-jobs debug rerun in run `41752978283` passed on attempt 2, including Runner package job `126328280650` and `verify` job `126332119116`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. The three-PR stack is awaiting operator merge approval. - Original attempt 1 of current-head CI run `41752978283` had 46 successful jobs, one skipped and two failed: Runner package tests and the downstream `verify` gate. GitHub reports lost communication with the Runner host; the test step was cancelled, its log returns HTTP 404, and no Runner diagnostic artifact is available. The underlying test outcome and disconnection cause are unknown. One failed-jobs retry with debug logging was accepted after verifying the unchanged current head and passed on attempt 2. No source change was made; earlier passing CI and local tests do not certify this incomplete Runner result. Current-head CI now passes; the earlier disconnection cause remains unknown. - Preceding head `6531caccf62` reached merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status was successful, and both required gates passed in run `41749512329`. Greptile reviewed that exact head at 5/5 with zero unresolved threads. Current head `228c1b68d30` carries later account-preference and bulk-selection fixes and now has its own passing current-head CI. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact current head `228c1b68d30`: ten outage, six durable-write and seven loading cases. The run used bundled Chromium, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned temporary server and database stopped. - Current integrated head `228c1b68d30` contains cache revision `fe4f682c879` and read head `09d1db29dea`. The exact application source passes all 1,198 focused tests across twenty-eight files, UI typecheck and both guards. Seventeen new cases cover restored-account settings, owned crown cleanup and pending bulk selections; sixteen fail the previous source and one preserves denial compatibility. All three browser specs pass standalone strict checking. All 23 current-head browser cases pass in 4.5 minutes. - Application revision `9c1a10f25d9` passed all 1,137 focused tests across twenty-one files, UI typecheck and both guards. Exact preceding application/spec head `4fec693ec94` passed all 23 built-UI outage/write/loading cases in 4.6 minutes and standalone strict checking of all three specs. It used bundled Chromium, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. Owned test processes stopped. - Preceding integrated head `870aa709657`, containing cache `cbb891dd37d`, passes all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. It used bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned test processes stopped. - The earlier head `763fe431b74` passes the affected legacy Inbox outage/reload case (34.3 seconds; 50.8 seconds including startup) and strict three-spec checking after the response-snapshot fix. The preceding head `973a12d77f4` passed all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. Its first-socket regression fails the previous helper because that socket stays open. The run used bundled Chromium, the built UI, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. That earlier isolated callback revision was verified with its affected case; the later application/spec head `4fec693ec94` also passes the full suite. Owned test servers and databases stopped. - The current base gives known skills/app errors priority over sibling loading. All 60 affected tests and both application UI typechecks passed. Its twelve new regressions failed the preceding source. All 47 recovery-worker tests and seven related Runner fixture variants also passed during the historical CI audit; those focused checks do not claim a full current server/Runner suite. - All 22 outage/write/loading cases passed in 4.7 minutes on preceding integrated head `b7deaaee5da`, including direct cache revision `97c0bc3966f`. The run used the built UI, bundled Chromium and a fresh local-trusted server with isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. It covered all nine outage, six durable-write and seven loading cases. The owned temporary server and database stopped. - All 1,038 focused tests passed on that direct cache revision, with UI typecheck and guards. Six regressions failed the previous source: online retries could trigger a hidden-tab worker reload, and sign-out failed without crypto.randomUUID. Both modes and stamp states now pass. Standalone strict checking passes for all three browser specs. - The restored cached-A/draft-B/server-C test asserts zero replay PATCHes and retains the conflict. The queued-comment case holds fresh session verification after general API recovery and asserts zero POSTs until release. Denial, logout and cached-navigation cases require saved records. Sign-out seeds several company records to exercise native Storage key reordering. Healthy navigation waits for conversation-history readiness. - Earlier runs exposed real cached-header, storage cleanup and replay defects in the base. Those failures and corrected fixture assumptions remain in the discussion. A prior run passed 21/22 while real sockets remained available during simulated HTTP failures. A diagnostic trace established the contradiction. The first repair gated new sockets, but review identified the first pre-existing socket still escaping. This revision intercepts before navigation and closes existing pairs too. The new regression fails the preceding helper, and all 23 current cases pass without weaker assertions. Shared-memory exhaustion prevented two earlier diagnostic startups; only unused segments from completed owned test databases were removed. - All six separate cross-organization link/upload variants passed on an earlier integrated base. Standalone strict checking of that unrelated spec reports five existing errors in its shared fixture; this PR does not claim that check passes. - CI on older head `53d979e384c` failed the legacy Inbox case with `apiResponse.json: Response has been disposed` inside a route.fetch callback. Its other 25 cases passed and one skipped. The seven other browser shards succeeded; the aggregate only reports that child failure. The new snapshot callbacks remove those fetched-response lifetimes. Original CI evidence remains recorded in the discussion. - Greptile Apex reviewed exact current head `228c1b68d30` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:26:14 UTC). The branch has no merge conflicts. Current-head CI now passes; the earlier disconnection cause remains unknown. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Commands run: ```sh pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts --grep 'a legacy Inbox reload retains' --reporter=list pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts --reporter=list pnpm exec tsc --ignoreConfig --noEmit --strict --target ES2022 --module NodeNext --moduleResolution NodeNext --skipLibCheck tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts ``` ## Risks The suite adds about three minutes to one e2e shard and depends on current accessible UI controls. It uses controlled outages and logout responses. A passing local run must not be treated as real deployment acceptance. There are no production behavior, schema, dependency, or API changes in this PR. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for related PRs and linked them above - [x] I have described the issue following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] All 23 built-UI cases pass on current head `228c1b68d30` - [x] All three browser specs pass standalone strict typechecking on the current head - [x] The current application source passes all 1,198 focused tests, UI typecheck and both guards - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
818d2bb661 |
fix(native): preserve closed post-result failure diagnostics (#19126)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution persists a result before it completes cleanup and workspace export. > - A later execution error enters the failure-write path. > - That write must reject an already selected result. The existing code reports lease loss for every rejected write. > - This can hide the category of the first error even when the same controller still owns an unexpired lease. > - This change adds closed diagnostic labels after exact ownership and accepted-result checks. It keeps the existing control error and recovery policy. ## Linked Issues or Issue Description **What happened?** A local real-PostgreSQL regression accepts a result through the production control plane, then injects a harmless filesystem failure. The production executor reports `native_session_lease_lost`. It loses the original failure category because the failure transition requires a null result ID. **Expected behavior** Keep the result guard and the existing lifecycle classification. Record a closed diagnostic category when the selected result is accepted and the controller still has exact, unexpired authority. Keep ordinary lease-loss fencing when that proof is absent or its diagnostic lock is busy. **Steps to reproduce** Run the new post-result cases in `server/src/services/heartbeat/runtime-execution.test.ts`. They use the real executor, control-plane result commit, heartbeat recovery path and PostgreSQL. The provider boundary is controlled. No live provider is used. **Paperclip version or commit** The regression fails on base `7d1e9b3477`. Related open PR #16917 changes legacy lease recovery. Open #19063 adds terminal finalizer reasons, and #15310 preserves database finalization errors. They do not change this executor diagnostic branch. This change only adds native diagnostic evidence and does not reclaim a lease. ## What Changed - Add a typed diagnostic with a closed stage, category and accepted-result flag. Do not retain the source message, path, arbitrary code or lease token. - After a rejected failure write, check the selected accepted result and the exact company, task, run, owner, attempt, generation, process identity and expiry. Use a coordinator-only `NOWAIT` lock. Fall back to the original control error if the diagnostic read fails. - Report the closed event after rollback. Keep `native_session_lease_lost` as the error message and leave every failure-write predicate unchanged. - Add real-PostgreSQL safety cases for stale and expired authority, owner replacement, Stop state, result selection, pending accounting, one provider invocation, closed labels and diagnostic sink failure. - Document the closed event, unchanged authority, and possible late diagnostic sink completion. - Add a competing-row-lock barrier. It verifies actual PostgreSQL lock refusal and control settlement before the competing lock is released. ## Verification - Permanent regression on unchanged base: RED because the closed diagnostic is absent. - NOWAIT mutation: RED when only NOWAIT is removed. PostgreSQL reports lock timeout instead of immediate lock refusal. The test uses barriers and the actual database error, with no elapsed-time assertion. - Final related suites: 56 tests passed across runtime execution and provider failure integration tests. - Full server source and test typecheck: passed. - Full `pnpm -r typecheck`: passed in an independent locked-dependency checkout. - Full `pnpm build`: passed. - Full `pnpm test:run`: running in the independent immutable checkout. All six chat shards passed; the remaining suites are still pending. - The first current-head CI attempt failed in two cases: workspace runtime initial fixture HTTP readiness and instructions-bundle error-message rendering. Both exact cases passed in isolation on unchanged PR head and base. The original causes remain unknown after bounded test-only observation. One controlled same-head rerun of each failed job passed. Current required CI and all other completed checks passed. Original failures remain recorded; no infrastructure or flaky-test claim is made. - Module boundary check and `git diff --check`: passed. - Exact-head Greptile: 5/5 with no actionable findings or open review threads. Automatic native review remains queued. That advisory check is not required by current master rules; no current-head native assessment is claimed. - Earlier fixture setup failures remain recorded. These included missing fixture fields and observing the Drizzle wrapper instead of its PostgreSQL cause. They are not the regression RED. An earlier broad typecheck was interrupted for the required NOWAIT revision; the final full typecheck passed. - No hosted export recovery or original hosted failure cause is claimed. ## Risks The extra read is diagnostic only. It uses NOWAIT and cannot wait for a competing coordinator lock. Its failure preserves the existing control classification. The event sink uses the existing 60-second execution-control bound. A slow sink can finish later and emit a late event. Result selection, Stop, accounting, retry eligibility and provider work do not depend on event success. No schema or lifecycle contract changes are required. ## Model Used OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2ccc7d37ef |
feat(ui): retain approved reads across outage reloads (#18825)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board already retains loaded reads during a temporary server failure. > - Reloading the page removes that in-memory data. > - A deploy can then leave a returning user without the shell or recent tasks. > - This pull request saves an explicit, limited set of reads in the browser. > - Cached identity supports display while fresh session checks protect writes. ## Linked Issues or Issue Description **What happened?** A reload during an API outage loses the board's loaded reads. An automatic worker update can also reload a hidden tab while a write is pending. **Expected behavior** Show the cached shell and recent tasks during a temporary outage. Limit saved content by user, company, age and build. Clear it on sign-out. Wait for a fresh session before sending account-bound writes. Defer automatic reloads while work is pending. **Steps to reproduce** Open a task and settings while online. Return 503 from the API and reload. Restore the API. Sign out and confirm that saved query records are gone. Discover a worker update while offline or during an autosave and confirm that it waits. Stacked on Refs #18822. Related durable-write foundation: Refs #17960. The outage browser suite follows in a separate test PR. ## What Changed - Persist an exact read allowlist through TanStack Query dehydration and hydration. Save shell preferences, company/sidebar data, agent/project lists, plugin UI metadata, the first inbox page, and 20 recent tasks. Task UUID and identifier reads share one recent-task slot. - Namespace records by user and company. Retain eligible reads across normal memory garbage collection, expire each read after 24 hours, and discard them after a build change. Unstamped builds disable hydration and persistence while retaining sign-out cleanup. Use the existing instance-wide plugin contribution query in the user's root record. - Exclude session credentials and bootstrap data, embedded plan documents, comments, preview work products, provider credentials and environment values, browser capabilities, pairing data, other queries, and mutations. Restore only display identity from session data. Preserve safe agent provider/model display fields and configuration object shapes. Apply the projection on save and restore. - Clear saved reads on sign-out, cross-tab account changes, and definitive session denial. Keep revocation working on plain HTTP without crypto.randomUUID. Use per-user revocation revisions to reject delayed writes and stale records. Preserve a newer account. Remove every UUID/identifier projection and nested inbox copy of a denied task. - Detach shared GETs when the account changes or signs out. Load complete, freshly authorized agents before opening bulk settings from a display-only roster. Retire a pending bulk-editor result or error when row, select-all or clear changes the selection, including a clear/reselect of the same agent. - Require a fresh server lookup before account-bound writes, pending task creation, or mounted autosaves resume. This includes automatic task read receipts, inbox archives, workspace preferences and primary-agent choices, with optimistic archive changes gated too. Cancel the old write scope when the verified user changes. Release completed archives through account-independent cleanup without restoring old cached rows; an old operation cannot clear a newer hide flag. Retry unconfirmed read receipts after a fresh session succeeds for the original account, and remember confirmed receipts across refreshes. Track pending reads per task so opening another task sends its receipt immediately; revisiting reuses an outstanding receipt and its original account scope. - Defer a hidden-tab worker reload until connectivity, mutations, durable sends, and autosaves are safe. Synchronously count each unresolved operation across online rate-limit retries. Release a cancelled target’s unused dirty token when an editor changes tasks; keep stored drafts, admitted saves and queued retries. Cover autosave debounce, newer edits, navigation and undo. ## Verification - Current head `fe4f682c879` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both `ci / verify` and `ci / e2e` pass in run `41752927822`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. All three PRs now meet the merge-ready bar; the stack remains unmerged pending operator approval. - Greptile Apex reviewed exact current head `fe4f682c879` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:24:40 UTC). Both settings-account and pending-selection findings are fixed, replied to and resolved. The branch has no merge conflicts. Current-head CI now passes in run `41752927822`. - Seventeen new cases cover settings restored from actual saved query records, matching/changed/signed-out identities, denied verification, late crown responses without overwriting newer reads, and bulk-selection changes during successful/failed reads. Sixteen fail the preceding source and one preserves denial compatibility. Both settings paths reuse the existing account-bound mutation guard; its optional explicit QueryClient supports the workspace hook’s existing detached-host mode. Primary-agent optimistic updates begin only after verification and cleanup checks ownership and the cache revision. - Earlier CI run `41749456480` on prior head `2cf7516f18c` has one real server test failure: the owned-runtime case in `idle-completed-history.test.ts` could not reach its HTTP child on port 34757 within 10 seconds (99 refused probes). Its other 1,829 cases pass and three skip; the only other unsuccessful job is the downstream `verify` gate. The exact selected case passes locally, and the entire 75-case file passes in 72.19 seconds on the exact `2cf7516f18c` source tree with isolated PostgreSQL and a real child process on macOS arm64/Node 25.6.1. The Linux CI startup cause remains unknown; these local results are not a fix or CI acceptance. One failed-jobs retry with debug logging was accepted on that unchanged head, then superseded by the later settings fixes. Current-head CI remains required; no source change was made for this failure. - Current cache head `fe4f682c879`, based on read head `09d1db29dea` and master `683fa4725e9`, passes all 1,198 focused tests across twenty-eight files, UI typecheck, token gates and the enforcing query-error guard. Dependent browser head `228c1b68d30` contains this exact application source and passes standalone strict checking of all three browser specs. Current-head CI now passes in run `41752927822`. - Three new navigation regressions fail the preceding application source and pass with the fix. They open three cached tasks before any receipt resolves, leave the page before success or failure, revisit an outstanding read without duplicating it, and recover an older failed receipt after a fresh session succeeds while a later task remains pending. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Nine new cases cover archive success/failure after sign-out and another account takes over, both mounted and unmounted; cleanup ownership when a newer archive uses the same task; session-check 429 exhaustion followed by matching, changed or signed-out identity; and a failed read receipt followed by a new session read. They verify no old-cache restoration, toast or navigation, retained newer hide flags, no failed-check loop and no duplicate confirmed receipt. Seven fail the previous implementation; two compatibility cases retain account-change suppression. Initial test error-constructor and query-notification timing assumptions were corrected before the passing checks. - The inherited skills/app error-priority fix passes all 60 affected tests and UI typecheck on both application branches. Its twelve new regression cases fail the preceding read implementation. The persisted-cache suite includes 48 cases. - Six new regressions fail against the preceding application source and pass with the fix. Actual worker-update and mutation-cache listeners cannot reload a hidden tab between online 429 retries; the original create key survives, and the update reloads once after a receipt. Real self-hosted and Cloud sign-out work without crypto.randomUUID, with stamped and unstamped builds. Saved reads are removed and account write scopes are invalidated. An initial test-fixture type annotation error was corrected before the passing typecheck. - The 48 cache tests cover actual five-minute memory GC, the 20-task bound across saved and live reads, original 24-hour expiry, denial before session verification, unstamped cleanup, cross-tab revocation and delayed flushes. Private GETs cannot cross account transitions. Bulk edits require complete current agent data. Undo releases the update blocker while preserving the Saved indicator. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact integrated head `228c1b68d30`, containing current cache revision `fe4f682c879`: ten outage, six durable-write and seven loading cases. The run uses bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. All three specs pass standalone strict TypeScript checking. Owned temporary server and database stopped. - Earlier browser runs exposed and fixed cached-header, native Storage cleanup and replay/session defects. A prior run passed 21/22 because real WebSockets contradicted the simulated HTTP outage; the fixture now closes existing socket pairs and holds new sockets through the same outage deadline. An older CI run exposed a disposed response in the Inbox route callback; pre-navigation snapshots remove that lifetime, and the current full run includes that repair. Earlier failures and their diagnosis remain in the discussion. The current full run passes. Broader historical workspace failures remain documented in Refs #18822; this does not claim a clean full workspace suite. - Current-head CI passes on `fe4f682c879`. Controlled local outages do not complete hosted authentication or a real Cloud restart/deploy acceptance. Commands for the current cache change and inherited read fix (the current focused run also includes src/lib/inboxArchiveCache.test.ts and src/pages/Inbox.test.ts): ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/hooks/useDurableSubmit.test.tsx src/hooks/useDurableAutosave.test.tsx src/hooks/useSignOut.test.tsx src/lib/persisted-query-cache.test.ts src/lib/service-worker-updates.test.ts src/lib/service-worker-cache-policy.test.ts src/lib/pending-send.test.ts src/lib/issue-execution-policy.test.ts src/components/InlineEditor.test.tsx src/components/MarkdownEditor.test.tsx src/components/NewIssueDialog.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/components/agents/AgentBulkRoster.test.tsx src/components/agents/AgentBulkSettingsSheet.test.tsx src/hooks/useAccountBoundMutation.test.tsx src/pages/IssueDetail.test.tsx src/components/TaskChatThread.test.tsx src/components/TaskChatThread.task-creation-order.test.tsx src/api/client.test.ts src/api/agents.test.ts pnpm --filter @paperclipai/ui exec vitest run src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/context/PrimaryAgentProvider.test.tsx src/components/CoordinatorWorkspaceSetting.test.tsx src/components/primary-agent/PrimaryAgentPresentation.test.tsx src/components/MobileBottomNav.coordinator.test.tsx src/components/CoordinatorSidebar.test.tsx src/pages/Issues.coordinator-routing.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This deliberately saves task content on the local browser for up to 24 hours. The allowlist excludes credentials and ephemeral capabilities. Local storage can be disabled or full; persistence then fails without blocking the app. Restored data can be stale and never grants server access. A user or access change clears cached account reads and invalidates pending writes. The browser outage suite and real hosted deploy acceptance remain release requirements. There are no migrations or new dependencies. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The focused local tests for this change pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b7ef307a65 |
fix(workspaces): wait for an ended writer's save instead of failing setup (#19074)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs can use a remote task workspace in a sandbox. One writer run at a time holds that workspace. > - When a writer run ends, its workspace save and sandbox release finish a few minutes later. The writer hold stays until then. > - A new run that started in that interval failed with `setup_failed`: "The workspace still has an unfinished writer or save operation." > - A live writer already makes the new run wait (`workspace_busy`, added in #18691). An ended writer that is still settling did not. > - This pull request reports the ended writer as the holder too, so the new run waits instead of failing. > - The benefit is that work continues automatically after a short settle interval. Users do not see a failed run and a blocked task. ## Linked Issues or Issue Description Refs #18691 **What happened?** A task run failed during workspace setup with `setup_failed` and the error "The workspace still has an unfinished writer or save operation." The previous run on the same task had ended a few seconds before. Its sandbox lease was released 4 to 17 minutes later. A staging soak recorded 31 of these failures over 14 tasks in about 29 hours. **What did you expect to happen?** The new run waits for the workspace, as it does when the previous writer is still running. It continues when the save and release settle. **Steps to reproduce** 1. Bind a task to a remote workspace whose authority has `dirty: true` and a `writerRunId` for a run with status `failed` that ended now. 2. Wake the task's agent. 3. The run fails with `setup_failed`. No retry is scheduled. ## What Changed - `reserveRemoteWorkspace` now throws `RemoteWorkspaceWriterActiveError` for an ended writer whose save or sandbox release has not settled. Before, it threw a plain 409 conflict. The error keeps the same message, status and `remote_workspace_writer_pending` code. It uses the writer's `finishedAt` as its last activity and sets `ended: true`. - Workspace preparation already converts that error to a `workspace_busy` deferral with the one-hour holder cutoff. It now passes `holderEnded` to the deferral. - `WorkspaceBusyDeferral` says "ended, but its workspace save has not settled" for an ended holder, instead of "is still running". - A writer that is missing or has no task still gets the plain conflict, as before. - `doc/execution-semantics.md` describes the wait and its cutoff. ## Verification - Red before the fix: the new test "waits for a writer that just ended while its save or sandbox release settles" failed. The run had `status: failed`, `errorCode: setup_failed`, `executionRecovery.kind: bootstrap`. This is the same result as the staging failures. - Green after the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts src/services/remote-workspace-reservation.test.ts` (16 tests pass.) - Safety cases in the new tests: - The waiting run does not change the workspace authority. The unsaved writer's files and hold stay unchanged. - A writer that ended more than one hour ago still causes the unfinished-writer setup failure, with no scheduled retry. The wait is bounded. - Existing cases still pass: live writer wait with Stop and reassignment, the concurrent reservation race (`remote_workspace_writer_pending`), and the unused-reservation takeover. - Also run: `remote-workspaces.integration.test.ts`, `heartbeat-workspace-busy.test.ts`, `remote-workspace-save-concurrency.test.ts`, `workspace-recovery.test.ts`. All 6 suites pass (442 tests). - Not done: a live run on staging at this revision. The fix is not deployed. ## Risks - A task whose ended writer never settles now makes up to about one hour of short `workspace_busy` waits (one scheduled retry every 1 to 2 minutes) before it fails. Before, it failed at once. The end state after the cutoff is the same. - Some of the staging failures have other causes that this change does not fix: - A previous run that stays `running` with no output for more than one hour still causes this failure, as intended. - A dirty workspace whose writer ended without a saved result (for example a workspace finalization retry that never runs) still blocks the task after the cutoff. These need separate fixes. - No schema, API or UI changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser reads of staging run records) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |