3899 Commits

Include renames

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	e2b337b330	fix(authz): name the Experimental settings page when an agent is denied an instance experiment ... An agent asked to turn on an instance experiment (for example Cases) reads or patches /api/instance/settings/experimental and gets the generic board-only denial, which names no page. Turning Cases off also denies the case API with a bare "Cases are disabled". In both cases the agent cannot tell the person where to act, so it cannot give a usable next step. - The board-only denial for the experimental settings route now links [Open Experimental settings](/<prefix>/company/settings/instance/experimental), unless the operator hides that page. - The case API gate keeps its message and adds a final remediation that names the same page (guarded the same way). - The API catalog entries for the experimental settings route name the page, say agents cannot read or change it, and give Cases/enableCases as an example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 12:17:15 -06:00
DottaandClaude Fable 5.2	ffe05b1b7c	fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 12:05:54 -06:00
Nicky LeachandPaperclip	b2edea5754	fix(workspaces): verify existing worktrees before fresh retries (#20876) ... ## Thinking Path > - Paperclip manages work for AI agents. > - Retry checks whether the task can keep its existing workspace. > - A provider can complete checkout before Core receives the result. > - Recreating a completed worktree costs time and can change its base commit. > - Core uses the same ownership and writer checks for local and provider workspaces. > - The provider verifies its checkout before Core creates a fresh workspace. ## Linked Issues or Issue Description **What happened?** A retry can allocate a new provider worktree even though the original checkout completed. **Steps to reproduce** Complete workspace checkout through a storage provider. Fail Runner startup or lose the prepare reply. Retry the task. **Expected behavior** Reuse the same worktree when the provider confirms checkout and the prior run can no longer write to it. Related: #20865 addresses timeouts while the original prepare is in progress. This PR checks an earlier checkout when the user retries. ## What Changed - Check the prior task binding and writer state before asking a storage provider to verify a checkout. - Extend the existing prepare operation with an optional verification mode. Providers opt in through their manifest. - Return through normal workspace reuse after verification. Keep the fresh retry fallback when verification cannot establish checkout. ## Verification - Core workspace preparation, plugin workspace ownership, retry continuity, and SDK workspace-provider suites passed locally (213 tests). - Focused cases pass for provider verification, retry through heartbeat dispatch, and real local Git checkout verification with no plugin worker. The heartbeat test confirms that ordinary preparation and adapter dispatch keep the original workspace, branch, and base commit. - Core server typecheck passed before the final audit and timing edits. GitHub CI covers the final revision and broader regressions. ## Risks Provider verification adds one bounded call before a fresh retry. Older providers keep their current behavior. Verification must preserve the workspace, branch, and recorded base commit. The normal execution preparation checks still run before admission. No database migration is needed. ## Model Used OpenAI GPT-6 (Codex), with source inspection and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 18:04:18 +00:00
DottaandClaude Opus 5.5	a52f622be8	fix(native): retry a remote provider startup failure once its runner exit is proven (#20874) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a Runner in the sandbox, and the Runner opens the provider session > - When the provider rejects `session.open` at startup, no provider work began > - #20808 ends every such remote rejection at once with Retry, because restart recovery could not prove that the failed runner was gone. A fault that clears by itself therefore needs one human click > - The Runner transport already settles the failed runner on close and observes its exit. That is the missing proof > - This pull request records that proof on the failure and lets the same-run bootstrap retry use it. It stops a startup error that repeats, so an image fault that a retry cannot fix does not loop > - The benefit is that a transient remote startup failure recovers without a person, and a permanent one still rests at once with a working Retry that names the error ## Linked Issues or Issue Description Refs #20808 Refs #20832 Refs #20820 **What happened?** A remote (Daytona) native run whose provider rejected `session.open` at startup ended as `native_provider_startup_rejected` with Retry. This happened even when the Runner had settled and exited, and no provider session, event or checkpoint existed. A person had to click Retry for a fault that the durable record proves did no provider work. **Expected behavior** With a proven runner exit and no provider work, the run retries its bootstrap on the same run automatically. A startup error that repeats on the next attempt stops at once with Retry and names that error. Failures saved without the proof keep today's behaviour. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-startup-rejected-retry.integration.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "remote startup rejection|session.open rejection"` 2. On master, 3 cases fail: the proven remote rejection is `terminal_failure`, the repeat does not name its error, and restart recovery holds the proven failure on `ambiguous_remote_recovery_evidence`. **Paperclip version or commit** master at 3ded7bbd91. ## What Changed - Runner transport (`runnerd-codex-transport.ts`): new `runnerExitConfirmed()`. It is true only after a completed close settled the runner (provider stopped, runner suspended) and the runner process exit was observed. - Codex driver (`codex-app-server-driver-impl.ts`): after it closes the transport for a failed session open, it sets `runnerExitConfirmed` on the error when the transport proves the exit. - Executor (`native-session-executor.ts`): - A remote `provider_session_open_rejected` with a startup `providerFailure` keeps the same-run `bootstrap_retry` only with that proof. The saved failure records `runnerExitConfirmed: true`. - Each failed attempt's recovery history entry records a startup failure signature: provider, category and message, with numbers ignored. When the next attempt fails with the same signature, it stops as `native_provider_startup_rejected`. Its next action names the startup error and keeps "Retry starts a new run". - Deterministic rejections (`requires_action` authentication, permission or invalid request) stop at once as before. - Local targets are unchanged. - Restart recovery (`native-restart-recovery.ts`): `remoteRunnerExitedBeforeProviderWork` also accepts a settled `provider_session_open_rejected` with `runnerExitConfirmed: true`. All its other checks still apply: no provider session, no provider event, no checkpoint, no runner-sent event, no successor. - Docs: `doc/execution-semantics.md` and `packages/paperclip-runner/docs/durable-recovery.md`. ## Verification - Red on master, green here: - `native-startup-rejected-retry.integration.test.ts`: proven remote rejection → `retryable_failure` with `runnerExitConfirmed`; unproven → terminal; repeat → terminal, the next action names `bwrap: loopback`, and Retry is offered. - `native-runner-restart-recovery.integration.test.ts`: proven failure → `bootstrap_incomplete` (`remote_runner_exited_before_provider_work`); an older release's failure without the proof stays held. - `codex-app-server-driver.lifecycle.test.ts`: the driver records the proof only when the transport reports it after close. - `native-session-executor.test.ts`: unit cases for the rule, the `cause` chain, the repeat signature and deterministic rejections. - Local runs: the 3 server files above pass (1380 tests); restart recovery `-t remote` 75 pass; driver lifecycle 25 pass. - `tsc --noEmit` clean for `server` and `packages/paperclip-runner`. - Not run: a Daytona chaos wave. There is no in-sandbox startup fault seam yet (see Risks). ## Risks - The proof depends on the transport's close settling the runner after a failed `session.open`. The Rust runner keeps serving after a failed `session.open` (lifecycle `Continue`) and accepts `runner.suspend`, so close should settle. If it does not settle, there is no proof and the run ends at once with Retry, which is today's behaviour. It is never less safe than today. - Retries stay bounded: the startup retry budget (3 attempts) and the repeat rule both apply. A repeated identical error stops after one automatic retry. - Saved state: failures saved by older releases have no `runnerExitConfirmed` and keep the old rule. History entries without a signature never match a repeat. A mid-run deploy is safe: an old executor never writes the proof. - Not covered by chaos: a remote "Codex fails at startup N times" fault needs a hook inside the sandbox Runner or Codex. DF1's seam is server-side only, and no open chaos PR adds a seam inside the sandbox. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 11:55:15 -06:00
Nicky LeachandClaude Opus 5.5	14cb69245f	fix(projects): do not offer a single project storage option (#20875) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project uses one workspace storage owner. The user chooses it when they create the project. > - The **Available in** selector shows the storage owners that the user can choose. > - Sometimes the default environment cannot be used, and only one storage owner is available. In that case, the dialog shows a selector with only one option. The user must select it before they can create the project. > - A selector with only one option is not a real choice. > - This pull request uses the only available storage owner as the default. The selector appears only when the user has two or more options. > - The benefit is that project creation asks for a choice only when there is one. ## Linked Issues or Issue Description Refs #20835 **What happened?** An example is an instance in managed-sandbox-only mode that has the agent-computer plugin but no managed sandbox. The local environment is hidden, and the managed-only rule finds no default. The project-environments endpoint returns one storage group and an error. The new project dialog shows **Available in** with only one option and disables **Create project** until the user selects it. An API request without `environmentId` fails with `project_environment_unavailable`. **What did you expect?** When only one storage owner is available, the project uses it. No selector appears. **Steps to reproduce** 1. Turn on `enableManagedSandboxOnly`. Install the agent-computer plugin. Do not add a sandbox environment. 2. Open **Create project**. 3. See an **Available in** selector with one option, and a disabled **Create project** button. ## What Changed - `projectEnvironmentOptions`: when the configured default is unavailable and only one storage group exists, the first environment of that group becomes `defaultEnvironmentId`, and `error` is null. When two or more groups exist, an unavailable default still produces the error. The server does not guess. - Project create uses the same default, so an API request without `environmentId` now succeeds in the single-group case. - `NewProjectDialog` shows the selector only when `groups.length > 1`. - Tests: - A new server test covers managed-only mode with only the agent-computer environment. - A new server test covers an unavailable default with two storage groups (the server still refuses to guess). - The archived-default test now expects the server to use the only remaining storage. - A new UI test checks that the dialog never shows one option. - Docs: `doc/project-environment-compatibility.md` and `docs/api/goals-and-projects.md`. ## Verification - `pnpm vitest run src/__tests__/project-environments.test.ts` in `server/`: 16 passed. - `pnpm vitest run src/components/NewProjectDialog` in `ui/`: 12 passed. - `tsc --noEmit` in `ui/`: clean. - Server `tsc` reports 9 errors on this machine. None are in the changed files. They are in `packages/adapter-utils` and a native-runtime test, because local dependencies are stale. CI typechecks with a fresh install. ## Risks - Low risk. The change applies only when the default is unavailable and exactly one storage group exists. In that case, the server picks the group that the user would have had to choose anyway. Run placement does not depend on the project, so this does not change where runs execute. - No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 09:49:33 -08:00
DottaandPaperclip	cda23f08ac	fix(db): preserve legacy project reads during rolling upgrades (#20860) ... ## Thinking Path > - Paperclip controls agent work while web servers change versions. > - The old and new servers can share a database during that change. > - Older servers still read and write the project environment column. > - Migration 0466 removes that column before those servers stop serving. > - A later repair alone leaves a committed missing-column window and loses values. > - This pull request applies the exact drop and repair in one transaction. > - Old reads and saved values survive while current placement behavior stays unchanged. ## Linked Issues or Issue Description Refs #20835 and #20852. **What happened?** Migration 0466 drops `projects.environment_id`. The previous server still selects that column and reads it during admission. Applying a separate repair afterward leaves a window in which those requests fail. The drop also removes saved values. **Expected behavior** A database that has not yet applied 0466 must keep the column and its values visible to concurrent old servers throughout the upgrade. Both published migration entries must be recorded truthfully. Current project responses must keep the new workspace placement contract. **Steps to reproduce** 1. Use a database before migration 0466 with nonempty project environment values. 2. Keep an old project writer transaction open and start the pending migrator. 3. Start an old project read, then commit the writer. 4. Without this repair, the saved values disappear. With this repair, the reader sees the preserved values and both migrations commit together. ## What Changed - Add migration 0467 to restore the legacy nullable UUID column and its original foreign key. Leave the published 0466 SQL unchanged. - Apply only the exact SHA-bound 0466/0467 pair in one transaction. Lock the project table, retain nonempty values in a temporary table, execute both real SQL files, restore values, and record both real journal entries before commit. - Apply the additive 0467 repair normally when 0466 was already committed. - Keep a deprecated database-only field and omit it from current project API responses. Current placement logic stays unchanged. - Add PostgreSQL regressions for concurrent old reads and writes, idempotence, rollback, and an already-applied drop. Document the overlap contract and retain the five newest generated snapshots. ## Verification - Node 24 cached agent preflight passed. - `pnpm exec vitest run packages/db/src/project-environment-overlap-migration.test.ts server/src/__tests__/project-environments.test.ts -t 'project environment rolling migration|saves storage, not'`: 4 passed in 32.57 seconds against real disposable PostgreSQL. - Before the owner fix, the regression had two failures: saved values disappeared and the restore-failure case committed. The fixed case proves that both DDL and journal entries roll back together. - `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter @paperclipai/db check:migrations`: passed. - `git diff --check`: passed. Migration 0466 bytes are unchanged. The generated snapshot set is 0463 through 0467. - Review found a raw DELETE response that still exposed the database-only field. The delete projection now omits it; a real authorized HTTP DELETE passed with successful cleanup. The concurrency test now observes the old reader waiting inside PostgreSQL before releasing the old writer. All three migration cases passed with that barrier. Earlier HTTP fixture setup/cleanup failures remain recorded as failed attempts; they were not product-regression passes. - Candidate `047178538c0df3a9e382259a51635a6e46da5646` has fresh Greptile 5/5 with no open findings. [Required final-head CI](https://github.com/paperclipai/paperclip/actions/runs/43062354612) passed, including build, typecheck, migration checks, all test shards and the aggregate verification gate. The separate Superagent security scan ended inconclusive/neutral; it did not produce a passed scan. Deployed acceptance remains pending. No hosted database or app deployment was changed by this branch. ## Risks The pair takes a table lock for its transaction. Operators must retain their normal lock and statement timeouts. Both SQL files and their exact hashes must match before the pair runs. A failed restore rolls back the drop and both journal entries. If 0466 was already committed, 0467 restores the column and foreign key. It cannot recover values that were already deleted by the earlier release. This repair does not claim to recover those values. ## Model Used OpenAI Codex (GPT-6), with tool-assisted implementation and real PostgreSQL qualification. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 11:08:30 -06:00
DottaandClaude Opus 5.5	e0e6462a73	fix(heartbeat): keep an agent's explicit resume comment that a stale continuation dropped (#20856) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents coordinate on tasks through comments. A comment posted with `resume: true` is an explicit request for more work; on a finished task it reopens the task, for a person or for another agent. > - A comment that arrives while the assignee's run is still settling queues its own run. If the settling run then closes the task, that queued run is cancelled before dispatch as a stale continuation (`continuation_task_ownership_changed`). > - For a person's comment, the outbox saves the unread message first and the release drain reopens the task (chaos s9401/t72). For another agent's explicit `resume: true` comment, nothing was saved. The request was dropped and the task stayed `done`. > - This pull request saves that agent comment too, with its explicit resume intent. The existing release-drain rule for an agent's explicit resume reopens the task and delivers the comment once. > - The benefit is that a parent agent's "do this next turn" request is not lost to a timing race. An agent comment without `resume: true` stays communicative, as before. ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, a parent agent posted a comment with `resume: true` on its child's task: "a changed requirement for your NEXT turn". The child's provider turn had finished 90 s earlier, but its run was still settling (workspace copy-back). The comment queued a new run. 40 s later the settling run applied its `done` decision. The queued run was then cancelled with `continuation_task_ownership_changed`. The task stayed `done` and no run ever read the comment. The cancelled run itself is not shown as a failure (the task is terminal, so task chat hides it, and Sentry skips expected cancellations), so nothing told anyone the request was lost. **Steps to reproduce** 1. Run an agent on a task. While its run is still settling, post a comment from another agent with `resume: true`. 2. Let the run close the task (`done`). 3. The comment's queued run is cancelled as a stale continuation. The task is not reopened and the comment is never delivered. **Expected behavior** The task reopens for the explicit resume request, and the comment is delivered exactly once, as for a person's comment. ## What Changed - `deferUndeliveredCommentsOfStaleRun` finds, per owed comment, the comment's own `issue.comment_added` activity record: written by its author (another agent, not the assignee) with `resumeIntent: true`. Merged runs, whose wake is rewritten as comments join, therefore keep each comment's own intent. - `saveUndeliveredUserComments` also admits those comments (`explicitResumeCommentIds`). When only such comments remain, the deferred receipt records the agent as requester and carries `resumeIntent`/`followUpRequested` in its payload and wake context. The release drain's existing rule for an agent's explicit resume on a `done` task then reopens and delivers it. Person-comment behaviour is unchanged. - `doc/run-log-events.md` describes the rule. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-held-comment-wake.test.ts -t "explicit resume comment"` fails: `expected [] to have a length of 1 but got +0` (the comment is never delivered after the stale cancellation). - GREEN: the whole file passes (29/29), including two-requester merged-run tests in both orders, the new negative test (`does not reopen a finished task for another agent's comment without resume intent`) and the existing person-comment and exactly-once tests. - Related suites `ownership-reconciliation-report`, `execution-continuation`, `chat-wake-conversation-batch` plus this file: 118/118 tests pass locally. One `afterAll` cleanup hook in `ownership-reconciliation-report` timed out under local machine load; CI runs the suite fresh. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - Not done: no restart-chaos cell. The harness posts follow-ups as the board user, which s9401/t72 already covers; an agent-authored `resume: true` comment needs a two-agent workload the harness does not have. ## Risks - Low risk. Only a comment whose own wake was an agent's explicit `resume: true` qualifies, only on a task an agent run closed, and only when no run read the comment. The assignee's own comments never qualify. - The ownership reconciler's durable backstop (`checkStaleDroppedComments`) still covers person comments only. If the fast path fails, an agent's resume comment is not recovered later. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:57:47 -06:00
DottaandClaude Opus 5.5	e6b538efd2	fix(native): retire a superseded warm owner instead of failing the new run (#20844) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep a warm Runner between turns of the same task, so the next run can start fast > - When the next run needs a new provider session (for example after an agent setting or skill changes), it first retires the previous run's warm Runner > - That retirement uses the previous owner's close. When the close cannot prove checkpoint suspension, the new run failed before any provider work, with the previous owner's error `provider_transport_failed: runner did not durably suspend before checkpoint` > - A governed-wait park (a saved question or review request) always leaves such an owner: its question callback stays unsettled by design. A failed Launcher stop RPC leaves one too > - Nothing retried the run, and the task rested as "recovery needed" with no provider work done > - This pull request lets the new run retire the old owner when that owner is contained, and forces the containment when the close left it unproven > - The benefit is that a fresh run never fails on its predecessor's close, and the chaos harness now covers this class ## Linked Issues or Issue Description **What happened?** A new run on a task whose previous run left a warm Runner retained failed within two minutes, before any provider turn. The error was `provider_transport_failed: runner did not durably suspend before checkpoint`. The run row recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Two runs of the same agent on two tasks failed this way 47 seconds apart, after a configuration change made both start new provider sessions. One predecessor had parked on a `request_human_input` question; its callback never settled. **Expected behavior** A run that has done no provider work does not fail because its predecessor's Runner could not prove suspension. The new session never resumes the old checkpoint. Core must only prove that the old owner is contained, and then start. **Steps to reproduce** 1. Run a warm native task that parks on a saved question (chaos workload Q1). 2. After the asking run settles, change the agent's adapter configuration, so the next run cannot resume the saved session. 3. Answer the question. The follow-up run retires the parked owner and fails with the error above. Related: #18373 added the stopped-owner reset path that this change extends. ## What Changed - `closeWarmNativeSession` (server native executor): during retirement for a different normalized session, accept an unrecoverable close when the old owner is contained. A plugin task owner must show its `executionStopped: true` receipt and a completed handoff. Without a task owner, the Runner's own durable suspension (`settlement.runnerSuspended`) is the proof. Unsettled Core tool callbacks of the discarded conversation no longer block the reset. - With a plugin task owner, any close error qualifies once the Launcher receipt exists. This includes a failed `stop` that `releaseRunnerProcessOwnership` rethrows, and "Controller journal drain timed out": the discarded session's journal lives in its own state root. Owners without a task keep the narrower rule. `--matrix` now rejects session faults. - `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When close left the receipt unproven, the new run sends one confirmed `stop` (not resent after a confirmed stop), takes the credential snapshot, and calls `complete`. It never reads the old state. - Same-session continuation, a handoff that still fails, other close errors and failed instruction collection still block replacement. - Chaos harness: add session fault SR1 (`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle warm` (or `--daytona-warm`), the board changes `adapterConfig.idleTimeoutMs` after the asking run settles and before it answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant. Q1's oracle applies, plus `session_rotated_before_answer` and `follow_up_new_session`. - Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README describe the new boundary and cell. ## Verification - RED on master: `native-session-executor.test.ts -t "permanently rejected plugin close during reset"` fails the `pending-semantic` and `stop-unconfirmed-at-close` cases with `NativeSessionCloseUnrecoverableError: provider_transport_failed: runner did not durably suspend before checkpoint`. GREEN with this change: all identity-transition cases pass (49). - Chaos SR1 local, master server code (`--env local --lifecycle warm --workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1 task): all tasks pass, and the follow-up started a new provider session. The local parked owner closes cleanly, so the local cell does not reproduce the failure. It needs a Daytona task owner. - Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one agent): 2/2 pass, `follow_up_new_session` true for both. - Not run: the Daytona SR1 variant. No runner image matches this checkout's runner content ID, and this change does not touch runner inputs. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts question-handoff scheduler config` passes. Server and harness `tsc --noEmit` pass. ## Risks - The reset no longer waits for unsettled Core tool callbacks of the discarded session. Their run is already terminal and their provider is stopped, so no result can reach a provider, and the new session never replays them. Core-side tool executions keep their own receipts under the old run. - A local owner without a task needs the Runner's durable suspension. A local owner that did not suspend still blocks replacement, as before. - Previous-release story: this changes no persisted state. Warm owners live only in controller memory, so a deploy drops them as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:53:50 -06:00
DottaandClaude Opus 5.5	3ded7bbd91	fix(native): do not claim a server restart for a held same-run retry (#20832) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) retry provider startup on the same run > - A due remote same-run retry re-enters the restart-recovery classifier > - When that classifier holds the run, the hold note says "Automatic recovery stopped after a server restart". No server restarted, so the user reads a false cause > - This pull request records the trigger of the hold and words a same-run-retry hold as a stopped automatic retry > - The benefit is that a held task names its real cause ## Linked Issues or Issue Description Refs #20808 Refs #20820 **What happened?** Two Codex runs on one Daytona sandbox failed in `session.open` (`bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`). The executor saved a `bootstrap_retry`. Thirty seconds later, the due retry entered restart recovery and stopped with `ambiguous_remote_recovery_evidence`. The task said "Automatic recovery stopped after a server restart". No server restart happened. The second run was the board's Retry of the first, and it failed the same way. **Expected behavior** A hold that a same-run retry creates must not claim a server restart. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "same-run retry"` 2. On master the recovery action has no `trigger`, and its note contains "server restart". **Paperclip version or commit** master at 5f38226290. ## What Changed - Restart recovery records `trigger: "same_run_retry"` in the recovery action evidence when the held run was a saved `retryable_failure`. It reads this from the coordinator, so a deferred check or a check after a real restart records the same trigger. - `executionRecoverySettlementNote` says "Automatic retry of this run stopped" for such a hold. It does not name a cause, because the classifier can also stop it for `execution_recovery_budget_exhausted`. Holds without the field keep the old text. - Tests: one integration case with the staging signature and one unit case for the note. ## Verification - Red on master: the integration case fails because `evidence.trigger` is absent; the unit case fails because the note says "server restart". - Green here: `npx vitest run src/services/execution-recovery-resolution.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "server restart|same-run retry|exited remote runner|remote bootstrap retry held"`: 19 pass on the rebased head. - `npx tsc --noEmit -p server/tsconfig.json`: clean. ## Risks - Low risk. The change adds one optional evidence field and one note branch. Recovery decisions do not change. - Saved state: holds that an older release wrote have no `trigger` and keep the old note. A mid-run deploy reads the field only when present. - Not fixed here: the Codex Planning startup fault itself (`bwrap: loopback: Failed RTM_NEWADDR`). #20820 changes the external-sandbox planning profile so Codex no longer uses its network-isolated filesystem helper. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no doc change needed) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:44:27 -06:00
DottaandClaude Fable 5.2	464e1de9b1	perf(issues): decide task reads for a list in one batch (#20858) ... The task routes that filter several rows for one actor (summaries, lists, blockers, relation edges and the chat bootstrap) asked for one authorization decision per task, all at the same time. Each decision sent a task-row read and a copy of the task-privacy statement, which is more than 1 KB of SQL, so a request for 50 summaries sent 127 to 279 statements on every pool connection at once. The routes now ask the authorization service for all decisions in one batched read. It reads the task rows and applies the same privacy predicate to all of them in one statement, and reads membership, agent and run policy once. Within a batch, the authorizing user's membership is read once, and the boundary checks of a low-trust agent, which stay per task, run at most eight at a time. A request for 50 summaries now sends 22 to 28 statements (65 for a low-trust agent). Tests compare the batch with one check per task for 24 actors. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 10:42:21 -06:00
DottaandClaude	c8ae02e5ce	fix(native): start a fresh session when an accepted run's provider state was not saved (#20811) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a per-turn remote sandbox saves a provider state backup after its result is accepted. The sandbox is then released, so the backup is the only copy of the provider session. > - When that backup fails to save (for example, #20772: storage throttling), the run keeps its accepted result. Nothing records that its checkpoint can no longer be resumed. > - The next run on the session rebinds that checkpoint and asks the Runner to resume. The Runner has no state and refuses (#20757). With the normal policy, the run opens a replacement session after that failed attempt. With a governed-wait or unfinished-goal checkpoint, the run fails with `native_provider_state_incompatible`, and only Retry or the next message starts fresh. > - The server already knew the state was gone when the backup failed. The failed resume is wasted work. > - This pull request marks the accepted run's checkpoint as not resumable at the moment the backup fails. The next run then starts a fresh provider session directly. > - The benefit is that a lost backup no longer costs a failed run or a failed resume attempt before the task continues. ## Linked Issues or Issue Description Refs #20757 (the missing-state refusal that the next run reaches today). Refs #20772 (the throttled backup upload that loses the backup). Refs #20809 (a related fix for governed-wait follow-ups that reach the missing state; this PR stops most of them from reaching it). **What happened?** 1. A native run in a per-turn sandbox finished, and its result was accepted. 2. `harness_state.backup.persist` failed. The run recorded `native.post_result_failure`. Its run row kept a normal `sessionCheckpoint`. 3. The sandbox was released. 4. The next run on the same session selected that run as its resume source, and `rebindNativeSessionCheckpoint` returned the checkpoint. 5. The Runner found no saved state and refused with `PRP provider resume state is unavailable`. The run then opened a replacement session after the failed attempt, or (governed-wait or unfinished-goal checkpoint) failed with `native_provider_state_incompatible`. Retry or the next message then started fresh. **Expected behavior** The next run starts a fresh provider session at once, because the server already knows the saved state is not there. The accepted result and the task outcome stay as they are. **Steps to reproduce** 1. Give a run row an accepted result and a normal `sessionCheckpoint`. 2. Make its close checkpoint fail in a per-turn sandbox (`reusableLeaseConfigured: false`). 3. Select the next run's resume source with `findNativeSessionResumeRun` and call `rebindNativeSessionCheckpoint`. 4. On `master` it returns the checkpoint, so the next run tries to resume state that does not exist. **Paperclip version or commit** `master` at `556b249897`. ## What Changed - `server/src/services/native-runtime/native-provider-state-unsaved.ts` (new): `markAcceptedRunProviderStateUnsaved` adds `nativeProviderStateUnsaved: { code: "native_provider_state_incompatible", reason: "harness_backup_not_saved" }` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it only when the run's coordinator holds a result with `schema_status = 'accepted'`. `clearRunProviderStateUnsaved` removes it. - `server/src/services/native-runtime/native-session-executor.ts`: the remote close checkpoint (`checkpointRemoteRunner`) now wraps the existing body. When the checkpoint throws in a sandbox without a reusable lease, it calls the marker, writes one run-log line, and throws the original error. A marker or log failure cannot replace that error. If a later checkpoint of the same run saves, it clears the mark. - `server/src/services/native-runtime/native-session-resume.ts`: `rebindNativeSessionCheckpoint` returns `null` for a source with the mark, as it already does for `errorCode = native_provider_state_incompatible`. `findNativeSessionResumeRun` projects the new key. - `doc/execution-semantics.md`: one additive paragraph. - Tests in `native-session-resume.test.ts` and `native-session-executor.test.ts`. Guard conditions. The mark is written only when all of these are true: - The close checkpoint threw before the durable backup was stored (after the upload's own bounded retries). A later local publication, stamp or cleanup failure does not mark the run. - The target is a remote `sandbox` with `reusableLeaseConfigured !== true`. A retained sandbox or an SSH host keeps the live state, so its resume path is unchanged. - The run holds an accepted result. A failure before acceptance changes nothing. Saved state from the previous release: run rows without the key keep today's resume path. An older server that reads a row with the key ignores it, and so it behaves as it does today. No schema change. User-visible: the next run's transcript shows "Paperclip session started" instead of "resumed", with a new session id. The source run's log shows `provider state was not saved after the accepted result; the next run starts a fresh provider session`. The fresh session gets the full task context, as for other fresh starts. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-resume.test.ts -t "not resumable only after|provider state was not saved"`: 9 pass (6 pure, 3 embedded Postgres). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "marks the provider state unsaved|uses the bundled Linux image authority for remote opencode-new-durable"`: 8 pass. They include a durable store failure (marked) and a local publication failure after the store (not marked). - Red on `master` (source files reverted, tests kept): the 3 "starts the next run in a fresh provider session directly" cases fail (`expected {…} to be null`), the DB "accepted" case fails the same way, and the executor per-turn case fails (`mark` called 0 times). - Guard removal checks: - Sandbox guard replaced with `true`: the "retained sandbox" case fails (`mark` called once). - `!checkpointBackupStored` removed: the "publish-failed" case fails. - `schema_status = 'accepted'` removed from the marker: the "rejected" case fails (`expected true to be false`). - Negative cases that pass: no result, rejected result, retained sandbox, records without the key or with another code (all three recovery policies), and a later save that clears the mark. - `cd server && npx tsc --noEmit -p .`: clean. - Not related to this change and failing on `master` in this worktree too: `runtime-selection.test.ts` connector skill cases (missing `.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts` "archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required CI is the authority for those suites. ## Risks - Low. The change only skips a resume that cannot succeed. The fresh session uses the same workspace and full task context, like a Retry after `native_provider_state_incompatible`. - If the backup failed but the provider state were still somewhere recoverable, the next run would lose provider context. For a per-turn sandbox without a reusable lease, the restore path uses only the backup, so this does not apply. Retained sandboxes and SSH hosts are excluded. - One extra conditional `UPDATE` runs only on the failure path. The normal path before `turn.start` has no new waits or work. - `doc/execution-semantics.md` is also edited by #20809. The paragraphs are at different places; whichever merges second rebases. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 10:36:00 -06:00
DottaandClaude Opus 5.5	1b39cf1794	fix(native): retry a contended provider-pack admission write (#20850) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=<SQLSTATE or native_* code>)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:28:55 -06:00
DottaandClaude Fable 5.2	381b862135	fix(native): let a finish through after one stale-result refusal; rebuild a missing instruction bundle (#20833) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native Codex run ends its task with a Paperclip terminal call (`paperclip_finish` or `paperclip_block`) > - The Runner refuses a terminal call that the agent decided before it saw earlier tool results (`terminal_call_before_results`). This protects against stale calls held across a restart. > - On staging, Codex cancelled a nested exec cell. Codex never sent `item/completed` for one Paperclip tool call in that cell. The Runner then refused every later finish and block in the turn, and the turn ended with no semantic result. > - The same-run recovery then resumed the run on a new controller container. The run's admitted instruction bundle is stored on the controller's local disk, so it was missing there, and runnerd failed with ENOENT. > - This pull request makes a refusal name a returned call once, and rebuilds a missing admitted instruction bundle before the Runner backend starts. > - The agent-directory lock part of the same failure chain ("Agent directory operation is already active") is fixed separately by #20829. > - The benefit is that the agent can finish after one refusal, and a resumed run does not fail because a deploy moved it to a host without the bundle. ## Linked Issues or Issue Description Refs #20829 (the agent-directory lock wait and dead-owner release; this PR does not change that lock). **What happened?** Staging Codex runs on Daytona sandboxes failed after about 30 minutes of work: 1. Early in the turn, the agent called `paperclip_search_assigned_tools` from a nested exec cell. Codex cancelled that cell. The controller returned the result, but Codex never sent `item/completed`, so the Runner never committed `tool.execution.completed` for it. 2. At the end, the agent called `paperclip_finish` four times and `paperclip_block` once. The Runner refused each one with `terminal_call_before_results` ("Those calls have now returned"). The old call stayed "returned but unseen" for the rest of the turn. 3. The turn completed with no semantic result (`native_finalization_missing`). Same-run recovery scheduled another attempt. 4. On the new controller container, the attempt failed with `runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir '/paperclip/instances/default/runtime-context-assets/bundles/<sha>'` (one run) or `ENOENT ... lstat` of the same kind of path (a run during a rolling deploy). In both, `<sha>` is the run's admitted instruction bundle. The asset root is controller-local, and the new containers had no volume. 5. The last attempt failed with "Agent directory operation is already active" (fixed by #20829). **Expected behavior** The agent finishes after one refusal. A resumed run rebuilds its exact admitted instruction bundle on the new host. **Steps to reproduce** 1. Run a Codex task that calls a Paperclip tool, then `paperclip_finish`. 2. Drop Codex's `item/completed` for the first tool call (new chaos fault `--faults PV2 --provider-fault-message tool_completed`). 3. On master, every `paperclip_finish` is refused and the run ends with `native_finalization_missing`. 4. Separately, delete `runtime-context-assets/bundles/<instruction digest>` and resume a run that pinned it: runnerd fails with ENOENT. **Paperclip version or commit** master at 1d67e309e0. ## What Changed - `packages/paperclip-runner/src/live/terminal-call-freshness.ts` and `runnerd-codex-transport.ts`: when the Runner refuses a terminal call with `terminal_call_before_results`, it records the refused call id with the returned-but-unseen call ids it named. A later terminal call ignores those calls only when the refused call's `tool.execution.completed` (the refusal reached the provider) is committed before the later call starts. A terminal call queued before the refusal reached the provider, a call that was pending at the refusal, and a call made after it are judged as before. The record is in memory, so after a Runner restart the agent can get one more refusal. - `server/src/services/native-runtime/runtime-context.ts`: new `rehydratePinnedInstructionAsset`. When the admitted instruction bundle is not present on this host, it rebuilds it from the agent's instruction revision whose content has the admitted digest (single-file bundles), or from the current instructions when they still produce the admitted digest. It never substitutes different content. When nothing matches, it leaves the context unchanged. - `server/src/services/native-runtime/native-session-executor.ts`: `createRunnerdBackendWithinSessionClaim` calls it before the backend starts. When the bundle is present this is two local file reads. - Restart chaos: new provider message `tool_completed` for the Codex shim (PV1/PV2/PV3). It targets Codex's `item/completed` for the first Paperclip tool call that is not a finish or block. Config accepts it with A1 and C1 on Codex only. The README documents it. ## Verification - RED on master (fix reverted, new tests kept): `vitest run src/live/terminal-call-freshness.test.ts` in `packages/paperclip-runner` fails with `expected [ 'paperclip_search_assigned_tools' ] to deeply equal []`. - GREEN: `terminal-call-freshness.test.ts` 13 passed; the transport tests for terminal calls pass, including a new test where the first finish is refused and the second is dispatched, and tests where a queued stale terminal call and a late-returning pending write are still refused. - `server/src/__tests__/runtime-skill-preservation.test.ts`: 9 passed, including the new test that deletes an admitted instruction bundle and its manifest, then rebuilds the exact bundle from the revision history while a newer revision exists. - `vitest --config tests/restart-chaos/vitest.config.ts provider-faults.test.ts`: 22 passed. - Chaos wave `w-20261111T150212-s301` (`--env local --adapters codex --workloads A1 --faults PV2 --provider-fault-message tool_completed --tasks 1 --seed 301`): 1/1 PASS, `provider_fault_applied` applied=1, R1–R6 and the A1 oracle pass. - `pnpm verify` asks for broader verification (a restart-chaos README change is a global CI input); required CI supplies it. ## Risks - The terminal-call change lets a later finish through when a returned call never reaches the agent. The earlier refusal named that call and reached the provider first, so the agent decides with that knowledge. - The instruction rebuild adds two local file reads before the Runner backend starts. It writes only when the bundle is missing, and only bytes whose digest equals the admitted digest. - Not covered here: a multi-file (legacy) instruction bundle whose content changed since admission still cannot be rebuilt; a skill bundle with neither a version id nor a preservation receipt is not rebuilt; the runner-side quarantine after an S3 `SlowDown` during the post-turn harness backup. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 10:25:37 -06:00
DottaandClaude Fable 5.2	8ab955776d	perf(github): check the requester provenance of an issue list's review waits in one batch (#20851) ... <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An agent can review a GitHub pull request. After the agent publishes the review, the task waits in review for the next GitHub reply. A later run of the same task can inherit this wait. > - Issue lists, issue summaries and inbox attention show these waits. For each waiting task, the server first checks the requester provenance of the waiting run (`chatRunSourceMessageId`). This check is an authorization check: it walks the identity-context lineage of the run and rejects a changed requester, another task, another company or another provider. > - #20673 batched the review reads and the publication reads of one list request. It kept the provenance check as one call for each waiting run, inside `Promise.all`. > - Thus a request with N waits still built and sent at least 2N statements at the same moment. The database is fast for each statement, but the server builds each statement on its main thread. > - On a hosted instance, a CPU profile of the server showed the main thread at 96% of one core. SQL text building, per-statement timers, socket writes and garbage collection had the largest shares. Connection samples showed 235 to 292 pool connections whose last statement was the source-message link read of this check, and 53 to 112 whose last statement was its identity-context read. > - This pull request checks the provenance of all the waiting runs of one request in one batch. The runs that need the same kind of read share one statement. Each run gets the same decision as a separate call, from the same walk and the same reads. > - The benefit is that a list request with 250 waits sends 17 statements, not 658, and uses about one quarter of the CPU time. The statement count no longer follows the number of waits. ## Linked Issues or Issue Description Refs #20673 (batched the review reads and the publication reads of the same request; this change does the same for the provenance reads that #20673 kept for each run). Refs #20551 (authorizes the waits of one issue page with one read). **What happened** `readInheritedGitHubReviewWaits` called `chatRunSourceMessageId` one time for each waiting run, inside `Promise.all`. Each call sent an identity-context read and a source-message link read. A run that continues an earlier run also sent a parent-context read and a source-run read for each continuation in its lineage. `listExternalReviewWaits` calls this function for the issue list, the issue summaries and the inbox attention of a company, so each of those requests repeated the same reads for every waiting task. Evidence from a hosted instance (sanitized): - CPU profile of the server process, 30 s, 5 ms samples: main thread at 96% of one core. Building SQL text for select statements: 17.7% inclusive. The idle-timer restart that the database client does for each statement: 7.6%. Socket writes: 6.2%. Garbage collection: 13%. - Three samples of the pool connections (about 494 connections). 235 to 292 connections had the source-message link read of `chatRunSourceMessageId` as their last statement. 53 to 112 connections had its identity-context read as their last statement. - The `DISTINCT ON` review read that #20673 added ran about 29 times each minute. It runs one time for each list request, so each request started hundreds of provenance reads. **Expected behavior** One list request sends a number of provenance statements that does not follow the number of waiting tasks. The decision for each run does not change. **Steps to reproduce** 1. Put many tasks of one company in review, each with an inherited GitHub review wait. 2. Open the issue list or the inbox. 3. Count the statements that the server sends. The identity-context read and the source-message link read run one time for each waiting task. The benchmark below does these steps on a local PostgreSQL. ## What Changed - `server/src/services/chat-channels/run-origin.ts`: - The provenance decision is now one walk (`sourceMessageWalk`). The walk asks for one read at a time and decides on the returned rows only. Its checks, their order and its error messages are the checks of the previous `chatRunSourceMessageId`. - The five reads of the walk (active identity context, source-message link, board message, parent context, source run) are defined one time in `provenanceReads`. Each definition takes its values either as literals or as SQL columns. - `chatRunSourceMessageId` (single run) drives the walk with one statement for each read, as before. The joins, the company scope and the filters are the same. The identity-context reads now select the seven columns that the walk uses instead of every column. - New `chatRunSourceMessageIds(db, runs, options)` (batch). It drives one walk for each run. In each round, the runs that wait for the same kind of read share one statement: `unnest(...)` of the request values, `CROSS JOIN LATERAL` the same single-row read with the request columns in place of the literals. Thus each run gets the row that its own single read returns, with `LIMIT 1` and the company scope on each join. The result for each run is `{ messageId }` or `{ forbidden }` (the 403 that the single call throws). A database error rejects the whole batch, as `Promise.all` did. - `server/src/services/chat-channels/github/review-wait.ts`: `readInheritedGitHubReviewWaits` checks the provenance of its runs with the batch. A 403 still gives `revoked`. The single-run `readInheritedGitHubReviewWait` does not change. - `server/src/services/chat-channels/run-origin.test.ts`: two new tests (see Verification). - `server/src/__tests__/github-review-wait.test.ts`: one new regression test for the statement count. - `server/scripts/benchmark-review-wait-provenance.ts`: the reproduction and the before/after measurement. It loads the pre-fix modules from `--baseline-ref` and fails unless both versions return the same waits. - `scripts/benchmark-github-review-wait.ts`: removed an assertion that `run-origin.ts` is byte-identical to its baseline. Both sides of that benchmark import the current helper, so the comparison stays fair. Without this change, the documented command of that benchmark fails. - `doc/DATABASE.md`: describes the batch provenance check and the new benchmark command. Statement count of one batch: the walk of one run does at most 3 + 4L reads, where L is the number of continuations in its requester lineage. The batch does the same number of rounds for its longest lineage. Each round sends at most one statement for each kind of read (5 kinds), for any number of runs. A request whose waiting runs accepted their GitHub comment directly sends 2 provenance statements. ## Verification Red/green regression test (`server/src/__tests__/github-review-wait.test.ts`): - `reads the waits of forty runs with the statements of four runs`. Before the fix: `expected { few: 10, many: 82 } to deeply equal { few: 4, many: 4 }`. After the fix: passes. Equivalence tests (`server/src/services/chat-channels/run-origin.test.ts`), on real PostgreSQL: - `gives every run of a batch the decision of its own check, in every provenance state`. 33 runs, each in a different provenance state, are checked with the single function and with the batch for five option sets (no options, legacy wake allowed, GitHub, GitHub with legacy wake, Slack). The test compares the decision of each run (message, `null`, or the 403 and its message) for: the full batch, the reversed batch, and a batch of one run. The states include the negative cases: a run of another company, a message that is linked only in another company, a message that is linked to another task, a conversation that moved to another task, a continuation on another task, a changed requester (run, identity context, continuation, board message), a link of another provider, a missing link, an outbound link only, a deleted message, an agent message, a rejected or pending identity context, a missing parent, another agent and a lineage cycle. The test also states the expected GitHub decision for each state, so the comparison cannot pass on two wrong answers. - `checks the runs of a batch with statements that follow the lineage, not the number of runs`. 4 runs and 40 runs each send 2 statements. 40 separate checks send 80. One continuation in each lineage gives 4 statements for the batch. - A mutation check: when the batch gives a row to the wrong request, both tests fail. Commands: - `pnpm vitest run server/src/services/chat-channels/run-origin.test.ts server/src/__tests__/github-review-wait.test.ts` - Focused run of 15 test files that import the changed modules, their callers or the shared seed helper, including `external-chat-wait.integration.test.ts`, `chat-github-tools.test.ts`, `slack-authority.test.ts` and `issue-list-review-wait-authorization.integration.test.ts`: 482 tests passed. In `idle-completed-history.test.ts`, `afterEach` hooks (a `TRUNCATE` of the test database) timed out at 30 s for 1 test in this run and for 3 tests in a second run of the file, while other agents loaded the machine. No assertion failed. The three tests of that file that reach the changed code pass when they run alone. CI runs the full file. - `server/src/__tests__/chat-channels.integration.test.ts` also uses the single check. It was not run locally (very large file). CI runs it. - Typecheck of the changed files and of the callers of `chatRunSourceMessageId` with a focused `tsc` program: no errors in these files. The full server typecheck was not run locally (shared machine). CI supplies it. - The first CI run of this pull request failed `ci / policy`: the module boundary check rejects an insert into the agents table outside the agent lifecycle module, and the benchmark seeded its fixture agent that way. The second commit seeds that agent with SQL. `pnpm check:module-boundaries` passes locally, and the benchmark gives the same statement counts. - `pnpm verify` returns exit 2 (broader verification required) because the change adds a file under `server/scripts/`. CI supplies the broad coverage. Before/after measurement: ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-review-wait-provenance.ts --baseline-ref=e2cf9a670fc5d5f5058fd0972beb71650f77c554 ``` One call of `listExternalReviewWaits` (the read of one issue list request). Embedded PostgreSQL 18.1, Node v26.4.0, `VACUUM ANALYZE` after seeding. One company, one agent, one GitHub repository. Every wait has a published review. Every tenth waiting run continues three earlier runs. 3 warm-up calls, then 15 pairs in alternating order. Both versions return the same waits and the same inherited wait for each run. | Waits | Version | Statements | SQL text bytes | Wall ms, median (min–max) | CPU ms, median (min–max) | | --- | --- | --- | --- | --- | --- | | 20 | before | 60 | 31,084 | 14.1 (12.8–19.6) | 13.7 (8.7–39.3) | | 20 | after | 17 | 15,847 | 12.5 (9.4–22.7) | 8.8 (4.1–87.1) | | 250 | before | 658 | 300,611 | 63.3 (58.1–89.6) | 78.1 (64.8–103.3) | | 250 | after | 17 | 17,171 | 20.8 (18.9–29.2) | 18.7 (14.1–27.3) | | 1000 | before | 2,608 | 1,179,710 | 256.0 (221.0–363.2) | 303.3 (262.2–507.8) | | 1000 | after | 17 | 21,695 | 54.9 (46.7–81.7) | 51.0 (41.6–72.2) | Of the 658 statements before the fix at 250 waits, 650 are provenance reads: 325 identity-context reads, 250 source-message link reads and 75 source-run reads. After the fix, 9 statements do the same reads. The other 8 statements of the request do not change. Sizes are set with `WAITS=20` and `WAITS=1000`. Other agents used the same machine during the measurement, so the maximum values have noise. - The batch statements use index scans only (`run_identity_contexts_pkey`, `run_identity_contexts_company_run_idx`, `chat_message_links_company_comment_idx`, `chat_conversations_pkey`, `chat_endpoints_company_id_uq`, `heartbeat_runs_pkey`). The 9 batch statements of a request with 250 waits take about 1 ms of server execution in total. - Wall time and CPU time are almost equal before the fix. This shows that the cost is in the server process, not in the database. Not verified: - The effect on the hosted instance. This pull request does not deploy. The share of the main-thread CPU that this path uses there is not known; the profile gives the share of all statement building. ## Risks - The provenance check is an authorization check. The risk is a batch decision that differs from the single decision. Mitigation: both paths run the same walk and build each read from the same definition. The batch read is the single read inside `CROSS JOIN LATERAL`, for each request row. The equivalence test compares both paths in every state. - A database error (for example an invalid UUID in a legacy task binding) rejects the whole batch. The previous `Promise.all` also rejected the whole request. The callers of the batch pass the native task of each run. - The statement count of the batch follows the longest lineage of its runs. A batch with one very long lineage sends as many statements as that run sent before, in sequence. It does not send more. - The batch sends its statements in sequence on one connection. Before, the reads of N runs used up to N pool connections at the same moment. - The single-run callers (GitHub tools, review publication, response comments, Slack authority, the finalizer, the status committer) keep one statement for each read. Their identity-context reads select fewer columns. - No migration. No new index. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Anthropic, used through Claude Code with tool use (code reading, editing, shell, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 10:19:29 -06:00
DottaandPaperclip	e625505ac9	fix(auth): preserve credential sign-in during server overlap (#20839) ... ## Thinking Path > - Paperclip controls AI agent work across software updates. > - Authenticated instances must keep human sign-in available during an update. > - Better Auth 1.7.16 identifies accounts by provider and account ID. > - It creates credential accounts without the issuer required by the previous reader. > - A new account can therefore fail to sign in through a surviving 1.7.2 process. > - This pull request keeps the legacy credential issuer on trusted account creation. > - The security upgrade and provider/account uniqueness stay in place. ## Linked Issues or Issue Description Refs #18906. **What happened?** After the Better Auth upgrade, a credential account created by the new version has a NULL issuer. A surviving Better Auth 1.7.2 process rejects its correct password because it selects only the `local:credential` account identity. **Expected behavior** A new credential account must sign in through both versions during an update. The new version must keep its provider/account lookup and uniqueness rules. **Steps to reproduce** 1. Apply migration 0465 to a disposable database. 2. Create an email/password account with the new auth mount. 3. Sign in to the same database through Better Auth 1.7.2. 4. Without this change, signup and new-version sign-in return 200, but old-version sign-in returns 401. ## What Changed - Register issuer as an optional internal account field with client input and response output disabled. - Set `local:credential` in the trusted before-create hook. Preserve any nonempty issuer supplied by internal code. - Leave unconfigured providers without an issuer on the new NULL-issuer behavior. No social provider is configured by this module. - Add credential integration and issuer-policy controls. Document the exact compatibility scope. ## Verification - Node 24: `node scripts/agent-preflight.mjs` passed using the existing build cache. - `node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts -- node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/better-auth.test.ts server/src/__tests__/better-auth-credential-signup.integration.test.ts`: 22 passed. - Separate local qualification used the actual cached Better Auth 1.7.2 SDK and the current 1.7.16 mount against one disposable PostgreSQL database. Before this change, old-version sign-in returned 401 for the new account. After this change, signup through either writer and sign-in through both readers returned 200. - The integration covers forged client issuer input, the frozen 1.7.2 credential selector, current-version sign-in for existing NULL-issuer rows, and the unchanged provider/account unique index. - `pnpm --filter @paperclipai/server typecheck`: passed, including all server and test-tree layers. - The first required Runner support run failed three SDK proof suites because two generated lock references were stale on the current base. The existing generator corrected those references and the three suites passed locally (58 tests in 17.07 seconds). Master then merged the same repair in #20834 and advanced its runtime/dependency lock. This branch was rebased onto current master; the duplicate proof commit was omitted. The final diff contains only the four auth/test/doc files. The exact-head checks and review listed below passed on the rebased candidate. - Final candidate `577fd7612a6d80ac35e2f5ee9ea2ce9b77169687`: focused auth verification passed again (22 tests, 51.81 seconds). [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43051216288) passed, including build, typecheck, server/Runner tests and all eight browser shards. One unchanged Rust process-lineage orphan-census test failed on the first attempt; the failed job was rerun after the workflow completed and passed. This PR changes no Rust source. - [Greptile review](https://github.com/paperclipai/paperclip/pull/20839#issuecomment-6799551882) is 5/5 on the final head with no actionable findings or open review threads. No deployment or original database change was performed for this branch. ## Risks The bridge applies to new credential accounts. It does not backfill existing NULL issuers or claim old-reader compatibility for unconfigured social providers. Trusted internal issuers retain their exact value. Account lookup, password checks, account linking, provider/account uniqueness, and migration 0465 remain with their existing owners. ## Model Used OpenAI Codex (GPT-6), with tool-assisted code editing and local PostgreSQL qualification. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 10:17:49 -06:00
DottaandClaude Opus 5.5	57f1afdf19	fix(daytona): retry a sandbox create that timed out before any agent work (#20828) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a Daytona sandbox environment, every new task run first creates a fresh sandbox > - When one Daytona create did not start, the plugin waited the full 300 s acquisition budget > - The run then failed `setup_failed` with `lease_acquire_failed`. No agent work had started, but nothing retried it > - On staging this hit two agents in 15 minutes. Each attempt used about 5 minutes, and the task stopped until a person pressed Retry. The manual Retry then got a sandbox in 28 s > - This pull request ends a stuck create early, adopts a create that started late, and routes the timeout to the bounded bootstrap retry that already exists for transient provider refusals > - The benefit is that a stuck create costs about 90 s and recovers by itself. The task shows Retry only after the bounded attempts are used up ## Linked Issues or Issue Description Refs #20626 (bounded resume of a reusable sandbox), Refs #20212 (wait out a Daytona state change), Refs #20609 (force a stuck stop). **What happened?** A run on a Daytona sandbox environment failed after 307 s with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending`. The run recorded `errorCode: setup_failed`, `failureCause: lease_acquire_failed`, and `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The task became blocked with Retry. No automatic retry was scheduled. **Steps to reproduce** 1. Use a Daytona sandbox environment with the default `timeoutMs` (300,000). 2. Start a run while Daytona keeps the new sandbox in `creating`. 3. See that the run waits 300 s, fails `setup_failed`, and is not retried. **Expected behavior** A create that has not started before provider work must not cost 5 minutes. The run must retry by itself within bounds. A late sandbox that has started must be used. A sandbox that was abandoned must be removed. **Root cause** - `packages/plugins/sandbox-providers/daytona/src/plugin.ts` (`onEnvironmentAcquireLease`): the only bound on the create step was the whole acquisition budget (`timeoutMs`, default 300 s). - `server/src/services/environment-provider-unavailable-error.ts` (`readTransientEnvironmentProviderFailure`): only an HTTP status or errno on the create call counted as transient. The timeout envelope carries neither, so the run fell through to generic `setup_failed`, which `isTransientWorkspacePreparationCode` does not retry. ## What Changed - Daytona: one fresh acquisition is now capped at 180 s. A create that has not returned after 90 s is looked up once by its creation name. A sandbox in `started` is adopted, and setup continues. A sandbox in `pending_build`, `building_snapshot`, or `pulling_snapshot` keeps waiting inside the cap. Any other state ends the attempt with the existing cleanup envelope and an acquisition diagnostic whose elapsed time reached the expired budget. - Daytona: a late SDK failure no longer runs the compensating delete for a sandbox that was adopted. - Host: `readTransientEnvironmentProviderFailure` reads the validated acquisition diagnostic. An acquisition whose first failure was its own deadline is transient. The run fails with `environment_provider_unavailable`, category `timeout`, and uses the bounded bootstrap retry (30 s, then 60 s). An earlier setup failure stays `setup_failed`. Message text is never parsed. - Host: the task reason for this case is plain: "The sandbox for ... did not finish starting in time. No agent work had started. Paperclip retries automatically ...". - Docs: `doc/execution-semantics.md` and `doc/plugins/SANDBOX_PROVIDER_CAPABILITIES.md`. ## Verification Red before the fix, green after: - `server/src/services/heartbeat/run-completion.test.ts` "routes a fresh sandbox create that timed out through bounded bootstrap recovery" uses the staging error shape through the orchestrator. Red: `errorCode` was `setup_failed`. Green: `environment_provider_unavailable`, category `timeout`, bounded retry scheduled. The `started` and `early_failure` cases stay `setup_failed` with no retry. - `server/src/services/environment-provider-unavailable-error.test.ts`: deadline diagnostic is transient (red, then green). An early failure is not. The task message is plain. - `packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`: "ends a create still pending at 90 seconds" (red: still pending at 90 s; green) and "adopts a create that started by 90 seconds" (red; green, and a late SDK rejection does not delete it). "keeps waiting for a create that is still preparing its snapshot" is a safety case. - Existing tests that encoded the 300 s acquisition budget now use the 180 s cap. Commands run locally: - `npx vitest run src/plugin.test.ts` (Daytona): 414 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/services/environment-provider-unavailable-error.test.ts src/__tests__/environment-acquisition-diagnostics.test.ts src/__tests__/task-interruption-query.test.ts`: 229 tests passed. The run-completion file reports an `afterAll` embedded-Postgres teardown timeout. The same timeout occurs on unchanged `origin/master`, so it is a local platform fault. - Daytona plugin `pnpm typecheck` and server `pnpm typecheck`: pass. - `pnpm verify`: asked for broader CI coverage, because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. Upgrade safety: nothing new is persisted. A task already blocked by this failure keeps `canRetry: true`. Its Retry starts a new run with a new creation record, and that run uses the new path. On staging, the manual Retry of the affected task got a fresh lease in 28 s. The creation record of the earlier attempt stays with the creation reconciler. Not verified: a live Daytona create that stays in `creating`. That cannot be forced on demand. ## Risks - A legitimately slow create that stays in `creating` or `starting` for more than 90 s now ends the attempt and retries. Snapshot preparation states still wait inside the 180 s cap. Staging creates normally take about 20 to 30 s. - An abandoned create is removed by the creation reconciler. It can stay alive until that reconciler settles the record (the existing settle window). This PR does not change that window. - Other sandbox providers get the host classification only when they send the same acquisition diagnostic. ## Model Used Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code with tool use (shell, file edits, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 10:04:17 -06:00
DottaandClaude	889b33da72	fix(native): apply a saved accepted result on Retry instead of a new turn (#20819) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox can finish with an accepted result, and then fail to save its workspace. Saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - #20758 made a save that finds the sandbox disk full stop at once as `sandbox_disk_full` and hand the task to the board with a Retry. > - When the user frees space and presses Retry, the save now succeeds. But the Retry then started a new agent turn (`strategy: continue`). That turn repeats work that already finished and spends provider budget again. The accepted result was not applied (`replacementDenied: explicit_user_continuation`). > - The decision for this case: when a durable record proves that the run's result was already accepted, Retry applies that saved result and starts no new turn. > - This pull request makes Retry admission wait for the run's saved-work holds and then applies the accepted result through the existing preserved-result finalization. > - The benefit is that a Retry after a failed save finishes the task from the work that was already done, with no repeated provider work or spend. ## Linked Issues or Issue Description Refs #20758 (full-disk save rests as `sandbox_disk_full` with a Retry), Refs #20810 (chaos fault DF1 for a full disk on the final workspace save) **What happened?** A native run had an accepted, succeeded result. Its workspace save failed because the sandbox disk was full, and the task rested at `sandbox_disk_full` with a Retry. The user freed space and pressed Retry. The save succeeded, and then Retry admission started a new run with `strategy: continue`. The accepted result stayed unapplied (the finalization coordinator stayed in `terminal_failure`). Two paths caused this: 1. Retry admission let the Retry through as soon as the saved files were verified. That can happen before preserved-result finalization runs: the recovery worker dispatches the Retry right after one hold settles, while the hold that Retry added for the same run is still scheduled. On master this is deterministic in that order, and a race otherwise. 2. Preserved-result finalization reads the newest saved-work hold of the run. While that hold was still scheduled, it declined, and nothing held the Retry back. **Expected behavior** After the save succeeds, and a durable record proves that the run's result was already accepted, Retry applies that result. The run and task finish from it. No new `turn.start` and no new provider spend. If the save fails again with a full disk, the task keeps the `sandbox_disk_full` rest and a working Retry. If there is no proven accepted result, Retry keeps the previous behaviour and starts one run. **Steps to reproduce** 1. Give a native run an accepted, succeeded result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make the saved-work recovery save fail with `remote_workspace_sandbox_disk_full`. The hold goes to the board. 3. Free space. Press Retry. Let recovery save the files, then dispatch the Retry before preserved-result finalization runs. 4. On master, a new run starts with `previousRunId` set to the source run, and the accepted result is not applied. ## What Changed - `preserved-result-finalization.ts`: the eligibility checks move into one read (`readPreservedResult`). They are unchanged: an accepted, succeeded result of this run, a completed saved-work recovery with a verified receipt, no Stop, no newer run, no current execution, and no applied generation. A new read-only `canResumePreservedResultFinalization` uses the same read without locks. `resumePreservedResultFinalization` keeps the finalizer's lock order (coordinator first) and takes the issue and run rows with NOWAIT, so it cannot deadlock with the finalizer, owner-stop confirmation or Retry dispatch; a busy row is retried briefly, and the last attempt waits for its locks. It also writes an activity entry, `issue.saved_result_finalization_resumed`, when it applies a result. - `explicit-native-continuation.ts`: Retry admission holds an accepted Retry (`evaluate`) of a run whose saved files are verified and whose accepted result is still held, with the wait `finalizing_saved_result`. It holds while a saved-work hold of that run is still `scheduled`, `running` or `awaiting_evidence` and the result is otherwise applicable (no Stop, newer run, other owner or current execution), or once `canResumePreservedResultFinalization` is true. In every other case admission is unchanged. - `heartbeat/run-control.ts` (`resumeTaskRetry`): after dispatch, if admission held the Retry with `finalizing_saved_result`, it calls `resumePreservedResultFinalization`. That coalesces the Retry into the source run with `strategy: finalize_result` and re-arms finalization. This is one key read after admission; the resumption runs only for a held Retry. The recovery worker already calls the same function after each saved hold, so either path finishes the work. - Tests: `workspace-recovery-disk-full.test.ts` now proves the new Retry behaviour (red on master), a dispatch that lands while another hold of the run is still saving (red on master), a Retry while the disk is still full, and three negative cases that keep one new run. `workspace-recovery-preparation-gates.test.ts` (s12 and RC-50b) and `explicit-native-continuation.test.ts` (copy-back + `task_retry`) encoded the old "new run" result for the same situation. They now expect the saved result to be applied. A fixture helper `recoverRunHolds` runs the recovery steps for every hold of a run, as the worker does. - `ui/src/components/WorkspaceExportRecovery.tsx`: at the `sandbox_disk_full` rest, the retained-file panel no longer shows its own "Recover saved files" button and no longer hides the task's Retry. Before, the API said `canRetry: true` and the note said "Retry saves the files again", but the board showed only "Recover saved files" (seen in the DF1 chaos wave). The panel now says to free space and use Retry. A new UI test covers it. `pnpm check:token-gates` is clean. - `doc/execution-semantics.md`: documents the Retry behaviour and its conditions. ## Verification Red/green (`cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts`): - On master source with the new tests: 3 failed, 3 passed (the 3 negative cases pass on master, as they should). Each failure shows a second heartbeat run (the new agent turn): `expected [ …(2) ] to deeply equal [ Array(1) ]`. - On this branch: 7 passed. A fourth negative case (an operator Stop while a hold is still settling keeps one new run) was added after review; it also passes on master. - Guard removal 1: with the `canResumePreservedResultFinalization` check replaced by `true`, the negative test "an operator Stop recorded on the run" fails (`expected [] to have a length of 1 but got +0`: the Retry waits forever instead of starting its one run). - Guard removal 2: with the "hold still saving" check removed, the test "holds a Retry dispatched while another saved-work hold of the run is still settling" fails (a second run starts). Related suites pass on this branch: `workspace-recovery-disk-full`, `workspace-recovery-preparation-gates`, `workspace-recovery-admission`, `native-exhausted-finalization-retry`, `explicit-native-continuation` (641 tests). On the first revision, `task-retry`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `preserved-native-response-wait.integration`, `deferred-execution-wait` and `deferred-execution-receipt-work` passed (317 tests). Other retry and recovery suites were not run locally; required CI `general_tests` covers them. After the review fixes, the five related suites above passed again (642 tests). `npx tsc --noEmit -p server/tsconfig.json` is clean. `pnpm verify --explain` selects a broad plan (exit 2) because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Behaviour change: an accepted Retry of a terminal run with an accepted, succeeded result and verified saved files now applies that result and starts no run. Before, it started one new run. This is the intended change. It also covers the s12 and RC-50b recovery-handoff cases, which used to start a new run. - A held Retry waits while any saved-work hold of the run is still saving. The recovery worker drives those holds to `complete` or `exhausted`. When a hold ends `exhausted` or the records do not prove the result, the wait ends and the Retry starts one run, as before. A hold that stays in `awaiting_evidence` keeps the Retry waiting with "Paperclip is finishing the saved result." - Upgrade-safe: no persisted shape changes. The checks read existing records (`native_run_results`, `native_run_finalizations`, the recovery action evidence, workspace receipts). A record written by an older server that lacks any required fact keeps the old behaviour. An older server that reads the new state sees a coalesced Retry and a re-armed finalization, which it already handles. - No new work before `turn.start` on the normal path. Admission adds the eligibility read only when it has already proven an accepted result and verified saved files for a Retry of that run. `resumeTaskRetry` adds one key read after admission. - Lock order: `resumePreservedResultFinalization` keeps the finalizer's order (coordinator, then issue, then run). Because it can now run beside Retry dispatch (which locks the issue first), it takes the issue and run rows with NOWAIT. A busy row ends the attempt, and it retries up to 9 times at 100 ms. The 10th attempt waits for its locks, as this resumption always did, so a recovered result is never left unapplied. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 10:03:53 -06:00
Nicky LeachandClaude Opus 5.5	369e238295	fix(projects): make workspace storage the project's only placement choice (#20835) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A repository project gets its working files from core staging or from a workspace storage plugin > - #20459 reported that an installed storage plugin attached every new repository project, so sandbox agents could not use those projects > - #20587 fixed creation with an **Available in** choice that selects storage > - #20651 then saved an environment on every new project and used it before the agent default to place runs. That solved a problem that did not exist: the project needs storage, not a run environment > - Creation saves the first environment of the chosen group, so a project silently sends every agent's runs to that environment. Project Configuration also shows two controls (Environment and Workspace storage) for one decision > - This pull request removes the project run environment and makes storage the only placement choice a project owns. Storage is fixed at creation > - The benefit is that runs follow the agent again, and project settings show one clear, read-only fact about storage ## Linked Issues or Issue Description Refs #20459. Follow-up to #20587 and #20651. Refs #11644. That issue asks for a project-level run environment override. This pull request removes the override that #20651 added for it, because the override is not necessary to fix #20459. #11644 stays a separate feature request. Related: #18176 (open) repairs placement through the legacy workspace-policy field. This pull request does not restore project placement, so it conflicts with the direction of #18176. ## What Changed - **Remove the project run environment.** Migration `0466` drops `projects.environment_id`. Run placement no longer reads the project. Runs use the agent default, then the instance default, then local execution. The low-trust task sandbox and the enforced sandbox policies keep their precedence. - **Keep the #20651 recovery and binding guards.** A recovered run still uses its admitted environment lease (source `run`). Company-binding checks before admission and during lease creation stay. - **Keep the creation choice.** **Available in** in New Project still selects storage. It still shows only when the environments need more than one storage owner. That can only occur when a storage plugin is installed. `POST /projects` with `environmentId` selects the storage of that environment and does not save the environment. - **Fix storage at creation.** `PATCH /projects/:id` rejects any storage change with `409 project_storage_locked`, from core to a plugin and from a plugin to core. Before, a core project could move to a plugin. - **Drop legacy writes.** Older clients can still send `executionWorkspacePolicy.environmentId`. The service removes it on create and update. - **Simplify Project Configuration.** Remove the Environment and Workspace storage selectors. Show an advisory **Available in** list of the environments that can use the project's storage. Show it when a storage plugin is installed or when the project uses one. When the project's plugin is not installed, show **Storage plugin unavailable**. Without a storage plugin, show nothing. - **Issue workspace card.** Remove the "project default" environment label. - **Shared contract.** Remove `Project.environmentId`, the `environmentId` project field, and `expectedStorageProviderPluginId` from the update schema. The create schema keeps `environmentId` and `expectedStorageProviderPluginId` from #20587. - **Docs and maps.** Update `doc/project-environment-compatibility.md`, `docs/api/goals-and-projects.md`, and the project feature and agent-access maps. - **SDK source pins.** Refresh `tests/runner-e2e/source-sdk-public-inputs.json` for the two shared files. The shared file blobs return to their values before #20651. The refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`: `master` pins an older lockfile blob than the one it contains. ## Verification - Targeted server suites pass (227 tests): `project-environments`, `project-workspace-provider.integration`, `heartbeat/environment-selection`, `execution-workspace-policy`, `environment-selection-route-guards`, `environment-run-orchestrator`, `openapi-routes`. - Related server suites pass (349 tests): `environment-service`, `environment-routes`, `plugin-workspaces.integration`, `issues-service`. - Targeted UI suites pass: `ProjectProperties.concurrency`, `IssueWorkspaceCard`, `NewProjectDialog`, `NewProjectDialog.managed-sandbox`, `ProjectDetail`. - New tests: - Creation saves storage, not an environment. Core and plugin projects both reject a storage change with `project_storage_locked`. - A legacy policy `environmentId` is dropped on create and update. - A new run follows the agent default for any project. A recovered run keeps its admitted lease after the agent default changes. - Project Configuration has no storage or environment selector. The note is hidden without a plugin. It lists the core or plugin environments for the project's storage. It shows **Storage plugin unavailable** for a removed plugin. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared typecheck` pass. - `pnpm --filter @paperclipai/db check:migrations`, `pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass. - Not run locally: the full `pnpm test:run`, `pnpm build`, and browser suites. CI supplies that coverage. - Manual check: open Project Configuration on an instance with a storage plugin. Confirm that there is no Environment or Workspace storage selector, and that **Available in** lists the environments. ## Risks - **Migration.** `0466` drops `projects.environment_id` and its foreign key. The migration deletes the saved project environments. Apply it before you serve the new revision. An old server revision that reads `projects` fails until it stops. This is the same pattern as earlier drop-column migrations (`0229`, `0289`). - **Behavior change for existing projects.** Projects created after #20651 shipped have a saved environment that overrides the agent. After this change, their runs follow the agent default again. This is the intended fix. Before #20651, a project environment selection was not applied to runs (#11644), so no older behavior is lost. - **No path from core to a plugin.** A core project, including one created before a plugin was installed, cannot move to the plugin. To use plugin storage, create a new project. - **API.** A `PATCH` that sends `environmentId` or `expectedStorageProviderPluginId` no longer has an effect. The schema strips these fields. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with tool use (shell, file edits, test runs). Extended thinking was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 08:00:21 -08:00
DottaandClaude Fable 5.2	39442400b7	fix(native): let a saved governed wait settle on its own same-run resume (#20838) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can save a governed wait (a question or confirmation for a person) and park the provider turn > - If shutdown of the retained execution cannot be confirmed in time, the run keeps its checkpoint, records `native_governed_wait_settlement_pending`, and schedules a same-run resume that settles the saved wait (#20336, #20494, #20519) > - That failure writes an active recovery action for the run. Its cause is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes that same run > - When the run is a resolved-interaction continuation, its dispatch gate checks execution blockers. The gate read the run's own resume action as a hold and cancelled the resume with `execution_reconciliation_required` > - The task then stayed blocked with no Retry and no continue control, and the saved question could not wake the agent > - This pull request makes the gate skip only the run's own scheduled-resume action > - The benefit is that a saved governed wait settles by itself, with no person involved ## Linked Issues or Issue Description Refs #20494, #20519, #20809 **What happened?** A staging Codex run on a remote sandbox was a continuation after an answered question. The agent saved a second governed wait (a confirmation). Shutdown confirmation of the retained execution was still pending, so the run recorded `native_governed_wait_settlement_pending` and scheduled its exact-checkpoint resume. The resume reattached to the remote runner and started. 32 s later it ended `cancelled` with `execution_reconciliation_required`. No person cancelled it. The lifecycle event of the cancel carried the run's own recovery action id. The task showed `blocked` with `canRetry: false` and `canContinue: false`, and no later run started. **Expected behavior** The same-run resume settles the saved governed wait. The task waits for the person's answer. No run is cancelled and no recovery action needs a person. **Steps to reproduce** 1. A native run starts as a resolved-interaction continuation (context `mutation: interaction`, `interactionStatus: answered`). 2. The run saves a governed wait, and shutdown confirmation does not arrive in time (`native_governed_wait_settlement_pending`). 3. The coordinator schedules the same-run resume. At dispatch, `decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns the run's own active action, and `cancelStaleRunInTx` cancels the run. ## What Changed - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` takes `resumingNativeRunId`. For that run it skips only the active action whose wake policy is `resume_native_run` for the same run and whose evidence names that run. Workspace-restore holds and all other actions still block. - `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch staleness decision passes the native run id. - `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the failing case and two controls (an action for another run, an action without a resume policy still cancel). - `tests/restart-chaos`: new opt-in workload Q4 (a second saved question asked by the first answer's continuation run). Provider-fault rules take `occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait. Codex shim and ACPX tap both honor it. README documents Q4. - `doc/execution-semantics.md`: states that the run's own scheduled resume is not a hold against it. ## Verification - Red on master: `postgres.test.ts -t "saved governed wait"` fails with `{ dispatched: false, status: "cancelled", errorCode: "execution_reconciliation_required" }` (the staging error code). - Green: the same test passes; the two controls still cancel. - Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`, `native-preclaim-continuation.test.ts`, `workspace-recovery-preparation-gates.test.ts`; server typecheck. - Harness: `question-handoff.test.ts`, `provider-faults.test.ts`, `acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck. - Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed 111` on master and on this branch (results in a comment below). ## Risks - Low. The exemption is exact: same run id in the wake policy and the evidence, status `active`, only for a native run being dispatched. A recovery action that names another run, has no resume policy, or is resolved keeps its current effect. - Saved state from older versions: actions already written have the same shape, so a resume that is still scheduled settles after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 09:59:22 -06:00
DottaandClaude Fable 5.2	1d52436bf8	fix(config): detect the tailnet bind host only when the bind reads it, one time per process (#20843) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server reads its configuration through `loadConfig()` in `server/src/config.ts`. Runtime code calls it often: `getStorageService()` calls it on each use, and the native runtime calls it for backup storage, response delivery and attachment reuse > - `loadConfig()` called `detectTailnetBindHost()` on each call. That function runs `tailscale ip -4` with `execFileSync`, which is a synchronous child process. The event loop stops until the child process ends > - A CPU profile of a busy hosted server showed this cost on the main thread, on a server that does not bind to a tailnet > - The tailnet address is an input only to the bind host, and only the listener that the server binds at startup uses the bind host. Most bind configurations never read the address > - This pull request makes `loadConfig()` detect the address only when the bind configuration reads it, and keeps the result for the life of the process > - The benefit is that a `loadConfig()` call no longer starts a child process on runtime paths, so the event loop stays free for requests ## Linked Issues or Issue Description No public issue exists. I searched open and closed pull requests for `tailscale loadConfig` and `detectTailnetBindHost` and found no related work. **What happened?** Each `loadConfig()` call started `tailscale ip -4` as a synchronous child process. A 30 second CPU profile of the main thread of a hosted server (commit 556b249897, Node 24, main thread at 96% of one core) showed `detectTailnetBindHost` ← `loadConfig` at 3.1% inclusive and the native child-process spawn at 3.6% self. The callers in the profile were `getStorageService`, `durableNativeHarnessStorage`, `restoreDurableHarnessForExecution` and `createRunnerdBackendWithinSessionClaim`. The server process was about 4.4 GB resident, which makes each spawn more expensive. The cost is the same when the `tailscale` command is not installed, because the spawn still runs. **Expected behavior** `loadConfig()` starts `tailscale` only when the bind configuration reads a tailnet address: `server.bind=tailnet`, or a host without a bind mode that is not loopback and not all interfaces. It starts it at most one time per process. A change to `PAPERCLIP_TAILNET_BIND_HOST`, to another environment variable or to the config file is still visible to the next `loadConfig()` call. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-config-load.ts --calls=200` on the parent commit. The benchmark puts a counting `tailscale` stand-in first on `PATH`, so it does not need Tailscale. 2. Read the `tailscale execs` column. Each scenario shows 200 execs for 200 calls. **Paperclip version or commit** `master` at e2cf9a670f. The profile is from 556b249897. **Deployment mode** All modes. The profile is from an `authenticated` hosted server. The benchmark covers `bind=lan`, the default loopback bind and `bind=tailnet`. **Node.js version** Profile: Node 24. Local benchmark: Node 26.4.0 on darwin arm64. ## What Changed - `server/src/config.ts`: `loadConfig()` calls `detectTailnetBindHost()` only when the bind configuration reads the address. With a configured bind mode, that is `tailnet` only. Without one, it is a host that `inferBindModeFromHost` cannot resolve to loopback or all interfaces without the address. - `server/src/config.ts`: the result of `tailscale ip -4` is kept for the life of the process, also when the command fails. `PAPERCLIP_TAILNET_BIND_HOST` is not kept: each call reads it first, as before. - `server/src/__tests__/config-tailnet-bind-detection.test.ts` (new): regression tests that count `tailscale` starts. - `server/scripts/benchmark-config-load.ts` (new): the reproduction and benchmark. - `doc/DEVELOPING.md`: removed the advice to set `PAPERCLIP_TAILNET_BIND_HOST` for local fixtures to prevent repeated discovery. The server no longer repeats discovery. - `cli/src/config/server-bind.ts` has its own copy of the detection. It is not changed: it detects only for `bind=tailnet` and runs in short CLI commands, not on a server runtime path. ## Verification Regression test, red then green: - `cd server && pnpm exec vitest run src/__tests__/config-tailnet-bind-detection.test.ts` - Parent commit: 11 of 11 tests fail. Examples: `expected 1 to be +0` for `bind=lan` (the command ran when it must not), `expected 3 to be 1` for `bind=tailnet` (the command ran on each call). - This branch: 11 of 11 tests pass. - The tests assert the number of `tailscale` starts and the resolved bind host. They do not assert time. They cover `bind=loopback`, `lan`, `custom`, loopback and all-interfaces hosts without a bind mode, repeated `getStorageService()`, `bind=tailnet`, bind inference from a host, `PAPERCLIP_TAILNET_BIND_HOST` (used without the command, each change visible), a bind mode change in the environment and in the config file, and the unchanged refusal when no tailnet address exists. Benchmark, same machine, 200 calls per scenario, 3 runs each (ranges shown): | Scenario | Before: `tailscale` execs | Before: ms per call | After: `tailscale` execs | After: ms per call | | --- | --- | --- | --- | --- | | `loadConfig()`, `bind=lan` | 200 | 4.5 to 6.0 | 0 | 0.063 to 0.069 | | `getStorageService()`, `bind=lan` | 200 | 4.0 to 4.5 | 0 | 0.043 to 0.060 | | `loadConfig()`, default loopback | 200 | 4.0 to 4.5 | 0 | 0.046 to 0.053 | | `loadConfig()`, `bind=tailnet` | 200 | 3.8 to 5.1 | 1 | 0.74 to 0.92 | - Command: `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-config-load.ts --calls=200` - Each call is synchronous, so the time per call is time that the event loop is blocked. For 200 calls the block went from about 750 to 1200 ms to about 9 to 14 ms where the bind does not read a tailnet address. - The `bind=tailnet` row after the fix contains the one detection (about 150 ms for the first start of the new stand-in file on macOS). The other 199 calls start no process. - `loadConfig()` also reads and validates the config file on each call. After the fix a full call costs about 0.05 ms locally, so the file read is not a material cost and this pull request does not change it. A hosted server reads a stored snapshot from memory, not a file. Other checks: - `cd server && pnpm exec vitest run src/__tests__/managed-runtime-public-url.test.ts src/__tests__/native-harness-backup-expiration-config.test.ts src/__tests__/chat-webhook-public-url.test.ts src/__tests__/managed-cloud-prerequisites.test.ts src/__tests__/managed-loopback-auth.test.ts src/__tests__/cloud-runtime-config.test.ts`: 84 tests pass. - `tsc --noEmit` over the three changed TypeScript files: clean. - `pnpm verify` exits 2 (broader verification required, because of the new file in `server/scripts`). Required CI supplies that coverage. - Not verified: the benchmark on Linux with a multi-gigabyte process, and the hosted server after deploy. The local numbers come from a 200 MB process, where a spawn is cheaper than on the profiled server. ## Risks - Low risk. The resolved `bind`, `host` and `customBindHost` are the same as before for each configuration. A configured bind mode other than `tailnet` never used the detected address, and a loopback or all-interfaces host resolved before the address was read. - Behavior shift: a server process detects the Tailscale address one time. If Tailscale starts, stops or changes its address later, `loadConfig()` in that process still returns the first result until restart. The server binds its listener one time at startup, so the listener already had this property. From the code (not reproduced): before this change, a `bind=tailnet` server whose Tailscale stopped after startup threw from each later `loadConfig()` call on runtime paths. It now continues with the address from startup. - A failed detection is also kept. With `bind=tailnet` the server refuses to start, as before, and the next start detects again. `PAPERCLIP_TAILNET_BIND_HOST` overrides a kept result at once. ## Model Used - Claude Fable 5.2 (Anthropic, model ID `claude-fable-5-2`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 09:59:09 -06:00
DottaandClaude	3914d0c94a	test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`, run through the two-slot queue): - `w-20261111T131314-s9411` (B1): the hook was armed in the server but no save reached it. B1 uses a local worktree synced into the sandbox, not a remote workspace. Graded `not_applied` as designed. The harness now requires `--storage s3` and the docs point at R1. - `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked` with `sandbox_disk_full`, `user_action_required`, board owner, API `canRetry: true`. t02 was cleared too early, while recovery still checked its source; that recovery then saved and the task finished. Fixed in 626f552e (wait for the disk-full verdict). - `w-20261111T134316-s9413` (R1, after the fix): 2/2 tasks rested with `sandbox_disk_full` and API `canRetry: true`. In both, the board showed **"Recover saved files"** and no Retry button, so the UI Retry click failed (`ui_retry_button_missing`). Retry-worked is not shown. The wave was interrupted by SIGTERM during the post-rest file reads, before final grading. - Local wave: not run. A local run has no sandbox workspace save, so DF1 cannot fire there; the harness refuses `--env local`. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona --storage s3` and a remote workspace (R1). A local run has no sandbox workspace save, so the harness refuses DF1 there. DF1 is refused in `--matrix` waves, and T1's restart check is skipped for DF1. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 09:57:59 -06:00
DottaandClaude Opus 5.5	0b8ba1ac92	fix(queue): show messages a booting run's first turn takes as sent (#20822) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task chat lets a person send follow-up messages to a running agent, and steer them into the live turn > - A sandboxed run (for example on Daytona) can take minutes to boot before its provider session exists > - A message sent during that boot cannot be steered, so it waits in the steering pane > - The runner then puts the message into its first turn, and the server removes it from the queue, but the view has no signal for this, so the message stays in the steering pane as "queued" > - That stale local entry also hides the queue id, so Steer and Discard fail for every later message > - This pull request tells the view which messages the first turn takes, and drops local queue entries whose run has settled > - The benefit is that a message sent while the sandbox boots shows as sent at once, and later messages keep working Steer and Discard ## Linked Issues or Issue Description **What happened?** A person sent a follow-up while the agent's sandbox was booting. The message stayed in the steering pane with Steer unavailable. After the agent started, the message stayed there. The agent had received it in its first turn. Later, Discard on that message failed with "Couldn't discard. Message is still queued.", and a new message also stayed in the steering pane. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent that runs in a Daytona sandbox. 2. While the sandbox boots, send a follow-up message. 3. Wait for the agent's first response. The message is still in the steering pane. 4. After the run hands off to a successor run, send another message and try to discard the first one. **Expected behavior** A message sent while the run is starting shows as a sent message. Later messages keep working Steer and Discard. **Actual behavior** The first message stays in the steering pane. Discard fails, and the new message cannot be steered until the page reloads. ## What Changed - `firstTurnQueuedCommentIds` (server, `first-turn-queued-input.ts`) returns the messages in a deferred queue that the target run's first turn already sealed, or will take while it is still starting. It uses the same eligibility rule as `claimFirstTurnQueuedInput`. It reads only lean columns, so the queue poll does not transfer run snapshots or large wake payloads. - The queue read path and the queue mutation path both put these ids in a new optional `firstTurnCommentIds` field on `IssueQueuedCommentQueue`. - The task view puts each of those messages in the conversation as input of that run, at its submission time, and not in the steering pane. - The task view no longer treats a saved message as a pending local queue entry after its target run is no longer live. A request that is still in flight keeps its original target. ## Verification - Red-green server test: `issue-queued-comments-routes.test.ts` "reports the messages a preparing/native run's first turn takes, before and after it seals them" and "reports no first-turn messages once the run's turn already started". Before the fix, `firstTurnCommentIds` was `undefined`. - Red-green UI tests: `IssueDetail.test.tsx` "shows a message the booting run's first turn takes as sent, not queued" (still starting, and sealed) and "keeps queue controls for a new message after the run that took an earlier one hands off". Before the fix, the queue held `['new-message', 'boot-message']` and the queue id was withheld. - Reproduced on staging first: the run's context held a first-turn receipt for the stuck message, and the server queue had no entries, while the view still showed it as queued. - `npx vitest run` for `issue-queued-comments-routes`, `queued-comment-consumption`, `queued-comment-postgres`, `queued-comment-use-cases`: 172 passed. The existing queue-poll byte budget tests still pass. - `npx vitest run src/pages/IssueDetail.test.tsx src/components/task-chat src/lib/issue-queued-comment-queue src/lib/optimistic-issue-comments`: 1288 passed. - `tsc --noEmit` for shared, server, and ui: no new errors. - `pnpm verify` exits 2 because the change touches `packages/shared`. Required CI supplies the broad coverage. ## Risks - Low risk. The new field is optional and only for display. Delivery still belongs to the first-turn claim and the deferred wake. - If a starting run fails before its first turn, the field goes away and the message shows in the queue again for the next run. This is correct, because the deferred wake still holds it. - The queue read adds one lean run read and one comment read only while a deferred queue targets a running run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 15:52:14 +00:00
DottaandClaude Fable 5.2	a4b872477e	fix(chat): do not wake the assignee for a decision its own run recorded (#20842) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task can be bound to a chat conversation (Slack, Telegram). Confirmation cards are published there > - The assignee's running turn can record the user's chat reply as the decision on a card (resolve-from-comment). That run already owns the decision, and the in-app route does not wake the agent again > - The chat continuation worker (`interaction_wakeup`) still woke the assignee for that decision > - The wake waited behind the running turn. When it was promoted, the reply was already delivered, so its context kept `source: chat:<provider>` but no chat message > - Reviewed-chat attestation then refused the run (`reviewed_chat_execution_binding_not_authorized`, `context_mismatch`). The refusal is correct; the wake should not exist > - This pull request makes the worker settle that action without a wake > - The benefit is no failed setup run on the task, and no change to any permission check ## Linked Issues or Issue Description Refs #20809 **What happened?** On staging, an automation-triggered run failed in about 5 s with `setup_failed` and error `reviewed_chat_execution_binding_not_authorized` (denial reason `context_mismatch`). The task was bound to a Slack thread and was `in_review`. The assignee's previous run had recorded the user's Slack reply as acceptance of a plan confirmation and kept working. The chat continuation worker then queued a second wake for the same decision. That wake carried no undelivered chat message, so attestation found zero admitted comment ids. **Expected behavior** No second wake. The responding run already acts on the decision, as in the in-app path. **Steps to reproduce** 1. Bind a task to a Slack conversation and publish a `request_confirmation` card with `continuationPolicy: wake_assignee`. 2. While the assignee's run is active, the user replies in Slack. The run calls `POST /api/issues/:id/interactions/:interactionId/resolve-from-comment`. 3. The worker processes the `interaction_wakeup` chat action and wakes the assignee. The wake is deferred behind the active run and loses the delivered reply. With the task `in_review`, the next run fails attestation. ## What Changed - `server/src/services/chat-channels/interactions/continuations.ts`: when the current interaction was resolved by the assignee's own run from a comment (`resolvedByAgentId` is the assignee, `resolvedByRunId` is set, no `resolvedByUserId`, and `result.commentId` is set; only resolve-from-comment writes that), the worker marks the action `processed` with `interaction_wakeup_owned_by_responding_run` and creates no wake. Actions saved before the deploy take the same path. - `server/src/__tests__/chat-interaction-publications.test.ts`: real resolve-from-comment flow (no wake) and the board-accept control (one wake). - Reviewed-chat attestation and every authorization check are unchanged. ## Verification - Red on master: the new test fails, `wakeup` called 1 time. - Green: `server/src/__tests__/chat-interaction-publications.test.ts` 32/32 pass. - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - Low. Only a self-resolved, comment-recorded decision skips the wake. A user decision through a chat button or the board still wakes the assignee. - Not covered here: a user decision made through a chat button while another run of the task is active can still be deferred and lose its chat comment id. That is a separate case and is not changed by this PR. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>	2026-11-11 09:47:43 -06:00
Nicky LeachandClaude Opus 5.5	ff8ec41e6a	feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 07:28:00 -08:00
DottaandClaude	fd1460bb24	fix(native): continue a governed-wait follow-up in a fresh session when its saved provider state is gone (#20809) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks and breaks the control plane on purpose, so failures show up before staging users see them > - A census of 48 hours of staging failures found runs whose result save-back failed because the sandbox disk was full (class F). The harness never filled a sandbox disk, so it missed that class > - #20758 made a full-disk save end recovery at once with cause `sandbox_disk_full` and a Retry. Nothing in chaos exercised that path end to end > - A real full disk cannot be made on demand. The owner chose a test-only hook that makes the save report a full disk, not a small disk > - This pull request adds the hook at the save's sandbox boundary and a DF1 fault that arms it, clears it, and grades the result > - The benefit is that every wave can now check the disk-full rest, the Retry offer, and that Retry works after space is freed ## Linked Issues or Issue Description Refs #20758 (product behaviour under test), #20678 (finalization hands a full-disk copy-back to recovery) **What is the problem?** The chaos harness had no fault for a full sandbox disk during result save-back. Staging saw this failure (census class F in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos could not reproduce it. **What is the expected behaviour?** A wave can schedule DF1. While DF1 is armed, the task's final workspace saves fail as a full disk. When DF1 is cleared, saves succeed again. The task must rest with `sandbox_disk_full` and a Retry, and that Retry must finish the task. **Why does it matter?** Without a fault, a regression in the disk-full path (no Retry, a wrong cause, or a Retry that does not work) reaches staging users before chaos sees it. ## What Changed - `server/src/services/sandbox-disk-full-fault.ts` (new, test-only): `sandboxDiskFullFaultTarget` wraps the target of a final save only when the harness armed that save. The capture commands then exit 1 with the exact message the in-sandbox script writes on ENOSPC, and the provider download fails with `No space left on device`. The gate follows `@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL` is read once at import and ignored under `NODE_ENV=production`. Disarmed, the hook is one comparison and returns the save's own target. - `server/src/services/remote-workspaces.ts`: the capture call in `saveRemoteWorkspace` passes its target through the hook. No other product change. `workspace-recovery.ts` is not touched. - `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the control-file seam (atomic writes), the rest evidence (`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`), the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry that must finish the task). - Harness wiring: `--faults DF1` (refused without `--env daytona`, with a storm, or with PV faults), no scheduled restart, arm at launch, clear at the first rest just before the Retry click, receipts in `<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: { faultedSaves, restedDiskFull, retryOffered, retryWorked }`. - Docs: a "Sandbox disk full (DF1)" section in `tests/restart-chaos/README.md`, and the census plan marks DF1 as built. ## Verification - `cd server && pnpm exec vitest run src/services/sandbox-disk-full-fault.test.ts`: the gate is off by default and under `NODE_ENV=production`, the save's own target comes back when disarmed, only a listed final save is faulted, the product capture turns it into the named disk-full failure, and clearing the file disarms the next save. - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/disk-full-faults.test.ts tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts tests/restart-chaos/provider-faults.test.ts`: planning, the control file, and grading. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the server `tsc --noEmit`: clean. - Live wave: a Daytona DF1 wave is running. Its result is added here before merge. ## Risks - Low risk for production. The hook reads its variable once at startup and is ignored under `NODE_ENV=production`. Without the variable it does one comparison and returns the same target object. - The hook fails every sandbox command of an armed capture, including the read-only fallback's inventory. A real full disk can let that inventory succeed and fail later at the download. The outcome is the same (`sandbox_disk_full`), but the fallback's inventory step is not exercised under DF1. - DF1 runs only with `--env daytona`. A local run has no sandbox workspace save, so the harness refuses DF1 there. ## Model Used - Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 08:47:06 -06:00
DottaandPaperclip	5f38226290	fix(build): report compiler signals in server typecheck layers (#20603) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its development checks help contributors verify changes. > - The server typecheck runs the compiler once for each source layer. > - A compiler child can stop on a signal before it reports type errors. > - The wrapper returned failure without the signal or the affected layer. > - This pull request reports the signal, layer, command, and working directory. > - Contributors can identify an interrupted check from its output. ## Linked Issues or Issue Description **What happened?** The server typecheck returned exit 1 after a layer banner when a compiler child stopped on a signal. The wrapper did not report that signal. Related layer work: #18737, #18738, and #18739. A search of open and closed PRs found no duplicate signal-reporting fix. **Expected behavior** Report the actual termination signal and affected layer. Keep compiler output visible and return failure. Do not infer a memory failure from SIGKILL alone. **Steps to reproduce** Run a controlled compiler child that sends SIGTERM or SIGKILL to itself. Run it through typecheckLayers and the real runTsc subprocess path. The added tests do this without memory pressure or timed waits. **Paperclip version or commit** Reproduced on 7368a0b15c. **Deployment mode** Source checkout on Linux with Node v25.9.0. ## What Changed - Print a stderr diagnostic before the existing early return for a compiler signal. - Include the layer, signal, compiler command, and working directory. - Add real subprocess regressions for SIGTERM in below and SIGKILL in cycle. - Check that child stdout and stderr remain visible and later layers stop. - Document the diagnostic and the limit of what a signal proves. ## Verification - Red: the two new subprocess tests failed because stderr omitted the signal and layer. - Green: `cd server && pnpm exec vitest run scripts/typecheck-layers.test.mjs` passed all 47 tests. This includes real compiler success, declaration output, and compiler error diagnostics. - `git diff --check` passed. - `pnpm verify` returned exit 2. The wrapper is a broad input, so CI must supply broader coverage. - Preflight installed dependencies and built shared/plugin-sdk. It could not build Rust outputs because cargo is absent. Rust builds are outside the focused local test scope. - Final head: `fdd68d7e13751573c661e80a285be86f1a1420bc`. - [CI passed](https://github.com/paperclipai/paperclip/actions/runs/42867062133). Typecheck, build, test suites, and all eight E2E shards passed. - [Greptile gave 5/5](https://github.com/paperclipai/paperclip/pull/20603#issuecomment-6774642345) for this head. No actionable findings or open review threads remain. ## Risks - Low risk. Only the existing signal branch gains a diagnostic. Failure propagation and compiler output handling stay the same. - The controlled signal tests require POSIX signal behavior. Local verification used Linux. ## Model Used - OpenAI Codex, based on GPT-6, with reasoning, repository tools, and code execution. The exact deployment model ID is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 08:39:15 -06:00
DottaandPaperclip	81c9143fe4	fix: upgrade Better Auth to 1.7.16 with account migration (#18906) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - The board uses Better Auth for credential accounts and sessions. > - The server pins Better Auth 1.7.2. > - The November security release fixes 12 issues in Better Auth. > - The new version stops writing the old required account issuer field. > - This change updates the packages and the database contract together. > - New users can sign up, and existing users can still sign in. ## Linked Issues or Issue Description **What happened?** The server uses Better Auth 1.7.2. The [security release](https://better-auth.com/blog/security-release-2026-11-03) fixes 12 issues in 1.7.16. Earlier update PRs #17909 and #17911 target 1.7.7, which is below this security release. **Expected behavior** Use the patched stable release. Keep new and existing credential accounts usable. **Steps to reproduce** Read the server dependency pin. Upgrade it without a schema change. Better Auth then rejects the required account issuer column. See the [upgrade guide](https://better-auth.com/docs/guides/1-7-upgrade-guide). ## What Changed - Pin Better Auth to 1.7.16. The repository refresh bot owns the committed Core lockfile. - Make the old issuer column nullable. Keep existing issuer values. - Replace the issuer index with a unique provider/account index. - Generate the migration and retain the five newest schema snapshots. - Test new signup, new-account sign-in, legacy issuer sign-in, duplicate rejection, and separate provider identities with real PostgreSQL. - Update the database schema test. Test the historical issuer backfill separately. Verify upgrades over existing rows and rollback on duplicate provider/account keys. - Document the duplicate check and migration order. - Add the approved two-line dependency resolution step before the backup job’s frozen install. Leave the frozen check and backup tests intact. ## Verification - Core head: `af70282c7e24335b8688c8f3cc88fdf65045632c`; parent `0f1481e4462100357a7caa496b875cf0bc500d95` includes master `84fc67317`. - Nine changed files. The only workflow edit is the approved two-line backup dependency resolution step; no managed lockfile edits. - Migration `0465_better_auth_account_identity.sql` is three SQL statements; snapshot changes only `public.account`, five newest snapshots retained. - DB typecheck and 23 focused tests passed on the parent. Both real PostgreSQL migration/signup suites were rerun November 11: all 4 tests passed. Application source and tests are unchanged in this workflow-only commit. - The parent’s sandbox-provider lane passed all 205 file-sync tests but reported an unhandled EPIPE during the symlink rejection case; its aggregate verify therefore failed. No unrelated source change was made. The new head reruns the full applicable CI. - Reproduced the old backup install failure locally. Resolving manifests followed by a frozen install passed locally; restored the bot-owned committed lock afterward. Hosted backup qualification remains required. - Current-head CI and review are running; no final-head green claim or merge yet. ## Risks Apply the migration before the new server starts. Check for duplicate provider/account pairs first. Duplicate rows stop index creation. Keep separate users separate when resolving duplicates. The migration keeps issuer data. Do not restore its NOT NULL constraint while new rows contain null. No deployment is included. ## Model Used OpenAI GPT-6 Codex, with reasoning and tool use. The runtime did not expose a more specific model ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 08:26:58 -06:00
DottaandPaperclip	aaf1337c7b	fix(slack): require mentions only for channel roots (#20821) ... ## Thinking Path > - Paperclip helps people manage AI agents for work. > - Slack connects people to those agents through channels and threads. > - A mention can start a conversation without making the bot answer all channel traffic. > - The current setting also requires mentions on every thread reply. > - This interrupts an existing conversation and can cancel a queued reply. > - This change requires mentions on channel roots and allows replies without a mention. > - People can continue a thread without adding the bot name to each message. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Slack connection setting that requires an at-mention. **Current behavior** When the setting is on, a channel thread reply must mention the bot. The delivery and execution checks reject a reply that has no mention. **Proposed behavior** Require a mention on a top-level channel message. Allow replies in existing threads without a mention. Keep the current DM access controls. **Reason and benefit** A person can continue an existing conversation without repeating the bot name. **Breaking changes** Existing connections with the setting on will accept thread replies without a mention. The saved setting, default value, and API field stay the same. ## What Changed - Rename the control to **Require at-mention in channels**. Explain the thread and DM rules in the help text and operator documentation. - Keep the mention policy in the Slack module. Use the thread root and message IDs to identify a reply. - Call that policy from durable admission, wake authorization, and queued run start. These shared paths must enforce the same setting when a delivery resumes. Other providers keep their current behavior. - Update PostgreSQL integration tests for thread replies, DMs, queued deliveries, and combined wake inputs. Add tests that reject incomplete reply identity. ## Verification - Reproduced rejected thread replies and cancelled queued reply runs before the fix. - Passed 16 focused PostgreSQL integration cases after the fix. - Passed 71 tests on revision `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04` after the rebase. This includes the full durable wakeup suite, Slack policy tests, and eight selected Slack delivery cases. - Command: `pnpm exec vitest run server/src/services/chat-channels/slack/wake-authority.test.ts server/src/__tests__/durable-chat-wakeup.test.ts server/src/__tests__/chat-channels.integration.test.ts -t "at-mentions by default|thread replies without mentions|rechecks queued Slack|preserves running Slack tool authority|Slack channel mention policy|durable inbound chat scheduler receipts"`. - Strengthened the queued-root test to check the mention-rejection reason and content redaction. A temporary change that disabled mention enforcement caused this test to fail with the unrelated-message reason. The real policy is restored. - Removed the added source comments to follow the contribution rules. - `node scripts/check-module-boundaries.mjs` and `git diff --check` pass. - `pnpm verify` returns exit 2 because the operator documentation is a global CI input. Required CI must supply the wider coverage. - Local preflight installed dependencies and built the plugin SDK. It could not build Runner Rust binaries because this environment has no `cargo` executable. The focused server tests run without those binaries. - Live Slack and browser tests were not run. This change uses the existing provider message and thread identities. - Required CI passed on `31b1adede2b92bad24b8fa21b8f6fffa9bd26d04`: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/43035551294), including typecheck, build, server/chat tests, Runner checks, and browser E2E. Current-head Greptile and Paperclip reviews are both 5/5 with no open findings. ## Risks - Existing connections with mention gating on will now accept unmentioned thread replies. This is the intended behavior change. - A missing or invalid thread or message ID does not grant the reply exception. - No schema migration is required. Destination, sender, and company access checks stay in place. - The existing chat integration test file is large. This change updates its current Slack cases to reuse the full delivery and authorization fixture. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 08:26:52 -06:00
DottaandClaude Opus 5.5	fa3cd6f330	fix(native): end provider startup rejections with Retry instead of an ownership hold (#20808) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner starts Codex for native runs. On a remote sandbox such as Daytona, the sandbox is the OS boundary. > - A Codex planning run on Daytona failed in `session.open`. Codex's read-only sandbox helper could not set up loopback in the container (`bwrap: loopback: Failed RTM_NEWADDR`), and Codex loads `AGENTS.md` through that helper. > - The server treated that definite provider rejection as an interrupted transport. Its same-run bootstrap retry went through remote restart recovery and stopped on `ambiguous_remote_recovery_evidence`. The task showed "Recovering saved work…" and no Retry, although no provider work was done. > - This pull request gives such startup rejections a named, permanent outcome with a working Retry, and pins ask and standard runs to the external sandbox policy. > - The benefit is that a provider startup refusal ends as a clear failure with Retry, never as an ownership hold. ## Linked Issues or Issue Description **What happened?** A Codex planning run on a Daytona sandbox failed in `session.open` with `error creating thread: failed to load AGENTS.md instructions ... bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`. The server recorded `native_session_interrupted` with recovery mode `bootstrap_retry`. Thirty seconds later, remote restart recovery blocked the run with `native_restart_recovery_blocked` / `ambiguous_remote_recovery_evidence`. No server restart happened. All default-mode Codex runs on the same image succeeded that day; only the planning run failed. **Expected behavior** A provider startup failure with no provider work must end as a clear failure with a working Retry, or get a bounded automatic retry that also ends with Retry. It must never become an ambiguous-ownership hold. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-startup-rejected-retry.integration.test.ts`. On master the deterministic case is `retryable_failure` / `native_session_interrupted`. 2. `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "provider itself rejected"`. On master the new code is not produced. **Paperclip version or commit** master at 84fc67317e. ## What Changed - Server (`native-session-executor.ts`): new permanent code `native_provider_startup_rejected`. - A definite `session.open` failure maps to it when its startup `providerFailure` is `requires_action` for `authentication`, `permission` or `invalid_request`. - On a remote execution target (`nativeRemoteStartupRejectionCode`), any definite `session.open` failure with a startup `providerFailure` maps to it. A remote same-run bootstrap after the runner launched cannot prove ownership and stops on `ambiguous_remote_recovery_evidence`. - The run ends as `terminal_failure` with a board owner and a plain next action. The board offers Retry, which starts one new run with a fresh provider session. - On a local target, unknown, transient, rate-limit, quota and model startup failures keep the bounded same-run retries and end as `native_session_retry_exhausted` with Retry. That code is unchanged because the chat-channel failed-run retry depends on it. - Runner: planning runs keep Codex's `paperclip-runner-workspace-read-only` profile, also in an external sandbox. Codex 0.160.0's plan collaboration mode is a prompt ("Plan Mode (Conversational)") and does not block writes or commands, so it cannot replace that enforcement. - Ask mode: an ask task builds `executionMode: "default"`. Its launch has `readOnly: false`, the external sandbox profile, and no collaboration mode. Ask mode restricts only Paperclip mutation tools, not the Codex sandbox. - Tests: - `native-startup-rejected-retry.integration.test.ts` (new, embedded Postgres, Codex provider): a deterministic rejection ends at once with `canRetry: true`, and Retry starts exactly one new run. Unknown and transient local failures stay `retryable_failure`. A spent retry budget ends `terminal_failure` with `canRetry: true`. - Unit tests for the deterministic, local-retry and remote rules. - `runner-provider-launch.test.ts`: ask, standard and planning launches in an external sandbox. - The Rust external-sandbox test now fails if a non-planning turn sends a collaboration mode (`--forbid-plan-collaboration` switch on the fake app server). - Docs: `packages/paperclip-runner/docs/codex-driver.md`, `packages/paperclip-runner/docs/durable-recovery.md`, `doc/execution-semantics.md`. ## Verification - Red on master, green here: the commands in "Steps to reproduce". - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1364 pass. - `npx vitest run src/live/runner-provider-launch.test.ts` (packages/paperclip-runner): 22 pass. - `cargo test -p paperclip-runner-core --test codex_provider external_sandbox`: pass. `cargo fmt --check`: clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` returned exit 2 (a global CI input changed). Required CI supplies the broad coverage. ## Risks - Codex planning on Daytona still fails at thread creation. With this change it ends at once with the provider message and Retry instead of an ownership hold. A follow-up must make Codex's read-only sandbox work in that container, or add a verified Paperclip read-only guard. This PR does not relax read-only enforcement. - A remote startup failure that could clear by itself now needs a Retry click instead of an automatic same-run retry. Before this change, that retry ended in an ownership hold, so no working automatic path is lost. - A false `native_provider_startup_rejected` costs automatic retries and shows Retry earlier. It never replays provider work, because `session.open` precedes every prompt. - Saved state: no persisted shape or launch argument changes. A run that an older release held on `ambiguous_remote_recovery_evidence` after the same rejection already offers Retry once its saved workspace is recovered. No migration is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) Opus 5.5, model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit, browser reads) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 07:48:18 -06:00
DottaandClaude Opus 5.5	6dd8611a2f	fix(queue): keep tool action outcomes out of the steering queue (#20582) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents ask a person to approve a governed tool action (for example, create a Notebook page) before the tool runs > - Approve means run: the tool runs as soon as the person approves it, and the approval card then shows Executed, Failed or Declined > - After the tool runs, the server wakes the agent with the recorded result. When another turn is active, that wake waits behind it > - The composer queue lists waiting interaction receipts as messages. The result wake carries the approval's interaction id and status, so the queue shows "Accepted: Approve tool action" with an Interrupt control, after the tool already ran > - People read this row as an approval that is still not done, and Interrupt stops their current turn only to deliver a tool result > - This pull request keeps tool action outcome wakes out of the composer queue and out of its steer, interrupt and discard actions > - The benefit is a steering pane that shows only messages the person can act on, while the agent still receives the tool result after the current turn ## Linked Issues or Issue Description **What happened?** After a person approves a governed tool action, the tool runs and its card shows Succeeded. When the person then sends a new message, the composer queue shows a row "Accepted: Approve tool action" with Interrupt and Discard controls. The row stays until the active turn ends. **What did you expect?** The steering pane does not show the tool result as a queued message. The approval card already shows the outcome. The agent receives the result after the current turn. **Steps to reproduce** 1. Let an agent request approval for a governed tool action (`request_confirmation` with `payload.toolAction`). 2. Approve it. The tool runs and the card shows Succeeded. 3. Send a new message so a new turn starts before the result wake runs. 4. See "Accepted: Approve tool action" in the composer queue. ## What Changed - Add `isToolActionOutcomeReceipt` in `server/src/services/queued-interaction-response.ts`. It identifies an interaction receipt wake that carries a tool action (`toolAction` identity or `toolActionRequestIds`). - `findQueuedCommentWake` in `server/src/routes/issues.ts` skips these wakes, so `GET /issues/:id/queued-comments` does not list them. The payload projection adds only the tool action's `actionRequestId`, `toolName` and `toolActionRequestIds`, never its result text. - The locked by-id queue lookup (steer, interrupt, discard) rejects these wakes with `queued_comment_not_pending`, so a stale client cannot interrupt a turn for a tool result. - Delivery, stranded-queue recovery and the handback hold (`readQueuedInteractionResponse`, `hasQueuedInteractionResponse`) do not change. The agent still receives the result. - Add route tests for accepted and declined tool action outcomes. They check that the queue is empty, the wake stays `deferred_issue_execution`, and the issue still has owed interaction input. ## Verification - `cd server && npx vitest run src/__tests__/issue-queued-comments-routes.test.ts src/services/issue-queued-comment-queue.test.ts`: 137 of 137 tests pass. The new tests fail without the fix (the queue lists the receipt). - On a heavily loaded machine (load average about 25), the embedded Postgres `afterAll` teardown timed out once at 30 s after all tests passed. This is not related to the change. - `cd server && npx tsc --noEmit -p .`: clean. ## Risks - Low risk. The change only filters the user-facing queue view and its by-id actions. - A tool action outcome wake no longer appears as an Interrupt target. Before, Interrupt could deliver a tool result into a new turn early. Now the result arrives when the current turn ends, as for other system deliveries. - #13160 refactors the same steering path. One of the two PRs may need a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser inspection of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 07:39:37 -06:00
DottaandClaude Opus 5.5	4c04c20c9e	fix(recovery): wake a parent whose refused completion waited on dependencies that later clear (#20803) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task that waits on blocker tasks resumes through the `issue_blockers_resolved` wake > - A periodic dependency sweep sends that wake when a blocked task's dependencies become ready and no wake covers the current blocked cycle > - The sweep skips a task that its own agent blocked, because that agent waits on something outside the blocker graph > - A native completion that Paperclip refuses for unresolved dependencies is also recorded as a block by the agent's own run (`durable_dependency_blocker_bound`). The sweep treated it as a self-block. It woke the task only if a blocker later completed or finalized on its workspace > - When readiness cleared another way, nothing woke the task. It stayed `blocked` with no next action > - This pull request makes the sweep treat that refusal as a dependency wait, and count a "Recover saved files" finalize without a workspace as the blocker settling > - The benefit is that a blocked task is not a silent dead end: it resumes once, automatically, when its dependencies clear ## Linked Issues or Issue Description Refs #20728 **What happened?** A parent task had three done child tasks that block it. One child was still listed as "saving its files", because a "Recover saved files" finalize had no workspace. The parent's agent tried to finish. Paperclip refused the completion because dependencies were unresolved and set the parent `blocked` (`durable_dependency_blocker_bound`). After #20728 was deployed, the parent's dependencies were ready. But the task stayed `blocked` with no wake, no Retry, and no next action. A person had to post a comment to wake it. **Expected behavior** When the last blocker of a blocked task clears, by status or by finalize settlement, Paperclip wakes the assignee once. This also applies to tasks already blocked before a deploy that changes readiness. **Steps to reproduce** 1. Block a task on a done blocker whose workspace finalize is still pending. 2. Let the assignee agent's native run report the task done. Paperclip sets the task `blocked` with `durable_dependency_blocker_bound`. 3. Make the blocker settle without a later blocker completion or a finalize on its workspace (for example, a recovery finalize with no workspace, or a deploy that changes the readiness rule). 4. Run the dependency sweep. Before this fix it reports `selfBlockedSkipped: 1` and sends no wake. ## What Changed - `server/src/services/recovery/service.ts` (`wasBlockedByAssigneeAgent`): a native block with reason `durable_dependency_blocker_bound` is not a self-block. The dependency owns its wake. Other native blocks and agent blocks keep the self-block rule. - The same function counts a later succeeded `workspace_finalize` with `owningService: workspace_recovery` and no workspace as the blocker settling, which matches the readiness rule from #20728. - `server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts`: regression tests for the refused-completion shape (one wake, no second wake), the recovery finalize without a workspace, and a safety case for a different native block reason. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: a characterization test that a real "Recover saved files" signals the dependency sweep for the company. This path already existed; the test keeps it. - `doc/execution-semantics.md`: documents when the dependency sweep wakes after the last blocker clears. ## Verification - Red (unfixed `service.ts`): `npx vitest run src/__tests__/heartbeat-issue-liveness-escalation.test.ts -t "refused for unresolved|still holds a native block|recovery finalize without a workspace"` fails 2 of 3 tests with `selfBlockedSkipped` 1 instead of 0. - Green (fixed): `src/__tests__/heartbeat-issue-liveness-escalation.test.ts` passes 44 of 44. - Related suites pass: `src/__tests__/issue-dependency-wakeups-routes.test.ts`, `src/__tests__/heartbeat-dependency-scheduling.test.ts`, `src/services/workspace-recovery-dependency-finalize.test.ts`, `src/__tests__/issue-recovery-dependencies.test.ts` (43 tests). Server `tsc --noEmit` passes. - `pnpm verify` exits 2 because a doc change is a global CI input. Required CI supplies broad coverage. - Staging evidence (read only): the affected parent's activity shows `native-status-committer` set it `blocked` with `durable_dependency_blocker_bound`, and no later blocker completion or finalize. ## Risks - Previous-release story: no persisted shape changes. Tasks already `blocked` with `durable_dependency_blocker_bound` and ready dependencies are picked up by the existing dependency sweep after deploy (it visits every blocked task at startup and on recovery signals). No migration. - Bounded: the wake uses the cycle-aware ready-state key, so one blocked cycle gets one wake. The sweep keeps its live-run, queued-wake, pending-interaction, and pause-hold checks. - No loop: if the woken agent blocks the task again while its dependencies are ready, the reason is not `durable_dependency_blocker_bound`, so the self-block rule applies. - The recovery finalize rule only applies to a finalize on the blocker task itself with `owningService: workspace_recovery` and no workspace. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 07:04:13 -06:00
DottaandPaperclip	556b249897	fix: preserve exact Markdown in native document writes (#20605) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents save Markdown documents through native tools and REST routes. > - The native write_document handler trims the supplied body. > - This removes final newlines and can change Markdown whitespace. > - This pull request preserves the original body after required-content validation. > - Agents can now write and read the same Markdown string across these paths. ## Linked Issues or Issue Description **What happened?** The native write_document tool removes leading and trailing whitespace. A body with one final newline loses that newline on read-back. The REST document PUT and write_task_document preserve it. **Expected behavior** The native write_document tool must preserve the supplied Markdown body exactly. Required-content checks and key and title normalization must stay in place. REST-backed paths retain surrounding whitespace and keep their existing escaped-line-break conversion for single-line payloads. **Steps to reproduce** 1. Write a disposable document through write_document with a final newline. 2. Read the document and compare the full body with the supplied string. 3. Repeat through write_task_document and REST. These paths preserve the string. Related work: Refs #18824. That change aligns document keys. This change preserves body strings. A search of open and closed PRs found no duplicate fix. ## What Changed - Use a separate untrimmed required-string helper for the document body. Keep the existing requiredString callback signature and all other field normalization. - Add real tool-handler, database, and authenticated HTTP tests for create, update, read-back, and replay. Cover leading spaces, hard breaks, blank lines, final newlines, no final newline, Unicode, and mixed line endings. - Keep blank and invalid bodies rejected. Document whitespace preservation and the existing REST escaped-line-break exception. - Fix the Runner support archive-scanner shutdown defect exposed by CI: rejected ZIP scanners now receive SIGKILL so an ignored SIGTERM cannot leave publication waiting for process closure. Keep all secret and archive limits unchanged. A real child-process regression fails before the change and passes afterward. - Address all three initial review findings: callback type compatibility, contribution comment policy, and the REST documentation limit. ## Verification Final tested head: `c9ff8cf3f83aa19e5d8313c65816a3a2e6562368`. - All applicable [CI checks](https://github.com/paperclipai/paperclip/actions/runs/43019885807) passed. This includes typecheck, build, Runner support, Rust, native Runner, general tests, serialized suites, all eight E2E shards, and the required aggregate gates. - [Greptile review](https://github.com/paperclipai/paperclip/runs/130520170309) passed on this head with confidence 5/5. All three review threads are resolved. The independent Paperclip review also reports 5/5 with no findings. - Red at `7368a0b15c2f02107778a961144d86e2eb59a952`: `pnpm --dir server exec vitest run src/services/native-runtime/task-document-body.integration.test.ts` failed because write_document removed the final newline. Cross-task and REST cases passed. - Green on the final source: 176 tests passed across the document body, document key, native tool authority, task scope, document service, and document route suites. The tests cover create, update, exact read-back, replay, blank rejection, revision guards, and company scope. - The ZIP shutdown regression failed before repair with “scanner did not stop”. After repair, 54 tests passed across zip-inspection-shutdown, private-reporter-artifacts, manual-private-cell-report-assets, and private-evidence-budget. The regression uses a real process that ignores SIGTERM. - Earlier CI runs had two archive-related timeouts. The source repair above replaces blind reruns. The Runner support suite and aggregate verify gate now pass. - A disposable live native document reproduced the original whitespace loss. REST preserved the same string. The fixed handler passed local integration tests. The running service has not been deployed. - Local environment: Linux x64, Node 25.9.0, Vitest 5.0.3, and embedded PostgreSQL. Local Rust preparation was unavailable because cargo is absent. Final-head CI passed Rust verification. `pnpm verify --explain` requested broad CI because evidence.ts is a global input. Required CI supplied that coverage. ## Risks Low risk. The change retains required-content validation. The product change affects document body whitespace handling. The CI repair forcefully stops only already-rejected archive scanners; it retains fail-closed publication and all existing scan limits. It does not change schema, authorization, revision guards, receipts, or persistence. ## Model Used CI repair continuation: GPT-6.1, using repository tools and code execution. OpenAI Codex, GPT-6 (the exact serving model ID and context window were not exposed in this session). Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 06:38:32 -06:00
DottaandPaperclip	84fc67317e	feat(runner): export authorized read responses as artifacts (#20606) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use authorized API reads to get task evidence. > - Small responses return inline JSON, so an agent must copy each document into a local helper. > - Copying changes bytes and uses model output for data transport. > - This pull request adds an explicit artifact export for non-sensitive GET reads. > - Helpers can download the exact response bytes and check their identity. ## Linked Issues or Issue Description **What existing behavior does this improve?** The native `call_api` response capture path. **Current behavior** Small JSON responses have no artifact reference. Local helpers need the model to copy the JSON data. **Proposed behavior** `saveResponse: true` saves a complete authorized GET response below or above the inline threshold. Zero-byte success returns an explicit empty receipt without storage. The receipt returns an artifact ID, byte size and SHA-256 for `download_artifact`. Other callers keep the current behavior. Related work: #17393 covers saved app-tool result size. #18105 covers mutation response paging. This change adds an explicit read export. ## What Changed - Add GET-only `saveResponse` validation and capture through the current artifact store. - Refuse sensitive operations, denied reads, partial responses and combined text paging. - Keep route authorization, actor visibility, storage budgets and download integrity checks. - Update the native tool contract, generated catalogs and API guide. - Add boundary and real-route byte comparison tests. - Preserve saved run state. This is an optional input field; old calls and receipts keep their current shape. No schema migration or identity reset is needed. The seeded catalog fixture is generated. ## Verification - Final published head `d47b52e1bf412418e24768f865cc139097a6d068`: [typecheck, build, all tests, Linux Rust and aggregate verify passed](https://github.com/paperclipai/paperclip/actions/runs/42918883477); [Windows Rust passed](https://github.com/paperclipai/paperclip/actions/runs/42918883348); [quality gate passed](https://github.com/paperclipai/paperclip/actions/runs/42918883016). Greptile completed successfully at this exact head, 5/5; all five review threads resolved. Automated Paperclip review remains queued due to a failed reviewer run; no final verdict is claimed for that service. - Current head: `d47b52e1bf412418e24768f865cc139097a6d068`. Merged current master to refresh GitHub's stale base snapshot; the PR now contains exactly nine intended artifact transport files, with no lockfile or UI changes. Fresh final-head CI/review is required. - Previous head `986ad87490fff93cfec313ba6fd56d3dc0bb665b` passed typecheck, build, Rust verification, artifact/native suites and completed test shards; remaining shards were still running when the base snapshot was refreshed. Greptile was 5/5, with one unrelated master UI finding resolved as outside the corrected PR diff. - Review fixes verified at `986ad87490fff93cfec313ba6fd56d3dc0bb665b`: 2,135 checks passed across the five focused suites; Runner TypeScript build and regenerated contract/catalog checks passed. This head replays the original implementation on current master and retains the original PR commit in its history. - Review red/green at replayed baseline `da3a1d4de9`: empty HTTP 200/204 tests failed with real storage's `File is empty`, then passed with zero new assets or quota reservations. Error-body tests failed without size/truncation fields, then passed. - Empty successes now return `empty: true`, `byteSize: 0`, `data: null` without storage. Nonempty JSON results such as `[]` export normally. Error receipts report byte size and truncation. Docs and tool guidance state the existing 10 MiB download limit and paged text alternative. - Previous-head CI typecheck/build/tests passed except the Rust process-lineage cleanup test (`unattributed_orphan`); final-head CI and fresh review remain required. - Red at `7368a0b15c`: the small-document export test failed with `Invalid call_api arguments` before production edits. The same check passed after the change. - Final post-rebase run at `6e7a35ed6a59bb74f4218312d9f10557099433c3`: all 2,124 checks passed across the API, capture, artifact boundary, real-route export and artifact download suites. - Real-route export test passed: HTTP bytes, stored bytes and downloaded bytes were equal. Foreign-company access and forged download metadata were rejected. - Existing API integration and artifact suites passed. Nearby API integration checks also passed (59 passed, 2 skipped). - Disposable native helper scan passed: 12 existing documents, 15 GET requests, zero mutations, 73,919 bytes, 17.985 seconds. All files passed size/hash checks. The bridge passed file references only. Production token savings were not measured. - Runner TypeScript build and generated contract checks passed. - `pnpm verify` returned exit 2 and required broader CI coverage. Local preflight stopped at missing Cargo after dependency and SDK setup. The local server typecheck stopped during the cycle layer without a diagnostic; the container recorded an OOM kill. This check has no pass verdict. Required CI must cover typecheck, build and test at the final head. ## Risks - Each explicit export uses the current capture budget and storage admission. Frequent small exports can reach that budget. - An artifact holds one HTTP response. Callers must still follow list pagination. - No database or saved-state migration is required. The option does not change default response behavior. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in Paperclip Runner. Used reasoning, shell execution and native tools. The active agent configuration supplied this exact model ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 06:26:41 -06:00
DottaandPaperclip	bc453342ef	fix(native): start a cold server from active database checkpoints (#20792) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs save their controller state in complete encrypted database checkpoints. > - A rolling update starts a replacement with an empty local home while the old server still owns active remote work. > - Startup required a local controller marker before it checked the active database authority. > - The replacement could report `native_controllers_require_migration` even when the checkpoint, workspace and live owner were valid. > - This change uses the existing complete checkpoint proof when the local marker is absent. A marker that exists keeps its strict validation. > - The benefit is that a cold replacement can start while supported remote work continues. ## Linked Issues or Issue Description **What happened?** A volume-free replacement failed startup with `native_controllers_require_migration`. The old server had a local external-control marker and a valid active database checkpoint. The replacement had the checkpoint but no local marker. **Expected behavior** An absent local marker must not make a complete, valid database checkpoint into a local migration obligation. Missing, corrupt or mismatched durable evidence must still refuse startup. **Steps to reproduce** 1. Save a full native checkpoint with its real checkpoint writer. Keep its source run active with a live generation-one owner and its bound remote workspace. 2. Remove only the local controller marker to represent a cold replacement home. 3. Inspect native controller obligations. Before this change, the inspection returns true. **Paperclip version or commit** Reproduced on `eb75c9bb855ce83915256efc9d6b308a272cdeff`. The fix is based on `c44a422607f9f08c39ad98f9c830fe6e3dba8dc2`. **Deployment mode** Volume-free server with native remote execution and database checkpoint persistence. ## What Changed - Let an absent local marker use the existing full checkpoint, run, session, workspace and owner checks. - Keep validation of every present marker and the final file reread that detects a change during inspection. - Add a cold-home variant of the existing real PostgreSQL active-checkpoint regression. Its live source owner has a different boot ID from the inspecting process. - Run the same refusal controls for missing checkpoint parts, invalid identity, stale leases, changed task/workspace bindings and invalid local markers. ## Verification - Genuine RED: the cold-home case failed on unchanged production code with `expected true to be false` after the real checkpoint was saved and only its local marker was removed. - Rebased GREEN, Node 24: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/native-runtime/native-controller-migration.integration.test.ts -t 'offline native controller migration \(active_(durable|durable_cold|warm)\)'` — 3 passed, 53 unrelated cases filtered out. These controls use disposable real PostgreSQL and the canonical encrypted checkpoint writer/reader. - Server layered TypeScript check: `cd server && node scripts/typecheck-layers.mjs` — PASS for all source and test layers. - `node scripts/check-module-boundaries.mjs` and `git diff --check` — PASS. - [Required CI](https://github.com/paperclipai/paperclip/actions/runs/43003635878) passed on `7c5cb8fc16f3f7e3bcfec0ce10bf0b43d638d09b`. All 60 check runs are terminal: 55 success, 4 skipped and one neutral 5/5 review. Greptile is 5/5 with no review threads. Live rolling-update acceptance remains a separate delivery check. ## Risks - No schema or saved-state format changes. State saved by the previous release uses the same checkpoint reader. - The absent-marker path still needs a complete valid checkpoint and the existing identity, workspace and owner proof. Present legacy or malformed markers still refuse. - This changes eligibility only. It does not take ownership, launch work, copy state, release a lease or alter admission. - No command or configuration changes need operator documentation. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use and code execution. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal issue id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-11 04:42:05 -06:00
DottaandClaude	c44a422607	fix(native): retry a follow-up whose own server stopped serving before its remote runner (#20786) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs each agent task through a Runner. On a graceful shutdown (SIGTERM), the server closes its runtime API listener first, then drains runs. > - A run that is still preparing can reach the runtime API lookup after the listener closed. It then fails with `local_runtime_api_listener_unavailable` before it reaches its Runner. > - For a remote (sandbox) Runner, the executor already settles this failure as a same-run bootstrap retry, so the next server restarts it with no manual Retry. > - A follow-up run is different. Admission rebinds the settled predecessor's provider session into the follow-up's run as its checkpoint. The recovery evidence saw "a checkpoint exists" and called the failure `ambiguous_state`. That blocked the task and the user had to click Retry. > - This pull request recognizes that carried checkpoint. When this server's own listener closed, the run has no Runner checkpoint and no provider event of its own, and a settled predecessor saved that exact provider session, the failure is a same-run bootstrap retry. > - The benefit is that a graceful restart during a follow-up's preparation no longer needs a manual Retry. ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave (codex, remote sandbox Runner), a follow-up run was created. About 2 s later a graceful SIGTERM drain began. The draining server kept preparing the run. `resolveLocalRuntimeApiUrl()` in `createRunnerdBackendWithinSessionClaim` threw `LocalRuntimeApiListenerClosedError`. The run failed with `local_runtime_api_listener_unavailable`. The task went to `blocked` with a board-owned recovery action, and it needed a manual Retry. The manual Retry worked. **Expected behavior** No provider work started for the run. The next server must pick it up automatically. **Steps to reproduce** 1. Finish a remote native run for a task, so that its provider session is saved. 2. Comment on the task to start a follow-up run. The follow-up carries the saved provider session as its admission checkpoint. 3. Send SIGTERM while the follow-up is in preparation, before it reaches its Runner. 4. The follow-up fails with `ambiguous_state` and the task is blocked. **Paperclip version or commit** `master` at `62b7733161` (wave) and `eb75c9bb85` (this branch base). Refs #20514, #20721 ## What Changed - `native-session-executor.ts` (`nativeProviderRecoveryEvidence`): add one narrow case before the generic checks. When all of these are true, return `bootstrap_retry` (with `checkpointExists: true` and `providerSessionEstablished: false`): - the source failure is `local_runtime_api_listener_unavailable` (this server stopped serving); - the target is remote, the run has the current preparation fencing version, and the run has no Runner controller checkpoint; - the run has no provider event; - the run checkpoint has no cursor, turn, terminal, result, or pending request of its own; - a settled earlier run of the same native session saved that exact provider session. - `native-restart-recovery.ts`: export the existing `carriesSettledPredecessorSession` proof, so that the executor uses the same proof that restart recovery uses for rebound sessions. - `native-controller-recovery.integration.test.ts`: add a regression test. It also shows that the verdict stays `ambiguous_state` for a different failure code, for an unknown provider session, for a checkpoint with its own turn, and for a run with a provider event. ## Verification - Red on master: `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "own server stopped serving"` returned `recoveryMode: "ambiguous_state", providerSessionEstablished: true`. - Green with the fix: the same file passes (25 tests). - `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts src/services/native-runtime/native-runner-restart-preparation.integration.test.ts`: 43 passed (the existing closed-listener bootstrap-retry and ambiguous cases are unchanged). - `cd server && pnpm typecheck`: passed. - Not run: a live restart-chaos wave. ## Risks - Low risk. The new case applies only to the closed-listener failure on a remote target. It needs durable proof that the run has no Runner checkpoint and no provider event, and that the provider session belongs to a settled predecessor. All other cases keep their earlier verdict. - If the retry later finds different evidence, the existing restart recovery and resumption checks re-prove it and fail closed. The task then keeps a working Retry. - Upgrade safety: no persisted shape changes. The proof reads only rows that older servers already write (`sessionCheckpoint`, `nativePreparationFencingVersion`, Runner checkpoints, run events). A run that an older server prepared without the fencing version keeps its earlier verdict. - No waits are added before `turn.start`. The new queries run only on the failure path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 04:22:49 -06:00
DottaandClaude	59ded526be	fix(native): ask once to re-check Paperclip before resting blocked on its own stall (#20779) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs report their result through a completion report. The server checks the report before it accepts it. > - When the Paperclip server pauses or restarts, the agent's Paperclip tool calls stay unanswered. They return later, all together. > - The agent can read that silence as an outage. It then reports a blocker owned by a "Paperclip ... service" with reasonCode `external_service_unavailable`. > - The server accepts that report. The task rests blocked with no Retry. Paperclip's own stall stops the task. Our rule says that must not happen. > - This pull request rejects that blocker report one time per run. The correction tells the agent that Paperclip is reachable now and to re-check. > - The benefit is that the agent continues in the same turn and reports the real outcome. ## Linked Issues or Issue Description **What happened?** In a restart-chaos test, the server was paused (SIGSTOP) for about 2 to 5 minutes while a Codex agent saved a task document. `write_document` and `read_document` did not return during the pause. After the pause, the save had succeeded, but the agent reported a blocker: owner "Paperclip document service", reasonCode `external_service_unavailable`. The run succeeded with disposition `blocked`. The task rested blocked with `canRetry=false` and no automatic recovery. A second test (two long server holds, then a same-run resume) gave the same result. **Expected behavior** A Paperclip stall or restart must not leave a task stuck. The agent must re-check Paperclip after it is reachable and finish the task. **Steps to reproduce** 1. Start a native Codex task that saves a task document. 2. Pause the server process with SIGSTOP for about 2 minutes during the save, then resume it. 3. The agent reports a blocker owned by a Paperclip service. The task rests blocked with no Retry. **Paperclip version or commit** master at 62b7733161 (test base) and eb75c9bb85. ## What Changed - `nativeCompletionFeedback` now rejects a `blocked` report one time per run when all of these are true: the blocker owner kind is `system`, the owner name contains "Paperclip", the reasonCode is `external_service_unavailable`, the run is still running, and it owns the task. - The correction tells the agent: Paperclip answered this report, so it is reachable now. Re-read the current state before writing again. Repeat a needed write with the same `idempotencyKey`. Report blocked only if a fresh Paperclip call fails now. - The second identical report in the same run is accepted as before, so a persistent real failure still rests blocked. - Added a regression test. It also covers concatenated owner names such as "PaperclipAI document service" and "paperclip_runner" (the match is a case-insensitive substring). ## Verification - `cd server && npx vitest run src/services/native-runtime/native-completion-feedback.test.ts` passes (32 tests). - Red on master: the new test fails with `promise resolved "'Blocker report accepted. ...'" instead of rejecting`. Green with this change. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Low risk. The change only adds one correction message inside the same turn. It does not change a status, a database row, or a schedule. - The once-per-run limit is kept in process memory. After a server restart, a run can get the correction one more time. This is bounded and harmless. - The match uses the agent's own owner name. That only selects the correction. It does not prove anything or replay a side effect. The agent's next Paperclip calls give the real proof. Writes use idempotency keys, so a repeated write does not apply twice. - This does not add a Retry to a task that already rests blocked from an older report. That is a separate product decision. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user docs change; the behavior is explained in code comments) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending on the latest head) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending on the latest head) - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 04:22:09 -06:00
DottaandClaude	eb75c9bb85	fix(native): resend a throttled harness backup upload instead of losing it (#20772) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - After a native run in a per-turn remote sandbox, the server saves a durable harness backup to object storage. The next run restores the provider session from it. > - On staging, two OpenCode runs had an accepted result, but `harness_state.backup.persist` failed. The run log for both says `remote checkpoint failed: Please reduce your request rate.` This is the S3 `SlowDown` (HTTP 503) answer. > - The backup and the OpenCode provider-artifact bundle were uploaded as one stream `PUT`. The S3 SDK cannot send a stream body again, and this path had no retry for a request-rate limit. One `SlowDown` lost the backup. > - The sandbox was then released. The follow-up run had no state to resume and failed with `PRP provider resume state is unavailable` (class P, see #20757). > - This pull request uploads both objects from their encrypted scratch file, as workspace snapshots already do. The existing throttled-retry path then sends them again. > - The benefit is that a short request-rate limit no longer loses the only copy of a provider session. ## Linked Issues or Issue Description Refs #20757 (the failure this causes on the next run). Refs #19917 (added the file-source upload path that this change uses). **What happened?** 1. An OpenCode run in a per-turn Daytona sandbox finished, and its result was accepted. 2. The server uploaded the encrypted harness backup with `storage.putFile` and a stream body. 3. Object storage answered `503 SlowDown` ("Please reduce your request rate."). Nothing sent the request again. 4. `harness_state.backup.persist` failed, and `native.post_result_failure` was recorded. 5. The next run for the same chat task started in a new sandbox and could not resume the provider session. **Expected behavior** A request-rate limit slows the upload down. It does not lose the backup. **Steps to reproduce** 1. Point the storage service at an S3 endpoint that answers the first `PUT` of each object with `503 SlowDown`. 2. Call `persistNativeHarnessBackup`. 3. On `master` it throws `SlowDown: Please reduce your request rate.` **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `server/src/services/native-runtime/native-harness-backup-storage.ts`: pass the encrypted scratch file as `sourceFile` to `putFile`. - `server/src/services/native-runtime/native-provider-artifacts.ts`: the same for the provider-artifact bundle. - `native-harness-backup-throttle.integration.test.ts` (new, focused file): a real S3 client against a local endpoint that throttles the first `PUT` of each object once. Cases for a Codex backup, and for an OpenCode backup with provider artifacts. Each case also throttles the first read-back of each object once. Each case checks that the backup is current and restores the provider session bytes. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-throttle.integration.test.ts`: both cases fail on `master` with `SlowDown: Please reduce your request rate.` and pass with this change. - `cd server && npx vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 68 pass (file unchanged). - `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify`: it selected the full server suite and failed 16 of 34,997 tests, all local-environment failures unrelated to object storage: missing `.claude/skills/*/SKILL.md` files in this worktree, `ENAMETOOLONG` under the macOS temp directory, and 30 s hook timeouts under load. Required CI is the authority for that suite. ## Risks - Low. A file source also sends a `Content-MD5` header, so the backend rejects any byte it did not receive intact. Workspace snapshots have used this path since #19917. - An upload above 16 MiB now uses the file multipart path. That path has the same verified-part retry. - The read-back check (`getObject`) needs no change: a GET has no body, so the S3 client resends it after `SlowDown`. The test now also throttles the first read of each object once and proves the backup still publishes. - The Rust Runner is unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 02:56:52 -06:00
Nicky Leach	4d141cdf64	Carry plugin task refusals and batch project reads (#20762)	2026-11-11 00:49:03 -08:00
DottaandClaude Opus 5.5	f125b823f4	fix(recovery): release a parent's Retry once a done child's recovered files are saved (#20728) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task waits for its done child tasks to save their workspace files before it runs again (the workspace-finalize barrier) > - "Recover saved files" saves the retained work of a done child and records a successful `workspace_finalize` > - That finalize row had no `execution_workspace_id`. The barrier reads finalize rows by the blocker's workspace, so it never saw the row > - The parent's saved Retry then waited forever with "Saving its files" on a child that had already saved its files > - This pull request lets the barrier accept the recovery's own finalize for the recovered task's finished operations of the same run. It does not make the recovery finalize a workspace-wide save > - The benefit is that recoveries work by themselves: the parent's dependency wait releases and its saved Retry starts on the next dependency pass, with no migration ## Linked Issues or Issue Description **What happened?** A parent task had three done children. One legacy child shared the parent's execution workspace. The operator pressed "Recover saved files" on that child. The recovery saved the files and marked the old run succeeded. The operator then pressed Retry on the parent. The server accepted the Retry, but the parent stayed on "Retry is waiting for a finished task this one depends on to save its files" for more than 15 minutes. `blockerAttention.pendingFinalizeBlockerIssueIds` named the recovered child. The child had no active recovery action and no execution blocker. The workspace was clean (`dirty: false`, `writerRunId: null`). The child's workspace operations showed the cause. The latest operation on the workspace was a succeeded `workspace_recovery`. The recovery's succeeded `workspace_finalize` existed, but with `execution_workspace_id = NULL`. Earlier finalize attempts of that run had failed. `pendingFinalizeBlockerCondition` only looks at operations on the blocker's workspace, so it found no successful finalize after the recovery. **Expected behavior** When "Recover saved files" saves a done child's work, the child must stop holding its dependents. The parent's saved Retry must start without another operator action. **Steps to reproduce** 1. Let a child task finish as `done` with a retained remote workspace whose finalize failed. 2. Run "Recover saved files" on the child, so recovery saves the remote workspace snapshot. 3. Block a parent task on the child. The parent can share the child's workspace. 4. Read the parent's dependency readiness. Before this fix it reports the child in `pendingFinalizeBlockerIssueIds` forever. **Paperclip version or commit** Seen on staging at `a46b9ac7`. Reproduced on `master` at `e039be9202`. ## What Changed - `server/src/services/issues.ts` (`pendingFinalizeBlockerCondition`): the blocker task's latest workspace operation is settled when a succeeded recovery finalize (`owningService: workspace_recovery`) from the same company, task, and run started at or after that operation finished. The operation must be finished (`succeeded`, `failed`, or `skipped`). The recovery finalize can have no workspace (all rows written so far) or this workspace. - The rule is scoped to the blocker task and its run. It does not change the workspace-wide `finalized_at`, so other tasks that share the workspace keep their own barrier. - `server/src/services/workspace-recovery-dependency-finalize.test.ts`: new regression tests that run the real remote "Recover saved files" path, make the child `done`, and block a parent that shares the child's workspace. - `doc/execution-semantics.md`: documents how the recovery finalize settles the dependency barrier. ## Verification - Red (unfixed `issues.ts`): `npx vitest run src/services/workspace-recovery-dependency-finalize.test.ts` fails 3 of 4 tests. The parent stays not ready with the recovered child in `pendingFinalizeBlockerIssueIds`. - Green (fixed): the same file passes 4 of 4 tests: - a real recovery releases the parent in `getDependencyReadiness` and in the issue list `blockerAttention`; - snapshot-fallback shape: the preservation receipt has no workspace and the child's latest operation is the run's failed finalize. The parent is released; - safety: a sibling task that started work on the shared workspace while the recovery waited for teardown still holds the parent; - safety: child work that starts after the recovery still holds the parent. - Focused suites pass: `src/__tests__/issues-service.test.ts`, `src/services/workspace-recovery-admission.test.ts`, `src/services/task-retry.test.ts`, `src/services/saved-bootstrap-continuation.test.ts`, `src/services/remote-workspace-recovery-receipt-fallback.test.ts`, `src/__tests__/issue-dependency-wakeups-routes.test.ts` (414 tests with the new file). - `pnpm verify` exits 2 (a doc change is a global CI input), so required CI supplies broad coverage. Server `tsc --noEmit` passes. ## Risks - Previous-release story: no persisted shape changes. Recovery finalize rows already saved (with no workspace) are read by the new rule. Affected parents become dependency-ready on the next read, with no migration or backfill. The deferred-receipt and dependency backstop passes then start the saved Retry. - The barrier keeps its protection. Unfinished operations, operations from another run, operations that start after the recovery finalize, and other tasks on a shared workspace still wait for their own save. - No schema change and no write-path change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution, read-only staging API inspection). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-11 01:42:29 -06:00
DottaandClaude	1ac8b6a711	fix(workspaces): stop re-saving a retained sandbox whose disk is still full (#20758) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a native run on a remote sandbox, Paperclip saves the sandbox workspace back to durable storage. When that save fails, saved-work recovery saves the files again from the retained sandbox, with no provider turn. > - A staging census of the last 48 hours (`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`, class F) found saves that failed because the sandbox disk was full. > - #20678 already ends finalization at the first full-disk copy-back and hands the work to saved-work recovery. Recovery then deletes inactive tool caches that it can prove safe, and saves again. > - When that save still found the disk full, recovery recorded a transient `export_interrupted` failure with cause `provider_error`. It then saved again on its automatic schedule (up to 5 retries in 30 minutes). Nothing frees space in a stopped, retained sandbox between those attempts, so each attempt failed the same way. The task showed "Recovering saved work…" and no Retry while this went on. > - This pull request ends the recovery generation at the first such failure with a named cause, `sandbox_disk_full`, and hands it to the board with a Retry. > - The benefit is that the task does not spend 30 minutes on saves that cannot succeed, the user sees why, and one Retry saves the files and starts one run. ## Linked Issues or Issue Description Refs #20678, #20695 (named sandbox disk-full failures for copy-back and restore) **What happened?** A native run on a remote sandbox finished, and its workspace copy-back hit a full sandbox disk. Finalization handed the task to saved-work recovery (#20678). Recovery's own save also hit the full disk. Recovery classified this as `export_interrupted` / `retry_transient` with cause `provider_error`, so it scheduled more automatic saves of the same sandbox. Each one failed the same way. The task showed "Recovering saved work…" with no Retry until the 30-minute window ran out. **Expected behavior** A save that still finds the disk full after the safe cache cleanup is not repeated automatically. The recovery ends at once with the cause `sandbox_disk_full`, and the board gets a Retry. The workspace keeps its previous saved version, and the run's unsaved files stay in the retained sandbox. One Retry saves the files once there is room and starts exactly one run. **Steps to reproduce** 1. Give a native run an accepted result whose copy-back failed (`native_workspace_sync_out_retry_exhausted`). 2. Make every save from its retained sandbox fail with `remote_workspace_sandbox_disk_full`, with no inactive tool cache to delete. 3. Run saved-work recovery. Before this change, it records `export_interrupted` with cause `provider_error` and schedules another save. ## What Changed - `workspace-recovery.ts`: a save that fails with `remote_workspace_sandbox_disk_full` after a confirmed cache cleanup (or with nothing to clean up) now fails the generation with `export_interrupted`, retry class `user_action_required`, and cause `{ label: "sandbox_disk_full", code: "ENOSPC" }`. That class is not retried automatically, and it is not a permanent-source class, so recovery never falls back to the last save and never drops the run's unsaved files. The next sweep hands the hold to the board. - A cleanup whose outcome is unknown (lost acknowledgement, interrupted journal) keeps the existing scheduled retry, because its journal resumes on the next attempt and can still free space. - `retained-provider-cache.ts`: `reclaimInactiveNativeProviderCache` returns `{ changed, confirmed }` instead of a boolean, so the caller can tell "nothing to delete" from "outcome unknown". It has one caller. - `system-recovery-state.ts`: the board handoff text for this cause says that the sandbox disk is full, that Paperclip deletes only inactive tool caches it can prove safe, and that Retry saves the files again. - New `workspace-recovery-disk-full.test.ts` covers the whole path: one failed attempt, no further automatic saves, the board handoff text, the previous saved version unchanged, the agent's files still in the sandbox, no unfinished in-sandbox capture attempt, a Retry offered in the task presentation and blocker, `POST /issues/:id/retry` accepted (202), the files saved after Retry, and exactly one run started (also with a replayed dispatch). - `doc/execution-semantics.md`: documents the behavior. ## Verification - Red on master: `cd server && npx vitest run src/services/workspace-recovery-disk-full.test.ts` fails. The recovery state is `scheduled` with cause `provider_error` instead of `exhausted` with cause `sandbox_disk_full`. - Green on this branch: the same command passes. - Related suites pass: `workspace-recovery-retry-copyback.test.ts` (79 tests, including the 8 cache-cleanup resume tests that need the scheduled retry after an unconfirmed cleanup), `system-recovery-state.test.ts`, `workspace-recovery-capture-policy.test.ts`, `workspace-recovery-fallback.test.ts`, `workspace-recovery-stop-proof.test.ts`, `native-run-finalizer-telemetry.test.ts` (167 tests in all). - `npx tsc --noEmit -p server/tsconfig.json` is clean. - `pnpm verify`: exit 2 (broader verification required), because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change touches only the saved-work recovery path when its save fails with the exact full-disk message. - Upgrade-safe: the failure code (`export_interrupted`), the retry class (`user_action_required`) and the cause label all pass the earlier release's recovery schema. An earlier release that reads this state shows its generic "Retry to try again" text. - A full-disk failure no longer gets automatic retries in its 30-minute window. If space became free by itself in that window, the user must now press Retry. Nothing in a stopped, retained sandbox frees space by itself, so this is expected to be rare. - No Runner (Rust) change, so no new sandbox image is needed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 01:31:50 -06:00
DottaandClaude	ea25af759e	fix(native): name a resume that finds no saved Runner state, so Retry starts a fresh session (#20757) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can continue the provider session of the previous run on the same task. In a remote sandbox, the Runner keeps the session state inside the sandbox, and the server saves a harness backup after each run. > - On staging, five runs in 48 hours failed with `native_session_recovery_failed: Error: PRP provider resume state is unavailable` (class P in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`). > - In each case we read, the previous run had an accepted result, but its `harness_state.backup.persist` failed. Its per-turn sandbox was then destroyed. The next run started in a new sandbox with no Runner state for the session. > - The Runner transport threw an untyped error. The runtime could not tell it apart from a transport drop. A governed-wait or same-session checkpoint ended as a generic `native_session_interrupted` with an "ambiguous state" hold that said not to open a replacement session. > - This pull request types that refusal and lets the driver classify it as `provider_state_incompatible` when nothing is in flight. The existing recovery policy then decides. > - The benefit is that the run ends with the permanent named code `native_provider_state_incompatible`. Retry, or the next message, continues in a fresh provider session in the same workspace. No provider turn or tool call can be repeated, because the follow-up never started a turn. ## Linked Issues or Issue Description Refs #20712, #20723 (the same classification for a moved remote workspace root). Refs #20691 (fixes the `provider_identity_incomplete` part of class P). **What happened?** 1. Run A on a chat task finished. Its result was accepted. Its harness backup failed to save (`harness_state.backup.persist` failed, `native.post_result_failure`). 2. Run B started for the next chat message in a new sandbox. It rebound run A's checkpoint with `providerRecoveryPolicy: allow_replacement_after_governed_wait`. 3. The Runner found no control-plane state and no archived authority epoch. It threw `PRP provider resume state is unavailable` as a plain `Error`. 4. The runtime refused a replacement for the governed-wait policy and failed run B with the generic `native_session_recovery_failed`. The task stayed blocked with an "ambiguous state" hold. **Expected behavior** The run ends with a permanent named code and a working Retry, as it does for other deterministic refusals. **Steps to reproduce** 1. Create a Runner transport that resumes a saved provider session from an empty state directory. 2. Request `thread/read`. On `master` it rejects with a plain `Error`. 3. Recover a saved Codex, OpenCode or ACPX session through the driver when the transport fails that way. On `master` the result has no `failureClass`. **Paperclip version or commit** `master` at `9574065a28`. ## What Changed - `packages/paperclip-runner/src/contracts/harness-driver.ts`: add `NativeRunnerResumeStateUnavailableError` (same message as before) and `isRunnerResumeStateUnavailable`. - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: throw the typed error at the three branches where the control-plane state is absent. The branch with no PRP identity and the quarantine branch are unchanged. - `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts`: classify the typed error as `provider_state_incompatible` when the saved session has no turn or runtime request in flight. Extract the in-flight test that the moved-root check already used, without changing it. - `doc/execution-semantics.md`: describe the new classification (additive). - Tests: a transport test (local and external Runner state), driver tests for Codex, OpenCode and ACPX Claude, in-flight cases that stay unclassified, and a server case that shows the blocked task offers Retry and that one click starts exactly one fresh-session run. ## Verification - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts -t "no saved state"`: 5 pass. With the driver change reverted, the 3 "reports … as provider state incompatible" cases fail (no `failureClass`). - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport-cold-restore.test.ts`: 5 pass. On `master` the error is a plain `Error`. - `cd packages/paperclip-runner && npx vitest run src/backends/native-backend-factory.test.ts src/drivers/codex/codex-app-server-driver.recovery.test.ts src/live/runnerd-codex-transport-cold-restore.test.ts src/native-session-runtime.test.ts`: 410 pass. - `cd server && npx vitest run src/services/workspace-recovery-retry-copyback.test.ts -t "does not continue automatically"`: 5 pass, including the new case. It checks `canRetry: true`, an accepted Retry, exactly one queued successor, and `forceFreshSession: true`. - `cd packages/paperclip-runner && npx tsc --noEmit -p .`: clean. - `pnpm verify`: exit 2 (broader verification required) because `doc/execution-semantics.md` is a global CI input. The focused checks above pass. Required CI supplies the broad coverage. ## Risks - Low. The change only adds a class to a failure that already failed. A checkpoint with `allow_replacement_after_resume_failure` already replaced the session for this failure, and still does. - For `same_session_only` and `allow_replacement_after_governed_wait`, the run now ends with `native_provider_state_incompatible` instead of a retryable `native_session_interrupted`. Same-run retries stop at once. They could not succeed, because the state is absent. - A session with a turn or runtime request in flight keeps the old unclassified behavior. This PR does not change it. - The server code path is unchanged. The Rust Runner is unchanged, so no new runner image is needed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution) and read-only staging API reads. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 01:30:27 -06:00
DottaandClaude	9574065a28	fix(workspaces): hand on a proven-unchanged preparation epoch instead of abandoning it (#20736) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox first restores its workspace's saved version, publishes that root as its dirty epoch, and only then starts work > - A restart that kills the run in that window leaves the epoch dirty. The ownership settle (`settlePreparationOnlyRemoteWorkspaceEpoch`) always abandons it: the root is forgotten and the retry restores the saved version into a new directory > - The saved provider session still names the old directory, which is what #20712 and #20723 now have to handle > - In the common case the killed preparation only restored the saved version and changed nothing. A durable proof for exactly that already exists (`proveRemoteWorkspaceUnchanged`, used when the next admission finds an ended writer) > - This pull request makes the settle use that proof first, and keeps abandoning whenever the proof fails > - The benefit is a stable workspace root across a restart during preparation, with no change for anything unproven ## Linked Issues or Issue Description Refs #20712 **What happened?** In restart-chaos waves, a server kill during a follow-up's preparation (right after the workspace restore published) made the ownership settle abandon the epoch. The retry restored into `<workspace>-<epoch>-<lease>` instead of the published root, and the inherited session checkpoint named a root that no longer existed. **Expected behavior** When durable records prove the killed preparation only restored the saved version, the workspace is handed on at that saved version and keeps its root. Anything unproven is abandoned exactly as before. **Steps to reproduce** 1. Run a remote native task to a saved version, then start a follow-up or continuation run. 2. Kill the server after the follow-up's restore publishes (crash point `remote_workspace.prepare.after_publish_before_provision`). 3. Let ownership reconciliation settle the dirty epoch, then let the retry prepare. ## What Changed - `recovery/remote-workspace-unsaved-files-settlement.ts`: before it abandons a preparation-only epoch, the settle calls the existing `releaseUnchangedRemoteWorkspaceWriterById` (lock the workspace, check the writer, then `releaseUnchangedRemoteWorkspaceWriter`). That releases the writer at its saved version (dirty false, writer cleared, location and root kept, baseline marked released, run reference set to `saved`) only when `proveRemoteWorkspaceUnchanged` holds. The proof requires all of these durable records: - the unchanged-baseline marker for this exact epoch, run and saved version. It is written in the same transaction that publishes the restored root, so a kill mid-restore leaves no marker; - no recorded process, process group, process start or provider session; - no execution-start run event, no native result and no native runner checkpoint (the controller writes one before every runner command); - no workspace operation other than the file-neutral restore, finalize and recovery receipts (provisioning and saves record theirs before they touch files); - every workspace repository already in the saved version. - If the proof fails for any reason, the settle abandons the epoch exactly as before. The existing settle preconditions (run terminal, still `preparing`, no finalization, no work-started event, no save in flight) are unchanged and run first. - `packages/shared/src/crash-point-registry.ts` and `remote-workspaces.ts`: add the crash point `remote_workspace.prepare.after_publish_before_provision`, so the chaos harness can kill exactly at this boundary. It counts only a preparation that published an unchanged baseline (a restore of a saved version), and it is a no-op unless a test arms it. `tests/runner-e2e/source-sdk-public-inputs.json` is refreshed because the registry file is a pinned SDK input. - Test `ownership-reconciliation-report.test.ts`: a proven-clean epoch is handed on with its root kept, with no abandonment receipt, and the writer hold is free for the retry. These cases are still abandoned exactly as before: no baseline (an epoch saved by the previous release), a baseline for another epoch, an already-released baseline, a provisioning operation. A run with a recorded process is still neither handed on nor abandoned by this settle. Previous-release story: an epoch saved by a release that did not write the baseline marker has no proof, so it is abandoned as before. No persisted shape changes; the marker already exists on master. ## Verification - Red: with the release branch disabled, the new test fails on the clean case (the epoch is abandoned). Green with the change. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/crash-point-call-sites.test.ts` passes. `packages/shared` `crash-points.test.ts` passes. `tsc --noEmit -p server` passes. - Live Daytona waves (claude, workload S1, the new crash point at the continuation's preparation (`:2` before the point was gated to saved restores; `:1` now), image built from master `e039be9202`, one task each): | | before (release disabled) | after | | --- | --- | --- | | Killed epoch 2 | abandoned (`workspace.remote_epoch_abandoned`) | handed on (`workspace.snapshot_skipped_unchanged`, epoch 2) | | Retry root | new root `…-4-ecf1c81e…` | same published root `…-1-33e93b90…` | | Retry `task.prepare` | 4598 ms | 4731 ms | | Retry restore / environment acquire | 1141 / 1381 ms | 1106 / 1478 ms | | Task result | done, no Retry | done, no Retry | | Stopped or orphan sandbox at rest | none | none | Preparation time does not change in this setup. The Chaos Lab environment uses ephemeral leases, so every run gets a fresh sandbox and a full restore with or without this change, and the killed run's sandbox is destroyed either way. The gain is the stable root, which keeps the saved session's working directory valid. ## Risks - Low. The release path is the same one admission already uses for an ended writer, under the same proof. The settle now tries it first. - A proof false positive would keep a root whose files changed. The proof refuses on every record that precedes a write, and on any missing or older-release evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 00:33:18 -06:00
DottaandClaude	f25fee3532	fix(native): hash runner artifacts and run archive tar off the event loop (#20722) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process prepares native runs, renews every controller lease on timers, and serves every request > - On a run path, the server hashed the 31 MB runner binary with `readFileSync` (more than once per run), hashed provider packs and harness backups synchronously, and ran `tar` with `execFileSync` (timeouts up to 60 s, one with no timeout) > - On a slow disk, each of these calls freezes the whole process. A 103 s stall on a hosted instance lapsed every lease. The last event before that stall was immediately before a synchronous read and hash of the runner binary (see #20694 and #20709) > - This pull request moves those reads, hashes and archive commands off the event loop. It computes the runner binary digest once per boot > - The benefit is that a slow disk delays only the run that waits for it. Turns also start faster, because the runner binary is no longer re-read for every run ## Linked Issues or Issue Description Refs #20694, #20709 **What happened?** On a hosted instance, the server process stopped for 103 s without a restart. Every controller lease lapsed, and two tasks needed a manual Retry. The last run event before the gap was immediately before `matchesControllerRunnerArtifact`, which does `createHash("sha256").update(readFileSync(controllerRunnerBinary))`. Four workspace restores were running at the same time, and file I/O was very slow (a 43 s archive hash, 20 s snapshot decrypts). The trigger is not proven. But each synchronous call below blocks the event loop for as long as the disk is slow. **Expected behavior** File hashing and archive commands on a run path must not block the event loop. A slow disk must delay only the run that waits for it. **Steps to reproduce** 1. Make a remote runner binary slow to read (the test uses a FIFO whose writer waits), or make `tar` slow (the test puts a `tar` wrapper that sleeps first on `PATH`). 2. Call `resolveRemotePaperclipRunnerBinary`, `stageRemoteRunnerDirectory`, or `syncRemoteRunnerDirectoryOut` while a 10 ms timer runs. 3. Before this change, the timer stops for the whole slow step (814 ms, 1,764 ms and 3,379 ms). After this change, it keeps firing. ## What Changed - `server/src/services/native-runtime/off-loop-file-work.ts` (new): - `sha256FileStreamed` - `cachedFileSha256`: a streamed digest, reused while the file's device, inode, size, mtime and ctime do not change - `binaryTargetFromHeader`: reads 64 KiB; reads the rest of the file only when the header cannot answer, so the result equals `runnerBinaryTarget(readFileSync(path))` - `sha256DirectoryTreeStreamed` - `execFileBuffered`: `execFile` with stdin input, `maxBuffer`, `timeout` and `env`. It rejects on a non-zero exit, a timeout or a `maxBuffer` overflow, like `execFileSync`. - `native-session-executor.ts`: - `matchesControllerRunnerArtifact` uses `cachedFileSha256`. Each boot reads the binary once, not once or twice for every run. - The remote Claude daemon target is read from the binary header. - The bundled remote runner binary is checked by `verifiedBundledRemoteRunnerBinary`: the same checks and errors, with a cached digest. - Provider-pack upload verification uses the new `verifyRemoteProviderPackManifest`. It runs the same checks, in the same order, with the same errors. One generator drives both the sync and the streamed readers. - Harness backup build and verify on run paths use streamed digests (`buildNativeHarnessBackupManifestStreamed`, `verifyNativeHarnessBackupStreamed`). The digest format does not change. - `stageRemoteRunnerDirectory` (3 calls), `assertSafeRemoteCheckpointArchive` (3 calls) and the checkpoint extract use `execFileBuffered`, with the same arguments, timeouts and buffer limits. `stageRemoteRunnerFile` uses `readFile`. - `packages/paperclip-runner/src/live/runner-binary.ts`: the default daemon lookup (`resolvePackagedRunnerBinary`, used by the local and remote resolvers) classifies the daemon with `runnerBinaryFileTarget`. It reads the 64 KiB header, not the whole file. - `packages/paperclip-runner/src/live/bundled-remote-provider-pack.ts`: the release-manifest checks now live in one helper, `bundledRemoteRunnerRelease`. The sync `bundledRemoteRunnerBinary` and the server's streamed check both use it. - `native-codex-runner.ts`: - `resolveRemotePaperclipRunnerBinary` reads only the header. - The local `runnerDigest` uses `cachedFileSha256`. - New `verifiedBundledRemoteRunnerBinary`. **Ordering and atomicity.** I checked each converted call for code that relied on the sync call being atomic with respect to other runs: - **Checkpoint copy-back.** It extracts into a private `mkdtemp` directory. The replace sequence (`existsSync`, `rename`, `rename`, `rm`) stays synchronous, so no other run can see a half-installed checkpoint. This function already awaited a remote command before the extract, so this change adds no new interleaving with the target. - **Harness backup verify then restore.** The verify was sync, but the restore after it was already async. The window between them existed before. One owner serializes the turns of a session, so no other turn publishes a backup for that session while a turn verifies it. - **Staging and provider-pack hashing.** These read build-owned, immutable trees, or the private staging copies of a run. ## Verification - Red/green: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/off-loop-file-work.test.ts` - With `native-session-executor.ts` and `native-codex-runner.ts` from `origin/master`: 3 failed. The longest timer gaps were 814 ms (runner binary target read), 1,764 ms (directory staging tar) and 3,379 ms (checkpoint list, size check and extract). - With this change: 6 passed. The gaps stay under 400 ms. The same file also checks the digest cache, the header fallback, and the `execFileSync`-like errors (exit code, timeout kill, `ERR_CHILD_PROCESS_STDIO_MAXBUFFER`). - `native-session-executor.test.ts`: - The streamed provider-pack verifier returns the same manifest as the sync reader, and rejects a changed asset tree with the same error. - The streamed harness backup manifest equals the sync one. - The streamed verify agrees with the sync verify, for a valid backup and for a corrupt one. - Run together: `native-session-executor`, `off-loop-file-work`, `native-codex-runner`, `setup-readiness`, `antigravity-connection-probe` (1,653 passed). `run-completion.test.ts` passed all 153 tests. Its `afterAll` embedded-Postgres cleanup timed out locally, which is not related to this change; CI runs it. - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. ## Risks - **Turn start.** This change adds no wait before `turn.start`; it removes work. The first run of a boot streams the runner binary once. Later runs reuse the digest after one `stat`. - **Cache staleness.** The cache keys on device, inode, size, mtime and ctime. An in-place rewrite changes mtime and ctime, and a replaced file changes its inode, so a changed binary is hashed again. - **stderr.** `execFileSync` wrote a child's stderr to the server's stderr. `execFileBuffered` captures it. Tar warnings no longer appear in server logs; errors and exit codes are unchanged. - **Persisted state.** No persisted shape changes. Old saved state and new code (and the reverse) read the same manifests and digests. - **Not converted in this PR:** small metadata syscalls (`existsSync`, `lstatSync`, `readdirSync`, small JSON reads and writes, renames) in the native state machines. Some of these rely on running synchronously. Also not converted: the sync harness backup verify inside the sync restart-authority checks, and the sync stamp verify in `native-harness-backup-storage.ts`. Both are listed for a follow-up. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (code comments carry the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 00:27:12 -06:00
DottaandClaude	94c7c83ce8	fix(runner): refuse a moved-root same-session checkpoint by name (#20723) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox resume a task's provider session from a saved checkpoint > - #20712 made a checkpoint that names an abandoned remote workspace root decline recovery, so a replacement session opens in the new root > - A checkpoint that requires the same session (an unfinished goal, `same_session_only`) cannot be replaced. It then failed with the generic `native_session_recovery_failed`, which recovery treats as a retryable interruption > - A retry cannot move a saved session to another root, so the failure repeats. The upgrade-safe rule requires a permanent, named code for this case > - Also, the working-directory errors said "Codex" on Claude and other ACPX runs, because one app-server driver serves all of them > - This pull request reports the moved root as provider state that is incompatible, and makes the errors name the agent generically > - The benefit is a named, permanent refusal with a working board Retry, and error text that is correct for every agent ## Linked Issues or Issue Description Refs #20712 **What happened?** After #20712, a `same_session_only` checkpoint with a moved remote root ended with `native_session_recovery_failed: persisted session working directory is not the assigned remote workspace`. That is not a named permanent code, so recovery could retry a run that cannot succeed. On ACPX Claude runs, path errors read "Remote Codex working directory ...". **Expected behavior** The run ends with the permanent code `native_provider_state_incompatible`. Automatic retries stop. A board Retry continues the task in a fresh provider session in the same workspace. Errors do not name Codex on other agents. **Steps to reproduce** 1. Save a native checkpoint with `providerRecoveryPolicy: same_session_only` under one remote workspace root. 2. Assign the run a different sibling root (the epoch was abandoned). 3. Resume the run. ## What Changed - `codex-app-server-driver-impl.ts`: the moved-root decline from #20712 now carries `failureClass: provider_state_incompatible`. The runtime already maps that class: - checkpoint allows replacement (`allow_replacement_after_resume_failure`): a replacement session opens in the assigned root, as before; - checkpoint requires the same session: `NativeProviderStateIncompatibleError`, code `native_provider_state_incompatible`. - The server already treats `native_provider_state_incompatible` as a permanent failure (no automatic retry, board action text "Retry the task to continue in a fresh provider session with the same workspace"). `rebindNativeSessionCheckpoint` returns no checkpoint after that code, so a board Retry starts a fresh session in the same workspace. No server change is needed. - A moved-root checkpoint with an active turn or a pending runtime request now reports `provider_side_effect_pending`. The runtime never replaces it under any policy, and the run ends with the permanent code `native_provider_side_effect_pending` for an operator to reconcile. Before, it threw the working-directory refusal, which recovery recorded as the retryable `native_session_interrupted`. - `harness-driver.ts`: the `PROVIDER_STATE_INCOMPATIBLE` doc now names the moved-workspace case. - `codex-boundaries.ts`: working-directory errors say "Agent working directory", "Remote agent working directory" and "Provider thread response" instead of "Codex". Tests updated to the new text. - Tests: - `native-session-runtime.test.ts`: the real app-server backend with `remote_runner` authority and a moved root, for each recovery policy. Same-session policies end with `native_provider_state_incompatible`, the replacement policy opens a replacement in the assigned root, an in-flight turn ends with `native_provider_side_effect_pending` under every policy, and no transport starts. - `native-backend-factory.test.ts`: the decline carries the failure class, and an ACPX Claude refusal reads "Remote agent working directory does not match the assigned workspace". Previous-release story: no persisted shape changes. Saved checkpoints are classified at recovery time. A saved same-session checkpoint under an old root now ends with the named permanent code instead of a generic one. ## Verification - Red: without the failure classes, the same-session and in-flight runtime cases fail (generic `native_session_recovery_failed`). Before the rename, the ACPX Claude message test fails. - Green: `cd packages/paperclip-runner && npx vitest run src/backends src/drivers/codex src/live/runner-prepared-launch-workspace.test.ts src/native-session-runtime.test.ts` passes (784 tests). - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Existing server coverage for the code: `native-session-resume.test.ts` (no rebind after `native_provider_state_incompatible`) and `native-session-executor.test.ts` (source-code mapping). ## Risks - Low. Error message text changes. A search found no code that matches on the old text outside the updated tests. - Same-session checkpoints with a moved root now stop for a board Retry instead of a generic interruption. That is the intended behavior: a retry cannot succeed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 00:14:03 -06:00
DottaandClaude	08ff0828d6	fix(native): offer Retry at once when the Runner refuses a turn start on a closed session (#20729) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) starts each turn with `turn.start`. When a start is cut off, the Runner reads the sidecar's dispatch journal (#20675). > - If the prompt never left, the Runner keeps the session reopenable, and the same-run resume sends the prompt once. If the prompt may have left, the Runner closes the session, because a resend could duplicate it. > - A closed session cannot reopen for the same run. Every same-run resume got `ACPX session is not open`. The server treated that as a generic interruption and spent three attempts (about 95 s) before the board offered Retry. > - This pull request gives that refusal a closed class in the Runner and a permanent, named failure code in the server. > - The benefit is that the board offers a working Retry on the first refusal, while every recoverable case keeps recovering by itself. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T011704-s9107`, task t05 (local, claude, workload C1, fault TM2). A freeze cut off a turn start, and the Runner saved the session `closed`. Three same-run resumes each failed with `PRP command turn.start failed: … "message":"ACPX session is not open"`. Recovery stopped with `native_session_retry_exhausted` at 01:25:14, and the board then offered Retry. The Retry run succeeded. **Expected behavior** When the Runner proves the session is closed for this run, the run stops at the first refusal with a named failure, and the board offers a Retry that works. A start that the Runner proved never reached the provider still recovers by itself. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-closed-session-retry.integration.test.ts` 2. On master, "offers Retry on the first refusal, and Retry starts a new run" fails: the coordinator is `retryable_failure` / `native_session_interrupted` (a same-run retry in 30 s). Refs #20675, #20720 ## What Changed - Runner (`durable/mod.rs`, `acpx_provider_backend.rs`): new closed class `provider_session_closed`. The Runner sets it on these refusals: - a turn start that saves the session `closed` (`ACPX turn start failed closed: …`), but not the rejected-checkpoint case, which stays `prepared` and reopens; - `ACPX session is not open` when the saved lifecycle is `closed` for this run; - the saved turn-end-restart refusal for this run (asserted in `acpx_turn_end_restart`). A session that waits for provider-exit proof can reopen after a restart. Its refusal gets no class. - Server (`native-session-executor.ts`): a `turn.start` failure with that class maps to the new permanent code `native_provider_session_closed`. The server reads the class from the authenticated command result, and from the command-failure text if the error was rewrapped. `nativeSessionFailureDisposition` treats the code as permanent, so the first refusal ends the run as `terminal_failure` with a board owner and a plain next action. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` states the class and the older-Runner behavior. - Tests: - `native-closed-session-retry.integration.test.ts` (new, embedded Postgres) drives `executePaperclipNativeSession` with the Runner refusal. On the first refusal, the board shows `canRetry: true`. `requestTaskRetry` is accepted, and `resumeTaskRetry` starts exactly one new run with a fresh provider session. A start proved undispatched stays `retryable_failure`. - Unit tests in `native-session-executor.test.ts`: both closed refusals, a rewrapped error, an undispatched start, an unclassified older-Runner refusal, and a class on another command. - Rust `acpx_cut_turn_start`: a start cut off before dispatch has no class. A start cut off after dispatch, and the later refusal, carry `provider_session_closed`. ## Verification - Red on master, green here: step 1 above. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts`: 1358/1358 pass. - `cargo test -p paperclip-runner-core --test acpx_park_preparation` (26 pass), `--lib acpx` (94 pass), `--test acpx_provider_turns --test acpx_provider_session --test acpx_warm_attachment` (all pass), `cargo fmt --check` clean. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - Not run: a chaos wave. ## Risks - This changes the Rust Runner. Daytona needs a new runner image before remote runs get the class. Until then, an older Runner sends the refusal without the class, and the server keeps today's bounded same-run retries. That path is unchanged and ends with a working Retry. The server does not trust the text alone, because an older Runner sends the same text for a session that can still reopen. - The class is set only where the Runner's saved state proves that no later start of this run can reopen the session. A false class would cost one automatic resume and show Retry earlier. It never replays provider work. - No persisted shape changes. The class is an optional field of a failed command result. Old saved state resumes unchanged. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, run in Claude Code with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-11 00:12:41 -06:00
DottaandClaude Opus 5.5	1143abaf92	fix(recovery): give due exhausted holds their own fair lane in the recovery worker (#20725) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The workspace recovery worker saves the files of failed runs. It also re-arms exhausted holds under the rules of #20295, #20569, #20641 and #20648 > - After #20680 and #20661 were deployed to staging, the worker still re-armed none of 29 exhausted holds in 36 minutes. One hold's id sorts in the first percent of all candidates, so the restart starvation that #20680 fixed was not the only cause > - The worker skipped every obligation while any unfinished save was selectable. Discovery is instance-wide. A save that another server owns stays unfinished on every run, so obligations never ran. The obligation pass also put due holds behind hundreds of settled holds whose saved input waits, at two per run. A completed pass never revisited a hold that became due by time alone > - This pull request gives due exhausted holds their own lane, and it shares the worker's slots fairly between unfinished saves, due holds and the obligation pass > - The benefit is that every due hold is reached within a few runs of the worker, whatever its other load. The holds save their files and free their tasks without a person ## Linked Issues or Issue Description Refs #20680 Refs #20661 Refs #20648 Refs #20641 Refs #20569 Refs #20295 **What happened?** On staging, 36 minutes after a deploy that included #20680 and #20661, the workspace recovery worker had touched none of the 29 exhausted holds of the company (22 capture-policy holds and 7 stop-proof or fallback holds). They had no `stopProofRecovery` or `capturePolicyRecovery` marker, no re-arm activity, and `updatedAt` from Nov 2 to Nov 10. The worker did run manual "Recover saved files" requests: one stop-proof hold saved its files 57 seconds after the request. So the worker was alive and the stop proof was available. The holds were not reached. **What did you expect to happen?** The worker reaches each exhausted hold whose re-arm or fallback is due, re-arms it, and saves the files, within a bounded time. **Steps to reproduce** 1. Create a scheduled save whose run's finalization lock another database session holds (another server owns the run). 2. Create an exhausted stop-proof hold and an exhausted capture-policy hold, in the evidence shape that the Nov 4 and Nov 10 releases saved. 3. Start the real workspace recovery worker. 4. The worker retries the busy save on every run and never re-arms either hold. ## What Changed - `server/src/services/workspace-recovery.ts`: `reconcileRetainedWorkspaces` draws candidates from lanes. Each lane is a page that its own cursor advances: unfinished saves; for the worker, exhausted holds whose re-arm or fallback is due; and the obligation pass. Before, the obligation pass ran only when no unfinished save was selectable (the gate at `const unfinished = rows.length > 0; if (!rows.length) { ... }`). - Each lane with work takes one free slot before any lane takes a second. The lane that goes first rotates on each run, so with one free slot (a long local export holds the other) each lane still gets it in one run of three. Actions of a run that one lane took join that run's job without another slot. - The due lane wraps at its tail. A hold that becomes due by time alone (a lease expiry or a re-arm backoff) is reached after the obligation pass completed with no committed change. - `retainedWorkspaceCandidates` takes a `dueOnly` filter (due re-arm or fallback of an exhausted hold). The obligation filter uses the same condition. One `now` is used for the whole query. - The startup sweep (no worker sweep state) also shares its slots between unfinished saves and its general pass. - `doc/execution-semantics.md`: document the lanes and the fairness rule. - New `workspace-recovery-passive-fairness.test.ts` with four real-Postgres regressions. ## Verification - Red on `origin/master` (156212cfd5), green with this change: - "re-arms both staging holds while a save that another server owns stays unfinished": the real worker, a busy unfinished save, and the two staging hold shapes. On master neither hold is re-armed in 20 s. - "reaches due holds within two runs however many settled holds with waiting input sort before them": ten settled holds with waiting user input sort before the two holds. On master the capture-policy hold is not re-armed after two runs. - "re-arms a hold that becomes due after its pass completed, with no committed change": on master the hold is never re-armed. - "re-arms the holds with one free slot while a long local job holds the other and a busy save stays unfinished" (from the Greptile review): red on this PR's first revision (05b261e5fe), green with the lane rotation. - `npx vitest run` of the twelve suites that use discovery or the worker (`heartbeat-observed-workspace-preservation`, `server-startup-feedback-export`, `execution-control-work`, `workspace-recovery-preparation-dispatch`, `-retry-copyback`, `-fallback`, `-warm-handoff`, `-preparation-gates`, `-capture-policy`, `-preparation`, `-stop-proof`, `-admission`) plus the new suite: 646 passed. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input. Required CI supplies the broad coverage. - Staging (read-only GET requests): the 29 exhausted holds, their saved shapes, and the manual Retry timeline. Staging exposes only one company to this account and no worker logs, so the exact unfinished row that starved the pass on staging is not named. The fix removes the whole class: no lane can starve another. ## Risks - Previous-release story: no persisted state changes. Lane positions stay in memory, as the existing cursors did. Holds saved by earlier releases (no marker, no capture policy version) are due at once and are reached in the first runs after deploy, with no migration. A deploy that lands mid-run only resets the in-memory cursors. - Behavior shift: obligations now share the two slots with unfinished saves. With two free slots, a new save or a manual Retry gets one in two runs of three. With one free slot, it gets it in one run of three. - A due hold that its re-arm refuses again is revisited once per due-lane pass. The worker already ran every interval while such a hold was due, so this adds no new wakeups. - One more bounded query per worker run (the due lane, 5 rows). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, code editing, browser GET reads) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-10 23:59:01 -06:00
DottaandClaude	16024ffa2a	fix(native): a same-boot stall never leaves a run waiting on itself (#20721) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep an execution lease. Restart recovery takes over a run only when it can prove that the recorded controller stopped. > - A stall (SIGSTOP) longer than the lease, with no restart, makes the executor lose its own lease. The executor then detaches for "the next controller" and exits. > - The next controller is this same process. Recovery asks the PID probe if the recorded controller is alive. The recorded controller is this process, so the answer is always "alive". > - The task then waits forever: `in_progress`, no Retry, an active run, and an active environment lease. > - A second gap: after back-to-back stalls mid-tool, the executor returns and nothing in this process ever looks at the run again. No terminal state, no recovery and no Retry. > - This pull request lets recovery recognise its own process. When this process no longer drives the run and its own lease lapsed, the prior owner is gone. The existing evidence checks then decide, as for any stopped controller. > - It also makes heartbeat schedule that recovery as soon as such an executor exits. > - The benefit is that a same-boot stall never needs a manual Retry. When provider work may have started, the run fails with a working Retry instead of a wait with no end. ## Linked Issues or Issue Description Refs #20694 (the same class of fix for legacy `preparing` runs), #20529, #20691. **What happened?** Chaos wave `w-20261111T034105-s9330`, task t02 (Daytona, OpenCode, fault TM1). The server process was frozen with SIGSTOP for about 84 s while run `89a3922b` prepared. After SIGCONT, `native.process_start_requested` was recorded and `native.session.execute` failed after 118 ms: the lease renewal found the lease lapsed, and the executor detached and exited. From then on, recovery recorded `native.recovery.transition` "waiting for safe ownership evidence" with reason `controller_still_alive` every 30 s for more than 20 minutes. The run stayed `running`, the issue stayed `in_progress`, no Retry was shown, and the environment lease stayed `active`. Chaos wave `w-20261111T040439-s9331` (Daytona, codex/claude/opencode, T1, TM1). Three stalls of about 90 s came back to back, with less than 1 s between them. Each run was in a tool call. After the second thaw, `native.session.execute` failed for all three runs (lease lost, executor detached). After that, no run had any event: no terminal state, no `native.recovery.transition` and no Retry, for 20 minutes. The in-sandbox turns kept going. A single 90 s stall (wave s9340) stays green, because a later recovery signal happens to find the run. With stacked stalls, that signal can land while the executor is still tearing down, so it skips the run and sets no deadline. **Expected behavior** When this process no longer drives a run that it recorded as its own, recovery takes the run back. No turn reached the runner here, so the bootstrap restarts on the same run. When a turn or runner work may have started, the run fails and shows a working Retry. **Steps to reproduce** 1. Start a native remote (Daytona) run. 2. Send SIGSTOP to the server during preparation, and hold it longer than the execution lease. 3. Send SIGCONT. The executor loses its lease, detaches and exits. 4. See that every recovery scan waits with `controller_still_alive`, and the run never settles. The new tests in `native-runner-restart-recovery.integration.test.ts` ("same-boot controller whose own lease lapsed") reproduce this at the recovery layer. ## What Changed - `claimNativeRestartRecoveries`: when the recorded controller is this exact process (same boot ID and PID): - If this process still drives the run (heartbeat execution or native executor registered), recovery leaves the run to it (`current_controller_preparing`). - If neither drives it and the self lease lapsed, the takeover is allowed (`current_controller_released`). The existing evidence checks then choose the outcome: bootstrap again on the same run when a durable record proves no command reached the runner, or fail with a Retry (`blocked`) when runner or provider work may have started. - An unexpired self lease still waits. A claim that has not reached its executor holds it, and it lapses by itself. - A run that this process handed to the next server (graceful drain `restart_suspended` record, or the in-process hot-restart handoff marker) stays with the next server. - `heartbeat.ts`: when the executor ends with `NativeControllerDetachedForRestartError`, there was no shutdown or hot-restart handoff, and this process's own lease lapsed (`ownNativeExecutionLeaseLapsed`), heartbeat schedules restart recovery for the run when the executor exits. Recovery then reattaches the remote runner whose turn is still running, restarts an unstarted bootstrap, or fails the run with a working Retry. - `claimNativeRestartRecoveries`: a self-owned lease that is still live, with no driver here, now returns a `retryAt` at its expiry, so the wait is bounded. - `heartbeat/shutdown-handoff.ts`: add `isRunHandedOffForShutdown`. Heartbeat recovery passes it to the claim. - `suspendedByCleanDrain` now uses the shared `handedToNextControllerSinceClaim` helper. Behavior is the same. - Heartbeat test: an executor detaches after its own lease lapsed, with no restart. The same process must take the run back. Red before (timed out, the run stayed `running`), green after. - Tests: five cases for the same-boot lapsed lease: bootstrap again with no runner command; fail with a working Retry when a runner identity was recorded; leave the run to a live in-process executor; leave a handed-off run to the next server; still wait for a different live controller. ## Verification - Red before the fix: the three same-boot cases returned `awaiting_evidence` / `controller_still_alive`. - Green after the fix: - `pnpm --dir server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/native-restart-recovery.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts` - Daytona wave `w-20261111T050504-s9331` from this branch (same seed and shape as s9331: 3 tasks, concurrency 3, three 90 s TM1 stalls with about 1 s between them). All 3 tasks reached `done` with no Retry. Run `632f0564` (claude) shows the exact s9331 sequence: `native.session.execute` failed after 216 s, then `reattach_remote_runner` 0.4 s later (before the third stall), then a second reattach after it, and the turn completed. Two tasks are graded FAIL only on the T1 crossing check: their own fault landed after their tool loop ended (for example loop ended 05:11:26, fault 05:11:44). Their status is `done`, with no Retry. - `tsc --noEmit -p server` is clean. ## Risks - A same-process recovery scan can now claim a run that this process recorded as its own. The guard needs all of these: the self lease lapsed, no in-process execution of the run, and no handoff to the next server. A claim that is still on its way to its executor holds a live lease, so it is not taken twice. - Upgrade safety: the decision uses only the coordinator row and in-process state. Old saved rows are judged by the same rules. A new process never matches an old boot ID, so deploys keep the existing proof path. - No new waits or file operations run before `turn.start`. - There is still no ceiling on `awaiting_evidence` for a genuinely different live controller (case c). This PR does not change that policy. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file edit) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 23:55:10 -06:00
DottaandClaude	08ea18085e	fix(workspaces): restore a saved workspace into one copy of sandbox disk (#20695) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A task on a remote (Daytona) sandbox continues from its saved workspace. Before the agent starts, the server uploads the saved archive into the sandbox and a sandbox script extracts it. > - The extraction wrote a full second copy of the workspace beside the uploaded archive. A 7.5 GB workspace on a fresh sandbox then needed about 15 GB, and the restore failed with `stage=extract code=ENOSPC disk=full`. > - The failure was classified as retryable, so every automatic retry and every Retry repeated it on a same-size sandbox. The task could not continue. > - On a retained sandbox it is worse: the old root, the archive, and the new tree all share the disk. > - This pull request makes the restore need about one copy of the workspace. The sandbox removes only data that control-plane storage already holds durably, and it never removes a root whose save the server did not prove. > - The benefit is that a workspace that fits once on the sandbox disk restores. A disk that is too small for one copy now gets a named failure that does not loop. ## Linked Issues or Issue Description **What happened?** A saved remote workspace of about 7.5 GB could not be restored on a new Daytona sandbox. The run failed with `remote_workspace_restore_failed: step=sandbox_restore; remote_workspace_operation_failed: operation=restore exit=1; remote_workspace_restore_failed: stage=extract code=ENOSPC disk=full exit=1`. The run marked the failure as retryable, and each retry failed the same way. **Steps to reproduce** 1. Save a remote workspace whose archive is larger than half of the free sandbox disk. 2. Start the next run on another sandbox, so that the server restores the saved archive. 3. The upload succeeds. The extraction writes the second copy and fails with ENOSPC. **Expected behavior** A workspace that fits once on the sandbox disk restores. A disk that cannot hold one copy gives a specific failure that is not retried automatically, and Retry works after the sandbox has more space. **What the disk holds during a restore (from the code)** - The uploaded archive (`<parent>/.paperclip-runtime/workspace-snapshots/<operation>/workspace.pcws`, about the workspace size, because the archive is not compressed). - The extracted tree (`<root>.restoring-*`, the workspace size). - On a retained sandbox only: the old root, which stayed until after the new root was published. - After a restore that died: its partial tree and its archive, which no later attempt removed. Refs #20678 (the save side of the same full-disk class). ## What Changed - `workspaceArchive` restore has a `consumeArchive` mode. It creates the directories and symlinks first, then writes the file bodies from the last to the first. After it verifies each file against the manifest hash, it truncates the archive behind that file. The archive and the tree together need the workspace size plus the largest file. The default mode, which the server uses for local restores, does not change. - The sandbox restore script uses that mode on a private rename of the uploaded archive. A later upload to the same path cannot change the file that is being consumed. The script removes the consumed file when it stops. - The helper's own staging is now inside the root-specific `<root>.restoring-*` folder, not in a shared `.workspace-restore-*` sibling. - While it holds the root's restore lock, and after the epoch fence, the script removes leftovers of an earlier attempt on the same root that died: `<root>.restoring-*` folders, `.consuming` archives in its own staging, and archive staging folders that a `restore-target.json` binds to the same root. Each restore writes that binding after the reclaim step. - When the server proved that a retained root is its own earlier, fully saved allocation (`replaceRetiredRoot` and a Paperclip receipt, as before), the script removes the root's files before the extraction, not after the publish. The receipt and `.paperclip-runtime` stay, so an interrupted restore proves the same root again and continues. - A full disk at any restore stage reports `reason=sandbox_disk_full`. The existing named-refusal rule makes it non-retryable, so it goes to the board recovery action and does not loop. - `doc/execution-semantics.md` describes the disk need, the reclaim, the early removal, and the new reason. ## Verification - `cd server && npx vitest run src/services/remote-workspace-transport.test.ts src/services/remote-workspace-snapshot-archive.test.ts` (287 passed). - `cd server && npx vitest run src/services/remote-workspaces.integration.test.ts src/services/remote-workspace-lifecycle.test.ts src/services/remote-workspace-archive-worker.test.ts src/services/remote-workspace-snapshot-archive-environments.test.ts` (122 passed, 16 skipped). - `cd server && npx tsc --noEmit -p tsconfig.json` (pass). - The new tests run the real sandbox scripts against a folder with a byte capacity. A `--require` preload makes each file write fail with ENOSPC when the folder would exceed the capacity. - Red on `origin/master`, green with this change: - Fresh sandbox, room for the archive plus one file: master fails with the exact production error `stage=extract code=ENOSPC disk=full exit=1`. The fix restores every byte and leaves only the restored files. - Retained root that the server proved, room for the old root, the archive, and one file: master fails with ENOSPC. The fix restores and keeps the runner state. - Too small for one copy: the failure is `stage=extract reason=sandbox_disk_full`, with no partial tree or archive left. A retry with more space then restores. - Both sandbox processes are killed mid-extraction (a restart): a retry with the same identity and a retry with a new identity and epoch both reclaim the leftovers and restore into the same capacity. - The command dies right after it removed a proven root's files: the next restore proves the root again from its kept receipt, restores, and keeps the runner state. - Integration test through `prepareRemoteWorkspace`: a full disk gives `retryable: false` with `reason=sandbox_disk_full`. On master the same case is `retryable: true`. - Guards (pass on master and with the fix): a retained root without the server proof, or without a Paperclip receipt, is refused with `reason=occupied_root`, and none of its files are removed. - Previous-release story: the archive format and the saved state do not change. A partial `<root>.restoring-*` tree from the previous release is reclaimed (test). Archive staging from the previous release has no binding, so it is kept (test). The previous release's shared `.workspace-restore-*` siblings cannot be bound to a root, so they are kept. ## Risks - The consumed archive cannot be read again. Each restore uploads a new copy from control-plane storage, and the early-return path for an already restored root does not read the archive. Every file is still checked against the manifest, and the manifest is checked against the saved digest. - The reclaim step removes data only under the root's restore lock and after the epoch fence. It removes only names that a restore of the same root creates. Staging that names another root, or no root, is kept. - The early removal of a retained root depends on the existing server proof (`isRetiredOwnAllocation`) and the receipt check. If a restore fails after the removal, the saved version in storage is the source of truth, and the next restore proves the same root again. - Files in a lower overlay layer of a resumed sandbox image do not free space when they are removed. Such a sandbox can still be full. - A workspace that is larger than the free disk after the upload, plus its largest file, still fails. It now fails with the named, non-retryable `reason=sandbox_disk_full`. - No change to `packages/paperclip-runner`, the database schema, or the API. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 22:19:43 -06:00
DottaandClaude	855e241887	fix(server): record a long event-loop block with the deltas that name its cause (#20709) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process owns the controller leases of its runs, renews them on timers, and serves every request > - On a hosted instance, the process stopped for about 103 seconds without a restart. Every lease lapsed, and two runs needed a manual Retry (the symptom is fixed in #20694) > - Nobody can say what blocked the process. The server records no event-loop delay, and it has no CPU, GC or memory record for that minute > - This pull request logs one local line for each event-loop block of 2 s or more. The line holds the deltas that tell the possible causes apart, and a likely-cause class > - The benefit is that the next stall names its class (main-thread CPU, GC, paging, CPU throttling, host suspend, or a blocking sync call) and its exact time, so we can match it to the run timeline and fix the cause ## Linked Issues or Issue Description Refs #20694 **What happened?** On a hosted instance, the merged run-event timeline of all active runs had no events from 08:19:52.04Z to 08:21:35.37Z (103.3 s). Events from streaming runs arrived about every 0.5 s before and after the gap. The process did not restart (the run diagnostic shows an observer uptime of about 24 minutes when the reaper ran). The first event after the gap was a span with a 60 s timeout that ended after 103.5 s. Thus the timer callback ran about 43 s late. This agrees with a blocked event loop, or with a frozen process. Nothing in the process recorded which one. **Expected behavior** A long event-loop block is a product failure. It lapses every lease and freezes every user. The server must record each long block with enough data to identify its cause. **Steps to reproduce** 1. Start the server. 2. Block the main thread for more than 2 s, for example with a synchronous read on a slow disk or a busy loop. 3. Before this change: the logs show nothing. After this change: the logs show `event loop blocked for <n> ms (likely <cause>)` with the measured deltas. ## What Changed - `server/src/event-loop-stall-monitor.ts` (new): one unref'd 1.5 s timeout, re-armed after each check, measures how late it runs. (Server startup creates no interval timers, so it is not a `setInterval`.) When a check is 2 s or more late, it logs once with these fields: - stall duration, start and end time (wall clock and monotonic clock) - process CPU (user and system), GC pause time and count (`perf_hooks` GC entries) - major and minor page faults, file-system reads and writes, voluntary and involuntary context switches (`process.resourceUsage()`) - cgroup v2 `throttled_usec` delta (Linux; read asynchronously, best effort) - RSS, heap and external memory - `likelyCause`: `host_suspended`, `gc`, `memory_paging`, `cpu_throttled`, `main_thread_cpu`, or `blocked_off_cpu` (sync file I/O, a sync child process, or another off-CPU wait) - `server/src/index.ts`: starts the monitor when the listener binds. It stops the monitor at the start of shutdown, and when a failed startup closes the listener. It writes the line with the existing `logger.warn`. - This is a local log line only. It is not the Telemetry path and not the Observability (OpenTelemetry) path. No data leaves the process. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/event-loop-stall-monitor.test.ts src/__tests__/server-startup-feedback-export.test.ts` (131 passed; the monitor file passed 5 runs in a row): - an 800 ms busy loop is recorded as `main_thread_cpu`, with the stall and CPU time - an 800 ms `Atomics.wait` (blocks without CPU, like a sync read on a stalled disk) is recorded as `blocked_off_cpu` - nothing is logged while the loop is responsive, or after `stop()` (injected clock) - the file-read delta of a stall is kept while the cgroup read is pending - server startup still creates no interval timer - a 100 s stall with 90 s of cgroup throttling is recorded as `cpu_throttled`, with the stall start and end times - each cause class is selected from its deltas - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - CI supplies the broad server suites. ## Risks - Low risk. The timeout reads only process counters. Every 10 s it reads one small cgroup file asynchronously. It does not query the database or the network, so it does not keep an idle instance or its database awake. It is unref'd and stopped on shutdown. - `process.cpuUsage()` counts all threads of the process, not only the main thread. A busy libuv thread pool during a stall can make an off-CPU block look like `main_thread_cpu`. The raw deltas are in the line, so a reader can check this. - If a platform suspends and resumes the process, each resume can log one `host_suspended` line. This is expected and useful. - The monitor names the class of a block, not the call stack. If the class is not enough, the next step is a bounded CPU profile around the stall time. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the module comment carries the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 22:19:13 -06:00
DottaandClaude	200c6e40e2	fix(native): restart a bootstrap whose Runner suspended before the provider opened (#20691) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run's first provider start fails before any session opens, Paperclip retries the bootstrap on the same run. Durable controller evidence proves that no provider session or provider event exists. > - A remote sandbox keeps the Runner state inside the workspace. A replacement sandbox gets that state back when the saved workspace is restored. > - When the provider refused `session.open` (for example an Anthropic 429), the Runner still suspended cleanly. Its state says `suspended`, but the provider state has no session identity. > - The same-run restart handled a Runner that never suspended. It did not handle a Runner that suspended before the provider opened a session. It failed with `runner_harness_state_mismatch: provider_identity_incomplete (provider_not_suspended)`, and the task showed "Recovery needed" without Retry. > - This pull request lets that proven restart set the unopened session aside and start the provider again. > - The benefit is that a rate-limited provider start continues by itself after the limit resets. ## Linked Issues or Issue Description Refs #20627 and #20683 (other restore paths). This PR changes only the same-run bootstrap restart. **What happened?** Two ACPX Claude tasks on remote Daytona sandboxes, on staging: 1. The first attempt sent `session.open`. ACPX rejected it with `classification=provider_rate_limit` (Anthropic 429). The Runner exited with `lifecycle=suspended`. 2. Paperclip recorded `recoveryMode: bootstrap_retry`, `providerSessionEstablished: false`, `providerEventsExist: false`, and a retry after the limit reset. 3. One hour later the retry ran as `bootstrap_incomplete` on the same run, in a new sandbox with the saved workspace restored. 4. Host preparation inspected the restored Runner state. The Runner was `suspended`, but the provider had no session identity. The inspection threw `provider_identity_incomplete (provider_not_suspended)` before the bootstrap restart branch ran. 5. The run failed. The task stopped at "Recovery needed" with only inspect actions. **Expected behavior** The restart sets the unopened session aside and opens a new provider session, as it already does for a Runner that never suspended. **Steps to reproduce** 1. Restart a remote run as `bootstrap_incomplete` with a proven bootstrap restart. 2. Give the sandbox a Runner state for this run with `lifecycle: suspended` and a provider state with no session identity. 3. Prepare the host. On `master` it throws `runner_harness_state_mismatch: provider_identity_incomplete`. The new `interrupted_bootstrap_unopened` cases of `bootstraps only an untouched provider session in a $outcome workspace lease` run these steps for a created and a resumed lease. **Paperclip version or commit** `master` at `7435abcc63`. ## What Changed - During host preparation, a Runner state that is suspended with no provider session identity returns the new incomplete reason `provider_unopened`, but only when all of these hold: - the run is restarting as `bootstrap_incomplete`; - the state belongs to this same run; - the controller already recorded the bootstrap restart proof (its durable journal shows no provider turn, and the old Runner is stopped or absent). - the provider state is the one saved before any session opened (`remoteProviderStateNeverOpened`): `prepared`, no session or identity, no turn in progress, no unconfirmed exit. Only ACPX and Codex-family states qualify. - The created-lease and resumed-lease restart branches share one check (`restartableBootstrapState`) and accept `provider_unopened` exactly like `not_suspended`. They move the old session aside and claim a new, untouched session root. - Every other case still throws `provider_identity_incomplete`: another run's state, a restart without the proof, a normal resume, and close checkpointing. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "bootstraps only an untouched provider session"`: 47 passed. - Red on `master`: the `created` and `resumed` `interrupted_bootstrap_unopened` cases fail with `runner_harness_state_mismatch: provider_identity_incomplete (identity_binding)`. Green with this change. - Fail-closed cases stay red-proof: `interrupted_bootstrap_unopened_unproven` (no restart proof) and `interrupted_bootstrap_unopened_other_run` (another run's state) still throw. - Full `native-session-executor.test.ts` and `pnpm --filter @paperclipai/server typecheck` pass. ## Risks - Low risk. The new branch needs the same proof that already authorizes a fresh bootstrap for a Runner that never suspended. The proof comes from the controller's durable journal, not from the sandbox. - The normal path is unchanged and has no added wait. - Not proven live: I did not confirm on staging that the controller recorded the restart proof for these runs. If it did not, the run still fails closed as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-10 21:35:38 -06:00