mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
e05bced39fe8e8bafe1d27bbf5fdcee99ad048af
571
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
785ead5c7a |
fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
a0ac81e114 |
fix(ui): show task image results in compact thumbnail grids (#19626)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task feeds show agent messages, documents, and files in time order. > - Each image currently uses a full feed card. > - A turn with seven screenshots makes the conversation hard to scan. > - This pull request groups adjacent images from the same run in a compact grid. > - Users can scan the results and open each image in the existing gallery. ## Linked Issues or Issue Description **What existing behavior does this improve?** Image results in task history and the live task feed. **Current behavior** Each image uses a separate full-width card. Native image upload receipt messages can also separate related screenshots. **Proposed behavior** Adjacent images from one run share a wrapping thumbnail grid. New image results join the grid. Documents, video, and authored messages keep their positions. The grid opens the existing gallery. **Reason and benefit** Users can see a batch of screenshots without scrolling past one large card for each file. **Breaking changes** The feed layout changes. The stored comments, files, APIs, and permissions do not change. Related public work: Refs #17228. Refs #8785. Refs #7667. These changes supplied the artifact cards and media gallery that this pull request reuses. ## What Changed - Add one image grouping function for settled history, turn history, and live results. Group only images with the same known run. Keep the first row key and each image key when results arrive. - Add a token-based grid that reuses the media artifact cards and image gallery. - Add an optional typed attachment-preparation marker to file registration and chat-file reuse. Preserve the marker through comment normalization. Use it with matching image, author, and run links to hide generated receipts. Keep the exact historical sentence as a fallback for old receipts. - Add nine production component stories with seven sample-account screenshots. Cover mobile, Classic, mixed results, single images, missing images, streaming, gallery navigation, and the complete feed projection. - Add grouping and production-feed regressions. Add 18 browser cases across light and dark themes. - Improve contrast for the shared fallback label and gallery counter after the accessibility checks found low contrast. - Match alternate original-image paths when opening the gallery. Keep a separate Open original link beside Download in the gallery. - Record the feed layout in DESIGN.md. ## Verification - Final qualification at `09b12381ce85bee6593050bd8a2fa18cbe2c45a6`: [required CI passed](https://github.com/paperclipai/paperclip/actions/runs/42317080212/job/128237094513), including typecheck, build, all test lanes, E2E, and verify. The snapshot has 58 successful check runs, three conditional skips, one neutral advisory job, and a successful legacy Snyk status. No pending or failed checks remain. Greptile and Paperclip Review both score this exact head 5/5; all review threads are resolved. - Current-head local checks: 544 focused UI tests passed across 11 files, with one existing skip. UI/shared typechecks and token gates pass. All 18 focused Storybook browser cases passed after the CSS merge, with retries disabled. - Two database integration cases verify file registration and chat-file reuse receipt markers. All 52 issue-validator tests and six upstream default-model tests passed. - Regressions failed before the fixes for unknown-run grouping, receipt wording, and authored progress text with an image. The repair uses a typed receipt marker and requires matching author, attachment, image, and run links before hiding a generated receipt. All five review findings are resolved. - Browser checks cover loaded images, wrapping columns, overflow, missing-image fallback, streaming stability, keyboard gallery navigation, original-file links, and downloads. Fixtures use production renderers. These checks do not prove deployed persistence or provider behavior and do not compare visual baselines. - Local limits: the preflight helper could not finish because Cargo was absent. The earlier low-memory server typecheck and dependency-free selector did not replace final-head CI. The now-passing final CI supplies broad typecheck, build, server, Runner, and E2E coverage. - Prior independent CI failures remain recorded: one Slack timer case passed a single diagnosed rerun; its exact timer cause was not proved. A later chat shard lost its self-hosted runner without a test assertion failure. The current revision passes that shard and all applicable gates. Master conflict resolution retained both image-grid and patch CSS rules. ## Risks - Images without a run ID remain separate. Messages and other resource types end each batch. - Receipt suppression requires a matching preparation marker or the exact legacy sentence, plus matching attachment, image work product, run, and author. Tests preserve human messages and native progress text with attachments. The optional marker uses existing JSON metadata and needs no database migration. - Shared gallery and fallback text now have stronger contrast. ## Model Used OpenAI GPT-6, through Codex. The agent used reasoning, repository tools, code execution, and browser tests. The runtime did not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
72f7b42fe3 |
Preserve settled failed controller state during online migration (#19678)
## Thinking Path
> - Paperclip controls agent work across web-server releases.
> - Volume-free servers need retained controller state in shared
storage.
> - Successful settled sources already support online migration.
> - Some retained failed sources have no accepted result and must stay
available for recovery inspection.
> - This change copies those sources under task, coordinator and
publication fences while unrelated work continues.
> - The original files, failed status and recovery warnings remain
unchanged.
## Linked Issues or Issue Description
Refs #19038. Refs #19631.
**What are you trying to improve?**
Preserve retained failed controller files without waiting for global
idle.
**What problem does this solve?**
The online producer skips terminal failed controllers. Their files can
still be needed for recovery inspection even when no accepted provider
artifact proves executable continuation.
**What would the improvement look like?**
A settled exact source can enter the existing encrypted archive store.
An accepted committed failed source can use the existing checkpoint
import. Busy, changed, unowned or foreign sources remain blocked.
## What Changed
- Add a source owner bound to the selected task session, run,
coordinator generation, admitted input and file digests.
- Check the task, physical finalizer lock, released coordinator,
canonical resume source and active related turns before upload and
before publication.
- Preserve failed sources through the existing encrypted archive format.
Do not treat archived bytes as execution authority or old-writer stop
proof.
- Allow the existing accepted committed checkpoint import for a finished
failed run without changing its status.
- Validate admitted input bindings before looking up local controller
files.
- Document the online preservation path and the final static writer
fence.
## Verification
- Real PostgreSQL: 125 cases passed across migration, controller
recovery and durable backup storage. One added assertion inspected an
unadmitted queued fixture; it was corrected while retaining the
queued-turn preservation and stale-publication assertions. The final
migration rerun passes 51/51; the other three affected suites pass
75/75.
- Genuine prepatch RED: online failed-source preservation and accepted
committed failed-source import both fail on the previous producer and
recovery owners.
- New cases cover independent restore after deleting the original
source, an expired but non-null coordinator owner, source mutation after
upload, a new related queued turn during upload, foreign input binding
and an accepted committed failed result.
- The full rebased server source and test-tree typecheck passes after
restoring its missing ignored workspace SDK link and building that
existing dependency.
- Module boundaries and whitespace checks pass.
- Rebased onto current master; both provider-artifact imports and the
new archive owner are preserved. Rebased type and module-boundary checks
pass. All six archive cases pass on the exact rebased head (39.68
seconds). All CI gates pass on exact head
|
||
|
Nicky LeachandPaperclip
|
d6f4155bad |
Refresh checkpoint writers when controller ownership changes (#19679)
## Thinking Path > - Paperclip manages AI agents and keeps their work attached to tasks. > - Core saves native Runner checkpoints so a session can continue under a later run. > - The checkpoint records can still belong to source run A after run B becomes the current owner. > - The writer cache currently keeps A's first write authority even after Core validates B. > - This pull request renews that writer when the authority, boot, generation, or lease owner changes. > - Repeated registration under the same authority keeps its work history. Stale and expired owners still cannot write. ## Linked Issues or Issue Description Refs: #19611. Related: #19225. **What happened?** A checkpoint writer is cached by source run ID. After Core validates a successor owner, a new registration still uses the cached writer for the old owner. Saving new work can then fail with `native_runner_checkpoint_owner_lost`. **Expected behavior** Use the validated current owner for later writes. Preserve the existing writer when its authority has not changed. Continue to reject stale or expired ownership. **Steps to reproduce** 1. Register a source checkpoint writer and save its first command. 2. Complete the source run and give its task to a successor with a current write lease. 3. Register the same source checkpoint under the successor and save a new command. The unchanged code rejects this write. ## What Changed - Store the write-authority fields with each cached writer. - Replace the writer only when its authority run, boot, controller generation, or lease owner changes. - Add a PostgreSQL regression for successor handover, repeated registration, generation renewal, stale writes, and expired leases. ## Verification - The new regression fails on base `913102870d` at the successor's checkpoint write. - The fixed regression passes: `cd server && node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-runner-checkpoint-authority.integration.test.ts`. - A focused strict TypeScript check covers the source, regression, and their imports. - The maintained general-server selector includes the regression. It needs no native process or new CI group. - The scoped diff passes whitespace and changed-line secret/PII checks. - Targeted database and focused strict TypeScript checks pass. Required CI will supply full regression coverage. - CI and standard review are pending for this draft. ## Risks A renewed writer starts a new in-memory work history. The authority comparison limits that reset to a real ownership change. Each database write still checks the current lease and generation in its transaction. This change adds no schema, API, or automatic session-release behavior. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, with repository inspection, code execution, and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes N/A: This internal correction changes no documented interface. - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a3eecf1fc9 |
Preserve admitted runtime skill bundles during server replacement (#19675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep the exact skill files admitted for execution and recovery. > - Older inputs can name immutable local bundles without a skill version ID. > - Installed skill heads alone do not preserve those admitted bundles. > - Server replacement needs a durable source for each retained bundle. > - This pull request copies authenticated bundles through the existing package owner and restores their exact projections. > - The benefit is preservation during online migration without stopping active work or settling recovery warnings. ## Linked Issues or Issue Description Refs #19656. Related: #19660. **What happened?** Older native execution inputs can contain skill bundles with a null version ID. Recovery skips those bundles. Startup installation checks can pass even when those retained inputs still require the old filesystem. **Expected behavior** Preserve each retained admitted bundle in object storage. Restore its original bytes, modes and digests on an empty replacement host. Keep its null pin and all run, controller and recovery state. Block startup and enrollment while a concrete local asset obligation remains. **Steps to reproduce** Retain a native run with an admitted local skill bundle and a null version ID. Convert its installed skill head to durable storage. Replace the server home and attempt recovery. The old bundle path remains in the recovered context. ## What Changed - Add a bounded `runtime-assets` phase to the existing skill package maintenance CLI. It copies exact admitted files under the runtime asset publication lock, verifies the durable package, and appends a run-bound receipt after source revalidation. - Keep original files, null pins, task intent, warnings and controller leases. Support an active run through copy-only preservation. Do not start a turn or claim provider ownership. - Verify and reuse a healthy canonical local bundle before storage reads. Restore missing or damaged bundles through the existing package and native asset materializers. Use the authenticated original manifest to preserve exact read and execute modes. Accept a later contract revision only when the run binding and exact skill selection remain unchanged. - Check retained recovery obligations and current task sessions during both startup and migration eligibility. Ignore ordinary admission that has no completed execution input. - Protect receipt-backed packages from orphan collection. Serialize receipt publication and deletion on the package row, then check references in a fresh statement after the lock. - Add real PostgreSQL and filesystem regressions. Replace a delete-method proxy with an actual upload-renewal row-lock race. Publish mutation activity only after commit. ## Verification - Baseline production startup gate failed the new concrete-asset regression as expected. The fixed gate passed. The actual before-fix preservation owner also failed recovery from a healthy local bundle during a simulated storage outage. A verified local-first path fixes that regression. - Eight preservation tests passed with real PostgreSQL, streaming storage, archive extraction and native materialization. They cover cold recovery, exact modes and binary bytes, active ownership, changed sources and bindings, contract revisions, missing or corrupt receipts, and publication/collection races. - Six affected package collection/deletion controls and three existing startup controls passed. The existing versioned recovery consumer passed a sequential retry. - One fixture attempt failed on its cleanup foreign-key order; the fixture was corrected. One existing suite hit its unchanged 30-second database startup hook before test execution. The unchanged single consumer passed on retry. These failures remain recorded. - Source, changed-test and CLI TypeScript checks passed. Rebase preserved both the local instruction-copy and runtime-asset guards. Current-head changed-test checks passed. Final PostgreSQL rerun passed all eleven preservation/startup cases. Current CLI check passed. Required CI remains pending at the trusted-runner gate. - Current canonical bundled creation uses package preparation plus atomic content/head publication. Native preparation requests immutable version snapshots before delivery. Converted installations and new canonical bundled creation therefore do not recreate the historical null-version asset dependency. - Fresh Greptile review on exact head `54c7dd048c9016c1af9fc68213d5ae053ac16499` passed at 5/5 with zero unresolved threads. - Module boundaries, whitespace, and credential/internal-reference scans passed. No live source writes, provider actions or migration apply were performed. ## Risks Missing, changed or unsafe local files remain blockers. This command does not repair a previously published package whose objects later disappear or become corrupt. It also refuses malformed preservation receipts. It retains the original local files and prior evidence and keeps removal blocked. Automatic repair requires a separate canonical package-owner recovery operation; direct database edits are not an accepted recovery path. Failed publication keeps the original files and any unreferenced package for normal collection. The readiness check is explicit startup or enrollment work, not a background scan. Its cost grows with retained admitted asset obligations. Older readers must be upgraded before the old volume is removed. This change preserves asset bytes; it does not settle business recovery or qualify unrelated volume dependencies. ## Model Used OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend model ID or context-window size. Used reasoning, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6c7cc2c5d6 |
fix(native): do not spend a recovery attempt resuming a cleanly drained run (#19625)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A graceful shutdown (deploy or SIGTERM) drains native runs: it suspends them for reattachment by the next server > - When the runner did not survive, the next boot resumed the turn and charged one attempt > - Restarts that the run did not cause used up its budget, so recovery stopped and the task needed a manual Retry > - This pull request does not charge an attempt for resuming a run that a graceful drain suspended > - The benefit is that deploys and graceful restarts no longer exhaust a healthy run's recovery budget ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, native runs reached `execution_recovery_budget_exhausted` (3/3) only because of server restarts. In one wave, two restarts crossed each of two runs. In a second schedule (K1, then K2 when ready, then K2 again), each drain correctly suspended the local ACPX Claude run, yet the fourth boot blocked it with `native_restart_recovery_blocked`. Every resume after a drain was classified `resume_dead_runner` and charged an attempt (`nextNativeProviderAttempt` in `native-restart-recovery.ts`, applied by the executor's execution claim). **Expected behavior** A restart that the run did not cause should not consume its recovery budget. A graceful drain records `restart_suspended` and `awaiting_runner_reattach`. The resume that follows should keep the attempt count. **Paperclip version or commit** `master` at `4ef3fa4271`. **Steps to reproduce** Start a native local run. Send the server SIGTERM three times while the run is active, with a full restart each time. The third resume is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `suspendedByCleanDrain()` in `native-restart-recovery.ts` recognizes a run whose most recent recovery claim is older than the drain's `restart_suspended` entry. Ownership waits and deferred blocks between the drain and the next claim do not erase it. The next claim consumes it. - A scan that waits for the old controller to exit (`awaiting_evidence`) records one history entry per reason, so repeated scans cannot push the drain record out of the bounded history. - Restart claims carry `chargesProviderAttempt`. It is `false` for a claim that follows a clean drain. - `nextNativeProviderAttempt()` takes that flag. The executor passes it when it takes the execution lease, so a drained resume keeps the current attempt. - The first execution is always counted: a run drained at attempt 0 (no execution claimed it yet) still charges its first provider start. - A run drained during its third attempt may continue that attempt. The local resume and remote bootstrap gates skip the `>= 3` exhausted check only for a non-charging resume, and the executor still refuses a fourth attempt. - A crash restart (no drain record) still charges one attempt. Reattachment to a live runner still charges none. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification - New test in `server/src/__tests__/heartbeat-native-restart-suspension.test.ts` (embedded Postgres, real heartbeat drain and restart recovery): - Red before the fix: the claim after the drain had no `chargesProviderAttempt: false`. - Green after the fix, for drained attempts 0, 1 and 3: the crash claim charges (`true`). After the drain, a controlled backend lets the executor take its execution lease and records the saved `native_run_finalizations.attempt` at provider dispatch: 1, 1 and 3. Attempt 3 is claimed instead of blocked. - Red at the executor: when the executor ignores the flag, the drained attempt-1 resume saves 2. - Ownership-wait case: the drained controller (the test process) is still alive, so two scans wait. The history keeps `restart_suspended` followed by one `awaiting_evidence` entry. After the controller is gone, the claim does not charge, and attempt 3 continues. - Classifier cases (local resume and remote bootstrap): attempt 3 after a crash is still `execution_recovery_budget_exhausted`. Both new cases were red on the previous revision. - `heartbeat-native-restart-suspension.test.ts` 12/12, `native-runner-restart-recovery.integration.test.ts` and `native-runner-ownership.test.ts` 94 passed. - `tsc --noEmit` for `server` passes. ## Risks - A run that itself makes the server shut down gracefully, again and again, would no longer be stopped by this budget. Graceful shutdowns are operator or deploy actions, so this is unlikely. A crash loop (SIGKILL, OOM) still charges an attempt on every restart, and a resumed turn that fails still goes through the existing failure and replacement budget. - This changes only the attempt count. Ownership checks, checkpoint verification and the exhausted gate are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
56c2b28c04 |
Settle saved monitor results after a newer user request (#19644)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native results must finish their durable finalization before later
work can reuse a saved workspace.
> - An accepted result can wait on a monitor while its workspace files
are recovered.
> - A newer user request can replace that old wait before finalization
resumes.
> - The old finalizer then reports lost monitor authority and blocks the
saved-workspace lineage.
> - This pull request proves the newer user request and settles only the
old result bookkeeping.
> - The benefit is released recovery work without replaying an old turn
or changing the current request.
## Linked Issues or Issue Description
Refs #17641 and #16975.
**What happened?**
An older successful native result has saved files and a server-issued
monitor receipt. Its task is now blocked after a newer user-comment
continuation failed during preparation. The old coordinator stays in
retryable_failure with monitor_wait_authority_lost. The retained
allocation ownership check correctly refuses the unresolved predecessor,
so later file recovery also stays held.
**Expected behavior**
The normal finalizer can settle the old accepted result when exact newer
user-request evidence proves the old wait is superseded. It must
preserve the current task, user input, ownership, schedule, accepted
result, and attempt history. It must not start provider work or publish
the old wait summary.
**Steps to reproduce**
Create an accepted successful monitor result and mark workspace
finalization successful. Leave the task blocked. Record a newer
canonical user comment and its ordinary user wake, either queued or
consumed by a same-task continuation that retains the exact comment
revision and source-run lineage. Call the real native finalizer. Before
the repair, both cases remain retryable_failure instead of settling.
**Paperclip version or commit**
|
||
|
DottaandClaude Opus 5.5
|
e89be6ee83 |
fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - After a restart, recovery gives a run with no provider evidence the `bootstrap_incomplete` disposition. The executor quarantines the old root and starts the bootstrap again, but only when the root proves that no provider work started. > - #19524 made that proof accept a root killed after `run.prepare` and before `session.open` (a hard kill). > - A graceful restart (SIGTERM) leaves different shapes. The drain sends `runner.drain` and `runner.suspend`, and the kill can land just after `session.open` failed or just after it started a fresh session. The proof refused both shapes. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, although no turn was ever sent. > - This pull request extends the proof to both drain shapes and keeps provider work fail-closed. > - The benefit is that a graceful restart during native bootstrap continues automatically. ## Linked Issues or Issue Description Refs #19524 **What happened?** Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after the run started) failed with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete`, and the executor proof refused the state root. Two real roots were kept: 1. `session.open` failed (the ACPX sidecar timed out), then `runner.drain` completed and `runner.suspend` completed. There was 1 connection, no committed or acknowledged event, an empty runner outbox, and `nextSourceSeq: 1`. 2. `session.open` completed with a freshly started session (`status: "started"`). The committed events were only `session.started`, `session.capabilities.updated` and `session.goal.snapshot`, and the runner outbox held one `harness.diagnostic` (the sidecar process). No turn command existed, and the sidecar process had exited. The database had not yet recorded the session events, so the classifier saw no provider evidence. **Steps to reproduce** 1. Start an ACPX Claude task on a local runner. 2. Send SIGTERM to the control plane about 10 s after the run starts, before the first turn. 3. Let the control plane restart. **Expected behavior** Recovery starts the bootstrap again, and the task continues without a user action. **Actual behavior** The run fails and the task stops. ## What Changed - `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that is pending or `failed` is now accepted, as are the runner lifecycle commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any state. The controller journal and the runner journal must both agree. Any committed or acknowledged event, any turn command, a completed `session.open`, and any queued runner event still fail closed. - New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed `session.open` only when all of these are true: - the session was freshly started (`status: "started"`, not resumed); - the commands are only `run.prepare`, `session.open` and runner maintenance commands; - every committed event and every event in the runner outbox is a session setup or runner lifecycle event; - `ackedSourceSeq` equals the count of committed events, with no gap; - the runner journal is present; - every recorded provider process ID has exited on this host. - `native-session-executor.ts`: the executor applies the opened-session proof only for a local execution target. Remote runners stay fail-closed for that shape, because their process IDs belong to another host. The remote checkpoint classifier still reads only commands 1 and 2. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain suspended a runner whose session.open failed" and "allows a fresh bootstrap after a drain that landed between session.open and the first turn" both failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 32/32. - Fail-closed counter-tests: - completed `session.open`; - committed event, acknowledged event, or queued runner event; - turn command, `turn.stop`, or a turn processed by the runner; - `turn.started` or `provider.event`; - resumed session; - compacted events; - no recorded provider process; - missing runner journal; - live provider process; - remote runner. - I ran the new proof read-only against the three real kept roots: the K1 root from #19524, the K2 `session.open` failed root and the K2 opened-session root. All three return `true` for a local runner. The opened-session root returns `false` when the processes are on another host. - `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to bootstrap, restart, remote and drain: 333 passed. `native-runner-restart-recovery.integration.test.ts` and `native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos replay: see the PR comment. ## Risks - After a `session.open` failure or timeout, the provider sidecar might have created an empty session. No turn command was sent, so no provider work can repeat. - An opened session before any turn is discarded, and a new one is started. The proof requires a fresh `started` session, so no earlier conversation is lost. Resumed sessions stay fail-closed. - The opened-session proof is local only. It requires every recorded provider process to be gone, so a live provider process blocks it. - Out of scope: the Retry offer when the proof fails legitimately (RC-05/06). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9197abf673 |
fix(native): preserve controller state when remote archiving fails (#19663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. **What happened?** A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. **Expected behavior** A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. **Steps to reproduce** 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0819f4ea78 |
fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task turns can keep a provider conversation from an earlier run. > - A copied conversation checkpoint names the new run before its controller has attached. > - A failed attachment can make recovery treat that copied identity as current execution authority. > - The current run needs its own durable controller checkpoint, or a valid staged attachment, before exact resume. > - This pull request checks that ownership and uses the normal Retry path to continue a saved user turn after verified cleanup. > - The task can continue once with its saved work and history. ## Linked Issues or Issue Description Refs #19399 and #18860. **What happened?** A new user comment resumed a completed native conversation. The retained runner process was gone before the new controller attached. The new run still had a copied provider session ID. Recovery selected exact resume for that new run and then failed its unchanged owner and lease checks. Ordinary first-attempt failure settlement also omitted the saved-user Retry handoff that restart recovery already uses. **Expected behavior** Exact resume requires current controller evidence. A saved user turn can request one normal Retry after this infrastructure failure. The normal path must verify provider stop, saved files, current user authority and all execution gates before it starts a fresh session. **Steps to reproduce** 1. Finish a native warm run and retain its provider checkpoint. 2. Admit a user comment as a new run and copy the provider checkpoint through the real resume producer. 3. Stop the retained runner before attachment. The real remote process check reports that the process is not running. 4. Observe that the copied provider ID selects exact current-run resume. Observe that ordinary failure settlement saves no Retry request. ## What Changed - Check a bounded encrypted controller header for fenced remote Runner recovery. Keep local Runner recovery on its existing disk-backed path. Match the run, conversation and runner identity. Preserve the existing staged-attachment path. - Share the existing saved-user Retry admission between restart recovery and ordinary terminal native settlement. Keep its exact infrastructure errors, user wake, comment, provider-turn and retry-chain checks. - Add real PostgreSQL and stopped-process regressions. Verify malformed and foreign checkpoint holds, current-owner resume, user authority and Stop. Verify saved-file recovery creates one fresh-session successor. - Document the ownership and Retry contract. ## Verification - RED: the real checkpoint rebind producer and stopped-process check selected exact current-run resume without a current controller checkpoint. - RED: ordinary settlement created zero Retry receipts for the real native failure code; expected one. - Review RED: a real current local controller file existed with no database checkpoint rows. The initial gate returned ambiguous state. The fix takes a required trusted execution-target kind and applies the database requirement only to the remote persistence path. - Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`, macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL. - Initial server Vitest: 42 selected cases passed across controller recovery integration, remote restart adoption, ordinary run completion and retained workspace recovery. The cases cover the stopped-process producer, current/foreign/malformed ownership, automatic user continuation, Stop, unsettled turns, system wakes and a manual Retry on first use. - `pnpm --filter @paperclipai/server typecheck`: passed with this worktree's Runner, shared and SDK builds and all seven server layers. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit: passed. - Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The real local/remote controller case passed against a fresh PostgreSQL cluster. It verifies local disk recovery without database rows, remote current-owner resume, and remote missing/foreign/malformed holds. Fresh `node scripts/typecheck-layers.mjs` passed all seven server layers. The existing offline chat reader now requests metadata only and retains its full independent native-state safety check. - A broader local recheck hit PostgreSQL setup/cleanup hook timeouts. These are retained as failed setup attempts; the fresh-cluster regression above passed. Required CI supplies broad final-head coverage. - Settlement-order correction: `dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL regression first proved both failures: Stop after an already-failed row created one Retry instead of zero; a DB-native promotion with an old legacy caller snapshot created zero instead of one. All six settlement scenarios now pass. The hook requires the local failed outcome and the canonical settled native row. Fresh seven-layer server typecheck passed again. - Required shared fixture base integrated at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills fixture cases and all seven server typecheck layers passed. Required CI passed for this head: [workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592), including all server layers, tests, build, Runner verification and E2E. The final snapshot has 57 successful checks, three intended skips and one neutral result. Current-head Greptile is 5/5; both concrete review threads are resolved. The independent re-review ended without a final response; its prior Stop finding has a real RED/GREEN repair. The focused command was run from `server/`: ```sh ../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically' ``` ## Risks - A terminal failure of a fenced Runner adds one exact-run header read. The read selects one encrypted header and limits inspection to 1 MiB. Invalid or missing ownership evidence stays held. - Saved user input can request one automatic Retry. It uses normal preservation, current authorization, budget, approval, Stop and cleanup gates. Unsettled provider turns and system wakes do not start an automatic successor. - This PR changes no provider process, credential, model or lease authority checks. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bee0f0e2fa |
fix(server): recognize recoverable result finalization during upgrades (#19651)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution saves an accepted result before it finishes
workspace export and status projection.
> - The upgrade report currently recognizes only a live execution in the
observed phase.
> - A durable accepted result can therefore block an upgrade after its
execution lease is released.
> - Existing finalization owners already resume that saved work with
task binding, physical worker stop proof, and an atomic owner claim.
> - This pull request lets the report inspect those existing contracts
without taking authority.
> - Qualified remote work can continue through a normal server
replacement.
## Linked Issues or Issue Description
Refs #18037 and #19596.
**What happened?**
An accepted remote native result remains in result acceptance, workspace
finalization, retry, or committed projection. The upgrade report marks
it blocking because it is no longer a result-less observed execution
with a live execution lease.
**Expected behavior**
Report a saved result as recoverable only when the existing replacement
owners can resume it. Preserve all task, result, contract, remote
workspace, provider lease, physical owner, decision, and workspace
barrier checks. Keep unsupported or unknown work blocking.
**Steps to reproduce**
Run the new PostgreSQL post-result readiness fixture. It persists a real
encrypted Runner header and an accepted terminal result with a bound
remote workspace. The unchanged report returns one blocker.
**Paperclip version or commit**
Regression base:
|
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
349b8b3f7b |
Migrate settled controller checkpoints while unrelated work runs (#19631)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Server replacement must preserve retained Runner sessions. > - Older sessions can retain controller checkpoints on the server volume. > - The existing importer requires all execution writers to stop, including unrelated runs. > - This change adds a separate online path for successful, settled sources. > - It locks the source task and finalizer while it reads and publishes that source. > - Unrelated work continues. The offline importer keeps its existing authority. ## Linked Issues or Issue Description **What happened?** A retained controller checkpoint cannot be converted by the existing maintenance importer while another task runs. Requiring a busy server to become globally idle prevents online migration. **Expected behavior** An operator can convert a successful retained source after its finalizer releases ownership, while unrelated tasks continue. An expired owner remains an owner. Related active work and uncertain evidence remain blocked. **Steps to reproduce** 1. Retain a valid legacy controller for a successful run with an accepted result and a released committed finalizer. 2. Start a different task. 3. Run controller preparation. The old importer refuses because execution writers are active. Related preservation contracts: #19038 and #19024. This change does not replace workspace migration or authorize volume removal. ## What Changed - Add `prestageNativeControllers` with task, finalizer, and source-run row locks. Recheck related runs and exact checkpoint identity under those locks. - Allow legacy checkpoint material through normal settled-owner import checks. Keep offline authority separate. - Compare captured local controller and supervisor bytes before commit. A changed source rolls back publication. - Add `--controllers-only` to the existing finite native preparation entry. This avoids re-reading provider archives during controller conversion. - Add PostgreSQL cases for unrelated work, expired owners, related active work, source-lock contention, and source changes. - Document the finite command and its limits. ## Verification - Real PostgreSQL affected suites: 56 passed across controller migration and recovery. A first per-server run was interrupted because it was slow; the completed run used the repository's shared PostgreSQL test wrapper. - Initial online cases: 3 passed, covering unrelated work, an expired owner, and related active work. - Source lock and changed-file cases: 2 passed. Publication rolled back on a source change. - Final source-evidence regression: 2 passed. A first unproven source deferred and a later valid source imported; changed-file rollback remained intact. - Server TypeScript check passed with the SDK and Runner declarations built. - Server package build passed, including the Runner vendor checks and compiled command. - Runtime bundling suite: 10 passed. - Full repository CI and exact-head automated review remain required before merge. This draft is not staging acceptance. - PASS: 7 CLI cases after adding the controller-only dependency fixture and explicit-mode coverage. The first CI run exposed missing mock/export coverage; the fixture repair passed after rebase. - The unrelated model-default failures were fixed on master; this branch now includes that fix. ## Risks - A source task can wait briefly on its own migration lock. A one-second lock acquisition budget defers a contended source. No global admission hold is acquired. - Only successful sources with a released committed finalizer are eligible. Failed or unsettled sources stay deferred and require their preservation owners. - The operator must run the command on the host that owns the original files. The command does not certify enrollment readiness or detach any volume. - The existing offline path remains unchanged and still requires stopped writers. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
15ee6a25a6 |
fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".
Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.
**Expected behavior**
When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".
Refs #19583, #16349
## What Changed
- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
|
||
|
DottaandClaude Opus 5.5
|
7941f6b051 |
fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model
## Linked Issues or Issue Description
**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".
**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.
**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".
## What Changed
- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.
## Verification
- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.
## Risks
- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
ee6c101854 |
Reject failover backups behind the accepted event cursor (#19613)
A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors.
Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on
|
||
|
DottaandPaperclip
|
44c60c7221 |
Verify default skill sync through two real cold sleep cycles (#19191)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Automatic skill sources check upstream every five minutes. > - The domain deadline contract can hand future work to a durable wake alarm. > - A saved deadline or a worker unit test does not prove that a stopped tenant resumes the real producer. > - This pull request checks the default cadence through the actual app and a matching service checkout. > - It checks two cold starts and three distinct persisted skill revisions with real clocks. > - The benefit is repeatable local evidence without changing production timing or authority. ## Linked Issues or Issue Description Refs: #19001, #17008, #17923. This is a test and documentation follow-up to #19001. #19001 is now merged; this PR targets master. Feature code remains in that merged foundation. #17008 covers the earlier source worker change. #17923 covers separate skill-cache retention; this PR does not change that behavior. A scoped search of public PRs and issues found no duplicate default-cadence cold-cycle acceptance test. No matching public issue was found. The enhancement fields follow. **Pre-submission checklist** This improves checks for an existing feature. It adds no product endpoint or execution authority. The public related-work search is complete. **What existing behavior does this improve?** Verification of automatic skill-source updates across a durable idle handoff and cold restart. **Subsystem affected** `server/` test fixtures and development documentation. **Current behavior** Domain tests cover claims, access and future deadlines. They do not prove the full default-300-second path from ordinary source attachment through signed tenant reports, durable alarms, stopped compute and a changed persisted skill. **Proposed behavior** An opt-in launcher checks bootstrap first, then two full real-clock cycles. Ordinary CI can run the transport controls without the host fixture. The launcher-specific worker assertion runs only when prewire is requested, and fails closed in that lane. **Reason and benefit** The test can detect an early refresh, a lost deadline, a stale proof, an ACK after physical stop, a failed cold restart or a producer that does not publish changed contents. **Breaking changes** None to product APIs, schema, cadence, authorization, claims or retry policy. Existing maintenance-helper defaults stay unchanged. **Additional context** The opt-in host flow needs an authorized matching service checkout, its normal compiled acceptance bridge, Node 22 and two owned PostgreSQL test databases. The standalone controls do not need that service or PostgreSQL. Roadmap review places this under existing Skills Manager and Cloud deployment verification, not a new core feature. ## What Changed - Add an opt-in default-cadence acceptance test and checked-in launcher. - Reuse real server startup, managed tenant authority and the ordinary project attachment route. Control only the remote Git snapshot boundary; keep actual authorization, scanning, claims and publication. - Use actual signed tenant routes and the matching service's gate, sleep controller and PostgreSQL wake-alarm store. Stop and restart real app children through a controlled local provider. - Preload the Node transport guard before coordinator, worker and child module work. Check worker-local and child PID, thread and source-hash receipts. Keep caught unexpected transport attempts fatal. - Require both persisted refresh times and actual upstream handler timestamps to be at or after each original deadline. - Join physical callbacks during bridge close, stop a late wake-created child, and preserve the original cleanup budget. Add a real IPC/HTTP-child counterfactual. - Accept valid owned HTTP options paths without allowing Unix sockets or unknown origins. Remove the unused App helper export. - Add finite transport and launcher controls. Defer only the launcher-specific parent assertion in vanilla CI. Keep all subprocess controls default-runnable. - Add optional child entry, preload and bounded pool settings to the existing test helper. Preserve its defaults and the real observer test body. - Remove added explanatory comments under the contribution policy. Preserve executable syntax, directives and literals. - Document pinned setup, fresh owned databases, bootstrap and full two-cycle commands in `doc/DEVELOPING.md`. The diff against the intended base contains nine test/helper/launcher files and this one document. It adds no production feature code or Cloud source to the App repository. ## Verification The latest two-file CI repair preserves both original failed jobs. A newer master workflow introduced two required step inputs absent from its extracted-step test environment; the faithful false/false inputs now pass all five existing target-authorization cases, after an actual missing-input counterfactual failed. A controlled socket case also reproduces legal reuse of the released runner port; the fixture now holds that reservation until the preview binds, releases it in finally, and asserts distinct ports. All original retry/result/accounting assertions and timeouts stay unchanged. The original Linux listener owner remains unknown; the control proves this fixture collision mechanism, not that original attribution. Fresh current-head full CI and complete reviews are pending. Current merge-integration candidate: `f6f686103b7bfd0a4ab83085634c622af3896ecb`, a normal merge after #19001 landed, including actual master `1c9767d323cf872a266962b9e6b4bd5cbcb57f1c`. All nine acceptance code files are byte-identical to148a82e5; both parents' documentation additions survive. Every other path equals actual master. The only conflict was the old foundation's startup block; current master was retained in full. Fresh exact-head CI and complete reviews are pending. The local measurements below belong to their explicitly named revisions; no fresh full614-second run is claimed. Current candidate: `148a82e54c502759785407357001c6b50c1439dd`, normally carrying the current base `8c912e1ba4fc66397a7341ade59373af8075068f`. All nine acceptance code blobs remain identical to qualified repair `e092746f4da181c8414c63be71474619a84b97e7`; documentation retains both parents. The matching built service for the fresh bootstrap is `389878364b42dd129835d263f91882c5774d5123`, using Node 22.23.4. The App uses Node 24.22.2 and pnpm 9.15.4. Use the exact controlled setup and commands in [doc/DEVELOPING.md](doc/DEVELOPING.md#controlled-default-cadence-sleep-acceptance). Fresh owned databases and the matching compiled service are prerequisites; the launcher does not create a cluster or build dependencies. - **Current affected semantic check passed**, using actual native TypeScript 7 and the current generated CI options/references (0.640 seconds). Normal declaration references passed in 11.726 seconds, with source-matched SDK/shared/kernel/Runner prerequisites and the frozen current lockfile. - **Current guarded controls: 44 passed**, plus the same single explicit real-PG observer deferral, in 4.790 seconds. The executing worker checked its own global PID/thread/preload source hash and matching receipt. Guard hash: `7daf9ae7d2f26e8fcd389b1d560fac1d4e218a8304a2c58e49f646af108165bc`. No unexpected attempts or PG sockets occurred. Vanilla controls on the identical repair files passed 22 cases with one explicit launcher-parent assertion deferral (2.044 seconds). - **Real cleanup counterfactual:** unchanged old helper failed the actual endpoint assertion in 1.568 seconds: its late HTTP child remained reachable after close returned. The same fixed-helper case passed in 1.485 seconds: close waited, the child exited, the port was unreachable and no IPC error occurred. The isolated driver records only the known legacy closed-IPC error long enough to inspect cleanup; current helper requires zero such errors. All recorded processes and ports were independently checked empty. - **Fresh current bootstrap passed once**, 16.159 seconds wall / 14.12 seconds Vitest / 12.237 seconds body. The ordinary enabled producer persisted useful revision 1, cleared its lease and retained exactly 300,000 ms between success and original next deadline. A signed current-generation report produced the matching generation/operation-lock SQL ACK before physical child stop (ACK 06:54:02.020 UTC; stop signal 06:54:02.025 UTC). The alarm retained the original D and D-120 eligibility. Both fixture databases ended with zero sessions; the coordinator independently audited and normally removed them. This bootstrap does not execute a due wake or the new v2/v3 timing assertions. - **Historical full two-cycle result remains 1/1 passed in 614.213 seconds wall**, at measured App `74f7c012ca17ad0d7302d76403f5033affb1430b` and service `2cb54f904031dcf69a32902cd853b43f74d6f301`. Two new processes published distinct immutable v2/v3 at original D+115 ms and D+98 ms. Exact signed SQL ACKs preceded both physical stops; real SQL D-120 claims caused both cold starts. Features, claims and original business deadlines stayed unchanged. - Both historical 10.002-second quiet windows kept app SQL, upstream and outgoing HTTP counts unchanged. Two awake PG sockets remained. Quiet counts do not mean all connections closed. - The stronger timing checks are additional assertions, not retroactively executed results. Offline inspection of the original receipts confirms persisted successes at D+115/+98 ms and earliest upstream handler entries at D+44/+30 ms; all five requests in each refresh group started after D. The exact new pre-sleep request counter and dispatch-before-handler time were not independently recorded historically. - Source review found 19 critical automatic Git producer/runtime files identical to the measured App. The current active cache helper change does not run on automatic Git publication and adds no pool. The matching service carries unchanged measured helper/sleep/controller/alarm seams through its normal master merge. The current bootstrap and focused controls qualify the changed cleanup/guard and parent integration. No fresh current-head 614-second pass or whole-tree identity is claimed. - Runtime is serialized: worker 1, per-pool limits at most 4, configured App potential 10 + shared service pool 2 + one observer against server max 40. Current bootstrap point samples observed at most 3 owned clients plus the observer; those samples are not a continuous aggregate peak. App/service guard peaks were 2 each, with zero remaining sockets at cleanup. Failures remain accounted for. The first historical bootstrap cleanup masked an unknown primary error. Later attempts exposed missing fixture-owned domain metadata and lost origin snapshots; test-only ownership/cleanup corrections preserved the original assertions and timeouts. The new causal driver first lacked a final observation, then encountered sandbox bind EPERM, then exposed the old closed-IPC error; only the final real endpoint failure above is labeled the cleanup counterfactual. An offline dependency miss and a generator-cleared ignored compiler config were retained and corrected without weakening compiler options. The former inherited policy failure (397 findings) is resolved by the genuine current-master base/scanner, with zero findings and seven scanner tests; no unrelated UI site was edited and the gate was not relaxed. Historical real TCP observer coverage passed 1/1 in 3.688 seconds. Comment-only and opt-in-selection carries remain documented in prior evidence; self-hash changes are metadata exceptions with fresh executing-worker checks. The full workspace typecheck, test, build and browser commands were not run locally for this test-only slice. The current base passes its full exact-head CI and Greptile 5/5; this leaf now has all 56 exact-head check runs terminal (54 success, one neutral, one skip), the additional legacy security status passing, complete current Greptile 5/5 and Ponytail 5/5 reviews, and all four review threads resolved across all pages. No live provider or deployed host/database suspension was exercised. ## Risks - The host test requires a matching service checkout and compiled bridge. An unrelated service revision is not a valid substitute. - Controlled localhost Git, S3 and physical process operations do not qualify deployed Railway or Neon suspension, external permissions or provider uncertainty. - The Node guard does not prove native subprocess transport isolation. Only finite local Git metadata and Node version commands were needed. - Two short SQL/HTTP quiet windows do not establish strict 90-second host acceptance. Separate plugin-host LISTEN, MCP wake and other domain gaps remain open. - Explicit test URLs are destructive fixture inputs. Use fresh, owned `_test` databases. The launcher does not start a cluster or choose a database implicitly. - Passing CI and review do not qualify deployed suspension; the current proof boundaries above still apply. > For core feature work, check `ROADMAP.md` first and discuss it in `#dev` before opening the PR. This PR adds verification to the existing feature in its base. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation, repository review and local code execution. OpenAI Codex, based on GPT-6, performed managing review and integration. The context-window sizes and exact served backend snapshots were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — fresh merge-head gates pending - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — fresh merge-head review pending - [x] I will address all Greptile and reviewer comments before requesting merge ### Current master reconciliation (2026-11-06) Merged actual master `e28158b353fd274f0322be76d62cd223e0df599f` normally. All nine qualified acceptance code files and the held-port repair remain byte-identical. Upstream independently landed the workflow input fix in #19602; the fixture now uses that exact upstream version once, avoiding duplicate object keys in a squash merge. All other master changes are preserved. The earlier two-cycle local evidence remains source-specific; no fresh 614-second run or deployed suspension is claimed. This new head requires fresh CI and bot reviews. ### Actual combined-tree typecheck failure and repair CI job `128156887548` failed with TS1117 at workflow-target.test.ts:22: two independently landed master fixes declared the same `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys. The complete original log is retained. A further normal master merge now removes the duplicate declarations, retaining exactly one false/false pair and every original assertion. All nine qualified acceptance code paths and the held-port repair remain byte-identical; unrelated master changes are preserved. This is a concrete compiler failure and source repair, with fresh exact-head CI/reviews required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e482f0f632 |
fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.
Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on
|
||
|
DottaandPaperclip
|
c09fb60ca6 |
fix(recovery): retain worker-stop audit in preparation retries (#19612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must preserve saved files before it admits another execution. > - A failed preparation can need a verified stop of its previous workspace worker. > - Recovery keeps that verified stop receipt in the run profile for audit. > - The preparation retry guard treats the retained receipt as unknown execution evidence. > - This pull request recognizes a valid same-run audit receipt while it retains all current execution and file checks. > - A saved Retry can then continue once after recovery completes. ## Linked Issues or Issue Description **What happened?** A native preparation failed before provider dispatch. The recovery worker verified that the previous workspace worker had stopped and saved its files. The same HTTP Retry stayed deferred because the run profile retained `nativeWorkspaceOwnerStopProof`. **Expected behavior** The saved Retry should continue once after all preparation, stop, file, ownership, and authorization checks pass. The old run and its audit receipt should remain unchanged. **Steps to reproduce** 1. Fail native preparation with a retained remote workspace and no provider activity. 2. Let the recovery worker acquire the old workspace owner through verified platform stop evidence. 3. Submit one HTTP Retry and let preservation complete. 4. Observe that the old guard retains the retry hold even after saved-file verification passes. **Paperclip version or commit** Reproduced against `d728be8fbf`. Related work: #18908, #19021, #19026, and #19466. This change covers the retained workspace-worker stop audit left by a later repair. ## What Changed - Add a typed reader for a host-persisted same-run worker-stop audit receipt. Keep signature and expiry validation at ownership acquisition. - Recognize only that validated audit field in failed-preparation settlement. Keep unknown profiles and active owners blocked. - Exercise real preservation, HTTP Retry delivery, native dispatch, saved files, accepted results, and task completion. Add malformed, foreign-run, unstopped-worker, and active-owner cases. - Document the distinction between retained audit evidence and current execution authority. ## Verification - Failing before the fix: the current-worker preservation case passed; the stopped-worker case produced zero successor runs after the same Retry, where one was expected. - Passing after the fix: 47 recovery cases, 10 native dispatch/preservation cases, and 6 platform-proof tests. The two preservation success cases overlap between the first two groups. - Server typecheck passed all seven layers after the final test changes. The Runner and SDK dependencies were built in this isolated worktree. - Final comment-free commit `44f8f33f8d917dddb51852283a479786102eab4d`: a fresh combined run passed all 61 affected cases in 67.92s. Fresh server typecheck passed all seven layers. - The earlier `95d0376c` combined run passed 60 cases and hit one `socket hang up` in the existing `catalog_provider_warning` HTTP test. That case passed in a bounded isolated run (12.48s), and the final 61-case combined run passed. The socket failure remains recorded as an intermittent HTTP harness limitation. No retry or weaker assertion was added. - Final commit `44f8f33f8d917dddb51852283a479786102eab4d` completed all required CI gates: 57 checks succeeded, 3 intentional skips, and the advisory Paperclip review is neutral with a substantive 5/5 verdict. Greptile is 5/5 and there are no unresolved review threads. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42293625696). The follow-up commit only removes six new code-comment lines to follow CONTRIBUTING. - The native provider and signed platform API are controlled test boundaries. The tests use real PostgreSQL, recovery, ownership, file persistence, retry admission, dispatch, and finalization services. Deployed recovery remains unverified until normal rollout. ## Risks - The main risk is mistaking an audit receipt for current stop authority. The reader checks its exact shape and run binding. The receipt cannot replace stopped leases, saved-file receipts, settled preparation, current authorization, or the absence of an active owner or provider evidence. - The negative matrix runs with the audit field present. Provider activity, unknown usage, bad file receipts, Stop, actor changes, and unsettled leases still block continuation. - No schema or migration change is required. ## Model Used OpenAI GPT-6 in Codex. The exact deployment model ID and context-window size are not exposed in this session. Used reasoning, code editing, shell execution, and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandClaude Opus 5.5
|
a7d3d4b78c |
fix(runner): rotate local and external authority through one resumable path (#19547)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.
**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.
**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.
The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
da9a4cb96c |
fix(native): retire a finished local Runner that stopped before suspending (#19597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs each per-turn agent turn through a
runnerd process, and the next turn must prove that the previous Runner
is finished before it takes over the saved session root
> - Under load, a local turn can have its result accepted while its
`runner.suspend` command does not finish, so the Runner's own receipt
stays unsuspended
> - The retirement proof that handles the same case for sandbox Runners
refused every local Runner with `target_not_sandbox_runner`
> - So the next turn failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and its session root was
quarantined
> - This pull request extends the retirement to a local Runner when the
prior Runner process is proven stopped by exact PID and start time
> - The benefit is that a follow-up turn (for example, after a user
answers a question) continues instead of failing
## Linked Issues or Issue Description
Refs #19525 (keeps the accepted result in the same incident; this is the
next run).
**What happened?**
In the runner E2E cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
the first turn asked a question and its result was accepted. The Runner
then logged `runner did not prove durable suspension before checkpoint`
(`runner.suspend` pending, Runner lifecycle `ready`). After a server
restart and the answer, the next run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`, and its root was
quarantined.
**Expected behavior**
When the prior local Runner is provably stopped and every other
retirement proof holds, the next run retires that owner as a continuity
break and continues. If the stop cannot be proven, the run keeps the
named rejection.
**Steps to reproduce**
1. Run a local per-turn native turn whose `runner.suspend` does not
finish after its result is accepted (host load average ~40).
2. Start the next turn for the same session.
3. See `retirement_refused=target_not_sandbox_runner`.
## What Changed
- `server/src/services/native-runtime/native-restart-recovery.ts`: add
`proveNativeLocalRunnerStopped`. It uses the existing restart-recovery
checks: the recorded Runner PID is dead or now belongs to a process with
a different start time; its recorded process group is not alive; each
recorded provider process is dead or recycled. A missing PID, start time
or process group, a live recorded process, or a start time that cannot
be compared is not proof. When settlement already cleared the run's
process fields, the saved `native.local_process_stopped` receipt is the
Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus
run events) so restart recovery and this proof read provider PIDs the
same way.
- `server/src/services/native-runtime/native-session-executor.ts`
(`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a
local target is now eligible. For a local Runner, the projection accepts
the Runner's own unsuspended receipt for the same identity and
unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`) only. All other proofs are unchanged: committed
result, released lease, no recovery state, stopped controller, delivered
terminal, other commands complete, tool inputs answered, quarantine
copies. The process proof runs before retirement and again just before
the archive.
- `doc/execution-semantics.md`: describe the local case.
- Tests: seven integration cases in
`native-recovered-owner-followup.test.ts` and ten unit cases in
`native-restart-recovery.test.ts`.
## Verification
- Red before the fix: the four new integration cases failed with
`retirement_refused=target_not_sandbox_runner`, which is the incident
error.
- Green after the fix: `pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`
35/35. The new cases prove that a stopped Runner is retired (root
archived under `continuity-breaks`, no quarantine), and that a live
recorded Runner (`runner_process_alive`), a missing identity
(`runner_process_identity_missing`), and a pending non-lifecycle command
(`journal_commands_unsettled(turn.start.pending)`) keep the rejection.
All existing sandbox cases still pass, including "keeps an unsuspended
Runner receipt decisive" for sandbox targets.
- `native-restart-recovery.test.ts` unit cases: recycled PID, live
process, unknown start time, live group, live provider, missing PID or
start time.
- `native-session-executor.test.ts` + `native-restart-recovery.test.ts`:
1190 pass. `native-runner-restart-recovery.integration.test.ts` +
`native-restart-recovery-isolation.test.ts` (shared provider-PID
extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass.
- Live: `pnpm test:e2e:runner -- --max-parallel 1
--max-automatic-retries 0 --id
local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
3 runs at
|
||
|
DottaandClaude Opus 5.5
|
586caa5f49 |
fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at |
||
|
DottaandClaude Opus 5.5
|
d728be8fbf |
fix(native): re-bootstrap a Daytona runner restarted before session.open (#19583)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. The local control plane keeps the runner's controller state in an encrypted Postgres checkpoint. > - When the control plane restarts, restart recovery classifies each live native run and decides to reattach, resume, bootstrap again, or hold the task. > - A restart chaos wave killed the control plane twice while a Daytona runner was connecting. The run was claimed and the sandbox runner was launched, but `session.open` never completed. No provider evidence existed. > - Recovery waited correctly for the old 60 s controller lease to expire. Then the classifier had no rule for a claimed remote bootstrap. It returned `ambiguous_remote_recovery_evidence` and blocked the task as "Workspace repair required". > - The controller writes each command to the Postgres checkpoint before it sends it. After the lease expires, that checkpoint is the complete record of what reached the sandbox runner. > - This pull request lets recovery prove from that checkpoint that no provider work exists, and then start the same-run bootstrap again. > - The benefit is that a restart during Daytona runner startup continues automatically instead of stopping the task. ## Linked Issues or Issue Description **What happened?** A restart during Daytona remote runner startup (native ACPX Claude, two SIGKILLs about 20 s apart) left the task blocked. The run history showed `remote_controller_lease_live` twice, then `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task showed "Workspace repair required" (`native_restart_recovery_blocked`) and did not continue. **Expected behavior** When the durable controller record proves that the runner received only bootstrap commands, recovery continues the run with a fresh bootstrap. Any evidence of provider work keeps the hold. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Kill the control plane after the sandbox runner connects and before `session.open` completes. 3. Restart the control plane and wait for the old controller lease to expire. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: `server/src/services/native-runtime/native-restart-recovery.ts` `classifyNativeRunnerRecoveryEvidence`, remote branch. It admitted only an unclaimed bootstrap (`remote_execution_never_claimed`, #16349) or a run with a provider checkpoint. A claimed bootstrap with no provider evidence fell through to `ambiguous_remote_recovery_evidence`. The decrypted checkpoint of the failed run held `connectionCount: 1`, `run.prepare` delivered once and pending, `session.open` pending and never delivered, no committed events, and `ackedSourceSeq: 0`. Refs #16349, #19524 ## What Changed - `native-bootstrap-proof.ts`: move the incomplete-bootstrap predicates from the executor into a small pure module. Restart recovery and the executor now use the same proof. No behavior change. - `native-runner-checkpoint.ts`: add `nativeRunnerCheckpointProvesIncompleteBootstrap`. It reads only a checkpoint with the bootstrap parts (header, cursor, `command:1`, `command:2`), checks the run, runner instance, and session identity, and applies the shared proof. A remote runner journal is not on this host, so the proof uses the controller record alone. - `native-restart-recovery.ts`: for a classic sandbox run whose owner lease has expired, with a durable checkpoint, no session checkpoint, no provider evidence, and no staged attachment, claim `bootstrap_incomplete` with the reason `remote_bootstrap_proven_incomplete` when the checkpoint proof holds. At three provider attempts, block with `execution_recovery_budget_exhausted`. - `native-session-executor.ts`: add `restoreRemoteBootstrapCheckpointForRecovery`. For a `bootstrap_incomplete` recovery on a sandbox target, it restores the Postgres checkpoint over the local external-state marker. The existing incomplete-bootstrap proof then checks the restored state before it quarantines the root and bootstraps again. - `doc/run-log-events.md`: document the new disposition reason. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "remote bootstrap"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven claimed bootstrap gets `bootstrap_incomplete` (attempt kept, generation +1, run still `running`, no recovery action). The hold stays for a queued `turn.start`, an acknowledged event, adopted state, no checkpoint, and a spent budget. The executor helper restores the checkpoint only for a bootstrap recovery on a classic sandbox and only over a marker for the same run. - Regression set: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration`, `heartbeat/recovery`: 1383 passed, 14 skipped. - Live Daytona replay of the failing chaos task (double SIGKILL during remote runner startup), 3 runs: REPLAY_RESULTS ## Risks - A fresh bootstrap acquires a new sandbox. The old sandbox keeps its lease until run settlement releases all run leases. The old runner gets no turn command, because only the controller issues one and the new controller never connects to it. - If `session.open` reached the old runner, the old provider session can exist without a turn. No provider work or tool call can happen in it. - Any checkpoint shape other than the bootstrap-only shape keeps the previous fail-closed hold. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
dc1adb7209 |
fix(tasks): advertise a native resume only while the scheduler would claim it (#19523)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native run cannot reattach to its provider after a control-plane restart, the native coordinator resumes the same run by itself after about 30 seconds, up to three attempts > - The run projection shows this correctly as "Retry scheduled" with only `inspect_run` permitted > - The task interruption shown on the issue read only system-recovery evidence and `scheduledRetryAt`. It never read the coordinator's `nextAttemptAt`, so it showed `state: failed, canRetry: true` and offered a manual Retry while the automatic resume was still pending > - The two surfaces disagree, and a user click can race the automatic resume. That is one way a Retry gets parked behind "execution checks" > - #19503 made the task interruption read the coordinator's pending resume time. Review found that the coordinator also pushes that time forward while it defers an ineligible candidate, so a reassigned or blocked task could hide Retry for good > - This pull request advertises the resume only while the scheduler's synchronous admission checks pass > - The benefit is one consistent story: while the automatic resume is pending, the task shows it with its time and no competing Retry. If recovery ends, exactly one Retry returns ## Linked Issues or Issue Description Follow-up to #19503. Refs #19487, #19497. **What happened?** A native run failed with `native_session_interrupted` after a restart. Its finalization row had `phase: retryable_failure` and `nextAttemptAt` 30 seconds later. The run projection said "Retry scheduled". The issue interruption said `state: failed, canRetry: true, nextRetryAt: null`. **What did you expect?** While the automatic resume is pending, the task shows that it is resuming, with the time, and no Retry. When the resume fails for good or never runs, exactly one Retry that works on the first click. **Steps to reproduce** 1. Run a native task and restart the control plane so the native attach fails. 2. Read `GET /api/issues/:id` while the coordinator's resume is pending. 3. Observe `interruption.canRetry: true` while `execution.phase` is `retry_scheduled`. ## What Changed - `task-interruption-presentation.ts`: the pending-resume read now also requires the task to be in a resumable status, assigned to the run's agent, not owned by another execution or checkout run, the run's question not cancelled, and the agent invokable (not paused, terminated or pending approval). These mirror `readNativeSessionResumptionAgent`. - `native-session-resumption-statuses.ts`: one shared status set used by both the scheduler admission and the read model. - No time cap: provider cooldowns and remote backoff schedules stay visible as a pending resume. ## Verification - `task-interruption-query.test.ts` (embedded Postgres): a reassigned task and a blocked task show one Retry; a paused agent asks to be resumed; in each case `readNativeSessionResumptionAgent` also refuses the candidate, so the read model and the scheduler agree. A resume an hour out (provider cooldown) stays `retry_scheduled` with no Retry. Red on master for the reassigned case. - All 57 presentation and query tests pass. Server typecheck passes. ## Risks - Low risk. Read model only. - A review-assignment resume (reviewer is not the assignee) shows Retry instead of the pending resume. That is the safe fallback. - Deferrals for a pause hold, dependency readiness or a competing run are not mirrored. Those gates also hold a manual Retry, so no competing action appears. - The `POST /issues/:id/retry` endpoint is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4938a930d6 |
fix(budgets): stop pending or restart-lost accounting from hard-stopping companies (#19497)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - A control-plane restart that kills a provider turn mid-flight can lose that turn's final usage receipt. ACPX Claude reports usage only at turn end, so the receipt never becomes complete > - Accounting kept such a run "awaiting receipt" forever. Every later admission in the company then refused with "completed runs await accounting", and the budget evaluator opened a hard-stop incident that paused the company. No product action could clear it: raising the budget refuses while runs await accounting, accounting retry finds no new data, and allowing unpriced usage does not apply to pending runs > - Restart chaos waves showed this as a dead task Retry, a new task blocked as "over budget" at 22 of 500 cents, and a hard stop that cancelled queued Retries and an unrelated running task > - This pull request closes the debt from evidence: once native recovery reaches terminal failure and the provider stop is verified, the known part settles as an unpriced charge. Until then the run holds new work as recovering accounting, without a hard stop > - The benefit is that a restart no longer bricks a company with a hard budget. The operator's unpriced-usage policy decides admission, and unknown spend still never shows as zero cost ## Linked Issues or Issue Description Refs #19399, #19426, #19431 (earlier restart Retry fixes; this is the budget layer under the same journey). **What happened?** After a control-plane SIGKILL during an ACPX Claude turn, the killed run ended `native_provider_terminal_failed` with `usage_json.accountingReceiptReady = false` and `cost_accounting_pending = true`. It stayed that way for good. The task Retry was accepted but parked with `budget.blocked` ("Company cannot start work while completed runs await accounting"), which the UI showed as "Retry is waiting for execution checks". A new Codex task was escalated to `blocked` with "the original recovery target is over budget" while the company had spent 22 of 500 cents. After the next restart the budget evaluator opened a hard-stop incident (`amount_observed` 22.08 < 500), paused the company and cancelled both queued Retries. **What did you expect?** A restart should not stop the company. Unknown usage from a stopped provider should become an honest, operator-governed state, and Retry should start a run. **Steps to reproduce** 1. Give a company a hard-stop budget. 2. Start an ACPX Claude task and SIGKILL the control plane mid-turn, so the resumed turn fails. 3. Observe the run stay `cost_accounting_pending` with an incomplete receipt, and every new admission refuse. ## What Changed - `budgets.ts`: completed runs that await accounting now hold new work only. They no longer open a hard-stop incident, pause the company, place an agent budget hold, or deliver Stop to running work. Only spend at the limit, or unpriced usage under a blocking policy, does that. Chaos wave A and a pre-restart lifecycle cell showed the wedge: a run that had just succeeded stayed terminal for about 100 ms before its charge committed, the evaluator paused the company in that gap, and the Stop it delivered left a cancelled run whose receipt never closed. - `run-cost-accounting.ts`: a failed native run is a closed receipt source only with positive stop evidence: its finalization is `terminal_failure` with no result (not a finalization-deadline failure), it no longer records a process identity, and its latest server-authored process event is `native.local_process_stopped`. One SQL predicate serves the reconciler scan and settlement. Accounting now settles its known part as an `unpriced` charge instead of waiting forever. The background reconciler selects these runs. The reservation settles. Other incomplete receipts stay pending as before. A cancelled run keeps its existing accounting drain, so Stop recovery can still deliver its final receipt. - `native-local-process-stop.ts` and its transactional callers: releasing a verified local or remote process stop wakes the accounting worker, so the closed debt settles promptly. - `usage-receipts.ts`: a charge settled without its receipt is marked `accountingReceiptUnavailable`. A later complete receipt that carries its own price (including a rate-card price) corrects it. A receipt with no own price and unchanged usage does not reopen the hold. - `budget-exposure.ts`: the blocking unpriced-usage policy no longer counts a charge settled from a lost receipt with no observed usage. That is unknown, not "usage with no reliable price". Complete receipts that report no tokens (for example Cursor Cloud) still count. A live wave under the default policy hard-stopped its company at 0 of 500 cents on one such charge. - `budget-exposure.ts`: a terminal-failure native run with an incomplete, not capture-failed receipt counts as recovering accounting. Admission holds new work ("New work must wait while a native run recovers its unfinished accounting") without a hard-stop incident, a company pause or cancellation of running work. - `recovery/service.ts`: the stranded-work comment names the actual budget gate instead of always saying "over budget". - `task-interruption-presentation.ts`: a Retry parked by `budget.blocked` says it waits for the budget, not for "execution checks". - `doc/SPEC-implementation.md`: documents the closed-receipt contract next to the existing incomplete-receipt rule. ## Verification - New embedded-Postgres suite `server/src/__tests__/closed-native-receipt-accounting.test.ts`, built from the persisted shape of the killed run: - Red before the fix: the run stayed pending, `reconcileRunCosts` accounted 0, and admission refused with "completed runs await accounting". - Debt stays pending with no verified stop event, with a finalization-deadline failure, with a recorded process identity, and while the coordinator can still produce a result. - Under a blocking policy, a lost receipt with no usage does not hard-stop; one with tokens still blocks. A run with a still-recorded process identity paused the company with a hard-stop incident. - Green after: the stopped run settles as one `unpriced` charge with the reservation settled; admission is clear under an `allow` policy and names "no reliable price" under `block`; a run with a recorded process identity or a non-terminal coordinator stays pending; the closing window holds admission without an incident or pause; a final receipt that arrives later still enters late-receipt reconciliation. - Pending accounting from a run that just finished holds admission with no incident, no pause, no enforcement version bump, and an unrelated running run stays running. Red before the `budgets.ts` change: the company was paused. - A later priced receipt corrects the unavailable charge to its price; an unpriced one leaves it settled. - `task-interruption-presentation.test.ts`: the `budget.blocked` Retry wait message. - `budget-enforcement-scoping.test.ts`: the accounting-hold release cases now assert that pending accounting places no budget hold and that admission clears after settlement. - Existing suites pass locally: all budget suites (control publication, enforcement scoping, enforcement worker, evaluation work, live attempts, history, service, live usage, budget cancellation), native-stop-accounting, explicit-native-continuation, native-case-handoff, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, native-usage-delivery. Server typecheck passes. - Runner e2e cell `lifecycle-baseline.runner-codex.local.lifecycle-dependency-restart-neutral`, which failed on master with a false agent budget hold before any restart, passes on this branch. - Live restart-chaos wave (local runner, K1 SIGKILL, codex + 2× ACPX Claude + OpenCode, 4 restarts, company policy `unpricedUsagePolicy: allow`) with the accounting fix: every killed run settled (no `cost_accounting_pending` left), the company stayed active, the OpenCode Retry completed the task on the first click and the Codex Retry finished its task. That wave also exposed a 1-second false hard stop in the closing window (observed 0 of 500 cents) that cancelled the running OpenCode turn; the `budget-exposure.ts` change fixes it and has its own red/green test. - Confirming live wave with both commits, same plan (2× ACPX Claude, OpenCode, Codex; 4 K1 restarts): both Claude tasks passed R3 (one Retry click started a run that completed the task), OpenCode passed R1 (continued by itself), and the Codex Retry completed its task. Zero budget incidents, zero cancelled runs, zero runs left pending accounting, company active; three killed runs recorded as `unpriced` charges. The only open signature is an unrelated workspace-durability check (`R2:pre_fault_files_lost` on the Codex long-bench task). ## Risks - Behavior change: a stopped native run's lost receipt now becomes an `unpriced` ledger charge with its known usage. If that usage has tokens, the default `block` policy still stops new work with "recorded usage has no reliable price". If it has none, the charge is visible in Costs but does not stop the company. Other unpriced charges keep their existing effect. Unknown spend is never recorded as a priced receipt. Budget enforcement stays best effort, as the spec says; killed OpenCode and Codex requests already lose their in-flight usage. - Remote runs and runs without a verified local stop keep their debt pending (an admission hold only, no hard stop). - Contract change: pending accounting no longer pauses a scope or stops running work. New work still waits with "completed runs await accounting". A scope's real hard stop (spend at the limit, or blocked unpriced usage) is unchanged. - If a provider stop can never be verified, the run stays "recovering accounting": new work waits without a hard-stop incident. Before, the same state paused the company. The hold is visible in admission errors and Costs. - A receipt that arrives after settlement posts a late correction. The original unpriced charge stays in the ledger. - No schema or migration change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests, live restart-chaos harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b55d8a9201 |
fix(runner): retain admitted OpenCode pack on ordinary continuation (#19558)
## Thinking Path > - Paperclip manages AI agents and preserves their work across runs. > - Native OpenCode turns can restore a saved provider session after the prior daemon stops. > - The Rust provider binds that saved state to exact qualified launch paths and artifact bytes. > - Ordinary continuation selected the current provider pack, although live reattachment retained the admitted pack. > - A new proxy or a diagnostics setting could therefore replace the launch profile behind saved state. > - This pull request retains the recorded pack for ordinary OpenCode continuation with a verified durable binding. > - The benefit is compatible continuation without weakening artifact integrity or granting replay authority. ## Linked Issues or Issue Description Refs: #17512, #18237. These added saved authority and compatible pack reuse. This change covers ordinary OpenCode continuation. **What happened?** An ordinary remote OpenCode continuation could select a newer pack while restoring its existing provider session. The Rust recovery guard then rejected the changed launch profile. The same error can occur after a controller release changes the bundled proxy. **Expected behavior** A continuation with a valid saved binding and pack receipt uses the same admitted artifacts. If those artifacts are unavailable, it holds the saved state instead of selecting another pack or creating a provider session. **Steps to reproduce** 1. Admit a remote OpenCode session with a qualified protected provider pack. 2. Persist its pack receipt and suspended durable Runner state. 3. Change the controller bundle to a valid pack with a different proxy digest. 4. Start an ordinary continuation with diagnostics enabled. The new executor regression fails before the repair because selection rejects the available original pack in favor of the new bundle. It passes after the repair. ## What Changed - Retain the saved provider pack for ordinary OpenCode continuation with a verified durable session binding. - Keep provider-pack retention independent of surviving-daemon identity: a new daemon records its current binary, and an authorized continuity break restores normal compatible pack selection. - Test retained pack selection after a proxy upgrade and fail closed when only a newer pack is available. - Test the real Rust executor with changed, valid proxy file bytes. It rejects drift without changing saved state and accepts the original artifacts again. - Document the selection rule and its limits. ## Verification - Initial prevention focused Vitest: 220 passed across native session execution, pack authority, pack verification and pack preparation. The executor regression was observed failing before the production change. - Rust native provider tests: all five OpenCode tests passed. These include the existing drift guard, actual changed proxy bytes, original artifact restoration and restart-lost turn behavior. - Advisory review follow-up: reproduced stale daemon identity in the actual executor and incompatible pack selection after a governed continuity archive. Both fail before their corresponding repairs and pass after them. Added an executor regression for a missing authority receipt; it remains at the current release integrity gate without creating a new receipt. - Updated head `639c080856` integrates master `17901d74ba` by normal merge. Affected Vitest: 202 passed across four files, including all new review regressions; all seven server typecheck layers passed. Exact-head CI and fresh Greptile/Paperclip review are pending. The prior head had 57 successful checks plus successful Snyk and Greptile 5/5; those results do not qualify this revision. - Environment: macOS arm64, Node 26.4.0, locked pnpm 9.15.4 dependencies, controlled remote command boundary and real Rust provider executor. No paid provider calls or hosted retries. - This is a bounded prevention change. It does not deliver historical provider artifacts to a replacement sandbox or replacement controller. Missing saved receipts or unavailable old artifacts remain a separate cold-recovery gap. No full hosted recovery is claimed. ## Risks - A continuation now requires its exact recorded pack when it has a durable binding. A newer installed pack cannot substitute for missing historical bytes. - Fresh launches, other providers and explicit continuity replacement keep their existing paths. Current ownership, company, model, permission and receipt binding checks remain in effect. - The repair does not relax the Rust launch-profile guard, replay prior tool effects, or authorize a new provider conversation. Existing published review receipts remain history. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose the exact model ID or context window. Capabilities used: reasoning, code editing, shell execution, tests and authenticated read-only browser diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
50038fac70 |
fix(native-runtime): contain backup verification deadline aborts (#19556)
Reject an already-aborted backup verification before constructing its file stream so cancellation does not raise an uncaught AbortError. Validation: 60 focused tests and all applicable CI pass. Greptile scored the exact head 5/5, and no review threads remain unresolved. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
1391f64551 |
Add gated release/resume methods to plugin Runner owners (#19584)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core owns Runner tasks through a lease-bound plugin owner. > - The SDK already defines release and resume in #19468. > - The owner still expects an accepted result for both operations, so it cannot use those SDK results. > - This pull request adds both methods behind the existing capability, warm-task and account checks. > - It uses the SDK parser to verify the result and task ID, and preserves the owner's cancellation checks. > - This prepares the owner boundary for later lifecycle integration. It does not start automatic release or resume. ## Linked Issues or Issue Description **Problem or motivation** The plugin Runner owner cannot forward the existing SDK release/resume pair. Its local result mapping rejects their valid results. Later lifecycle code needs both operations on the same checked task owner. **Proposed solution** Pass the existing description capability to the owner. Add explicit release and resume methods and use the SDK's operation-result parser. Require warm-task support and a selected account. Reject calls and late receipts after owner cancellation or detachment. **Alternatives considered** Adding automatic release here would also require connection quieting, settled-event drain, pending-ticket retry ownership and safe disposal. Keep those changes together in the next integration. Do not add a new flag, schema, timer or lifecycle mode here. Related: #19468. I checked ROADMAP.md and found no duplicate planned item. An open-PR title search for release/resume found no overlapping pull request. ## What Changed - Add the capability-gated owner facade for Codex and ACPX. Resume returns only `resumed` or `unavailable`. - Use the SDK parser for exact task-ID and operation-result validation. - Pass `releaseResumeVersion` from the checked description and test that it does not cause an automatic operation. - Cover account-preserving owner rebind, missing prerequisites, malformed results, ambiguous errors, cancellation and late replies. - Document the caller's remaining settlement, connection, ticket and attachment duties. ## Verification Passed locally on Node 26.12.0 with the pinned workspace dependencies: - In `server/`: `pnpm exec vitest run src/services/native-runtime/plugin-runner-task.test.ts` — 74 tests passed. - In `server/`: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t 'uses a task-only'` — 3 cases passed; 1,160 unrelated cases were not selected. - Built the real workspace SDK dependencies and Runner TypeScript. A focused strict TypeScript program for the owner and its tests passed. - `git diff --check` and the added-line secret/PII scan passed. The first root test command selected a nonexistent Vitest project name and ran no tests. Running the test from the server package fixed the command. The first focused TypeScript setup could not find Node types; pointing it to the package's installed type definitions fixed that setup. The full regression suite and release checks run in GitHub. ## Risks The facade has no automatic caller. Existing warm timeout, bootstrap, outbound connection and pool behavior stay in place. No plugin capability is advertised or task control channel enabled by this pull request. Future integration must serialize release with new work and disposal. It must quiet the owned connection, keep the exact pending ticket after an ambiguous reply, authenticate the source again and finish ordinary run attachment before new work. This change does not qualify provider parking, cold recovery, retention policy or deployment. ## Model Used OpenAI GPT-6.1 Sol, exact model ID `gpt-6.1-sol`, High reasoning. Used code execution and targeted tests. The context-window size was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
c6eecac1e0 |
fix(heartbeat): keep a shutdown-suspended native recovery for the next boot (#19530)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow tests the Runner in Daytona sandboxes from a signed, content-addressed image > - Every change to an audited Runner input needs a new image before a Daytona cell can test it > - The image job took 10 to 16.5 minutes today, and a restart chaos campaign waits on it after each runner-side fix > - Measurement shows two serial waits: a full Rust dependency compile (about 5 of 6.6 minutes) and a 3.5 to 4 minute download of cached base layers that starts only after the builder stages finish > - There is also no way to build only the image: every dispatch reaches a paid gate that the operator must reject > - This pull request removes both serial waits and adds an `image_only` dispatch input > - The benefit is a shorter fix-to-Daytona-test loop, with every image security check unchanged ## Linked Issues or Issue Description **What happened?** The `Publish verified Daytona image` job is on the critical path of the control-plane restart chaos campaign. Builds today took 16.5 min (run 42240434011), 15 min (42253920727), 10.5 min (42259118276) and 16 min (42265637650). Run 42264611573 failed: its RunsOn runner lost communication with GitHub 7.6 minutes into the build step, during the Rust compile. GitHub has no log for that job. **Expected behavior** A Runner source change should not recompile the locked third-party Rust graph. Cached image layers should download while the builder stages compile. An operator should be able to publish only the image, without a paid gate. **Steps to reproduce** Read the BuildKit timeline in the logs of job 127970125495 (run 42240434011). See the stage table below. ## What Changed - `docker/daytona-runner/Dockerfile`: compile the locked Rust dependency graph in its own layer against placeholder targets. The layer depends only on `Cargo.toml`, `Cargo.lock` and the crate manifest. The step deletes the placeholder outputs and fingerprints, so the next step always compiles the real Runner sources. The final stage still receives only the stripped executable. - `docker/daytona-runner/Dockerfile`: move the content-ID / source-revision check above the two `COPY --from` steps. It does not depend on either builder stage, so BuildKit now downloads the cached base and CLI layers in parallel with the compile. Both copies use `COPY --link`. - `.github/workflows/runner-full-stack-e2e.yml`: add the boolean `image_only` dispatch input. `authorize`, `target_lock`, `catalog` and `daytona_image` run unchanged. Every other campaign job gets `&& inputs.image_only != true`. `authorize` rejects `image_only` together with TestOps inputs. - `tests/runner-e2e/workflow-image-only.test.ts` (new): pin the job classification, the top-level guard on each campaign job, the guard on every protected environment, and the TestOps rejection. - `tests/runner-e2e/README.md`: document the layer layout and the `image_only` command. ## Verification Measured on the same RunsOn `m7i-flex.xlarge` (4 vCPU, 16 GiB, native x64) runner class. "Before" is master job 127970125495 (run 42240434011). "After (cold)" is this branch, job 128077687214 (run 42271236140), with no dependency layer in the registry cache yet. Its paid gate was rejected. | Stage | Before | After (cold dependency cache) | |---|---|---| | Whole image job | 16.5 min | **11.6 min** | | Rust dependency graph | 311 s (inside one 399 s step) | 395 s (own layer; reused from cache after a master build) | | `paperclip-runner-core` compile | ~87 s | 65 s | | Cached base and CLI layer download | 236 s, **after** cargo finished | 257 s, **in parallel** with cargo | | Copy runnerd + provider pack | 20 s + 50 s | 9 s (`--link`) | | Provider pack stage | 143 s (parallel) | 152 s (parallel) | | Export + push | 82 s + 10 s | 81 s + 11 s | | Dependency layer size | n/a | `target` 852 MB + registry 333 MB (builder stage only) | With a warm dependency cache, a Runner source change compiles only the 65 s crate step. The critical path then becomes the ~4 min cached-layer download or the provider pack. The expected job time is about 7 to 8 min. I measure this after merge with an `image_only` dispatch (master first, to write the cache, then a Rust-only branch change). Disk: the instance starts with 38.76 GiB free. The dependency layer adds about 1.2 GB uncompressed to builder storage. The final image does not change. - `vitest run tests/runner-e2e/workflow-image-only.test.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/daytona-image.test.ts`: the new test and all image tests pass. Three tests in `workflow-security.test.ts` (`gates every provider-secret job with stable actor IDs`, `keeps provider credentials inside explicitly gated paid workflows`, `isolates private archive trust from workflows that execute target-branch code`) fail on `origin/master` without this change, because of `chat-catalog-testops-case.yml`. This pull request does not change them. - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - No local Docker build. The Dockerfile is measured with a branch-targeted CI dispatch whose paid gate was rejected. ## Risks - The dependency layer keeps the Rust `target` and registry trees in a builder-stage layer, and default-branch builds export it to the registry cache. This adds builder disk and cache upload when `Cargo.lock` changes. The build prints the size (`du -sk target /usr/local/cargo/registry`). - The dependency cache helps only after a default-branch build writes it. Branch builds read it but never write it. This is the existing cache policy. - `image_only` changes the workflow, so it works only after merge to master. - No check is removed. Lockfile verification, cosign sign and verify, the anonymous pull check, the provider-pack manifest match, the default-branch-only cache write and the actor allowlist are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
fd4b197f83 |
fix(ui): hide runner selection during agent setup (#19517)
## Thinking Path > - Paperclip lets people manage AI agents for work. > - Agent setup connects a harness, model, account, and computer. > - The system already selects a qualified runner for a new agent. > - Setup still asks users to select that internal runner. > - The runner dropdown can be the only field in Advanced. > - This PR removes the dropdown and hides an empty Advanced section. > - Users keep their saved execution settings and see fewer setup controls. ## Linked Issues or Issue Description **What existing behavior does this improve?** Agent creation, onboarding, built-in agent setup, company import, and agent configuration. **Subsystem affected** The UI, setup error messages, tests, and Storybook fixtures. **Current behavior** Users can select a runner from a dropdown. Some setup screens place only that dropdown in Advanced. Errors can direct users to that control. **Proposed behavior** Remove runner dropdowns from setup and configuration. Use the existing automatic selection for new agents. Retain saved runner settings. Show Advanced only when other fields remain. **Reason and benefit** Users can configure agents without an extra internal execution choice. Setup also avoids an empty Advanced section. **Breaking changes** Users can no longer select a runner through these UI dropdowns. The API and CLI retain their runner options. Existing agents keep their saved execution settings. **Additional context** Related: [Default new agents to the qualified runner for their harness](https://github.com/paperclipai/paperclip/pull/15422). This PR changes the existing setup UI. It adds no harness or runtime default. The master merge retains the custom gateway placement from [#19510](https://github.com/paperclipai/paperclip/pull/19510). ## What Changed - Remove the shared runner dropdown and its setup/configuration callers. - Hide Advanced when it has no remaining fields. Keep command and argument settings where supported. Preserve custom gateway setup under Advanced on the Claude and OpenAI API key paths. - Keep setup failures visible and keep the retry action outside the removed control. - Update errors and documentation that referred to the removed dropdown. - Update component tests and Storybook fixtures for the new UI states. - Make Legacy first-task evals resume an API-configured setup record. Record that path separately from fresh automatic onboarding. ## Verification - Latest master integration (`187c0f6d6e5`): 276 onboarding/provider-connection tests passed across six files. UI typecheck and build passed. Token gates, query-error rendering checks, and the Storybook catalog check passed. The conflict resolution retains master's bounded connection tests and cancellation while removing the obsolete runner instruction. The first test command used the wrong working directory and failed before collecting tests; the corrected invocation from `ui/` passed. - Earlier branch verification passed: full workspace build and typecheck, 1,037 focused UI/service tests, 49 heartbeat provider contract tests, eval typecheck, and 30 map/catalog tests. - The full local `pnpm test:run` run passed all six chat integration shards (1,931 tests). It was stopped for the master merge before the complete suite finished. The [complete CI run](https://github.com/paperclipai/paperclip/actions/runs/42264662642) passed on the previous head `72e1699`. - Current head `a097c8369d079c839e78661a2206cfa62c5210e0`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42273168268) passed. PR checks total 55 passed and 3 skipped, with none failed or pending. Greptile reviewed this head at 5/5 with no actionable findings or unresolved review threads. - Live browser/provider evals have not run. The Legacy eval change has no live qualification claim. ## Risks - Removing the dropdown changes the setup UI. Existing API and CLI runner choices remain available. - Conditional Advanced sections must retain valid command and argument fields. - Legacy first-task evals now prove setup resumption. They do not prove the fresh-agent automatic default. - No schema change or data migration is included. ## Model Used OpenAI `gpt-6.1-sol`, through Codex in T3 Code. Reasoning effort: `xhigh`. The model used file editing, shell commands, and test tools. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
07ddf9a5c2 |
fix(native): keep accepted results when runner close cannot prove suspension (#19525)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist an accepted result first. The Runner then closes and suspends its session. > - Under heavy host load, the Runner close can fail to prove a durable suspension after the result is accepted. > - The executor then kept its finalization lease. The heartbeat finalizer got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. > - The provider work was complete. Only Runner bookkeeping failed, so the run must not fail. > - This pull request hands the exact lease to finalization and completes the run from its accepted result. > - The benefit is that an accepted result stays accepted, independent of host load. ## Linked Issues or Issue Description **What happened?** A native ACPX Claude run asked a structured question. The control plane accepted its result (`Waiting for…`) and recorded `run.terminal` with `runTerminalState=succeeded`. The Runner close then took about 8.6 s. Its `runner.suspend` command ended `failed`, but the durable Runner state was `suspended` with a matching identity. The close threw `NativeSessionCloseUnrecoverableError`. The executor emitted `native.post_result_failure` (`category: runner_settlement_incomplete`, `resultAlreadyAccepted: true`) and rethrew `native_session_lease_lost`. Its finalization lease stayed held (its TTL is 20 minutes). `finalizeNativeRun` got `native_finalization_lease_busy`, and the run ended `failed/adapter_failed`. No restart was in progress. Host load average was about 40. **Expected behavior** After the control plane accepts a result, finalization bookkeeping must not fail the run. The run settles from its accepted result. Repair of the Runner checkpoint stays with the existing next-run admission rules. **Steps to reproduce** Run `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts -t "accepted local result"` in `server/`. The test uses the real executor, the real control-plane result commit, and embedded PostgreSQL. The provider boundary accepts a result and then throws `NativeSessionCloseUnrecoverableError`. On the base revision, the finalization row keeps the executor lease (`phase: workspace_finalizing`) and the finalizer reports `lease_busy`. **Paperclip version or commit** Base `ff1d1dea5d`. The incident was on `819420a60a`. ## What Changed - `native-session-executor.ts`: after the rejected failure write proves an accepted result under the exact, unexpired lease, the executor releases that lease to finalization. It uses the same exact-authority predicates as the normal release. Before, the lease stayed held until its TTL expired. - `runtime-execution.ts`: for `runner_settlement_incomplete` on a local workspace (no workspace sync and no remote workspace authority), Paperclip records `workspace_finalize` as succeeded, finalizes the accepted result, and returns it to the ordinary completion path. - New `native-accepted-result-settlement.ts`: rebuilds the adapter result from the durable accepted result row, its terminal, the run's native session, and the accepted usage checkpoint. The result carries the closed `nativePostResultFailure` diagnostic. It contains no provider text. - Other post-result categories (for example `io_not_found`) keep their workspace-failure handling. They now also reach finalization at once, instead of after the lease TTL. - `doc/execution-semantics.md`: describes the settlement. ## Verification - Red, then green: `settles an accepted local result when runner close cannot prove suspension` and `keeps workspace failure handling for other accepted-result failures on a local workspace` in `server/src/services/heartbeat/runtime-execution.test.ts`. Both failed on the base revision with the executor lease still held. Both pass with this change. - Safety case `settles an accepted local result ... (workspace-check-failed)`: when the `workspace_finalize` check fails, the run keeps the existing failure path. Case `terminal-unreadable`: when the accepted result cannot be rebuilt, only a failed `workspace_finalize` record is written. - `pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts`: all pass (some cases time out at 30 s only under host load above 40; they pass on a narrow rerun). This includes the 14 existing post-result diagnostic and authority cases (expired, replaced owner, generation, attempt, boot, pid, process start, stopped, lock busy, and others). - `pnpm --filter @paperclipai/server typecheck`: pass. - Live cell `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` (real ACPX Claude, local runner): 3 runs at host load average about 40. Runs 1 and 2 passed. Run 3 hit the fault naturally: the first run emitted `native.post_result_failure` (`runner_settlement_incomplete`) and now ended `run succeeded` (before this change it ended `failed/adapter_failed`). The cell then failed later, in the resume run after the answer: `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=target_not_sandbox_runner`. That is a separate follow-up (see Risks). ## Risks - A changed lease handoff. The executor releases the lease only after the rejected failure write proves an accepted result under the exact owner, attempt, controller generation, controller identity, and an unexpired lease. A stale or replaced owner does not release anything. A failed release is bounded and best effort. The old behavior then applies. - The run now completes as `succeeded` while the local Runner checkpoint is not proven suspended. The next run on the same session then meets the existing fence: the continuity-break retirement (`retireFinishedPerTurnRunnerdOwner`) accepts only sandbox Runners, so a local Runner is refused with `target_not_sandbox_runner` and its state root is quarantined. This PR does not change that fence. Extending the retirement to local Runners needs its own proof that the local Runner process stopped, and is a follow-up. - Settlement happens only when the accepted result and its terminal are rebuilt first, and the `workspace_finalize` record then succeeds. A failed branch or plugin workspace check keeps the failure path. - Synced and remote workspaces are not changed. They keep their copy-back retry path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Related: #19126 added the closed post-result diagnostic that this change uses. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
beaa9a622d |
fix(native): re-bootstrap a runner killed before session.open (#19524)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - When the control plane restarts, restart recovery classifies each live native run. A run whose runner died before any provider identity existed gets the `bootstrap_incomplete` disposition. Recovery then quarantines the old state root and starts the bootstrap again. > - The executor accepts that disposition only when the durable state root proves that no provider work started. The old proof required `connectionCount === 0` and every command `pending`. > - A restart chaos wave killed the server about 2 s after runnerd started. The runner had connected and completed `run.prepare`, but `session.open` was still pending. No event was committed, so no provider session or turn existed. > - The proof rejected that root. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, and the task stopped with `native_continuation_requires_reconciliation`. Nothing continued automatically. > - This pull request widens the proof to the exact window where provider work cannot have started, and keeps every other shape fail-closed. > - The benefit is that a restart during native runner bootstrap continues automatically for every native provider. ## Linked Issues or Issue Description **What happened?** A control plane restart that lands after the native runner connected and acknowledged `run.prepare`, but before `session.open` completed, fails the run with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete` (controller generations 2 and 3), but the executor proof in `migrateRunnerdStateRootForExecution` refused it. The durable control-plane state had `connectionCount: 1`, `run.prepare` completed, `session.open` pending, no committed events, and `ackedSourceSeq: 0`. The runner journal had `nextSourceSeq: 1`, an empty outbox, and the provider state was `prepared` with no provider session ID. **Steps to reproduce** 1. Start a native task (seen with OpenCode on a local runner; the window applies to every native provider). 2. Kill the control plane about 2 s after runnerd starts, after `run.prepare` completes and before `session.open` completes. 3. Let the control plane restart. **Expected behavior** Recovery quarantines the state root and starts the bootstrap again. The task continues without a user action. **Actual behavior** The run fails and the task is blocked with `native_continuation_requires_reconciliation`. ## What Changed - `runnerdStateIsIncompleteBootstrap` (native-session-executor.ts) now also accepts a connected root when all of these are true: no committed event, `ackedSourceSeq === 0`, only `run.prepare` and `session.open` commands, `session.open` not completed, `run.prepare` pending or completed, no warm transition or staged attachment, and the bootstrap was not adopted. - When `runner/runner-state.json` exists, the runner journal must agree: empty outbox, `nextSourceSeq === 1`, `ackedSourceSeq === 0`, and no completed `session.open` or other processed command. An unreadable runner journal stays fail-closed. This covers events that the runner emitted but the controller never committed. - The remote "claim an untouched session" path keeps the old never-connected proof (`runnerdStateIsUnconnectedBootstrap`), so its behavior does not change. - Tests: a red test with the production state shape (connected, `run.prepare` completed, `session.open` pending), plus fail-closed counter-tests for a completed `session.open`, committed or acknowledged events, a turn command, a failed `run.prepare`, an adopted bootstrap, a warm transition, an uncommitted runner event, a runner-completed `session.open`, a runner-processed turn, and a torn runner journal. ## Verification - Red before the fix: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t "native incomplete-bootstrap evidence"` failed on "allows a fresh bootstrap after connect and run.prepare when session.open never completed" (`expected false to be true`). Green after the fix: 13/13. - The proof run against the kept durable state root from the failing wave (read only) returns `true` after the fix. - Related suites: `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to `bootstrap|restart|remote|unstarted|untouched`: 301 passed. - `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos proof: see the comment on this PR. ## Risks - The proof now accepts a root where the runner received `session.open` but did not complete it. The runner can have started a provider process or created an empty provider session in that window. Recovery only proceeds when restart classification already proved that the runner process and its group are dead and no known provider process is alive. No provider turn can exist, because the controller never issued a turn command. The worst case is an orphaned empty provider session, not repeated work. - Every state with committed or acknowledged events, a completed `session.open`, a turn, a warm transition, an adopted bootstrap, or an unreadable runner journal still fails closed. - Out of scope: when the proof fails legitimately, the task offers the existing execution-blocker Retry. In the failing wave the issue interruption also showed a budget wait (`userAction: budget`). That is the RC-06 accounting issue and belongs to a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
e3fb4bc73a |
fix(native): recover saved listener startup failures (#19479)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a0cba70f18 |
fix(runner): continue restart-interrupted ACPX and OpenCode turns (#19501)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their provider conversation when the control plane restarts > - A local Codex turn lost in a restart already continues once in the same conversation (#15297) > - ACPX (Claude and the other ACPX profiles) and OpenCode took a different path: the runner closed the lost turn as a provider failure > - The task then blocked with "Provider turn failed", and the user's queued follow-up stayed unread > - A restart must not need a user action when continuing is safe > - This pull request gives ACPX and OpenCode the same recoverable restart-loss cause and the same single continuation as Codex > - The benefit is that work interrupted by a controller restart finishes on its own ## Linked Issues or Issue Description Refs #15297 (Codex continuation), #19399 (restart-interrupted turn hook), #19471 (harness fix used for the live reruns). **What happened?** After a control-plane restart during an active native ACPX Claude turn, the task ended `blocked` with `native_provider_terminal_failed: Provider turn failed`. The queued follow-up comment was not consumed. No continuation started. The next run then failed with `native_session_cleanup_quarantined`. A native OpenCode run killed mid-turn ended the same way (`turn.failed` with `error: null`). **Expected behavior** When the old provider process is gone and the conversation can be restored, continue the task once in the same conversation. When that is not safe, record an honest restart cause instead of a generic provider failure. **Steps to reproduce** 1. Start a native ACPX Claude task on a local runner. 2. Submit a follow-up comment while the turn is active. 3. Restart the control plane while the turn is still running. 4. Run `pnpm test:e2e:runner -- --max-automatic-retries 0 --id everyday-workflows.runner-acpx-claude.local.recover-controller`. **Root cause** - `runner-core/src/acpx_provider_backend.rs` `restore_session_if_needed`: a saved `turn_active` state always closed the session. It recorded `turn.failed` with only a top-level `code: acpx_active_turn_recovery_closed` and no `error` object. It also emitted `run.terminal`. - The runnerd transport forwards only `error`, so the controller saw `error: null` and reported generic `provider_turn_failed`. - `runner-core/src/provider_backend.rs` gave the recoverable `provider_turn_lost_on_restore` cause only to `provider_label == "codex"`. OpenCode shares that backend. - `nativeRestartInterruptedTurnId` accepted only `codex_app_server`. The runtime continuation gate accepted only `provider.kind === "codex"`. - Two follow-on defects appeared once the continuation ran live: - The restored ACPX provider rejected the controller's result for the lost turn's open semantic call. That call never settled. - The restored provider's MCP request ids restart. The controller keys semantic calls by `hash(runId, callId)`. The continuation's first Paperclip tool call reused an earlier id, was treated as already answered, and hung. After the result was accepted, `runner_settlement_incomplete` failed the run. ## What Changed - ACPX runner: for an active turn lost to a runner restart, prove that the original provider lifetime lost its fence quorum. Then restore the same conversation and record `turn.failed` with `error: { code: "provider_turn_lost_on_restore", recoverable: true }`. Keep the run open for the controller's continuation. An accepted result, an unanswered runtime request, or an active goal keeps the old close behavior. So do an unproven lifetime fence and a failed restore. - ACPX runner fallback close: add a structured `error` (`acpx_active_turn_recovery_closed`, not recoverable, with an honest restart message), so the cause survives the transport. - ACPX runner: track open semantic calls durably. On restore, retire the lost turn's open calls. Acknowledge their controller results as `retired` and do not send them to the new provider. - ACPX sidecar: when the sidecar restores an existing session, prefix forwarded semantic `callId`s with a per-process namespace (`r<hex>:<id>`). Fresh sessions keep raw ids. - OpenCode runner: give OpenCode the same recoverable cause and open run as Codex (`restart_lost_turn_is_continuable`). Cursor Cloud stays excluded because its agent runs remotely. - Runner TS: `nativeRestartInterruptedTurnId` accepts `acpx_runtime` and `opencode_server`. The runtime continuation gate accepts `acpx` and `opencode`. All other Codex safety rules are unchanged: one continuation, admitted dead-runner restart only, no compaction, goal, semantic result, or pending runtime request. - Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`. ## Verification Red, then green, at each layer: - Rust `tests/acpx_warm_attachment.rs` `restores_the_conversation_after_losing_an_active_turn_to_a_runner_restart`. Red: status `closed`. Then red on the lost call's result: "tool result does not match a pending provider call". Green after the fix. - Rust `tests/native_provider_backend.rs` `restart_lost_opencode_turn_records_a_recoverable_cause`, with the qualified OpenCode launch fixture. Red: `error` was `null`. Green after the fix. - Rust unit test `restart_lost_turns_continue_only_for_locally_restored_providers` (Codex, OpenCode yes; Cursor Cloud no). - TS `native-session-runtime.test.ts` "continues a restart-lost acpx_runtime / opencode_server turn once". Red: `NativeProviderTerminalFailure provider_turn_lost_on_restore`. Green: exactly one continuation, which does not resend the task. - TS `acpx-runtime-sidecar.test.ts`: a restored provider's call ids are namespaced and resolve. - Server `native-runner-restart-recovery.integration.test.ts`: a checkpointed `acpx_runtime` or `opencode_server` restart interruption is claimed as `resume_dead_runner`. Red without the driver gate. Green with it. - Two `native_provider_backend.rs` tests asserted the old close-on-restore result. They now assert the new contract. Their real invariant still holds: the delivered response is never replayed. Suites run locally on the final revision: - runner-core: `cargo test --no-fail-fast acpx`, `--test native_provider_backend` (38), `--test acpx_warm_attachment`, `--test codex_provider restart_lost` - runner TS: `native-session-runtime`, `runnerd-restart-continuation`, Codex driver recovery, contracts, and `acpx-runtime-sidecar` (all pass) - server: `native-runner-restart-recovery.integration.test.ts` (80 passed) - `pnpm --filter "@paperclipai/paperclip-runner..." build` (all generated-contract checks pass) Live runner E2E (local, real Claude via ACPX, macOS arm64): - `everyday-workflows.runner-acpx-claude.local.recover-controller`: PASS twice in a row on the final runner code. The restart crossed an active turn. The task finished `done`, the queued follow-up was consumed, the ZIP download was verified, and there were no failure codes. - `stock-harness.runner-acpx-claude.local.continuity-restart` regression: PASS. - Earlier failing reruns are kept as evidence of the follow-on defects: one harness `kill EPERM` while stopping the server (unrelated to this change), one hang on the colliding call id, and one `runner_settlement_incomplete`. ## Risks - A continuation can run more work in the restored conversation. Its prompt requires inspecting state before repeating an uncertain action, as for Codex. - The ACPX restore waits up to 5 s for the original provider lifetime fence before it falls back to the old close. - The ACPX provider state file gains two fields with defaults. Older saved states still load. A downgraded runner cannot read state that a newer runner wrote with these fields. - Restored ACPX sessions now forward prefixed call ids. Only fresh sessions keep raw ids. Live proof covers Claude. Codex-ACP, Pi, Cursor, Copilot, Grok, Kimi, and Antigravity share this path, but this PR proves them only by unit tests and code review. - OpenCode is covered by the runner fixture and gate tests. The live OpenCode restart-chaos cell was not rerun here. - `runner-core` `unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend` is flaky when run in parallel. It also fails with the new path disabled, so it predates this change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
7840d894fa |
fix: retain files after accepted remote copy-back exhausts retries (#19491)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces keep unsaved files bound to the run that wrote them. > - An accepted result can exhaust its copy-back retries before those files are saved. > - Failed-workspace preservation rejects every accepted result, so a stopped writer can keep the workspace held after a newer setup attempt fails. > - This pull request lets the existing owner retain files from that exact stopped remote writer. > - File recovery can then release the hold without applying the old result or repeating its provider work. ## Linked Issues or Issue Description **What happened?** A failed native remote writer had an accepted result and exhausted copy-back retries. Its sandbox was stopped with a valid retained-source receipt. A newer attempt failed on the writer's unsaved workspace. Task Retry could not register the missing file-recovery obligation because preservation returned as soon as it found the old accepted result. **Expected behavior** Task Retry must let the existing recovery owner save the exact retained files. The accepted result, failed attempts and current task must stay intact. The old provider work must not run again. **Steps to reproduce** 1. Finish native work in a remote task workspace and accept its completion result. 2. Exhaust copy-back retries, stop and retain that exact sandbox, and leave the workspace dirty. 3. Let a newer setup attempt fail on the old writer's hold. 4. Use ordinary task Retry. Before this fix, the missing file obligation is never recorded. **Paperclip version or commit** Reproduced against master at `51a65af9b81f3c60f8210047e9a3eeaada7bc155` with isolated PostgreSQL and real files. Related: #19385 preserves accepted copy-back interrupted by Stop. #19422 admits saved input after verified recovery. #19464 and #19466 repair preparation and Retry continuation. This change covers an older remote writer whose accepted copy-back has exhausted its retries. ## What Changed - Admit only terminal, exhausted accepted copy-back on a failed remote writer with a finished run and no live controller or finalizer. - Require the exact stop-and-retain receipt, accepted completion contract and current dirty workspace binding. Keep native-sync sources on their existing export path. - Use the existing file-preservation recorder and worker. Add no new recovery worker, schema, polling or replay path. - Cover bound and orphaned entry points, reusable leases, changed proof and the ordinary HTTP Retry journey. Document the recovery behavior. ## Verification - Causal baseline: the corrected bound-writer and orphaned-lease tests both fail without the source change and pass with it. - Assembled acceptance: authenticated HTTP Retry records the old writer's missing obligation; the real recovery service saves its files; concurrent continuation calls queue exactly one successor for the newer failed setup. PostgreSQL, filesystem, tar, Git and the API/service path are real. Provider transport is controlled, and no model turn is dispatched. - `pnpm check:module-boundaries`, `pnpm -r typecheck`, `pnpm build` and `git diff --check` pass. - Four full affected recovery suites pass: **659 tests**. Command: `pnpm --dir server exec vitest run --config vitest.config.ts src/services/workspace-recovery.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-cancelled-workspace-preservation.test.ts src/services/native-runtime/native-workspace-export-retry.test.ts`. This includes the final 16 new regression, refusal and assembled journey cases. - `pnpm verify --explain` completed. It selected broad transitive coverage; this is planning evidence only. Hosted CI will supply the remaining broad coverage. - Dependency cost: extra indexed source/result reads run only for this exact terminal retained-result path, inside the existing bounded transaction. Existing notifications activate the existing recovery owner. No idle scan is added. - Fixture failures are accounted for: the first fixture used an active-only stop helper on a released lease. The corrected fixture produced the causal baseline above. One negative fixture tried to remove a contract while an accepted result referenced it and failed the database foreign key. That impossible fixture was removed; the positive journey passed in the same run. Neither fixture failure is reported as a product regression or a passing full run. - Hosted CI is fully green at the reviewed commit: 54 successful checks and 3 intentional skips, with no incomplete or failed checks. Greptile is 5/5 with no unresolved threads. - Hosted rollout and a live task continuation remain pending. Local API admission does not prove a deployed model run. ## Risks This expands file preservation for a narrow terminal failure. Wrong source identity could retain unrelated files; unchanged writer, epoch, location, provider lease, stop receipt and accepted contract checks fail closed. Active ownership, unsafe archives, rejected results and native-sync sources stay on their existing paths. Existing recovery owns physical source locking, copy verification and bounded retries. It preserves unknown action outcomes and leaves the accepted result uncommitted. There is no migration or configuration change. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning effort. Context window size is unavailable. Used repository tools, isolated tests and code execution. No delegated agents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
026dc6d135 |
fix(native): release a session-resume owner when the resumed run fails before its provider claim (#19371)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a finalization row. Its lease owner tells the
control plane that an execution may still be live.
> - Pausing an agent waits for `agentExecutionsHaveStopped`. That check
refuses to finish while any run of the agent has a lease owner, unless
the run was cancelled and has stop proof.
> - A same-run session resume claims that lease before it dispatches the
run. If the run then fails before the native executor takes its own
provider claim, nothing releases the resume owner.
> - The leaked owner blocks pause and resume for good, and the task
keeps looking like it has a live recovery path.
> - This pull request settles those rows at the source, and it adds a
stop-proof-gated release for rows that already leaked.
> - The benefit is that agents can pause and resume after an ordinary
resume failure, without database edits, and the cancelled-run stop-proof
rule stays as it is.
## Linked Issues or Issue Description
**What happened?**
An agent stayed in `pausing` with `hostComplete=false` and climbing
attempts. Its runs had `native_run_finalizations` rows with `phase:
observed`, `recovery_state: resuming_session`, a `lease_owner` like
`default:resume:<uuid>`, and no `result_id`. Each heartbeat run had
ended `failed`: either `setup_failed` (`Failed to acquire lease for
environment … Sandbox creation has failed`,
`native_remote_recovery_lease_mismatch`) or `adapter_failed`
(`runner_state_identity_mismatch: …`). The lease expired about 20
minutes after the run finished. Nothing settled the row. The only fix
was to set `lease_owner` to null by hand.
**Expected behavior**
When a resumed run fails before any provider turn starts, its
finalization row goes to `terminal_failure` and its lease owner is
released, as the native executor does for failures it sees. A row that
leaked before this fix is released in the background once its lease
expires and the run has stop proof. Pause then completes.
**Steps to reproduce**
1. Leave a native run in `retryable_failure` with a persisted
checkpoint, on a task that already has an active recovery incident.
2. Let session resume claim it (`claimNativeSessionResumptions` sets
`phase: observed` and a `…:resume:…` owner).
3. Make the resumed run fail before the executor claims it, for example
when the environment driver cannot create the sandbox.
4. The heartbeat failure path marks the run `failed`. The issue release
path does not settle the row, because it settles only when it opens a
new incident. The owner stays set, and `agentExecutionsHaveStopped`
returns false indefinitely.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
06c0566a08 |
fix: make task Retry work after a refused automatic attempt (#19466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry stores a durable request before it starts a new run. > - An automatic attempt can stop at the execution gate before a provider starts. > - That cancelled attempt becomes the source of the next Retry click. > - Admission rejected that source, or chose an older run that did not match the stored Retry authority. > - This pull request verifies the refused attempt and keeps it as the exact authority source. > - The user can start one new turn after the older execution holds pass their existing checks. ## Linked Issues or Issue Description **What happened?** Retry can remain deferred when the latest run was cancelled with `execution_reconciliation_required` before dispatch. An older failed preparation has an execution recovery hold. The Retry receipt names the cancelled child, but admission only accepts a failed preparation as an extra source. It can also choose the older run as `previousRunId`, which fails the exact source check at dispatch. **Expected behavior** Retry starts one new turn when the latest attempt has a verified pre-dispatch refusal and every older execution hold is ready. Provider activity, retained execution resources, missing receipts, or a live earlier process keep the hold. **Steps to reproduce** 1. Seed a failed native preparation and its recovery hold. 2. Let a later automatic attempt be cancelled by the stale queued execution gate before dispatch. 3. Click Retry against that latest cancelled attempt. 4. Before this change, continuation admission returns no receipt. After this change, one successor starts with authority bound to the clicked attempt. **Paperclip version or commit** Reproduced on `2f35eb1080`. Rebased onto `6cafcaf89b` for submission. **Deployment mode** Local source checkout with embedded PostgreSQL and the native Runner on macOS arm64. Refs #15383 and #16939. Those changes cover legacy conversation cancellation and queue promotion. This change covers the exact source of a task Retry receipt after a gate refused its automatic child. ## What Changed - Reuse the existing database-backed pre-dispatch refusal proof when admitting the exact Retry source. - Keep the clicked source as `previousRunId`. Continue to check every older execution hold. - Add database regressions for safe refusal, unsafe evidence, concurrent dispatch, and duplicate clicks. - Add an opt-in acceptance test that drives the authenticated Retry route through a real Codex turn and task finalization. - Document the admission contract. ## Verification - TDD: the new database regression returned `null` on the original code. It passes with the repair. - Final head `c43a84b1bf`: 487 tests pass across task Retry, explicit native continuation, and execution continuation. Server TypeScript and module boundary checks pass. - Live acceptance: `PAPERCLIP_LIVE_REFUSED_RETRY=true PAPERCLIP_RUNNER_BINARY=<current-source-runnerd> pnpm --dir server exec vitest run --config vitest.config.ts src/services/native-runtime/refused-task-retry.live.test.ts`. - The real HTTP request returned 202 on the final rebased head `c43a84b1bf`. Codex `gpt-6.1-sol` with high effort produced the requested file. The task became done, both task locks cleared, and finalization committed. A repeated click reused the same receipt and did not create another run. The source file stayed unchanged. - Server TypeScript check and module boundary check. - `pnpm verify --explain` returned 2 because the execution contract doc requires broad CI coverage. The required CI checks supply that coverage. ## Risks This changes execution admission. The existing proof requires the stored refusal receipt and no provider work or retained execution resources. Earlier recovery holds, process stop checks, decisions, actor authority, and task ownership still apply. No schema change or migration. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The host does not expose an exact author model ID or context window. The live acceptance provider was `gpt-6.1-sol` with high reasoning effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — all GitHub Actions checks pass on `c43a84b1bf`; Paperclip Review completed at 5/5 with no findings (neutral check conclusion). - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — reviewed `c43a84b1bf` with no actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e680fa6814 |
fix(native): retry startup interrupted by a closed runtime API listener (#19462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs prepare a remote workspace, then start a runner and a provider session > - A graceful server shutdown closes the HTTP listener before it drains runs > - A run that is still preparing then asks for the local runtime API origin, and the lookup fails with `local_runtime_api_listener_unavailable` > - Recovery did not know this code. It treated the failure as an ambiguous provider state and blocked the task behind a board-owned action, although no runner or provider process existed > - This pull request classifies the closed listener as a definite pre-session failure > - The benefit is that a deploy no longer strands a preparing task. The same run gets a bounded bootstrap retry that the next server process claims ## Linked Issues or Issue Description **What happened** A native Codex task on a remote sandbox spent about ten minutes in workspace preparation. A deploy started a graceful shutdown during that time. When preparation finished, the executor called `resolveLocalRuntimeApiUrl()` to build the tool authority. The listener was already closed, so it threw `local_runtime_api_listener_unavailable`. No runner backend or provider session had started. The failure classifier returned the generic `native_session_interrupted` code. `nativeProviderRecoveryEvidence` then returned `ambiguous_state`. The run became a terminal failure, the task became blocked, and the recovery action told the board to resolve an ambiguous session and to not open a replacement session. The same signature occurred on three other runs during an earlier deploy. **Expected behavior** A failure that occurs before any runner or provider process exists is safe to retry on the same run. The task must stay in progress with a scheduled, bounded bootstrap retry. If a provider event or checkpoint exists, recovery must stay operator-owned. **Steps to reproduce** Start a native run. Close the server's HTTP listener before the executor creates the runner backend. Observe a `terminal_failure` with `recoveryMode: "ambiguous_state"` and a blocked task. The new integration test reproduces this with the real `resolveLocalRuntimeApiUrl()` and a real Postgres database. **Environment** Server native runtime, remote sandbox target, `codex_app_server` driver. Related recovery work: #19249, #19304, #19305. ## What Changed - `nativeSessionFailureSourceCode` returns a new `local_runtime_api_listener_unavailable` source code for this error. - `nativeProviderRecoveryEvidence` treats that code as definitely pre-session. The existing guard stays: the run must have no checkpoint and no durable provider event. - The chat failed-run retry accepts the code in its list of ordinary pre-session failures, so an exhausted startup retry can still be retried from chat. - New integration tests in `native-startup-retry.integration.test.ts` cover the retry path and the operator-owned path when a provider event exists. ## Verification - Red before the fix: the new test failed with `phase: "terminal_failure"`, `recoveryMode: "ambiguous_state"`, `recoverable: false` and a board-owned recovery. This matches the failed run on the deployed instance. - Green after the fix: `cd server && npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` passes 26 of 26 tests. The retry test asserts `retryable_failure`, `bootstrap_retry`, backoff timing, an agent-owned `resume_native_run` wake, and a claim through `claimNativeSessionResumptions`. - Safety case: when a durable `turn.started` event exists, the same error stays `terminal_failure` with `ambiguous_state` and a blocked task. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "listener|classif|pre-session"` passes. - Not run: a live deploy that interrupts a preparing sandbox run. The test uses the real listener lookup and the real recovery tables, but not a real sandbox. ## Risks - Low risk. The change only affects runs that fail with this exact error and have no checkpoint and no provider event. - The retry uses the existing bootstrap budget of three attempts with backoff. Repeated failures still stop and hand the task to the board. - The GitHub callback bridge that started in the sandbox before the failure is a per-attempt helper. The existing cleanup path releases it. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use, code execution and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
11bc490cda |
fix(recovery): keep a stopped copy-back's files instead of holding the workspace (#19385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run can have its result accepted before the copy-back of its sandbox files finishes. If the copy-back fails, it retries. > - A board Stop during those retries ends the run. It sets the run's finalization to `terminal_failure` / `native_retry_cancelled`, and the result stays accepted. > - After that, nothing exports the files. The finalization worker skips terminal rows, and cancelled-workspace preservation skipped any run with an accepted result. > - The task workspace stays dirty and bound to the stopped run. Every later run on the task waits for the workspace, then fails setup with the unfinished-writer error. > - This pull request lets cancelled-workspace preservation also admit an accepted result whose copy-back a Stop ended. > - The benefit is that stopping a stuck copy-back saves its files and frees the task, like stopping any other run. ## Linked Issues or Issue Description Refs #16700, #17677, #18820, #19304 and #19305 (cancelled-workspace preservation). This pull request adds the accepted-result case to that path and keeps its other checks unchanged. **What happened?** On a hosted instance, a native run's result was accepted, but its copy-back kept failing and retrying. A board operator pressed Stop to unblock the task. The run became `cancelled`, its sandbox was stopped and kept, and its finalization moved to `terminal_failure` / `native_retry_cancelled` with the result still accepted. The next run on the task was cancelled with "Shared execution workspace is busy: run … ended, but its workspace save has not settled". About 24 automatic retries followed over an hour. The last one failed with "The workspace still has an unfinished writer or save operation." After that nothing ran: the task stayed `in_progress` with no run and no recovery action. A Retry or a new comment would hit the same block. The workspace stayed `dirty: true` with the stopped run as its writer, and no `workspaceRestoreRecovery` was ever recorded. The reason: `preserveCancelledNativeWorkspaces` requires `resultId IS NULL`, and `registerCancelledWorkspacePreservationWork` and `preserveNativeFailedWorkspace` both return early when `resultId` is set. **Expected behavior** Stopping a run during its copy-back retries keeps its files, as stopping any other run does. Paperclip's workspace recovery saves them and releases the workspace. Nothing commits or replays the stopped result. **Steps to reproduce** 1. Accept a native result for a run whose copy-back is in `retryable_failure`. 2. Stop the run with the board Stop. The finalization becomes `terminal_failure` / `native_retry_cancelled`. 3. Start another run on the task. It waits for the workspace and then fails setup. No save of the stopped run's files is ever recorded. The new database tests run steps 1 and 2 against PostgreSQL and check that the save is recorded. **Paperclip version or commit** `master` at `9ceaec6f97e`. **Deployment mode** Any deployment that uses remote (sandbox) task workspaces. Seen on a cloud-managed instance. ## What Changed - `server/src/services/native-runtime/native-failed-workspace-recovery.ts`: cancelled-workspace preservation now also admits an accepted result when the finalization is `terminal_failure` with `native_retry_cancelled`. That is the state a Stop leaves after ending copy-back retries. All three entry points use the same rule: - the startup scan `preserveCancelledNativeWorkspaces`; - the worker `registerCancelledWorkspacePreservationWork`; - `preserveNativeFailedWorkspace`, only when it is called with `requireCancelledStop`. Preservation still requires everything it required before: a cancelled run, an acknowledged Stop, no live finalization owner or controller lease, a prepared workspace reference, and the exact retained source. - An accepted result with any other failure code is still skipped; those runs keep their own export repair path. Failed (not cancelled) runs are unchanged. - The accepted result stays recorded and uncommitted. Preservation does not change the task's status or queue a run. - `doc/execution-semantics.md`: describe the stopped copy-back case in the paragraph on writers stopped mid-run. - `native-cancelled-workspace-preservation.test.ts`: move the accepted-result fixture into an `acceptResult` helper, and add a test with three cases. ## Verification - Failing first. The new test "preserves files of an accepted result whose copy-back was stopped" has three cases: worker, startup scan, and a sandbox that the Stop already stopped and kept, through the real `prepareSandboxStopAndRetain` and `settleStopOnlyCleanup`. All three fail on `master`, with no `workspaceRestoreFailure` recorded, and pass with this change. - Each case checks that: - the save obligation and recovery action are recorded for the exact lease; - `inspectWorkspaceRecoverySource` finds one source; - the finalization keeps its `resultId`, `terminal_failure` and `native_retry_cancelled`; - the workspace stays held by the stopped writer until the save publishes; - no run or wake is created, and the task status is unchanged. - The existing "does not admit preservation for accepted-result" case still passes. It uses an accepted result with a different failure code. - `cd server && npx vitest run src/services/native-runtime/native-cancelled-workspace-preservation.test.ts`: 44 passed. - Related suites, 10 files and 854 passed: - `native-failed-workspace-recovery`, `workspace-recovery`, `heartbeat/run-control` - `native-cancellation-request` (unit and integration) - `heartbeat-remote-workspace-busy`, `native-workspace-export-retry`, `native-workspace-reconstruction` - `native-stop-accounting`, `task-workspace-binding` - `cd server && npx tsc --noEmit -p tsconfig.json`: pass. - Not run end to end on a live instance. Diagnosed from that instance's run, finalization, lease, workspace and recovery-action rows. ## Risks - On deploy, the startup scan and the worker will pick up existing cancelled runs in this state and record their saves. That is the intended repair, and it uses the same retention checks as other stopped writers. - A copy-back interrupted by a server restart can leave a `remote_workspace_snapshot` operation marked `running` after its process is gone. This change does not settle such rows. If one blocks the later save, the save follows its existing retry and board hand-off. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs, read-only database queries). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
240d46e2ec |
fix(native-runtime): join the deferred observer seen when a duplicate append arrives (#19397)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runtime writes runner events through
`PaperclipControlPlanePort.appendEvent`.
> - A terminal event can defer its post-commit observer. A duplicate
append of the same event must join that observer. It must not run
`onDuplicateEvent`.
> - The duplicate looked up the observer only after its own event
transaction finished. If the original observer settled during that
transaction, the observer left the map. Then the duplicate ran
`onDuplicateEvent`.
> - This made the test "joins the exact deferred observer on duplicate
append..." fail on slow CI (job 127633060467, and once on Windows).
> - This pull request captures the pending observer when the duplicate
arrives. The join decision no longer depends on database latency.
> - The benefit is a deterministic join and a stable test.
## Linked Issues or Issue Description
**What happened?**
The server test `paperclip-control-plane-port.test.ts > joins the exact
deferred observer on duplicate append and replays it after a new port
opens` failed intermittently with `Error: must join the original owned
observation`. The test gives the duplicate append 30 ms and then
releases the original observer. On a slow runner, the duplicate's event
transaction took longer than 30 ms. The original observer settled and
its `.then` handler removed the map entry. The duplicate then found no
observer after its transaction and ran `onDuplicateEvent`. (The CI
failure took 90 ms.)
**Expected behavior**
A duplicate append that arrives while the original deferred observer is
pending joins that observer. Database latency does not change this
result.
**Steps to reproduce**
1. Add a 60 ms sleep inside the `appendEvent` transaction when the
receipt disposition is `duplicate`.
2. Run `pnpm vitest run
src/services/native-runtime/paperclip-control-plane-port.test.ts -t
"joins the exact deferred"` from `server/`.
3. The test fails with `must join the original owned observation`. This
is the same error and stack as CI.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
958f9aa9f3 |
feat(issues): limit agent-created task trees to parent and child (#19312)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents split work into tasks. A task can have a parent task. > - Today agents can nest tasks to any depth: parent, child, grandchild, and deeper. > - Deep agent-made trees are hard to follow, and an agent that works on a child task makes grandchildren by default. > - This pull request limits agent-created trees to two levels: a top-level task and its children. > - A request for a grandchild does not fail. The server puts the new task beside the requester and tells the agent. > - Board users keep full control. They can still nest to any depth. Existing deep trees stay valid. > - The pull request also records which task created each new task, so the audit trail stays complete when the parent changes. ## Linked Issues or Issue Description No public issue exists. This is the feature description. Related: #16853 (budgets that include descendant work) also reasons about task trees; this change does not alter its traversal. **Subsystem affected** Issues (tasks): create, child create and move in the issue service and routes. Agent tools, chat suggestions and creation provenance use these paths. **Problem or motivation** Agents create subtasks as deep as they want. An agent that runs on a child task gets that child as the default parent, so its follow-ups become grandchildren. The trees become deep and hard to read. The parent link is also the only record of which task created a follow-up, so that record is lost when the parent changes. **Proposed solution** Limit agent-created trees to a parent and its children. When an agent asks for a grandchild, put the new task under the requester's parent and return `parentAdjusted`. Do not reject the request. Keep board users unrestricted. Record the originating task in a new server-set `origin_issue_id` column. Keep `origin_run_id` as the originating run. **Alternatives considered** Reject grandchild requests with an error. We did not do this, because agents then fail mid-task. Add a database constraint. We did not do this, because existing deep trees must stay valid and nesting can return later. ## What Changed - Add `MAX_AGENT_TASK_HIERARCHY_DEPTH` (2) and the `IssueParentAdjustment` type in `packages/shared`. - Add `resolveNewTaskParent` and `planIssueParentChange` in `server/src/services/issues.ts`. `create`, `createChild` and `update` use the agent limit by default (`parentDepthPolicy: "flatten"`). - The issue create, `/children` and PATCH routes pass `allow_nested` for board users. For agents, the routes check authorization against the parent that the server actually writes. - The requester keeps its behavior when the server moves a task beside it: `blockParentUntilDone` blocks the requester, and the new task inherits the requester's workspace and privacy. - The server never moves a task above a low-trust boundary root that contains the requester. - These subsystems keep any depth, because they find their work by parent: watchdog reviews, monitor recovery, email-channel placement, and routines that a person configured. - An agent move of a task that has children skips only the parent change and returns `reason: "has_children"`. - Add `issues.origin_issue_id` (nullable, `on delete set null`, indexed) in migration `0434`. Only the server sets it. The `createdFrom` detail field, the `createdFromIssueId` filter and task-creation provenance read it first. - Chat prompts now let a chat agent create one task, a task with children, or separate tasks. The conversation is never a parent. - The runner `create_task` tool accepts an optional `parentTaskId`. A chat agent can create a top-level task and then put subtasks under it. The run must be able to read and change the parent. A conversation is never a parent. The depth limit still applies. Project tools map the field to `parentId` through the issue route. The protocol manifest, semantic contracts and eval fixture are regenerated. - Watchdog runs that have an active watchdog keep their requested parent on all create paths, because their access is limited to the watched subtree. - Fix a bug that the test drive found: the chat (MCP) `create_task` ignored the requested `status`, so a `backlog` task became `todo` and started to run. It now uses the same rule as the native tool. - Update `skills/paperclip`, `doc/SPEC-implementation.md`, and add the plan `doc/plans/2026-11-05-two-level-task-hierarchy.md`. ## Verification - `cd server && npx vitest run src/__tests__/agent-task-hierarchy-depth.test.ts src/__tests__/agent-task-hierarchy-depth-routes.test.ts`: 19 new tests for service and route behavior, all pass. - Related suites (64 files: issues service, thread interactions, privacy, watchdogs, provenance, runner tool authority, decisions, routines, project tools): 1208 tests pass. Two tests changed because of the intended behavior and pass now. Two `email-channels.integration` tests fail locally because the test cannot find a bundled skill file from this worktree path. This change does not touch that code. - `packages/db`: migration safety, snapshot drift and journal tests pass. - Typecheck passes for `server`, `packages/shared`, `packages/db`, `cli` and `ui` (local `ui` errors come only from generated Storybook catalog files that are absent). - Local test drive on an isolated instance with real Claude agents: 29 of 29 use cases pass, including an agent on a child task (follow-ups placed beside it, with the notice) and a chat that creates a top-level task with three subtasks. - I did not run the full `pnpm test:run` or `pnpm build` locally. CI runs them. ## Risks - Behavior change for agents: some follow-ups that were grandchildren are now siblings. The parent of the requester is woken when all its children finish, which now includes these tasks. - The runner `create_task` input contract adds one optional field, so the protocol manifest and the pack contract change. The tool output does not change: it shows the actual `parentId`, and the activity log records `parentAdjusted`. - Migration `0434` adds a nullable column, a foreign key and an index. It does not backfill rows. - `IssueCreationSource.run` is now nullable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
36643d9378 |
fix(native): retire finished owners whose continued turn rotated authority (#19373)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner runs per-turn agent sessions in managed sandboxes.
A controller keeps a durable session root for each session.
> - When a per-turn owner finishes without a saved runner checkpoint,
the next turn retires that owner. Retirement first checks that the
session root holds only known entries.
> - The runner transport writes two entries that the check does not
know: `authority-epochs/` and `runtime-context.json`. A continued turn
writes `authority-epochs/` when it rotates authority from the turn
before it.
> - So the third turn of an ephemeral-lease task cannot retire the
second turn's owner. The run fails with
`native_continuation_requires_reconciliation` and the task waits for a
user message.
> - This pull request classifies every session-root entry in one
contract module, and adds a test that scans the writers so the list
cannot drift again.
> - The benefit is that follow-up turns continue without manual help,
and any future refusal names the unexpected entry.
## Linked Issues or Issue Description
Refs #18846, Refs #19061, Refs #19355
**What happened?**
On staging, the third turn of every task for one agent (gpt-5.6-sol,
native runner, ephemeral sandbox lease) failed. The error was
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=unexpected_root_entry`. The first two turns
succeeded. The task then went to `blocked` with "Saved work recovered.
Send a message to continue."
**Expected behavior**
The third turn retires the finished second-turn owner and continues the
conversation.
**Steps to reproduce**
1. Assign a task to a native agent that uses an ephemeral sandbox lease.
2. Let the first turn finish. Reopen the task with a comment so a second
turn runs and finishes.
3. Reopen the task again. The third turn fails with the error above.
**Paperclip version or commit**
Staging near 9a79384958; reproduced locally on master
|
||
|
DottaandPaperclip
|
7c0922a7b6 |
fix(native): preserve provider backups during slow workspace saves (#19359)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions save provider state so another web server can continue the same conversation. > - A provider can suspend while its accepted result is still saving a large workspace. > - The run has released its live lease at that point. A new backup-owner lookup rejects this uncommitted state. > - This pull request captures the real owner during backend creation and retains it with the current session authority. > - The existing database checks still reject a changed boot, generation, result, or successor before inspection and during publication. > - The benefit is that a slow workspace save no longer prevents the provider backup. ## Linked Issues or Issue Description **What happened?** A remote session can fail to save its provider backup with `native_harness_storage_owner_missing`. The provider has suspended, but its accepted result is still in `workspace_finalizing`. The close callback tries to acquire a new owner after the run released its lease. A later allocation cannot use a backup that was never published. **Expected behavior** Save the suspended provider with the authority acquired during execution. Recheck that authority before provider inspection and before and after upload. Keep result finalization and provider-work permissions unchanged. **Steps to reproduce** 1. Start a remote native session with durable provider storage. 2. Accept its result and release the live lease while its workspace save remains pending. 3. Close the provider transport. The old owner lookup rejects the save before provider inspection. **Paperclip version or commit** Reproduced on master `bf10bc695abaef6b20e85b287f456b5ce2edb266`. **Deployment mode** A web server with object storage and remote managed Runner sessions. Related: #19225 retains failed-turn ownership through close. #18436 restores an external controller checkpoint before a changed warm process is replaced. This change fixes late backup-owner acquisition while an accepted result saves its workspace. ## What Changed - Capture the provider backup owner when the remote backend still owns its execution lease. - Bind that owner to the current session authority. An older close callback selects the current warm attachment's owner. Revalidate the captured fence before inspecting or skipping provider state. - Add a regression for the close callback during workspace finalization. - Restore encrypted provider files with a captured owner while workspace finalization remains pending. Reject boot, generation, result, and successor changes during upload. - Document the ownership contract. ## Verification - Observed the callback regression fail on the old code with `native_harness_storage_owner_missing`. The same test passes with this fix. - `cd server && node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 59 passed, using real PostgreSQL and encrypted object write/readback. - `node scripts/check-module-boundaries.mjs`: passed. - Full workspace typecheck: passed with the complete production fix. - Native execution: 1,151 passed. Managed warm lifecycle: 9 passed. These tests require a lost-owner error during shutdown, including before an incomplete provider checkpoint is skipped. The final change to the lifecycle test also requires rejection after lease renewal is lost. - Full local `pnpm build`: passed on the final head. - The full local `pnpm test:run` stopped in chat shard 6 on five existing deadline assertions. Earlier chat shards passed; the PR CI chat shards passed. This is not a full local-suite pass. - The first CI server shard 4 attempt failed the unchanged legacy environment-inheritance fixture at its five-second completion wait. The same failure reproduced locally on the repair head and on an earlier checkout without this repair. One unchanged CI retry passed. No fixture assertion or timeout change is shipped. - Latest-head CI has passed. Greptile reviewed the final head at 5/5 with no open threads. The separate Paperclip Review is paused. This PR does not claim staging acceptance. ## Risks The backup owner now lives with its session authority. A wrong authority selection must fail rather than publish provider state. Database publication still checks the original company, task, run, boot, generation, and accepted result before and after upload. Tests change the owner during upload and verify that the selected backup stays intact. No schema, data migration, or new provider-work permission changes. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning and code execution tools. The host does not expose the exact runtime model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
a6ab6a301c |
fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
789a71bc17 |
fix(runner): retry quarantined tasks in their saved workspace (#19340)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain execution ownership and action history across turns. > - Task Retry must prove that old execution has stopped before it starts another turn. > - An attempt can inherit its predecessor's identity and fail before it launches a provider. > - The cleanup proof searched under that failed attempt and compared unrelated event sequences. > - This pull request verifies the retained owner and starts one fresh authorized turn. > - Saved results and uncertain action outcomes remain intact. ## Linked Issues or Issue Description **What happened?** Task Retry accepted repeated clicks but kept the canonical request deferred with `executionWait.reason: local_cleanup`. It created no successor, although the retained local worker and provider had stopped. **Expected behavior:** After exact cleanup proof, admit one fresh turn. Preserve the old journal and result. Keep unresolved action outcomes unknown. **Steps to reproduce:** 1. Finish a local native turn with retained cleanup state and a saved result. 2. Let the next attempt inherit its identity and fail with `native_session_cleanup_quarantined` before any launch. 3. Upgrade across a tool-contract change, then click task Retry. 4. The old code requires the failed attempt's stop receipt and journal identity. It also requires the current tool fingerprint and equal sequence numbers across raw and normalized events. Those requirements cannot match the retained owner. **Paperclip version or commit:** Reproduced on `31bf6e4482`. **Deployment mode / installation method:** Authenticated, self-hosted source checkout. **Agent adapter(s) involved:** Native `paperclip_runner`, local Codex app-server. The shared physical-stop proof also covers ACPX. **Database mode:** Embedded PostgreSQL 18.1. **Access context:** Board task Retry. **Node.js version / operating system:** Incident: Node 24.20.0, Linux ARM64. Local verification: macOS ARM64. **Relevant logs or output:** Accepted Retry stays in `deferred_issue_execution`; no new run or provider launch occurs. **Relevant config:** Per-turn local native execution. The old tool contract differs from the current contract. **Additional context:** Related: #19311 prevents the oversized tool result that can lead to retained cleanup. #18217 covers broader cleanup isolation and retained-result maintenance. #19134 admits stopped Runner sources under the current run lease. This patch fixes explicit continuation admission on current master. ## What Changed - Resolve the retained journal's owner under the same company, task, agent, workspace, native session, and runner instance. - Require terminal ownership, settled controllers, released local leases, and exact inherited process birth. Refuse any launch evidence on the failed attempt. - Accept a current host's verified process absence from its exact launch receipt when an older server lacks a stop receipt. A later launch invalidates that observation. - Validate normalized receipts against their own IDs, sequences, and digests. Match ordered provider identities to the independent raw journal. - Keep tool compatibility at the session reuse boundary. Explicit continuation always uses a fresh session. - Bind the stop proof to its exact successor and canonical Retry receipt. Reuse the proven local workspace through normal preparation. Refuse changed bindings, closed or remote workspaces, pending operations, and other live leases. Projectless run-ID placeholders keep the existing fresh-workspace fallback. - Retain journals, results, files, and uncertain action outcomes. Add regression and opt-in live acceptance coverage. - Make the split-handoff browser test wait for both child-completion notices and all run settlement before teardown. Assert the final task set stays unchanged. ## Verification - Red: canonical Retry on the original code accepts one saved intent but creates zero successors and records `local_cleanup`. - Red: distinct raw and normalized event sequences reproduce that wait after owner resolution alone. After admission was repaired, live acceptance then exposed the empty-attempt workspace problem. A PostgreSQL regression also reproduced that wrong workspace selection. - Green at `6684df5e82`: **479 targeted cases**, including 26 new PostgreSQL cases for duplicate clicks, exact process and workspace ownership, controller/lease fences, changed state, and independent receipt integrity. The complete 1,150-test native executor suite passed earlier. - `pnpm -r typecheck` and `pnpm build` pass again at final head `432a52bea8`, including its build stamp. Token gates pass. Master now contains the unrelated directory-fixture type correction found during the first typecheck. - Real authenticated HTTP Retry with Codex `gpt-6.1-sol`, high effort, and per-turn execution passes again at final head `432a52bea8` (93.3 seconds). It preserves the first file line, appends the second once, starts a fresh session in the existing workspace, keeps the retained journal unchanged, commits finalization, and clears task locks. - The strengthened split-handoff browser scenario passes **three consecutive local runs**. Its original CI teardown and one same-head rerun failed when cleanup cancelled an active legacy completion-notification run and hit a PostgreSQL row-lock timeout. The scenario now waits for the complete handoff before cleanup. [Initial failure](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127523693634), [same-head rerun](https://github.com/paperclipai/paperclip/actions/runs/42106466518/job/127531698321). These failures remain part of the verification record. - The local full `pnpm test:run` passed all six chat integration shards. Its obsolete pre-rebase attempt was stopped during the general server phase; no complete local-suite pass is claimed. Full CI, including both required gates, passed at `6684df5e82`. At final head `432a52bea8`, all initial CI groups passed except the unchanged status-card deadline test, whose post-wakeup clock assertion missed its bound by 10 ms. That exact case passed three unchanged local repetitions (16.8/27.0/18.3 seconds); the single isolated shard rerun passed all **2,218 tests** (123 files), with no source changes. [Initial timing failure](https://github.com/paperclipai/paperclip/actions/runs/42114473652/job/127550722187). - Greptile reviewed `6684df5e82` at **5/5**, with no actionable findings. A second reviewer identified the supported projectless placeholder case. A new real PostgreSQL regression reproduced its workspace-binding failure before the one-line fallback correction; all **45 focused cases** now pass, including default environment selection and changed binding/cwd refusal. Greptile reviewed final head `432a52bea8` at **5/5**, with no blocking findings. The second reviewer’s thread is resolved. - The patched Linux build verified the actual retained worker/provider ownership and process absence in a transaction that was rolled back. No retained state was retired or provider started during that pre-deployment check. ## Risks - This changes an execution admission proof. Incorrect ownership could permit a second executor. The proof retains process, controller, lease, scope, journal, and receipt fences, and duplicate admission is tested through the real queue. - Missing or conflicting evidence remains deferred. Windows, remote execution, automatic replacement, and session reuse retain their existing policies. - No database migration or UI change. The original affected task still needs deployment of the fix before recovery can be claimed. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, SSH diagnostics, and live provider testing. This session does not expose the exact backend model build or context-window size. The live test agent uses OpenAI `gpt-6.1-sol` at high effort. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
2506bcdaf4 |
Admit exact stopped Runner sources under the current run lease (#19134)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution keeps provider state and a durable controller journal. > - A completed process can stop before the next run attaches. > - The current run must own recovery of that exact source. > - An old controller must lose permission to change the consumed source journal. > - This pull request adds explicit source admission and protected task dispatch. > - The benefit is a narrow recovery boundary that preserves the original successful result. ## Linked Issues or Issue Description Native prerequisites #19132 and #19153 are merged into master. This PR targets master and updates the TypeScript Runner driver contract alongside Core and SDK admission. This change adds the SDK and Core admission boundary. Executor selection and fresh source preparation are a separate change. Related #17916 supplies the workspace export holds preserved here. ## What Changed - Plugins advertise `codexSourceRecoveryVersion: 1` for Codex App Server and `acpxSourceRecoveryVersion: 1` for ACPX. Hosts can qualify either, both, or neither. Core selects the matching field, because a stopped-source check for one provider cannot establish the other provider’s recovery guarantees. ACPX orchestration is tracked separately in #19196. Use the existing exact source identity and `resumeFrom` under the authenticated current run. No public recovery mode is added. Require the claimed source, current account and prepared launch at protected dispatch. - Keep the first source binding pinned on the consumer lease, including after uncertain dispatch. A newly acquired coordinator can refresh only the same source. - Claim one stopped successful source under the current run's acquired native lease. Store the token-free consumer reference in existing lease metadata. Preserve the source status and result. - Check the exact six source identity fields, encrypted journal header, company, task, agent, environment, driver, workspace, account selection, and current coordinator generation. Keep workspace export and legacy settlement holds. - Serialize claims and checkpoint writers in the same coordinator lock order. Reject stale source writes after consumption. Check the current consumer before recovered source writes. - Check source admission before and after protected worker dispatch. Keep the request and lease owned by the current run while the native identity remains the source. - Allow the same owner to prepare B after a fresh source submission, using the exact consumed-source proof instead of a generic warm-reuse flag. Recheck that proof after dispatch. - Preserve ordinary cached adoption and repeated preparation after launch. The current authenticated owner can differ from the adopted PRP identity without requesting a new source launch. Keep ordinary suspended recovery and live task attachment checks. Extract the shared source resolver to avoid a module cycle, and make ordinary selection honor the same durable source consumer. ## Verification - Current head `cac9cf188a` includes merged #19300 and the symmetric capabilities introduced in `b7e25a64b4`. The task-owner and warm-rebind fixtures now identify their physical task. All 29 SDK schema cases, 95 Core ownership and source-admission cases, 25 native Codex and ACPX recovery cases, and the binary metadata check passed on this head. The SDK and Runner builds and server typecheck passed. Both native providers reject the other provider’s capability. GitHub CI is green and [standard Greptile scored this commit 5/5](https://github.com/paperclipai/paperclip/pull/19134#issuecomment-6645574114) with no open threads. - Previous head `bda49271a4ae362a15baf3fe6d6d297e25fd2000` is directly based on master after #19132 and #19153 landed. [GitHub workflow 42072192023](https://github.com/paperclipai/paperclip/actions/runs/42072192023) passed on its first attempt. All check runs completed successfully or as intentional workflow skips; Snyk passed. Standard Greptile scored this exact head 5/5. That review had no unresolved findings. - Runner and SDK TypeScript builds and server typecheck passed. The focused SDK, task-owner, task-runtime, PostgreSQL source and authority suites passed all 161 cases. The production transport-factory getter regression and 14 native-lane selection checks also passed. - The Linux native lane executed the new ordinary warm A→B process case. It kept one Runner/provider process and completed two turns on the same thread. The eager-getter mutation fails with `native_runner_source_attachment_pending`; preserving the live getter passes. The production factory test changes the current owner after transport creation and checks that same options object. - PostgreSQL cases cover both concurrent source-writer/claim orders, exact source submission under the current caller, source-switch denial after uncertain dispatch, same-source coordinator refresh, stale same-boot generation, corrupted journal headers, competing consumers, account changes and export holds. - The protected dispatch sequence submits A, prepares B and retries B preparation. It rejects undispatched sources, changed accounts and foreign target bindings. Source status and accepted result remain unchanged. - Ordinary adopted A preparation under current B, repeated ordinary preparation after launch and ordinary suspended continuation retain their existing path. Absent or false claims preserve ordinary identity denial; malformed claims fail closed. - Earlier development attempts exposed ordinary preparation regressions and the live-getter omission; the final regressions cover those fixes. The first embedded PostgreSQL attempt needed missing library aliases repaired in ignored dependencies. Those attempts are not passing evidence. - Historical, superseded CI heads had unchanged UI fake-timer and Playwright health-response-lifetime failures. Their isolated cases and scoped retries passed. The response-lifetime cause remains unproven. No UI/test timeout was changed. The final rebased-head workflow above passed without a retry. - Whitespace and added-line secret/PII scans passed. Full workspace checks ran on GitHub. No live vendor probe or deployment was performed. ## Risks - Any driver that used the earlier Codex TypeScript option must use `codexSourceRecoveryVersion`. The new plugin description rejects the generic spelling. The Launcher’s existing Codex-specific HTTP header can map to this property when the plugin adopts the new SDK. The change requires no journal migration or ROM change. - This is an inactive admission foundation. Executor orchestration is separate in #19157. Native restore integrity and provider stopped-source/account validation must be present before executor activation. The native prerequisite uses ordinary recovery followed by attachment. The composed executor proofs are in #19157. - The source claim is durable. A later controller must recover the same consumer or keep the hold. It cannot select another source after an ambiguous launch. - Admission adds bounded database checks around protected startup calls. Busy issue locks fail closed; other lock waits have a one-second bound. - No migration, snapshot format, proactive receipt, recurring worker, or deployment is included. Full workspace checks and platform coverage run in CI. These tests do not prove vendor history durability or a hosted rollout. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning, through Codex with repository tools and code execution. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the current head - [x] Standard Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
22281d4b3a |
feat(task-chat): approve harness-proposed plans (Codex plan mode, Claude ExitPlanMode) (#19291)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run through the native runner. Codex and Claude Code each have their own plan mode, and each produces a proposed-plan document. > - Paperclip already has a planning work mode, an issue `plan` document, and a `request_confirmation` approval gate for plans. > - These parts were not connected. The Codex proposed-plan item was transcript text only. The server path never put Codex into plan mode. Claude was never put into plan mode, and its ExitPlanMode plan was dropped. A planning run could close its task as done with no approval. > - This pull request turns a harness's own proposed plan into the issue's Plan document and the existing approval gate, for Codex and Claude. > - The benefit is that users approve or send back the agent's real plan from the task chat. Approve resumes the agent. Request changes sends the reason back to the agent. ## Linked Issues or Issue Description Refs #19252 **Subsystem affected** Native runner provider events (Codex and ACPX Claude), the runner Codex provider (Rust), server native plan synchronization, and the task chat UI. **Problem or motivation** Codex and Claude both produce a proposed-plan document in their native plan mode. Paperclip did not use it. In live tests on the server path, a Codex planning run was not in Codex plan mode, saved its own `implementation-plan` document, and marked the task done without approval. Claude planning runs were not in Claude plan mode, and the ExitPlanMode plan text never reached Paperclip. **Proposed solution** Add a canonical `plan.proposed` event for a harness's proposed plan. Put Codex and Claude into their native plan mode for planning runs. Save a completed proposal as the issue `plan` document with the existing `request_confirmation` gate. Agents can also propose a plan on a standard task; the work mode stays standard. **Alternatives considered** Reuse `plan.updated`. Its schema is the within-turn checklist and is not the durable Plan document, so a separate event keeps the two clear. Stronger prompt text without native plan mode. Live tests showed that the model kept its own document workflow. **Roadmap alignment** This reuses the issue plan document, planning work mode and approval gate that already exist. It adds one canonical provider event. ## What Changed - `packages/paperclip-runner/protocol/schemas/*`: add the `plan.proposed` event (`paperclip.plan.proposed.v1`: `proposalId`, `status`, `markdown` up to 60k characters, `source`). Regenerate the schema bundle, validators and manifest. - `packages/paperclip-runner/src/provider-events.ts`: map the Codex `plan` item and a `<proposed_plan>` block in a completed Codex message to `plan.proposed`. Add `claudePlanProposalEvent`. - `packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs`: send Codex's native `collaborationMode: plan` on `turn/start` for a read-only (planning) Codex run. Before this change the server path never sent it. - `runner-core/src/acpx_event_payload.rs`: admit `plan.proposed` as ACPX rich display activity. - `drivers/acpx/codex-runtime-adapter.ts`: set Claude Code to native plan mode for a read-only run. Capture Claude's ExitPlanMode plan from its permission request, record it, and decline the native "Ready to code?" prompt. - `drivers/acpx/codex-acpx-driver.ts`, `cli/acpx-runtime-sidecar.ts`: emit the captured proposal as a rich event. Complete that turn instead of reporting it as cancelled. - `backends/codex-native-backend.ts`: per-harness plan-mode instructions, and a short opt-in hint for Claude on standard tasks. - `server/src/services/native-runtime/native-session-executor.ts`: synchronize a completed `plan.proposed` into the `plan` document and a `request_confirmation` in planning and standard mode. Do not change the issue status during the run (that revoked the run's lease). The run finalizer turns the pending confirmation into the review wait. - `server/src/services/heartbeat-run-summary.ts`: keep a `<proposed_plan>` block out of the posted reply. - UI: new `plan_proposal` provider family. The task chat shows a live "Writing plan" card while a proposal streams and a one-line record when it completes. Resolved plan reviews keep a receipt ("Approved Plan v1", "Requested changes to Plan v1: reason"). Plan previews skip section headings. - Storybook: `Tasks and conversations/.../Task chat/Proposed plan approval` shows the journey. - `doc/SPEC-implementation.md`: describe harness-proposed plan approval. ## Verification - `cargo test -p paperclip-runner-core --lib` (new: `read_only_codex_turns_use_native_plan_collaboration_mode`) - `cd packages/paperclip-runner && npx vitest run src/provider-events.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts` - `cd server && npx vitest run src/__tests__/native-plan-proposal-sync.test.ts src/__tests__/heartbeat-run-summary.test.ts` - `cd ui && npx vitest run src/components/transcript/native-run-events.test.ts src/components/task-chat` - Live test on a local worktree instance with real harnesses: - Codex (`gpt-5.6-sol`), planning task: Codex asked one question, then produced a native plan item. The task showed Plan v1 with Approve plan / Request changes and moved to in review. Approve changed the work mode to standard. Codex then implemented the plan and its tests passed. - Codex, Request changes with a reason: Codex woke in planning mode and produced Plan v2 that includes the change. - Claude (`claude-opus-5-5`), planning task: Claude was in plan mode. ExitPlanMode became Plan v1 and the turn completed. Approve resumed Claude in standard mode, and Claude implemented the plan. - Claude, standard task that asks for a plan first: Claude stopped before code changes and asked for plan approval. The work mode stayed standard. - Storybook: open `Proposed plan approval` and check each stage. ## Risks - Codex planning runs now really use Codex plan mode on the server path. Codex can ask a clarifying question before it proposes. This matches Codex plan mode. - Claude planning runs now use Claude plan mode. A declined ExitPlanMode ends the Claude turn; the runner reports it as completed. Claude's tool row for that request records a rejection from Claude's side. - Codex does not stream the proposed plan on the server path yet, so the live card shows a placeholder until the plan completes. - A `<proposed_plan>` block in any completed Codex message on a standard task now asks for approval. This is the documented opt-in path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, browser). Long context. No extended thinking mode was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b86f55f0ba |
test(server): remove two more server test races (#19330)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server Vitest shards gate every pull request > - Two more server tests failed on pull requests that do not touch them (#19309, #19265) > - #19314 fixed four other flakes. These two have different causes > - Random CI failures slow every contributor and hide real regressions > - This pull request removes both root causes without retries or skips > - The benefit is fewer false CI failures ## Linked Issues or Issue Description Refs #19309, #19265, #19314 **What happened?** - #19309, job 127459915333: `workspace-recovery.test.ts` > "keeps ancestor Retry held when current proof changes: sibling_hold" received `requestId: null` from a second Retry request. - #19265, job 127481105563: `paperclip-control-plane-port.test.ts` > "retains a durably recorded governed terminal while its postcommit observer is pending: checkpoint timeout" failed with "native session checkpoint timed out after 100ms" before the terminal was recorded. **Root causes** 1. Workspace recovery (test race). `POST /issues/:id/retry` returns 202 and then starts `heartbeat.resumeTaskRetry` without waiting. The proof-change tests change state after the receipt. When the background nudge runs first, it admits the Retry and queues a run before the change. The new queued run is then the latest run, so the second Retry in `sibling_hold` is stale. I proved this locally: a 1.5 s pause after the receipt made every gate in the family fail with a queued run in the issue history. 2. Control-plane checkpoint timeout (budget too tight). `checkpointTimeoutMs: 100` applies to every checkpoint in the session, including the normal checkpoints before the terminal. On a loaded runner, one of those took more than 100 ms. The test expects only the checkpoint that the pending observer blocks to time out. ## What Changed - `server/src/services/workspace-recovery.test.ts`: add `receiptOnlyRecoveryApp`. It builds the same route app, but the route nudge does nothing. The two proof-change families ("keeps ancestor Retry held…" and "holds a failed-preparation HTTP Retry…") use it and call `resumeTaskRetry` explicitly, as before. - `server/src/services/native-runtime/paperclip-control-plane-port.test.ts`: the checkpoint budget is 1 s. The timeout case now waits for the run to fail with the timeout while the observer is still blocked, instead of a fixed 700 ms sleep. This also proves that the timeout fires while the observer is pending. ## Verification - With the forced 1.5 s pause and the fix applied: both workspace-recovery families pass (80 passed). - `npx vitest run src/services/workspace-recovery.test.ts -t "keeps ancestor Retry held when current proof changes|holds a failed-preparation HTTP Retry when current proof changes"`: 80 passed. - `npx vitest run src/services/native-runtime/paperclip-control-plane-port.test.ts`: 94 passed, run 3 times. ## Risks - Low risk. The changes are only in tests. The route nudge is a latency hint ("Maintenance redelivers it"). Other tests still cover it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |