mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
f0ddccff71f3bbda975cac8a86053a67c5d554f3
7726
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Devin FoleyandPaperclip
|
94fccc3180 |
fix(workspaces): preserve snapshots beside runtime sockets (#19159)
## Thinking Path > - A workspace can contain saved source files alongside transient runtime IPC endpoints. > - The portable archive supports files, directories and confined symlinks; treating a Unix socket as a fatal entry prevents every file beside it from being saved. > - Keep the existing archive owner and validation. Record socket paths as exclusions, leaving the source endpoint and adjacent work intact. ## Linked Issues or Issue Description **What happened?** Retained workspace recovery repeatedly fails with `remote_workspace_archive_unsupported_entry` when the source tree contains a Unix socket, preventing adjacent source files from being saved. A real socket reproduces the error in local capture and the serialized sandbox helper. **Expected behavior** Save regular workspace files, record runtime socket paths as exclusions, and leave the source endpoints and new file edits intact. **Steps to reproduce** 1. Create a temporary workspace with an uncommitted text file. 2. Open a Unix domain socket in that directory with a Node server. 3. Capture the workspace through the local archive function or generated sandbox helper. Before this fix both captures fail; afterward the saved file restores and the socket exclusion is recorded. **Paperclip version or commit** Reproduced on `6fa54d55779295d90c455389cebdd3da96607d20` before this change. The socket fixture establishes this failure class; it does not establish the entry type behind every historical unsupported-entry occurrence. ## What Changed - Exclude Unix domain socket endpoints from portable workspace captures and include their paths in the existing exclusion manifest. - Keep source sockets in place. Preserve normal files and the existing rejection of named pipes, device entries and unsafe links. - Exercise local and serialized capture, an unchanged source endpoint, restoration of adjacent uncommitted work, FIFO rejection, and a socket becoming a real file during capture. A changed source still fails safely and retains that new file. - Document the portable snapshot behavior. ## Verification - Red regression: both local and serialized captures failed with `remote_workspace_archive_unsupported_entry` before the change. - `pnpm exec vitest run server/src/services/remote-workspace-snapshot-archive.test.ts`: **43 passed** using real filesystem sockets, pipes, Git repositories and archives. - `pnpm -r typecheck` and `pnpm build`: passed. - Module boundaries and `git diff --check`: passed. Added source and PR text were scanned for credentials, private incident identifiers and personal data before push. - Snapshot storage and save-concurrency consumer suites pass **23 tests**. Their first local runs exposed missing workspace SDK build links and unhydrated macOS embedded-Postgres library symlinks in the new checkout. Repairing those prerequisites made the unchanged consumer suites pass; the earlier failures remain recorded and no assertion was weakened. - Greptile reviewed `e313565a5088abce448f11b29982462aa338a692` at **5/5** with no unresolved threads. All current-head CI checks are green and the branch is conflict-free. Hosted recovery needs a qualified deployment and original-task verification. ## Risks Socket endpoints represent runtime IPC and are not recreated by archive restore. Their paths are visible in the existing exclusion manifest. This adds no archive entry type, subprocess, recurring work, provider call or dependency. It does not alter source authorization, stop receipts, ordinary file validation, restore safety or recovery budgets. Unix socket/FIFO fixtures are skipped on Windows; existing portable archive coverage still runs there. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, xhigh reasoning. Used repository tools, code execution and real local filesystem/process fixtures. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] Added focused behavioral and preservation regression coverage - [x] Updated the execution contract and documented limits - [x] Typecheck and build pass locally - [x] Scanned the diff and description before push - [x] Current-head CI is fully green - [x] Greptile is 5/5 with no unresolved comments Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
7edbc1fc07 |
fix(server): keep a successor's issue lock when a finalizing native run releases its checkout (#19156)
`release_checkout` now clears `checkoutRunId` and the execution lock only when they belong to the committing run, so a successor that claimed the issue while its predecessor was finalizing keeps its claim and passes reviewed-chat attestation. `attestReviewedExternalChatRun` reports the first failed check as a closed label. Setup failures store it in `resultJson.reviewedChatBinding.denialReason`, and Sentry run-failure diagnostics export it as `execution.reviewedChatBindingDenialReason` (no IDs or text). The error message is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1105.0-canary.2 |
||
|
Devin FoleyandClaude Opus 5.5
|
6fa54d5577 |
fix(codex-local): route typed ACP credential rejections to sign-in recovery (#19146)
Classify a typed Codex ACP `access` failure as `acpx_auth_required` when the provider text reports `401 Unauthorized`, `invalid_api_key`, or an incorrect/invalid API key. Other `access` failures keep `acpx_turn_failed`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
482ec0aea9 |
fix(native): run saved copy-back retries on schedule behind a slow export (#19145)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs copy their sandbox workspace back after the provider finishes. A failed copy-back saves a retry 30 seconds later, with a bound of 3 attempts. > - One shared delivery queue runs these retries. A pass of that queue ends only when its slowest check ends. > - A legitimate sandbox export can take 10+ minutes. While it runs, all other due retries wait. > - On staging, a run active across a deploy showed "Reconnecting" with an overdue retry for about 24 minutes. Its retries ran at +24 min and +3.5 min instead of +30 s. It settled only after about 32 minutes. > - This pull request lets idle check slots admit entries that come due during a pass. It does not overlap a running check's resource and does not start work after abort or drain. > - The benefit is that the existing 3-attempt bound settles a failing copy-back in about 1 minute, not after the slowest unrelated export. ## Linked Issues or Issue Description **What happened?** A native run whose result is durable but whose workspace copy-back failed stays in `running` with the label "Reconnecting" for a long time. Its saved retry time is in the past. The retry does not run until an unrelated slow copy-back in the same queue pass finishes. **Steps to reproduce** 1. Start two native runs on different tasks with accepted results. 2. Let the copy-back of run A take many minutes. 3. Let the copy-back of run B fail. It saves a retry 30 seconds later. 4. Watch run B. Its retry does not run until run A's copy-back ends. **Expected behavior** Run B retries on its saved schedule. After 3 failed attempts it settles with `native_workspace_sync_out_retry_exhausted` and a recovery action, while run A continues. **Paperclip version or commit** master at |
||
|
|
c2cc69d1e4 |
fix(recovery): resume and settle preserved native results (#19034)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution can save an accepted result before workspace export fails. > - Recover must preserve the files and settle that same result without another agent turn. > - The export failure can own the task's Blocked transition and its recovery notice. > - Recovery must prove preservation, stop, source identity and current authority before settlement. > - Recover can also transfer repair ownership to the system, which must keep explicit board export repair available. > - This change restores those paths while preserving later task state, live owners and unknown action outcomes. ## Linked Issues or Issue Description Related #17916 protects unfinished exports. Related #19004 preserves deliberate Retry authority. This change repairs accepted-result finalization and its recovery notice. **What happened?** A native run accepted its result, then exhausted workspace export. Normal Recover could save the files but leave the result and old recovery notice unsettled. A saved Retry could also use a different generation from preservation. Separately, Recover changed the action owner from board to system. Explicit export Retry then rejected that same source as stale, even when its accepted result and stopped allocation still matched. **Expected behavior** Preserve files, finish the original accepted result and close only its obsolete notice after exact proof. Keep explicit board export repair available after system adoption. Preserve current authority, Stop, later task changes, newer execution and every physical ownership fence. **Steps to reproduce** 1. Accept a native result and exhaust its workspace export. 2. Request ordinary Recover, optionally with an existing saved Retry. 3. Let file preservation complete. Inspect the task and source recovery notice. 4. To reproduce the explicit-repair defect, call export Retry after Recover adopts the action. Before this repair it rejects the valid system-owned source before any provider call. ## What Changed - Record the exact accepted result, contract, task status version and prior decision when export exhaustion creates Blocked. - Bind passive finalization to complete, source-bound preservation and the current chat/review authority. Recheck the proof under status commit locks. - Keep the saved Retry identity separate from its preservation generation. Retain coalescing and reject malformed or mismatched history. - Preserve an existing published GitHub assessment for the no-publication status-unproven wait. Keep all semantic, result, authority and publication checks. - Close an old export notice only after its own accepted result, export or preservation receipts, effects, projection and old Runner ownership settle. Keep newer work and files unchanged. - Permit explicit board export Retry after Recover transfers the exact source to the system. Require valid system and workspace recovery records for that same run, plus every existing result, stop, allocation, task and shared physical-lock check. Reject missing proof, another source and agent ownership before provider access. - Merge current master while retaining the stronger Stop, source and verified preservation checks. - Update the execution contract. No schema, endpoint, Runner protocol, model, trust or provider artifact change. ## Verification - Current head: `9be5a73489801095c9f39c9bfcb285a22b055912`. - The explicit-repair regression fails before the new guard change: ordinary Recover adopts the real database action, then export Retry rejects it as stale. It passes after the repair. - On the merged source, three owning PostgreSQL/file suites pass **220 tests**: `preserved-native-response-wait.integration.test.ts`, `native-workspace-export-retry.test.ts` and `idle-completed-history.test.ts`. The export suite includes four unrelated-owner/source rejection controls and both board-owned and system-adopted active-exporter controls. Results and run counts prove that repair creates no provider turn. - Server layered typecheck, module boundaries, `git diff --check`, and the added-line secret/private-incident scan pass. - Full workspace typecheck (`pnpm -r typecheck`) and build (`pnpm build`) pass. The complete local `pnpm test:run` is still running; its completed serialized chat partitions pass. Current-head CI is green (53 successful checks, 3 expected skips). Greptile reviewed `9be5a73489801095c9f39c9bfcb285a22b055912` at 2026-11-05 00:58 UTC with confidence **5/5**; all review threads are resolved. - Original qualification remains supporting evidence: the accepted-result fixture exercises HTTP Recover, real files, the finalizer, saved Retry coalescing, eight commit races, current principal/resource/head changes, Stop, later task changes, notice closure and repeated delivery. No accepted response, provider turn or publication is replayed. - Earlier failures remain recorded. Original red controls exposed wrong-action preservation, unrelated finalization barriers, generation mismatch and a lost GitHub assessment; the original repair added controls for them. The prior CI run [41908572248](https://github.com/paperclipai/paperclip/actions/runs/41908572248) failed an unchanged completed-history runtime fixture because its loopback service refused connections through the original ten-second readiness window. That owning suite passes unchanged in the 220-case run above. The earlier readiness failure's cause remains unproven; no wait or assertion was weakened. - Local and CI tests do not establish hosted task recovery. The serving revision and original-task outcomes require separate verification after rollout. ## Risks - Historical failures without the exact export-owned Blocked receipt keep their current task status. A saved result does not grant execution or publication authority. - Explicit export repair still resumes only the same retained allocation under the shared physical lock. It cannot take over an active exporter, create another agent turn, choose a replacement sandbox or discard saved files. - Retiring a source notice does not release physical resources or modify a newer task owner, later Done state or saved snapshot history. - The proof adds bounded reads only for concrete recovery obligations. It adds no recurring worker or polling timer. ## Model Used OpenAI GPT-6 through Codex for the original repair. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge, integration verification and system-adopted export repair. Both used repository tools, code execution and PostgreSQL tests. The runtime does not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and searched for duplicate or related work - [x] I have described the issue and linked related public PRs - [x] I have included no internal task identifiers or private incident links - [x] I have added behavioral and safety regression tests - [x] I have documented verification limits and risks - [x] I have updated the execution contract - [x] All Paperclip CI gates are green on the current head - [x] Greptile is 5/5 on the current head with no unresolved comments - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
9e5039f6c8 |
fix(daytona): retain duplex close ownership after lease cleanup (#19150)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote agents use provider plugins to manage sandbox leases and duplex channels. > - The host verifies the exact route and session when a bound channel closes. > - Daytona lease teardown removed that binding before a later host close. > - The incomplete reply made the host retire the shared worker. A later command then failed during setup. > - This pull request retains bounded close receipts and joins the original cleanup. > - Later work can use the same worker while invalid or unconfirmed closes still fail closed. ## Linked Issues or Issue Description **What happened?** A normal lease teardown can close a duplex channel before the host closes its bound route. Daytona then replies with only the route ID. The host requires the original session ID too. It logs `duplex channel close not acknowledged; retiring worker`, stops the worker, and rejects a later `environmentExecute` call as stopped. **Expected behavior** A later close must identify the original route and session and join its existing cleanup. It must not revive the channel or restart the worker. Unknown ownership and an unconfirmed close must retain the existing failure behavior. **Steps to reproduce** 1. Acquire two leases through one Daytona provider worker. 2. Open a duplex channel on the first lease. 3. Release that lease, then close the bound channel through the host. 4. Execute a command on the second lease. **Paperclip version or commit** The real host/worker regression fails on `69dea878c0b920b335594debb0286443ba9f6c35`. This branch includes master `3a72efebe5b050e7594bb309e71a0d5f3072db00`. **Deployment mode** Managed Daytona sandbox execution in Cloud. The regression uses a local real host and SDK worker. Only the external Daytona SDK and PTY are simulated. Related work: Refs #19004, Refs #10429, Refs #18065, Refs #19050. These cover task Retry, managed worker startup, drain release, and stale provider sessions. ## What Changed - Retain at most 4096 retired route/session bindings with their original close promises. - Remove live channel authority before cleanup starts. Concurrent and later closes join the same promise and return the stored binding. - Reject reuse of a retained route ID. Keep unknown and evicted routes unable to provide a session acknowledgement. - Add a real child-process host/SDK-worker/Daytona regression with only the external Daytona SDK and PTY simulated. - Cover release, cancellation, stop, destroy, resume, a stalled close, concurrent closes, failed cleanup, forged IDs and bounded eviction. - Document close receipt scope and its separate provider-termination checks. ## Verification - RED before the repair: the actual host and SDK worker reproduced the retirement log, SIGKILL, stopped state, and later `environmentExecute` rejection. The direct provider cases also failed because the session ID was missing. - Combined-branch provider suite: 364 passed. - Combined-branch host manager, duplex, managed readiness and composed suites: 142 passed. The final test-only manifest-loading change passed all 6 composed cases again. - Full repository typecheck (`node scripts/typecheck-low-memory.mjs`), focused server types and the standalone Daytona build passed. The provider build uses pinned SDK `0.203.0` and TypeScript `5.9.3`. - The composed cases check the same worker/PID, one later command, preserved saved content, revoked channel writes/stops, and continued rejection of commands on closed leases. A stalled close still retires the worker and dispatches no later command. - Reproduce with `pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/plugin-worker-manager-daytona-close.test.ts` from a checkout with the SDK build dependencies prepared. Run the provider suite with its own Vitest config. - An initial local typecheck lacked the excluded providers' SDK links after an install with scripts disabled. The standard root postinstall linker restored those local links, and full typechecking then passed. The fixture loads its actual built manifest. Failed setup output is retained; no type configuration was relaxed. - The first two composed attempts stopped during fixture setup. They are not the behavioral RED above. Their failed outputs are retained. - The first broad local typecheck stopped at an IPC permission error during preflight. It passed after rerunning with allowed local IPC and the standard SDK links. - [Full repository CI](https://github.com/paperclipai/paperclip/actions/runs/41945022881) passed all 48 jobs on this head. Selection fell back to the complete task plan, so the run was unfiltered. The complete check inventory has 54 successes and two expected Storybook skips; the separate Snyk status passed. Fresh Greptile review on `daebbeb5005b9d9bd59d32ea30094279048b9213` is [5/5 with no actionable comments](https://github.com/paperclipai/paperclip/pull/19150#issuecomment-6646621914). Independent production and fixture review found no actionable findings. Hosted acceptance of this repair has not run. ## Risks - A receipt removed at the 4096-entry bound cannot confirm an old bound close. The existing host guard remains effective in that case. - Close acknowledgements identify channels. They do not certify remote process termination or clear unknown external work. Provider termination receipts and existing recovery gates remain in force. - Hosted evidence confirms the unacknowledged-close retirement class. The historical log does not distinguish a missing session ID from a timeout or another close error. The regression proves and repairs the missing-binding path; it also preserves timeout retirement. - The provider is outside the main pnpm workspace. A new Cloud image must compile its changed source. The unchanged plugin version alone does not prove activation. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source editing and tool execution. The exact model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a7a1059149 |
fix(evals): limit review restrictions to noncurrent artifacts
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
133c95da1b |
fix(evals): scope mobile review prohibitions and restrictions
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
204601dd19 |
fix(evals): bind installation claims in both directions
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ca648d97ae |
fix(evals): reject contradictory missing-skill recovery claims
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a400fd8a61 |
test(evals): scope question inspection to timeline opener
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e7af60dd98 |
fix(evals): scope timeline intent and possessive relations
Co-Authored-By: Paperclip <paperclip@paperclip.ing> |
||
|
DottaandPaperclip
|
ee7283efb5 |
test(evals): preserve pending question during summary inspection
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
74e89a073f |
test(evals): grade rendered question context text
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ed4f01b15a |
test(evals): observe visible multipart continuation summaries
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f1c8709326 |
fix(evals): preserve factual timeline handoff and known actors
Co-Authored-By: Paperclip <paperclip@paperclip.ing> |
||
|
DottaandPaperclip
|
e3e43e2fab |
fix(evals): bind mobile waivers to review targets
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dc93a04715 |
fix(evals): retain mobile review action waivers
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4c8672ab20 |
fix(evals): distinguish mobile check waivers from protection
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5de738b8aa |
fix(evals): accept named unresolved skill occupancy
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d52f38db7a |
fix(evals): accept bound reduced-passive timeline handoffs
Version the measured timeline oracle to v8 without changing query, evidence joins, or execution bounds. Co-Authored-By: Paperclip <paperclip@paperclip.ing> |
||
|
Dotta
|
5991e10109 | test(evals): select catalog-bound revision links | ||
|
Dotta
|
91517d9709 | test(evals): resolve displayed hire model identifiers | ||
|
DottaandPaperclip
|
0e8d141e54 |
fix(evals): retain request context through excluded destinations
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
32447178fe |
test(evals): version negative-destination admission repair
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
155ce1420a |
fix(evals): retain explicit negative chat destinations
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
11f2a5293d |
fix(native): name the proof that holds a recovered per-turn owner (#19061)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - A follow-up must prove that the previous Runner owner is finished before it starts a new Runner. #18846 added a retirement path for an owner that finished without a saved Runner checkpoint. > - On a test deployment, follow-ups still fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The retirement path has about twenty separate fences, and each refusal gives the same error. > - The read-only run data cannot tell which fence held these runs. A code fix without that evidence would be a guess. > - This pull request names the first failed retirement proof in the error. Admission behavior does not change. > - The benefit is that the next failure on a deployment identifies the exact fence, so the follow-up fix can target it. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #18992 (settlement from live state). **What happened?** On a test deployment, 16 runs in one day failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. All were per-turn `acpx_runtime` runs in a Daytona sandbox, and each got a new sandbox. In the runs that I inspected, the previous run committed its result. In one of them, `harness_state.backup.persist` and `session.checkpoint.persist` ran for about 346 seconds and failed, so the Runner left no suspension receipt or failover backup. That is the case that #18846 addresses, but the follow-ups still failed. The run log does not show the retirement message, and the error is the same for each fence. Thus the cause cannot be found from the run data. **Expected behavior** When admission holds a follow-up, the error names the proof that failed. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result without a Runner suspension receipt. 2. Make one retirement proof fail. For example, leave the owner phase `observed`, or leave a `runner.suspend` command pending. 3. Admit a follow-up in a new sandbox. 4. Before this change, every case fails with the same error text. **Paperclip version or commit** Observed on a deployment before `e107a75755`. The same error text is on `master`. ## What Changed - `retireFinishedPerTurnRunnerdOwner` returns the refusal reason instead of `false`. `readSettledRunnerdProjection` returns the journal refusal reason instead of `null`. - When admission tries the retirement and it fails, the error is `prior_owner_terminal_state_indeterminate; retirement_refused=<reason>`. A warm session gets no suffix, because retirement does not apply to it. - Reasons use only fixed lists of known names: owner phase, recovery state, controller fence reason, unsettled command type and status, sandbox outcome, and quarantine copy result. Any other stored value appears as `other`. Reasons do not include payloads, file names, or paths. - The same proofs apply in the same order. Admission accepts and refuses the same states as before. - `doc/execution-semantics.md` describes the suffix. ## Verification - **Failure before the change:** I ran the updated `native-recovered-owner-followup.test.ts` against the unchanged executor. 18 tests failed. Example: expected `...indeterminate; retirement_refused=owner_observed`, received `...indeterminate`. - **After the change:** `cd server && pnpm exec vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 passed. Each held case now asserts its reason: `owner_retryable_failure`, `controller_remote_controller_lease_live`, `owner_recovery_resuming_session`, `journal_commands_unsettled(runner.suspend.pending)`, `journal_tool_input_unanswered`, `quarantine_journal_identity_mismatch`, `runner_lifecycle_not_suspended`, and `quarantine_runner_lifecycle_not_suspended`. - The success cases still retire the owner and admit the follow-up. - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 1227 passed, 20 skipped. - `pnpm exec vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts src/__tests__/task-workspace-binding.test.ts`: passed. - `tsc --noEmit` reports no errors in the changed lines. This worktree did not build every workspace package, so the full-package typecheck shows unrelated missing-module errors. CI runs the full typecheck. - **Not verified live:** no deployment has run this change. ## Risks - Low risk. Admission logic is unchanged. Only the error text gets a suffix. - Consumers that match the error by prefix still match. `failed-run-retry.ts` uses `startsWith("runner_state_identity_mismatch: ")`. No code compares the full string outside tests. - This change does not fix the held runs. It finds the fence. A second change is needed after a deployment reports the reason. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution, and read-only browser access to a test deployment. The harness does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1105.0-canary.1 |
||
|
DottaandClaude Opus 5.5
|
3a72efebe5 |
fix(native): confirm no runner process when remote staging fails before launch (#19059)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native sandbox runs supervise a remote `paperclip-runnerd` process. When that process exits during a turn, the transport restarts it inside a reconnect grace window. > - The server launcher stages the runtime assets first. Then it sends one launch RPC that starts runnerd. > - When staging failed, the launcher reported the exit as unconfirmed. This happened although no launch RPC was sent and no process could exist. > - Reconnect recovery treats an unconfirmed exit as a possible second owner. So it stopped at once with `runner_process_termination_unconfirmed` and did not retry. > - This pull request confirms the exit for failures before the launch RPC. Failures after dispatch stay unconfirmed. > - The benefit is that a transient staging failure, such as a full sandbox disk, uses the normal bounded retry. A real failure ends with `transport_reconnect_grace_exceeded` and keeps the staging error as the detail. ## Linked Issues or Issue Description **What happened?** On a hosted staging deployment, 7 native ACPX runs failed in 24 hours with `Provider notification transport failed closed.` (`native_provider_terminal_failed`). In every run, the runner process exited during the turn. The replacement launch then failed while it uploaded the runtime context into the sandbox: `runner_process_termination_unconfirmed: replacement runner health could not be verified: ... .paperclip-upload-<uuid>.tar.gz: create: ... no space left on device`. The sandbox disk was full. No replacement process was started. But the launcher said that it could not confirm the exit. So recovery did not retry inside its grace window. **Expected behavior** A staging failure before the launch RPC proves that no replacement process exists. Recovery must retry within the configured reconnect grace. If the failure continues, the run must end with the reconnect-grace failure and the staging error. **Steps to reproduce** 1. Create a remote runner launcher whose `ensureArtifact` step throws, for example with an ENOSPC upload error. 2. Wait for the handle completion to reject. 3. Before this change, `isExitConfirmed()` returns `false`, and no launch RPC was sent. The transport then fails closed with `runner_process_termination_unconfirmed`. **Paperclip version or commit** Observed on staging revision `89703c3`. Reproduced on `master` at `be20d4ad97`. Related: #19035 and #19014 change other runner paths (replayed terminal turns and redacted external tool results). They do not change remote launcher exit evidence. #18999 adds a free-space reserve for host-volume seed copies. It does not change sandbox disk use or this classification. ## What Changed - `createRemoteRunnerProcessLauncher` records when it dispatches the launch RPC. If the completion fails before that point, the handle reports `isExitConfirmed() === true`. The original error is kept unchanged. - A failed or timed-out launch RPC still reports an unconfirmed exit, because the RPC may have started runnerd. - Added a launcher regression for a staging failure before dispatch. - Added safety cases for a failed and a timed-out launch RPC. - Added a transport regression with the real `paperclip-runnerd`. A replacement whose staging fails is retried, and the next launch restores the durable session without a notification failure. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote runner process supervision"` failed `confirms no replacement process exists when staging fails before the launch RPC` with `expected false to be true`. The two launch-RPC safety cases passed. - Green after the fix: `npx vitest run src/services/native-runtime/native-session-executor.test.ts` passes 1131 of 1131 tests. - `npx vitest run src/services/native-runtime/setup-readiness.test.ts` passes 98 of 98 tests. - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts` passes 246 of 246 tests with the real debug `paperclip-runnerd`. This includes the existing case where a replacement with a dispatched but unverifiable process still fails closed. - `npx tsc --noEmit -p server` and `pnpm --filter @paperclipai/paperclip-runner typecheck` pass. - Not run: a live hosted sandbox with a full disk. The staging data showed the exact error, but this change was not deployed there. ## Risks - Low risk. The change affects only failures that occur before the launch RPC. In that window, the launcher has not run any sandbox command that can start runnerd. - Retries now run for the reconnect grace window, which is bounded and uses the existing backoff. Each failed staging attempt cleans its own upload scratch. - This change does not free sandbox disk space. If the disk stays full, the run still fails, now with `transport_reconnect_grace_exceeded` and the ENOSPC detail. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use, shell execution, and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d8dbe55368 |
fix(workspaces): let a missing task repository ref be corrected (#19072)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can ask Paperclip to prepare a task repository at a specific
ref
> - Paperclip records that request and clones the repository at the next
admission
> - Paperclip did not check that the requested ref exists
> - When the ref was absent, every admission retried the same checkout
and failed setup
> - A corrected request was refused as a conflicting receipt, so the
task stayed blocked with no repair path
> - This pull request identifies a missing ref and lets a corrected
request replace it while nothing was published
> - The benefit is that a bad ref becomes a repairable error instead of
a permanent setup failure
## Linked Issues or Issue Description
**What happened?**
An agent asked for a catalog repository at ref `main`. The repository
default branch is `master` and has no `main` branch. Each later
admission failed with `setup_failed`: `git checkout main` reported
`pathspec 'main' did not match`. The task failed admission for more than
a week and became blocked. A new request with ref `master` returned 409
because the receipt with ref `main` already existed.
**Expected behavior**
Paperclip should report that the requested ref does not exist. A new
request with a valid ref should replace the bad ref when no checkout
exists.
**Steps to reproduce**
1. Use a repository whose only branch is `master`.
2. Request it as a task repository with `ref: "main"`.
3. Admit the task. Preparation fails, and the receipt goes to
`needs_repair`.
4. Request the same repository with `ref: "master"`. Before this change,
the request returns 409.
## What Changed
- `managed-repository-checkout.ts`: When `git checkout <ref>` fails with
Git's unknown-ref message, and each quiet `rev-parse --verify` probe
(`<ref>^{commit}`, `refs/remotes/origin/<ref>^{commit}`) exits 1, throw
`RepositoryRefNotFoundError`. The message tells the caller to request
the repository again with an existing branch, tag, or commit. Other
checkout failures keep their original cause.
- `execution-workspace-repositories.ts`: Record `failureCode:
"repository_ref_not_found"` for that error. Other failures keep
`repository_preparation_failed`.
- `execution-workspace-repositories.ts`: A request for the same
repository with a different ref replaces the ref only when all of these
are true: the receipt is `needs_repair` with `repository_ref_not_found`,
it has no pinned commit, the workspace is local (no remote authority),
and the checkout path is absent. The update is conditional on that state
and writes a `workspace.repository.ref_corrected` activity entry. All
other ref changes keep the existing 409 conflict.
- `execution-workspace-repositories.ts`: Local admission re-reads each
receipt inside the checkout lock and claims it with a conditional update
on its current ref. A correction that commits after the receipts were
listed is used, not discarded.
- Tests: one regression for the full missing-ref path, and one safety
test that keeps the receipt immutable when files exist at the checkout
path or when the failure was not a missing ref.
## Verification
- Red before the fix: `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/task-workspace-repositories-persistence.test.ts` failed
with `expected ... /ref "main" does not exist/ but got 'Failed to
prepare managed checkout fo…'`.
- Green after the fix: the same file passes (11 tests). The new test
admits with ref `main` (fails, `repository_ref_not_found`, no checkout),
corrects to `master`, admits again (ready, pinned to the source commit),
and then confirms that a later ref change after publication still
returns a conflict.
- The regression also holds the checkout lock while admission waits,
corrects the ref, and then releases the lock. Without the in-lock
re-read it fails with 409; with it, admission publishes the corrected
ref.
- `heartbeat-managed-clone-credentials.test.ts` passes: an
authentication failure during checkout keeps its cause and is not
reported as a missing ref.
- Other checkout suites pass (9 files, 317 tests): reopen, base storage,
expiration, project env, rehome, nested clone migration, object cache,
trust, source copy privacy.
- Neighbor suites pass: `managed-repository-checkout.test.ts`,
`task-workspace-repositories.test.ts`,
`task-workspace-repositories-trust.test.ts`,
`heartbeat-project-repositories.test.ts` (47 tests). In one parallel run
under heavy machine load, `heartbeat-project-repositories.test.ts` had
timing failures in different tests on each run. The same file passed
alone with this change (47/47) and on master.
- `tsc --noEmit` for `server` passes.
## Risks
- Low risk. The new correction path is narrow and fails closed. Remote
(sandbox) workspaces keep the existing conflict because the control
plane cannot inspect the remote checkout.
- Existing receipts that failed before this change have
`repository_preparation_failed`. They need one more failed admission to
get `repository_ref_not_found` before a correction is accepted.
- No schema change. `failure_code` is free text.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edit, browser read-only diagnostics).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5723fd82ca |
fix(native): release a fresh claim when task preparation fails (#19065)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run claims its finalization coordinator before task preparation. Then it starts the provider turn. > - Task preparation can fail after the claim and before any provider turn. > - In that case the executor kept its claim. Failure settlement read the claim as a live owner and stopped. Nothing failed the run. > - The 60-second execution-control watchdog then failed the run as `execution_finalization_deadline_exceeded`. The original error was lost. > - This pull request releases only that exact fresh, unstarted claim after renewal stops. > - The benefit is that ordinary settlement fails the run at once with its real error and keeps the workspace files. The watchdog stays a backstop. ## Linked Issues or Issue Description Refs #19028. That PR keeps the earlier failure code when the watchdog fails a run. In this case no earlier code is saved, so it cannot help. This PR removes the wait for the watchdog. I searched open PRs for duplicates. #18820, #19034 and #19038 touch near code, but none release a preparation claim. **What happened?** A native task with a prepared sandbox workspace failed during preparation, after the controller claim committed. The coordinator stayed `observed` and kept this controller's lease owner. The heartbeat failure path calls workspace preservation. Preservation refuses any lease owner without a matching restart claim, so it raised `NativeWorkspaceFinalizationBusyError`. The heartbeat then treated ownership as held and returned without a terminal write. About 60 seconds later the watchdog failed the run with a generic "verify the provider stopped" action. The real error (for example `native_harness_storage_owner_missing`) stayed only in the workspace operation record. **Expected behavior** When no provider turn started under a fresh claim, settle the failure like a failure before the claim: retain the source, fail the run with its own error, and do not wait for the watchdog. **Steps to reproduce** 1. Start a native run with a prepared remote workspace. 2. Make task preparation fail after `native.coordinator.claim` commits (for example, a preparation callback throws). 3. See that the run stays `running` until the control deadline fails it as `execution_finalization_deadline_exceeded`. ## What Changed - `native-execution-lease.ts`: add `releaseUnstartedNativeExecutionClaim`. It clears the lease owner and expiry only for the exact run, company, issue, owner and attempt, while the phase is `observed` and no result exists. - `native-session-executor.ts`: when task preparation fails after a fresh claim, mark that claim. The startup owner releases it after renewal stops. A restart-recovery claim, a startup-cleanup context, Stop, restart detach, or a different owner keeps the old behavior. - Tests: an executor regression for the fresh and restart cases, and a real-database test for release and workspace retention against the exact owner, a foreign owner, a different attempt and an accepted result. - `doc/execution-semantics.md`: state that the deadline is a backstop and describe the release. ## Verification - Red before the fix: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "releases only a fresh claim"` fails with "releaseUnstartedNativeExecutionClaim ... Number of calls: 0". - Green after the fix, with the same command. - `npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-failed-workspace-recovery.test.ts src/services/native-runtime/native-execution-lease.test.ts src/services/execution-control-reconciliation.test.ts` passes (embedded Postgres for the retention test). - `pnpm --filter @paperclipai/server typecheck` passes. - Not done: a live run against a real sandbox at this revision. The staging evidence is read-only. ## Risks - The release is fenced by the exact owner, attempt, `observed` phase and no result. Renewal stops before the release, so a renewal cannot see a missing owner and detach or cancel the run. - The fix covers failures inside task preparation only. Failures later in startup, or after a restart-recovery claim expires, still reach the watchdog. Those keep fail-closed behavior and need separate evidence. - If the release write fails, the run keeps the old watchdog path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9cdbcbeddc |
fix(workspaces): restore archived commits from verified project objects (#19138)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An ephemeral web server must preserve historical task files outside its local disk. > - Archived isolated Git workspaces can have a successful branch-deletion receipt. > - Their private checkout can be gone while the project cache still owns the exact commit. > - The importer rejects that cache because it applies private-checkout validation. > - This pull request validates the managed object source for receipt-based restoration. > - The importer restores the recorded commit without using the current shared working tree. ## Linked Issues or Issue Description Refs #18826, #18943, #19038. **What happened?** An archived isolated workspace has a valid successful cleanup receipt and a retained commit. Offline migration rejects it with `archived_git_source_unavailable,local_source_unsafe_or_missing`. The private checkout was removed. The company and project cache still contains the exact commit. **Expected behavior** Restore the immutable commit from the verified project object cache. Keep strict validation for saved private edit recipes. Reject missing receipts, a foreign cache, a different origin and revoked cleanup authority. **Steps to reproduce** 1. Create an isolated Git workspace from a managed project cache. 2. Archive it and retain its successful branch-deletion receipt. 3. Remove its private checkout while the verified project cache retains that commit. 4. Attempt offline workspace migration. ## What Changed - Extract the existing managed Git object-source validator. - Use that validator only for exact successful deletion receipts that refer to a canonical company and project cache. - Keep private restore-point source validation unchanged. - Fetch immutable objects at the recorded commit into private scratch. Recheck receipt authority before publication. - Add real Git and database tests for restoration and rejected authority changes. Document the preservation path. ## Verification - 129 focused migration, concurrency and isolated-source tests pass across three files. - The new restoration case fails against the previous importer with the reported source-unavailable error. - Tests remove the old home and restore the retained commit. A newer shared working tree and unrelated files are excluded. - Tests reject a foreign company cache, wrong origin, missing receipt and receipt revocation during upload. - Local server typecheck has no diagnostics in changed files. Stale linked dependency declarations prevent a clean local whole-server result. Current-head full CI passes (54 passing checks, three skipped). Greptile is 5/5 with no unresolved review threads. - A replay of the 32 affected records is running in an isolated staging copy. Two committed owner overlays are under test. This is not released-image or serving-instance acceptance. ## Risks - A deletion receipt must identify an exact commit that still exists. Missing objects continue to block migration. - Shared working-tree contents must never substitute for archived private files. Only the recorded commit is fetched through this path. - The change does not recreate private edit authority, retire recovery records or alter serving configuration. ## Model Used OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning, code editing, shell tools and test execution. The runtime does not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
cda9460ae9 |
fix(native): restart remote bootstrap after a shutdown during preparation (#19056)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs in a remote sandbox go through workspace preparation
before the Runner process starts.
> - When the server restarts, restart recovery decides if a remote run
can safely restart its bootstrap.
> - A remote run that never started a Runner process, never had a lease
owner, and has no provider evidence can restart its bootstrap. Recovery
calls this an "unclaimed bootstrap".
> - A graceful shutdown (SIGTERM) suspends every running native
coordinator. It sets `leaseExpiresAt = now`, also when the coordinator
has no lease owner.
> - The unclaimed-bootstrap check requires `leaseExpiresAt` to be null.
So a deploy during workspace preparation turns a safe bootstrap restart
into `native_restart_recovery_blocked` /
`ambiguous_remote_recovery_evidence`. The task is blocked and a board
recovery action is created.
> - This pull request lets the check accept an elapsed expiry when the
coordinator has no lease owner.
> - The benefit is that a deploy during remote workspace preparation no
longer blocks the task. Every other contradictory-evidence guard stays
in force.
## Linked Issues or Issue Description
**What happened?**
On a staging instance, three remote-sandbox runs failed with
`native_restart_recovery_blocked` and
`ambiguous_remote_recovery_evidence` after one deploy. Each run was in
native workspace preparation (`snapshot.git`, `pack`, `seed.verify`,
`stage.workspace`). No run had a process start, a process identity, a
provider event, or a checkpoint. The graceful shutdown event was the
only coordinator change. The next server then blocked each run in
restart recovery.
**Expected behavior**
A remote run that was suspended during preparation, before any Runner
launch or lease claim, restarts its bootstrap on the same run. Recovery
does not block the task.
**Steps to reproduce**
1. Prepare a remote native run (prepared native workspace sync, fenced
preparation, no process, no checkpoint).
2. Call `drainRunningRunsForShutdown("SIGTERM")` for the run.
3. Call `claimNativeRestartRecoveries` as a successor controller with
`restartKind: "hard"`.
4. Observe `{ kind: "blocked", reason:
"ambiguous_remote_recovery_evidence" }`. Without step 2, the same run
gets `bootstrap_incomplete`.
**Paperclip version or commit**
Reproduced on `master` at `be20d4ad97`.
**Deployment mode**
Remote sandbox environments with PostgreSQL.
Related: #18877 (remote checkpoint retry reclaim) and #18854
(pre-provider startup retry). Neither changes this predicate.
## What Changed
- `claimNativeRestartRecoveries`: the unclaimed remote bootstrap check
accepts `leaseExpiresAt` when the coordinator has no lease owner and the
expiry has elapsed. A null owner with an elapsed expiry grants no
authority.
- New regression: a remote run suspended by the real graceful-shutdown
drain during preparation is claimed as `bootstrap_incomplete`, with no
recovery action.
- New safety case: an ownerless coordinator with an unexpired lease
expiry still blocks with `ambiguous_remote_recovery_evidence`.
## Verification
- Red before the fix: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "graceful shutdown suspended during preparation"` returned `blocked`
/ `ambiguous_remote_recovery_evidence`.
- Green after the fix: the same test returns `bootstrap_incomplete` with
`providerAttempt: 0` and `controllerGeneration: 1`.
- The existing contradictory-evidence cases (`claimed`, `generation`,
`process`, `launch`, `provider-event`, `checkpoint`,
`invalid-workspace`, `unfenced`) and the new `unexpired-lease` case
still block.
- Full `native-runner-restart-recovery.integration.test.ts` suite and
server `tsc --noEmit` pass locally.
- Not run: a live deploy on a remote sandbox. The fix is not deployed.
The original blocked tasks are not recovered by this change.
## Risks
- Low risk. The change relaxes one field of one predicate. All other
unclaimed-bootstrap conditions stay: prepared workspace, fenced
preparation, attempt 0, generation 0, no controller identity, no lease
owner, no process identity, no session, no checkpoint, no provider
evidence, and no launch or tool event.
- The executor still requires the durable PRP root to prove an
incomplete bootstrap before it starts a fresh bootstrap.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, browser reads of staging). Standard context
window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
367aa16800 |
fix(daytona): drop the stale session id on every sandbox restart (#19050)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs can execute inside a Daytona sandbox through the Daytona sandbox provider plugin > - The plugin runs every command for a lease in one persistent Daytona process session and keeps that session id in memory > - Daytona auto-stops an idle sandbox, and a stopped sandbox loses all of its process sessions > - Only the execute and resume hooks dropped the stored session id before a restart. Workspace realize, sync-in, sync-out, runner ingress, and interactive setup also restart a stopped sandbox, but they kept the dead id > - When host-side workspace preparation stays idle for longer than the auto-stop interval, sync-in restarts the sandbox, and the next command sends the dead id. The native run then fails with `native_session_interrupted: session not found` > - This pull request moves the session-id clear into the one shared start helper, so every restart path drops the stale id > - The benefit is that a run that waits through a long preparation opens a fresh session instead of failing ## Linked Issues or Issue Description **What happened?** Native runs on Daytona failed with `native_session_interrupted` and the error `session not found`. The failure came about 4 seconds after the provider session started. In every observed run, the workspace pack phase took longer than 15 minutes (917 s to 1442 s). This is longer than the default Daytona `autoStopInterval` of 15 minutes. **Expected behavior** After Daytona restarts an auto-stopped sandbox, the next command opens a new process session and the run continues. **Steps to reproduce** 1. Run one command on a Daytona lease, so the plugin opens and stores a session. 2. Let the sandbox stop (Daytona auto-stop after an idle gap). 3. Call workspace realize or sync-in on the lease. The hook restarts the sandbox. 4. Run the next command. It uses the stored session id, and Daytona answers `session not found`. No related open PR exists. The undeployed native fixes on master (#18992, #19025, #19026, #19004) do not touch the Daytona session store. ## What Changed - `ensureSandboxStarted` in the Daytona plugin now takes the lease scope. When the sandbox is not running, it clears the stored session id before it starts or recovers the sandbox. - All seven callers (execute, resume, realize workspace, interactive setup, runner ingress, sync-in, sync-out) pass their lease scope. The compiler enforces the scope on any new caller. - The execute hook no longer has its own copy of the clear. The resume hook keeps its clear because it also closes duplex channels before the restart. - Added three regression tests: a fresh session after realize restarts an auto-stopped sandbox, a fresh session after sync-in restarts it, and no new session when realize finds the sandbox already running. ## Verification - Red before the fix: `pnpm exec vitest run src/plugin.test.ts -t "auto-stopped sandbox|already running"` in `packages/plugins/sandbox-providers/daytona`. The two restart tests failed with `expected "vi.fn()" to be called 2 times, but got 1 times`. The running-sandbox safety test passed. - Green after the fix: the full Daytona plugin suite (`pnpm exec vitest run`) passes, 542 passed and 8 skipped. - `pnpm typecheck` in the Daytona plugin package passes. - Not done: a live Daytona run that idles past the auto-stop interval. The fix needs deployment before the staging failure can be checked again. ## Risks - Low risk. The clear runs only when the sandbox is not in the `started` state. A stopped sandbox has no live session, so no live shell can leak. A running sandbox keeps its stored session, and a test covers this case. - The recover path for a recoverable `error` state now also drops the id. A recovered sandbox restarts, so its old session is also gone. - Duplex channels on a restarted sandbox are still closed only by the resume hook. This PR does not change that behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
9c6993f0aa |
test(evals): pin corrected chat catalog and restore retained attachments (#19135)
## Thinking Path > - Paperclip uses real Product E2E journeys to verify agent work. > - The trusted TestOps controller must bind the reviewed source and retain inspectable evidence. > - The last uniform attempt exposed bounded observation and evidence-retention defects. Its original outcomes remain unchanged. > - The reviewed successor corrects those boundaries with the original model, requests, counts, and clocks. > - Its private framework report stores canonical attachments once. The controller must validate and restore those references before normal reporting. > - This PR ports that reader support and pins the qualified source in one infrastructure change. It does not merge the full catalog or its core repair PRs. ## Linked Issues or Issue Description Refs #19127, #19083, #19086, #19084, #19093, #19095, #19100, #19112. **What happened?** The last uniform attempt retained real failed outcomes. It also lost the multipart-question original when evidence packaging exceeded its unchanged physical byte allowance. Review found independent boundaries in skill navigation, announcement observation, timeline attribution, hire history, credential questions, and team visibility. **Expected behavior** The next run must execute the same ten requests against one reviewed source. The controller must retain complete original evidence, validate private attachment references, and report the actual outcome. Historical failures and missing-original UNKNOWN grades must remain unchanged. **Steps to reproduce** Use the focused source controls linked above. The private framework control uses a real failed Playwright test and a 140 MiB attachment. Legacy duplicate carriers hit the old cap. Canonical original evidence passes both seals and strict archive restoration. The controller restores derived files and the existing TestOps exporter preserves the failed result. **Paperclip version/commit** Qualified candidate: `155dd1bc1c4b85ae107f6a0cd429720caa281367`. The controller port is based on current master. The full candidate catalog stays on its own immutable branch. **Deployment mode** Existing AWS/TestOps job 100, plan 102, branch master. This PR does not dispatch a run. ## What Changed - Pin the published qualified source and its exact launcher and descriptor fixtures. - Preserve current manifest history. Update twelve declaration fields across eleven cases: oracle versions and skill source-file provenance only. Preserve all 244 identities, 242 canonical declarations, queries, setup, human/native counts, clocks, profile, model, retry policy, and global cap of 8. - Port the exact source-owned compact framework helper and private-transfer reader. Keep the three existing required source/controller protocol helpers byte-identical. - Validate both sealed inventories before restoring registered attachments into the derived selected report tree. Verify exact bytes and hashes. Preserve seals and unregistered evidence. - Keep deep JSON, decoded-body, sensitive-key, path, and original/final status checks. Plain-text bodies and registered plain files fail closed if existing diagnostic redaction would change them; JSON remains under structured inspection. Keep all existing physical byte, entry, ZIP, time, privacy, and archive bounds. - Preserve registered clean metadata and attachment bytes. Reject content that needs privacy redaction instead of silently rewriting immutable evidence. ## Verification - 131 existing Node pin, selection, admission, adapter, reconciliation, and security controls pass. The explicit new target assertion and drift negatives remain active. - 86 current-controller controls pass. They cover legacy reports, privacy and static assets, canonical reference guards, both seals, strict archive restoration, derived files, the actual report normalizer, and the actual TestOps/Allure exporter. The original failed result stays failed; screenshot and trace hashes resolve. - Actual P1 controls reject both an unknown password in a decoded plain-text body and an unknown command token in a registered text file. Parent code fails both controls; corrected code passes. The composed source passes 87 affected controls, fresh server declarations, and full E2E typechecking. The old cfe head completed all 49 CI jobs without adverse conclusions, but its review found this real privacy defect. That CI result is not credited to this successor. - The existing report-lock preflight returns already-reviewed at `60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`. - The existing source SDK guard confirms 506 registered inputs, unchanged registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`, and unchanged runtime pins. No registry renewal or SDK build was needed. - Source launcher SHA-256: `2b9a6953ee51f0ea0dbbd475b0491d9a6c39a1d6835eeaa912e0dbb7cf987724`. Descriptor SHA-256: `e026d5c000102f8d50397dc59055d187388304f91dbca9a6ccfbe747004cd16d`. - No model call, evaluation dispatch, evidence download, reporting retry, core merge, or permission change was performed. Full current-head CI and review are pending. Separate source PR failures remain disclosed; local controls do not establish universal source CI readiness. - Seven cases have verified passing historical attempts. A uniform passing ten-case run remains unverified. The old lost multipart-question original remains UNKNOWN. ## Risks This changes a trusted evidence reader and the source selected for future measurements. The new helper is included in the controller import closure. Compact references require matching originals; missing, foreign, changed, unsafe, or redaction-required evidence fails closed. Legacy reporter formats remain valid. A single canonical evidence set can still exceed a bound. Current-head CI and material review must pass before normal protected merge. Actual AWS/TestOps evidence remains required afterward. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, code editing, and tool use. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
7d46532c6d |
fix(runtime): bound parallel offline harness backup transfers (#19108)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server must retain provider history outside its
local disk.
> - Offline enrollment moves verified local harness backups to object
storage.
> - A cold rehearsal reached 699 current backup rows while processing
1,953 sources before its 25-minute deadline.
> - The serial transfer loop prevents this existing dataset from
enrolling.
> - This pull request overlaps at most four transfers within the
existing total scratch limit.
> - The benefit is faster preservation with the same ownership and
publication checks.
## Linked Issues or Issue Description
Refs #19038, #18943, #18932.
**What happened?**
Offline enrollment processed 1,953 local provider backups one at a time.
The normal Cloud image reached 699 current backup rows (including nine
pre-existing rows) before maintenance expired. The serving instance
remained unchanged during the isolated test.
**Expected behavior**
Enrollment must preserve every source within bounded time and disk use.
It must stop all in-flight transfers before it reports failure.
**Steps to reproduce**
1. Prepare a settled instance with multiple verified native harness
backups.
2. Run offline enrollment against shared object storage.
3. Observe that the serial owner waits for each archive, upload,
readback and publication before starting the next source.
## What Changed
- Process groups of up to four verified offline harness backups.
- Reserve space for the archive and encrypted copy before each group
starts. Keep the total scratch limit equal to one maximum-size serial
transfer.
- Derive a smaller archive limit from a closed manifest directory set.
Preserve unlisted files with an exclusive maximum-size reservation.
- Enforce the reservation while streaming. Keep the existing maximum for
ordinary runtime backups.
- Cancel and await peer transfers after failure. Keep completed
publications valid for retry.
- Preserve the original archive-stream failure when the tar writer also
fails after destruction.
- Document the migration bound and add integration regressions for
overlap, restoration, cancellation, retry, extra files and archive
limits.
## Verification
- Serial baseline: the new overlap regression fails with one active
transfer when four are required.
- Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem
archives and restored provider bytes. The bundled-entry test is skipped
locally because this source worktree has no runtime bundle. The full
normal image rehearsal remains required.
- Follow-up: 48 archive/storage checks pass after correcting the
PAX-header allowance. The real encoder produces a 972,801,024-byte
full-entry projection, above the old 805,306,368-byte allowance and
below the new 1,024,001,024-byte bound. The existing global object limit
remains in force. The old formula fails the new regression. Exact-size
restoration and one-byte-over-limit rejection pass.
- The previous revision passed full CI after one setup-route timeout
retry. That failure remains recorded; the corrected current revision
|
||
|
|
fb96537e28 |
test(ui): verify disconnect recovery through browser outages (#18832)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - Deploys can interrupt reads and writes while people use the board. > - Component tests cannot prove the browser keeps task content and saved drafts through a reload. > - The read and cache changes need a regression suite against the built app. > - This pull request drives those journeys through temporary API failures. > - CI discovers the new suite through the existing e2e shard plan. ## Linked Issues or Issue Description **What happened?** Outage recovery lacked one browser suite covering retained content, write recovery, cached reloads, access errors and sign-out clearing together. **Expected behavior** Keep loaded task content visible through a 20-second API outage. Show one connection banner and its recovery state. Save comments once and retain document edits. Restore approved reads after a reload. Keep real error responses readable and clear query records on sign-out. **Steps to reproduce** Open a task or settings. Return 503 with `tenant_app_unavailable` from browser API requests for 20 seconds. Type a comment, edit a document, or reload. Restore the API and compare the visible result with the saved server state. Repeat with real 403/404 statuses and sign-out. Stacked on Refs #18825, which depends on Refs #18822. Related durable-write foundation: Refs #17960. ## What Changed - Add `disconnect-resilience.spec.ts` with ten browser cases for comments, a queued comment across reload, document autosave, cached settings/sidebar/task reads, legacy Inbox reloads, 403 and 404 responses, sign-out clearing, restored-draft conflict detection, and first-socket outage control. - Assert one connection banner, readable recovery, no transient error boxes or raw codes, retained task text, one saved comment, and the saved document after reload. - Use the real local server and isolated embedded PostgreSQL. Intercept browser API traffic for outages. Control the logout/session response for the sign-out test. Create no agents or model runs. - Register realtime interception before navigation. Track both ends of current sockets, close them at outage start, and keep reconnects unavailable through the HTTP outage deadline. Preserve normal close forwarding and close tracked pairs during teardown. The regression captures the first browser socket without waiting for history or open, then checks that the outage closes it. - Prepare complete server-derived experimental settings and workspace preferences before the legacy Inbox navigation. Its route callbacks serve per-test snapshots across reloads, avoiding reads from disposed route.fetch responses. - Document the direct test command. The existing CI e2e shards already discover this file exactly once; no workflow or package script is added. ## Verification - Current head `228c1b68d30` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both required gates pass. The single failed-jobs debug rerun in run `41752978283` passed on attempt 2, including Runner package job `126328280650` and `verify` job `126332119116`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. The three-PR stack is awaiting operator merge approval. - Original attempt 1 of current-head CI run `41752978283` had 46 successful jobs, one skipped and two failed: Runner package tests and the downstream `verify` gate. GitHub reports lost communication with the Runner host; the test step was cancelled, its log returns HTTP 404, and no Runner diagnostic artifact is available. The underlying test outcome and disconnection cause are unknown. One failed-jobs retry with debug logging was accepted after verifying the unchanged current head and passed on attempt 2. No source change was made; earlier passing CI and local tests do not certify this incomplete Runner result. Current-head CI now passes; the earlier disconnection cause remains unknown. - Preceding head `6531caccf62` reached merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status was successful, and both required gates passed in run `41749512329`. Greptile reviewed that exact head at 5/5 with zero unresolved threads. Current head `228c1b68d30` carries later account-preference and bulk-selection fixes and now has its own passing current-head CI. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact current head `228c1b68d30`: ten outage, six durable-write and seven loading cases. The run used bundled Chromium, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned temporary server and database stopped. - Current integrated head `228c1b68d30` contains cache revision `fe4f682c879` and read head `09d1db29dea`. The exact application source passes all 1,198 focused tests across twenty-eight files, UI typecheck and both guards. Seventeen new cases cover restored-account settings, owned crown cleanup and pending bulk selections; sixteen fail the previous source and one preserves denial compatibility. All three browser specs pass standalone strict checking. All 23 current-head browser cases pass in 4.5 minutes. - Application revision `9c1a10f25d9` passed all 1,137 focused tests across twenty-one files, UI typecheck and both guards. Exact preceding application/spec head `4fec693ec94` passed all 23 built-UI outage/write/loading cases in 4.6 minutes and standalone strict checking of all three specs. It used bundled Chromium, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. Owned test processes stopped. - Preceding integrated head `870aa709657`, containing cache `cbb891dd37d`, passes all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. It used bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. Owned test processes stopped. - The earlier head `763fe431b74` passes the affected legacy Inbox outage/reload case (34.3 seconds; 50.8 seconds including startup) and strict three-spec checking after the response-snapshot fix. The preceding head `973a12d77f4` passed all 23 built-UI outage/write/loading cases in 4.6 minutes: ten outage cases, six durable-write cases and seven loading cases. Its first-socket regression fails the previous helper because that socket stays open. The run used bundled Chromium, the built UI, fresh isolated PostgreSQL and local-trusted auth on macOS arm64/Node 25.6.1. That earlier isolated callback revision was verified with its affected case; the later application/spec head `4fec693ec94` also passes the full suite. Owned test servers and databases stopped. - The current base gives known skills/app errors priority over sibling loading. All 60 affected tests and both application UI typechecks passed. Its twelve new regressions failed the preceding source. All 47 recovery-worker tests and seven related Runner fixture variants also passed during the historical CI audit; those focused checks do not claim a full current server/Runner suite. - All 22 outage/write/loading cases passed in 4.7 minutes on preceding integrated head `b7deaaee5da`, including direct cache revision `97c0bc3966f`. The run used the built UI, bundled Chromium and a fresh local-trusted server with isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. It covered all nine outage, six durable-write and seven loading cases. The owned temporary server and database stopped. - All 1,038 focused tests passed on that direct cache revision, with UI typecheck and guards. Six regressions failed the previous source: online retries could trigger a hidden-tab worker reload, and sign-out failed without crypto.randomUUID. Both modes and stamp states now pass. Standalone strict checking passes for all three browser specs. - The restored cached-A/draft-B/server-C test asserts zero replay PATCHes and retains the conflict. The queued-comment case holds fresh session verification after general API recovery and asserts zero POSTs until release. Denial, logout and cached-navigation cases require saved records. Sign-out seeds several company records to exercise native Storage key reordering. Healthy navigation waits for conversation-history readiness. - Earlier runs exposed real cached-header, storage cleanup and replay defects in the base. Those failures and corrected fixture assumptions remain in the discussion. A prior run passed 21/22 while real sockets remained available during simulated HTTP failures. A diagnostic trace established the contradiction. The first repair gated new sockets, but review identified the first pre-existing socket still escaping. This revision intercepts before navigation and closes existing pairs too. The new regression fails the preceding helper, and all 23 current cases pass without weaker assertions. Shared-memory exhaustion prevented two earlier diagnostic startups; only unused segments from completed owned test databases were removed. - All six separate cross-organization link/upload variants passed on an earlier integrated base. Standalone strict checking of that unrelated spec reports five existing errors in its shared fixture; this PR does not claim that check passes. - CI on older head `53d979e384c` failed the legacy Inbox case with `apiResponse.json: Response has been disposed` inside a route.fetch callback. Its other 25 cases passed and one skipped. The seven other browser shards succeeded; the aggregate only reports that child failure. The new snapshot callbacks remove those fetched-response lifetimes. Original CI evidence remains recorded in the discussion. - Greptile Apex reviewed exact current head `228c1b68d30` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:26:14 UTC). The branch has no merge conflicts. Current-head CI now passes; the earlier disconnection cause remains unknown. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Commands run: ```sh pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts --grep 'a legacy Inbox reload retains' --reporter=list pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts --reporter=list pnpm exec tsc --ignoreConfig --noEmit --strict --target ES2022 --module NodeNext --moduleResolution NodeNext --skipLibCheck tests/e2e/disconnect-resilience.spec.ts tests/e2e/disconnect-write-recovery.spec.ts tests/e2e/task-loading-frame.spec.ts ``` ## Risks The suite adds about three minutes to one e2e shard and depends on current accessible UI controls. It uses controlled outages and logout responses. A passing local run must not be treated as real deployment acceptance. There are no production behavior, schema, dependency, or API changes in this PR. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for related PRs and linked them above - [x] I have described the issue following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] All 23 built-UI cases pass on current head `228c1b68d30` - [x] All three browser specs pass standalone strict typechecking on the current head - [x] The current application source passes all 1,198 focused tests, UI typecheck and both guards - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
ea6bd15e2a |
fix(native): record why finalization failed a run (#19063)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run finishes in two steps: the provider turn, then
finalization. Finalization saves the workspace and commits the result.
> - When finalization exhausts its retries, `recordRetryableFailure`
sets the run to `failed` and blocks the task.
> - That code writes the cause only to `resultJson.failureCode`. It does
not write the run's `errorCode` or `error`.
> - So the run list, inbox, and agent status show a failed run with no
reason. Operators cannot tell why the run failed.
> - This pull request writes the finalizer's terminal failure code and a
readable message to the run row when the finalizer fails the run.
> - The benefit is that every failed run now names its cause.
## Linked Issues or Issue Description
**What happened?**
A hosted instance had 11 assigned native runs with status `failed`,
`errorCode` null, and `error` null in one day. In each run, the provider
turn succeeded. The remote workspace copy-back then failed three times.
The finalizer set the run to `failed` and blocked the task. The run kept
no error code. `resultJson.failureCode` was
`native_workspace_sync_out_retry_exhausted`, but no screen shows that
field as the failure reason.
**Expected behavior**
A failed run has an `errorCode` and an `error`. The failure surfaces
show why the run failed.
**Steps to reproduce**
1. Create a native run with a finalization coordinator.
2. Call `recordNativeFinalizationFailure` three times with
`native_workspace_sync_out_failed` and `failureScope: "workspace"`.
3. Read the run. Its status is `failed`, and its `errorCode` and `error`
are null.
**Paperclip version or commit**
`master` at
|
||
|
Dotta
|
155dd1bc1c | fix(evals): reject redaction-required plain attachments | ||
|
Nicky LeachandPaperclip
|
a0c0851766 |
Runner: add private host control channel (#19129)
## Thinking Path > - Paperclip coordinates AI agents and their execution. > - A hosted Runner can remain alive between turns. > - Its host needs to park, claim, or retire that process without a persistent Core connection. > - Those requests need a private channel with bounded input and safe retries. > - This PR adds the channel codec and socket handling as an isolated foundation. > - Later lifecycle changes will validate and execute the requests. ## Linked Issues or Issue Description **Current behavior** The Runner has no task-private host control channel. The earlier combined lifecycle change was split for review. **Proposed behavior** Add a private Unix stream protocol for host lifecycle requests. This PR does not wire it into Runner startup or advertise support. It can be reviewed independently of stopped-process recovery. **Reason and benefit** Review frame parsing, credential handling, and retry behavior before adding park and claim transitions. This replaces the transport portion of #19060. The source-recovery work supersedes #19057 and #19058. ## What Changed - Add `host_control.rs` and enable the existing rustix dependency's socket APIs. - Accept only a connected Unix stream. Set close-on-exec so provider children do not inherit it. - Bound request frames to 16 KiB. Reject unknown fields, malformed requests, and EOF with a credential-free error. - Add socket-pair tests for framing, deadlines, retries, conflicts, and receipt-window reset in a separate test file. ### Transport contract Each request is one newline-delimited JSON object. Each response is `{ "id": "request-id", "status": "..." }` followed by a newline. | Request | Required binding | Intended effect in the later lifecycle consumer | | --- | --- | --- | | `park` | Six Runner identity fields, last controller command sequence, acknowledged source sequence | Retain the process and mounts after the controller has settled that exact cut | | `claim` | Park request ID and a fresh bootstrap ticket | Reconnect the retained process through authenticated bootstrap | | `retire` | Park request ID; optional claim ID and identity/sequence cut | Retire only the matching parked or claimed process | The six identity fields are Runner instance, environment lease, run, normalized session, turn, and item IDs. This module only parses requests and sends responses. The lifecycle consumer must validate these bindings before performing an effect. Before applying a request, the consumer checks the in-memory response cache. An identical retry receives its prior status. Reusing a completed request ID with different content returns `conflict`. Conflict responses do not reserve an ID. A successful `parked` response starts a new cache window; the consumer's identity, sequence, and park-ID checks must reject requests from older windows. Credentials are neither persisted nor included in errors. Active reads are nonblocking. Parked reads block on socket input and may have a caller-supplied receive deadline. Partial frames do not extend that deadline. No idle polling is added. ## Verification - `cargo test --offline --locked --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --lib host_control::` — 6 passed. - Confirmed the repeated-chunk test fails when reads incorrectly reset the original deadline. Restored production code and reran the targeted suite. - The initial policy gate did not recognize inline Rust tests. Moved the same tests to the existing `src/tests/` convention. - GitHub CI passed on `aaac996aba76`. Greptile is 5/5 on that commit, with no unresolved review threads. - A local merge-tree check against current master `1a7eb70a37bf` is conflict-free. - Reviewed changed lines for secrets, personal data, and private deployment references before push. ## Risks The module is dormant in this PR. It does not make a VM sleep, detach Core, or evict a Runner. Integration must enforce the documented identity and sequence fences before replying with success. The response cache is process-local and does not provide recovery across process death. Linux and Windows compilation are covered by CI; local socket tests run on macOS. ## Model Used OpenAI Codex with tool use and code execution. GPT 6.1 Sol with High effort handles the parallel recovery work; this isolated channel change is implemented by the coordinating Codex agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1105.0-canary.0 |
||
|
DottaandPaperclip
|
818d2bb661 |
fix(native): preserve closed post-result failure diagnostics (#19126)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution persists a result before it completes cleanup and workspace export. > - A later execution error enters the failure-write path. > - That write must reject an already selected result. The existing code reports lease loss for every rejected write. > - This can hide the category of the first error even when the same controller still owns an unexpired lease. > - This change adds closed diagnostic labels after exact ownership and accepted-result checks. It keeps the existing control error and recovery policy. ## Linked Issues or Issue Description **What happened?** A local real-PostgreSQL regression accepts a result through the production control plane, then injects a harmless filesystem failure. The production executor reports `native_session_lease_lost`. It loses the original failure category because the failure transition requires a null result ID. **Expected behavior** Keep the result guard and the existing lifecycle classification. Record a closed diagnostic category when the selected result is accepted and the controller still has exact, unexpired authority. Keep ordinary lease-loss fencing when that proof is absent or its diagnostic lock is busy. **Steps to reproduce** Run the new post-result cases in `server/src/services/heartbeat/runtime-execution.test.ts`. They use the real executor, control-plane result commit, heartbeat recovery path and PostgreSQL. The provider boundary is controlled. No live provider is used. **Paperclip version or commit** The regression fails on base `7d1e9b3477`. Related open PR #16917 changes legacy lease recovery. Open #19063 adds terminal finalizer reasons, and #15310 preserves database finalization errors. They do not change this executor diagnostic branch. This change only adds native diagnostic evidence and does not reclaim a lease. ## What Changed - Add a typed diagnostic with a closed stage, category and accepted-result flag. Do not retain the source message, path, arbitrary code or lease token. - After a rejected failure write, check the selected accepted result and the exact company, task, run, owner, attempt, generation, process identity and expiry. Use a coordinator-only `NOWAIT` lock. Fall back to the original control error if the diagnostic read fails. - Report the closed event after rollback. Keep `native_session_lease_lost` as the error message and leave every failure-write predicate unchanged. - Add real-PostgreSQL safety cases for stale and expired authority, owner replacement, Stop state, result selection, pending accounting, one provider invocation, closed labels and diagnostic sink failure. - Document the closed event, unchanged authority, and possible late diagnostic sink completion. - Add a competing-row-lock barrier. It verifies actual PostgreSQL lock refusal and control settlement before the competing lock is released. ## Verification - Permanent regression on unchanged base: RED because the closed diagnostic is absent. - NOWAIT mutation: RED when only NOWAIT is removed. PostgreSQL reports lock timeout instead of immediate lock refusal. The test uses barriers and the actual database error, with no elapsed-time assertion. - Final related suites: 56 tests passed across runtime execution and provider failure integration tests. - Full server source and test typecheck: passed. - Full `pnpm -r typecheck`: passed in an independent locked-dependency checkout. - Full `pnpm build`: passed. - Full `pnpm test:run`: running in the independent immutable checkout. All six chat shards passed; the remaining suites are still pending. - The first current-head CI attempt failed in two cases: workspace runtime initial fixture HTTP readiness and instructions-bundle error-message rendering. Both exact cases passed in isolation on unchanged PR head and base. The original causes remain unknown after bounded test-only observation. One controlled same-head rerun of each failed job passed. Current required CI and all other completed checks passed. Original failures remain recorded; no infrastructure or flaky-test claim is made. - Module boundary check and `git diff --check`: passed. - Exact-head Greptile: 5/5 with no actionable findings or open review threads. Automatic native review remains queued. That advisory check is not required by current master rules; no current-head native assessment is claimed. - Earlier fixture setup failures remain recorded. These included missing fixture fields and observing the Drizzle wrapper instead of its PostgreSQL cause. They are not the regression RED. An earlier broad typecheck was interrupted for the required NOWAIT revision; the final full typecheck passed. - No hosted export recovery or original hosted failure cause is claimed. ## Risks The extra read is diagnostic only. It uses NOWAIT and cannot wait for a competing coordinator lock. Its failure preserves the existing control classification. The event sink uses the existing 60-second execution-control bound. A slow sink can finish later and emit a late event. Result selection, Stop, accounting, retry eligibility and provider work do not depend on event success. No schema or lifecycle contract changes are required. ## Model Used OpenAI GPT-6.1, model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
0e27369fa4 |
fix(native): restart sandbox prep interrupted before its sync descriptor (#19054)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on sandbox environments sync the host workspace into the
sandbox before the provider starts
> - Runtime selection writes the native execution input first. It writes
the workspace sync reference only after the sandbox is seeded, and
before any provider dispatch
> - A server restart between those two writes leaves a run with an input
but no sync reference
> - Restart recovery then fails the run during workspace preparation
with `native_workspace_sync_descriptor_binding_missing`, although the
sandbox never received any work from that run
> - This pull request lets recovery re-admit the host workspace in that
state, so sync preparation starts fresh on the same run
> - The benefit is that a deploy or restart during sandbox preparation
no longer turns queued work into a terminal `setup_failed` run
## Linked Issues or Issue Description
**What happened?**
After a server restart, six native sandbox runs failed with
`setup_failed` and the error
`native_workspace_sync_descriptor_binding_missing`. Each run timeline
shows the same order: run started, `snapshot.git`, `snapshot.baseline`,
then "Server shutdown suspended native controller ownership", then a
restart recovery claim ("Restarting an incomplete native runner
bootstrap" or "Resuming the durable native provider session"), then the
error. No run had a `nativeWorkspaceSync` reference. Each run had a
persisted `nativeExecutionInput`.
**Expected behavior**
When sandbox sync preparation did not finish, restart recovery prepares
the sandbox workspace again from the host workspace and continues the
run.
**Steps to reproduce**
1. Start a native run on a sandbox environment.
2. Restart the server after runtime selection persists
`nativeExecutionInput` but before `prepareNativeWorkspaceSync` persists
its reference.
3. Restart recovery re-enters workspace preparation and fails with
`native_workspace_sync_descriptor_binding_missing`.
**Paperclip version or commit**
Reproduced on `master` at
|
||
|
DottaandClaude Opus 5.5
|
ac57b183f0 |
fix(evals): run campaign cells when TestOps selection is skipped (#19053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner Full-Stack E2E workflow runs the paid Product E2E cells, including the Product regression TestOps plan > - #18983 added an optional `testops_selection` job and made `catalog` depend on it > - That job is skipped in every ordinary manual or scheduled campaign > - GitHub's implicit `success()` also fails when an ancestor job was skipped, so every job below `catalog` was skipped > - Since #18983 merged, ordinary campaigns run no cells: the merge reports every selected cell as "No result artifact was uploaded" > - This pull request gives each job below `catalog` an explicit status condition > - The benefit is that ordinary, nightly and branch-targeted campaigns run their cells again, and TestOps campaigns are unchanged ## Linked Issues or Issue Description Refs #18983, #19000 **What happened?** Run https://github.com/paperclipai/paperclip/actions/runs/41914430034 selected 20 Product regression cells with the `id` input. `catalog` passed. `daytona_image`, `build_runner_artifacts`, `build_remote_provider_pack` and every matrix job were skipped. The merge failed with 20 "No result artifact was uploaded" rows. **Expected behavior** An ordinary manual campaign runs its selected cells when `testops_selection` is skipped, as it did before #18983. **Steps to reproduce** Dispatch `runner-full-stack-e2e.yml` on `master` without `testops_case` or `ALLURE_JOB_RUN_ID`. The scheduled daily campaign has the same path. ## What Changed - `daytona_image`, `build_runner_artifacts`, `build_remote_provider_pack`, `test`, `test_recurring` and `test_manual_private` now start with `!cancelled()`. - Each of those jobs now requires `needs.<job>.result == 'success'` for each of its real dependencies. A failed or skipped build still skips the cells, as before. The existing schedule, cadence and evidence-route conditions are unchanged. - A new regression test parses the workflow. It requires every job below `testops_selection` to use a status function and to require each real dependency result. ## Verification - `node --test tests/runner-e2e/product-regression-testops.test.mjs`: 15/15 pass. - Negative control: the new test fails against the current `master` workflow. - `daytona-image.test.ts`, `workflow-target.test.ts` and `workflow-timeout.test.ts` pass. `workflow-security.test.ts` has 5 failures. The same 5 fail on unchanged `master` (they concern `pr-trusted.yml` and `chat-catalog-testops-case.yml`). - A workflow change takes effect only from `master`, because paid campaigns run only from the default branch. After merge, dispatch an ordinary `id=` campaign and confirm the cells run. ## Risks - Low risk. The conditions keep the previous failure propagation: a job runs only when each direct dependency succeeded and the run is not cancelled. - `catalog` success already implies `authorize` and `target_lock` success, so those are not repeated. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
23cd123ff8 |
fix(e2e): retain the closed cause code of a journey cleanup failure (#19079)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Product E2E journey harness runs the QA staging smoke plan and
reports each case to TestOps
> - Each case runs a journey, then releases its owned target resources
(cleanup)
> - Three recent cleanup failures had a passed journey and all checks
passed, but each result only says `target_cleanup_failed`
> - The adapter throws an error with a closed identifier code, and the
harness discards that code, so no one can find the failing cleanup step
> - This pull request keeps the identifier-shaped cause code as an extra
cleanup phase
> - The benefit is that the next cleanup failure is diagnosable from
TestOps without private runner files
## Linked Issues or Issue Description
**What happened?**
QA staging smoke cleanup failures show only `target_cleanup_failed`.
This occurred for `question-answer-resume` (launch 2390), and for
`company-access-isolation` and `create-skill-studio` (launch 2693). In
each case, the journey and all behavioral checks passed. The private
runner report is deleted with the runner, and the adapter error code is
not kept in the retained result. The cause is not known.
**Expected behavior**
A cleanup failure keeps a closed, identifier-shaped cause code (for
example `fixture_cleanup_not_confirmed`). The result never keeps the raw
error message.
**Steps to reproduce**
1. Run a Product E2E journey with a target session whose `release()`
throws an error that has a `code` property.
2. Read `normalized-results.json`.
3. Before this change, the cleanup phase only contains
`target_cleanup_failed`.
**Paperclip version or commit**
master at
|
||
|
DottaandPaperclip
|
552140495c |
fix(evals): retain bound skill contradiction chains
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ba8418fc4d |
fix(evals): bind adjacent team skill recovery pronouns
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
b1a71044af |
fix(evals): observe chat handoff replies and restarted task pages (#19073)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Product regression TestOps plan runs 20 Product E2E cells
through `runner-full-stack-e2e.yml`
> - The full 20-case TestOps launch on `master`
|
||
|
|
2ccc7d37ef |
feat(ui): retain approved reads across outage reloads (#18825)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board already retains loaded reads during a temporary server failure. > - Reloading the page removes that in-memory data. > - A deploy can then leave a returning user without the shell or recent tasks. > - This pull request saves an explicit, limited set of reads in the browser. > - Cached identity supports display while fresh session checks protect writes. ## Linked Issues or Issue Description **What happened?** A reload during an API outage loses the board's loaded reads. An automatic worker update can also reload a hidden tab while a write is pending. **Expected behavior** Show the cached shell and recent tasks during a temporary outage. Limit saved content by user, company, age and build. Clear it on sign-out. Wait for a fresh session before sending account-bound writes. Defer automatic reloads while work is pending. **Steps to reproduce** Open a task and settings while online. Return 503 from the API and reload. Restore the API. Sign out and confirm that saved query records are gone. Discover a worker update while offline or during an autosave and confirm that it waits. Stacked on Refs #18822. Related durable-write foundation: Refs #17960. The outage browser suite follows in a separate test PR. ## What Changed - Persist an exact read allowlist through TanStack Query dehydration and hydration. Save shell preferences, company/sidebar data, agent/project lists, plugin UI metadata, the first inbox page, and 20 recent tasks. Task UUID and identifier reads share one recent-task slot. - Namespace records by user and company. Retain eligible reads across normal memory garbage collection, expire each read after 24 hours, and discard them after a build change. Unstamped builds disable hydration and persistence while retaining sign-out cleanup. Use the existing instance-wide plugin contribution query in the user's root record. - Exclude session credentials and bootstrap data, embedded plan documents, comments, preview work products, provider credentials and environment values, browser capabilities, pairing data, other queries, and mutations. Restore only display identity from session data. Preserve safe agent provider/model display fields and configuration object shapes. Apply the projection on save and restore. - Clear saved reads on sign-out, cross-tab account changes, and definitive session denial. Keep revocation working on plain HTTP without crypto.randomUUID. Use per-user revocation revisions to reject delayed writes and stale records. Preserve a newer account. Remove every UUID/identifier projection and nested inbox copy of a denied task. - Detach shared GETs when the account changes or signs out. Load complete, freshly authorized agents before opening bulk settings from a display-only roster. Retire a pending bulk-editor result or error when row, select-all or clear changes the selection, including a clear/reselect of the same agent. - Require a fresh server lookup before account-bound writes, pending task creation, or mounted autosaves resume. This includes automatic task read receipts, inbox archives, workspace preferences and primary-agent choices, with optimistic archive changes gated too. Cancel the old write scope when the verified user changes. Release completed archives through account-independent cleanup without restoring old cached rows; an old operation cannot clear a newer hide flag. Retry unconfirmed read receipts after a fresh session succeeds for the original account, and remember confirmed receipts across refreshes. Track pending reads per task so opening another task sends its receipt immediately; revisiting reuses an outstanding receipt and its original account scope. - Defer a hidden-tab worker reload until connectivity, mutations, durable sends, and autosaves are safe. Synchronously count each unresolved operation across online rate-limit retries. Release a cancelled target’s unused dirty token when an editor changes tasks; keep stored drafts, admitted saves and queued retries. Cover autosave debounce, newer edits, navigation and undo. ## Verification - Current head `fe4f682c879` is merge-ready: all 54 check runs completed (52 successful, one skipped, one neutral), the legacy Snyk status is successful, and both `ci / verify` and `ci / e2e` pass in run `41752927822`. Greptile Apex reviewed this exact head at 5/5 with zero unresolved threads; the branch has no merge conflicts. All three PRs now meet the merge-ready bar; the stack remains unmerged pending operator approval. - Greptile Apex reviewed exact current head `fe4f682c879` at 5/5 with no actionable findings and zero unresolved threads (summary updated 2026-11-04 at 01:24:40 UTC). Both settings-account and pending-selection findings are fixed, replied to and resolved. The branch has no merge conflicts. Current-head CI now passes in run `41752927822`. - Seventeen new cases cover settings restored from actual saved query records, matching/changed/signed-out identities, denied verification, late crown responses without overwriting newer reads, and bulk-selection changes during successful/failed reads. Sixteen fail the preceding source and one preserves denial compatibility. Both settings paths reuse the existing account-bound mutation guard; its optional explicit QueryClient supports the workspace hook’s existing detached-host mode. Primary-agent optimistic updates begin only after verification and cleanup checks ownership and the cache revision. - Earlier CI run `41749456480` on prior head `2cf7516f18c` has one real server test failure: the owned-runtime case in `idle-completed-history.test.ts` could not reach its HTTP child on port 34757 within 10 seconds (99 refused probes). Its other 1,829 cases pass and three skip; the only other unsuccessful job is the downstream `verify` gate. The exact selected case passes locally, and the entire 75-case file passes in 72.19 seconds on the exact `2cf7516f18c` source tree with isolated PostgreSQL and a real child process on macOS arm64/Node 25.6.1. The Linux CI startup cause remains unknown; these local results are not a fix or CI acceptance. One failed-jobs retry with debug logging was accepted on that unchanged head, then superseded by the later settings fixes. Current-head CI remains required; no source change was made for this failure. - Current cache head `fe4f682c879`, based on read head `09d1db29dea` and master `683fa4725e9`, passes all 1,198 focused tests across twenty-eight files, UI typecheck, token gates and the enforcing query-error guard. Dependent browser head `228c1b68d30` contains this exact application source and passes standalone strict checking of all three browser specs. Current-head CI now passes in run `41752927822`. - Three new navigation regressions fail the preceding application source and pass with the fix. They open three cached tasks before any receipt resolves, leave the page before success or failure, revisit an outstanding read without duplicating it, and recover an older failed receipt after a fresh session succeeds while a later task remains pending. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Nine new cases cover archive success/failure after sign-out and another account takes over, both mounted and unmounted; cleanup ownership when a newer archive uses the same task; session-check 429 exhaustion followed by matching, changed or signed-out identity; and a failed read receipt followed by a new session read. They verify no old-cache restoration, toast or navigation, retained newer hide flags, no failed-check loop and no duplicate confirmed receipt. Seven fail the previous implementation; two compatibility cases retain account-change suppression. Initial test error-constructor and query-notification timing assumptions were corrected before the passing checks. - The inherited skills/app error-priority fix passes all 60 affected tests and UI typecheck on both application branches. Its twelve new regression cases fail the preceding read implementation. The persisted-cache suite includes 48 cases. - Six new regressions fail against the preceding application source and pass with the fix. Actual worker-update and mutation-cache listeners cannot reload a hidden tab between online 429 retries; the original create key survives, and the update reloads once after a receipt. Real self-hosted and Cloud sign-out work without crypto.randomUUID, with stamped and unstamped builds. Saved reads are removed and account write scopes are invalidated. An initial test-fixture type annotation error was corrected before the passing typecheck. - The 48 cache tests cover actual five-minute memory GC, the 20-task bound across saved and live reads, original 24-hour expiry, denial before session verification, unstamped cleanup, cross-tab revocation and delayed flushes. Private GETs cannot cross account transitions. Bulk edits require complete current agent data. Undo releases the update blocker while preserving the Saved indicator. - All 23 built-UI outage/write/loading cases pass in 4.5 minutes on exact integrated head `228c1b68d30`, containing current cache revision `fe4f682c879`: ten outage, six durable-write and seven loading cases. The run uses bundled Chromium, the built UI, a fresh local-trusted server and isolated embedded PostgreSQL on macOS arm64/Node 25.6.1. All three specs pass standalone strict TypeScript checking. Owned temporary server and database stopped. - Earlier browser runs exposed and fixed cached-header, native Storage cleanup and replay/session defects. A prior run passed 21/22 because real WebSockets contradicted the simulated HTTP outage; the fixture now closes existing socket pairs and holds new sockets through the same outage deadline. An older CI run exposed a disposed response in the Inbox route callback; pre-navigation snapshots remove that lifetime, and the current full run includes that repair. Earlier failures and their diagnosis remain in the discussion. The current full run passes. Broader historical workspace failures remain documented in Refs #18822; this does not claim a clean full workspace suite. - Current-head CI passes on `fe4f682c879`. Controlled local outages do not complete hosted authentication or a real Cloud restart/deploy acceptance. Commands for the current cache change and inherited read fix (the current focused run also includes src/lib/inboxArchiveCache.test.ts and src/pages/Inbox.test.ts): ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/hooks/useDurableSubmit.test.tsx src/hooks/useDurableAutosave.test.tsx src/hooks/useSignOut.test.tsx src/lib/persisted-query-cache.test.ts src/lib/service-worker-updates.test.ts src/lib/service-worker-cache-policy.test.ts src/lib/pending-send.test.ts src/lib/issue-execution-policy.test.ts src/components/InlineEditor.test.tsx src/components/MarkdownEditor.test.tsx src/components/NewIssueDialog.test.tsx pnpm --filter @paperclipai/ui exec vitest run src/components/agents/AgentBulkRoster.test.tsx src/components/agents/AgentBulkSettingsSheet.test.tsx src/hooks/useAccountBoundMutation.test.tsx src/pages/IssueDetail.test.tsx src/components/TaskChatThread.test.tsx src/components/TaskChatThread.task-creation-order.test.tsx src/api/client.test.ts src/api/agents.test.ts pnpm --filter @paperclipai/ui exec vitest run src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/context/PrimaryAgentProvider.test.tsx src/components/CoordinatorWorkspaceSetting.test.tsx src/components/primary-agent/PrimaryAgentPresentation.test.tsx src/components/MobileBottomNav.coordinator.test.tsx src/components/CoordinatorSidebar.test.tsx src/pages/Issues.coordinator-routing.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This deliberately saves task content on the local browser for up to 24 hours. The allowlist excludes credentials and ephemeral capabilities. Local storage can be disabled or full; persistence then fails without blocking the app. Restored data can be stale and never grants server access. A user or access change clears cached account reads and invalidates pending writes. The browser outage suite and real hosted deploy acceptance remain release requirements. There are no migrations or new dependencies. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The focused local tests for this change pass - [x] I have added tests and updated relevant documentation - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
|
35d086213e |
fix(native): explain failed accepted results on the run (#19091)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run ends when the server projects the accepted result onto the run row. > - Operators read `errorCode` and `error` on a failed run to learn why it failed. > - Three native writers can set a run to `failed` without these fields. #19063 repairs the finalizer retry-exhaustion writer. > - The two result-projection writers still set `failed` with no cause when the provider failure is not classified. > - This pull request makes both projection writers record a cause when they project `failed`. > - The benefit is that, with #19063, no native writer can leave a failed run unexplained. ## Linked Issues or Issue Description Refs #19063. **What happened?** A staging soak found 11 failed native runs with `errorCode` null and `error` null. In each run the provider turn succeeded. Workspace copy-back then failed three times, and the finalizer retry-exhaustion path set the run to `failed`. #19063 fixes that writer. An audit of every `heartbeat_runs` status writer found two more writers with the same defect. `finalizeNativeRun` and `projectCommittedRun` project an accepted result with `runTerminalState: "failed"`. They write the error fields only when a classified provider failure exists. **Expected behavior** A run with status `failed` always has an `errorCode` and an `error`. A cause that is already recorded stays. **Steps to reproduce** 1. Accept a native result with `runTerminalState: "failed"` and no `turn.failed` event. 2. Call `finalizeNativeRun` with `projectRunStatus: true`, or project the committed run through `projectCommittedRun`. 3. Read the run. On master its status is `failed`, and its `errorCode` and `error` are null. ## What Changed - Add `explainFailedRun` in `server/src/services/native-runtime/native-run-failure-explanation.ts`. It builds SQL SET expressions. They keep a non-empty existing `errorCode` or `error`. Otherwise they write `native_run_failed` and a readable message. By default the expressions leave a run that is already `cancelled` unchanged, as the status writers do. - `finalizeNativeRun` uses the helper when it projects `failed`. The existing provider-failure and recovered-failure values remain the preferred causes. - `projectCommittedRun` uses the helper for a `failed` accepted result. This writer sets status unconditionally, so it does not use the cancelled branch. - Writer audit: all legacy and heartbeat callers of `setRunStatus*` that pass `failed` already set both fields. The other direct native writers also set both fields. ## Verification - Red, with the source change stashed: 3 new cases failed with `errorCode: null`. These were the finalizer projection, recovery projection with no cause, and recovery projection with an empty error. The control case with a recorded cause passed. - Green: `npx vitest run src/__tests__/native-run-projection-work.test.ts src/services/native-runtime/native-run-finalizer-telemetry.test.ts src/__tests__/native-status-arbiter-corpus.test.ts` (225 passed). - Wider run: all `server/src/__tests__/native-*.test.ts` and `server/src/services/native-runtime/*.test.ts` (7416 passed). Two failures come from the local environment. `plugin-runner-probe` needs the Rust runner source. `native-ingress-bootstrap-retry` needs the controller-host OpenCode proxy. - This change was not exercised on staging. It changes only fields that are written at terminal projection. ## Risks - Low risk. The change affects only rows that are projected to `failed`. A recorded cause is never replaced. Succeeded and cancelled projections are unchanged. - The staging failures need #19063 too. This PR alone does not change their writer. - The cause behind the staging copy-back failures is still unknown. The sandbox command reports the redacted `remote_workspace_operation_failed`, which also appears in a separate signature. This PR does not change that diagnostic. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, browser reads of staging run records). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Saurabh Sungh
|
66b0aded28 |
fix(cli): add KillMode=mixed to generated systemd unit (#18598)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The CLI subsystem provides background service management (`paperclipai service install`) generating systemd user units on Linux. > - When systemctl stops or restarts the service, systemd uses default `KillMode=control-group` and sends SIGTERM simultaneously to all cgroup processes, including embedded PostgreSQL. > - This races the server's clean shutdown sequence and triggers a smart shutdown in PostgreSQL instead of a fast shutdown, leading to ungraceful exit. > - This pull request adds `KillMode=mixed` to the `[Service]` section of the generated systemd unit definition. > - The benefit is that SIGTERM is delivered only to the main server process, allowing the server to cleanly drain and shut down embedded PostgreSQL in order. ## Linked Issues or Issue Description Fixes #18595 ## What Changed - Added `KillMode=mixed` to `renderSystemdUnit` under `[Service]` in `cli/src/services/service-manager.ts`. - Added test assertion for `KillMode=mixed` in `cli/src/__tests__/service-manager.test.ts`. ## Verification - Automated test: `cli/src/__tests__/service-manager.test.ts` checks that `renderSystemdUnit` generates a unit containing `KillMode=mixed`. - Manual verification: Checked rendered systemd unit output to ensure valid systemd syntax and correct section placement under `[Service]`. ## Risks - Low risk: `KillMode=mixed` only applies to systemd user services on Linux generated via `paperclipai service install`. If the main process fails to exit within `TimeoutStopSec` (300s), systemd sends SIGKILL to all remaining child processes. ## Model Used - Google Antigravity (Gemini 3.8 Flash High) with chain-of-thought reasoning and codebase tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green |
||
|
|
1a7eb70a37 |
fix(run-dispatch): save a redirect sent while a stopped run settles (#19122)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can stop a running agent and then send a new message that
redirects the work
> - The stopped run keeps settling for some seconds after Stop (backup
and checkpoint persist)
> - A message sent in that window is queued as a normal run before
settlement records the workspace recovery hold
> - The dispatch gate then withholds the unstarted run, but it saved the
message only for sources with a native workspace-sync baseline
> - For other sources (remote workspace, snapshot-only) it marked the
wake `skipped`, so the message was lost and the task stayed in progress
with no run and no wake
> - This pull request saves the withheld message for any source when the
message was written after the Stop
> - The benefit is that a redirect after Stop is acted on exactly once
instead of disappearing
## Linked Issues or Issue Description
**What happened?**
After Stop, a user posted a redirect comment with resume intent. The
follow-up run was created while the stopped run was still persisting its
checkpoint. When settlement finished, the dispatch gate cancelled the
follow-up run with `execution_reconciliation_required` and an
empty-looking error ("Recovering saved work…"). The user's wake was
marked `skipped`. Nothing woke the task again. It stayed `in_progress`
with no active or queued run.
**Expected behavior**
The message sent after Stop runs exactly once after the stopped run
settles. The task never rests in progress with no run and no pending
wake.
**Steps to reproduce**
1. Start a native run whose stopped source has no native workspace-sync
baseline (for example a remote-workspace sandbox run).
2. Stop the run.
3. Post a comment with resume intent while the stopped run still
persists its checkpoint.
4. Let settlement record the workspace recovery hold.
5. Observe that the queued follow-up run is cancelled with
`execution_reconciliation_required` and its wake is `skipped`.
**Root cause**
`deferUnstartedUserComment` in
`server/src/modules/run-dispatch/adapters/postgres.ts` saved a withheld
user comment only when the stopped source had a native workspace-sync
baseline (the copyback replay case). The enqueue path saves or admits
the same message when it arrives after the hold exists. The gap was only
timing.
Refs #18332 (added the native-baseline-only deferral this PR widens).
**Paperclip version or commit**
Reproduced on `master` at
|