3407 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	33ca62f776	Keep external-agent feature reads on their admission transaction (#19750) ... ## Thinking Path > - Paperclip manages work for AI agents. > - External native agents must pass current feature and authority checks. > - The broker checks feature settings inside its admission transaction. > - The old feature reader uses the root database pool. > - Two active transactions can leave no connection for that read. > - This change uses the admitted transaction for the same policy read. > - It keeps the existing company, principal and controller checks. ## Linked Issues or Issue Description Refs: #19340. That merged change fixes retained native Retry ownership. This change leaves that recovery contract intact. Related: #18744 and #17833 fix caller-owned transaction reads in other domains. Root searched live GitHub for open externalBotEnabled changes and found no duplicate. **What happened?** An external-agent admission can wait for another pool connection while it holds a row lock. A second transaction can occupy the remaining connection and wait for that lock. The feature read must use the transaction that already owns admission. **Expected behavior** Dot, Grok and Muse read current feature settings on the admitted transaction. Disabled providers remain disabled. Current authority checks remain in force. **Steps to reproduce** 1. Use a real PostgreSQL pool with two connections. 2. Lock the settings row in the admission transaction. 3. Occupy the second connection with a transaction that waits for that row. 4. Read the existing provider feature policy through the admitted owner. 5. Require the read to finish before the competing lock is acquired. **Paperclip version or commit** The fix was tested at `d8dece250b75cffee67d3b7f4f0e669826282164`, based on master `839d41aabbc6d042250548aedcaa5c50d369dca3`. It now includes master `131b566ca4fa866c4348739c75cfe48c805b3398` through normal merge `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a`. All seven fix files retain their tested blobs and modes. ## What Changed - Pass the admitted select owner through the broker identity contract. - Use that owner for Dot, Grok and Muse feature settings. - Add a read-only settings projection that reuses the existing managed and staging view. It does not create the settings row. - Add six real PostgreSQL controls for enabled and disabled providers. - Add three broker-path PostgreSQL regressions for exact transaction forwarding through Dot, Grok and Muse. - Keep all newer master broker changes and existing authority checks. ## Verification Local controlled regression passed at tested revision `d8dece250b75cffee67d3b7f4f0e669826282164`: - Baseline: restore only the five original master production leaves. Keep the same read-only settings projection and regression file. One enabled-Dot case failed with the intended `57014`; it still read the correct `true` policy. Five cases were unselected. - The trace held the admission lock at 2.441 ms. The second connection waited for that lock at 9.494 ms. The root-pool feature read finished at 2015.585 ms, after the competing statement timed out at 2014.982 ms. - Candidate: all six enabled/disabled Dot, Grok and Muse controls passed, with zero skips, in 14.528 s. Each read returned the configured policy before the competing lock was acquired. Every competing transaction committed without an error. - Fresh normal candidate production declarations passed in 17.080 s. The complete 615-root owning test program passed in 4.756 s. Own dependency setup and TypeScript export builds had passed before these controls. - Known worker groups and sockets closed. The single post-owner audit found no other sessions in the test database. No forensic database was changed. An earlier selector attempt skipped all six cases and opened no test database connection. It is not a regression pass or failure. Source inspection proved that Vitest 5 filters names with ` > ` separators, while its JSON reporter joins names with spaces. The selected control then used the unique title-only pattern; its separate baseline failure is preserved above. The later normal master merge preserves all seven tested files. The 46 incoming paths change Native/workspace recovery, types and presentation. They do not change the direct feature/settings read graph. The direct-policy evidence therefore carries from its tested revision. It was **not rerun at the merged revision**. Fresh merged-head checks at `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a` passed: the content-aware Shared/SDK producer (1.325 s), normal Runner core and surfaces TypeScript programs (0.936 s and 0.693 s), and the full normal App graph (28.933 s). The App check included all production layers and all four test programs: 616 normal test roots and 397/409/412 tree roots. Known build owners and sockets closed. Database/Eval and provider asset outputs remained unchanged. No install, Cargo, binary build, provider execution or PG test repeated. Fresh final-head remote CI and review remain **pending**. The test entry is `server/src/modules/external-agents/feature-reader-owner.postgres.test.ts`. It requires a dedicated external `PAPERCLIP_FEATURE_OWNER_TEST_DATABASE_URL` ending in `_test` and an owned `PAPERCLIP_FEATURE_OWNER_TRACE` path. Its normal migration hook, two-connection pool, case deadlines and assertions remain unchanged. No provider or Runner binary is needed for these controls. The review also identified a wiring coverage gap: the six contention controls call the readers directly. A new whole-three file now traverses the actual Dot/Grok/Muse broker factories, normal pending-pairing producers, dispatch and real execution-admission transaction. Pass-through observations compare the exact admission transaction with the owner supplied to the normal readonly settings projection. Each provider is disabled through normal settings and must return real 403, read actual false policy, leave the queued run without Native authority, and create no assignment. No grant, ready binding, policy result or successful authorization is mocked. With only the original five master production leaves restored, all three new cases failed at the exact missing-owner assertion (`readOwners` expected length 1, actual 0) after passing the real 403 and admission-scope checks. The full baseline had0 pass / 3 fail / 0 skip in 12.691 s. With the exact candidate leaves restored, the unchanged whole-three file passed 3/3 with zero skips in 10.101 s. Both variants used actual Node 24 / Vitest 5 forks and a real PG pool of two; all known owners/groups/sockets closed. The candidate's private temporary directory was removed; the failed baseline database/home remain retained. Root independently read all three raw baseline failures, rehashed source and outputs, verified known owner/group closure, then observed no remaining test sessions. The successful database was normally removed; the original failing database and home remain retained. The new test entry is `server/src/modules/external-agents/broker-feature-owner.postgres.test.ts`, using dedicated `PAPERCLIP_BROKER_FEATURE_OWNER_TEST_DATABASE_URL`. The complete normal 617-root owning test program passed in 4.883 s after a normal exported config refresh; prior production declarations and unrelated configs remained byte-identical. No compiler or dependency/Rust work repeated for the PG variants. These results were obtained with the seven production/source files at c203 plus the new test; they are not claimed as a new published-head remote CI result. The review repair is committed at `d043860eb93a26c4416f184697a212f027d538e2`; it adds only this 117-line test file to `c2031b0f`. Tracked production bytes are unchanged. Exact published head `d043860eb93a26c4416f184697a212f027d538e2` now passes 56 checks with four expected skips. Greptile reviewed this exact head at 5/5 and confirms that the broker-test gap is addressed. The complete review thread is resolved. Ponytail stopped without a verdict; one normal retry was requested and no active turn or new finding was observed. No earlier-head result is used as a new-head pass. The controls prove feature policy and transaction ownership. They do not prove deployed suspension, hosted Dot/Muse eligibility or full Native task execution. The separate private useful-case pass does not supply missing current-master or accounting evidence. ## Risks The broker identity contract changes for all three providers. A missed adapter could use the wrong owner. Current semantic, real-policy and exact broker-owner identity tests cover that error. The read-only projection reuses the normal settings view. It preserves managed and staging rules. An absent row is projected without a write; the existing initializing getter stays unchanged. No schema, public API, queue, timer, rate limit or provider deadline changes. No historical Retry or physical-stop proof is changed. ## Model Used OpenAI `gpt-6.1-sol` assisted with source changes and tool use. An OpenAI GPT-6 family coordinator reviewed the work; its exact runtime ID was not exposed. Context-window sizes were unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (tested revision and source-carry limits stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing command or contract change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 01:02:03 -06:00
DottaandClaude Opus 5.5	6d112c74cf	fix(native): do not charge a restart resume that never started the provider (#19747) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - #19720 stopped charging a crash restart when the previous restart resume made progress > - In a crash loop (several kills within seconds), the controller can die again before the resumed execution starts its runner > - The next claim then charged that resume, although it used no provider work > - This pull request does not charge a resume that never started; the next claim replaces it > - The benefit is that fast restart bursts do not exhaust a healthy run's budget, and the generation cap still bounds real loops ## Linked Issues or Issue Description Refs #19720, #19625 **What happened?** In a restart-chaos storm wave, 8 runs still reached `execution_recovery_budget_exhausted`, mostly in K6 crash loops (three kills in about 20 seconds). Example: a Codex run was claimed at generation 9, attempt 3. The controller was killed 6 seconds later, before the executor recorded `native.process_start_requested`. The next claim found no progress after generation 9's position, charged, and blocked the run at attempt 3. The recorded history of the blocked runs shows that almost every charge after the first restart was for a resume with no start evidence (no runner request and no runner event) before the next claim. Two runs were blocked at generation 12, the existing cap. **Expected behavior** A resume that never started the provider consumed nothing and must not be charged. A resume that started and then made no progress must still be charged. The generation-12 cap stays. **Steps to reproduce** Start a native run. Kill the server during the run. Let a restart resume do work. Then kill the server repeatedly within seconds, before each new controller starts the runner. After one or two such bursts the run is blocked with `execution_recovery_budget_exhausted`. ## What Changed - `native-restart-recovery.ts`: `restartResumeKeepsAttempt()` replaces `previousResumeMadeProgress()`. It reads the previous claim's recorded `progressWatermarkSeq` and `progressAttempt` (`lastRecoveryClaimResume()`). - New `nativeExecutionStartedSince()`: after the watermark, it looks for `native.process_start_requested`, `native.process_identity_recorded`, `adapter.invoke`, `tool.execution.started`, or any event with a runner `source_event_id`. This is the same start evidence as `hasNativeExecutionStartEvidence()`, limited to events after the watermark. - When the previous resume never started, the claim keeps the attempt if the run is already at the attempt that resume was to run as. If the executor never applied that charge (killed before its execution lease), the claim charges once, so the run reaches that attempt and never goes above it. - When the previous resume started, the #19720 progress rule applies unchanged. - The generation-12 cap applies to both rules. - `doc/execution-semantics.md` states the rule. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): - `does not charge for a resume killed before it started the provider (after the executor took its lease)`: one progress resume, then four crash claims with no start evidence. The attempt stays 2. A resume that starts and fails then charges to 3. Red on master (charged to 3, then blocked). - `... (before the executor took its lease)`: the same with the lease never taken. Green on master and with the fix, as a guard. - `charges the owed attempt once when an unstarted first resume is replaced`: the replacement claim charges once (attempt 2). - `bounds a crash loop that never starts the provider`: blocks after the generation cap. Red on master (blocked at the third claim). - The #19720 tests now record `native.process_start_requested` in their resumes, as a real resume does. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts`, `native-remote-retry-admission.test.ts`: 231 passed, 14 skipped. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A crash that happens before the runner start, every time, gets free resumes until generation 12. Then the budget applies. This is the same bound as the progress rule. - Start evidence is the executor's durable `native.process_start_requested` record, written before any backend may spawn a runner. If an older path launched a provider without it, that resume would be treated as unstarted until the generation cap. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 01:00:34 -06:00
DottaandPaperclip	f673bd3cb5	fix(native): verify legacy harness filesystem independence (#19748) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native sessions retain provider checkpoints and recovery state across turns. > - Server replacement must preserve every remaining local data dependency. > - A legacy backup stamp can outlive a destroyed provider and an absent local backup. > - Surviving skill files can already have exact immutable package copies. > - Raw controller files also contain stream events that database checkpoints omit. > - This pull request records verified filesystem independence under the original-volume maintenance owner. > - Replacement startup checks durable bindings while existing resume failures and policies remain intact. ## Linked Issues or Issue Description Refs #19715, #19729, #19733. **What happened?** Startup treats every retained legacy backup stamp without a current backup row as a local filesystem dependency. This also blocks sessions whose provider was already destroyed and whose remaining local skill bytes have exact durable copies. **Expected behavior** An authorized maintenance owner can prove that an exact retained source has no uncovered local harness payload. Startup can use that proof after replacement. The proof must preserve failed recovery state and must not invent a provider backup. **Steps to reproduce** 1. Retain a terminal native session, its checkpoint, and its legacy backup stamp. 2. Record exact canonical destroyed-provider receipts. Keep only admitted immutable skill files on the original host. 3. Publish the owner-verified independence receipt. The old startup predicate still blocks; the updated predicate validates its durable bindings. ## What Changed - Add an internal original-volume capability. It requires a private nonce anchor, independently reviewed attachment evidence, current reserved runtime identity, and a witness that is at most 120 seconds old. - Verify retained and selected-source inputs, checkpoints, leases, destroyed-provider receipts, released ownership, and exact immutable skill references. - Inspect canonical persistence roots and compare surviving skill bytes and modes. Refuse any provider directory present outside the current persistence profile, including an old Codex home after a preconfigured-home switch. Unknown data, backups, history, owners, or source changes still block publication. - Preserve raw controller and supervisor bytes through the existing encrypted evidence archive owner. Match recovery records to the database checkpoint, authenticate the exact archive row, and compare the original bytes again before publication. The archive grants no executable authority. - Publish one audited receipt with source locks and a final comparison. Keep original files, stamps, warnings, statuses, and session pointers. A late capture retains a valid published winner and refuses an invalid receipt. - Validate published receipts from current database bindings on replacement hosts. Add no startup filesystem probe, object read, provider call, or recurring worker. ## Verification - Genuine baseline regression: restore the original startup predicate while keeping the actual receipt owner. The positive PostgreSQL/filesystem case fails with expected `false`, actual `true` after valid publication. Restore the changed predicate; the case passes. - Final rebased head `e78f4b4424`: 42 affected PostgreSQL/filesystem cases pass. This includes 32 owner guards, three paired continuation cases, six existing failed-source archive controls, and one existing lease-verified backup migration control. The other 113 tests in the selected existing files were explicitly unselected. - Genuine controller baseline: the unchanged prior owner publishes no raw archive and accepts missing checkpoint parts. Both new regressions fail. The repair passes encrypted byte-for-byte controller/supervisor restore, missing checkpoint, foreign source, active source, and changed-after-upload controls. - Genuine omitted-provider baseline: the unchanged owner accepts old Codex, OpenCode, and ACPX history after a preconfigured-home switch. All three refusal cases fail. The fix passes those cases and permits preconfigured execution with no omitted directory. - Genuine late-publication baseline: both concurrent winner tests fail before the repair. The repair keeps the existing valid receipt, refuses an invalid one, and keeps one audit event. A delayed raw upload also retains the first authenticated archive and its receipt. - Capability controls: 12 passed, including empty replacement refusal, stale or future evidence, changed deployment, missing volume, path aliases, changed anchor, and current remount identity. - The actual server-to-Runner boundary preserves explicit recovery policy on original and empty homes. The separate actual CREATED-sandbox preparation callback preserves checkpoint identity and policy without a backup. Provider I/O and final transport construction are controlled. - Warm sessions retain the same canonical refusal on both homes, make no backend/provider attempt, and retain the saved checkpoints. - Server TypeScript, module boundary checks, diff checks, and a private-data scan pass. The Runner TypeScript build was refreshed after rebase to include two new upstream exports. - Fresh Greptile review is 5/5 with no unresolved threads on `e78f4b44240b0fc33c2fd7cae2d5b4a2578e3153`; check `128330103712` completed successfully. Required CI remains pending. No live preservation or enrollment action ran. Focused command from `server/`: ```sh node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts src/services/native-runtime/native-harness-filesystem-independence-continuation.integration.test.ts src/services/native-runtime/native-controller-migration.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts --testNamePattern 'owner-verified|legacy harness continuation|preserves a failed source online|migrates a lease-verified' --hookTimeout 90000 --maxWorkers 1 --fsModuleCache ``` ## Risks - This proves that no uncovered harness payload remains local. It does not recover provider history or certify that no other copy exists. Existing governed-wait and warm-session failures remain. - Unknown providers, retained owners, extra payloads, missing references, or changed source bindings fail closed. Invalid receipts require investigation; this change does not repair or replace them automatically. - The maintenance capability requires independently reviewed Railway attachment evidence. The original anchor remains stable across normal releases; each new staging invocation needs current deployment and mount evidence. - Startup validation adds bounded per-scope database reads. It pages candidate scopes and caps each source history at 500 rows. Measure the affected cohort's actual startup latency on the qualified serving build before enrollment. - Raw archives use the existing backup ledger, encryption, retention, and restore machinery. They preserve evidence only. They do not repair missing provider history or grant a new resume policy. - Local tests control provider and transport boundaries. Live staging remains a separate step after review, merge, normal delivery, and a fresh original-volume capability. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and local code/test execution. The exact runtime variant and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 00:57:20 -06:00
3 people	96bc021edb	fix(server): certify settled legacy processes for future retries (#19377) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Bounded retries must preserve current authority, budgets and deadlines. > - The tenant can wait for supported future work after the source process and its cleanup stop. > - Retained process identity alone cannot prove that cleanup finished. > - This change records host-owned settlement and checks its exact identity before a retry can cover its wait. > - Uncertain process state, changed authority, live claims and cleanup debt keep the tenant awake. ## Linked Issues or Issue Description Refs #19001. Related dependencies: #19019 and #18988. Related #19340 covers explicit Native task Retry; this change covers future Legacy process retries. Current integrated source: `1645970a9322d1213ba25fc7b52366c285682977`, normally merged with actual master `131b566ca4fa866c4348739c75cfe48c805b3398`. The prerequisites above are merged. This PR targets `master`. **What happened?** A future issue, company or review retry can retain its failed source process identity. The deadline reader could not prove that host cleanup finished. Supported waits stayed blocked. Ordinary issue retries also needed an exact source-settlement check, and represented coalesced requests needed exact input coverage. **Expected behavior** A supported future retry covers only its existing wait, under current company and actor authority, after exact host settlement. Missing, malformed, stale, cancelled, Native or unfinished source state stays blocked. No retry gains new execution authority. **Steps to reproduce** Fail an ordinary process or fail environment preparation before adapter entry. Let normal cleanup schedule the existing retry. Inspect its settlement and deadline, stop and reconstruct the owner, and admit one useful successor at the saved deadline. Hold cleanup or change authority to verify that the reader refuses coverage. ## What Changed - Add nullable internal `legacyProcessSettlement` state through normally generated migration `0439_cultured_impossible_man`. No backfill or adapter result creates the record. - Write exact host settlement after stopped-process and cleanup checks. Compare the current run and reject live claims or cleanup debt. Optional persistence failure preserves normal cleanup and keeps sleep coverage blocked. - Add a separate preparation receipt for a claimed Legacy run that failed before any adapter or Native dispatch. Require the same boot, run times, preparing stage, released claim and completed cleanup. A null PID alone proves nothing. - Clear settlement on a new claim, process metadata or Native selection. Omit the internal record from public serialized data. - Check exact source lineage and current authority for ordinary issue, company and taskless retries. Cover completed coalesced requests only when the admitted retry represents their exact inputs. - Add the closed `current_review_retry` wait with current reviewer, stage and existing issue authority checks. Require a nonempty current stage ID. - Add actual producer, cold reconstruction, SQL negative, redaction and optional-persistence controls. - Repair CI fixture expectations for the additive column, startup mock, budget owner and popup return URL. Bind the controlled Native comment server explicitly to loopback. - Replace temporary Slack clock logging with an exact persisted-deadline assertion. Wait until that real deadline before the existing single default poll. Keep both forced polls, business assertions and timing limits. ## Verification - On final integrated source `1645970a`, normal Shared/SDK prerequisites, Runner TypeScript and the entire normal App layered typecheck pass. Tracked source and lockfile remain unchanged. - All 26 whole-file settlement, recovery-reader and bootstrap controls pass. Seven separate real PostgreSQL producer/negative cases pass at `cf16c6a`; each has one selected passing case and 18 filtered registrations. Filtered registrations are not acceptance. - Master then changed the preparation source-lock scope. Its three intersecting cases were rerun on final `1645970a`: an actual failed company process, an actual failed ordinary issue process, and held preparation cleanup. Each preserves its source identity and original deadline and admits one useful successor. Issue and preparation cases exercise normal automatic retry; the company case uses the normal explicit bounded-retry producer. - The unchanged 26 controls and four other PostgreSQL cases retain their `cf16c6a` proof after a scoped source comparison. Optional settlement/preparation persistence failures keep certification blocked while normal cleanup completes. An entered adapter without process identity stays uncertified. This is scoped carry, not a claim that every earlier case ran again on the final head. - All recorded test owners and controlled groups closed. Final read-only SQL audits returned no test sessions. One earlier postcase resource barrier observed a transient unattributed database backend; that database and original evidence remain preserved. Its later disappearance is recovery, not an explained cause. No assertion, deadline or test body was relaxed. - Migration0439 was generated normally after master history. All executable SQL and journal history are preserved; the five retained snapshots are0435–0439. Every Legacy-owned patch and unrelated actual-master blob/mode is preserved. - Published exact head `1645970a9322d1213ba25fc7b52366c285682977` passes all current required checks: 68 successful contexts and eight expected skips after exhaustive pagination and superseded-run accounting. The fresh complete Greptile review scores this exact head 5/5. All four inline review threads are resolved. The latest Ponytail turn stopped without a verdict; one normal retry was requested, with no active turn or new finding observed. No old-head score is used. The original old-base policy failure, its dependent aggregate failure, and the corrected description-gate failure remain recorded. No full local repository suite is claimed; remote typecheck, test and build gates pass. To repeat the real database coverage, set both `PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL` and `PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL` to the same fresh dedicated test database and run the selected `recovery-deadline-acceptance.test.ts` cases with one worker. An unset URL, skipped case or filtered registration is not acceptance. The fixtures use controlled local processes and real PostgreSQL. Reconstruction creates new service owners inside the same test process. They do not prove deployed Cloud alarms, physical host/database suspension, new-host recovery or Native capacity settlement. The original30-second retry remains below the Cloud270-second minimum wake lead. The merge projection also includes actual master `c3180398faca815fed7a62674bd4b0c4e88884ae`, whose common completion path now sends deferred/recovery notifications. Its unpublished normal merge tree is `16b028d57150f8caa576e913f38343a365b542dd`. Normal Shared/SDK, Runner TypeScript and the full App graph pass on that combined source. One unchanged ordinary-issue acceptance case was rerun there: an actual failed process retains the original automatic retry deadline, admits one useful successor, and passes the settlement, authority and no-duplicate assertions. It passed in 45.055 s; 18 filtered registrations are not acceptance. All known owners and groups closed, and the independent final database audit found zero remaining test sessions. The successful database was removed normally. Recorded worker socket peak is unavailable for this run; parent-process zero values are not used as worker measurements. All 29 Legacy-owned paths and every unrelated master blob and mode are preserved. Two outside-diff review observations concern existing waiting-chat compaction and cross-host Native result-lease recovery. Their files are byte-identical to current master. This focused Legacy change does not fix or qualify those separate paths. ## Risks Settlement proves exact local closure. It does not cover live provider work, Native ownership, cancellation, costs, claims, cleanup debt or changed authority. Independent checks remain. Ship normal migration0439 with the reader and writer. Preserve executable history and the newest five snapshots0435–0439. Generate the MCP child's next migration normally after this parent. The prerequisites are merged; the final remote checks must use this integrated exact head. This proof uses controlled local processes and external PostgreSQL. It does not establish deployed Cloud alarm acceptance, physical host or database suspension, cold host recovery, or R5 Native capacity settlement. The30-second retry is below the Cloud minimum wake lead. This focused repair follows the roadmap's bounded recovery and Cloud deployment work. It adds no new provider policy. ## Model Used OpenAI `gpt-6.1-sol` assisted with readers, fixtures, source review and verification through Codex. A coordinating GPT-6-family agent implemented and reviewed shared lifecycle wiring and integration. Tools included Git, native TypeScript checks and controlled tests. Exact served revisions and context limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and its available capability details - [x] I have checked ROADMAP.md and related work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public work and described the issue - [x] I have not included internal issue IDs or private links - [x] My branch name describes the change - [x] Required focused checks pass at their explicitly recorded source/carry scopes above - [x] I have added or updated meaningful tests - [x] I have recorded the internal contract and migration requirements - [x] I have documented risks and evidence limits - [x] All required published exact-head Paperclip CI gates are green - [x] Published exact-head Greptile is 5/5 with no open findings in this focused diff; separate unchanged master observations are identified above - [x] I will address reviewer findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-11-07 00:56:04 -06:00
4 people	c3180398fa	fix(native): resume a remote Runner after a settled suspension ... fix(native): resume a remote Runner after a settled suspension A scheduled checkpoint retry can retain a remote reattach claim after closing its failed predecessor cleanly suspends and exits the Runner. Resume only that exact settled source after confirmed exit and a fresh identity/state check. Preserve pending work and uncertain-exit fences. Admit the last already-charged provider attempt and share one bounded recovery deadline with the final state read, retaining closed diagnostic labels on read failures. Cover fresh ACPX Runner relaunch on attempts two and three through actual processes and the durable close barrier. Six affected suites pass 1,579 tests, with 14 environment-gated skips; all server typecheck layers pass on Node 25.6.1. The complete provider turn continuation failure remains preserved as follow-up #19751. Refs: #19366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>	2026-11-06 22:39:40 -08:00
Nicky Leach	08b049f381	Admit qualified Claude sources through the existing Core claim (#19196)	2026-11-06 22:25:22 -08:00
DottaandClaude Opus 5.5	e620ce3678	fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 00:24:09 -06:00
DottaandPaperclip	18b6b71d83	fix(native): refresh per-turn projections from the selected checkpoint (#19744) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Per-turn native sessions save controller checkpoints and suspended Runner backups under separate owners. > - A local controller projection can still name the first turn after the next turn has saved its checkpoint and backup. > - The old per-turn restore path selected that stale local reference instead of the newer canonical source. > - Admission then saw different turn identities and archived the valid backup with the old projection. > - This change uses the canonical selected checkpoint and the existing finished-owner proof before it replaces a stale projection. > - The benefit is that a third turn keeps the valid saved conversation and backup. ## Linked Issues or Issue Description **What happened?** After two successful per-turn remote runs, a third turn could fail before provider execution with unavailable provider resume state. The selected database checkpoint and suspended backup belonged to the second run. The local controller projection still belonged to the first run. **Expected behavior** The selected checkpoint must govern controller rehydration. Preserve the older projection and the selected run's backup. Do not overwrite current, newer, foreign, changed or unsettled ownership evidence. **Steps to reproduce** 1. Save two real controller checkpoints for one per-turn conversation and finish both owners. 2. Leave the first checkpoint's readable projection or external marker beside the second run's suspended backup. 3. Admit the third run through the real canonical selector and backend owner. 4. Before this change, admission archives the active backup with the stale projection and leaves no active controller state. Related: #18018 added per-turn controller restoration. #18718 refreshed warm projections. #18846 added finished-owner retirement. This change keeps the selected per-turn source from entering that retirement path because of an older projection. The open warm-only change #18436 concerns a different restart branch. ## What Changed - Resolve the canonical source before refreshing a prior per-turn controller projection. - Re-read it under the existing sorted coordinator locks. Replacing an older owner reuses finished-owner checks for both runs and verifies their full session scope. A same-source marker only rehydrates its exact selected identity before the normal retained-sandbox verification. - Recheck the exact initial local bytes after asynchronous reads and lock waits. Preserve the old control-plane directory in the existing archive, while keeping the newer backup and provider files active. - Restore a current-run external marker only from its own checkpoint before the existing bootstrap retry proof. A missing current checkpoint cannot select a previous turn. Retain readable current-run, legacy file-backed, backup integrity and provider resume checks. - Stage the existing OpenCode proxy bytes into the controlled fake-sandbox filesystem in the real ingress-retry fixture. Keep remote artifact path and digest checks unchanged. - Add real PostgreSQL/filesystem coverage for Codex and ACPX, readable projections and external markers, missing or corrupt parts, held owners, foreign scope, newer history and a current owner arriving during a real coordinator-lock wait. ## Verification - Baseline RED: the two new readable-projection cases failed against unchanged production code because active control-plane state was absent after admission. The real selector selected the second run. Setup failures were separate and are not counted as RED. - Exact final existing per-turn restore, active-owner refresh and retained-owner wait controls: 19/19 passed in 5.91s. - Exact final local source (`fd8d9d9021`, based on `131b566ca4`): 58/58 real PostgreSQL/filesystem and native Runner tests passed in 31.58s. This includes both existing ingress bootstrap retries through the real Runner and fake Codex/OpenCode providers, plus 56 focused owner cases. Cargo validated and rebuilt current source from a private APFS clone of an unqualified local cache; the original cache was untouched. A private local configuration resolves workspace DB, Shared and Runner sources without replacing behavior. The local CLI allowed a 90s setup hook; no default timeout changed. - Exact final full server and test-tree TypeScript check with Runner source paths: passed. The stock fresh-worktree check first failed on unbuilt Runner declaration entries; no product types were changed to hide that setup gap. - Exact final module-boundary and diff-whitespace checks: passed. - Native-server CI RED on the previous head: both real ingress retry cases failed because a current-run marker was returned without hydration. The aggregate gate failed from that one underlying lane. The correction preserves the own-checkpoint bootstrap path. The first local OpenCode attempt hit a controller-host fixture path guard before its intended failure; staging the same proxy bytes into the fake sandbox made both native cases pass. - Review regression RED: the real retained-sandbox fixture failed at `sandbox_resumed` on the first PR head. GREEN: the corrected owner hydrates the same-source marker and reaches the existing exact retained-lease/suspended-runner read. A mismatched item identity still refuses before any remote read. - Local environment: macOS arm64, Node 26.4.0, PostgreSQL 18.1. Provider execution is deliberately outside this admission fixture. The real three-turn workflow still needs verification on the normally deployed fixed source. ## Risks - Refresh requires settled, committed, released and stopped or fenced owners. Missing or changed evidence remains blocked. - The refresh holds existing coordinator locks while it validates bounded controller state. It performs no provider calls. - The old control-plane bytes remain in the archive. A restore write failure still fails admission; it does not reset provider state. - Final-head full CI is green, including typecheck, build, Rust, native-server, server and browser suites; Greptile is 5/5 with no unresolved threads. Hosted deployment and continuity verification remain separate acceptance steps. A successful replacement conversation is not proof of provider continuity; hosted continuity acceptance is still pending. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The host does not expose the exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 00:14:17 -06:00
DottaandClaude Opus 5.5	24cbea136f	fix(native): export OpenCode launch artifacts through a confined workspace copy (#19743) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native recovery saves the qualified OpenCode launch files so a cold allocation can resume the same session (#19622) > - Core reads those files from the sandbox with the provider's bulk `syncOut` > - The launch root is under the sandbox account home, not under the workspace remote dir > - Daytona, Kubernetes, and CreateOS confine every export source to the workspace remote dir, so the export always fails > - Every remote OpenCode run on a durable-storage instance now fails 40-90 s after start with `native_session_interrupted` > - This pull request exports a confined scratch copy of the three launch files from inside the workspace runtime dir > - The benefit is that remote OpenCode runs work again, and the provider confinement guard stays unchanged ## Linked Issues or Issue Description Refs #19622 **What happened?** After #19622, every remote OpenCode run on Daytona fails at provider admission. The run error is `Daytona sync source path escapes the workspace remote dir: /home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/<digest>`. The error code is `native_session_interrupted`. The last successful run of the affected agent was immediately before #19622 reached the deployment. **Expected behavior** Core saves the qualified launch files and the run continues. The provider confinement guard still refuses any export source outside the workspace remote dir. **Steps to reproduce** 1. Configure an OpenCode agent on a Daytona environment with durable harness storage. 2. Start a run. 3. The run fails when Core captures the provider artifacts, before the first turn. ## What Changed - `captureRemoteNativeProviderArtifacts` now takes the workspace `remoteCwd`. - Before the export, a sandbox command copies the three launch files into a new scratch directory at `<remoteCwd>/.paperclip-runtime/paperclip-runner/provider-artifact-export/<uuid>/`. The command refuses a symlink or a non-regular file. It uses `cp -P`, so it never follows a link. - `syncOut` reads from that scratch copy. A `finally` step removes the scratch copy after success or failure. A failed removal is logged and does not replace the transfer result. - The launch root does not move. OpenCode still binds to the same absolute paths. Core still verifies each downloaded byte against the manifest SHA-256 values. - `.paperclip-runtime` is already excluded from workspace sync, so the scratch copy never enters the user workspace. - The Daytona `assertConfinedSandboxPath` guard is not changed. ## Verification - New regression `server/src/services/native-runtime/native-provider-artifact-capture-confinement.test.ts`. It runs the real sandbox commands on a local directory tree that has a home-rooted launch root and a separate workspace. Its `syncOut` applies the same workspace confinement rule as the Daytona provider. - Before the fix: it fails with `Daytona sync source path escapes the workspace remote dir: .../home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/dfe9e4b1d660/bin/nodeCommand`. This is the staging error. - After the fix: it passes. The published bytes match, every export source is under the workspace runtime dir, and the scratch directory is empty after the capture. - Safety cases: a failed transfer removes the scratch copy and does not publish. A launch file replaced by a symlink is refused before any export, and nothing is published. - `native-session-executor.test.ts`: the durable OpenCode admission case now asserts that the capture receives `remoteCwd` and that every `syncOut` source is under `/workspace/`. - `pnpm vitest run` for the capture, confinement, remote OpenCode artifact, and session root entries tests: 14 passed. - `vitest run native-session-executor.test.ts -t "opencode|provider artifact|durable"`: 62 passed. - `pnpm --filter @paperclipai/server typecheck`: passed. - Not run: a live Daytona run with this revision. The fix is not deployed to staging. ## Risks - Low risk. The capture now runs two short `sh` commands in the sandbox and holds a second copy of the launch files during the transfer. The files are bounded by the existing 512 MiB limit and are deleted afterwards. - The scratch copy uses `cp -P`, `mkdir`, and `rm -rf` from POSIX `sh`. The Daytona image already uses these tools for the workspace sync. - If the sandbox stops between the copy and the removal, a scratch directory can remain under `.paperclip-runtime`. Workspace sync excludes that directory. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and read-only browser access to staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 00:04:21 -06:00
DottaandClaude Opus 5.5	9f0ebf6f70	fix(accounting): close a run whose turn was lost on restore, and stop succeeded runs holding all work (#19724) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets hold new work while the cost of completed runs is unknown > - In a restart chaos wave, a native ACPX Claude run crossed a server kill, resumed, and succeeded. Its usage receipt never became complete. > - For the 10-minute closure bound, admission refused all work in the company. This included tasks that never crossed a fault. > - This pull request fixes the receipt for a turn that was lost on restore. It also stops a succeeded run from holding all new work. > - The benefit: one late usage report no longer freezes a company, and hard stops still apply. ## Linked Issues or Issue Description Refs #19700 (the accounting closure bound that this follows). **What happened?** A native ACPX Claude run crossed a server SIGKILL in the middle of a turn. The Runner restored the session, and the run succeeded 54 seconds after it started. All 17 usage receipts stayed incomplete. Accounting settled the run only at the 10-minute closure bound. During those 10 minutes, admission refused all company work with "Company cannot start work while completed runs await accounting". **Root cause** ACPX reports provisional usage (`runDeltaComplete: false`) until a turn ends. The kill lost the turn that was active, so that turn never got a complete report. The restored Runner failed that turn with `provider_turn_lost_on_restore`. Native turn accounting (`createNativeTurnAccounting`) required every terminal turn to have usage, so this turn kept every later receipt incomplete. This was true even after the continuation turn completed with full usage. The admission gate then counted the succeeded run as a pending terminal run for the whole company. **Expected behavior** The receipt closes when the run's turns end. A turn that was lost on restore makes the usage a lower bound. A succeeded run with only usage outstanding does not hold other work. ## What Changed - `native-usage-accounting.ts`: a turn that failed with `provider_turn_lost_on_restore` and has no complete usage no longer holds the receipt open. The snapshot marks `accountingUsageLost`. - `native-session-executor.ts` and `native-usage-delivery.ts`: such a receipt carries the provenance note "a provider turn was lost on restore and never reported its usage; observed usage is a lower bound". - `anthropic-pricing.ts` and `codex-pricing.ts`: the rate-card pricers keep a producer's provenance note. - `native-usage-delivery.ts` and `usage-receipts.ts`: remote producer delivery keeps a `lostOnRestore` marker on the receipt's producer event, so replay also excuses the lost turn. Receipts saved without the marker still replay. - `budget-exposure.ts` and `budgets.ts`: admission waits only for pending runs that ended without succeeding (`waitingRunCount`). A succeeded run whose provider reported observed usage is bounded like a running run: its observed cost counts toward spend and reservation capacity, and its settled charge can still reach the hard stop. - `budget-exposure.ts`: a closure that timed out because of a control-plane restart keeps the unpriced-usage exemptions, whatever its token count. Restart evidence is a replaced receipt source, a native recovery transition that is not an ordinary same-run retry, or a turn lost on restore. A second chaos run showed the need. It recovered and succeeded, but its final usage report was lost with the restart, so it settled as unpriced 0 cents. The blocking policy then opened a hard incident and paused the company. A timed-out closure with no restart evidence still counts. - `run-cost-accounting.ts`: a lost receipt that carries a positive provider-reported cost settles at that cost as an estimate marked as a lower bound, not as unpriced. - `paperclip-runner-tool-authority.ts` and `budgets.ts`: a running run's own tool mutations are evaluated as admitted work. Hard stops and pauses still refuse them. Accounting holds, recovering runs and budget inputs that await evaluation do not. In a third chaos run, a live run's write to its task document was refused during a temporary hold, so the run proposed a blocked result. - `doc/SPEC-implementation.md`: documents these rules. Policy choice: a succeeded run's provider work is over, so it has less unknown spend than a running run. Running runs never held admission. Failed, cancelled, timed-out and interrupted runs still hold new work, because their provider may not have stopped. A per-agent gate would still freeze unrelated work of the same agent. A per-reservation gate does not help when `reservationCents` is 0, which was the case in this wave. ## Verification - Live: a restart chaos wave ran master plus this PR (c2a48fb1e0) with no masking. It passed 6/6, and every task crossed a restart. There were 0 accounting holds, 0 refused agent writes, 0 company pauses, and every cost settled as estimated. - Red then green: `native-usage-accounting.test.ts`, "closes a run whose earlier turn was lost on restore, as a lower bound". It uses the event sequence of the chaos run. - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for a succeeded run with only usage outstanding". It also checks that the hard stop still applies. - I changed "neither pauses nor holds admission while a succeeded run's charge commits" to the new policy. A failed run in the same state still holds new work. - Red then green: `closed-native-receipt-accounting.test.ts`, "never hard-stops a blocking policy for a restart-derived timed-out receipt". It uses the zero-token row shape of the chaos run and a case with unpriced tokens. Without the fix, the company was paused. - Red then green: `runner-api.integration.test.ts`, "lets a running run write during a temporary accounting hold, but not after a hard stop". - Red then green: `closed-native-receipt-accounting.test.ts`, "does not hold other work for the s7504 succeeded row shape". - Red then green: `closed-native-receipt-accounting.test.ts`, "settles a timed-out restart receipt at its provider-reported cost, without holding or pausing". It uses the OpenCode/OpenRouter row shape from a fourth chaos run. - `anthropic-pricing.test.ts`: the rate-card provenance keeps the lower-bound note. - Local: `closed-native-receipt-accounting`, the budget suites, `native-usage-accounting`, `native-usage-delivery.integration`, `native-runner-restart-recovery.integration` and `anthropic-pricing` all pass (187 tests). Server `tsc --noEmit` is clean. `pnpm verify` asks for CI coverage because the spec changed. ## Risks - A succeeded run whose final charge is higher than its observed cost can let one more task start before the charge commits. The hard stop then pauses the scope as it does for a running run that overruns. - A run that includes a lost turn settles with a lower-bound usage figure. The provenance says so, and a later receipt still reconciles through the late-receipt path. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:57:48 -06:00
4 people	131b566ca4	fix(workspaces): report safe Git restore diagnostics (#19736) ... Return the restore stage and failure category with the existing HTTP409 rejection. Preserve the unsaved comment and editable draft. Keep nested saved-commit and clone-timeout diagnostics through preparation wrappers, and distinguish local Git file errors from authentication failures. Retain only safe process flags privately; no raw command cause is exposed. All seven affected local suites pass (288 tests) and the layered server typecheck passes on supported Node25.6.1. Reviewed head 022eb7c2eb has Greptile Apex5/5, zero unresolved findings and all applicable CI passing (55 successful checks and4 intentional skips). Co-Authored-By: Sentry Bot <sentry-bot@paperclip.invalid> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-06 21:33:59 -08:00
DottaandClaude Opus 5.5	464e6e16e0	fix(native): resume copyback after a restart kills its same-host owner; one board action otherwise (#19717) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save the agent's workspace files after the provider turn finishes (copyback) > - One controller owns copyback at a time. The run records that owner in `runnerProfileJson.nativeWorkspaceFinalizationOwner` (host, PID, token) > - A control-plane restart can kill the controller during copyback. The dead controller can never confirm its own stop > - Recovery then waited for a board confirmation. No surface offered that confirmation, so the run stayed `running` and the task stayed `in_progress` with no visible action > - This pull request lets the next controller on the same host prove the stop by itself. It also gives the board one clear action when that proof is not possible > - The benefit is that a restart during copyback resumes by itself, and a task never waits silently for a person ## Linked Issues or Issue Description **What happened?** In a Daytona restart-chaos wave, a kill during workspace copyback left the run `running` and its task `in_progress`. Recovery reported `native_workspace_finalization_owner_unverified`. The run was still `running` when the wave timed out. **Steps to reproduce** 1. Start a native remote (sandbox) run. 2. Kill the control-plane process (SIGKILL) after the provider result is accepted, during copyback. 3. Restart the control plane on the same host. **Expected behavior** If the previous controller and its copyback children are proven stopped on this host, recovery resumes copyback without a new agent turn. Otherwise, the task shows one board action to confirm that the previous server stopped. **Actual behavior** The recorded owner was cleared only by `resumeNativeWorkspaceAfterOwnerStop`. Same-host recovery also required a completed-copyback barrier, which an interrupted copyback never has. The task presentation showed `working`, and the UI had no control for the confirmation. Refs #19578 (boot-ID fencing for unstarted runs; same proof style). ## What Changed - Root cause: `server/src/services/native-runtime/native-workspace-finalization-ownership.ts` sent a dead same-host owner to `requireStopVerification` when no `workspace_finalize` operation had succeeded. That case is exactly a restart during copyback. - New `native-workspace-owner-stop-proof.ts`. The owner receipt now records the controller's process group (`processGroupId`). Copyback children (tar, Git) are not detached, so they keep that group. A successor on the same host releases the receipt only when the receipt names another controller boot, the PID is dead or recycled (process start time), and the group is empty or has no member outside the successor's own process tree that started after the old controller. Age alone never proves a stop. - A still-live group member returns `local_stop_pending`. The finalization worker rechecks it on its existing ownership recheck interval. The board action stays visible meanwhile. - A successful local proof writes `issue.workspace_owner_stop_verified` with the proof and marks the recovery `resuming`, like the existing platform proof. - A receipt from another host, a receipt without a process group, or a host without `ps` keeps the fence. The task interruption is now `waiting_for_user` with `userAction: "confirm_owner_stop"` (new shared enum value), not `working`. - UI: the task chat shows one **Confirm the previous server stopped** action. It asks for stop evidence and calls the existing `resume-workspace-finalization` API. It is shown only to roles that hold `runtime:confirm_owner_stop` (company owner or administrator, local board, instance administrator). The inbox names the same action. - The copyback advisory lock, the coordinator finalization lease, and remote save fencing are unchanged. - Docs: `doc/native-workspace-finalization-recovery.md`. ## Verification - Red then green on embedded Postgres (`server/src/services/native-runtime/native-workspace-finalizer.test.ts`): - same-host controller killed with SIGKILL, its process group empty: before `owner_unverified`, after `acquired` with the proof logged and no agent wake - a non-detached orphan child of the dead controller still runs: `local_stop_pending`, no copyback; after the child exits, `acquired` - a receipt without a process group: still `owner_unverified` - a foreign-host owner: the task presentation was `working` (red), now `waiting_for_user` / `confirm_owner_stop`; after the board confirmation it is `working` again - The existing orphan test (a detached child) and all other finalizer tests pass: `npx vitest run src/services/native-runtime/native-workspace-finalizer.test.ts src/__tests__/native-workspace-finalization-work.test.ts src/services/task-interruption-presentation.test.ts` (135 passed). - UI: `npx vitest run src/components/WorkspaceExportRecovery` (34 passed), including the action call body and the non-administrator case. - `pnpm check:token-gates` clean. Server and UI typecheck clean for the changed files. - Live Daytona restart-chaos wave: results are added in a comment below. ## Risks - A process-table check that reports a member wrongly as live only keeps the existing fence. The board action is still available. - A subreaper that adopts orphans does not change their process group, so the proof does not depend on the parent PID. - A child that a future change spawns with `detached: true` would leave the group. Copyback paths do not do that today. The existing detached-orphan test still fails closed, because a receipt without a process group is not released. - Old receipts have no process group. They keep the board action. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:31:07 -06:00
DottaandPaperclip	7a72caeda1	perf(workspaces): batch online companion discovery (#19735) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace migration checks all local aliases before it captures a source. > - Each alias must retain its prepared runs and recovery holds in that check. > - The current reader scans the same company run table once for every alias. > - Shared Git sources repeat that scan even after the complete alias group is known. > - This pull request reads those narrow companion rows once per inspection. > - The existing owners still validate every source and recovery hold before migration can proceed. ## Linked Issues or Issue Description **What happened?** Online source qualification repeats the prepared and finalized run query for each workspace alias. A shared Git source with four aliases makes four copies of this query. **Expected behavior** One inspection should read the complete alias group once. It must return the same source and recovery proofs, and a later inspection must read fresh rows. **Steps to reproduce** 1. Create a shared Git source with four aliases and a valid settled prepared run. 2. Add a finalized companion and a recovery hold. Add a foreign-company run that names the same workspace ID. 3. Inspect the source and count companion discovery queries. Before this change, the regression sees four queries instead of one. Related public work: [source ownership guards in #19688](https://github.com/paperclipai/paperclip/pull/19688) and [bounded accounting closure in #19700](https://github.com/paperclipai/paperclip/pull/19700). ## What Changed - Read prepared and finalized companion rows in one company-scoped query after the complete alias group and foreign aliases pass validation. - Group the narrow rows by workspace ID. Keep run order and all existing source and recovery owner checks. - Add a PostgreSQL regression for proof equivalence, one scan, company isolation, and a new source obligation between inspections. - Align an existing Stop-accounting test with the durable closure deadline from #19700. Keep the incomplete debt, quiet database, cancellation, and single-charge checks. Product workers and timers are unchanged. ## Verification - Before the source change, the new PostgreSQL test passed its proof comparisons and failed because four companion scans ran instead of one. - The two workspace suites pass: 77 tests in `remote-workspace-online-source.test.ts` and `online-legacy-workspace-recovery.test.ts`. - CI exposed an old Stop-accounting assertion that expected no timer while an incomplete receipt had an intentional closure deadline. Both automatic CI attempts failed. The unchanged local variants passed, so that run did not reproduce the failure. Waiting for the real owner deadline then proved the stale assertion red. - After the test alignment, both Stop-accounting variants and all 77 workspace tests passed together: 79 tests across three files. The three unrelated Stop-recovery cases were not selected locally. Fresh CI covers the complete suites. - The affected suites cover retained recovery owners, foreign aliases, physical source locks, and changes before publication. - Tests use a disposable PostgreSQL 18.1 database. The local hook allowance is 90 seconds. Product timeout settings are unchanged. - Final head `0eed2e5062cbfb62df7f76168c27d2172fc2d7de`: [full CI](https://github.com/paperclipai/paperclip/actions/runs/42341640259) passed, with 56 successful checks and four configured skips. [Fresh Greptile review](https://github.com/paperclipai/paperclip/pull/19735#issuecomment-6702476739) is 5/5 with no open threads. This PR makes no claim about deployed latency. ## Risks - Each inspection now holds the narrow companion rows for the complete alias group in memory. The existing alias group bound still applies. - The query keeps the same company, workspace, and sync-state predicates. There is no runtime filter, row limit, schema change, or cache between inspections. - No public command or API changes. Public documentation does not need an update. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Used reasoning, code editing, shell tools, and test execution. The context window size is not exposed by this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 23:26:53 -06:00
DottaandClaude Opus 5.5	fc7e3a6503	fix(recovery): never replace a native session after a deterministic refusal (#19733) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs keep a saved provider session, and same-run recovery resumes it after a failure > - When resume fails and the checkpoint allows it, recovery archives the saved continuity state and opens a fresh replacement session > - Recovery used that path for every resume failure, including deterministic refusals that left the saved state unchanged and repeat on every retry > - In five staging tasks a refusal (a protected launch argument, or a predecessor tool call with no result) escalated into a replacement. The remote archive move failed part way, the next attempt found no runner state, and each task was blocked for workspace repair > - This pull request stops before replacement after a closed-class refusal, keeps the saved state, and hands the run to an operator with a named reason > - The benefit is that a deterministic refusal can no longer damage saved state or open a replacement over an unsettled external action ## Linked Issues or Issue Description Refs #19659, Refs #19516, Refs #19334 **What happened?** Five Codex tasks on Daytona sandboxes failed on staging. The first attempt failed with a deterministic refusal: - `run.attach cannot change protected launch arguments` (four tasks, fixed separately in #19659), or - `native_runner_source_semantic_reconciliation_required` with a predecessor `paperclip_call_assigned_tool` call still pending (one task). The same-run retry then opened a replacement session. `archiveContinuityState` failed with `runner_continuity_break_archive_failed` and no detail. The third attempt failed with `runner_remote_state_unavailable: exit=1`, and the task was blocked with "Workspace repair required". **Steps to reproduce** 1. Save a native session whose checkpoint policy is `allow_replacement_after_resume_failure`. 2. Make the resume fail with a refusal that does not change state (an attach refusal, or a pending tool call). 3. Same-run recovery opens a replacement session and archives the saved continuity state. **Expected behavior** A deterministic refusal stops the run for an operator. The saved state stays intact. A run with a pending side-effecting call never reaches replacement. Transport drops keep the existing retry behavior. **Root cause** `executeNativeSession` checked only the checkpoint's replacement policy. The closed failure class (#19334) only had an effect when replacement was forbidden. Attach refusals and pending tool calls had no class at all, and the source-reconciliation refusal was a plain `Error`. ## What Changed - Runner (`provider_backend.rs`, `durable/mod.rs`): `run.attach` refusals that would change the durable Codex profile or a protected launch argument carry the closed classification `provider_attach_refused`. An attach blocked by `durable_pending_tool_calls` carries `provider_side_effect_pending`. These refusals already leave the saved state unchanged. - Contracts (`harness-driver.ts`): new closed classes `PROVIDER_ATTACH_REFUSED` and `PROVIDER_SIDE_EFFECT_PENDING`. A typed `NativeRunnerSourceReconciliationRequiredError` replaces the plain `Error` at the five `native_runner_source_*_reconciliation_required` throw sites. `runnerRecoveryFailureClass` reads the authenticated command-result classification or the typed error, never message text. - Runtime (`native-session-runtime.ts`): after either class, recovery never opens a replacement session, whatever the checkpoint policy. - An attach refusal ends with the existing `NativeProviderStateIncompatibleError`, which maps to the existing operator path `native_provider_state_incompatible`. - A pending side effect ends with the new `NativeProviderSideEffectPendingError`. - Server (`native-session-executor.ts`, `native-session-resume.ts`, shared `EXECUTION_RECONCILIATION_CAUSES`): `native_provider_side_effect_pending` is a permanent failure and an execution-reconciliation cause, so the board gets a reconciliation (record whether the action ran) instead of a generic Retry. The reconciled continuation does not rebind the saved session, which still refuses attachment. It starts a fresh provider session in the same workspace. The saved session and workspace are unchanged. - Archive (`archiveContinuityState`): a failed remote move now records the bounded (512 characters), redacted exit status and output in `runner_continuity_break_archive_failed: exit=… timedOut=… <output>`. The remote move already ran before the local move on `master`, so a failed remote move leaves the local state in place. `restart-interrupted-turn.ts` accepts the detailed message for its reattachment-loss retry. - Docs: `doc/execution-semantics.md` and `packages/paperclip-runner/docs/durable-recovery.md` describe the new rule. ## Verification Red before the fix, green after (each test first run against unchanged source from `origin/master` 644ee66f5d): - `native-session-runtime.test.ts`, test "never replaces a session after a deterministic, non-destructive recovery refusal". Red: replacement was attempted (`TypeError: Invalid value used as weak map key` from the stub replacement). Green: typed error, and `openReplacementSession`, `openSession` and `openRun` are not called. - `harness-driver-backend.test.ts`, test "keeps the runner's closed class of a failed recovery". Red: the typed error did not exist. Green: both new classes are read from the command result and from the typed error. The same message text without the type stays unclassified. - `tests/codex_provider.rs` (real durable executor and fake Codex app-server): - `attach_with_a_pending_tool_call_is_classified_as_a_pending_side_effect`. Red: `left: None, right: Some("provider_side_effect_pending")`. - `saved_codex_thread_attaches_after_a_release_changes_its_feature_toggles` (protected sandbox change). Red: `left: None, right: Some("provider_attach_refused")`. - `native-session-executor.test.ts`: - Test "stops same-run retries for an operator while a saved side-effecting call has no result". Red: `expected 'native_session_interrupted' to be 'native_provider_side_effect_pending'`. - Test "retains the local controller checkpoint until remote continuity archiving succeeds: failed/timeout". Red: the message had no detail. Green: the message has the exit status and output, a token in the output is redacted, and the local runner and control-plane files are intact. Safety and regression checks: - New test "still replaces a session after an unclassified resume failure when the checkpoint allows it". It passes before and after: a transport drop still reaches replacement. - Existing test "types a runner-proven incompatible session when same-session recovery is required" still passes. - `native-session-resume.test.ts`: a source stopped with `native_provider_side_effect_pending` is not rebound. - `run-completion.test.ts` "replays saved user intent through normal Retry …" passes with the detailed archive message. - Runner Vitest: `native-session-runtime.test.ts` and `harness-driver-backend.test.ts` (266 passed); `runnerd-codex-transport-warm-attachment.test.ts` and `codex-app-server-driver.recovery.test.ts` (74 passed). - Rust: `--lib` (560 passed, plus one ACPX timing test that failed under load and passed when rerun alone), `codex_provider` (118 passed), `native_provider_backend` (45 passed), `durable_recovery` (3 passed). Not run: a live staging recovery at this revision. The local server typecheck could not resolve the unbuilt runner package, so required CI provides typecheck. ## Risks - A run that a replacement session would have continued after an attach refusal now stops for an operator. The refusal repeats on every retry, so the replacement was the only way past it. It also destroyed saved state, which is what blocked the staging tasks. A board Retry still continues the task in a fresh session in the same workspace. This is the existing `native_provider_state_incompatible` behavior. - A new failure code `native_provider_side_effect_pending` appears in run records and recovery actions. It joins `EXECUTION_RECONCILIATION_CAUSES`, so the UI shows the reconciliation flow, not Retry. - An attach refusal used to reach an automatic replacement; it now needs one board Retry. This is intended: the replacement archived saved state and failed part way on staging. - The pending-side-effect class means "no result delivered to the runner". The controller replays a recorded tool outcome only after attach. A call whose outcome the server already recorded therefore also stops for an operator. That is conservative; the reconciliation shows the recorded outcome. - Only a pending tool call classifies an attach blocker. Other blockers (active turn, ambiguous turn start, active goal) keep their current behavior. - The archive error message is longer. The one exact-match consumer now accepts the `code: detail` form. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser reads of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:25:54 -06:00
DottaandClaude Opus 5.5	c97c0dd248	fix(workspaces): give run preparation its own source lock scope (#19734) ... <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run prepares its task workspace under per-source advisory locks (`workspace-source-lock.ts`). The locks live in a transaction on a dedicated connection. An `AsyncLocalStorage` scope lets nested calls reuse that transaction. > - Timers and detached promises keep the async context that created them. A native recovery timer or dispatch can start a run from inside another operation's source scope. > - When that run prepares its workspace, it sees the other scope as "held". If that scope already committed, its dedicated connection is closed. The lock query then fails with `Failed query: select pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that scope is still open, the run's lock is tied to the other transaction and is released early. > - In a restart chaos wave, a reattached follow-up run failed setup this way 50 ms after "Recovering the existing native runner process". The setup-failure log did not show the driver cause. > - This pull request makes only a live scope count as held, and makes run workspace preparation take its locks in an independent scope. > - The benefit is that a recovered or dispatched run gets its own source lock, and a lost-connection failure is diagnosable from the log. ## Linked Issues or Issue Description **What happened?** After a control-plane restart, a native run was reattached (`reattach_remote_runner`). Its workspace preparation failed at once with `Failed query: select pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params: isolated_repository_source:<company>:<issue>`. The run ended `failed`/`setup_failed`. The stack showed the "already held scope" branch of `withWorkspaceSourceLocks`, not a new acquisition. The same failure occurred earlier in the same wave on a different issue without a restart. **Steps to reproduce** 1. Call `withIsolatedRepositorySourceLock` for issue A. In its action, start a detached promise that later calls `withIsolatedRepositorySourceLock` for issue B. 2. Let the outer action return so its transaction commits. 3. Let the detached call run. It fails with `Failed query ... pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`. **Expected behavior** Work that starts after a source scope ends, or independent work such as a run's preparation, acquires its own locks on its own connection. **Environment** Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1, comment trigger), embedded Postgres. Reproduced locally on embedded Postgres by the new tests. Related: #14910 (deferred comment wake reopens a closed issue). This PR does not change wake promotion. ## What Changed - `server/src/services/workspace-source-lock.ts`: a scope has an `active` flag. The flag is cleared when its transaction ends. `withWorkspaceSourceLocks`, `retainManagedProjectSource`, `retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld` use only a live scope. - `server/src/services/workspace-source-lock.ts`: new `withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope and ignores an inherited one. - `server/src/services/heartbeat/workspace-preparation.ts`: run workspace preparation uses `withIndependentWorkspaceSourceLocks`. - `server/src/services/heartbeat/run-completion.ts`: the "heartbeat execution setup failed" log records a bounded driver `causeCode` (for example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause text. - Tests: two embedded-Postgres tests in `isolated-remote-workspace-source-lock.test.ts`, one unit test in `run-completion.test.ts`. ## Verification - `cd server && npx vitest run src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed. - Red (old scope rule): "gives work detached from a finished scope its own source lock" failed with `Failed query: select pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the production error. - Red: "keeps an independent preparation's lock after the scope that started it commits" failed because the lock was released with the other transaction. - `cd server && npx vitest run src/services/heartbeat/run-completion.test.ts` — 95 passed. - `cd server && npx vitest run src/services/remote-workspace-online-source.test.ts src/services/heartbeat/workspace-preparation.test.ts src/services/isolated-repository-rehome.test.ts src/services/isolated-task-directory-retention.test.ts src/services/isolated-task-directory.test.ts src/__tests__/execution-workspace-reopen.test.ts` — all passed. - `cd server && npx tsc --noEmit` — clean. - No live Daytona replay was run. ## Risks - Low. A nested call inside a live scope behaves as before. - A call that inherited an ended scope now opens its own dedicated connection and lock transaction. Before, it failed. - Run preparation never joins another operation's lock scope. No caller relies on that today. If one did, it would now take a second, independent lock and could see `workspace_source_busy` instead of silently sharing. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 23:08:57 -06:00
3 people	839d41aabb	fix(server): retire owned plugin listeners before idle sleep (#19358) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Plugins receive durable agent lifecycle events through a registered host. > - The host uses a PostgreSQL LISTEN connection to wake delivery. > - An idle report could return `none` while that host-owned connection stayed open. > - This change gives that listener an exclusive owner and closes it during the current idle hold. > - Release or expiry restores listening before journal replay. > - Failed close and pending work continue to block sleep. ## Linked Issues or Issue Description Refs #18988, which is now merged. This focused ten-file change now targets master. Current integrated candidate: `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`, with actual master `644ee66f5d4ae8365fe8b9eca57df2be7405b5ab` preserved through normal merges. No history was rewritten. **What happened?** A normal local App with an installed lifecycle plugin could return an idle `none` report while its lifecycle LISTEN backend and socket stayed open. **Expected behavior** The host must close its own listener before it reports idle safety. It must retain other subscriptions, durable events, company scope, retry deadlines and useful reactivation. **Steps to reproduce** Start the normal server with a lifecycle plugin. Create an agent and wait for its durable acknowledgment. Stop the server, create another agent while it is stopped, and restart with the same files. Request an owned idle hold after ordinary replay. Check the exact listener backend and matched Node socket before shutdown. Let the valid hold expire naturally, then make an ordinary authorized agent update and check its durable plugin acknowledgment. Live executing or persistent preview services remain active. The separate hosted preview dependency [paperclip-cloud#478](https://github.com/paperclipai/paperclip-cloud/pull/478) retains its own App and operator prerequisites. This PR does not claim deployed preview or suspension acceptance. ## What Changed - Add a private, exclusive database listener owner that inherits the normal client policy. - Prepare plugin lifecycle retirement after native retirement under the same current idle hold. - Use one delivery-eligibility helper for dispatch and idle journal inspection. It preserves the same manifest, capability, company-disable query, and skip rules. - Close only the owned listener. Keep failed close ownership and inspection failures visible. - Resume listening on normal release or expiry, before journal replay. - Preserve disabled-feature retry history and the original deadline on re-enable. - Await lifecycle disposal during normal shutdown and consumer teardown. - Add owner regressions and document the lifecycle contract. ## Verification **Current integrated source:** `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`. Fresh normal Runner TypeScript passed in 0.785 seconds. The complete normal App layered semantic check passed in 28.284 seconds. All tracked source, the lockfile and the 18 separate private witness inputs stayed unchanged. Known compiler owners closed and the isolated temporary home was removed. **Current product qualification:** at measured integration `896cc9b62993e6ed10418d7a7417e1ddbf8dc89c`, all three whole files passed with no failures or skips: lifecycle delivery 40, runtime hooks 20, and runtime assignments 24. The actual physical LISTEN and natural-expiry controls passed. All owned clients and captured processes closed. Independent final session checks were zero before normal deletion of each successful test database. **Normal full-App physical retirement:** the original case passed in 47.574 seconds. It exercised real startup, readiness, SDK callbacks, durable ACK, foreign-company exclusion, physical stop, offline input and cold replay. The initial unknown hold and its ordinary release remain in the evidence. After the original 30.5-second cadence, a new valid hold returned quiescent. The exact current cold-App LISTEN backend and its matched Node socket closed before server stop. **Normal full-App natural expiry:** the separate original case passed in 77.391 seconds under its original 90-second body cap. The final valid 30-second hold expired naturally. A new listener backend and matched current App socket appeared before fresh input. An ordinary authorized agent-name PATCH produced journal event 20, the actual SDK callback and durable ACK. Foreign exclusion and readiness assertions passed. The final measured hold had no manual release, replacement hold or shortened TTL. The original preparatory unknown hold, ordinary release and 30.5-second cadence remain explicit. Both App cases used the normal current server helpers and actual SDK bundling/worker entry. A private compatibility change supplied the same owned HOME at the exact existing child-fork boundary. It did not change the product helpers, business assertions or timing limits. These separate acceptance fixtures remain outside the focused product diff. **Source carry to the current candidate:** all nine other owned files, listener mechanics, actual producer paths, SDK paths and measured hold behavior are unchanged. The sole owned intersection from newer master is an independent environment-delete cleanup callback in `app.ts`. The plugin witnesses do not call that route. Normal merges preserve every unrelated master blob and file mode. The fresh current semantic checks above qualify the integrated source. New master Native crash-recovery and saved-copyback Retry capabilities are not claimed by these plugin cases. **Actual master merge target:** newer master contains retained plugin Runner release/resume and other Native recovery fixes. The projected combined target passed fresh normal Runner TypeScript in 0.799 seconds and the full App semantic check in 29.715 seconds. All ten plugin-owned blobs and modes remain exact to the published head; every unrelated master change is preserved. The measured plugin cases create no Native provider turn or warm Native session. The unchanged native-first/plugin-second ordering still checks the native result before listener retirement. Later sparse-checkpoint and Native response-publication changes do not enter those paths. This is a scoped source carry, not new hosted Runner acceptance. **Failures retained:** an earlier reactivation fixture's real timer returned one millisecond before its original wall-clock expiry. Its private fixture now waits any positive remainder and retains the same expiry assertion and limits. The earlier partial-composition `unknown` result remains distinct from normal-startup proof. Current qualification also exposed missing normal workspace dependencies and a private guard that assumed loose Git refs. Normal owned dependency setup and an exact loose-or-packed reader fixed those setup failures. No business assertion, timeout, permission boundary or product sleep gate was relaxed. Repeat focused product coverage with a fresh external fixture database and the normal server Vitest command, one worker: ```sh pnpm --filter @paperclipai/server exec vitest run src/__tests__/plugin-lifecycle-delivery.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-hooks.test.ts --maxWorkers=1 --no-file-parallelism pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-assignments.test.ts --maxWorkers=1 --no-file-parallelism ``` Exact current head `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1` has 60 successful checks, three skips and one neutral result, with no pending or failing contexts. Greptile completed its full review at 5/5 with no actionable issue. All root and nested review pages were read; the earlier eligibility finding is resolved. Ponytail's latest turn stopped before a verdict. A normal retry was requested; no current Ponytail verdict or score is claimed. Previous `3ca86ec` CI and scores are historical. A full local repository typecheck/test/build or browser run is not claimed. Controlled local App/SDK/PostgreSQL proof does not establish live provider, deployed Railway/Neon suspension, signed Cloud alarms or global exactly-once delivery. ## Risks - A listener close failure or uncertain inspection must continue to block sleep. - Release, expiry, configuration changes, and journal arrival can race with retirement. Epoch and current-hold checks retain conservative results. - Delivery remains at least once across uncertain ACK boundaries. This change does not add global exactly-once delivery. - The database helper closes only its exclusive listener client. Other owners and subscriptions keep their resources. ## Model Used OpenAI `gpt-6.1-sol`, reasoning and code/tool use. The managing Codex agent implemented shared database/composition seams and reviewed the change; its exact model ID was not exposed. The runtime does not expose a verified context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-11-06 22:56:47 -06:00
DottaandPaperclip	1c91bacf97	fix(native): admit independent replies during contended publication (#19726) ... ## Thinking Path > - Paperclip manages work done by AI agents. > - A completed native run must deliver its saved final response. > - The response worker also recovers retained replies that can wait on publication locks. > - A pass waited for its old publication groups even when other check slots were free. > - This could delay an independent final reply beyond a full page of retained candidates. > - This PR enables the existing shared coordinator option that admits independent due work during an active pass. > - Existing authority checks, resource keys and the four-check limit still control publication. ## Linked Issues or Issue Description **What happened?** A native run completed after replacement recovery. Its final response and accounting were saved, but its final issue reply was missing. The response queue contained more than 100 retained chat replies and repeated endpoint-lock failures. The existing worker did not admit independent due entries until its current groups finished. **Expected behavior** A contended publication owner must not hold free worker slots. An independent accepted final reply must publish once, including when its candidate is on a later SQL page. **Steps to reproduce** Create 100 accepted, committed replies that share one publication owner. Create an independent reply after them in candidate order. Hold the old coordinator rows with a database transaction and start the real response worker. Before this change, the independent reply stays unpublished until the old group is released. Related: #18977 changes response selection in standing conversations. This PR changes only worker admission and preserves response selection. ## What Changed - Enable `admitDueWhileActive` in the native response worker's existing shared coordinator configuration. - Add a real PostgreSQL regression with the full 100-row SQL page, the default dispatch batch, real row-lock contention and a later-page independent reply. - Verify one comment per reply after settlement and no further SQL or scheduled deadline across 60 seconds of idle time. - Update one existing principal/rescope integration assertion to observe the reconciled invalidation revision. Keep its actual per-run replies, authorization, publication and idle checks. ## Verification - Regression on the unchanged worker: the independent reply remained unpublished while the first 100 replies were locked. The real PostgreSQL test failed as expected. - Final head `748fb9609104c016c33ff7caf282ba9e15e88495`: all six publication regressions and both principal/rescope controls passed (8/8, 47.96 seconds). Checks cover later-page admission, exact per-run comment counts, retry and rollback behavior, authorization, and 60 seconds with no idle SQL or scheduled deadline. - The tests used the canonical PostgreSQL wrapper and one owned disposable PostgreSQL 18.1 fixture. All clients and test databases were closed before fixture cleanup. - The current-head [full CI run](https://github.com/paperclipai/paperclip/actions/runs/42338953241) supplies typecheck, build and broad test coverage. All current-head checks passed; the merge gate is green. - Local full typecheck was blocked by unbuilt dependencies; it is not counted as a pass. The prior timer-state CI failure and the final successful rescope-generation regression remain recorded. - Greptile reviewed the final head at 5/5 with no unresolved threads. `git diff --check` passed. - Deployed acceptance remains separate. No serving instance was changed by the PR tests. ## Risks The worker can now use free slots during a slow pass. The shared coordinator still limits checks to four and preserves task, endpoint and principal exclusions. Authorization, result identity, retry limits, response selection and transaction ownership are unchanged. The regression checks idle quiet after all concrete obligations settle. ## Model Used OpenAI GPT-6.1 (`gpt-6.1-sol`), with reasoning, code execution and tool use. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 22:47:30 -06:00
DottaandClaude Opus 5.5	c45c7cb854	fix(workspaces): ignore killed-transfer scratch in sparse checkpoints (#19730) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run in a remote sandbox copies its workspace back to the host after the turn. A sparse checkpoint copies only changed entries, and the host validates each entry against the sandbox's own manifest > - Every Daytona transfer stages a `.paperclip-upload-<uuid>` scratch file at the sandbox workspace root and removes it only in the same process > - A control-plane kill during any transfer leaves that scratch behind. The sparse checkpoint then recorded it as a workspace file, but providers never export it. Every later copyback and every saved-work recovery attempt failed with "Workspace checkpoint payload mismatch" > - The failed recovery then left an accepted Retry showing "Paperclip is finishing the saved result" with no run > - This pull request stops recording transfer scratch, makes the checkpoint export an exact copy, names the failing entry, and gives a Retry behind a spent recovery an honest reason and a working action > - The benefit is that a restart during a transfer no longer makes a task's saved work permanently unrecoverable ## Linked Issues or Issue Description Refs #19721 (RC-50: the killed copyback owner; this PR covers the separate RC-50b cause) **What happened?** Restart-chaos waves s12/t01 and s14/t05 (Claude on Daytona). After a SIGKILL of the control plane, the native result was accepted, and then workspace copyback failed with "Workspace checkpoint payload mismatch". In s14 all 6 saved-work recovery attempts failed the same way. A Retry accepted during recovery kept `retryWait.reason = finalizing_saved_result` after the recovery budget was spent. No run was created. **Steps to reproduce** 1. Start a native Claude run in a Daytona sandbox. 2. SIGKILL the control plane while any provider transfer is in flight. 3. Let the run finish its turn and copy its workspace back. **Expected behavior** Copyback succeeds. Provider scratch is not workspace content. A Retry behind a spent recovery names the failure and can start a new recovery. **Actual behavior** The s14 sandbox (6f4a0b6a) still held `/home/daytona/paperclip-workspace/.paperclip-upload-d99fe36b-5acc-437b-96ad-85d5ff9e79c4.tar`. Its mtime was 03:29:23, the second of the K4 SIGKILL. The Daytona provider stages transfer scratch at `remoteDir`, which is the lease `remoteCwd` (the workspace root). The runtime workspace wipe deliberately preserves `.paperclip-upload-*`. The checkpoint helper (`workspace-checkpoint.ts`, `checkpointMain`) recorded the stale tar. The ordinary export excludes `.paperclip-upload*`, so the host received nothing for that entry. `validateCheckpointPayload` then threw "payload mismatch" on every attempt. The file-mode hypothesis was disproved: the sandbox umask is 022 and the scratch is mode 644. ## What Changed - `packages/adapter-utils/src/exclude-patterns.ts`: new `isSandboxTransferScratchPath`, which matches a root-level `.paperclip-upload-*` entry. This agrees with the wipe's preserve rule in `buildWorkspaceTarExtractCommand`. - `packages/adapter-utils/src/workspace-checkpoint.ts`: the sandbox checkpoint helper skips that scratch. A mismatch error now names the entry and its recorded, baseline and received forms (kind, octal mode, short hash; never contents). - `packages/adapter-utils/src/sandbox-managed-runtime.ts`: the checkpoint export requests `preserveSourceEntries: true`. The checkpoint output is an exact copy that the host validates entry by entry. An ordinary export applies the host umask and drops `._*` and `.paperclip-upload*` names at any depth. That gives the same permanent mismatch for a legitimate user file such as `._summary`, `notes/.paperclip-upload-plan.md`, or a `0775` script. - `server/src/services/explicit-native-continuation.ts` and `task-interruption-presentation.ts`: when the run's saved-work recovery is exhausted, a waiting Retry reports `saved_work_recovery_failed` with Retry available, instead of `finalizing_saved_result`. The existing task Retry path starts one new recovery generation. - `doc/execution-semantics.md`: a short additive note. ## Verification - `packages/adapter-utils/src/workspace-checkpoint-capture.test.ts` uses the real checkpoint helper and an export that behaves like Daytona's ordinary export: - Before the fix: `Workspace checkpoint payload mismatch at ".paperclip-upload-d99fe36b-….tar": sandbox recorded file mode=644 …; baseline absent; received absent`. That is the wave error. - After the fix: the scratch is not recorded, and validation passes. - A second case shows that reserved-looking user names and `0775` modes need the exact export. - `packages/adapter-utils/src/workspace-checkpoint.test.ts` is an end-to-end `prepareSandboxManagedRuntime` restore through a provider model. In that model, ordinary exports drop reserved names and apply the umask, and preserve exports copy exactly. Two red checks: - Without `preserveSourceEntries`, it fails with the mismatch on `._summary`. - Without the scratch skip, the stale tar leaks into the host workspace. - Green: the agent file, the `._summary` file and the `0775` mode arrive, and no scratch arrives. - `server/src/services/workspace-recovery-preparation-gates.test.ts`, "names a spent saved-work recovery behind an accepted Retry, and the next Retry finishes it" (s14 shape): - Red: `finalizing_saved_result` after 6 failed attempts. - Green: `saved_work_recovery_failed` with `canRetry: true`. The next Retry starts a new generation, recovery completes, and the Retry continues in exactly one new run. - Suites: adapter-utils checkpoint, capture, sandbox-managed-runtime, seed-reuse and managed-seed (168 passed). Server preparation-gates (47), explicit-native-continuation and task-interruption-presentation (455), workspace-recovery admission and retry-copyback, and native-workspace-finalizer. `tsc --noEmit` is clean for adapter-utils and server. - Live evidence: I started the stopped s14 sandbox read-only, listed its workspace, and stopped it again. No Daytona replay was run. The runner content ID of this checkout matches no built image. ## Risks - A user file named `.paperclip-upload-*` directly at the workspace root is no longer captured by a sparse checkpoint. Providers already drop that name from every ordinary export, and the runtime already reserves it at the root. - The checkpoint export now uses the provider's exact-copy mode. A workspace file with setuid, setgid or sticky bits is refused by that mode, as it is for the existing exact-copy transfers. The ordinary export would have stripped those bits and failed validation anyway. - The stale scratch tar stays in the sandbox. It is ignored, not deleted, because a concurrent transfer can own a file with the same prefix. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres, a read-only Daytona sandbox listing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:46:16 -06:00
Nicky Leach	b21884bdcf	Release and resume retained plugin Runner sessions in Core (#19611)	2026-11-06 20:45:02 -08:00
DottaandPaperclip	9efee7ee16	fix(runtime): save dynamic skill bundles before native admission (#19729) ... ## Thinking Path > - Paperclip manages agents, tasks, and their saved execution state. > - A replacement server must restore each admitted skill from shared storage. > - Installed company skills already have immutable versions. > - Connector skills add assigned resources and tool definitions for each run. > - Those dynamic bytes keep a null version ID and differ from the static skill. > - Native admission saved their input before it saved a durable package receipt. > - This change stages the exact packages first and publishes the input and receipt together. > - A new connector turn can then survive an empty server home. ## Linked Issues or Issue Description Refs #19675 for the existing runtime asset preservation owner. **What happened?** A new native run with an assigned Slack connector creates a concrete local skill bundle after the old runtime assets were migrated. Admission saves its execution input without a preservation receipt. The new run creates a local durability dependency again. **Expected behavior** With S3 configured, the exact dynamic bundle must be durable before the execution input becomes visible. Its null version ID and admitted bytes must remain unchanged. Local storage must keep its volume-backed behavior. **Steps to reproduce** 1. Configure S3 and a native agent with an assigned Slack resource. 2. Prepare a new native task through the normal runtime selection owner. 3. Inspect its runtime skill receipt or restore its skills with an empty server home. The previous owner saved no receipt. **Paperclip version or commit** Reproduced on master `d1ee25d0f7`. The fix is tested on `4a3396d9b9`. **Deployment mode** S3-configured server. Local tests use PostgreSQL 18.1 and a private disk transport with the S3 receipt contract. ## What Changed - Extend the existing runtime asset owner with package staging and transaction-bound receipt validation. - Stage exact unversioned skill packages before native admission. Keep storage I/O outside the run transaction. - Publish the frozen input and receipt together under the existing run, cancellation, and package collection locks. - Reject changed input bindings and retired package metadata. - Reuse valid same-run receipts without reading the previous host's bundle path or requiring a new object read. - Keep existing online preservation and local storage behavior. - Add a bounded preparation span and document its work. ## Verification - Genuine unchanged-production RED: the actual Slack overlay and real admission transaction save no preservation receipt. - Final rebased head: 68 tests pass in three suites in 25.36 seconds. These include all runtime-selection, runtime-preservation, and connector controls. - The new positive checks that no execution input or coordinator is visible during package readback. It restores exact skill and tool bytes, modes, digests, and null version IDs in a cold home. - Guards cover an uncertain storage failure, cancellation during upload, a changed binding, retired package metadata, and local mode. An existing receipt remains usable after removing the original bundle and during a storage outage. - The canonical PostgreSQL wrapper uses one owned disposable PostgreSQL 18.1 container. Each suite's database is cleaned up. Real package archives, streams, and readback run locally; no external provider or model is called. - Strict server TypeScript, module boundaries, changed-line private-data checks, and `git diff --check` pass. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/42337473062) passes all required build, typecheck, test, and Runner gates. Fresh Greptile is 5/5 with no open review threads. ## Risks New S3-backed admission now requires its dynamic packages to be saved before execution can proceed. A storage failure can therefore stop preparation. Existing resource cleanup keeps its authority. Failed or cancelled staging retains original files and the package owner's pending receipts. Same-run recovery retains its original capture provenance and cache behavior. The tests replace the external S3 transport; they do not qualify live S3 permissions or provider execution. No schema, API, workflow, company skill version, or provider protocol changes are included. ## Model Used OpenAI Codex (GPT-6 family; the exact runtime model ID and context capacity are not exposed in this session), with reasoning, repository editing, code execution, and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 22:34:44 -06:00
DottaandClaude Opus 5.5	db41bfcb38	fix(recovery): let an accepted Retry finish after a killed copyback owner (#19721) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run saves the agent's workspace files after the provider turn (copyback). One controller owns copyback at a time and records a durable owner receipt on the run (`nativeWorkspaceFinalizationOwner`) > - When copyback of an accepted result fails, retained saved-work recovery exports the files from the kept sandbox. A task Retry waits for that recovery before it continues > - A control-plane restart can kill the controller while it owns copyback. The dead controller can never confirm its own stop, and nothing in the retained recovery path could release its receipt > - The recovery worker then spent one attempt on each recheck of that receipt until it was exhausted, and the accepted Retry kept saying "Paperclip is finishing the saved result" with no run. The task waited forever > - This pull request stops spending attempts on that recheck, names the real wait, and lets the board release the exact receipt so the same recovery and the same Retry continue > - The benefit is that an accepted Retry never waits silently: it finishes, or it shows one honest action that makes it finish ## Linked Issues or Issue Description Refs #19717 (same-host automatic proof for a killed copyback owner; this PR covers the retained-recovery path and the case where no automatic proof is possible) Refs #19713, #19609, #19694 (related restart-recovery fixes; no overlap in files except this doc) **What happened?** Two Daytona restart-chaos waves. In each, a native run's copyback failed and the run failed with `native_workspace_sync_out_retry_exhausted`. The agent's result was accepted. Retained recovery started. A SIGKILL of the control plane landed while a recovery attempt owned copyback: - Wave s13, task t06 (Claude, crash loop): Retry was accepted at 02:30:35. A SIGKILL landed 2 s later. All later attempts failed with `copyback_stop_unconfirmed`. After 6 attempts the recovery was exhausted. The Retry showed `retryWait.reason = finalizing_saved_result` for more than 15 minutes. No run was created. - Wave s12, task t01 (Claude): attempt 1 failed with "Workspace checkpoint payload mismatch". The SIGKILL at 01:43:54 killed the owner of attempt 2. Attempts 3 to 5 failed with `copyback_stop_unconfirmed`. The Retry was accepted with HTTP 2xx and produced no run. In both kept databases the run carries an owner receipt for the killed process. **Steps to reproduce** 1. Accept a native result and make its workspace copyback fail, so that retained recovery owns the export. 2. Press Retry on the task. 3. Kill the control plane while a recovery attempt owns copyback, so that its owner receipt stays on the run. Use another host, or a receipt without automatic stop proof. 4. Let recovery run. **Expected behavior** Rechecking the receipt does not spend attempts. The task names the wait and does not offer a Retry that cannot help. A board member can confirm that the previous server stopped. The recovery then finishes, and the waiting Retry continues by itself. **Actual behavior** `withNativeWorkspaceFinalizationOwnership` returned `owner_unverified` for a retained repair without creating any board action (`native-workspace-finalization-ownership.ts`, `requireStopVerification`). `recoverRetainedWorkspace` reserved an attempt for each recheck and failed it as `copyback_stop_unconfirmed` (`workspace-recovery.ts`), until the budget was exhausted. The Retry gate in `explicit-native-continuation.ts` returned `finalizing_saved_result` for an accepted result with no verified repair, with no check for this hold. `resumeNativeWorkspaceAfterOwnerStop` accepted only an `owner_unverified` action on a `running` run, so no route could release the receipt of a finished run. The exhausted handoff told the board to press Retry again, which repeats the same failure. ## What Changed - `server/src/services/workspace-recovery.ts`: when retained recovery finds an unverifiable copyback owner, it records the recheck without reserving an attempt. The 30-minute recovery window still bounds it. The retained action records the exact receipt as `workspaceOwnerRecovery: { state: "operator_required", owner }`. A newer receipt replaces the record and clears an older confirmation. - `server/src/services/explicit-native-continuation.ts`: a Retry that waits for an accepted saved result reports `workspace_owner_stop_required` when the run's active retained action has that hold. It no longer reports `finalizing_saved_result` for this case. - `server/src/services/task-interruption-presentation.ts`: the new wait has a fixed message, and Retry is not offered for it, because another click cannot release the receipt. - `server/src/services/native-runtime/native-workspace-owner-recovery.ts`: the existing board route `POST /issues/:id/recovery-actions/resume-workspace-finalization` (permission `runtime:confirm_owner_stop`) also accepts a retained recovery action with that hold. The confirmation must name the receipt that is still on the run. A live owner on this host still rejects it. It clears only that receipt, records the stop proof and an activity entry, and returns the same recovery to the worker. A spent budget gets one new generation, as a deliberate recovery request does. It accepts no result and starts no agent turn. The owner check is shared with the existing path. - `server/src/services/system-recovery.ts`: an exhausted `copyback_stop_unconfirmed` handoff names the board confirmation instead of another Retry. - `ui/src/api/issues.ts`: the route receipt's `resultId` can be `null` (a retained recovery has no live result binding). - `doc/native-workspace-finalization-recovery.md`: an additive paragraph describes this path. ## Verification - Red then green on embedded Postgres: `server/src/services/workspace-recovery-preparation-gates.test.ts`, "lets an accepted Retry finish after a killed copyback owner instead of waiting forever" with one case per wave (s13, s12). Each case uses a real accepted remote export, a Retry through the HTTP route, a killed-owner receipt, and 8 recovery passes. - Before the fix (both cases): `attempts: 6, state: "exhausted"`. That is the wave state. - After the fix: no attempt is spent on the recheck. The hold is recorded. The Retry wait is `workspace_owner_stop_required` with `canRetry: false`. The board confirmation through the route returns 202 and clears the receipt. The same recovery completes. The same accepted Retry then finishes the saved result (s13: the task is `done`) or continues in exactly one new run (s12). - `npx vitest run src/services/workspace-recovery-preparation-gates.test.ts -t "killed copyback owner"`: 2 passed. - `pnpm verify` (affected selection): see the comment below. Server `tsc --noEmit` is clean. - No live Daytona replay. Both waves ran on old signed images. The local proof uses the same accepted-export fixture as the existing Retry tests, with the receipt state copied from the kept databases. ## Risks - A copyback owner on another host that is still alive is now rechecked within the recovery window without spending attempts. Before this change it spent them. The window still bounds the wait. - The board confirmation for a retained action is an operator statement, the same as the existing owner-stop path. It is limited to roles with `runtime:confirm_owner_stop`. It never clears a receipt other than the one it names, and a live same-host owner still rejects it. - #19717 changes the same ownership area and adds a UI action for the running-run case. That UI does not show a retained action yet. After both merge, a small follow-up can show its button for this hold. Until then the API route and the task's wait message cover it. - Not addressed here: the "Workspace checkpoint payload mismatch" that started wave s12. The checkpoint outbound transfer extracts without `-p` (`sandbox-managed-runtime.ts` checkpoint sync-out, `extractHostTarball` in the Daytona provider), so the host umask can change file modes that the remote manifest recorded. That is a separate signature. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test runs, embedded Postgres). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:07:00 -06:00
DottaandClaude Opus 5.5	5cc89b1e4d	fix(native): do not charge a restart resume that made progress before the next crash (#19720) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs have an execution recovery budget of three provider attempts, so a run that keeps failing stops retrying > - A control-plane crash (no graceful drain) makes the next server resume the run, and that resume charges one attempt > - #19625 stopped charging a resume after a graceful drain, but crash restarts still charge, even when the previous resume did real work > - On a deploy day with several restarts, a healthy run used up its budget and the task needed a manual Retry > - This pull request does not charge a crash resume when the previous restart resume made recorded progress > - The benefit is that restarts alone do not exhaust a run that keeps making progress, and genuine loops stay bounded ## Linked Issues or Issue Description Refs #19625 **What happened?** In restart-chaos testing (storm wave, OpenCode run), a run crossed four control-plane restarts: K1, K1, K2 (drain), K1. Each resume was clean: the runner restored the session, recorded `provider_turn_lost_on_restore`, and the #19699 continuation turn then ran tools for minutes. The two K1 resumes each charged one attempt (attempt 1 to 3). The drain resume did not charge (#19625). The fourth restart found attempt 3 and blocked the run with `native_restart_recovery_blocked` / `execution_recovery_budget_exhausted`. Seven runs in the same wave failed in the same way. In every case, each resume had accepted a new turn and completed tool work before the next restart. **Expected behavior** A run must not fail because it survived several controller restarts that each resumed cleanly and made progress. The budget must still stop a resume that fails each time, a crash loop that makes no progress, and a provider that fails repeatedly. **Steps to reproduce** Start a native local run. Kill the server (SIGKILL) while the run is active and let it resume. Let the resumed turn run tools. Repeat three times. The third crash resume is blocked with `execution_recovery_budget_exhausted`. **Paperclip version or commit** `master` at `8a34c1b62c`. ## What Changed - `claimNativeRestartRecoveries()` (`native-restart-recovery.ts`) records `progressWatermarkSeq` (the run's highest `heartbeat_run_events.seq`) in the history entry of each recovery claim. - New `previousResumeMadeProgress()`: the next claim reads the watermark of the most recent claim. It finds progress when, after that watermark, a `turn.accepted` event names a turn, and a later `tool.execution.completed` or `item.completed` event (not a `usage` item) belongs to the same turn. The evidence is committed run events, not timing. - When that progress exists, the claim is not charged (`chargesProviderAttempt: false`, and the `>= 3` exhausted gate is skipped). This uses the same path as the clean-drain rule from #19625. The history entry records `attemptKeptForProgress: true`. - Restore activity does not count: lineage items without a turn, `turn.failed`, work on the interrupted turn, `turn.submitted`, and an accepted turn with only usage or a failure. So a resume that dies or fails before it does work still charges. - The credit stops at controller generation 12 (`NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT`). A run that does some work and then crashes the controller every time still reaches the budget. - The first restart (which interrupts the original execution, not a resume) still charges. A history entry from an older release has no watermark, so it gets no credit. - `doc/execution-semantics.md` states the rule next to the three-attempt budget. ## Verification New tests in `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`; between claims the test applies the executor's attempt charge and marks the controller dead): - `keeps recovering a run whose restart resumes each completed work before the next crash`: five crash resumes, each followed by restore noise and one completed tool in the continuation turn. Charges are `[true, false, false, false, false]` and the attempt stays 2. Red on master (attempt 2, 3, then blocked). - `still exhausts the budget when restart resumes complete no work between crashes`: resumes followed only by restore noise and a failed continuation turn. Attempt 2, attempt 3, then `execution_recovery_budget_exhausted`. Green on master and with the fix. - `bounds a crash loop that makes progress before every crash`: credit stops at the generation limit, the last attempt is spent, and the run blocks. Red on master. - `native-runner-restart-recovery.integration.test.ts`, `heartbeat-native-restart-suspension.test.ts`, `native-restart-recovery-isolation.test.ts` and `native-remote-retry-admission.test.ts`: 218 passed, 14 skipped. - I ran the progress query against the live database of the failing wave. For all seven blocked runs, every resume had progress before the next restart, so this rule would have kept each of them recovering. - `tsc --noEmit` for `server` has no errors in the changed files. ## Risks - A run that makes progress and then makes the controller crash each time gets up to about ten extra resumes before the budget applies. The generation limit bounds this. - The rule depends on the PRP event shape (`payload.prpEvent.turnId` on `turn.accepted` and on completed work). Events without that shape get no credit, so the failure mode is the old behavior (charge), not an unbounded loop. - The query runs once per claim, on the `(run_id, seq)` unique index range after the watermark. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 22:04:19 -06:00
DottaandPaperclip	1a60d47804	Preserve settled native log files during online migration (#19719) ... ## Thinking Path > - Paperclip manages work for AI agents. > - The web server must retain run history across replacements. > - Older native runs can have local log files with no final size or digest. > - The normal importer correctly refuses to call those files complete. > - The existing raw archive mode requires the entire instance to stop work. > - This change gives settled sources a separate, guarded online archive path. > - Active work can continue while old evidence moves to object storage. ## Linked Issues or Issue Description Refs #18864, #19678, #19688. The offline historical log archive path cannot migrate a busy instance. This change preserves exact surviving bytes without claiming complete native-event coverage or changing the original run outcome. ## What Changed - Add an explicit online option for settled native log files. Keep the default importer and offline option unchanged. - Require a released, closed native finalizer and no pending attempt, allocation, physical owner, or task binding. A stale lease or terminal run status alone is insufficient. - Use the canonical physical finalization lock and row locks. Check the source generation before claims and publication. - Read only the canonical, private local file. Reject aliases, symlinks, hard links, changed files, and changed owners. - Verify remote bytes before publishing a read-only archive and activity receipt in one transaction. Keep original files, errors, run status, event rows, and completeness metadata. - Document the command and its authority limits. Add real PostgreSQL and loopback S3 transport tests. ## Verification - The original implementation failed the new archive regression. It returned blocked instead of migrating the settled source. - Final source integration suite: 15 tests pass on disposable PostgreSQL and the bounded loopback S3 fixture. Coverage includes active unrelated work, failed and successful sources, a 4 MiB segment boundary, owner changes, file changes, unresolved owners, task bindings, allocations, and filesystem aliases. - The qualified source TypeScript check and module boundary check pass. - All 73 tests in five affected existing log-store, object-journal, cancellation, and migration-worker suites pass. The first borrowed dependency setup loaded work contexts from another worktree and failed four tests. Explicit database subpath aliases resolved that setup fault; no worker code changed. - Earlier test fault injection used the wrong storage method, and the first segment test reused a closed fixture endpoint. Those fixture defects were corrected before the final 15-test pass. - Local tests use Node 26. CI supplies the standard full typecheck, build, and test gates. Production object-storage credentials, live Railway replacement, and provider continuation are separate deployment acceptance steps. - Final exact-head CI: 59 successful contexts and 3 skipped, with all typecheck, build, test, E2E, security, and review gates complete. [Workflow evidence](https://github.com/paperclipai/paperclip/actions/runs/42332349644). - The first CI attempt had 12 jobs interrupted by runner shutdown. Their logs end in cancellation, with no test assertion found. The failed/cancelled-only retry passed all 12 jobs and recomputed aggregates. The 38 original successful job records kept their original execution times and were not rerun. Original interruption logs remain retained. ## Risks - This path preserves raw history. It cannot recover missing source bytes or prove a complete native transcript. - Uploads can finish before a later source check fails. The run reference stays local in that case; a retry must still satisfy all authority checks. - The new option is explicit. It adds no startup worker, recurring poll, or background scan. - Source transactions hold scoped locks only for ownership checks and publication. Streaming and remote verification run outside those transactions. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use, and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 21:37:18 -06:00
DottaandClaude Opus 5.5	aed053633e	fix(environments): run pending sandbox cleanup when an environment is deleted (#19716) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Operators delete a sandbox environment, optionally with consent to destroy its sandboxes (`DELETE /api/environments/:id?destroyReusableSandboxLeases=true`) > - A sandbox whose teardown failed or was interrupted is kept as a `pending_cleanup` lease, and it blocks the delete > - The cleanup sweep retries such a lease only after a cooldown of up to 5 minutes > - A consented delete therefore returned 409 `pending_sandbox_cleanup` for minutes. A caller with a shorter deadline gave up, and the environment and its sandbox stayed behind > - This pull request makes the consented delete run one teardown attempt for the environment's pending cleanups now, then re-check > - The benefit is that a consented environment delete completes as soon as the provider teardown succeeds ## Linked Issues or Issue Description **What happened?** After a restart chaos wave, the harness deleted the wave's Daytona environment with `destroyReusableSandboxLeases=true`. The server returned 409 `pending_sandbox_cleanup` every 3 seconds for the whole 120-second client deadline. A run lease had entered `pending_cleanup` after its run failed, and the sweep's retry cooldown had not passed. The environment was not deleted. **Expected behavior** With explicit consent, the delete tears down the environment's own pending sandbox cleanups and completes when that teardown succeeds. A teardown that fails, or one already in flight elsewhere, still blocks the delete. **Steps to reproduce** 1. Leave a sandbox lease of an environment in `pending_cleanup` with a future retry time. 2. `DELETE /api/environments/:id?destroyReusableSandboxLeases=true`. 3. On master, the response is 409 `pending_sandbox_cleanup` until the cooldown ends. Refs #19696 (restart recovery for creation and teardown claims; this builds on it). ## What Changed - `heartbeat/recovery.ts`: `sweepPendingCleanupLeases` accepts `environmentRetry: { environmentId, actorId }`. Like the existing run-scoped explicit retry, it bypasses the cooldown for one attempt, never takes over an in-flight attempt, and logs `environment_lease.cleanup_retried` with reason `environment_deleted`. It touches only that environment's leases. - `routes/environments.ts`: when the delete has consent and the only blockers are pending cleanup and reusable leases, it runs that retry first, then re-reads the blast radius. The existing reusable-lease destroy and the final guard are unchanged. The new dependency is a narrow, optional `retryPendingEnvironmentCleanup` option; without it, behavior is unchanged. - `app.ts`: wires the option to the existing heartbeat service instance. ## Verification - Red, then green: - `environment-routes.test.ts`: "runs the environment's pending sandbox cleanup now and deletes with explicit consent" and "keeps rejecting when the consented pending cleanup attempt does not finish" fail before and pass after. "does not run pending sandbox cleanup without consent or past another blocker" passes in both. - `heartbeat-pending-cleanup-sweep.test.ts`: "runs one attempt for a deleted environment's cleanup during its cooldown, scoped to that environment" fails before and passes after. It checks that another environment's lease and an in-flight claim are untouched, and that the activity log is written. - `sandbox-creation-operations.test.ts`: "reclaims an issue-bound creation after its run failed and the issue finished on a later run" covers another chaos finding. A creation intent bound to an issue was interrupted by SIGKILL 0.5 s after it committed, its run ended `process_lost`, and a later run finished the issue. The test fails on the code before #19696 and passes on master, so #19696 already covers this case. The test is kept as a regression. - After rebasing on master, `environment-routes`, `heartbeat-pending-cleanup-sweep`, `agent-runner-qualification-routes` and `sandbox-creation-operations` pass (192 tests). `tsc --noEmit` reports no errors in the changed files. ## Risks - A consented delete now makes provider teardown calls for pending cleanups. Those sandboxes were already queued for destruction, and the operator consented to destroy the environment's sandboxes. - One explicit attempt counts toward the lease's attempt cap, the same as a user Retry. - The delete request now waits for those teardown attempts. Each attempt uses the existing plugin timeouts. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:28:54 -06:00
DottaandClaude Opus 5.5	51eb4ff705	fix(recovery): finalize an accepted native result after a hard kill (#19713) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run hands its accepted result to a controller lease, and the controller finalizes the task from that result > - A graceful shutdown releases that lease for the next server (#19591), but a hard kill (SIGKILL, OOM, host crash) cannot > - After a hard kill between result acceptance and finalization, the next server waited for the dead controller's 20-minute lease, so the run stayed `running` and the user's follow-up comment waited with it > - This pull request releases that lease at startup when the controller is proved dead by its boot ID and process identity, never by age > - The benefit is that the saved result finalizes within seconds after a restart, without another provider turn, and queued follow-up work continues ## Linked Issues or Issue Description Refs #19591 (the graceful-shutdown counterpart) and #19578 (boot-identity proof for legacy runs). **What happened?** A restart-chaos wave sent K1 (SIGKILL) 20 ms after a Claude run recorded `run.result.accepted` and `run.terminal`. The run then stayed `running` for more than 10 minutes, through four more restarts. The finalization row was `workspace_finalizing` with the stored result. Its lease belonged to the killed server: a different `controller_boot_id`, a `controller_pid` with no process, and `lease_expires_at` 20 minutes ahead. Startup recovery skips runs that already have a result. The result-commit worker waits until the lease expires. The task's follow-up comment could not run until then. **Steps to reproduce** 1. Let a native local run accept its result. 2. SIGKILL the server before finalization finishes. 3. Restart the server. The run stays `running` until the old lease expires. **Expected behavior** After a restart, a result lease held by a stopped controller of an earlier boot is treated as released at once. The stored result finalizes once, with no new provider turn, and the follow-up wake runs after it. ## What Changed - New `server/src/services/native-runtime/native-dead-controller-result-lease.ts`: `releaseDeadControllerResultLeases` finds `running` native runs with an accepted result and an unexpired lease. It requires the lease to belong to a different controller boot. It proves the controller stopped with `evaluateNativeControllerTakeover` (`requireStoppedController`): the process is dead, or its PID now names another process. It then expires that lease with a compare-and-set on the exact owner token, boot ID, and PID, and notifies the result-commit and finalization workers. - `server/src/services/heartbeat/recovery.ts`: `recoverNativeRunsAfterRestart` calls this first and records a `native.recovery.transition` event (`awaiting_result_finalization`) for each released run. - Only executor leases are released, because only the executor claim writes the controller identity together with the lease. Finalizer (`native-finalizer:`) and cleanup (`native-cleanup:`) claims keep an earlier executor's identity, so they are never released this way and expire on their own short leases. A finalization-resume token (`:resume:`) on a result row always belongs to the executor that adopted it, because a pending resume claim is taken only on a row without a result. - Also not released: a live owner, an owner without a recorded process, a lease of the current boot, a run this process is executing, and remote workspaces (`nativeWorkspaceSync` or `remoteWorkspace`), which keep their fenced lease. - The compare-and-set does not compare the stored expiry. Postgres stores microseconds that a JavaScript `Date` drops, and an exact-expiry comparison silently skipped the real lease in the live reproduction. - `doc/execution-semantics.md`: documents the rule. ## Verification - Red then green on embedded Postgres (`server/src/__tests__/heartbeat-native-restart-suspension.test.ts`): - A hard kill with a dead controller's executor lease (first claim, restart claim, or resumed claim) now finalizes on the next startup: run `succeeded`, finalization `committed`, task `done`, one result, one run, no provider dispatch. These tests failed before the fix with `status: running`. - A lease expiry stored with microsecond precision is released. This test failed against the first version of the fix and reproduces the live miss. - A live controller of another boot keeps its lease, and so does a lease of the current boot. - A live claimant's finalizer or cleanup lease is kept when the recorded earlier executor is dead. - `npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 20 passed. - Related suites pass: `native-session-resumption`, `native-runner-restart-recovery.integration`, `native-run-projection-work`, `live-recovery-work`. One `live-recovery-work` legacy case timed out once under load and passed when run again by itself. - Live, local: `tests/restart-chaos/main.ts --workloads C1 --triggers result --faults K1 --rc06-masked`. The failing case above came from such a wave. I started its kept database and confirmed the stored lease state and the compare-and-set miss. Against that same database, the fixed release expires the stuck executor lease. Two later waves (12 tasks, `--triggers result`) passed every task. Run events were not kept for passing tasks, so I cannot confirm that a kill landed in the narrow pre-finalization window in those waves. - Live proof on head `8971874654`: local ACPX Claude, workload A1, seed 7411, `--triggers result` (a local variant that fires on A1 2.5–3.5 s after the run's result proposal), `--keep`. The run passed 4 of 4 tasks. In two tasks, K1 landed 1.5 s and 2.0 s after `run.result.accepted`. On the next startup, each run recorded "Server restart released the accepted native result of a stopped controller for finalization" (`controller_process_dead`). Each finished `succeeded` about 1 s later, with its task `done` and exactly one run per task (no extra provider turn). Before this fix, the same shape stayed `running` until the 10-minute budget ran out (seeds 7411, 7422, and 7423 from the RC-33b investigation in #19691). ## Risks - Startup could take a lease from a finalizer that is still running. Mitigations: the lease must belong to another boot, the controller process must be proved stopped on this host (the same proof the existing runner takeover uses), and the update matches the exact owner. Remote workspaces are excluded. Multi-host deployments keep the existing PID-liveness assumption that `claimNativeRestartRecoveries` already makes. - Low behavioral risk otherwise: finalization uses the existing reconciler path, which #19591 already exercises after a graceful drain. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:22:24 -06:00
DottaandClaude Opus 5.5	08f7ac03dc	fix(recovery): keep a never-started task waiting through a temporary accounting hold (#19702) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Admission checks every wake. A short company accounting hold can refuse new work after a control-plane restart. > - When the hold refused a new task's assignment wake, stranded-work recovery found the task with no run while the hold was still active. It moved the task to `blocked` with a board recovery comment. > - A `blocked` task leaves the stranded-assignment sweep, so nothing woke it after the hold cleared. The task stayed `blocked` with no run, no interruption and no Retry. > - The hold was temporary. Tasks created a few seconds later (after the sweep) started normally once accounting settled. > - This pull request keeps a never-started task `todo` while a temporary budget hold lasts. The existing assigned-todo liveness dispatch then admits it once the hold clears. The task shows the hold it waits for. > - The benefit is that a restart no longer silently drops new tasks created during the accounting settle window. ## Linked Issues or Issue Description **What happened?** In a restart-chaos storm on master `06bbcea12f`, three new tasks were created while the company accounting hold was active. Each assignment wake got 409 "Company cannot start work while completed runs await accounting." (or "...native run recovers its unfinished accounting."). One task rested `blocked` with 0 runs, interruption `null` and `canRetry=false`. The other two ran later. **Root cause** `reconcileStrandedAssignedIssues` (`server/src/services/recovery/service.ts`, the `readInvocationBudgetBlock` branch near line 4826) escalates any assigned task to `blocked` when a budget block exists. It does this before the assigned-todo no-run branch, which already skips a budget-blocked task and dispatches it later. So a task that never started was treated as failed recovery work. The refused wake itself is stored as a `skipped` `budget.blocked` diagnostic, which is correct; the escalation removed the only path that re-admits it. **Expected behavior** A wake refused by a temporary admission hold waits and is admitted once the hold clears, with one run. A real hard stop is still a board decision. Removed assignees and cancelled tasks still do not wake. **Steps to reproduce** Create a company with a hard-stop budget and a completed run that has `cost_accounting_pending = true`. Assign a new `todo` task. The wake gets 409. Run the stranded-assignment sweep. Before this change the task becomes `blocked` and never runs. Related: #19700 bounds the accounting hold itself. This PR is independent: it handles what happens to a task whose first wake the hold refused. ## What Changed - Stranded-work recovery skips a `todo` task with no run while a temporary budget hold blocks admission. It does not escalate it to `blocked`. The existing assigned-todo liveness dispatch admits the task once the hold clears. - A hard stop (spend at the limit, or blocked unpriced usage) keeps the existing board escalation. Recovery checks this with the read-only `policyBlocks` for the policies of the blocking scope. It does not change budget hold semantics. - The task interruption projection now covers a `todo` task with no run whose latest wake was refused with `budget.blocked`. It shows "Waiting for the budget to allow new work. The task starts automatically when it does." with no Retry. A budget pause shows the existing "budget" user action. - The no-run projection applies the same person-owned gates as the run-backed projection: a budget pause shows the "budget" action, and an ordinary agent, project, company or task pause shows "resume". Only an accounting hold shows the automatic-start text. - `ExecutionBlockerNotice` shows that wait text for a `working` presentation that carries a wait reason. - `doc/execution-semantics.md` records the rule. ## Verification - New embedded-Postgres regression `server/src/__tests__/heartbeat-refused-assignment-wake.test.ts`: - hold active, new task assigned, wake refused (409, `skipped` `budget.blocked`), two recovery sweeps: the task stays `todo` with no run and shows the wait reason. Before the fix this failed with `expected 'blocked' to be 'todo'` (the CHA-9 shape). - hold cleared, two sweeps: exactly one provider turn starts, from `issue.assigned_todo_liveness_dispatch`, and the task finishes. A later sweep does not start another turn. - counter-tests: a terminated assignee and a cancelled task get no run after the hold clears. - Existing `creates a board recovery action for budget-blocked assigned work` (real hard stop) still passes unchanged. - `pnpm exec vitest run src/__tests__/heartbeat-refused-assignment-wake.test.ts src/__tests__/heartbeat-process-recovery.test.ts src/services/task-interruption-presentation.test.ts src/__tests__/heartbeat-issue-liveness-escalation.test.ts` (server) and `src/components/ExecutionBlockerNotice.test.tsx` (ui). - Server typecheck and `pnpm check:token-gates`. - Live restart-chaos storm (local, codex/claude/opencode, 6 in flight, K2 every 3-4 minutes for 20 minutes, no RC-06 mask), revision e38bb5c8f7: 17 tasks, 10 pass. - Two new tasks (CHA-16, CHA-17) were created while the company accounting hold was active. Their assignment wakes got 409 "Company cannot start work while completed runs await accounting." Both stayed `todo` (not `blocked`) and showed `state: working`, `canRetry: false`, `retryWait.reason: budget.blocked`, "Waiting for the budget to allow new work. The task starts automatically when it does." No task rested `blocked` with 0 runs. - The hold itself never cleared in this wave (`pendingRunCount: 1` at wave end), so these two tasks did not start before their deadline. That stuck hold is the separate accounting-closure defect that #19700 bounds. The other 5 failures (`R3:retry_no_new_run:B5`) are Retries refused by the same stuck hold. The re-admission after the hold clears is proved by the embedded-Postgres regression above. ## Risks - Low. The change narrows one escalation to tasks that never started, under a budget block that is not a hard stop. Those tasks now stay `todo` and are re-checked by the existing 30-second stranded-assignment sweep, which already re-checks every admission gate. - The hard-stop check reads the blocking scope's active policies once per budget-blocked never-started task in a sweep. It runs only while a hold is active. - The interruption projection adds one indexed read (company + payload issue id) for `todo` tasks in the batch. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:12:00 -06:00
DottaandPaperclip	d1306f338b	fix(runtime): preserve settled failed harness backups online (#19715) ... ## Thinking Path > - Paperclip coordinates agents, tasks, and their saved execution state. > - A replacement server needs native provider backups in shared storage. > - The online pre-copy owner accepts successful runs with a committed result. > - A sealed backup from a failed or cancelled run can still contain user history. > - The existing owner defers those files even after their source ownership is settled. > - This change adds a separate preservation owner with exact source and publication checks. > - The copy retains the failed outcome and original files while making the backup readable from a fresh home. ## Linked Issues or Issue Description Refs #19137 and #19108 for the existing pre-copy and encrypted backup owners. **What happened?** Online pre-copy reports `source_not_successful` for a sealed legacy backup from a settled failed or cancelled run. The provider files remain tied to the old disk. **Expected behavior** Copy the exact recorded backup when the source has no remaining writer ownership. Preserve the run status, warnings, result, allocation, selected session, and local files. **Steps to reproduce** 1. Create a lease-stamped suspended native backup. 2. Settle its source as a failed run with a terminal-failure coordinator and no owner. 3. Keep an unrelated execution active. 4. Run canonical online pre-copy. The previous owner defers the source. ## What Changed - Add a narrow preservation owner for terminal source files. It checks the physical finalization lock, task, coordinator, run, selected session, source lease, and source fingerprints. - Recheck that owner before the encrypted upload receipt and current-backup publication. - Reuse the finite workspace keepalive during long source checks and defer source-local invalid archives. - Add optional exact-root stamp verification. A valid previous backup cannot authorize a changed current payload. Existing callers keep their fallback behavior. - Bind the archive input to the checked owner and record an activity event after publication. - Keep pending object receipts after an uncertain or failed publication. - Add focused PostgreSQL and filesystem controls. Update the existing migration runbook. ## Verification - Real PostgreSQL baseline on unchanged production: the new positive case fails with `source_not_successful` instead of publishing the backup. - Same case passes after the change and decrypts/restores the exact provider history. - Final head 4b6cf093be: three focused suites pass, 81 tests in 62.26 seconds, including existing storage and stamp consumers. The canonical PostgreSQL wrapper exited zero and completed cleanup. - Controls cover cancelled and failed sources, real committed results, expired nonnull owners, physical owner/lock contention, late source/owner changes, previous-tree fallback, and archive root/scope binding. - Both review regressions fail with the committed pre-fix owners: a real 30-second database idle timeout expires during a 32-second exact-source check, and a stamp-valid invalid archive stops the pass. The fixed owner keeps its transaction live and copies the next valid source. Source hashes and automatic file restoration were recorded for each leg. - Final-head strict server TypeScript and module-boundary checks pass after the master rebase and required Runner/shared declaration rebuild. - No source migration, provider mutation, or archive apply was executed. ## Risks This path does not reconstruct missing backups or prove that missing provider state is recoverable. Such sources remain blockers. It does not take expired owners or report failed work as successful. Source hashing adds bounded disk work while source row locks are held; unrelated executions remain active in the PostgreSQL fixture. Existing archive size limits, authenticated encryption, object readback, and publication fencing remain in place. No schema, API route, scheduler, or rollout-policy change is included. ## Model Used OpenAI Codex (GPT-6 family; exact runtime model ID and context capacity are not exposed in this session), with reasoning, repository editing, code execution, and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 21:08:27 -06:00
DottaandClaude Opus 5.5	8a34c1b62c	fix(budgets): bound the accounting hold so one missing receipt cannot wedge a company (#19700) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 21:02:01 -06:00
DottaandClaude Opus 5.5	4612fa9fb4	fix(native): restart a drained Daytona run whose runner opened its session (#19711) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. A graceful shutdown (every rolling deploy) suspends live native runs "without cancelling provider work". The next boot then decides how each run continues. > - #19583, #19616 and #19684 let restart recovery start a remote bootstrap again when durable evidence proves that no provider work exists. > - A chaos replay found another window. The drain suspended a remote run just after its sandbox runner opened the provider session, before the controller sent a turn and before a session snapshot existed. > - Recovery accepted only an unfinished `session.open` in the controller checkpoint, so it blocked the task as ambiguous. A Retry recovered it, but automatic continuation was lost. > - A related replay showed that the same-run bootstrap reuses the sandbox. The interrupted runner can still hold the runner ingress port there (`runner_ingress_bind_conflict`). > - This pull request proves the "session opened, no turn" case and stops the interrupted runner before relaunch. > - The benefit is that a deploy during Daytona startup continues automatically, without a port conflict. ## Linked Issues or Issue Description **What happened?** A chaos replay (ACPX Claude on Daytona) sent SIGTERM while a remote run was starting. The drain recorded `restart_suspended`. The dying controller still launched the runner and finished `session.open`. On the next boot, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The decrypted controller checkpoint of that run showed: - `run.prepare` completed and `session.open` completed (provider session opened). - Committed events `session.started`, `session.capabilities.updated` and `session.goal.snapshot` (`ackedSourceSeq: 3`). - No turn command. No run-profile session snapshot and no projected provider events. On the next run in that replay, a same-run bootstrap relaunched the runner in the same sandbox (one lease for all attempts). It failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`, because the interrupted runner from 51 s earlier was still listening there. The bounded same-run retry recovered it on its last attempt. **Expected behavior** When the durable controller record proves that no turn was ever sent, recovery starts the bootstrap again. It first stops the interrupted runner in the reused sandbox. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane while the runner opens its provider session. 3. Restart the control plane after the controller lease expires. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the remote checkpoint proof (`remoteBootstrapProvenIncomplete`, #19583) required an unfinished `session.open` with no committed events, and no provider events at all. A completed `session.open` with only session-open events had no rule, so it fell through to `ambiguous_remote_recovery_evidence`. Refs #19583, #19616, #19684, #19524 ## What Changed - `native-bootstrap-proof.ts`: new `runnerdStateHasNoProviderTurn` proof. The state must hold only `run.prepare` and `session.open` (pending or completed) and no warm, staged or adopted state. Every committed event must be a session-open event, with contiguous sequence numbers from 1, so no event is hidden. Only the controller issues turn commands, and it writes each one to Postgres before sending it. - `native-runner-checkpoint.ts`: `nativeRunnerCheckpointRemoteBootstrapProof` replaces `nativeRunnerCheckpointProvesIncompleteBootstrap`. It also reads a bounded number of `event:N` parts and returns which proof holds. It still rebuilds the state in memory. - `native-restart-recovery.ts`: a remote run can now claim `bootstrap_incomplete` with reason `remote_session_opened_without_turn`. Projected provider events may only be session-open events (`harness.ready`, `session.started`, `session.resumed`, `session.updated`), and only together with the stronger session-opened proof. A turn or provider event still blocks. - `native-session-executor.ts`: - For a remote sandbox target, the executor accepts the same proof on the restored record. - Before the same-run bootstrap relaunches, `stopStaleRemoteRunnerBeforeBootstrap` stops the interrupted runner in the reused sandbox. It uses the verified identity marker and process check from reattachment, which signals only a process whose identity matches this run: SIGTERM, a wait of up to 10 s, then SIGKILL. - The stop is best effort. If no prior runner can be verified, nothing is stopped, and the bounded same-run retry still handles any conflict. - `doc/run-log-events.md`: document the reason and the stop. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "drained remote"`. Before the fix, the opened-session case (with a projected `session.started`) returned `blocked` / `ambiguous_remote_recovery_evidence`. - These keep the hold: a `turn.start` command, a committed `turn.started`, an acknowledged event that is missing from the record, and a projected `turn.started`. The executor test restores the checkpoint and checks that the local proof is false and the remote proof is true. - Regression: `native-runner-restart-recovery.integration`, `native-session-executor`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `heartbeat/recovery`: 1460 passed, 14 skipped. - Stale runner stop: controlled sandbox-runner tests (`native-session-executor.test.ts`, "stale runner stop before a same-run remote bootstrap") run the real reattachment identity and process scripts against a fake sandbox runner: - a verified runner that exits on SIGTERM gets SIGTERM only; - a runner that ignores SIGTERM gets SIGTERM, then SIGKILL; - a runner with another identity gets no signal; - no live prior runner means no signal. - Live Daytona proof (wave `w-20261107T021837-s29`, ACPX Claude, `--faults K2 --triggers time,result`, 4 tasks, concurrency 1). The image was built from current master (`paperclip-daytona-runner@sha256:71f711ab…`), and its content id matches this branch. - All 4 tasks crossed a graceful restart, and all passed R1. Two of the K2s fired on the `result` trigger, during a running remote turn. - The K2 during a turn reattached to the live remote runner (`reattach_remote_runner`, generation 2), as designed. - Task t01 failed R6 with an orphan Daytona sandbox that no lease references. The graceful restart came during sandbox creation. This is a separate signature, outside this change. - The timed faults did not land in the exact "session opened, no turn" window. The integration tests cover that window with the evidence from the saved replay database. ## Risks - The proof relies on two facts: only the controller issues turn commands, and the controller writes each command before sending it (`#sendJson` waits for the checkpoint flush). A turn always leaves a `turn.start` command in the record. - The provider session opened in the old sandbox runner is not resumed. The new bootstrap opens a fresh session. No turn ran in the old one, so no work is lost. - The runner stop sends signals inside the run's own sandbox. It signals only after the existing reattachment identity and process checks pass. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:58:32 -06:00
3 people	18be7ac696	fix(server): retire idle native sessions before tenant sleep (#18988) ... ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Native agents can retain a warm session between useful turns. > - That owner must close and save its files before the tenant can sleep. > - A failed close must keep the owner visible until retirement succeeds. > - This PR prepares idle resources under the exact current admission hold, then takes a fresh safety snapshot. > - Busy turns, uncertain closure and changed holds still prevent sleep. > - CI also exposed a chat authorization race. The fix repeats full authorization after the exact delivery commits. ## Linked Issues or Issue Description Refs #18639 for local work and incoming wake boundaries. Related #18829 covers suspension receipts after admission. **What happened?** Idle preflight did not retire warm Native owners. Three failed-close paths could forget an owner without proving that its provider stopped. CI also found a false chat rejection when a delivery changed from processing to processed between two validation reads. **Expected behavior** Retire only idle owners under the exact bounded hold. Wait for closure and file collection. Retain failed owners. Scan safety after preparation. Keep diagnostic reads passive. A newly committed chat delivery must pass the full current authorization checks before attestation succeeds. **Steps to reproduce** Finish a warm Native turn. Start an idle drain and request safety with its owner ID. Delay or reject provider closure, or replace the hold during preparation or scanning. The report must keep the instance awake. For the chat race, commit the exact delivery between the first binding read and its fallback read. **Paperclip version or commit** Current head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` normally integrates actual master `1aa0c3639714f4976cfa56f95aae6e2b7d1235ac`. The merge preserves every unrelated master path. Fresh exact-head CI and complete review remain required. ## What Changed - Retain uncertain Native owners after failed restart, idle timeout and failed-turn closure. - Prepare Native retirement through the existing heartbeat facade. Reuse provider closure and stopped-file collection. - Track preparation under the exact idle owner and expiry. Reject new activity, changed holds and expired holds before and after awaited work. - Take the final durable and local safety snapshot after preparation. Keep diagnostic inspection passive. - Cover ephemeral workspace retirement and useful next use, supported plugin drain and SDK persistence, busy owners, failed closure and hold replacement. - Keep dedicated recovery storage explicit. Use the same validated options for support and storage. Probe-free helpers retain the original implementation bodies and preserve default callers. - Complete the Native fixture through real retirement and budget evaluation. Keep its rejection, file, credential, accounting and strict NONE assertions. - Repeat the complete chat authorization once if the fallback sees the exact batch now processed. Preserve owner locks, principal and binding checks, denial handling and the original retry deadline. ## Verification - The support-mode regression reproduces one failure with four passes under the old policy, then passes all five controls. It retains prior allocation and cleanup cases. The explicit mode does not probe embedded PostgreSQL or use an admin alias. - The deterministic chat race baseline reproduces two authorized failures with all four current-authority denial controls passing. Both Slack and Discord use real PostgreSQL and the real authorization function. The negative setup commits before the attestation transaction; it does not claim to revoke access through a row lock. - The product fix passes normal own App declaration emission with no public declaration changes, plus both current normal owning compiler programs (399 and 611 roots). The same six controls pass after the fix. The full original file plus these controls passes all 30 tests with no skips. - The original controlled Native continuation passes: one test, no failures or skips, 1.718 seconds in its body and 23.138 seconds in its launcher. The original 40-second case and 180-second outer bounds are unchanged. It verifies held closure, strict NONE, an ordinary authorized comment, saved session/files, useful output, encrypted credentials, accepted results and settled accounting. All ten observed processes and groups close; worker connections and listeners reach zero. A separate post-owner audit finds zero sessions; the successful owned database is normally removed. - Earlier preparation, HTTP, lifecycle, workspace and plugin controls remain source-specific evidence. They are not a claim that every final-tree test ran locally. - The original CI assertion log is retained. The controlled race proves a possible cause; the original CI SQL schedule was not captured. Two other jobs lost their self-hosted runners and provided no test verdict. Earlier asset timeouts, readiness failures and unexplained later recovery remain separately recorded. No timeout, failure predicate or authority check was weakened. Focused public commands: ```sh pnpm exec vitest run server/src/__tests__/idle-sleep-preparation.test.ts server/src/__tests__/instance-settings-routes.test.ts --maxWorkers=1 --no-file-parallelism pnpm exec vitest run server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts --maxWorkers=1 --no-file-parallelism ``` Local database qualification uses fresh owned test databases on existing PostgreSQL, bounded pools and one worker. The local allocation adapter changes only database allocation for this test module; it runs normal migrations and client cleanup. It is not part of the published source. No new cluster is created. Local acceptance measured the `0a69efa6` integration with the same committed authorization bytes. The later normal master merge adds startup-timeout classification and recovery changes. Those paths are outside the controlled healthy-backend witness; retirement and authorization hunks are preserved. Exact-head remote CI at `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` has completed with 56 successful contexts and four skips. The complete current Greptile summary is5/5 with no new actionable findings; all review threads are resolved. Ponytail’s current turn is explicitly paused without a current-head verdict; earlier actual findings are fixed and accounted for. No whole final-tree local repository build/test run, live-provider restore or deployed suspension is claimed. Actual master subsequently advanced to `d5c4f7dd02d408a261cedd9a1926b9900a4aedc4`. Its local normal-merge projection with this reviewed head preserves all20 owned paths and every unrelated master blob and mode. Current Runner TypeScript, App declarations and their full owning compiler programs pass. Focused Native cleanup and warm-continuation composition checks are being completed before merge; the new upstream suspension-retry behavior is not inferred from the older controlled-backend witness. The remote PR head remains unchanged. ## Risks - Privileged idle preparation can close idle warm Native sessions. It requires the exact current hold. - Failed closure retains its owner and can keep the tenant awake until normal retry succeeds. - A changed hold cannot authorize sleep. Started cleanup remains tracked until it settles. - The chat repair can perform one extra full authorization read. It keeps the existing deadline and does not grant authority from the fallback query. - Unsupported resources and active work retain their existing blockers. This PR does not certify every transport, plugin or deployed host. ## Model Used OpenAI `gpt-6.1-sol` produced the delegated changes and verification. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The coordinator's exact backend snapshot and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green at exact head `7f6f02b611dcf5a9a2b05173a44e9631eaf78058` - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups at the exact current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>	2026-11-06 20:49:30 -06:00
DottaandPaperclip	55c9c44e41	fix: preserve database-only failed controller evidence (#19714) ... ## Thinking Path > - Paperclip preserves failed work as well as successful work. > - Ephemeral enrollment must retain each selected controller source. > - A failed source can already exist entirely in an encrypted database checkpoint. > - Its original inline snapshot does not create a local filesystem dependency. > - The migration owner must prove that both legacy files are absent before accepting that case. > - This change preserves the failed evidence without creating execution authority. ## Linked Issues or Issue Description Refs: #19678. **What happened?** Startup still reports `native_controllers_require_migration` for a failed retained source with a valid database checkpoint, its original inline snapshot, and no local controller or supervisor. The online archive owner reports that the source is already absent. The existing database-only preservation case accepts only a missing inline snapshot. **Expected behavior** Accept the exact database-only failed evidence after full checkpoint and identity validation. Keep a present local payload blocked. Retain the failed run, original snapshot, coordinator, task pointer and queued work. **Steps to reproduce** 1. Store a valid encrypted checkpoint for a failed native run. 2. Keep its matching inline session snapshot and terminal-failure coordinator. 3. Remove both legacy controller files. 4. Check local controller obligations. 5. The old owner reports an obligation even though the evidence is already in the database. **Paperclip version or commit** The fix builds on current master `d5c4f7dd02`. **Deployment mode** Hosted native Runner with durable controller checkpoints. The local regression uses the existing real PostgreSQL and encrypted checkpoint fixture. ## What Changed - Add a narrow database-only failed-source preservation case to the existing migration owner. - Require the decoded checkpoint, protocol header, admitted execution, selected run and inline snapshot to match their exact scope. - Require both the controller and supervisor paths to be absent. A file or link keeps the local obligation blocked. - Retain the earlier null-snapshot predicate and all original failed execution evidence. Create no recovery metadata or executable authority. - Exercise the actual startup and maintenance consumers. Document the preservation boundary. ## Verification - Final revision `1b1da543dafd576e3c420657e3c2c6d0c882172d`: all checks are terminal, with 56 successful and four skipped, zero pending and zero failures. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42329501690/job/128275664414) passed. Build, types, server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head review](https://github.com/paperclipai/paperclip/pull/19714#issuecomment-6701018015) is 5/5 with no open threads. - Review found that the first non-recovery assertion stopped at the queued fixture's missing current binding. The fixture now binds the exact native issue/session before calling the real resolver and asserts `unsupported:metadata_format` from the selected failed source. That corrected case passes in 4.58 seconds, with wrapper exit 0 and cleanup complete. The production predicate is unchanged. The original startup RED remains valid. The final source/test type graph, module boundaries and diff checks pass. Fresh revision CI and review remain required. - The actual startup regression failed before the source change: one case, 16.62 seconds, expected the absent-files obligation to be false but received true. The wrapper exited 1 and completed cleanup. This failure is retained. - The final focused owner run passed nine cases in 10.79 seconds. It covers the new DB-only case, controller/supervisor presence, every inline scope field, foreign checkpoint Runner/session identities, the unchanged null-snapshot case, and existing online/offline archival controls. The wrapper exited 0 and completed cleanup. - The new case verifies unchanged run, coordinator, checkpoint and task-session rows, retained queued work, no upload, and no invented recoverable session. - The full qualified server/test source type graph passes with db/shared/Runner source aliases. Module boundaries and diff checks pass. Required broad package builds and regression suites remain in CI. - Read-only source diagnosis validated one exact existing checkpoint with the canonical decoder and matching header, run, Runner and session bindings. No source profile or secret values were emitted. No hosted mutation, admission hold, cancellation, status change, or writers-stopped claim was performed. ## Risks - A false database-only classification could hide local bytes. Both exact legacy paths must be absent, and every protocol and database binding must match. - Corrupt or mismatched checkpoints still fail closed. Existing failed evidence does not become successful or executable. - The original null-snapshot case is unchanged. Present local files still require the existing preservation path. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 20:45:29 -06:00
DottaandClaude Opus 5.5	a5fe5deebe	fix(environments): reclaim sandboxes left by a restart during acquire or teardown (#19696) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Remote agent runs execute in provider sandboxes (for example Daytona) that Paperclip creates and must clean up > - Before each provider create, Paperclip saves a durable creation intent (`sandbox_creation_operations`). Recovery uses it to find and destroy a sandbox whose lease was never committed > - Recovery waited 30 minutes after creation before it checked an intent, because a live server could still be waiting for the provider > - After a restart, the creating server is gone, but the intent still waited the full 30 minutes. A short-lived or redeployed instance never reclaimed those sandboxes > - The teardown that follows has the same problem: a teardown claim fences other sweeps for 15 minutes, and a restart during the claim kept the sandbox alive until the fence expired > - This pull request saves the server process identity with each intent and each teardown claim, and ends the wait when same-host proof shows that process exited > - The benefit is that a restart during sandbox acquire or teardown no longer leaves paid provider sandboxes behind ## Linked Issues or Issue Description **What happened?** A control-plane restart during Daytona sandbox acquire left the created sandbox without an `environment_leases` row. In one restart chaos wave, 6 creation intents stayed `pending` with `next_attempt_at` = creation + 30 minutes. The wave ended (and its environment was deleted) before that time, so 4 sandboxes stayed `stopped` in Daytona. Only Daytona's own auto-delete would remove them. Two of the runs did not exist in `heartbeat_runs` because they were non-run acquires; their intents still had the run id. **Expected behavior** After a restart, each sandbox that this instance and company created, with no committed lease and no live creator, is found by its exact creation name and labels and destroyed through the plugin, after the existing ownership checks. Sandboxes from another instance, another host, or a live server stay untouched. Nothing acts org-wide. **Steps to reproduce** 1. Start a Daytona-backed run. 2. Kill the server (SIGKILL) after the creation intent commits and before the lease commits. 3. Restart the server. On master, the intent stays `pending` for 30 minutes and the sandbox stays in Daytona. 4. The regression tests in `server/src/__tests__/sandbox-creation-operations.test.ts` reproduce this with the controlled provider fixture. Refs #18357 (added the durable creation intent this change completes). **Paperclip version or commit** master 07a548073a (live proof at 8d525dba7d + this change, content-matched to the runner image) ## What Changed - New `server/src/services/controller-process-identity.ts`: records the server boot id, PID, a PID-space hash and the process start time. The PID-space hash covers the hostname, the kernel boot id and the PID namespace on Linux (the hostname and boot time elsewhere), so containers that share a hostname but not a PID namespace never judge each other. It proves that a recorded process exited only in the same PID space: the PID is gone, or the PID is live with a definitely different start time. Another PID space, an unreadable identity, an unidentified record, or this server never counts as exited. - `sandbox-creation-operations.ts`: each creation intent stores the creating process and its settle deadline. Each recovery pass first moves intents whose creator exited to "due now". If the early lookup finds no sandbox before the settle deadline, the intent stays `pending` and retries. It does not become manual reconciliation, because the provider can still materialize an aborted request. - `heartbeat/recovery.ts`: each pending-cleanup teardown claim stores the claiming process, bound to its claim attempt id. Before the sweep reads due leases, it ends the 15-minute fence of a claim whose claimer exited. A newer claim from another path (workspace repair, export retry) that kept an older record has a different attempt id, so it is never released. - Both recovery passes page through candidates with a stable id cursor. Live, foreign or unidentified owners therefore cannot hide an abandoned record behind one page. - This covers every `createOwnedSandbox` caller, including non-task probe and qualification acquires. Their intents have no issue and a generated run id, which is why their run ids are missing from `heartbeat_runs`. - Recovery uses only the existing paths: exact `paperclip-create-<operation id>` name and label inspection in the plugin, `pending_cleanup` lease, and plugin teardown. No org-wide listing or deletion was added. ## Verification - Red, then green (controlled provider fixture, real embedded PostgreSQL): - `sandbox-creation-operations.test.ts` → "after the creating server exits mid-acquire". Before the fix, the exited-creator case left no cleanup lease. After the fix, the next sweep creates the `pending_cleanup` lease and the teardown destroys the sandbox. Counter-tests: a creation owned by this live server, another live process, another host, or a committed lease is not inspected or destroyed. An early "not found" keeps the intent pending, and a later sweep still finds a late allocation. - `heartbeat-pending-cleanup-sweep.test.ts` → "a teardown claim whose server stopped mid-attempt". Before the fix, a lost claim blocked the teardown. After the fix, it retries now. Counter-tests: claims from this live server, another host, or without an identity keep their fence. - Review regressions: "keeps the fence of a newer claim that kept an earlier attempt's server record" and "pages past more fenced claims than one sweep page" fail on the previous commit and pass now. "pages past more unprovable creations than one page" guards the creation-side cursor; the old unordered page happened to include the row, so it was not red. - After the review changes, `sandbox-creation-operations`, `heartbeat-pending-cleanup-sweep`, `live-recovery-work` and `environment-cleanup-work` pass (155 tests), and `tsc --noEmit` for `server` passes. - Related suites pass: sandbox-creation-operations (28), environment-cleanup-work, task-creation-recovery, native-startup-cleanup-recovery, sandbox-orphan-cleanup-spool, plugin-warm-tasks, idle-completed-history (187 total), and the pending-cleanup sweep suite. `tsc --noEmit` for `server` passes. - Live Daytona restart chaos wave `w-20261107T001609-s11` (codex, A1, K2+K4, concurrency 2, RC-06 masked, image `sha256:2767e91c…`, first fix only): 4/4 tasks pass, all restart-crossing, R6 pass for every task, `orphanSandboxes: []`. The server log and database show two creation intents killed by K4 being reconciled right after restart and destroyed through Paperclip. A third reclaimed sandbox lost its teardown claim to the next K4 kill in the same second, so its 15-minute fence blocked the environment delete. The second commit fixes that. - Live Daytona wave `w-20261107T012048-s11`, same plan, both commits: **4/4 pass, all restart-crossing, R6 pass for every task, `orphanSandboxes: []`, no held leases.** The environment delete through Paperclip succeeded with `remainingSandboxes: []`. One restart-time sweep destroyed one sandbox through the plugin. ## Risks - Same-host proof relies on PID liveness and process start time, as native controller takeover already does. A false "exited" result is limited to recovery running early. The plugin still checks exact name and labels before any delete. An early not-found result stays pending. - Multi-host deployments sharing one database keep the old deadlines, because a different PID space is never judged. A redeploy into a new host or PID namespace also keeps the old deadlines (fail closed). - The live waves ran before the review commits. The review commits only narrow when a record is judged, and they add paging. - Intents and claims written before this change have no process identity, so they keep the old deadlines. - No schema change. The new fields live in existing JSON metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:39:34 -06:00
Nicky Leach	03498f90ee	Add explicit Runner session release and resume support (#19683)	2026-11-06 18:31:23 -08:00
DottaandClaude Opus 5.5	f09b4704b6	fix(recovery): keep the agent's work when Retry continues after exhausted or blocked recovery (#19666) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery restarts agent work after a control-plane restart or a failed run > - When automatic recovery runs out or is blocked, the user clicks Retry > - Retry then started a fresh isolated attempt cloned from committed HEAD, so the agent lost its uncommitted files, its task branch and the worktree Git link > - A restart must never lose the agent's work > - This pull request makes Retry keep the task's own workspace when that is safe, and tells the agent where the old files are when a fresh attempt is still required > - The benefit is that a Retry continues the real work instead of starting over ## Linked Issues or Issue Description **What happened?** After `native_restart_recovery_blocked` or an exhausted recovery generation, a Retry ran in `isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That directory was a detached clone of the base ref. It did not contain the uncommitted edits from the issue's git worktree. It was not on the task branch. The agent was not told where the old files were. A restart-chaos wave graded this as an R2 failure (`pre_fault_files_lost`). A second defect: when a restart hit the bootstrap of a fresh attempt, preparing the same run again failed with `isolated_task_directory_binding_mismatch`, so the Retry could not finish. **Expected behavior** Retry continues in the task's own workspace with its edits and `.git` intact when no other writer can hold it. When a fresh attempt is really needed, the old directory stays as it was and the agent gets its path and its changed files. Preparing the same attempt again after a restart is idempotent. **Paperclip version or commit** `master` at `e482f0f632`, restart-chaos waves on 2026-11-06. **Steps to reproduce** 1. Start a native task in a git worktree. Let the agent write an uncommitted file. 2. Kill the control plane so recovery ends in `native_restart_recovery_blocked` or an exhausted generation. 3. Click Retry. 4. The successor runs in a fresh clone without the uncommitted file or the task branch. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `canSelectFreshWorkspace` returns a fresh intent for every task Retry that has a system recovery record. - `server/src/services/explicit-native-continuation.ts` (fresh selection in the source loop) then always persisted `fresh_isolated_workspace`. - `server/src/services/fresh-workspace-seed.ts` `materializeFreshWorkspaceSeed` clones the project source at the base ref with `--no-local`, checks out a detached commit and removes `origin`. Uncommitted work and the branch are not carried over. - RC-35: `server/src/services/heartbeat/workspace-preparation.ts` rejected any reusable existing workspace while `fresh` was true, including the attempt directory that the same run had already created and bound. ## What Changed - New continuation mode `reuse_issue_workspace`. At Retry admission, after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that the task's bound workspace is the single local directory the failed run used, owned by the task, not a shared workspace, not under remote storage authority, with no other lease, failed cleanup or running workspace operation, a settled recovery generation, no workspace-export failure, and a readable Git state rooted at that directory. Then the Retry continues there with a fresh session. Otherwise the existing fresh attempt is kept. - The reuse decision consumes the generation's one alternative attempt (`freshAttemptRunId`), sets the Retry strategy to `continue`, and records `issue.retry_workspace_reused`. - Environment selection pins the reused workspace binding and fails with `retry_workspace_binding_mismatch` if the binding changed before dispatch. - Fresh fallback: the run's continuation context gets `priorWorkspace` (path, branch, bounded list of uncommitted files). Git status is read with optional locks disabled. The prompt tells the agent to inspect it and carry over the work. The old directory is not changed. - RC-35: preparing the same fresh attempt again after a restart reuses the attempt directory that this run already bound. - Reuse checks run Git through the shared workspace scan scheduler with short deadlines, because admission can hold the issue lock. - `doc/task-recovery.md` and `doc/execution-semantics.md` describe the new contract. ## Verification - `server/src/services/retry-workspace-continuity.test.ts` (embedded Postgres, real git worktree with uncommitted edits): - exhausted-generation Retry and blocked-recovery Retry continue in the same worktree, `.git` file and branch intact. Both failed before the fix (readiness was `fresh_isolated_workspace`). - Blocked recovery without a generation and duplicate holds for one run also reuse the worktree. A running operation orphaned by a restart of the stopped source does not block reuse. - The fresh attempt is kept when the workspace is unsafe: live lease, running workspace operation, corrupt `.git`, missing directory, remote authority, shared workspace, running export generation. `priorWorkspace` is omitted for the unsafe writer and shared cases. - `workspace-preparation.test.ts`: - A `reuse_issue_workspace` Retry prepares in the original worktree with its uncommitted file and branch. - RC-35 red/green: re-preparing the same fresh attempt reuses its bound directory and keeps files written before the restart. Another run still gets `isolated_task_directory_binding_mismatch`. - The fresh fallback sets `priorWorkspace` with the old path and `unpublished.txt`. - `environment-selection.test.ts`: the new mode pins the binding and rejects a changed binding or cwd. - `packages/adapter-utils/src/server-utils.test.ts`: the prompt names the retained directory and its files. - Regression files run locally: `workspace-preparation`, `environment-selection`, `task-retry`, `quarantined-task-retry`, `explicit-native-continuation`, `workspace-recovery`. - Restart-chaos (`tests/restart-chaos`, local, B1+B5, codex+claude, K1/K4/K6, concurrency 2, rc06-masked): - Final head: wave `w-20261107T014633-s7206` on `c6776283f3`: 8/8 PASS, all 8 restart-crossing. 5 tasks needed a Retry (t03-t07: codex B1 K6, codex B1 K1, codex B5 K6, codex B5 K1, claude B5 K6). All five finished in the original git worktree with `.git` and every pre-fault file present (`workspaceMovedTo` empty, `lostFiles` empty); no "Retry keeps a fresh attempt" log line. - Wave `w-20261107T004829-s7205` on `29373a6b6a`: 8/8 PASS, all 8 restart-crossing. 3 tasks needed a Retry (t02, t06, t07). All three finished in the original git worktree with `.git` and every pre-fault file present; no "Retry keeps a fresh attempt" log line. (`cc178492ef` only makes the operation filter stricter for run-less jobs.) - Wave `w-20261107T000412-s7204` on `d197082107`: 8/8 PASS, 7 restart-crossing. 3 tasks needed a Retry (t02, t04, t06). All three finished in the original git worktree (`workspaceMovedTo` is empty), R2 passed with `.git` and every pre-fault file present, and no "Retry keeps a fresh attempt" log line appeared. - Wave `w-20261106T231416-s7203` on the first commit: 6/6 PASS, but its t01 Retry still moved to `isolated-workspace-attempts`. That instance was not kept, so the exact rejection was not recorded. The likely cause is a `running` workspace operation that a killed server left on the stopped source (the source restarted three times). The second commit treats such restart orphans as not live and logs the rejection reason. A `--replay` of that t01 on `d197082107` finished in the original worktree with its `.git` file and no attempt directory. - Review follow-ups in `d197082107`, `29373a6b6a`, `cc178492ef` and `c6776283f3` (run-less workspace jobs still block reuse; red/green covered by the `running_operation` case): holds are grouped by source run; the readiness fallback (blocked recovery without a generation) also reuses the task workspace; Git probes go through the shared scan scheduler with 10-15 s deadlines; `priorWorkspace` is only set for a task-owned directory with no other writer; the reuse rejection reason is logged; `priorWorkspace` uses the same task-owned, settled, single-writer predicate as reuse; Git probes share one budget per workspace; the RC-35 check is a pure path check; renames are parsed from either porcelain column. ## Risks - A Retry now writes into the old directory instead of a new one. The checks above prevent a second writer. A stale binding fails the run with `retry_workspace_binding_mismatch`; it does not write to another directory. - If the cause of the failure was inside the workspace itself, reuse could repeat it. Workspace-export failures, restore failures and required workspace recovery keep the fresh attempt. - A workspace operation of the stopped source that started before the source settled and before the current server process no longer blocks reuse (a restart orphan). Operations with no run, of another run, or started in the current process still block it. - `priorWorkspace` is an optional new field in the continuation envelope. Older adapters ignore it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:27:22 -06:00
DottaandPaperclip	d5c4f7dd02	fix: preserve shared Git sources during online workspace migration (#19688) ... ## Thinking Path > - Paperclip preserves the work produced by its agents. > - Remote workspace migration must preserve legacy Git history and local edits. > - Linked checkouts can share administration and objects with other tasks. > - A task lock alone cannot exclude writers of a shared or borrowed store. > - The online source owner must derive and retain the full physical ownership closure. > - This change reuses task, project, lifecycle and finalizer locks for that closure. > - Idle legacy Git work can then migrate while unrelated remote turns continue. ## Linked Issues or Issue Description Refs: #19642. **What happened?** Online migration keeps legacy Git sources blocked because several task checkouts can share a project base or a private Git cache. A read-only source observation found 160 pending Git records, 120 distinct paths and nine common stores. Its largest project cohort contained 126 records and 110 task owners. An ordinary-directory fence cannot preserve these sources by itself. **Expected behavior** Derive the physical source owners from stable Git metadata. Hold their existing source and lifecycle keys through publication. Retain all history and recovery obligations. Reject foreign aliases, unknown stores and changed sources. Let unrelated object-backed work continue. **Steps to reproduce** 1. Create a managed project checkout and linked task worktrees. 2. Keep a settled prepared recovery source on a sibling task. 3. Keep unrelated remote work running. 4. Request an online import of an idle leaf checkout. 5. The existing owner keeps the shared source blocked. **Paperclip version or commit** This branch builds on merged #19642 (`7ac99718a4`) and fresh master `fb3cf1a072`. It adds the Git closure. It does not add a writers-stopped shortcut. **Deployment mode** Hosted native Runner with remote workspace storage. Local qualification uses owned synthetic Git repositories, disposable PostgreSQL databases and the actual archive and storage owners. ## What Changed - Read bounded stable Git pointers, common-directory metadata and local alternates. Derive the complete physical source ownership cohort. Preserve all object stores. - Retain common and borrowed store owners in preparation, lifecycle, runtime, expiration and actual reopen paths. Keep company boundaries and the existing 128-key bound. - Hold exact sibling native recovery proofs under their existing physical finalizer keys. Recheck descriptors, allocations, recovery actions and source identities before capture and publication. Preserve original states and warnings. - Require leaf-first nested imports. Reuse the existing immutable child download, restore and source comparison before parent exclusion. - Bind the self-contained archive owner to the held Git namespaces. Keep its three walks, history normalization, fsck, bytes and digest format. Reject bare, detached, network, unknown or changed Git storage. - Check both foreign path fields. Keep foreign metadata private. Document the proof, bounds, source observation and remaining eligibility limits. ## Verification - Final revision `2934e41c923b93e16508b978c849b747c86be645`: all checks are terminal, with 56 successful and four skipped, zero failures and zero pending. The [required CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/42326464408/job/128266279210) passed. Build, types, all server/Runner/serialized suites, end-to-end aggregate and Snyk passed. [Fresh exact-head Greptile review](https://github.com/paperclipai/paperclip/pull/19688#issuecomment-6699760104) is 5/5 with no open threads. GitHub reports MERGEABLE and CLEAN. - Review then found two correctness gaps: an ordinary managed directory inside an unrelated host repository was blocked, and a Git parent could miss a completed nested operation after its child snapshot became missing or deleted. Three actual-owner regressions reproduced both gaps in 9.70 seconds. The corrected owner passes 12 focused cases in 17.08 seconds. It imports and restores only the ordinary root bytes, preserves the outer repository, rejects both missing/deleted child receipts before a parent journal, and retains valid/changed child coverage and private-cache contention. Its wrapper exited 0 and cleanup completed. The first bootstrap attempt failed before tests ran because the host exhausted its 32 shared-memory segment limit; no shared services were stopped and no pass is claimed for that attempt. The final qualified source/test type graph, module boundaries and diff checks pass. - Receipt scans keep canonical path resolution. A lexical SQL prefix can omit children reached through symlinked ancestors. The observed full metadata closure completed in 1.17 seconds; no measured bottleneck justifies weakening that alias proof. Writer admission still rejects uncertain Git ownership before mutation. Expiration returns its typed skip, and reopen returns its structured failure. Runtime and lifecycle do not bypass an unknown source. No broad error-contract change was added. - Full CI at `720104ea975e` exposed five existing boundary cases across three server shards. Four exact cases reproduced locally as failures in 25.98 seconds. Reopen now preserves its structured failure when source inspection rejects the path, without reconstructing it. Runtime recognizes only the exact regular ignore-only `.git/info/exclude` fragment; migration still rejects that fragment. The original arbitrary-path and file-URL import/delete-host/resume fixtures keep both supported input forms and use the genuine offline setup path after their sole writer finishes. The production mode gate is unchanged. A new real-owner control proves the offline flag still rejects a running legacy database writer. - The final focused run passed all ten cases across four suites in 22.75 seconds, including all five CI failures, invalid-pointer admission, fragment rejection, offline writer exclusion and actual nested reopen lock ordering. Its wrapper exited 0 and cleanup completed. The known invalid-pointer case remains fail-closed; the separate unexpected-I/O diagnostic retains safe error metadata without command text or credentials. Existing Git stderr redaction coverage remains intact. The final metadata reader suite passed 20 cases in 2.68 seconds. The qualified server/test source type graph, module boundaries and diff checks pass. No production assertions or timeout defaults were weakened. Fresh CI and review are required for the repair revision. - A later review found that the ordinary early return still used only one foreign path field. The actual ordinary providerRef-only regression failed as incorrectly eligible. One source-path owner now enumerates both local locators for company/foreign ordinary and Git scans, layout checks, early admission, lifecycle/reopen/runtime and expiration/move/preparation retention. Capture and display keep their original preferred path. The actual owner canary passed 49 cases across two suites in 109.15 seconds. It covers cwd-only/ref-only/different/symlink locators, same-company mismatches, foreign owners, lifecycle ordering, Git/recovery imports and expiration/reopen. Three selected-locator fallback cases also failed before their guard; the final eight-case control passed in 31.05 seconds, including the configured alias and valid import/private-child controls. All wrappers exited 0 and completed cleanup. The metadata/locator suite passed 19 cases. The final source/test type graph, module boundaries and diff checks pass. - Review found two P1 gaps. Both configured-directory symlink cases failed before canonicalization. The actual parent import lost an already-imported child's unique private cache and failed its archive fence instead of retaining that owner. The corrected real-owner canary passed four cases in 20.78 seconds: configured symlink runtime provisioning, private-cache contention before the parent journal, successful parent publication after release, and unchanged/changed child coverage. The metadata suite passed 15 cases in 4.14 seconds. Both wrappers exited 0 and cleanup completed. - A fresh source read-only observation found 160 pending Git records and nine common stores. An additional exact receipt query found 1,243 completed migration receipts and 557 nested source paths under pending Git roots. All 126 records in the largest project were present in the captured Git metadata set. For all 110 present candidate workspace records, the derived closure fits the unchanged 128-key cap. The worst case needs 110 issue keys and one project key, with 17 keys of headroom. Its 173 imported children introduce no physical issue owner outside those 110 logical task owners. This checks the observed key bound only. It does not claim migration eligibility or a coherent snapshot. - The initial stacked CI did not run four Runner test lanes: trusted master passed new --package-lane/--native-server options to the old base lane script. The branch is now rebased onto actual merged master and retargeted before synchronization. It includes the canonical current lane parser without an unrelated Runner patch. Fresh new-head CI is required. - The actual importer/source owner run passed 36 cases across three suites in 187.75 seconds. It covers sibling prepared recovery, unrelated running remote work, common-cache contention, changed coordinator/allocation/path state during upload, exact nested coverage, dirty unpushed work, and existing expiration/reopen owners. Wrapper exit 0 and cleanup completed. - A separate previously hanging lifecycle-worker restore passed in 17.33 seconds with the unchanged 15-second test-body limit. An earlier run timed out in three expiration/reopen cases and blocked cleanup; its wrapper later exited 1. These failures remain recorded. They are not counted as passes. - Two boundary controls first failed: a foreign providerRef-only alias was incorrectly eligible, and actual reopen did not admit its distinct cache owner at the initial source boundary. The narrow guards now inspect both foreign locators and derive physical Git keys before the raw lifecycle transaction. The foreign alias and existing real nested reopen contention passed. The private-cache control first retained contradictory recorded source provenance and stayed fail-closed after release. The corrected present legacy-worktree fixture passed in 12.61 seconds. These failed attempts remain recorded; they are not passes. - The metadata reader and serialized/local archive controls use actual Git repositories. They compare archive bytes and digests, remove every original source, restore staged and unstaged work, and run fsck. The metadata reader passed 13 cases in 2.85 seconds. The local/serialized archive controls passed nine cases in 9.27 seconds. All processes exited 0. An additional actual importer control changed a relative commondir pointer to the absolute name of the same store during upload. Publication was rejected, the uploaded journal remained available, and original files stayed intact (one case, 5.86 seconds; wrapper exit 0 and cleanup complete). - The final rebased full local server/test type graph passes with db/shared/Runner source aliases because reused dependency declarations predate new master exports. The first rebased attempt reported those stale declaration fields/tables, and is not counted as a pass. Module boundaries and diff checks pass. Standard package builds, types and broad regression checks remain required in CI. - No hosted import, admission hold, cancellation, provider mutation, source filesystem write or shared-cache Git command was performed for this change. The source observation is read-only and is not a locked eligibility proof. ## Risks - A missing writer owner could make physical source exclusion incomplete. Existing mutation owners retain the same company/task/project keys, and uncertain metadata fails closed. - A project can contain a large cohort. Limits remain finite. A bound failure keeps the source blocked. - Bare, detached, unowned or network Git stores remain blocked online. Missing archived sources retain their existing authenticated reconstruction path. - Exact recovery proofs preserve authority and obligations. They do not infer provider termination or mark a prepared descriptor finalized. - This change does not claim all observed Git records are immediately eligible. Existing preparation, provider, recovery, runtime and source-change blockers remain. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The host does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 20:08:19 -06:00
DottaandClaude Opus 5.5	a894b11a2b	fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run's close must prove that its runner durably suspended before the session's next run may start. If it cannot, the runtime quarantines the session (`NativeSessionCloseUnrecoverableError`) > - In a restart chaos storm, an ACPX Claude run succeeded and its normal close quarantined its own session. The follow-up run on that session failed 2 s later with `native_session_cleanup_quarantined`, and no control-plane restart came between the two runs > - The close sent `runner.suspend` once, the ACPX sidecar rejected it right after the turn completed, and the barrier gave up at the first rejected command. No log line showed the quarantine or its cause > - This pull request retries a rejected suspension within the close deadline, and logs every cleanup quarantine with its cause when it is created > - The benefit is that a successful run no longer blocks its own follow-up because of a one-time suspension rejection, and any remaining quarantine can be traced ## Linked Issues or Issue Description **What happened?** Run A (ACPX Claude) succeeded. The run log of its close shows: - `runner did not prove durable suspension before checkpoint` - `native_session_settlement_incomplete {"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}` The runner journal shows `runner.drain` completed, then `runner.suspend` failed with `failed to suspend ACPX provider: ACPX sidecar command session.suspend was rejected (retryable=false, classification=unclassified)`. The follow-up run B on the same session was created 200 ms after A finished and failed 2 s later at admission (`retryQuarantinedSessionCleanups`, operator recovery branch). The server log had no line for the quarantine. One Retry completed the task. **Steps to reproduce** 1. Run an ACPX task whose sidecar rejects the first `session.suspend` after the turn completes. 2. Start a follow-up run on the same session. **Expected behavior** The close asks again, proves suspension, and the follow-up runs. If a quarantine is still needed, it is logged with its cause. **Actual behavior** The close fails on the first rejection, quarantines the session, and the follow-up fails with no logged cause. ## What Changed - `runner/.../durable/runner.rs` (`process_command`): a failed `runner.suspend`, on first execution or replay, now keeps the runner in `Continue` instead of `Suspend`. Before, the runner recorded `suspended` over a provider whose session was still open, then exited. The RC-34 journal shows exactly that: runner lifecycle `suspended`, ACPX provider `session_open`. A failed `runner.shutdown` still stops the runner. - `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend` now lifts the connection's terminal-lifecycle fence, so the next pending command (the fresh suspend) is delivered on the same connection. Completed suspensions and every shutdown keep the fence. New test: "delivers a fresh suspension after a rejected one on the same connection". Before the fix, the retry was never sent and the test timed out waiting for the frame. - `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a rejected `runner.suspend`, the barrier sends a fresh suspend command, up to 3 attempts, 250 ms apart, inside the same close deadline. An exited runner or a spent budget still fails closed. A rejection changes no state: the runner returns before it records suspension. - `native-session-runtime.ts`: new `onNativeSessionCleanupQuarantined(listener)`. It is called when a cleanup quarantine is created or escalates to operator recovery, with company, run, issue, session, `operatorRecoveryRequired`, a bounded cause (error name, code, message) and bounded settlement facts. - `server`: `logNativeCleanupQuarantines` logs each quarantine as a warning. The subscription is process-wide and installed once, so closes that fail during shutdown, after HTTP closes, are still logged. ## Verification - Red before the change: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected suspension"`. - "retries a rejected suspension…" got `false`. - "stops retrying…" saw 1 attempt instead of 3. - Green after the change: both pass. - Rust: new test `rejected_suspend_keeps_the_runner_serving_its_controller` covers a failed suspend, its replay, and an accepted retry with a fresh command. `failed_lifecycle_commands_replay_their_terminal_transition` now expects `Continue` for suspend and `Shutdown` for shutdown. The `pnpm run check:runner` unit suites pass (559). `acpx_park_readiness` has two timing tests (`< 500ms`) that fail at this host load on unchanged master as well. - New runtime test: "announces every cleanup quarantine with its cause when it is created" (operator-required quarantine, cause and settlement reported once). `src/native-session-runtime.test.ts` passes (212). - The suspension cases in `runnerd-codex-transport.test.ts` pass, except "proves local suspension after an event backlog… (1024 suffix deltas)". That case also times out on unchanged master at this host load (27–35); I confirmed this by stashing the change and rerunning. - Server `tsc --noEmit` passes after the runner package build. ## Risks - A provider that keeps rejecting suspension now gets three tries instead of one, within the existing close deadline. The final outcome and quarantine are unchanged. - The cause of the ACPX sidecar's first rejection is not proven. The logged evidence fits the sidecar's "not at a safe suspension point" guard (`turnId`, tools or inputs still set) racing the just-completed turn. This change makes close tolerate it. It does not change the sidecar. - Log volume: one warning per quarantine. - The Rust change ships in the runner binary. Hosted images need a rebuild to pick it up. - After a failed suspend, the runner stays alive with its provider. If every retry fails, close still treats suspension as unproven and kills the owned runner process, exactly as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532, #19609. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:06:54 -06:00
DottaandClaude Opus 5.5	30857db55e	fix(native): recover a run killed between runner launch and its identity record (#19694) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The server records the request to start it, then records its process identity after it spawns. > - If the server is killed between those two records, there is no process ID to check after the restart > - Restart recovery then restarted the bootstrap, but the executor failed with `durable_identity_unreadable` because the runner never wrote its state > - The user's Retry then waited forever with "Retry is waiting for Paperclip to verify that the previous execution stopped", because the stop check needs a process ID > - This pull request restarts that bootstrap and proves on the host that no runner for the run exists > - The benefit is that a restart in the runner launch window no longer loses the task or leaves Retry permanently stuck ## Linked Issues or Issue Description **What happened?** In restart-chaos testing, an ACPX Claude run was created and the server was SIGKILLed 1.3 s later, right after `native.process_start_requested` and before the runner identity was recorded. On restart, recovery claimed `bootstrap_incomplete` (`dead_runner_before_provider_identity`). The executor then threw `runner_state_identity_mismatch: durable_identity_unreadable`: the state root existed, but `control-plane/control-plane-state.json` had never been written. The run failed with `native_continuation_requires_reconciliation`. A Retry was accepted (HTTP 200), but no run was ever created. The issue stayed blocked with `retryWait.reason=process_identity_missing` for over 20 minutes and through another restart. **Expected behavior** When the runner never wrote state, no provider started, so the bootstrap can start again. A Retry should be admitted once the host proves that no runner for the run exists. **Paperclip version or commit** `master` at `fb3cf1a072`. **Steps to reproduce** Create a native local task and SIGKILL the server about 1 s after the run starts, after the runner launch is requested and before its identity is recorded. Restart the server, then click Retry. ## What Changed - `migrateRunnerdStateRootForExecution` (`native-session-executor.ts`): for a `bootstrap_incomplete` restart on a local target, a root is quarantined (kept for inspection) and the bootstrap starts again only when both of these hold: - the root has no `control-plane-state.json`; - no live runner for the run exists on this host. A state file that exists but cannot be read, or a runner that started but has not connected yet, still fails closed with `durable_identity_unreadable`. - `unidentifiedLocalRunnerProvenAbsent` (`native-session-executor.ts`) proves that a runner is absent when both of these hold: - no host process carries the run's `--run-id <runId>` argument (`paperclip-runnerd` is always started with it, detached and unreferenced); - no saved control-plane state for this run exists, in the canonical root or in quarantine. A provider starts only after its runner connects and writes that state. - `native.process_start_requested` now records the launching `hostname`. - `recordUnidentifiedLocalRunnerStop` (`native-local-process-stop.ts`) applies the proof only when all of these hold: - the run is native; - the run has no process ID; - its latest launch event is `native.process_start_requested`, from this host; - all of its leases are local. A launch recorded without a host stays blocked. - The process listing runs asynchronously with a 5 s deadline. A failed or slow listing proves nothing. It then writes `native.local_process_stopped` with null process IDs and `basis: "unidentified_runner_absent"`. A later launch request invalidates it as usual. - The continuation and Retry gate (`explicit-native-continuation.ts`) uses this proof before returning `process_identity_missing`. A dry run checks the proof but records nothing. - `doc/run-log-events.md` documents the new receipt basis. ## Verification - `native-session-executor.test.ts`: - The bootstrap restart with absent state was red before the fix (`durable_identity_unreadable`) and is green after it (root quarantined, backend admitted). - Unreadable state, and a live runner that has not connected, still reject. - Proof cases: absent → true; a live process with `--run-id <runId>` → false; this run's saved state → false; another run ID → false. - `explicit-native-continuation.test.ts` (embedded Postgres, real admission): - The `absent` and `dry_run` cases were red before the fix (blocked) and are green after it (admitted). Only the real admission records the receipt. - The `live_runner`, `saved_state`, `foreign_host` and `unbound_launch` cases stay blocked. - Broader local run after the review fixes: `explicit-native-continuation`, `native-session-executor`, `native-recovered-owner-followup`, `native-runner-restart-recovery.integration` and `task-retry` gave 1836 passed, 0 failed. - A chaos replay of the original task (`--replay w-20261106T232854-s7103/t26`) passed (R1 PASS). Its kill landed after the runner identity was recorded (`resume_dead_runner`), not in the launch window, so the tests above are what reproduce that window. - `tsc --noEmit` for `server` passes. ## Risks - The proof depends on `ps -axww` listing every process on the launching host. If `ps` fails or times out, the proof fails closed (still blocked). Remote runners and other hosts are excluded. - A runner that was spawned but has not yet written its state could be mid-start when admission runs. It would still carry `--run-id` in its arguments, so the proof refuses. - Only the bootstrap path with no state file changes. Ownership, checkpoint and reattach checks are unchanged. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 20:06:36 -06:00
DottaandClaude Opus 5.5	803cb3bfe3	fix(native): re-bootstrap an OpenCode runner drained before its first turn (#19704) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd`. After a restart, recovery may start the bootstrap again when the durable state root proves that no provider work started. > - #19665 added a proof for a graceful drain that lands after `session.open` started a fresh session and before the first turn. It requires every recorded provider process to have exited. > - OpenCode records provider startup ownership twice during `session.open`. The first record comes before launch, with `processId: null` and `processGroupId: null`. > - The proof treated `processId: null` as a process it could not prove gone. An OpenCode run drained in this window still failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. > - This pull request treats a null process id as "no process yet". The benefit is that a graceful restart during OpenCode bootstrap continues automatically, as it already does for ACPX. ## Linked Issues or Issue Description Refs #19665, #19524 **What happened?** A restart chaos wave sent SIGTERM to the control plane about 11 s after an OpenCode run started. The drain suspended the run. At the next boot, recovery verified that the local runner had stopped and chose `bootstrap_incomplete`. The executor then failed the run with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. The kept state root had: - `run.prepare` and `session.open` completed, with a freshly started session (`status: "started"`); - only `session.started`, `session.capabilities.updated` and `session.goal.snapshot` committed; - two `harness.diagnostic` events of type `provider_startup_ownership` in the runner outbox, the first with null process ids and the second with the server pid and process group; - no turn command, and the OpenCode server process had exited. **Steps to reproduce** 1. Start an OpenCode task on a local runner. 2. Send SIGTERM to the control plane after `session.open` completes and before the first turn. 3. Let the control plane restart. **Expected behavior** Recovery starts the bootstrap again, and the task continues. **Actual behavior** The run fails. The task needs a manual Retry. ## What Changed - `native-bootstrap-proof.ts`: the process-id collector skips `processId`, `pid` and `processGroupId` values that are `null`. A null value records a launch that has not produced a process yet. Any other non-number value still fails the proof, and every numeric id must still be proven gone. - Tests: a red test with the OpenCode startup-ownership shape, plus counter-tests for a live OpenCode server process, a non-numeric process id, and a started turn. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain once OpenCode opened its session before the first turn" failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 36/36. - I ran the proof read-only against kept real roots: - the OpenCode drained root now returns `true`; - an OpenCode root from the same wave where a turn ran (108 events) still returns `false`; - the three roots from #19524 and #19665 still return `true`. - `native-runner-restart-recovery.integration.test.ts` and `native-restart-recovery.test.ts`: 134 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live replay: see the PR comment. ## Risks - Low. The change only skips null ids. Numeric ids still need a proven exit. Any turn, provider output, resumed session, or remote runner still fails closed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:59:34 -06:00
Devin FoleyandPaperclip	1aa0c36397	Fix execution after onboarding plan approval (#19698) ... ## Thinking Path > - Paperclip manages agents and their work. > - First-task onboarding turns a user's goal into an approved proposal. > - The interview offers a plan and a team to carry out that plan. > - The instructions leave execution after acceptance unclear. > - An agent can add a plan-only approval boundary and finish without doing the requested work. > - This change makes the approval scope explicit and tests both execution and plan-only outcomes. ## Linked Issues or Issue Description **What happened?** An onboarding agent can treat an interview about building a product as a request for a plan only. It can then retain the approved plan and close onboarding without creating execution tasks. The interview evaluation previously accepted that result. **Expected behavior** Acceptance of an execution proposal starts its selected work. An explicit plan-only request stops after planning. User-requested separate approvals and existing permission and budget controls still apply. **Steps to reproduce** 1. Choose the onboarding interview and describe a deliverable. 2. Accept the saved plan proposal. 3. Check that the agent starts the approved work instead of closing with only a plan. ## What Changed - Clarify execution intent, approval wording, continuation, and completion in the first-task prompt. - Require assigned child work and saved output in the interview acceptance evaluation. Use checkbox acceptance in the regular interview journey and retain conversational acceptance in the approval-reply suite. - Add an explicit plan-only acceptance journey and nine grader regression checks. - Update evaluation counts, grading versions, and onboarding verification guidance. ## Verification - Passed: all applicable CI checks on `198a6b47cac7`, including typechecks, the general and serialized server tests, Runner checks, build, and browser E2E shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42324855376). - Passed locally: the full Runner support suite, 6,860 tests across 323 files (three skipped), plus 13 onboarding asset tests. - Passed locally: the full workspace build, workspace package typechecks (server rechecked after restoring excluded-plugin SDK links), and the Runner evaluation TypeScript check. - Passed: map validation and whitespace checks. - Initial broad local checks failed during checkout preparation because PostgreSQL library symlinks and runtime/SDK assets were missing. Restoring the dependency setup made the affected controls pass. The full root test suite passed in CI; it was not rerun locally after that repair. - Live model journeys have not been run. Grader tests do not prove model compliance. ## Risks - Prompt behavior needs live qualification across providers. This change adds no server-side enforcement or permission bypass. - The new plan-only case adds four local evaluation cells. A full campaign now has 56 first-task cells. - Existing proposals that reserve later approval keep that restriction. This change does not rewrite tenant data or start existing onboarding tasks. - No schema migration or deployment is included. ## Model Used - OpenAI `gpt-6.1-sol`, xhigh reasoning effort, with repository tools and local test execution. Context window size is not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and full Runner support locally; full root coverage passed in CI) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 17:56:27 -08:00
DottaandClaude Opus 5.5	c37bc749ac	fix(native): keep restart continuations alive through early results and repeat restarts (#19699) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs continue a turn lost to a control-plane restart once, in the same conversation (#15297, #19501) > - In restart storm 3, two Codex runs each crossed one restart and still needed a Retry > - The continuation was attempted, but the driver failed it with `turn_binding_mismatch`, and the run then reported "runnerd exited before provider turn startup" > - A second restart that loses the continuation turn itself also needed a Retry, because the one-shot marker was already used > - A restart should not cost a click when continuing is safe, and a failure should name its real cause > - This pull request fixes the start race, reports the driver's own failure, and gives each admitted restart its own continuation > - The benefit is that interrupted tasks finish without a Retry, within the existing recovery budget ## Linked Issues or Issue Description Refs #19501, #15297, #19658, #19539 **What happened?** Restart storm 3 (seed 7103, RC06 mask on, load 9–12). Codex B5 Retry runs crossed one K1 restart mid-turn. After the restart, recovery resumed the dead runner. The runner restored the conversation and recorded `turn.failed` with `provider_turn_lost_on_restore` (`recoverable: true`). The control plane started the same-conversation continuation (`resumeInterruptedTurn` → `startTurn`). About 1 s later the transport closed with `protocol_failure:turn_binding_mismatch`. The run failed as `native_runner_process_exited: runnerd exited before provider turn startup`. The task needed another Retry. I reproduced it with the focused cell (`--workloads B5 --faults K1 --triggers retry`) and a temporary diagnostic. The failure is "Provider semantic result did not name the active turn": a `paperclip/runResult` reached the Codex driver while the continuation's `turn/start` was still pending and no turn was bound. The driver waits for a pending start before it handles `turn/completed`, but not before it handles `paperclip/runResult`. **Expected behavior** After a restart that loses a turn, the run continues automatically when that is safe. Each admitted restart attempt can continue once, within the provider-attempt budget. A failure names its real cause. **Steps to reproduce** 1. Run `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults K1 --triggers retry --tasks 2 --concurrency 2 --seed 4040 --rc06-masked`. 2. When a lost turn's continuation receives a semantic result before its start settles, the run fails with `turn_binding_mismatch` and the task needs a Retry. The race is intermittent: 1 of about 14 lost turns in my runs. **Root cause** - `packages/paperclip-runner/src/drivers/codex/codex-session-notifications.ts`: the `paperclip/runResult` branch checked the active-turn binding without waiting for `state.turnStartSettled`. The `turn/completed` branch already waits. - `codex-harness-session.ts` `startTurn`: the driver's own protocol failure closed the transport. The pending start then reported the runner exit, which hid the cause. - `codex-session-terminal.ts` `mapTerminalTurn`: a consumed one-shot marker blocked every later continuation in the run, even when the newly lost turn was that continuation itself. The one-continuation allowance is per run, not per issue or session lineage: attaching a new run resets the marker. The storm runs did attempt their continuation; the start race made it fail. ## What Changed - Codex driver: a `paperclip/runResult` waits for a pending turn start to settle before it is bound, like a terminal. - Codex driver: when this session's own protocol failure ends a pending turn start, `startTurn` throws `codex_turn_start_protocol_failure: <code>: <message>` instead of the runner exit. CodexRpc errors and protocol-integrity errors keep their own types. The server now classifies this as `native_session_interrupted` instead of `native_runner_process_exited`; both are retryable. - A later restart that loses the continuation turn itself re-arms the one-shot marker: the driver resets it as it was before the first continuation. It does this when it maps the lost terminal, including during recovery from a checkpoint taken while the continuation ran. The checkpoint gate `nativeRestartInterruptedTurnId` stays strict: a consumed marker still means the allowance is used. Each continuation still needs an admitted dead-runner restart. The existing provider-attempt budget bounds the repeats, and an exhausted budget still blocks with Retry. - `doc/execution-semantics.md`: an additive paragraph. ## Verification Red before the fix, green after: - `codex-app-server-driver.events.test.ts` "binds a semantic result that arrives before its pending turn start settles". Red: `session.failed`. Green: the result binds to `turn-1` and the turn completes. - `codex-app-server-driver.events.test.ts` "names its own protocol failure when that failure ends a pending turn start". Red: "runnerd exited before provider turn startup". Green: `codex_turn_start_protocol_failure: turn_binding_mismatch`. - `codex-app-server-driver.recovery.test.ts` "offers another restart continuation when a later restart loses the continuation turn itself". Red: `null`. Green: `turn-2`. A checkpoint with a consumed marker still returns `null`. - `codex-app-server-driver.recovery.test.ts` "re-arms the continuation when recovery finds the checkpointed continuation turn lost". This recovers from a checkpoint taken during the continuation, then starts the next continuation. Red without the re-arm (the marker stays consumed); green with it. - `native-runner-restart-recovery.integration.test.ts`: the same checkpoint (continuation active, consumed marker naming it) is claimed as `resume_dead_runner`. This is regression coverage. Suites run locally on the final code: - Runner: Codex driver, `native-session-runtime`, `runnerd-restart-continuation` and contracts (745 tests). - Server: `native-runner-restart-recovery.integration.test.ts`. - `tsc` for the runner tests and the server. - A few real-process tests timed out at about 6 s while six live agents ran. Each passed when rerun alone. The settlement suite passed both with and without this change at the same load. Live (local, real Codex, macOS arm64), focused restart-chaos cell `--adapters codex --workloads B5 --faults K1 --triggers retry,time --tasks 6 --seed 4042 --rc06-masked`: 6/6 crossed a restart and passed. R1 passed for all 6, so each continued without a Retry. ## Risks - A run can now continue more than once in the same conversation, once per admitted restart. Each continuation asks the agent to reconcile unfinished actions before repeating them. The provider-attempt budget still caps the repeats. - A semantic result that arrives during a pending start now waits for that start. If the start fails, the result is still rejected by the existing binding check. - The start race was intermittent (1 of about 14 lost turns), so the live cell shows absence of the failure, not a forced repro. The driver regression reproduces it deterministically. - No schema, API or migration change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, code execution, and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:55:18 -06:00
DottaandClaude Opus 5.5	11c07c622c	fix(native): restart a remote bootstrap retry that shutdown cut before runner launch (#19684) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on Daytona start a `paperclip-runnerd` process in a sandbox. Restart recovery decides what to do with each live or due run after the control plane restarts. > - #19616 added the `remote_runner_never_launched` proof. A claimed remote run with no runner process request and no controller checkpoint starts its bootstrap again. > - A graceful shutdown closes the local runtime listener before its drain. A remote run that is still starting then fails with `local_runtime_api_listener_unavailable`. The executor settles it as a same-run `bootstrap_retry` and releases its lease. #19596 treats this as a restart detach only for local runners. > - When the server restarted with that retry due, restart recovery claimed the row. The proof accepted only rows that still held a lease, so recovery blocked the task as ambiguous. > - This pull request lets the same proof also admit that settled `bootstrap_retry` row. > - The benefit is that a graceful restart during Daytona startup continues automatically. ## Linked Issues or Issue Description **What happened?** A chaos wave restarted the control plane gracefully (SIGTERM) while an ACPX Claude run on Daytona was starting. At 2 s after `stage.workspace`, the run failed with `local_runtime_api_listener_unavailable`, and the executor persisted "same-run resume persisted" with `recoveryMode: bootstrap_retry`. After the restart, restart recovery returned `ambiguous_remote_recovery_evidence` with disposition `blocked`. The task rested blocked. Evidence in the saved database at that point: - Coordinator `phase: retryable_failure`, `attempt: 1`, `controllerGeneration: 1`, controller identity recorded, and no lease owner. - `failureDetail` recorded `recoveryMode: bootstrap_retry`, `checkpointExists: false`, `providerEventsExist: false` and `providerSessionEstablished: false`. - The run had no runner checkpoint rows, no `native.process_start_requested`, no process identity, and no runner-sourced event. It also had no PID and no provider session. **Expected behavior** Restart recovery starts the bootstrap again, as it does for a claimed run that holds a lease (#19616) and for a local run. **Steps to reproduce** 1. Start a native run on a Daytona environment. 2. Send SIGTERM to the control plane after workspace staging and before the runner is requested. The run settles as a same-run `bootstrap_retry`. 3. Restart the control plane while that retry is due. 4. See that restart recovery blocks the task with `ambiguous_remote_recovery_evidence`. Root cause: in `server/src/services/native-runtime/native-restart-recovery.ts`, the `remoteRunnerNeverLaunched` condition required `phase === "observed"` and a lease owner. The settled `retryable_failure` / `bootstrap_retry` row failed that check. The remote classifier then returned `ambiguous_remote_recovery_evidence`. Refs #19616, #19596, #19583 ## What Changed - `native-restart-recovery.ts`: `remoteRunnerNeverLaunched` now accepts one of two coordinator shapes: - an `observed` claim that holds a lease (unchanged); - a `retryable_failure` with `recoveryMode: bootstrap_retry`, run status `failed`, and no lease owner. - All other proof conditions are unchanged. The run must still have no process start, process identity, adapter, tool or runner-sourced event. It must still have no process fields, no provider session, no provider evidence and no checkpoint. The same three-attempt budget applies. - An `exact_checkpoint_resume` retry keeps its own admission path. - Admission: before claiming a settled retry, restart recovery applies the same admission as the resumption reconciler (`readNativeSessionResumptionAgent`): task pause hold, agent status, dependencies, ownership, and competing runs. An ineligible retry is deferred with `native_retry_admission_ineligible` and keeps waiting. - `doc/run-log-events.md`: note the settled `bootstrap_retry` case. ## Verification - Red, then green: `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "bootstrap retry"`. Before the fix, the proven case and the budget case returned `blocked` / `ambiguous_remote_recovery_evidence`. - New cases: the proven case gets `bootstrap_incomplete`. The coordinator returns to `observed`, `nextAttemptAt` is cleared, the run returns to `running`, and no recovery action is created. A requested process start or a committed runner event keeps the hold. A spent budget blocks with `execution_recovery_budget_exhausted`. An `exact_checkpoint_resume` retry does not take this path. - Regression: `native-runner-restart-recovery.integration`, `heartbeat/recovery`, `native-restart-recovery`, `native-bootstrap-retry.integration` and `native-remote-retry-admission`: 337 passed, 14 skipped. - Admission: a paused agent's due `bootstrap_retry` is deferred with `native_retry_admission_ineligible`. It is not started (red before the follow-up commit, green after). - Live Daytona replay of `w-20261106T234719-s12/t02`, 3 runs, ACPX Claude B1, using the recorded K4, K2 and K1 sequence. Image: the wave's own `paperclip-daytona-runner@sha256:2767e91c…`, with runner inputs pinned to its revision e482f0f632 locally. The replays ran the first commit of this PR. - Runs 1 and 3 passed R1. Run 2 passed R3: one Retry worked on the first click. - Runs 1 and 2 failed R6 because the wave-end sweep found a run lease still in `pending_cleanup`. In run 1 the environment delete also timed out. These failures are not in this change's path. - The restarts landed in other startup windows, so none hit this exact `bootstrap_retry` state. The integration tests cover that state with the same evidence as the saved failing database. - Run 2 also exposed a separate gap. A graceful drain that suspended a remote run after the runner launched (`awaiting_runner_reattach`) was blocked as ambiguous on the next boot. The Retry recovered it. That case is outside this PR. ## Risks - The proof still depends on the executor writing `native.process_start_requested` before a backend may spawn a runner. The fail-closed tests pin that gate. - Restart recovery and the same-run resumption reconciler can both see a due row. Both claim with a conditional update on phase and attempt, so only one can win. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:54:04 -06:00
DottaandClaude Opus 5.5	d26372dc8a	test(restart-chaos): add a result trigger and recovered-run comment-wake guards (#19689) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness kills the control plane at chosen moments and grades whether each task still finishes correctly > - One wave lost a user's follow-up comment after a kill that landed after the run accepted its result but before the run settled > - No existing trigger aims a fault at that moment, so the case was hard to reproduce on purpose > - The recovery path that settles such a run had no regression test that covers the saved comment > - This pull request adds a `result` trigger to the harness and a regression test for the recovery path > - The benefit is that the harness can aim faults at the result-accepted window, and a change that drops, repeats, or re-wakes a saved comment fails a test ## Linked Issues or Issue Description **What happened?** In a restart-chaos storm wave, a user posted a comment while a native run was active. A hard kill (K1) landed after the run accepted its result and before the run settled. The run settled `succeeded` and the task stayed `done`, but no follow-up run read the comment. **Steps to reproduce** The original failure did not reproduce. Three targeted waves ran 12 tasks with K1 at comment time or in the new result window. Every follow-up comment was consumed exactly once. The new regression test also passes on current `master`. The test and the trigger guard the path and do not claim a fix. **Expected behavior** A comment posted during a run is consumed exactly once after the run ends, including when restart recovery settles the run. ## What Changed - `tests/restart-chaos`: add `--triggers result` (C1 only). K1 fires 0–3 s after the run that received the follow-up proposes or accepts its result (`run.result.proposed` or `run.result.accepted`). The fault fires only while that run is still `running`: the observer checks the latest sample and rereads the runs right before dispatch. A run that settled during the delay, or a follow-up posted at rest, records a `result` miss. Add poll-driven unit tests for these cases and a README row. - `server/src/__tests__/native-run-projection-work.test.ts`: add embedded Postgres tests in which background recovery, not the executor, settles the run (projection, environment settlement, and deferred-receipt workers), with a deferred user comment and an active local lease left by the killed executor. The tests check: - The comment is promoted into exactly one follow-up run, and a second restart adds none. - A comment the run already received as a same-turn steer is not woken again. The other saved comment still is. - A queue whose only comment was steered stays cancelled, and no follow-up starts. - A Retry click does not consume the comment a second time. A click that observed the source is stale. A click that observed the follow-up joins it. ## Verification - `cd tests/restart-chaos && npx vitest run --config vitest.config.ts` (96 tests pass) - `cd server && npx vitest run src/__tests__/native-run-projection-work.test.ts` (54 tests pass) - Live, local, ACPX Claude, Codex and OpenCode: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --workloads C1 --triggers comment --faults K1 --adapters codex,claude,opencode --tasks 4 --concurrency 2 --rc06-masked` passed 4 of 4. Two waves with `--triggers result` consumed each follow-up exactly once. One task in those waves found a separate defect: a kill 1.4 s after the result commit left the run `running` behind the dead controller's 20-minute finalization lease. A separate pull request fixes that defect. ## Risks - Low risk. These are test and harness changes only. No production code changes. - The `result` trigger depends on the observer's poll interval (about 3 s), so the exact kill point within the window varies. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:53:58 -06:00
DottaandPaperclip	5e4d24b6c8	fix(recovery): preserve admitted workspace on native restart (#19693) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A Retry can admit a new workspace before native provider work starts. > - The native execution input records that admitted workspace. > - A server restart must recover the same run under that recorded contract. > - Retry readiness previously evaluated fresh admission again after the run had started. > - This could replace the workspace and make the retained lease fail its correct identity checks. > - This change keeps the admitted workspace and leaves reattachment authority with the existing lease and controller checks. ## Linked Issues or Issue Description **What happened?** A native run started by Retry could do useful work, then fail after a server restart. Preparation evaluated the original fresh-workspace receipt again. It reserved a replacement workspace under the Retry intent ID. The existing runner still owned the original workspace, so setup failed with `native_remote_recovery_lease_mismatch`. **Expected behavior** Restart recovery must retain the workspace recorded in the same run's native execution input. A later configuration or admission decision must not replace it. **Steps to reproduce** 1. Admit a remote workspace for a Retry run. 2. Persist its native execution input and keep its original controller and environment lease records. 3. Recover the same run while its canonical Retry receipt selects fresh admission. 4. Preparation must reach the original lease with the original workspace. Related: #19340 preserves verified Retry workspaces. This change handles an already admitted native run instead of selecting a workspace for a new turn. Public duplicate searches for admitted/fresh workspace restart found no matching repair. ## What Changed - Stop fresh admission when a parsed native input names a real workspace in the run's company. - Validate the company, run, agent, and task binding before using that input. - Keep synthetic run-ID bootstraps on the existing admission path. Hold missing real bindings. - Test both recovery-action and ordinary user-Retry receipts against real workspace, controller, and lease rows. Check malformed, foreign, missing and opaque bindings, synthetic bootstrap behavior, and no-files API controllers. - Align the resume-owner fixture with the real transient run-ID binding. Give each invocation a unique issue label so a CI test retry cannot mask the original failure. ## Verification - RED on unchanged production: the new reservation regression failed with `native_remote_recovery_lease_mismatch` instead of reaching the admitted lease. - Review RED: an opaque task-workspace ID reached the UUID query. The new guard returns the controlled missing-admission error before the query. - Review RED: a valid no-files Cursor controller was treated as a missing task workspace. The existing task-workspace projection now keeps that controller on its prior path. - Clean final production-source regression: both restart cases passed in a fresh embedded PostgreSQL cluster. These cover both receipt paths and all binding, bootstrap and API-controller controls. - The initial focused checks passed 11 cases. Seven existing ordinary Retry and environment selection cases also passed. A later combined final-base run had 17 assertions pass but failed workspace teardown at 90 seconds and task-Retry setup at 30 seconds. That attempt is retained as a failure; the two new restart cases then passed alone with exit 0. Earlier PostgreSQL support-probe failures are also retained as setup failures. - The worktree's own Runner, shared and SDK dependencies were built. All seven server typecheck layers passed for the final production source. The final additional commit changes only the existing resume-owner fixture. - Required CI found an invalid fixture binding in [server shard 11](https://github.com/paperclipai/paperclip/actions/runs/42322379917/job/128250151095). The first attempt failed before lease acquisition because the fixture named a random workspace with no row. The automatic test retry then failed on its fixed issue label. The fixture now uses the transient run-ID contract and a unique label. Its exact original acquisition and owner-release assertions pass locally with exit 0. - Whitespace and added-code-comment checks passed. - Final commit `f0e41e8c6b949f43863ec60fc7b96b97025eafc6` passed required [CI](https://github.com/paperclipai/paperclip/actions/runs/42323826555): 56 successful checks, four intended skips, one neutral result, and no pending or failed checks. Greptile and Ponytail both gave this exact head 5/5 with no actionable findings. No review threads remain open. Hosted deployment and original-task recovery remain separate from local proof. ## Risks Malformed or missing native admission records now hold rather than selecting fresh work. Valid initial admissions and synthetic run-ID bootstraps retain their prior path. The change does not grant reattachment authority or settle provider effects. Existing controller, workspace writer, lease, Stop, and trust checks still apply. No API, schema, or configuration contract changes. ## Model Used OpenAI GPT-6 family through Codex. The runtime does not expose the exact serving model ID or context window. No model override was used. The assistant used reasoning, code editing, shell tools, and local tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 19:37:48 -06:00
DottaandClaude Opus 5.5	9a52a2dfeb	fix(native): retry ACPX session.open startup timeouts and give cold Claude a 60 s bound (#19661) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start an ACPX provider through Runner. `session.open` opens the provider session before any prompt. > - On a loaded host, a cold ACPX Claude `session.open` took longer than its 30-second bound. The controller reported `runner_local_connect_failed: PRP command session.open timed out`. > - The server classified this as the generic `native_session_interrupted` with `ambiguous_state`, so the task was blocked at once. A person had to click Retry, sometimes several times. In one restart-chaos storm, one task needed three Retry clicks, and on another task both Retry runs timed out too. > - The server already has a bounded same-run bootstrap retry for failures that come before provider work (#18854). It did not recognize this timeout. The 30-second bound also cut off normal slow startups. > - This pull request gives cold ACPX Claude a 60-second `session.open` bound, like Pi and OpenCode (#19487). It also retries a startup `session.open` timeout automatically on the same run, from clean state. > - The benefit is that a slow cold start on a busy host completes, or retries itself, instead of waiting for a person. ## Linked Issues or Issue Description Refs #18854 (bounded bootstrap retry for definite `session.open` failures), #19487 (60-second cold OpenCode admission), #19665 (the bootstrap proof that this change reuses). **What happened?** ACPX Claude runs failed with `runner_local_connect_failed: PRP command session.open timed out` about 35 to 40 seconds after they started. The coordinator recorded `native_session_interrupted` with `recoveryMode: "ambiguous_state"` and `providerSessionEstablished: false`. The task was blocked with a board-owned action. No provider event and no checkpoint existed. The host 1-minute load was 17 to 44, sometimes with no restart nearby. **Expected behavior** A normal slow cold start completes. A `session.open` that still times out during runner startup, with no provider event, no checkpoint and no turn, retries on the same run with backoff. At most two automatic attempts follow the first. A run where provider work may have started is never retried automatically. **Steps to reproduce** 1. Run ACPX Claude tasks on a host with a 1-minute load of about 25 or more. 2. Observe a run fail after about 35 to 40 seconds with `PRP command session.open timed out`. 3. Observe the task blocked with `ambiguous_state` and a manual Retry. **Paperclip version or commit** `master` at `6c7cc2c5d6`. ## What Changed The bound (evidence below): - `coldAdmissionTimeoutMs` gives cold ACPX Claude 60 seconds (live adoption keeps 30 seconds). The runner's Rust sidecar `session_open_timeout` and the sidecar's provider handshake (`acpxSessionHandshakeTimeoutMs`) give Claude and Pi 60 seconds too. All three bounds must agree, or the shortest one fails first. Claude's warm run attachment uses the same cold budget, like Pi. The automatic retry: - `nativeSessionFailureSourceCode` returns `provider_session_open_timeout` only for exactly `<startup failure code>: PRP command session.open timed out`. The startup failure codes are `runner_local_connect_failed`, `runner_direct_wss_failed` and `runner_ingress_unavailable`. The transport uses them only before startup completes. After startup the code is `provider_transport_failed`, which is unchanged. - `nativeProviderRecoveryEvidence` treats the cause as pre-session. Durable evidence still decides: a bootstrap retry needs no provider event (`session.started`, `turn.started` and the others) and no checkpoint. The existing budget applies: three attempts with backoff of about 30 seconds, then about 60 seconds. When the budget is used, the task is blocked with the existing "Provider startup failed on 3 attempts before any provider work began" action. - A timeout is indeterminate: the runner can open a provider session after the controller stops waiting. Live runs showed both leftovers, so the retry now starts from clean state: - `prepareNativeSessionOpenTimeoutRetry` (server) runs when the failure settles. It quarantines a local runner root only when the restart-recovery proof (`runnerdStateProvesIncompleteBootstrap`, #19665) shows that no turn began and every recorded process is gone. Any other root, and any remote runner, stays `ambiguous_state` as before. - `resetAbandonedBootstrapRecord` (runner ACPX adapter) runs on a fresh open (no runner-recorded identity). If ACPX has a saved record for the session key with no messages, no prompt and no events, it is marked `reset_on_next_ensure`. Claude saves nothing before the first prompt, so reusing such a record fails the first turn with `session_resume_required`. A record that holds any conversation is unchanged. - A chat retry of an exhausted `provider_session_open_timeout` run is accepted like the other pre-provider causes. - `doc/execution-semantics.md` and the Runner README describe the cause and the bound. ## Verification **Measured startup distribution** (ACPX Claude `runner.session.bootstrap` / `runner.session.startup` spans from restart-chaos run events, 1-minute load 8 to 44): | Source | Bound | Successful cold bootstraps (s) | Failures | | --- | --- | --- | --- | | Earlier storm evidence | 30 s | 2.9, 4.7, 7.4, 8.6, 8.7, 26.2, 27.4, 27.6, 29.2 | startups ended at 34.9 to 40.3 s (the 30 s `session.open` bound plus spawn and `run.prepare`) | | This branch, wave s3601 | 60 s | 6.3, 7.8, 8.1, 13.4, 14.4, 22.3, 22.8, 22.9, **37.3** | 0 `session.open` timeouts | | This branch, wave s3602 | 60 s | 13 samples, p50 10.5, p90 17.9, max 25.7 | 0 `session.open` timeouts | | This branch, wave s3610 (final head, load 28 to 42) | 60 s | 10.4, 11.3, 11.7, 12.0, 13.3, 14.2, 29.6, **30.4**, **30.9** | 0 `session.open` timeouts | Under load, successes clustered just under 30 s and failures sat just above it, so the 30 s bound was cutting the distribution off. With 60 s, cold starts of 30.4, 30.9 and 37.3 s succeeded that the old bound would have failed. 60 s is about 1.6 times the largest observed success, and it matches Pi and OpenCode. **Live restart-chaos waves** (local, ACPX Claude only, 4 in flight, `--rc06-masked`, real provider, K1/K2 restarts): - s3601 (8 tasks, load 19 to 32): 7/8 PASS. No `session.open` timeout. The one failure is an R3 Retry signature after K1 (`native_restart_recovery_blocked`), not this cause. - s3602 (8 tasks, load 17 to 39): 8/8 PASS. No `session.open` timeout. Three Retry clicks were for `native_provider_terminal_failed` (`provider_turn_lost_on_restore`), a different restart signature. - Forced timeout, s3609 (4 tasks, B5, K2): a local, uncommitted fault injection gave every other cold startup a 1.5 s `session.open` deadline. Result 4/4 PASS. Run 4fe6537a timed out, retried itself on the same run, and then succeeded through four K2 restarts with no person involved. Run 02799a85 hit the injection three times in a row; the budget stopped it after attempt 3, and one Retry completed it. - Before the two clean-state fixes, the same forced wave (s3607 and s3608) showed the retry reusing leftover state: `turn.start` failed with `session_resume_required`, and then "ACPX session is not open". Those runs are the red evidence for the two clean-state fixes. - Final head, s3610 (8 tasks, K1/K2, no injection, load 28 to 42): **8/8 PASS**, 0 `session.open` timeouts. The one Retry click (t06) was for `provider_turn_lost_on_restore`, a different restart signature. **Tests:** - Red before the fix: `native-startup-retry.integration.test.ts -t timeout` (embedded PostgreSQL, real executor, real resume claim). The 3 retry cases failed with `native_session_interrupted` / `ambiguous_state`. - Green: `native-startup-retry.integration.test.ts` passes 36/36. New cases: - for each startup failure code, a timeout retries on the same run with backoff, keeps the task `in_progress` with an agent-owned `resume_native_run` wake, and blocks after attempt 3 with a board-owned action; - no automatic retry when `session.started` or `turn.started` is durable, when the code is `provider_transport_failed`, or when another command timed out; - a real runner root with an opened session and no turn is quarantined; a root with a turn command stays `ambiguous_state` and intact; a remote runner is never prepared. - `codex-runtime-adapter.test.ts`: 113 pass. New cases: a 45 s delayed handshake is admitted for Claude and rejected for Codex; a never-prompted record is reset only on a fresh open; records with messages, a prompt or events are kept. - `abandoned-bootstrap-record.test.ts` (real `createAcpRuntime` with `createRuntimeStore` and a controlled ACP peer that, like Claude, cannot load a never-prompted session): without the reset, a fresh open reuses the record and the first turn fails with SESSION_RESUME_REQUIRED after `session/load`. With the reset, the provider receives `session/new` and the turn completes. - `vitest run src/drivers/acpx src/cli` in the runner: 107 files, 1,935 tests pass. - `native-session-executor.test.ts -t "session.open|startup|bootstrap"`: 144 pass. `runnerd-codex-transport.test.ts -t "cold process admission"`: 4 pass. `cargo test -p paperclip-runner-core --lib admission_budget` passes. - `npx tsc --noEmit -p server` and the runner `tsc -p tsconfig.test.json` pass. ## Risks - **This changes Runner Rust code. The Daytona runner image must be rebuilt after merge** for remote sandboxes to get the 60-second sidecar bound. Until then, a remote Claude `session.open` still fails at 30 seconds in the sidecar (as a definite failure, which already retries). A remote timeout stays operator-owned, because a remote root cannot be proven clean here. - A cold ACPX Claude start that hangs now fails after about 60 seconds instead of 30 seconds. - A timed-out `session.open` may have created an empty provider session. No prompt was sent. The retry quarantines that runner root and asks ACPX for a new session. - Same-run retries of a fresh-attempt Retry run can still fail with `isolated_task_directory_binding_mismatch`. This is RC-35 (workspace re-preparation), which has its own fix in progress. - No schema or API change. The new error code appears on runs and in run error codes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 19:37:11 -06:00
DottaandPaperclip	b94963f91e	fix: retry temporary repository catalog failures before dispatch (#19690) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A task must check repository access before it starts agent work. > - Repository discovery can fail while the user's existing grant remains valid. > - Discovery previously replaced a known temporary provider refusal with an empty catalog. > - The task then reported lost access and did not schedule recovery. > - This change keeps temporary discovery failures distinct and uses the existing bounded retry owner. > - Each retry must pass a fresh access check before it can use the saved repository. ## Linked Issues or Issue Description **What happened?** A task with a saved catalog repository stops with an access denial when GitHub returns an exhausted-quota response or HTTP 503. The task does not start provider work, but the discovery layer discards the temporary failure and returns an empty catalog. **Expected behavior** Known temporary failures before provider dispatch use the existing bounded infrastructure retry. The successor checks the same responsible user's current grants and reads the saved files only after fresh discovery succeeds. A completed denial remains terminal. **Steps to reproduce** 1. Bind a task to an authorized catalog repository with saved local changes. 2. Make its authorized GitHub discovery return HTTP 403 with zero remaining quota, or HTTP 503. 3. Let discovery recover before the existing retry deadline. 4. Observe one successful successor using the original repository bytes. **Paperclip version or commit** Reproduced on master at `b8d94b3ad1`. A search found no existing PR for this temporary catalog failure. ## What Changed - Keep a safe typed failure for known HTTP availability and quota responses. Honor the later provider deadline and bound it to one day. - Propagate that failure only when fresh discovery cannot find a required repository. Preserve successful partial catalogs and the existing connection setup error contract. - Classify the failure only before provider dispatch. Reuse existing bounded retry, interaction ownership, and bootstrap settlement rules. - Keep current grant checks, saved repository selection, normal Stop, and provider-started failure policy. - Keep provider deadlines when accepted interactions use their own retry owner. Recognize secondary HTTP 403 rate limits with valid retry advice while keeping ordinary permission denials terminal. This matches [GitHub rate-limit guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api). - Keep direct repository request responses outside the automatic bootstrap retry contract. Test successful partial discovery and missing-required-repository recovery with two independent eligible connections. - Document task repository discovery recovery. ## Verification - Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual HTTP 403 and503 discovery responses, started no adapter, and found zero successors instead of one. Both cases failed before the production change. - Behavioral GREEN: both cases waited for the real retry deadline, used the normal queue worker, ran the controlled adapter once after fresh discovery, and read the original saved bytes. Two cases passed in 74.73 seconds. No external model or provider request was made. - Deadline RED/GREEN: a combined Retry-After and quota-reset response initially selected the earlier deadline. The regression now selects the later deadline. - Initial repair at `d8d4714a18` (same production source as `018e8c52d345d79972d7907468080dc9908c299f`): seven affected files passed, 69 cases in 185.16 seconds. That test-only cleanup then passed all five real-database acceptance cases in 134.76 seconds. - Commands: `pnpm exec vitest run` with `heartbeat-repository-catalog-recovery.test.ts`, `github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`, `project-repositories-persistence.test.ts`, `task-workspace-repositories-persistence.test.ts`, `github-repository-catalog-failure.test.ts`, and `services/heartbeat/retries.test.ts` from the server source tree. The acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final eight-case file passed fresh at `e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real owned embedded PostgreSQL. It covers automatic recovery with saved bytes, genuine denial, provider-started classification, normal Stop, the direct HTTP route, and both partial-catalog outcomes. - Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both reported P1s reproduced RED. The real database interaction scheduler chose a retry after 24 seconds before the provider deadline of one hour. A secondary HTTP 403 with valid Retry-After lost its retry classification. The narrow fixes then passed 44 cases across scheduler, header, and metadata-contract files in 37.66 seconds. - A real direct request through the task workspace router and normal error middleware reproduced a 500 response instead of the existing 403. Required-ID retry mode now applies only to admission and its rechecks. The route and both two-connection partial-catalog cases then passed in 29.09 seconds. Both partial cases use actual owned grants and credential bindings. - All seven server type layers passed again at final head `e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer passed again after the test-only cleanup. Environment: macOS arm64, Node 26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18 on loopback. The current checkout's shared packages, plugin SDK, and Runner were built first. - One local acceptance run under concurrent owned test/typecheck load hit the unchanged 15-second Stop-case timeout. Its four other cases passed. The unchanged complete file then passed serially. The timeout receipt remains recorded; no assertion or timeout was weakened. - A provider-started failure retains the existing generic adapter retry policy. It does not receive the catalog no-provider certificate or a provider retry deadline. Fixture cleanup cancels its scheduled retry through the normal owner. - Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59 successful checks, 3 allowed skips, no pending or failed check, and Greptile 5/5 with no open findings. [Completed CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977). - Additional local diagnostic for the late recovery-owner advisory: on unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted` with attempt 3/max 2. The normal stranded sweeper then escalated once, created no extra run, and started no adapter, even after catalog access recovered. One case passed in 17.32 seconds. The existing failed-predecessor path delegates to that bounded scheduler and returns; it does not enter a new generic retry budget. No production registry change was made. This diagnostic was not a CI case. Its local harness is preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on `codex/catalog-exhaustion-acceptance`; its production source is identical to `e5b2efab`. - Module boundary and whitespace checks passed. ## Risks - A temporary failure from another eligible connection can delay a truly missing repository for the existing limit of two attempts. It cannot grant access. - Unknown network errors and genuine access denials retain their existing behavior. This change handles known HTTP failures only. - Historical provider quota causality is unproven. This repair has a separate local producer regression. It does not rewrite past failures or queued user intents. - Acceptance uses real local database and heartbeat owners with controlled HTTP and adapter fixtures. It does not claim a live native-provider recovery. ## Model Used OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository tools, code execution, and controlled local PostgreSQL tests. The host did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 19:29:40 -06:00
DottaandPaperclip	254b44f72b	fix(remote-workspaces): recover retired roots that cannot be renamed (#19674) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces must retain saved files across process and sandbox replacement. > - Restore uses identity receipts to distinguish a retired owned root from an unrelated directory. > - Some overlay filesystems reject a directory rename with EXDEV even within one mount. > - That error currently prevents a valid saved workspace from being restored. > - This pull request replaces only a verified retired root in place when that rename fails. > - Runtime state stays in place. Each restore holds a root-specific OS lock. > - A durable workspace epoch fence rejects delayed older commands. A pending marker keeps interrupted cleanup recoverable. ## Linked Issues or Issue Description **What happened?** Workspace restore fails with `remote_workspace_restore_failed: stage=publish code=EXDEV syscall=rename` when the retired directory comes from a lower overlay layer. Retry repeats the same rename. **Expected behavior** Restore should publish the verified saved workspace and preserve runtime state. An interrupted restore should remain safe to retry. **Steps to reproduce** 1. Create an owned retired root with a valid identity receipt, runtime state, and stale files. 2. Make directory rename return EXDEV for the existing lower-layer root. 3. Restore a saved workspace through `restoreRemoteWorkspaceDirectory`. 4. Repeat with an interruption before the new restore receipt is published. **Paperclip version or commit** The regression fails before this change on base `a2aa69bc6230c8b81de3588d7d5571b38db4dd04`. **Deployment mode** Remote sandbox with an overlay filesystem. Local regression: macOS arm64, Node 26.4.0, source checkout. Refs #19231. Refs #19251. Those changes establish the retained-root restore path. This fix handles its EXDEV publication failure. ## What Changed - Keep the EXDEV fallback limited to a verified retired workspace root. Preserve runtime state and the existing ownership checks. - Hold a private SQLite OS lock across receipt checks, file replacement, cleanup, and receipt publication. Keep the lock file outside the replaced root. Process exit releases ownership; a lost reply does not. - Pass the existing workspace authority epoch from the real restore caller. Write and sync a separate epoch fence before root changes. Reject lower epochs, conflicting identities within one epoch, and missing epochs on fenced roots. The fence survives SQLite rollback after process death. - Publish a pending marker without a completed identity until stale preparation-receipt removal and directory-mode cleanup finish. A retry can finish that cleanup. A higher authorized epoch can then restore its selected save. - Preserve later file and permission edits when adopting a completed restore. Refuse unsafe lock files and malformed authority records. Document compatibility and the stop-evidence boundary. ## Verification - RED before the original fix: real restore commands failed with EXDEV. The test preload changes only the kernel rename result. - RED before the review fixes: a live command with an unconfirmed timed-out reply overwrote edits made after retry. Tests also reproduced incomplete cleanup after receipt publication. - RED before the authority fence: a delayed older command replaced a newer completed restore. It also passed after the newer command was killed before root mutation. The final test uses real separate Node processes, OS lock contention, SIGKILL, and a durable on-disk fence. - Final local checks: 142 transport filesystem tests, 16 real PostgreSQL restore-caller cases, and all seven server typecheck layers. The caller cases verify the published receipt uses the current workspace epoch. - Late review RED: an actual separate SQLite owner made the real PostgreSQL restore caller mark an expired lock wait non-retryable. Corrupt fence and receipt bytes also produced a raw syntax error without the required refusal reason. Both regressions now pass. Only the test clock advances the lock deadline; actual OS lock contention and the production caller run unchanged. - Lock contention stays retryable only under the existing same clean writer, epoch, and snapshot proof. Four negative cases change those fields and remain non-retryable. After the owner stops, normal restore returns the useful saved files. Corrupt authority records stay untouched and non-retryable. - Safety cases cover same and different authorized identities, process-death lock release, an unrelated root making progress, pending cleanup interruption, malformed or missing fences, invalid input, and unsafe lock parents/files. - The first final transport attempt passed 139 cases and failed only because the restricted shell could not open a Unix socket (`listen EPERM`). The full suite was then run with the required local socket permission. No production change was made for that setup failure. - Reproduce the filesystem checks from `server/`: `../node_modules/.bin/vitest run --config vitest.config.ts src/services/remote-workspace-transport.test.ts`. Run the caller file with `-t "restores a newer save over this workspace|preserves the published workspace path"` and the repository test PostgreSQL setup. Run `node scripts/typecheck-layers.mjs` from `server/` for all seven layers. - Agent preflight, module boundary audit, and `git diff --check` passed. No added code comments or private issue references occur in the patch. - Local EXDEV proof uses macOS arm64 and Node 26.4.0. It does not claim a new Linux overlay-kernel run or hosted recovery. Final source: [`8092cc6b2ec448ff372337b1e7a43854eae7bb08`](https://github.com/paperclipai/paperclip/commit/8092cc6b2ec448ff372337b1e7a43854eae7bb08). [Exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/42321313715) completed with 59 successful checks, three conditional skips, and one neutral check. Greptile gave this exact head 5/5. The independent review passed. No review threads remain open. ## Risks The EXDEV path changes files in place. It must finish before execution admission. Participating restore commands use a private permanent SQLite lock file and a separately synced epoch fence. The qualified sandbox Node runtime must support `node:sqlite`. Lock contention has a 120-second limit and does not clear live ownership. A legacy command that never acquired this lock remains outside its protection. Hosted recovery must confirm that command stopped; a timeout is insufficient. The new fence does not backfill that evidence. A conflicting identity in the same epoch or a malformed fence stays blocked. A higher epoch can recover an interrupted pending restore only with the existing retired-root replacement authority. No controller database schema, execution permission, or provider-state guard changes. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) authored the recovered patch. OpenAI Codex, GPT-6 family, recovered, reproduced, repaired the review findings, and verified the change with filesystem tests, shell tools, and code execution. The Codex exact serving model ID and context-window size are not exposed in this session. No value is inferred for them. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-06 19:12:21 -06:00
DottaandClaude Opus 5.5	06bbcea12f	fix(budgets): price lost receipts and stop restart-derived accounting from hard-stopping companies (#19673) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission and run accounting decide whether any task in a company may start > - #19497 made pending accounting an admission hold and settles a stopped run's lost receipt as an unpriced charge > - Restart-chaos waves still stopped whole companies. A lost Codex receipt with complete token counts settled as unpriced because the rate card only prices complete receipts. Under the default blocking policy that hard-stopped the company at 0.38 of 3000 cents. In another storm, a resumed run's continuation was refused by its own task's recovering-accounting hold. Separately, unpriced charges with zero tokens or a known partial cost hard-stopped a company at 120 of 2000 cents > - This pull request prices a lost receipt's observed usage from the rate card, lets a task's continuation pass its own recovering accounting, and stops restart-derived or evidence-free unpriced charges from tripping the blocking policy > - The benefit is that a restart no longer pauses a company by itself, while genuinely unpriced usage still blocks ## Linked Issues or Issue Description Follow-up to #19497. Refs #19606. **What happened?** - Wave A (K6 crash loop): native Codex run `5037b8d1` was killed mid-turn. Its receipt was lost, but `usage_json` held gpt-6.1-sol with 28690 input and 220 output tokens. The charge was written `cost_status=unpriced`, `pricing_provenance.source=unknown`, because `priceCodexReceipt` returns early when `!receipt.complete`. 0.2 s later the default `unpricedUsagePolicy=block` opened a hard incident at 0.38 of 3000 cents. The company was paused, and 6/6 tasks were blocked with `canRetry=false`. - Wave B (K1 storm): Claude run `2a7a8abf` resumed after the restart. `reserveRunBudget` refused its interaction continuation with "New work must wait while a native run recovers its unfinished accounting", so the task could not settle its own recovery. A 0-token unpriced charge and an unpriced charge with a known 0.81-cent cost then hard-stopped the company at 120 of 2000 cents. **What did you expect?** A restart-lost receipt with known numbers is priced as an estimate, and no restart-derived accounting state pauses a company by itself. A task's own continuation is never held by its own accounting. Real usage above the limit, and genuinely unpriced usage, still stop work. **Steps to reproduce** 1. Give a company a hard-stop budget with the default unpriced-usage policy. 2. Kill a native Codex turn after it has reported tokens, so the final receipt is lost and native recovery ends in terminal failure with a verified stop. 3. Observe an unpriced charge for known tokens, a hard incident, and a paused company. ## What Changed - `run-cost-accounting.ts`: when a lost receipt settles and its usage has known tokens and no known cost, price it with `priceCodexReceipt`/`priceAnthropicReceipt` as a complete per-run checkpoint. The charge is `estimated`, its provenance keeps the rate-card source, and its evidence says "final receipt lost; observed usage is a lower bound". The estimate is also written to the run's usage so a later receipt reconciles against it. Zero observed tokens, and per-model usage, stay unpriced (unknown is never shown as zero, and aggregate totals never take one model's rate). - `budget-exposure.ts`: - The pending/recovering accounting hold excludes the run being admitted and earlier runs of the same task (same `native_issue_id`). A task's continuation and finalization can therefore settle its own accounting. Other tasks stay held, and those runs still count as spend. - The blocking unpriced-usage policy no longer counts these unpriced charges: the settled final charge of a lost receipt (restart-derived, any usage; later growth still counts), a charge with a known partial cost, and metered API or subscription billing with no observed tokens. Unknown billing with no tokens (for example Cursor Cloud) and complete-receipt usage that has no price still block. - `budgets.ts` / `budget-reservations.ts`: admission accepts the admitted `runId` and passes it to the exposure read. `reserveRunBudget`, the final dispatch recheck and the read-only admitted recheck all supply it, so a continuation that reserves can also dispatch. - Daytona wave s12 (run f1d98917, a blocked remote recovery with a 0-cent held reservation and every lease destroyed) is closed by #19606. A Retry for that same task passes the hold through the same-task exclusion here. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres): - Lost Codex (gpt-6.1-sol) and Claude (claude-sonnet-5) receipts with observed tokens settle as `estimated` rate-card charges with the lower-bound evidence. The company stays active and admission is clear under the blocking policy. - A lost receipt for a model with no rate card stays unpriced but opens no incident and no pause. - A task's continuation reserves budget past its own recovering accounting, while another task's run is still held. - Unpriced charges with no tokens on metered billing, or with a known partial cost, do not hard-stop. - Complete-receipt usage with no price still blocks. - The six new cases fail on master and pass with this change. - Existing budget and accounting suites pass locally: budget-enforcement-scoping, budgets-service, heartbeat-cost-accounting, run-cost-accounting, native-stop-accounting, cost-data-quality, budget-live-attempts, budget-live-usage. Server typecheck passes. ## Risks - A lost receipt's estimate is a lower bound. The provenance says so, and a later complete receipt still reconciles. - Restart-derived unpriced charges no longer trip the blocking policy. Usage lost with a killed provider is not counted against that policy, which is consistent with budget enforcement being best effort. Unknown-billing and complete-receipt unpriced usage still block. - The same-task exemption relies on `native_issue_id`. Legacy runs without it keep the old behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:55:26 -06:00
DottaandClaude Opus 5.5	785ead5c7a	fix(tasks): hold a Retry while the task's own native session close is retained (#19609) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:52:46 -06:00
DottaandClaude Opus 5.5	29e05f7e06	fix(heartbeat): sweep orphaned run scratch directories (#19671) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each heartbeat run gets a scratch directory in the system temp folder > - Each process attempt makes a new directory with a marker file > - Cleanup removes only the directory of the current attempt > - A restart kills the old attempt, so its directory stays forever > - One restart storm left 83 directories across 78 runs > - This pull request sweeps those directories when the run ends and at startup recovery > - The benefit is that temp space no longer fills with leftover directories ## Linked Issues or Issue Description **What happened?** Run scratch directories made before a restart are never removed. A restart storm left 83 `paperclip-run-*` directories across 78 runs. Each holds only `.paperclip-run-scratch.json`. **Steps to reproduce** 1. Start a run that makes a scratch directory. 2. Kill the server process and start it again. 3. Let the run finish. 4. List `$TMPDIR/paperclip-run-*`. The old directory is still there. **Expected behavior** The old directory is removed after the run ends, or at startup. ## What Changed - Add `sweepOrphanedRunScratch` in `server/src/services/run-scratch.ts`. - The sweep removes a directory only if it has a valid regular-file marker and its run is terminal in this instance database. It keeps a directory whose run it cannot see, because another instance can share the temp root. - The sweep keeps active runs, unmarked directories, symlinks, and directories outside `paperclip-run-*`. - The sweep checks each directory again before removal and bounds the scan (500, with a rotating start so kept entries do not starve later ones) and removals (100). A per-run sweep filters by directory name first. Marker run ids that are not UUIDs never reach the database query. - Call the sweep after the current attempt's cleanup in `heartbeat.ts`, for that run only. - Call the sweep in `reapOrphanedRuns` (startup recovery) with run status from the database. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/run-scratch.test.ts` - New tests cover: leftover directories of a terminal run are removed, an active run is kept, an unmarked directory is kept, a symlink is not followed, and removals are bounded. ## Risks - Low risk. The sweep acts only on marked directories of terminal runs. - A directory left by a previous attempt could hold a live grandchild process. The sweep does not check old process groups. The run is already terminal. ## Model Used - Claude, claude-opus-4-5 class model via Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 18:51:31 -06:00
Nicky LeachandClaude Opus 5.5	0b73c805e8	feat(environments): fill an unset instance default with a managed plugin environment (#19551) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document, which can declare plugin environments next to the managed sandbox > - A platform that runs managed instances wants a new instance to start with one of those plugin environments as its default, without changing the default of any instance that already has one > - #19632 made boot fill the instance default only when it is unset, local or deleted, in one locked transaction, and removed the revert, so every existing managed instance keeps a set default > - This pull request adds an `instanceDefault` flag to a plugin entry. That entry fills an unset default through the same locked fill, before the managed sandbox does > - The fill follows the same readiness rule as an admin's default write, so it never moves ready agents onto a plugin that has not prepared them > - The platform also needs a way to know whether a release understands plugin entries, so the release now ships a small capabilities file > - The benefit: new instances get the plugin environment as their default, and existing instances and admin choices are never changed ## Linked Issues or Issue Description Refs #19632 (the locked fill-if-unset rule this builds on), #19508 (plugin entries in the managed config) **Subsystem affected** Server environments: the managed config parser and the managed environment bootstrap. **Problem or motivation** The managed config can declare plugin environments, but it can't make one the instance default on a new instance. An external platform also has no way to tell whether a given release supports plugin entries before it sends one. **Proposed solution** - Add `instanceDefault: true` to at most one plugin entry. Right after that entry's boot ensure succeeds, it fills the instance default through `fillUnsetInstanceDefault`. The managed sandbox fill runs after it. - On a new instance the plugin environment wins. On an existing instance the default is already set, so neither fill changes it. - The fill declines while an active agent that follows the default hasn't been prepared by that plugin. This is the rule the instance-settings route enforces. When the plugin fill declines, the managed sandbox fills as usual. - Ship `server/src/services/managed-config-capabilities.json`, which lists `pluginEnvironments`. **Alternatives considered** - A recorded one-time decision with a retry worker. It needed pending states and extra checks for many writer and trigger combinations. Filling only an unset default, under the existing lock, needs none of that. - Rejecting the fill whenever agents exist. That is stricter than the admin route and blocks harmless cases, such as agents that are paused or already prepared. **Roadmap alignment** This extends existing managed-environment behaviour. It adds no UI. ## What Changed - `server/src/services/managed-config.ts`: - `ManagedPluginEnvironmentSpec.instanceDefault` (boolean, default `false`). - The parser refuses `instanceDefault` on more than one plugin entry, on the sandbox entry (even `false`), or as a non-boolean. - `MANAGED_CONFIG_CAPABILITIES` and `MANAGED_CONFIG_CAPABILITIES_SCHEMA_VERSION` constants. A plugin entry is refused unless `pluginEnvironments` is listed. - `server/src/services/managed-config-capabilities.json`: `{ "schemaVersion": 1, "capabilities": ["pluginEnvironments"] }`. A test pins it to the constants and to this literal contract. - `server/src/services/managed-environments.ts`: - New `fillPluginInstanceDefault`. It runs right after a successful boot ensure of the flagged entry and logs a fill or a decline. Recovery after boot never fills. - The managed sandbox fill now runs once, after the plugin entries. - `server/src/services/managed-instance-default.ts`: `fillUnsetInstanceDefault` runs `agentsNeedSetupForEnvironmentChange` inside its locked transaction and returns `declined: "agents_need_setup"`. A sandbox target never needs agent setup, so the managed sandbox fill behaves as before. - `server/src/services/plugin-environment-driver.ts`: the readiness query is factored into `agentsNeedSetupForEnvironmentChange`. `assertAgentsReadyForEnvironmentChange` wraps it and still throws the same conflict. - `doc/DEVELOPING.md`: documents `instanceDefault`, the fill order, the readiness decline, and the capabilities file. ## Verification - `cd server && npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres), under `instanceDefault (fill only an unset default)`: - A new instance gets the plugin row as its default. - An existing sandbox default is kept, and so is an admin's choice. - If the plugin was down at the first boot and the sandbox filled, a later recovery and boot keep the sandbox default. - An admin-cleared or local default is declined while a ready or verifying agent follows the default without the plugin's setup. The managed sandbox then fills it. - It fills when the agents that follow the default are paused or already prepared, or are pinned to another environment. - Without the flag, nothing happens. - A new agent that follows the default gets the plugin in `requiredPluginIds`. - `npx vitest run src/__tests__/managed-instance-default.test.ts` (embedded Postgres): the earlier fill, lock and overlap cases pass. A sandbox fill also succeeds while active agents follow the default. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts`: - The fill order is plugin then sandbox. - When the plugin is down there is no fill, and recovery never fills. - A failed ensure gives only the sandbox fill, and a failed fill keeps the ensure counted. - When the plugin fill declines, the sandbox still fills. - `npx vitest run src/__tests__/managed-config.test.ts src/services/managed-config-capabilities.test.ts`: parser refusals and the capabilities contract. - `npx vitest run src/services/managed-environments.test.ts src/__tests__/instance-settings-routes.test.ts src/__tests__/environment-routes.test.ts`: pass. The route still throws the same readiness conflict. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - An instance whose default is unset at the first boot after this release gets the plugin environment, if its agents are ready for it. That covers a new instance, and an instance whose admin cleared the default or chose the local environment. This is the intended behaviour. - The readiness check doesn't lock agent rows. If an agent becomes ready during the boot fill, it can miss the check, which is the same race the admin route has. Agents created after the fill are prepared for the plugin, because setup reads the current default. - If the plugin is down at a new instance's first boot, the instance keeps the managed sandbox default from then on. - No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-06 16:48:13 -08:00