208 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	4e715eb7ea	fix(server): schedule historical unsafe export recovery (#17687) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain accepted results after an unsafe workspace export fails. > - Historical recovery can retire an old notice or resume the accepted result without another provider turn. > - The periodic reaper repeatedly scans failures that cannot currently recover. > - Existing rows and committed changes can identify the actual recovery obligation. > - This pull request wakes recovery from those changes and retries only owed work. > - Dormant history becomes quiet while ownership and explicit operator decisions remain protected. ## Linked Issues or Issue Description **What happened?** The native finalization reaper repeatedly visits historical unsafe workspace-export failures. Cancelled tasks, newer runs, changed contracts, and explicit operator decisions can make those rows permanently ineligible. They still produce periodic database reads. **What did you expect?** Retire stale unsafe-export notices and recover eligible accepted results. Stop reading dormant history until a relevant committed change occurs. Keep a retry while a real obligation is blocked by ownership or a transient failure. **Steps to reproduce** Retain an unsafe-export terminal failure whose task is cancelled or superseded. Leave the instance idle after startup. The periodic reaper continues reading that history. Prerequisite #17680 is merged into master. Related: #14402 established safe export finalization; #14329 was an earlier historical recovery proposal. #17679 owns separate policy-review cleanup. This PR changes only historical unsafe-export scheduling and its producers. ## What Changed - Register a scoped worker through the existing coordinator and scheduler. Remove this duty from periodic reaping. Preserve the manual/startup recovery entry point. - Distinguish stale notices, eligible accepted results, dormant history, and concrete ownership contention. Reuse the existing mutation owner and lock order. Preserve explicit settlement vetoes and exact result/contract identity. - Signal relevant task, run, action, and deletion writes before they commit. Existing transaction boundaries handle nested commits and uncertain outcomes. Notifications contain only scope IDs. - Retain a retry only for owed work. Reconstruct work at startup, preserve drain/resume, and stop through the shared lifecycle. - Add concurrent, idempotent partial index migration 0410 for unsafe terminal failures. Keep the five newest generated schema snapshots. - Document producer coverage, query plans, supported writer boundaries, and test failures in the unsafe-export audit. ## Verification - Source `2a13dd08a6cc52f0adab09e3df9ce31e97b746b9` passes all 2188 cases across 24 complete files, without skips. The main selection passes 2129 cases in 192.58 seconds; safe-replacement passes 59 in 26.04 seconds. Coverage includes all 968 native-session cases, real export/action producers, explicit continuation, workspace reconstruction, interaction routing, Stop, queue and Slack consumers. The audit gives per-file counts. - Twenty separate fresh PostgreSQL 17 databases migrate through 0410 before setup. Four embedded-only fixture imports temporarily use their own external database on this IPC-exhausted host, then are restored byte for byte. Assertions and timing limits stay unchanged. This is qualified external-database acceptance, not an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. Normal integration preserves the complete original runtime/schema patch and unrelated upstream blobs. Original concurrent index SQL, all prior executable history, journal entries and retained snapshot bytes are unchanged; five snapshots 0406 through 0410 remain. Added comment removal preserves the full non-comment TypeScript token streams. - A populated 0409-to0410 upgrade preserves all 408 prior migration receipts and 38 companies. It appends exactly one receipt; replay adds none. The index is valid and ready on fresh and upgraded databases. - The owning audit retains earlier fixture failures, skipped checks and notice-retirement regression history. Older combined stack checks retain their original scope in their owning audits. Actual parent `73d24aa5acfcf17b3f676b4da7cd05eb70b60b99` is reconciled through a normal merge. Its tree equals the reviewed ancestor and the complete accepted child tree is unchanged. Fresh published-head CI and Greptile 5/5 remain required. No deployed sleep, fleet-cost or live provider qualification claim is made. - Inherited prefix correction: current integration `8f43c9f81fac23b50997e7f0e1d0d9cf383a8109` changes only two values in a previously unselected corpus fixture. All production code, the complete package tree and all 24 accepted test files remain byte-identical to the 2188-case source above. The additional complete 72-case corpus passes in 14.00 seconds and again on the same database in 13.08 seconds. Retained PSC/OUT and sentinel rows remain unchanged. The repeat is not double-counted. The parent audit retains the original collision and rejected cleanup attempt. - Latest parent context: integration `4d51231abe8b68c21ce896d0d2ac16865978c8ab` carries only the parent's one-argument GitHub observation correction and an independent master cleanup assertion. All production code, packages, 24 accepted suite files and the additional corpus are unchanged. The two focused native-presentation cases pass in 15.14 seconds and the complete cleanup-removal suite passes all six cases in 8.31 seconds on two fresh 0410 PostgreSQL fixtures. This does not claim the full parent workflow was rerun here. Current head `b28399babac72ba9db1c4a4ef882c5e87aded38c` has passing CI and Greptile 5/5 with no actionable findings. The first CI attempt lost runners: fourteen jobs explicitly received simultaneous shutdown signals, two reported lost runner communication, and two aggregate gates failed downstream. One failed-job retry on unchanged source passes. The underlying host shutdown cause remains unconfirmed; no code fix or extra retry is claimed. [CI run and attempts](https://github.com/paperclipai/paperclip/actions/runs/40819408528). No Ponytail verdict is present. ## Risks - A missing future producer can delay dormant-history discovery until restart. The audit names current producers. Independent SQL writers and other server processes remain outside the process-local signal contract. - Physical ownership and uncertain operations retain retries. Time alone does not authorize taking workspace ownership. No provider turn or unsafe archive retry is added. - The recovery owner remains authoritative if state changes after discovery. Terminal UI status does not authorize resource destruction or accepted-result replay. - The partial index adds matching-row write/storage cost. It builds concurrently through the existing restart-safe migration runner. - This proves the changed worker becomes quiet. It does not prove host suspension, external wake coverage, or fleet cost savings. Other sleep migrations remain. This slice still requires fresh current-head CI and reviews. ## Model Used OpenAI Codex, based on GPT-6, with reasoning and code-execution tools. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 00:13:22 -05:00
DottaandPaperclip	51c3f8924c	fix(server): schedule cancelled workspace preservation (#17677) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled native run may still owe preservation of prepared workspace files. > - The existing recovery receipt and source lease record that obligation. > - The orphan reaper still searches for these cancellations on every pass. > - This PR registers committed cancellations with the shared deadline worker. > - The existing retention owner keeps file pinning and ownership checks. > - Idle and completed history no longer causes this preservation scan. ## Linked Issues or Issue Description Refs #17674 and #17145. Stopped instruction collection is merged into master. It owns file-preservation discovery only. The existing environment cleanup worker continues to own stop-only teardown. **What needs to be improved?** `preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass. It queries even when no cancelled native workspace needs a file-recovery hold. Current foreground Stop can also defer retention after a lock or database failure, so startup-only recovery would be insufficient. **Why is this improvement important?** Empty queries keep idle tenants active. Removing them must not lose user files, weaken Stop ownership, replay cancelled work, or turn a retained allocation into destructive cleanup. **Proposed solution** Register the existing durable obligation from committed run-status, cancellation-acknowledgement and workspace-reference writes. Reconstruct it at startup. Keep retries while actual preservation remains unresolved. Disarm after the existing owner records the hold. Related-PR search and ROADMAP.md review found no duplicate of this scheduling migration. ## What Changed - Add a company/run-scoped preservation worker with 100-row keyset reconstruction and fair batches of five. - Separate pending file debt from immediate coordinator/lease readiness. Retain a 30-second retry for known unfinished preservation and shared failure backoff for errors. - Signal the existing status, acknowledgement, workspace-reference, deletion and settings transactions. Add no table or independent timer. - Keep exact source checks, no-workspace contracts, task/coordinator/run lock order, retention receipt and no-replay recovery card with their owners. - Recheck cancellation acknowledgement, run status and coordinator phase under the retention locks. Preserve the existing foreground preterminal API. - Keep startup/manual recovery; skip only this duty in periodic reaping. Preserve drain/standby and shutdown behavior. - Add concurrent, replay-safe partial index migration 0409 and retain the five newest snapshots. - Add the producer and lifecycle audit in `doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`. ## Verification - Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472 cases in 16 complete files pass without skips. The owner passes 84 cases in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46 seconds, including all 968 native-session cases and Stop, dispatch, restart, workspace-sync and upgrade-readiness coverage. - Twelve separate fresh PostgreSQL 17 databases migrated through 0409 before setup. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. The complete original runtime/schema patch and unrelated current-parent blobs are preserved. Original concurrent index SQL, all earlier executable history and retained snapshot bytes are unchanged; five snapshots 0405 through 0409 remain. - A populated 0408-to0409 upgrade preserves all 407 prior receipts and 38 companies. Exactly one receipt is appended; replay adds none. The preservation index is valid and ready on fresh and upgraded databases. - Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged through a normal ancestry reconciliation. Its complete tree equals the reviewed parent. The entire accepted child tree and all source hashes are preserved. The audit retains earlier failures, qualifications and per-file counts. Published head `b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57 checks passing or skipped. No Ponytail review is present. These checks do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Signals cover instrumented single-process writers. Independent SQL writers and additional replicas need integration. - Status signals may inspect a run without relevant file debt once. They do not create recurring work. Discovery and memory still scale with actual candidates. - Missing ownership or stop evidence remains pending. Active retries are intentional while a concrete file obligation is unresolved. - Retry timing is process-local; startup reconstructs durable debt. The worker does not overwrite another native-finalizer deadline. - Admitted database operations may finish after drain begins. Shutdown joins them. Provider teardown remains with the existing cleanup owner. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage, sleep eligibility and host suspension are unchanged. This proves worker behavior, not fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:27:11 -05:00
DottaandPaperclip	8252974141	fix(server): schedule stopped instruction collection (#17674) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runs can stop while their private instruction edits still need collection. > - Existing copy rows and stop evidence describe that unfinished work. > - The orphan reaper still queries for it on every maintenance pass. > - This PR activates collection from committed copy and run transitions. > - It uses saved retry deadlines when stop proof or collection remains pending. > - Active copies and completed history no longer cause collection polling. ## Linked Issues or Issue Description Refs #17671 and #17670. Both prerequisites are merged into master. Captured saves and post-save release remain separate reasons with their own workers. **What needs to be improved?** `recoverStopped` queries instruction copies on every orphan-reaper pass, even when no terminal run owes collection. A run can become terminal without changing its copy row, so copy-only notifications would miss real work. **Why is this improvement important?** Repeated empty queries prevent idle tenant instances from becoming quiet. Removing them must preserve edits, stop proof, current ownership, and recovery after restart. **Proposed solution** Use the existing transaction signals, scoped deadline worker, and scheduler. Reconstruct eligible copies at startup, signal supported copy and terminal-run writers, and retain saved retry deadlines for unresolved work. Preserve the existing collection and directory owners. Related PR search found the warm agent-file ownership work (#14695), verified stop boundary (#14311), and task-workspace cleanup (#16464). This PR does not replace those contracts. ROADMAP.md has no duplicate migration. ## What Changed - Add scoped stopped-instruction collection with keyset reconstruction and fair batches of 20. - Signal before copy preparation, mutation and adoption writes, and before terminal run writes in their existing transactions. - Cover heartbeat, native finalization/restart/session recovery, execution-control, queue and dispatch producers. - Signal settings changes and company/agent deletion. Preserve drain, standby, restart and shutdown ownership. - Recheck terminal-run eligibility in the due-time claim. Keep terminal status separate from process-stop proof. - Retain existing stop checks, attempt limits, saved candidate bytes and remote-resource fencing. - Skip only this lane in periodic orphan recovery. Preserve manual/startup recovery. - Add concurrent, replay-safe partial index 0408 and retain the five newest snapshots. Add no table. - Record the owner and producer audit in `doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`. ## Verification - Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases across 14 complete files pass without skips in 141.69 seconds. This includes all 968 native-session cases, filesystem saves, startup, terminal-run producers, dispatch, queue, execution-control, finalization and lifecycle. - Eleven separate fresh PostgreSQL 17 databases migrated through 0408. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent index SQL is unchanged. All prior executable migration history, journal entries and retained snapshot bytes are preserved; the five newest snapshots are 0404 through 0408. - A populated 0407-to0408 upgrade preserves all 406 prior migration receipts and 38 companies. It appends exactly one receipt. Replay adds none. The collection index is valid and ready on fresh and upgraded databases. - The normal merge of actual parent `267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete patches, all independent sidebar changes from #18117, and the full accepted server/package/test trees. The complete sidebar test file passes all 36 cases on the combined tree. React act warnings remain. - Prior failures and fixture qualifications remain in the audit. Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile 5/5, all 57 checks passing or skipped, and no unresolved review threads. No Ponytail review is present. These local tests do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Process-local signals cover instrumented writers. Independent SQL writers and additional replicas still need integration. - Terminal-run signals can cause one scoped inspection even if no instruction copy exists. They do not create a recurring scan. - Reconstruction and memory scale with actual pending candidates; active candidates are filtered by the run join. - Admitted file/provider operations can outlive a later drain request. Shutdown joins them. Uncertain ownership remains pending rather than being discarded. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarms, sleep eligibility and provider suspension are unchanged. Worker quiet is not a claim of deployed host/database sleep or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:13:57 -05:00
Nicky LeachandPaperclip	c57af3c564	Reuse prepared Runner storage and installed skill selections (#18089) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/207/changes/paperclip/207 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and their work. > - The native Runner prepares selected skills, instructions, and provider storage. > - Repeated preparation imports the company skill inventory and rebuilds Claude storage. > - Execution environments can already maintain prepared resources. > - This pull request lets them reuse account storage while Core keeps execution policy. > - New retry runs resolve current skills and keep the failed attempt's history. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native Runner preparation and retry skill selection. **Current behavior** Explicit installed selections still trigger company inventory reconciliation. Plugin preparation receives account identity only on warm transitions. Claude's isolated mode requires a session-specific configuration directory. **Proposed behavior** Skip inventory imports when explicit selections are installed, while retaining current authorization and version resolution. Send account identity on initial preparation. Allow an execution environment to provide persistent Claude storage without enabling preconfigured permission behavior. A new retry resolves current skill assignments. **Reason and benefit** Remove repeated preparation work without adding environment-specific policy to Core. Related: #13984 describes a similar storage-lifetime problem in the separate claude_local adapter; this PR does not change that adapter. ## What Changed - Derive Claude’s configuration path from the single authoritative prepared-storage input; separately supplied CLAUDE_CONFIG_DIR values are overridden. - Add an explicit prepared Claude storage binding, forward it to the sidecar, and retain recovery and provider lifetime fencing. - Refresh Core-owned Claude settings after acquiring the lifetime lease. Reuse environment-prepared skills. - Reuse installed explicit skill selections and preserve missing-selection initialization and current access checks. - Reserve the prepared-home marker for environment provisioning; task overrides are rejected and configured bindings are stripped before native launch. - Clear inherited skill selection snapshots on new retries. Keep admitted execution recovery and failed attempt metadata. ## Verification - Current single-input follow-up `8dbc2f6ec`: 154 focused Runner checks pass with one existing skip; four Grok identity checks and four Pi packaged-probe checks pass. Shared immutable source identities advance to Grok10/Pi32/Copilot74. All56 current-head CI checks pass (four intentionally skipped); standard and enhanced reviews are both5/5 with no unresolved threads. - Focused Runner sandbox tests: 29 passed. Focused environment tests: 16 passed. - company-skills-empty-runtime: 10 passed. - Plugin Runner and runtime selection database tests were attempted. Local execution is blocked because embedded PostgreSQL cannot load libzstd.1.dylib; these checks remain for CI. - New database regressions cover installed-selection reuse, initial account preparation, and retry snapshot removal. - Prepared-home trust-boundary regressions: 4 targeted checks passed (task schema, all env binding scopes, native executor). - Focused Pi public-install probe checks: 4 passed. The build smoke probe now asserts the current Pi 32 profile and exact digest. - CI found and fixed retry-context TypeScript inference, stale Pi identity fixtures, Docker source identity inputs, and the skill-observability test’s old assumption that every selection imports inventory. Cold and reused selection coverage remain. - CI chat shard 2 initially failed three GitHub/Discord idle-database assertions. An unchanged-head rerun passed; the original cause remains unexplained. - Runtime head `f5b53679a` passes all 56 current CI checks (four jobs intentionally skipped), including the full build/public-install probe, TypeScript, Rust, Runner and server matrices. Greptile is 5/5 with no unresolved review threads. - The requested enhanced review scored 5/5 on `f5b53679a`, with one nonblocking documentation request. Documentation follow-up `1ac5d26a8` adds only the documented existing ownership/0700 requirement; the thread is resolved and all 56 current-head checks pass (four intentionally skipped). The subsequent single-input runtime follow-up8dbc2f6ec repeated the complete CI → standard5/5 → enhanced5/5 cycle successfully. - Broader verification is delegated to GitHub CI as requested. The initial policy check rejected a generated lockfile repair; that change was removed because Core CI already resolves stale importer entries. ## Risks Prepared storage must be provisioned by the execution environment before launch. It is not credential isolation. Core replaces managed Claude settings under the existing provider lease. No new per-run harness homes are created by this option. New retries may select a newer skill version; recovery of an already admitted provider execution keeps its saved input. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and tool use. The exact model revision is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked related context above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run focused local tests; database and full verification remain for CI as explained above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 17:09:24 -07:00
DottaandPaperclip	f8484a9f8d	fix(native): deliver selected skills to remote Codex homes (#18080) ... ## Thinking Path > - Paperclip coordinates company agents and preserves their assigned instructions and approval rules. > - Native runs pin an immutable skill inventory for each execution. > - Built-in remote Codex launches prepare that inventory on the controller, but transfer only authentication, configuration, and the separate context assets. > - Codex needs the selected skills under its isolated home. An explicit skill path does not load a skill outside its registered discovery roots. > - This pull request transfers the sealed selected inventory to the path already used by explicit skill inputs. > - Each launch replaces stale assignments inside the authorized workspace while preserving provider history. ## Linked Issues or Issue Description Refs #18025. The corrected first-task instructions exist in the selected bundle, but remote Codex must receive those bytes before it can follow them. The legacy skill resolver work in #16646 and #9323 uses different paths and does not repair native remote launch staging. **What happened?** A fresh remote native Codex run could not read its assigned first-task skill. It produced an approval question instead of the required confirmation. The Product E2E journey correctly rejected that question before child creation. **Expected behavior** Codex receives the selected immutable skill at the same provider-local discovery path used by explicit skill input. Removed or unassigned skills do not remain available after a new launch. **Steps to reproduce** 1. Prepare an isolated Codex home with one assigned skill. 2. Run the built-in remote launch staging path against a fresh remote filesystem. 3. Resolve an explicit invocation of that skill. Before the repair, its provider-local file is missing. ## What Changed - Transfer only the prepared selected skill inventory into the isolated remote Codex home before other launch writes. - Reject writable files, symlinks, missing entries, and extra unassigned packages in the controller projection. - Validate each provider-home path component below the authorized workspace. Reject redirected retained paths before deleting stale skills. - Create the private discovery root before native sandbox transfers, which omit the source-root metadata and can ignore the requested directory mode. - Replace stale or empty inventories without changing provider conversation history, preconfigured homes, other providers, or plugin-provided launch preparation. - Exercise the real home materializer and turn resolver through the existing test-only Runner surface. Document the remote discovery contract. - Synchronize an existing Telegram attachment-failure test with the durable provider-effect commit, preserving its required final states. This CI repair is a separate test-only commit. ## Verification - The new production-staging regression failed before the repair because the resolved skill file was absent. - All 23 focused staging checks pass. They cover delivery, file modes, empty/revoked inventories, history retention, extra packages, writable files, source symlinks, target symlinks, redirected ancestors on sandbox/SSH, and an authorized workspace alias. The filesystem fixture packs and extracts actual tar bytes without applying the requested root mode; it preserves exact projected-file digests and executable bits. - The unchanged materializer and prepared-launch contracts passed 26 tests. Eight selected native caller checks passed; other cases in that filtered run were not selected. - Independent probes used official Codex 0.160.0, the real materializer, production staging, the real resolver, and a local model endpoint. The original staging supplied no skill message. The final staging supplied the complete selected skill body with its exact SHA-256. A final probe also ran production Daytona `performSyncIn`, backing only its SDK exec/upload/chmod edge with the local filesystem under umask `022`: the pre-review source reproduced root `0755`, and the final source retained root `0700`, package `0555`, file `0444`, and the exact selected digest. All model requests returned local 403; there were no upstream requests or credentials. - Full local `pnpm -r typecheck` and `pnpm build` passed on verification commit `e7d4e7a9`, based on `60b8089a`. The full local `pnpm test:run` on that frozen tree was not completed: later embedded PostgreSQL startup failed because the host could not allocate semaphore sets (`semget`, SEMMNI/SEMMNS), and the owned attempt was stopped with its log preserved. Its earlier four caller-fixture failures and one workspace-conflict failure are recorded separately below; the later resource error is not an explanation for them. Delivery commit `567d838d` replayed the initial six-file patch onto `ee67d2ec`; production was last changed in `23456f51`, followed by test-only commits `002fa4a7` and `6e8fa595`. All 24 staging checks, all 26 materializer/prepared-launch checks, and the complete native executor file (968 tests, zero skipped) pass locally. Server typecheck also passed after these test edits. The complete CI matrix passed on `002fa4a7`; current-head CI is tracked separately. - Initial-head CI exposed four caller fixtures that manually wrote auth/config without preparing the skills inventory. The same four failures reproduced locally. Those fixtures now use the real home preparer for their existing empty inventory and assert its transfer; all 34 retained/replacement scenarios and the complete 966-test caller file pass. A sandboxed full-file attempt could not bind local PostgreSQL (`EPERM`); it was stopped with its log preserved, then the identical command passed with local fixture access. Initial-head CI also recorded timeouts in workspace recovery, approval-queue backoff, and heartbeat attribution. Their three complete files pass unchanged locally (102 tests); the next CI head also passed those cases. The original slowdown remains unexplained. The consequent approval cleanup foreign-key failure is a timeout teardown race already recorded in the owning worker documentation. - The frozen full local run also observed an existing workspace-conflict test fail. The test and owning workspace code are unchanged between the verification and delivery trees, and the isolated delivery case passes. The original failure remains unresolved: stopping the later resource-blocked run prevented the final assertion detail from being emitted. That isolated pass does not replace the earlier result. - CI on `23456f51` caught a Telegram test reading `provider_effect=processing` after observing the provider mock call. Production commits `processed` later, so the test now waits for the unchanged complete durable-state assertion. All 18 selected Telegram attachment/media cases pass (the other 1,670 cases were not selected). The same CI attempt also recorded an E2E runner shutdown followed by cancellation, without a failed Playwright assertion; the necessary new test-only head reruns the matrix. Earlier failures remain preserved and are not reported as passes. - Review requested an executor-level preconfigured-home negative test. Retained and replacement workspace cases now drive the actual launcher, assert successful context staging, and exclude the isolated home's skills/auth/config transfers. A counterfactual that removes only the executor exclusion makes both tests fail; the exact production bytes were restored before the complete 968-test pass. Flat and nested selected package delivery is also covered. Optional arbitrary remote-stderr expansion is deferred; the repair keeps the existing closed error code and report surface. - Final head `6e8fa595` passes the complete CI matrix, including typecheck, build, Runner, browser, and aggregate gates. Two first-attempt jobs were interrupted by runner shutdown/lost communication; one unchanged-head rerun of those jobs and their aggregates passed. The underlying host interruption cause remains unknown. Both current-head automated reviews are 5/5 with the implemented finding resolved. - The original checked-in lockfile fails a fresh frozen install on the baseline. The repository's maintained resolution-then-frozen sequence passes. No lockfile change is included. - Independent source review found and then verified the retained-ancestor guard. This PR does not claim that the full live Cloud onboarding campaign has passed. It changes no journey assertion or approval gate. ## Risks Launch preparation adds an asynchronous inventory walk, one bounded remote cleanup command, and one transfer of the selected skill tree. Those costs grow with selected files and bytes and occur only at provider launch. A failed validation, cleanup, or transfer prevents launch. Remote Node is already required by the shared SSH/sandbox launcher. The selected packages remain read-only; the private discovery root retains Codex's normal writable mode. No database, API, credential-routing, or deployment change is included. ## Model Used OpenAI Codex based on GPT-6, with reasoning, source inspection, tool use, and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 16:37:13 -05:00
Nicky LeachandPaperclip	a2fbce9305	Fix plugin session and directory reuse on follow-up turns (#18045) ... ## Thinking Path > - Paperclip preserves task state when agents receive follow-up work. > - Plugin computers keep their process state outside the controller. > - A local receipt proves that a previous plugin run suspended. > - A cold resume left that receipt in place after the session became live. > - The next warm follow-up rejected the stale receipt as an identity mismatch. > - A separate follow-up path also treated a fresh task directory as a Git worktree. > - This PR consumes the obsolete receipt after admission and preserves the bound directory on follow-up. ## Linked Issues or Issue Description Related: #17820. This change retains the controller restoration added by #18018. **What happened?** A plugin session resumed from a suspended checkpoint and completed a turn. Its next warm follow-up failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The journal had the completed run's identity. The local suspension receipt still had the preceding run's identity. Two separate task histories showed this sequence. After a fresh retry completed in a repository-free plugin directory, an ordinary follow-up could instead fail with `The selected plugin environment does not support Core Git worktree preparation.` The saved task settings contained only `mode: isolated_workspace`, which introduced the default Git strategy on the next admission. **Expected behavior** Keep the same admitted provider session during warm follow-ups. Reuse the saved plugin task directory without Core Git preparation. Retain ownership checks and explicit strategy settings. **Steps to reproduce** 1. Complete a warm plugin task and let its session suspend. 2. Resume the checkpoint and complete another turn. 3. Send a follow-up while that session remains warm. 4. Separately, complete a fresh retry in a repository-free plugin directory and send an ordinary follow-up. **Paperclip version or commit** Both regressions reproduce against base `820c45846` with the added tests. **Deployment mode** Hosted plugin execution. Local reproduction uses the real filesystem and a disposable PostgreSQL 18 database, with synthetic provider execution. ## What Changed - Remove the local plugin suspension receipt after provider recovery and durable run admission publish a live session. Admission still checks the receipt first. Normal suspension writes a new receipt. - Recognize a saved fresh plugin directory through its company, task, workspace, owner, attempt and path. Use the existing directory-selection path. Explicit strategy settings still apply. - Add cold-to-warm, database-backed workspace-selection and ownership regressions. Update execution semantics and session verification recipes. ## Verification - Red: the cold-to-warm regression failed on unchanged executor code with `prior_owner_terminal_state_indeterminate` on the follow-up. - Red: the PostgreSQL regression selected a Git strategy on unchanged environment-selection code. The explicit-strategy denial case already passed. - Green: all 95 tests in `workspace-preparation.test.ts` and `provider-fresh-workspace.test.ts` pass against a disposable local PostgreSQL 18 database, selected with the existing `PAPERCLIP_WORKSPACE_PREPARATION_TEST_DATABASE_URL` test seam. - Green: 92 focused executor cases pass, including plugin checkpoint recovery, warm rotation and denial cases. The full `plugin-runner-task.test.ts` suite also passed (38 tests). - Green: direct server check, `node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit`. - Green: `node scripts/check-maps.mjs`, `git diff --check`, and a local diff scan for secrets, personal data and live identifiers. - Full local native suites: 988 of 1,004 cases passed. Sixteen integration cases could not start their bundled PostgreSQL cluster. The failure persisted outside the sandbox. These cases are not claimed as passing. An earlier sandboxed attempt also denied process and loopback operations; those denials disappeared when run outside the sandbox. - The first broad local commands could not locate Cargo on `PATH`. Rust is installed under `~/.rustup`; its Cargo executable was verified. Per the requested scope, subsequent local verification stays targeted to changed TypeScript paths; no Rust source changed. CI Typecheck + Release Registry and Rust verification passed on the initial commit. The complete repository test suite was not run locally. - Review regression: an explicit agent-level Git strategy reproduced the incorrect directory fallback before correction. After including the raw agent strategy in the existing check, all three targeted cases (implicit, task-level explicit, agent-level explicit) pass. The server TypeScript check passes after this correction. - CI on `a81668c4b`: the first workspace-a 2/2 attempt failed in unchanged UI `AgentActionButtons.test.tsx` (`calls clearError and refreshes agent-related queries`, zero mock calls). [Original failed job](https://github.com/paperclipai/paperclip/actions/runs/40717870938/job/122921463517). That shard passed on `294c5dfc3`. The failure is retained here. The UI shard passed on final commit `687000287`; no unrelated UI edits or local UI suite runs were added. - CI also exposed three existing local-directory recovery failures on `a81668c4b`: the review correction applied the raw agent strategy guard to local directories. [Original failed server job](https://github.com/paperclipai/paperclip/actions/runs/40717870938/job/122921463229). Narrowed that guard to adapter-managed directories, preserving local recovery behavior. All 20 targeted cases (three plugin selection cases and 17 local recovery scenarios) pass afterward, and server TypeScript verification passes. - Final commit `687000287d3339b7e15e35d95515b3c75c273d7a`: [hosted CI is green](https://github.com/paperclipai/paperclip/actions/runs/40720169342), including the corrected server shard, all other server/workspace shards, Runner checks, typechecking and build. All 55 non-skipped PR checks pass. Greptile scores this exact commit 5/5, with no unresolved review threads. GitHub reports no merge conflicts. - Pre-merge hosted acceptance was requested. The maintained deployment CLI stopped during artifact discovery: no published private Cloud composition exists for `687000287`. The current private build requires canonical Core master readiness and standard-image provenance. The Core-only branch preview omits the private computer plugin needed for the journey. No campaign or tenant update was started; private-preview deployment support is a separate prerequisite. No live deployment or provider acceptance run is claimed. ## Risks The receipt is removed only after the existing runtime publishes an admitted session. Earlier admission failures keep their proof. A later normal suspension creates a new receipt. The change preserves existing identity, lifecycle and controller checks. Directory selection remains limited to the exact bound, active, repository-free plugin recovery directory. Foreign owners, changed paths, wrong attempts and explicit Git strategies do not gain this fallback. No schema, API, credential, permission or fleet-default change is required. Existing failed runs and recovery holds remain history. This PR does not edit quarantined state or replay an interrupted task. Rollback restores the previous code and can reintroduce the two admission failures. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. A more specific model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 10:26:12 -07:00
DottaandPaperclip	820c458465	fix(runner): restore per-turn controller checkpoints before resume (#18018) ... ## Thinking Path > - Paperclip coordinates agents and preserves their work between turns. > - Remote native sessions save controller checkpoints in the database. > - External persistence leaves a small identity marker on local disk. > - A later `per_turn` continuation must restore that checkpoint before it reads the controller identity. > - The restore path handled warm sessions only, so valid per-turn continuations failed before provider execution. > - This PR restores the referenced checkpoint and retains the existing ownership and lifecycle checks. ## Linked Issues or Issue Description Related: #17439 added external controller persistence and the warm restore path. #18011 repairs warm preparation and fresh bootstrap checks; it does not cover ordinary per-turn continuation. **What happened?** A remote OpenCode run completed with `per_turn` lifecycle and saved its controller checkpoint in the database. Its next turn reused the normalized session. Admission read the remaining external-persistence marker as a full controller file and failed with `runner_state_identity_mismatch: durable_identity_unreadable`. The prior run's complete checkpoint and harness backup still existed. The failure occurred before provider work started. **Expected behavior** Restore the exact prior controller checkpoint before identity inspection. Then apply the existing company, task, session, prior-owner and suspended-state checks before a provider can start. **Steps to reproduce** 1. Complete a remote native OpenCode task with `per_turn` lifecycle and external controller persistence. 2. Send a follow-up that resumes the same saved session without a session reset. 3. Observe admission reject the valid external controller marker before provider execution. **Paperclip version or commit** Observed at `40c1ab36cdac70c85dc8fe5e912431834fdb691b`. The same warm-only restore remains on current master. **Deployment mode** Hosted Linux with Daytona and native OpenCode. The original model was `openrouter/deepseek/deepseek-v4.1-flash-1017`. ## What Changed - Restore external controller state for ordinary remote per-turn continuations through the existing checkpoint loader. - Bind that read to the current company and task, the marker's run, and the expected normalized session. - Keep local, warm and restart-recovery paths, prior-owner checks and harness backup checks intact. - Refresh a prior-run projection from the latest checkpoint after a rejected active-owner attempt. Keep current-run full state and the existing legacy file-backed fallback intact. - Add restoration and denial regressions, execution documentation and session verification recipes. ## Verification - Red: the real controller generated an external marker; a valid per-turn OpenCode continuation failed with the exact `durable_identity_unreadable` error before tool-authority construction. The repair passed the same case and its ownership guards. - Review regression red/green: a real controller advanced its checkpoint after an active-owner rejection. Before the follow-up fix, a later valid continuation retained a null attach seed. After the fix, it restored the newer attach template and prepare command. Legacy full-file roots with no database checkpoint retain their existing behavior. - At current head `1f6589233e6121d082a0c0c4444ea895b6e8cd7c`, 86 focused executor cases and all 14 execution-driver cases pass. Coverage includes checkpoint restoration, direct/backup suspension, Codex, legacy files, warm admission and canonical bootstrap, missing or malformed evidence, wrong bindings, active/foreign owners and unsuspended runners. Tests stop before provider launch. - The real PostgreSQL encrypted-checkpoint integration passes, including normalized-session refusal before write and stale-writer fencing. Independent source review found no actionable safety defect. - `node scripts/check-maps.mjs`, `git diff --check`, full `pnpm -r typecheck` and `pnpm build` pass on current head. - The local full suite did not complete: native PostgreSQL startup failed under the host sandbox, an initial Docker rerun lacked migrations, and the migrated retry crossed the review-fix revision change. Logs are retained; that mixed-revision run was stopped and its owned database removed. Full test evidence comes from current-head CI. - Current-head CI passes: 53 successful checks, two intended Storybook skips, none pending or failing. Greptile is 5/5 on that exact head with no open review threads. The only manual merge conflict with #18011 was in the verification recipes; both recipes and the unchanged per-turn restore helper are preserved. **Live continuation acceptance remains unverified, so this PR stays draft.** Two disposable OpenCode/Daytona attempts never reached turns two and three: - At `72b49208b1`, the first turn executed tools and produced the exact 33-byte workspace note, but exceeded its four-minute settlement deadline. The first completion lacked a registered attachment; the second decorated a typed attachment reference with filename/size/hash text. The strict UUID parser rejected that reference before checking a valid criterion reference. The scoped attachment, asset and publication receipt exist. Replaying the actual guard rejects the original input before a database lookup; correcting only that reference passes against the recovered row and receipt. This did not show a publication-persistence or controller defect. The task remained unsettled. - At `1f6589233e`, a separately approved replacement used a temporary prompt-only fixture change requiring exact `deliverable:<attachmentId>` references, with metadata in `verification.detail`. Production code, grading and deadlines were unchanged; the fixture patch was removed afterward. This attempt had no recorded provider authentication, harness-ready, model, tool or result event. Ingress acquisition took 84.072 seconds; transport connection took 205.835 seconds within the 239.832-second failed native execution. A local settlement GET also timed out after 30 seconds. The underlying startup stall is unresolved; these observations do not establish a remote provider outage. Final recovery was blocked with `ambiguous_remote_recovery_evidence`; no successful settlement or continuation is claimed. Both attempts used a Darwin arm64 source host with Node 26.4.0, isolated PostgreSQL 18, a Linux x64 runner and OpenCode 1.18.34. The representative qualified model was `openrouter/deepseek/deepseek-v4-flash-0731`; the original incident used `openrouter/deepseek/deepseek-v4.1-flash-1017`. The controller defect occurs before provider work. Runtime artifacts came from immutable build `e7ea2e2961`, whose Runner package source is identical to the candidate. The current-source manifest, artifact, distribution, asset and all 110 archive-hash checks passed. Each attempt had $10 company/agent model budgets and zero automatic workflow/task retries. Accounting was incomplete; zero recorded spend is not proof of zero cost. Earlier setup failures are also retained: host PostgreSQL bootstrap denial, cached-pack compatibility admission before provider work, and an [official artifact build](https://github.com/paperclipai/paperclip/actions/runs/40696838122) that stopped during an unrelated Pi npm install at its 300-second bound. No manifest or integrity guard was weakened. Owned local processes and databases are stopped, and the temporary fixture patch is removed. Standard fixture cleanup had left one replacement sandbox started; it was stopped and deleted through the SDK. Every owned sandbox was then verified absent with a final 404 readback. Resource cleanup does not establish successful task finalization. An earlier head retained one [voice-recovery timing failure](https://github.com/paperclipai/paperclip/actions/runs/40695892339/job/122843005098): a deadline test observed two failed-provider inspections instead of one. A single isolated shard rerun passed all 1,579 tests without a source change. Its timing explanation remains an inference. The current reconciled head passed without a retry. ## Risks Restoration writes a local projection before the existing admission checks. The encrypted checkpoint loader validates company/task/run and normalized session before writing. A later ordinary continuation refreshes a prior-run projection so rejected admission cannot freeze an active owner's older checkpoint. Active, foreign or unsuspended prior owners still cannot start a successor. Missing or malformed evidence remains blocked. The change adds no retry, credential, provider binary or lifecycle-policy change; it preserves the existing legacy fallback. This PR does not deploy the repair or recover the original hosted tasks. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and collaborating agents. The session exposes GPT-6 as its model version; a more specific model ID and context-window size are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:49:32 -05:00
DottaandPaperclip	25f1070fbd	fix(runner): preserve checkpoint proof across warm preparation (#18011) ... ## Thinking Path > - Paperclip coordinates agent work and preserves its execution state. > - Native Runner checkpoints preserve authority across controller changes. > - Warm sessions also need checkpoint writes while their successor prepares. > - External persistence replaces the local state file with a marker. > - Two existing checks treated those valid states as lost ownership or prior activity. > - This PR uses the existing preparation lease and canonical controller state to keep those checks accurate. > - The result is safe warm-session retirement and fresh-session startup on reused sandboxes. ## Linked Issues or Issue Description Related: #17439 and #17320. The open #17401 changes the recovery welcome window and does not cover these paths. **What happened?** A configuration change could fail before the next provider turn with `runner did not durably suspend before checkpoint`. The task already pointed to its preparing successor, but that successor did not yet have a native coordinator. The old completed runner's maintenance checkpoint was rejected. A separate fresh session on a reused sandbox could fail with `resumed_unavailable`. Checkpoint registration had replaced the full local state file with an external-persistence marker. The bootstrap check still required the old full-file format. Process supervision then reported unconfirmed termination before launch. **Expected behavior** Retire the completed warm runner under the same process's live preparation lease. Admit a fresh remote session only when the registered controller proves exact identity, no prior activity and pending bootstrap commands, and the remote session directory can be claimed atomically. **Steps to reproduce** 1. Complete a native warm turn with external checkpoint persistence. 2. Change configuration so instruction preparation must retire that session, then submit a follow-up. 3. Separately, admit a fresh native session on a reused sandbox lease after external checkpoint registration. **Paperclip version or commit** Reproduced in the checkpoint implementation introduced by #17439. Fixed against current master. **Deployment mode** Source checkout tests and a hosted Docker/Daytona incident. No deployment or original-task retry is part of this PR. ## What Changed - Accept completed-run maintenance under the exact successor's live, same-process preparation lease only before a native coordinator exists. - Preserve original controller generation, accepted-result, no-new-work, company and task checks. - Pass the registered controller through the existing connection callback and read its current canonical state for untouched-session proof. - Keep marker-only, identity, activity, backup and atomic remote-directory guards. - Add regressions and update execution semantics and session verification recipes. ## Verification - Warm retirement red: the real Postgres writer rejected the preparing successor's `runner.suspend` checkpoint with `native_runner_checkpoint_owner_lost`. - Warm retirement green: 14 Postgres ownership cases pass, including stale/foreign/expired owners, conflicting native authority and new provider work. - Bootstrap red: real controller registration and external persistence caused the resumed and replacement positives to fail. Six external negative cases passed. - Bootstrap green: all 34 bootstrap cases, the bootstrap-state predicate and 13 connection-driver cases pass (48 total). - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full `pnpm -r typecheck` and `pnpm build` pass. - Full local `pnpm test:run` remains partial/interrupted. The default attempt could not initialize embedded Postgres on this Mac. A dedicated Postgres 18 attempt ran for 17 minutes without a completed file result before interruption; the generic helper separately confirmed unsupported embedded Postgres (`initdb` failed after five attempts) and ignores the external database URL. The generic probe skipped one test. Setup and interrupted-run logs are retained; neither is claimed as a behavioral red or a full local pass. - Real source-hosted Codex/Daytona acceptance was attempted at `6ce7fcacc4` with `gpt-6.1-sol`, `xhigh`. The actual task stopped before provider work at `runner_ingress_auth_failed`, with zero authenticated connections and zero source events. A separate environment-qualification handshake reached challenge/response; that does not prove task authentication. The three-turn warm reuse, configuration retirement, useful output and reload workflow remains unverified, so this PR stays draft. - The live attempt used the cached Linux amd64 Runner binary `7437a4e32a099180889be534fcfb2a086046514490c1448aba46c55848983ed8` from the immutable Daytona image. The current staging binary could not be copied because SSH timed out, and the configured current EXE image returned registry 403. No compatible current local or CI-downloadable artifact was available. Earlier setup failures and cleanup receipts are retained; no provider success is claimed. Disposable test resources were cleaned up. - Exact-head Greptile completed at 5/5 with no open feedback: https://github.com/paperclipai/paperclip/pull/18011#issuecomment-6469515952. All current-head CI checks pass: 53 successful checks and two intended Storybook skips. ## Risks The preparation fallback affects maintenance checkpoints only after an accepted result. An existing native coordinator prevents that fallback. Stale controller generations, expired or foreign preparation leases, wrong tasks and new provider work remain denied. The bootstrap check retains the live controller object and reads current state each time. It does not trust a marker or restore local state beneath a live controller. Pending turns, prior events/connections, backups and existing remote directories retain recovery gates. No schema, provider binary, credential, permission or lifecycle-policy change is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and collaborating agents. This session does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:09:09 -05:00
DottaandPaperclip	09ad9adce2	feat(server): adopt running remote agents through durable controller checkpoints (#17439) ... ## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:39:23 -05:00
DottaandPaperclip	39e811c9c5	fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native turns reconcile company skills and select verified remote runtime packs. > - Catalog imports repeat folder transactions for skills in the same bundled category. > - Diagnostic requests can also reuse an older image proxy that lacks the phase collector. > - This pull request reuses categories within one import and selects the current proxy for fresh diagnostics. > - Each import has its own category map, and recovery keeps its admitted runtime identity. > - The result is less preparation work and a runtime that can provide the requested measurements. ## Linked Issues or Issue Description Refs #17918. **What happened?** A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but emitted no provider phase batch. Its saved pack authority selected an older image proxy. The deployed server proxy contained the collector. The image proxy did not. Bundled inventory reconciliation also repeats folder transactions for skills in the same category. The paired local benchmark exposes that preparation cost. **Expected behavior** A fresh diagnostic request must use a verified proxy that supports the current collector. Bundled skill reconciliation should resolve each category once per import. **Steps to reproduce** 1. Use a protected image pack with the same OpenCode interface revision and an older proxy hash. 2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh remote run. 3. Observe that the current backend accepts the older proxy and cannot emit the requested phase batch. ## What Changed - Reuse each bundled category during one import. Every skill still runs its existing validation, access lookup, value comparison, and write path. - Compare the installed proxy hash with the current build when a fresh run requests diagnostics. - Keep compatible-pack reuse for ordinary runs and exact admitted identity for recovery. - Cover selection through the real backend, including recovery after diagnostics are enabled. - Document runtime preparation costs and admission limits for diagnostic requests. The skill importer owns category reuse. The provider verifier owns the artifact comparison. The native backend passes the resolved diagnostic requirement at the existing admission seam. The changes stay in those existing owners and share one latency investigation and rollout. ## Verification - Before the fix, the new verifier and backend cases failed because both accepted a stale proxy. - After the fix, 63 focused cases passed in 8.10 seconds. They cover normal reuse, current diagnostic proxy selection, stale proxy rejection, selected ACPX profiles, and admitted recovery identity. - 110 skill and folder service cases passed against isolated real PostgreSQL databases. Only cluster bootstrap used the existing local PostgreSQL service. Queries, migrations, authorization, file handling, and transactions ran on their real paths. - A paired benchmark used the same immutable skill snapshot, authenticated actor, 170-skill inventory, and 3.5 ms delay in each network direction. It checked every delivered file and executable bit. Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81 in each matched sample, from 241–243 to 160–162. The bundled phase median fell from 2.596 s to 1.721 s. The final measurement uses commit `37eac8d8de`; only the skill importer differs from baseline `7951455073`. These local results are not hosted acceptance. - `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess toolchain mismatch, a local IPC restriction, and a fixture type error. Each was corrected before the passing run. - Module boundary and whitespace checks passed. - `pnpm build` passed for all workspaces. - `pnpm test:run` did not complete. Embedded PostgreSQL initialization failed after repeated startup attempts. The owned test subtree was stopped after preserving the failure log. This is not a full local test pass. On final commit `37eac8d8de`, all 53 successful check contexts passed; two optional Storybook checks were skipped. Greptile is 5/5 with no unresolved findings. The draft awaits merge approval. - The catalog change has a measured local gain. The proxy change fixes missing phase capture. Real hosted response latency remains above the target and needs a new matched measurement after delivery. ## Risks The category map belongs to one import and one company. The importer still resolves current skill data on every invocation. Fresh diagnostic runs can prepare a current verified pack, which can add startup time. Admission can fail when matching runtime assets are unavailable. Ordinary runs keep the existing fast compatibility path. Recovery keeps the already admitted identity, even if diagnostics are enabled later. The operator endpoint gate, collector limits, and closed span fields are unchanged. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:00:05 -05:00
DottaandPaperclip	f60ac29cf1	fix(runner): migrate Grok Build with native reasoning effort (#17922) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Harness selection maps supported agents to Paperclip Runner. > - The native Runner already supports Grok Build, but its migration mapping excludes Grok. > - Grok agents also have a thinking-effort setting that the native execution contract does not carry. > - This pull request adds the migration path and preserves explicit Grok effort through startup and resume. > - Operators can upgrade an existing Grok agent without silently losing its model, account, or effort. ## Linked Issues or Issue Description **What happened?** Existing Grok Build agents have no Runner selector or upgrade action. Adding only the mapping would route conversion through the Codex fallback and lose Grok execution semantics. Native execution also ignores the legacy reasoning-effort field. **Expected behavior** A qualified Grok agent can explicitly upgrade. The saved model, account, instructions, history, and supported effort remain intact. Unsupported legacy settings fail before saving. The provider must acknowledge the selected effort before inference. **Steps to reproduce** 1. Open the runtime page of an existing Grok Build agent on a qualified Linux x64 or macOS ARM64 computer. 2. Observe that the Runner selector and upgrade action are absent. 3. Compare the native execution input with the saved legacy reasoning-effort setting. Related work: #13882 introduced native Grok Build. #15422 covers the earlier default-runner work. This change extends the current shared selection contract. ## What Changed - Add the Grok harness mapping, qualified platform limits, upgrade action, and company-scoped computer qualification. - Translate Grok settings to ACPX without changing the selected account or explicit model. Resolve the legacy default model to the native default. Preserve the effective legacy search policy. Reject incompatible command, argument, permission, and turn-limit settings. - Add native execution input v10 for explicit Grok effort and search policy. Preserve historical schemas. Enforce disabled search through Grok requirements and bind it across startup and recovery. - Apply and verify ACP `reasoning_effort` before prompting. Bind the value to provider descriptors, checkpoints, runtime identity, and resume checks. - Use the Grok catalog and effort choices in the task composer. Apply task model and effort overrides while retaining the agent account and search policy. Update runtime controls, portability/default handling, documentation, and capability maps. - Advance Pi and Copilot source attestations for changed shared transport and recovery code. Keep historical identities intact and update all Rust integration fixtures. ## Verification - Migration and review regressions were reproduced before the fixes. Coverage includes the Grok mapping, default model, explicit effort, implicit search policy, task overrides, and incompatible legacy settings. The broad migration, route, immutable-input, and composer check passed 478 tests. The final catalog and task-input check passed 135 tests. - Runner startup, recovery, search policy, and profile checks passed 361 tests, with one platform skip. The final source-identity, search-admission, and transport event-pump check passed 89 tests. - All 757 Rust workspace tests passed for the broad implementation. After the final source attestation update, all 55 ACPX profile/session tests passed. - `pnpm -r typecheck` and `pnpm build` passed for the implementation. Server and Runner typecheck passed after the final production change. Server `tsc --noEmit` passed after the final test-fixture change. Docker context integrity, token gates, maps, generated protocol data, and replay goldens passed. - The v10 startup fixture initially used an ambient local Grok login and failed on CI. An explicitly empty credential home reproduced all four failures. The fixture now supplies a synthetic key and a private empty home. A separate case checks that missing credentials fail closed. All seven selected Grok v10 and Claude v9 startup cases pass without host credentials. - The full local suite could not complete because embedded PostgreSQL failed before database-backed fixture setup. The broad executor-suite attempt hit the same limit and was stopped. Neither attempt is counted as passing. Current-head CI must verify those paths before merge. - Earlier CI found missing Rust input fields, stale source attestations and exact fixture pins, a missing Docker context entry, and an outdated import expectation. Those failures were fixed and checked locally. The CI logs retain the original failures. - Different earlier CI runs also failed unchanged cleanup pagination, device-login timeout, and chat lost-response recovery cases. Later attempts passed the cleanup and login cases without changes to those implementations. Their causes remain unproven. The original evidence remains in [the first run](https://github.com/paperclipai/paperclip/actions/runs/40563402840), [the login run](https://github.com/paperclipai/paperclip/actions/runs/40571128328), and [the chat run](https://github.com/paperclipai/paperclip/actions/runs/40573072163). A passing retry does not establish a root-cause fix. - Both reviews rate current head `16fba9b4cdf12bc7945e6628912e79da42959f35` 5/5. All current-head CI gates pass in [run 40576580699](https://github.com/paperclipai/paperclip/actions/runs/40576580699). All 60 check runs completed cleanly, including the four intentional skips and the neutral 5/5 Paperclip review. The legacy security status is successful. No review threads remain unresolved. - Staging migration and a real shell task are pending deployment. This PR does not claim full Grok provider qualification. Reviewers can verify the operator path by upgrading a qualified Grok agent, checking its saved model and effort, and running a shell task through native execution. ## Risks - Explicit effort and search policy require the matching Runner and sidecar build. Older immutable execution inputs keep their existing schemas. - Providers that omit or reject the selected effort fail before a prompt. Unsupported effort does not fall back silently. - Existing agents retain their saved runner until an explicit change. New Grok agents use native execution only on qualified platforms. - Grok token and cost reporting remain unverified, as before. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window capacity are not exposed in this session. Used reasoning, source editing, shell execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and the focused suites pass; the full local suite has the environment limitation recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 15:28:42 -05:00
3 people	3cd3af5d75	Trace native preparation and OpenCode execution phases (#17918) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each native run prepares its workspace and skills, then starts its provider. > - A long preparation span does not show whether inventory refresh, a cache lock, repository reads, or file checks caused the delay. > - One hosted run spent 81 seconds in this phase. The same deployed service and pinned files took 2.55 seconds in a separate diagnostic process. > - Another hosted turn spent 69 seconds before its first output. A matched public-workspace local model replay took 7.64 seconds. > - We need skill preparation and provider-internal measurements before choosing another fix. > - This pull request adds skill spans, a real-service benchmark, and opt-in filtered provider timing. > - The benefit is a faster and more precise diagnosis loop. ## Linked Issues or Issue Description Refs #17620. That PR owns the batched read optimization. This PR adds diagnostics and does not change its read policy. Refs #17907. That PR protects long cache lock transactions from the database idle timeout. **What happened?** Native skill configuration can take much longer than matched standalone resolution. Provider execution also has long gaps with no internal timings. Existing spans cannot separate inventory refresh, cache acquisition, repository reads, context loading, Git snapshots, or model streaming. **Expected behavior** An operator-configured trace shows these phases under the existing run context. Without an OTLP endpoint, the new spans stay off. A local benchmark can reproduce the service path with real PostgreSQL, files, an agent actor, and a responsible user. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --actor=agent --staging-shape=true`. 2. Inspect cold and warm phase timings, SQL counts, and file verification results. 3. Set the existing operator OTLP configuration and add `--export-otlp=true` to inspect the same hierarchy in the collector. ## What Changed - Add bounded skill inventory, refresh wait, cache acquisition, row, version, repository, file check, and consumer spans. - Keep the existing endpoint gate, attribute allowlist, cache locks, authorization, file validation, and durable pin boundary. - Add a standalone benchmark with a full repository fixture, immutable snapshot input, calibrated local transport delay, and optional OTLP export. - Verify every delivered file and executable bit outside the timed section. Create its fixture agent through the lifecycle API. - Add an opt-in authenticated loopback sink for OpenCode. Discard raw attributes, logs and content. Retain only closed phase names and timestamps. - Carry one bounded diagnostic batch through the real proxy, Rust normalization, durable journal and server trace projection. Group provider phases under their execution parent. - Freeze the Date clock in an existing delegated Slack rate-limit test. This fixes a CI failure when consecutive requests cross a rate-window boundary. It does not change rate-limit behavior. - Document phase boundaries and the difference between SDK startup and confirmed ingestion. ## Verification - Skill service, runtime selection, concurrency, cache and timing suites: 122 tests passed with real PostgreSQL and migrations. Enabled-span integration tests also passed. They verify parent relationships and a real advisory-lock timeout. - A captured 208-file repository produced 209 verified runtime files. The benchmark uses real agent and responsible-user authorization. The lifecycle fixture correction passed with the staging-shaped inventory. - Affected OpenCode driver, collector and transport tests: 72 passed. The tests cover authentication, content removal, size/count bounds, disabled behavior and the rebuilt native Runner boundary. One Rust projection test and 18 server trace tests also passed. A held run-log sink proves that a full 128-span diagnostic batch adds no duplicate database writes before terminal consumption. - The two CI-failing delegated Slack rate-limit cases passed against real PostgreSQL with only the Date clock frozen. - Runner TypeScript and server TypeScript checks passed. Rust formatting and module-boundary checks passed. - Real Honeycomb readback confirmed the 14-span skill waterfall and a 40-span provider waterfall. Provider phases have the correct execution parent. The provider export uses synthetic model replies and is instrumentation proof only. - Earlier full local checks failed on missing package-local executables, then on the default Node heap and a missing generated service worker. The dependency links were repaired. The server check passed with a larger heap. The full local test attempt could not start embedded PostgreSQL on this host and was stopped after retaining its failures; real database coverage uses an isolated database on an existing local server. These attempts do not count as a full local pass. - CI on the prior revision passed typecheck, build, Runner and all E2E shards. It failed the benchmark lifecycle boundary and one existing rate-window test. Those fixes passed CI. A later Runner assertion exposed the Copilot source identity pin on the shared transport file. The diagnostic flag now uses the existing configured-environment projection, preserving that file byte-for-byte. All 66 selected installation and native transport tests passed. New-head CI and review remain required. - The under-15-second hosted response target remains unmet. This PR adds diagnostics and does not claim a hosted latency fix. ## Risks - Skill spans and Honeycomb export retain the operator endpoint gate. OpenCode phase collection requires a separate explicit diagnostic flag. Without an endpoint, its closed measurements stay in the instance run log. - Raw OpenCode OTLP can contain prompts and authorization attributes. The new sink rejects unauthenticated requests and discards all attributes before any measurement reaches PRP or the operator collector. Request, event and per-phase limits bound memory and journal growth. - Collection does not wait for a provider flush. Late or oversized exports can be absent; these measurements are not a complete profiling guarantee. - The benchmark's inventory contents and transport delay are explicit fixtures. They do not reproduce server contention or prove hosted response time. - The one durable diagnostic batch remains the local evidence. Its child span export does not add per-phase run-log rows. - No database migration, authorization policy, model selection or inference transport changes. The Rust Runner and OpenCode proxy must be shipped together for the new opt-in diagnostic. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing>	2026-10-27 14:53:32 -05:00
DottaandPaperclip	f6ba9afab0	perf(server): narrow explicit continuation run reads (#17896) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must prove that old execution stopped before it admits new user input. > - Explicit continuation reads the source run before and after it takes the run lock. > - Both reads transfer the saved prompt and the full Runner profile. > - The preflight needs only identity, comment, adapter, and workspace receipt fields. > - This pull request narrows those reads and keeps the complete locked profile for session proofs. > - The benefit is less database transfer with the same admission and recovery rules. ## Linked Issues or Issue Description **What existing behavior does this improve?** Explicit native continuation admission after a failed or stopped run. **Current behavior** The preflight and locked reread select complete heartbeat run rows. Large saved contexts and execution profiles cross the database connection twice. **Proposed behavior** Read only required scalar fields and typed context keys. Read only historical adapter identity and workspace sync receipts before the lock. Keep the complete Runner profile in the locked reread. Keep lock order, company scope, text lookup of historical run IDs, stop proofs, queued input, and replay rules. **Reason and benefit** A synthetic PostgreSQL 17 fixture removes 74.9% of the two queries' DataRow bytes. This result is local query evidence. It is not a claim about deployed latency or monthly transfer. **Breaking changes** None intended. No schema, API, migration, workflow, or dispatch contract changes. **Additional context** Related: #17491 narrows the separate execution blocker read path. This PR covers explicit continuation admission. The PR has no dependency on the other byte reduction PRs. ## What Changed - Select required continuation fields and typed context keys instead of complete run rows. - Read a small historical adapter/workspace profile during preflight. Preserve the full locked profile for exact suspended-session and quarantined-cleanup proofs. - Narrow four helper input types to their actual field requirements. Their runtime behavior is unchanged. - Add real PostgreSQL admission comparison coverage and a reproducible baseline/treatment benchmark. - Document the query boundary and benchmark command. ## Verification - Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`. - Disposable PostgreSQL 17.11. One source run has a 192 KiB incompressible context and two 96 KiB profile payloads. - Command: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-run-reads.ts --baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. Set `PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL` to a disposable database ending in `_test` to use an external server. - The harness captures SQL generated by both real service versions. Baseline imports use current helpers; only their TypeScript input types changed in this PR. Their runtime behavior stays fixed. - Preflight DataRow bytes: 394,067 to 466 (99.88% less). Locked reread: 394,067 to 197,168 (49.97% less). Combined: 788,134 to 197,634 (74.92% less). - Byte counts cover exact PostgreSQL text-format DataRow frames. They exclude RowDescription, command metadata, TLS, TCP, and other queries in the service. - Seven measured samples per query follow two warmups. The order alternates. Final-head median preflight time: 6.53 to 2.28 ms (ranges 5.34–7.41 and 1.84–4.46 ms). Median locked read: 8.06 to 5.26 ms (ranges 5.94–13.84 and 4.61–6.81 ms). These are warm local SQL measurements under shared machine load. The deterministic byte counts are the acceptance criterion; these noisy timings do not establish a deployed latency gain. - The harness compares the admitted continuation, blocked reasons, every retained scalar column, kept JSON, and the full locked profile. It fails unless the fixture removes more than half of the two reads' DataRow bytes. It records SQL, parameters, revision identity, samples, and EXPLAIN plans. - All 291 continuation tests pass together on the final head with no skips in a clean isolated PostgreSQL 17 database. This includes all seven new projection comparisons and six malformed/missing JSON cases. - Edited server TypeScript check passes: `node node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json`. - `pnpm -r typecheck` and `pnpm build` pass on the final head. The normal server build verifies the complete pinned Copilot closure. Local full `pnpm test:run` is not claimed: host embedded PostgreSQL startup limits prevent that gate. The isolated PostgreSQL regressions pass; the full exact-head test matrix runs in CI. The targeted exact-head test retry passes all 4,086 tests in 98 files. All CI verification checks and the dependent aggregate are complete and successful on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`. Greptile completed cleanly on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`: 5/5, no findings, and no unresolved review threads. - Local setup evidence: embedded initdb could not load its ICU data library. Tests therefore use a dedicated PostgreSQL 17 container. A run against reused test fixtures produced stale sweep failures; a new clean database passed all 285 cases. The first comparison mock replaced an unrelated aliased query; it was narrowed to the two admission reads before the clean pass. - The first [server test shard 6/12](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122277218565) failed one existing workspace-recovery case: a third export stayed `scheduled` at its 20-second poll deadline; 4,085 other tests passed. The exact case passes on the pinned baseline and final head with the same PostgreSQL 17 fixture. Temporary spies recorded zero calls to both changed admission functions in either replay. Other touched helpers have type-only changes. This supports one timing-flake retry; the exact scheduler delay remains unproven. The single [targeted shard retry](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122285744135) passes all 4,086 tests in 98 files. The exact case completes in 17.25 seconds. The dependent aggregate is complete and successful. No timeout or unrelated runtime code was changed. - The separate Paperclip Review check ended `NEUTRAL` without a final report. The completed source review is the exact-head Greptile 5/5 result above. ## Risks - A future admission check may need an additional saved context key. Keep the projection and helper contracts in sync with that consumer. - Exact suspended-session and quarantined-cleanup proofs still require the complete Runner profile. This PR retains it under the lock rather than weakening those checks. - JSON values retain their types and missing keys stay absent. Malformed-context comparisons preserve the existing controller hold. - These measurements use synthetic local data. Deployed transfer savings require a fresh capture after rollout. Nothing was deployed. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this session. The work used repository tools, code execution, and real PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 12:05:51 -05:00
DottaandPaperclip	db70c5acd9	fix(runner): retain startup Stop and exhaust reconnect recovery (#17812) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators must be able to stop a remote runner while it starts. > - Remote artifact staging can finish after the transport's bounded cleanup wait. > - The launcher discarded an early Stop request and could dispatch after staging. > - A separate path missed the runner's plain-text reconnect expiry receipt. > - This change retains startup cancellation and ends an exhausted reconnect incident. > - Public package imports and a source-bound profile keep the regressions buildable and admission checks intact. ## Linked Issues or Issue Description Related work: Refs #16835, Refs #16843, Refs #16855. **What happened?** The HTTP Stop route returned 200 and recorded a cancelled run. After bounded bootstrap cleanup, releasing artifact staging still dispatched one launch RPC. A separate reconnect test accepted a replacement after the runner reported that its grace period had expired. The first PR head also failed CI. The HTTP regression imported Runner source across the server TypeScript boundary. Its event receipt mock read a field that is not present on both event forms. The changed transport also invalidated the pinned Copilot source identity. **Expected behavior** A cancelled launch must remain cancelled after staging. An exhausted reconnect receipt must end that incident. Regression tests must use public package contracts. The source identity must cover the changed transport without replacing historical profile bytes. **Steps to reproduce** 1. Hold the remote launcher's artifact staging promise. 2. Call the board Stop route and let bootstrap cleanup reach its preparation deadline. 3. Release staging and count launch RPCs. 4. For reconnect expiry, end an owned runner with its coded or plain-text expiry message. 5. Run the server typecheck and the Copilot profile identity check. ## What Changed - Retain cancellation through staging, trace publication and an in-flight launch RPC. - Apply a deferred signal through the existing identity checks when the launch identity arrives. - Preserve an earlier KILL request when a later TERM request arrives. - Recognize both coded and plain-text reconnect expiry receipts. - Test the real board Stop route and heartbeat cancellation with PostgreSQL. Use public Runner exports and handle both event receipt forms. - Generate source-bound Copilot profile v65 after integrating current master. Preserve exact published v62, v63 and v64 fixtures, retain the earlier v63 reconnect candidate as historical evidence, and reject historical profiles for a current launch. - Document process evidence, retained uncertainty and the limits of cleanup and continuation evidence. ## Verification - At current integration head `2b96ee3adcb3be642c538436113e1159d92f843d`, launcher and HTTP Stop regressions: 21 passed. This includes the existing process-generation and exact-exit safety cases on the integrated base. The HTTP case includes bootstrap expiry; it is not isolated Stop-only settlement proof. - At current integration head, reconnect expiry regressions: 2 passed, for coded and plain-text receipts. - At current integration head `2b96ee3adcb3be642c538436113e1159d92f843d`, the Copilot installation and identity suite passed all 63 tests and the canonical generator check passed. Before regeneration, the merged-tree admission test failed with a transport source hash mismatch and the generator reported a stale identity. Published v62, v63 and v64 bytes remain unchanged; the prior reconnect candidate is preserved separately. The v62 fixture SHA-256 remains `faa4d1b35267b54e134a8e0c84586fe1196d8fad5525cb53aa195aae53ca6e83`. - Current integration head passed all 86 focused tests, the full recursive repository typecheck and the root repository build. The normal server typecheck also passed, including its full Runner release rebuild. - Historical full recursive repository typecheck and the root repository build passed at `187bb85b4fa1fbd2ffaec992650a1c8557df9ebe`. Server typecheck also passed through the normal package script and a direct full server TypeScript check. The prerequisite full Runner release build passed. - The historical HTTP baseline returned 200 and dispatched one late launch RPC. The repair returned 200 and dispatched zero in the bounded fixture. - Earlier failures remain part of the evidence: the first head failed TypeScript package boundaries and source identity. One local HTTP run failed before migrations; it passed after the disposable test database received normal migrations. Earlier sandbox-wide checks ran out of memory or lacked Cargo. Those runs did not pass. - Historical head `187bb85b4fa1fbd2ffaec992650a1c8557df9ebe` received Greptile 5/5 with no actionable findings and all hosted gates passed in run `40412261957`: 53 successful check runs, four expected skips and a successful legacy Snyk status. Those results predate the master integration and do not establish current-head readiness. - Current integration head `2b96ee3adcb3be642c538436113e1159d92f843d` includes master `bd382eee536f4ada680072b1cc19100163c3d04a`. All current-head hosted gates passed in run `40512885858`: 53 successful check runs, four expected skips and a successful legacy Snyk status, with zero pending or failed checks. Build passed at 13:42:36 UTC and the final `ci / verify` aggregate (job `122224566238`) passed at 13:43:00 UTC on 2026-10-27. Greptile reviewed exact head `2b96ee3adcb3be642c538436113e1159d92f843d` at 5/5 (updated 2026-10-27 13:36:41 UTC), with no actionable findings and no open inline threads. The hosted partitions passed full-suite verification, including browser and release gates. The wider qualification limits below remain in force. - Local setup on current master initially failed frozen install with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` because `patchedDependencies` differs from the lockfile. Pinned pnpm 9.15.4 hydrated dependencies with `--no-frozen-lockfile --ignore-scripts`; its generated lockfile change was restored. This PR changes no lockfile or workflow. - Reproduce the Stop tests from `server`: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t 'remote runner process supervision|HTTP Stop during remote runner staging'`. The test accepts a guarded disposable `_test` database through `PAPERCLIP_TEST_DATABASE_URL`, or starts its normal embedded database. - Reproduce reconnect tests from `packages/paperclip-runner`: `pnpm exec vitest run src/live/runnerd-codex-transport.test.ts -t 'does not restart an exhausted reconnect'`. Build the normal test runner first. - Reproduce profile checks from `packages/paperclip-runner`: `node scripts/copilot-profile-identity.mjs --check` and `pnpm exec vitest run src/drivers/acpx/copilot-installation.test.ts`. ## Risks - The wider qualification is incomplete. Verified-stopped replacement and concurrent recovery plus Stop each entered two physical provider processes. Both fail the original requirement of at most one physical provider entry. Passing fixture assertions do not make all incident scenarios accepted. - HTTP concurrent recovery, shared provider-attempt accounting, server-admitted restored continuation, live-provider behavior and product cleanup settlement remain unrun or unproved. - Accepting a deferred signal is not proof that a dispatched process stopped. The existing monitor remains authoritative. - The HTTP test uses the real route, heartbeat cancellation and PostgreSQL state. Execution lease and event ports, plus the launch RPC, are controlled fixtures. Bootstrap expiry participates in this case. - The local incident matrix uses an installed runner binary and a scripted provider. It does not qualify a live provider. Local fixture teardown does not certify product cleanup settlement. - This change does not authorize historical turn replay. It does not change ownership, pause, approval or provider-attempt gates. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. The agent used reasoning, repository inspection, shell execution and connected GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 08:45:00 -05:00
Nicky Leach	ea727dc3c2	Preserve plugin session continuity through warm execution and checkpoint recovery (#17820)	2026-10-26 20:25:10 -07:00
DottaandPaperclip	32ec84db2e	fix(native): restore plugin checkpoints for same-run retries (#17814) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native sessions retain provider state across execution attempts. > - A same-run retry can receive a replacement plugin task lease. > - The executor loaded the old task checkpoint only for crash recovery. > - An ordinary retry could fail its local authority check before reading that checkpoint. > - This change restores the checkpoint through the existing authorized lease transfer. ## Linked Issues or Issue Description **What happened?** A native provider turn ended without its completion result. Paperclip scheduled an exact-checkpoint retry on the same heartbeat run. The retry acquired a new plugin task lease, then failed with `runner_state_identity_mismatch: authority_indeterminate`. The checkpoint restore branch required a `resume_dead_runner` restart claim. Ordinary same-run retries have no such claim. A controlled fixture with a retained remote checkpoint and no local lifecycle receipt reproduces the error. The original controller file contents were unavailable, so that part of the incident mechanism remains inferred. **Expected behavior** Load the retained plugin checkpoint after the existing policy proves the lease transfer is safe. Keep the original run, session, and saved control journal. Reject an unverified predecessor or a changed checkpoint identity. **Steps to reproduce** 1. Start a native session through a plugin task environment. 2. End its provider turn without a completion result and retain its checkpoint. 3. Retry the same run on a replacement task lease without a restart claim. 4. With no local lifecycle receipt, observe the authority error before the checkpoint read. **Paperclip version or commit** Reproduced against source `507dc70be26c191e372f09e58dc0965e73bf8f57`. **Deployment mode** Native Runner with a plugin task environment. The observed provider used ACPX Claude. The controlled executor fixture uses the shared native boundary. Related work: #17439 concerns recovery after controller replacement. This fix uses the existing checkpoint contract and does not depend on that PR. ## What Changed - Restore prior plugin state for an ordinary same-run attempt when its lease changes. - Keep the existing company, agent, workspace, retained-state, and termination checks. - Extend executor regressions for ordinary retries, mismatched checkpoints, and missing stop proof. - Exercise the same-run checkpoint read through the database-backed task API. - Document the recovery rule and verification recipes. ## Verification - Failing before: the ordinary retry reproduces `authority_indeterminate`. Both new denial cases also fail because the old branch never reaches the transfer policy. - Passing after: all six focused checkpoint cases pass. The saved control journal stays byte-for-byte unchanged. Rejected cases do not construct a provider backend. - `pnpm -r typecheck` passes. - Map and module-boundary checks pass. - All 858 executor, database lease-authority, plugin task, and ownership tests pass. `pnpm build` passes. - The broad local `pnpm test:run` attempt did not complete. It hit a 20-second timeout in the existing historical workspace-recovery HTTP Retry case. That exact case then passed in isolation on the unchanged head. The broad local attempt was stopped after the full CI matrix passed; it is not counted as a local pass. - Greptile reviewed current head `5bc11fa978ab87d3d9c5d575fa9666185aaafc2c` at 5/5 with no actionable findings. The full current-head CI matrix passed on attempt 2: 55 checks passed, no pending or failed checks, and two expected Storybook jobs skipped. - The first broader local attempt could not bind embedded PostgreSQL inside the sandbox (`EPERM`). It was stopped and restarted with loopback access. This was a setup failure, separate from the observed red regression. - The first CI attempt failed two existing checks: the process-adapter signoff happy path exhausted its run wait budget, and the workspace export scheduler did not reach `complete` within 20 seconds. Both tests pass in isolated local runs on the unchanged head. Both failed shards passed on their single rerun without code changes. The initial failures remain recorded. Their timing cause is not proven by the later passes. - Live acceptance at the fixed revision has not run. The hosted workspace still serves the earlier release. No deployment or retry of the original task is claimed. ## Risks The new checkpoint read runs only for same-run plugin recovery with a replacement lease. Missing termination proof, changed scope, or a changed Runner identity still stops execution. The lease resolver owns these checks. The fix does not replay tools or change retry budgets. A turn that omitted its completion result still needs the existing bounded continuation to finish useful work. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The runtime does not expose a more specific backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 18:01:55 -05:00
DottaandPaperclip	718c73dafe	fix(native): require verified remote process exit receipts (#17799) ... Require fresh ESRCH or an exact matching zombie generation before treating a remote Runner as stopped. Preserve uncertain ownership through adoption and transport close, retire local controller resources after failed close, and check Node before remote launch. Bind the shared transport change in Copilot profile 62 while retaining historical profile decoding. Add red-green process, permission, launch, adoption and cleanup regressions. Thanks to @hsm-lv for the original investigation in #17073. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-26 21:39:04 +00:00
DottaandPaperclip	9ae4c25e7e	perf(runner): reuse compatible protected provider packs (#17749) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts provider runtimes in remote execution environments. > - Sandbox images already contain qualified provider packs. > - Startup required the image pack to match the entire current app release manifest. > - An unrelated app commit could therefore cause pack verification, transfer, and installation again. > - This pull request selects protected packs by their provider interface and platform. > - The benefit is no extra pack probe, directory scan, version command, link, or upload on the compatible image path. ## Linked Issues or Issue Description **What happened?** Starting a remote task can reject a compatible image pack because its source revision or unrelated provider differs from the controller. Pack replacement added about 16 seconds in the investigated startup. Startup also repeated full content checks. **Expected behavior** Use an already qualified, protected pack when its selected provider interface and target platform are compatible. Keep the selected artifact identity for process launch and recovery. **Steps to reproduce** 1. Install a qualified provider pack in a sandbox image. 2. Start a controller built from another app commit. 3. Start a remote OpenCode task. The old complete-manifest comparison rejects the image pack even when the provider interface still works. Related repairs: #16493 reduced transfer cost. #17512 retained exact artifact authority during reattachment. This change preserves that recovery rule and changes fresh-launch pack selection. ## What Changed - Add explicit OpenCode and ACPX interface revisions to produced pack manifests. App Git SHAs and dependency pins remain provenance. - Read bounded protected-installation metadata in the existing final runner metadata query. Accept only root-owned paths that the execution user cannot write, including ancestors and manifest. Reject writable aliases. - Select and prepare the pack in server-owned environment setup before constructing the existing transport. Bind provider launch paths and digests from the selected installation. Save that identity for exact reattachment. - Keep full content verification for uploads, writable installations, and legacy packs. Reuse successful verification within one startup. Keep older attached runners compatible with the optional metadata request. - Seal image packs and document the one-time image refresh that enables reuse across later compatible releases. Keep selected ACPX agent qualification checks. ## Verification - Red/green regression: the old code rejects a protected pack from another app revision with `manifest mismatch`; the fixed path selects it without executing verification commands. - Native startup regression: select the installed pack's own Node digest with one existing runner metadata request and no pack verification or version commands. - Targeted pack, artifact, recovery, execution-driver, and Rust metadata tests pass. Incompatible interfaces/platforms, malformed identities, writable roots, path aliases, and exact reattachment are covered. - Final revision `91dc1676ff67bdbe5866c8586daac10240530cca`: workspace typecheck and build pass. All 53 current-head checks are successful, with no pending or failed checks. Greptile is 5/5 with no open findings. - The broad local `pnpm test:run` attempt reported a workspace-runtime conflict test failure; its isolated rerun passed. The broad run was stopped after that failure. Full CI now passes, including the server test shards. This is not a claim that the full local run passed. - Linux filesystem evidence: the production metadata functions read a real root-owned preinstalled pack as a non-root user in 16 microseconds median and 20 microseconds p95 across 100 warm reads. This measures metadata admission, not model or hosted response latency. - All 910 affected server tests and 57 Copilot attestation tests pass locally on the final implementation. Full CI covers the unchanged Runner transport. No provider qualification hash was refreshed. - Built the actual Linux daemon at `416e7ec7425d0d3565b4a5e1808002d650a60414` and ran its metadata command as UID 65534. The protected installation returned its real manifest with `immutable: true`; writable alias and missing-pack cases returned `immutable: false`. This is metadata/permission evidence, not provider execution. - Hosted rollout and a hosted end-to-end latency measurement have not run. ## Risks - Existing images need one rebuild to publish the compatibility declaration. Packs without it keep the legacy content-verification path. - Fast admission relies on image publication producing a qualified protected tree. Root execution and task-writable installations retain full verification. - Bump an interface revision when required runner/pack behavior changes incompatibly. Deploy a new image to roll out compatible runtime fixes or upgrades. - This removes provider-pack preparation delay. Other runner startup, context, model, and finalization costs remain separate. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol`, reasoning effort `xhigh`, with tool use and code execution. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 14:43:12 -05:00
DottaandPaperclip	063a498b54	Bound native OpenCode assigned-tool catalog projection (#17740) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native agents use assigned connection tools through a governed gateway. > - OpenCode received the gateway catalog directly, outside the bounded server tool projection. > - A large catalog can exceed a provider grammar compiler's capacity even when it fits the transport limits. > - Synthetic requests reproduced the Fireworks grammar timeout on the configured DeepSeek model. > - This pull request gives OpenCode two assigned-tool discovery and call schemas through the existing server relay. > - The gateway keeps the same grants, validation, approvals, revocation, and audit rules. ## Linked Issues or Issue Description **What happened?** A native OpenCode run could fail during provider grammar compilation before its first assigned-tool call. OpenCode advertised the full assigned gateway catalog. A synthetic catalog of 741 tool schemas reproduced `Grammar processing took too much time` on Fireworks through OpenRouter with `deepseek/deepseek-v4.1-flash-1017`. The same model and backend accepted a compact two-tool discovery/call catalog. This reproduces the mechanism; it does not replay a historical private request. **Expected behavior** Assigned-tool discovery must keep provider grammar size bounded. Every call must retain the run's original gateway binding and permissions. **Steps to reproduce** 1. Give a native OpenCode run a large assigned connection catalog. 2. Use a provider that compiles tool schemas into a constrained output grammar. 3. Observe the full catalog in the provider tool surface before this change. 4. After this change, observe only `paperclip_search_assigned_tools` and `paperclip_call_assigned_tool` for assigned tools. Discover and call a permitted tool through its pinned alias. **Paperclip version or commit** Source checkout based on current `master`. **Deployment mode** Native local or sandbox execution with OpenCode 1.18.34. Related work: Refs #14258 (discovery paging), #16678 (local adapter MCP launch), and #17688 (gateway capability advertisement). This change covers native OpenCode catalog projection. ## What Changed - Route native OpenCode assigned tools through the existing server-owned authenticated relay. - Use bounded discovery/call schemas for OpenCode, including small catalogs. Preserve transport capacity checks. - Keep assigned gateway credentials on the controller. Ignore gateway and credential values in tool arguments. - Preserve exact declared MCP operation names through admission and driver dispatch. Resolve transport prefixes only against the closed catalog. Reject unknown or shortened names. - Refresh former OpenCode direct-catalog and shortened-name checkpoints. Keep the existing fingerprints for other providers. - Add local and remote wiring, catalog capacity, alias, scope, work-mode, approval, and revocation tests. - Allow the native test files to use a validated dedicated external test database when embedded PostgreSQL is unavailable. - Document the assigned-tool execution contract. - Bound OpenCode discovery pages to 32 KiB and large-schema chunks to 16K characters. Keep other harness budgets unchanged. - Test medium pages and escaped large schemas through the actual HTTP MCP bridge on the explicit Runner testing surface. ## Verification - Red before the catalog fix: OpenCode did not create the server relay, and forced compact delivery advertised 1 or 741 direct schemas. - Red before the alias repair: HTTP MCP changed declared operation names, and the driver sent `search_assigned_tools` instead of the authorized `paperclip_search_assigned_tools`. - Current repair tests: 112 OpenCode/proxy tests pass, plus one actual Rust/proxy/HTTP MCP regression. Two checkpoint regressions fail with the former scoped revision and pass with the new revision. Other providers keep their prior fingerprints. - The combined QA candidate with this exact PR head passes all 916 native tests and all 113 transport tests on supported Node 24.22. An earlier Node 26.4/macOS run passed the 113 assertions but had an uncaught networking error. That non-clean run remains recorded; its stack matches [the upstream Undici defect](https://github.com/nodejs/undici/issues/5544). No production workaround or error suppression was added. - `pnpm -r typecheck`, `pnpm build`, and `git diff --check` pass on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`. The maps structural check passes. - The local `pnpm test:run` attempt stopped after host embedded PostgreSQL initialization failures. The relevant database suites pass with isolated PostgreSQL 18. All repository CI gates, including aggregate `ci / verify`, pass on the current head. - Synthetic live provider comparison: 741 invented schemas produced the exact Fireworks grammar timeout in 20.9 seconds. Two compact invented schemas produced a discovery call in 585 ms on the same model and backend. - The first hosted QA attempt exposed the alias failure described above. On the patched source, a real local Rust runner → proxy → OpenCode 1.18.34 → exact DeepSeek smoke completed discovery, one synthetic gateway read, and `paperclip_finish`, then emitted `turn/completed`. It used the production compact projection and unchanged Rust allowlist. - Two earlier local smoke attempts exposed missing terminal authority and completion-contract fields in the smoke harness. Those failures remain recorded. The corrected ordinary bindings produced the clean result. - Fresh Greptile: 5/5 on `bc41ba72be2195eabeb0d7727227ee0cd02e8532`, with no new actionable findings or unresolved threads. - Hosted QA publication and lifecycle acceptance remain pending. The separate deployed-agent `Paperclip Review` check is also pending; it follows a task that failed on the earlier deployed runtime. - Local OpenCode/OpenRouter acceptance at `bc41ba72`: 530 synthetic tools were authorized and installed. A simple model-ID reply completed in 17.3 seconds with no app calls. A discovery task completed in 37.6 seconds, used the search/call relay, made one upstream call, and returned the exact synthetic value. This is local source acceptance, not a staging sandbox timing comparison. - The large-result review finding has a failing-before/passing-after HTTP MCP regression. Five medium schemas now page without invalid JSON. A large schema with escaped text reconstructs exactly through bounded chunks. All 35 assigned-tool tests and the server typecheck pass after this follow-up. - Final head `d84ce4856a6e65a35ad5fb8b561492598b525f4e`: all CI gates pass. Greptile and Paperclip Review both rate the change 5/5. No review threads remain open. The final full build, server typecheck, and 35 assigned-tool tests pass locally. ## Risks - OpenCode uses discovery before it calls an assigned tool, including for small catalogs. This can add a model turn. - An existing OpenCode provider checkpoint rotates once because it retains the former tool declarations. Paperclip task history remains available. - The historical filtered catalog was not captured. The synthetic comparison proves the failure mechanism, not the exact cardinality of an earlier private request. - No model, provider route, permission, or automatic retry policy changes. ## Model Used The bounded-result follow-up also used OpenAI Codex (GPT-6 family). The exact backend model ID and context-window size were not exposed for that session. It used reasoning, code execution, and tool use. OpenAI `gpt-6.1-sol` in Codex, with reasoning, tool use, code execution, and source inspection. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 10:49:23 -05:00
DottaandPaperclip	5823e9a664	fix(runner): preserve disabled Codex web search (#17744) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Codex agents use the saved agent settings to prepare each provider launch. > - Onboarding saves web search as off, but native configuration conversion removes that setting. > - Codex 0.160.0 includes a hosted web-search tool by default, even in an authentication probe that asks it not to use tools. > - A gateway that permits only priced function tools can reject that request before budget admission. > - This change preserves the explicit restriction through native execution and disables web search for authentication probes. ## Linked Issues or Issue Description **What happened?** A native Codex authentication probe can include hosted web search. Normal task execution also loses a saved `search: false` setting. A restricted Responses gateway can reject these requests before it makes a model call. Related repairs: Refs #17445, Refs #17529. The roadmap and related PRs were checked; no duplicate repair was found. **Expected behavior** Authentication probes check the selected account and model without hosted web search. A saved search-off setting remains effective in native task launches, prepared launches, and session recovery. Existing behavior for an absent setting remains unchanged. **Steps to reproduce** 1. Create a Codex agent with search off and choose Paperclip Runner. 2. Use a Responses gateway that rejects hosted tools. 3. Inspect the request from the native authentication probe or task launch. Before this fix, the pinned Codex SDK includes `web_search` despite the saved setting. ## What Changed - Preserve an explicit search-off setting when native agent configuration is built and converted. - Carry the restriction in the durable native input and provider launch requirements. - Apply the restriction to authentication probes and normal launch paths, including custom arguments and recovery. - Reject prepared launches that discard the required restriction. Keep gateway billing rules unchanged. - Admit only the restrictive literal value in the strict plugin task API. Exercise the real plugin setup path. - Add regression coverage and document the policy. - Add the immutable Copilot v60 source identity required by the changed shared files. Preserve v59 and all original live evidence. Copilot authentication probes keep their prior launch requirements. ## Verification - Regression cases were run against the unfixed source and failed at the configuration and launch boundaries. - An isolated capture using the real Codex 0.160.0 app-server reproduced hosted-tool rejection. Disabling web search reached a mocked budget reservation. The helper and child processes permitted only loopback network access and used synthetic credentials. It made zero upstream calls. - Focused checks passed: 284 server cases, 61 native form cases, 198 backend/security/launch/parser cases, and 36 authentication/launch cases. A separate complete transport/launch/parser run passed 393 cases. These groups overlap. - Head `617b98f` passed workspace typecheck/build and 28/28 plugin contract and actual Runner probe tests with no skips. Final rebased head `20ff0c1` passed all current-head CI checks, including full isolated test lanes, and fresh Greptile and Paperclip reviews at 5/5. - The additive Copilot identity update passed 134 compatibility/policy/protocol checks and nine authentication probe checks. The real pinned Codex SDK capture was repeated after the final policy change: all five cases passed with zero upstream calls. - Two local full-suite attempts stopped after native PostgreSQL loader failures. Seven local dependency aliases were restored, and an isolated cluster initialized successfully. A third run later encountered host PostgreSQL semaphore exhaustion and was interrupted before source-profile regeneration. Its zero exit code after SIGINT is explicitly not a full-suite pass. Final full-suite verification runs in isolated CI; the local attempts remain recorded. - The first CI run exposed a missing strict plugin API field. Its five failures were reproduced locally, then passed after repair. The next run passed every underlying job except the source-identity guard. That guard now binds an additive identity with v59 retained. Final-head CI and both fresh reviews passed; addressed review threads are resolved. - The offline capture runs on Darwin ARM64. It does not prove the contents of a prior unretained Linux request or complete Cloud onboarding acceptance. ## Risks - The restriction must survive prepared launches and resumed sessions. Regression tests cover those boundaries. - Existing absent settings retain their prior behavior. This change does not add native search-on support or loosen gateway admission. - Older durable inputs without the optional restriction must remain valid. - Copilot now uses profile v60 for exact launch admission. Historical v59 inputs remain readable. The new declaration adds no live Copilot qualification and does not change its executable or admission flags. - A deployed Cloud acceptance run is still required after this source repair. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code execution, and delegated review. The exact backend model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 10:29:51 -05:00
Devin FoleyandPaperclip	b792165ba4	Expire native runner continuity archives and quarantine (#17616) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner sessions keep durable state under `<instance>/runtime/paperclip-runner/durable-sessions/` > - When a session starts a fresh provider session, the retired state is moved into `<session>/continuity-breaks/<epoch>-<uuid>/`; when state loses authority, it is moved into `quarantine/<session>.<reason>.<epoch>.<pid>.<uuid>/` > - Nothing ever removes either, and each can hold a full Codex home of several gigabytes > - A sleep-aware sweep already expires failover backup copies in the same tree > - This pull request extends that sweep to remove both kinds of evidence after a retention window, while keeping anything recovery may still read > - The benefit is that native runner disk use stops growing without bound ## Linked Issues or Issue Description Bug report described in-PR (bug report template). **What happened?** On one instance, one day of native runs produced 20 continuity-break archives totaling 4.8 GB, one session alone holding 4 GB. The quarantine folder held 12 entries totaling 2.1 GB, the oldest a month old. Neither is read again by normal operation (continuity-break archives are written by `archiveContinuityState` and never read), and nothing deletes them. They were removed by hand to free space and will regrow. **Expected behavior** Evidence older than a bounded window is removed automatically, as failover backup copies already are, without removing anything a live session or recovery path still needs. **Steps to reproduce** 1. Run a native Codex session that hits a continuity break (fresh provider session), or whose state is quarantined (for example `identity_indeterminate`). 2. Inspect `durable-sessions/<session>/continuity-breaks/` and `durable-sessions/quarantine/`. 3. Wait any length of time: the entries stay forever. **Paperclip version or commit:** `master` at e335265568. **Deployment mode:** any with native runner sessions. **Agent adapter(s) involved:** native runner (Codex, and any provider whose state lands in these folders). ## What Changed - `server/src/services/native-runtime/native-harness-backup-expiration.ts`: the existing tick also retires evidence. - **Continuity-break archives** (`<epoch ms>-<uuid>` under each session): removed when older than the window. Age is the epoch in the name, or the folder mtime when that epoch is in the future. `archiveContinuityState` refuses to archive state with warm-transition evidence, and no code path reads `continuity-breaks`. - **Quarantine entries** (`<session>.<identity_indeterminate|identity_mismatch>.<epoch ms>.<pid>.<uuid>`): removed by the same age rule. Native cleanup source archives (`<session>.identity_indeterminate.cleanup.<uuid>`) are referenced from recovery history, have no epoch, and keep their original inode's mtime, so they are always kept. - Every entry is skipped while its session is executing, initializing, warm, or closing in this process; this is checked again just before removal. Symlinked entries and symlinked `continuity-breaks`/`quarantine` folders are reported as `unsafe_path` and never followed. - Removal renames to `.expired-<uuid>` in the same folder and then deletes; later sweeps finish interrupted deletions. An empty `continuity-breaks` folder is removed. - Archives, quarantine entries, and leftover deletions count toward `maxEntries` and `maxRemovals`, and stop on `shouldContinue`. Paging resumes inside a session's archives and then in quarantine, so a bounded or dry-run batch always makes progress and never recounts a session's backup. - New result fields: `evidenceCandidates`, `evidenceCandidateBytes`, `evidenceRemoved`, `evidenceReclaimedBytes`, and skip reasons `recent_evidence` and `cleanup_archive`. The service also exposes `noteEvidence(writtenAt)`. - `server/src/services/native-runtime/native-runner-evidence-signal.ts` (new): a process-local signal. `native-session-executor.ts` raises it with the entry's epoch right after `archiveContinuityState` creates an archive and after `quarantineRunnerdStateRoot` moves state into quarantine; `server/src/index.ts` subscribes the sweep and unsubscribes on shutdown. - `server/src/config.ts`: `PAPERCLIP_NATIVE_RUNNER_EVIDENCE_RETENTION_DAYS` (default `7`, minimum `1`). - `server/src/index.ts`: passes the window and logs a batch when it found evidence candidates. - `doc/recoverable-disk-cleanup.md`, `doc/DEVELOPING.md`: document the variable and behavior. **Sleep-aware background work** (`rules/sleep-aware-background-work.mdc`): - *Trigger:* each entry's own expiry deadline (its epoch plus the window), plus a direct call from the executor when it writes new evidence. No timer, interval, or polling loop is added; the pass runs inside the existing sweep's tick and keeps its drain, warm-standby, and shutdown handling. - *Source of truth:* the filesystem entries themselves. After a restart the first sweep reconstructs the next deadline from them. - *Pending-work predicate / next deadline:* after a full pass the sweep keeps the earliest due time of the entries it retained; ticks before that time skip the evidence pass entirely (no readdir, no SQL). A busy session, a failed removal, or a full batch makes it due on the next sweep while that obligation remains. - *Quiet condition:* no evidence, or only evidence not yet due. The pass never queries the database, so retained evidence never causes recurring SQL. Dry run reports each candidate once rather than every hour. No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/native-harness-backup-expiration` (in `server/`, embedded Postgres): 52 passed. New cases: - old archives and quarantine entries are removed, recent ones kept; the empty `continuity-breaks` folder is removed; - age comes from the name, with mtime used for a future epoch; non-archive folders are ignored; - a busy session keeps its evidence, including when it becomes busy after the first check, and it is retried on the next tick; - dry run reports candidates and bytes, changes nothing (including leftovers), and reports each candidate once until new evidence comes due; - symlinked entries and symlinked evidence folders are skipped without touching the target; - an old quarantine entry is removed even when its session's issue is still open; cleanup source archives are kept; - the pass stays quiet until retained evidence is due, and a `noteEvidence` signal schedules newly written evidence; - inspected archives and leftovers count against `maxEntries`, and leftover removal stops on `shouldContinue` and resumes next tick; - a due pass stopped by drain before any session stays pending; one-entry batches finish without recounting a resumed session; - interrupted `.expired-*` deletions are finished; paging across ticks with `maxRemovals`, in normal and dry run, with no recount of the backup when resuming. - `native-session-executor.test.ts`: quarantining a legacy root raises the evidence signal with the epoch in the entry name. - `native-harness-backup-expiration-config`: default, explicit, minimum, and fallback parsing. - **Mutation checks**, each failing at least one test: removing the age check, the future-epoch fallback, either busy check, the busy retry deadline, the recent-entry deadline, the due gate, the `noteEvidence` update, merging the pass deadline, the entry and both folder symlink checks, the dry-run return, the dry-run leftover guard, leftover removal, the leftover budget check and its retry deadline, counting archives against `maxEntries`, the cleanup-archive skip, the empty-folder removal, the resume cursor, the per-archive budget check, the resume skip of the backup pass, not charging a resumed session, keeping the pass active across drain, and the executor's quarantine signal. - `pnpm exec vitest run config native-harness native-workspace-sync-retention native-session-executor server-startup` (in `server/`): 1272 passed on the previous commit; `config native-harness native-workspace-sync-retention server-startup`: 450 passed on the head commit. - `pnpm run typecheck` (in `server/`): pass. - Not run: server build and boot smoke. The continuity-archive signal call is exercised only by a runner integration test that needs the runnerd binary. ## Risks - **Quarantine as a fail-closed fence:** recovery (`recoverQuiescentRunnerdState`) and failed-run retry (`nativeFailedRunRetryStateIsSafe`) treat a quarantined entry for a session as a reason not to bootstrap fresh. Once such an entry ages out, a later run of that session can start a fresh provider session instead of failing with `quarantine_owner_unverified`. This applies only after the window, matching what backup expiry already accepts for closed work. - **Signal coverage:** the signal is process-local. Evidence written by another process sharing the same state directory is picked up when the next retained deadline is due or after a restart, not immediately. - **Legacy-path quarantine:** entries quarantined from legacy roots carry a legacy digest, so the in-process busy check cannot match them; they are removed by age only. - **Remote sandboxes:** archives moved aside on a remote runner (`<root>.continuity-break-<token>`) are outside this sweep. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-25 09:20:30 -07:00
Devin FoleyandPaperclip	34409a92c4	fix(native-runtime): stop failover backups from carrying throwaway provider state (#17618) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner sessions on managed sandboxes keep a failover backup of provider state, so a replacement sandbox can resume the session > - Each backup copies the provider's runtime directory, with a per-directory sha256 manifest, minus `excludeEntries` > - Codex already excludes credentials and re-downloadable plugin caches; OpenCode excluded nothing, and ACPX excluded only per-agent credentials > - OpenCode also leaked a temporary HOME on every launch, because removing it failed on the read-only skill snapshot inside and the error was swallowed > - So every backup carried every earlier launch's HOME (with its npm cache) plus re-installable plugin `node_modules` > - This pull request removes those homes correctly, sweeps homes left by earlier versions, and excludes the throwaway entries from OpenCode and ACPX backups > - The benefit is much smaller backups and live session directories, while session history is still backed up and restored ## Linked Issues or Issue Description Bug report described in-PR (bug report template). **What happened?** On one instance, current native failover backups totalled 24 GB across about 300 sessions. For OpenCode sessions: - `opencode/<session>/home-XXXXXX/`: 170 per-launch HOME directories across 67 sessions, 9.3 GB. Each held `.npm` (about 111 MB of npm cache), `.claude` (the skill snapshot) and `.local`. - `opencode/<session>/config/opencode/node_modules`: 68 sessions, 4.3 GB of OpenCode plugin installs. One ACPX session also carried `acpx/<session>/data/pnpm/store` (229 MB in the current backup, several GB in older copies). Cause of the leaked homes: `startRuntime` in the OpenCode driver creates a fresh `mkdtemp` HOME for each launch and seals the assigned skills inside it (directories `0555`, files `0444`). `close()` and the startup-failure paths removed it with `rm(..., { recursive: true, force: true }).catch(() => undefined)`. That fails with `EACCES` on the sealed directories, and the failure was ignored, so each launch left its HOME behind. **Expected behavior** Per-launch homes are removed when the runtime closes, and backups hold only state needed to resume a session. Caches that the provider re-creates are left out. **Steps to reproduce** 1. Run an OpenCode native session with at least one assigned skill on a managed sandbox, over several turns or runs. 2. Inspect `<session>/` in the OpenCode runtime directory: one `home-XXXXXX` per launch remains. 3. Inspect `failover-backups/current/opencode/<session>/`: the same homes and `config/opencode/node_modules` are in the backup. **Paperclip version or commit:** `master` at e335265568. **Deployment mode:** any with native OpenCode/ACPX sessions on managed sandboxes. **Agent adapter(s) involved:** OpenCode (native), ACPX. ## What Changed - OpenCode driver (`packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts`): - `removeIsolatedHome` unseals the skill snapshot with the existing `releaseMaterializedNativeRuntimeSkills`, then removes the HOME. Close and every startup-failure path use it. It only unseals when `.claude` is a real directory, not a symlink. - Per-launch homes are now created under `<session>/homes/home-XXXXXX`, so one exact path can be excluded. - When a session opens, `home-*` directories left at the session root by earlier versions are removed. History is not in them: OpenCode keeps it in `XDG_DATA_HOME` (`<session>/data/opencode/opencode.db`), and HOME only holds caches, state locks and the skill copy. - Exports `opencodeRuntimeSessionDirectoryName` (the existing session-directory naming) and `OPENCODE_ISOLATED_HOMES_DIRECTORY` for the server. - `resolveNativeHarnessPersistenceProfile` (`server/src/services/native-runtime/native-session-executor.ts`): - OpenCode excludes `<session>/homes` and `<session>/config/opencode/node_modules`. - ACPX (every agent) excludes `acpx/<session>/home/.npm` and `acpx/<session>/data/pnpm/store`, in addition to the existing per-agent entries. - `excludeEntries` are matched as exact relative paths (wildcards are rejected), so the session directory is computed in the same way as the existing ACPX entries. - Exposes the two new exports through the server's runner vendor shim. No code comments were added. ## Verification - `pnpm exec vitest run src/drivers/opencode src/drivers/runtime-context-materializer` (in `packages/paperclip-runner/`): 108 passed. The 1 failure, `uses provider structure rather than prose length to select a post-tool answer`, also fails on unmodified `master`. - The existing skill-tree test, with and without a startup retry, now asserts that homes are created under `homes/` and that `homes/` is empty after close. - New test: a sealed legacy `home-*` at the session root is removed when the session reopens, and `data/opencode/opencode.db` is kept. - `pnpm exec vitest run src/services/native-runtime/ src/__tests__/native-harness-backup-expiration.test.ts` (in `server/`): 5080 passed. `plugin-runner-probe` could not run locally because of a cargo manifest-path problem in this environment; it is unrelated to this change. - Profile tests cover the new OpenCode and ACPX exclusions, including a session id that needs sanitizing. - New checkpoint/restore round-trip tests for OpenCode and ACPX. They use `syncRemoteRunnerDirectoryOut` for the backup and `stageRemoteRunnerDirectory` for the replacement. Excluded caches are absent from both the backup and the replacement, while history, config and workspace files are restored and the live directory is unchanged. - **Mutation checks**, each failing its test: - removing the skill unsealing (homes leak again; 3 runner tests fail); - removing the legacy sweep (1 test fails); - emptying the OpenCode/ACPX exclusion lists (4 server tests fail). - Checked by hand against the bundled OpenCode 1.18.34 binary, with the driver's environment: - launching with a fresh HOME and no `config/opencode/node_modules` still lists earlier sessions; - OpenCode reinstalls the plugin dependencies on launch. - `pnpm run typecheck` in `server/` and `packages/paperclip-runner/`: pass. - Not run: server build and boot smoke. ## Risks - **Rollout:** existing backups still contain legacy `home-*` directories. After a restore, the sweep removes them when the session opens, and the next backup no longer carries them. - **First launch after restore:** OpenCode reinstalls its plugin dependencies and npm repopulates its cache. This needs network access to the npm registry, the same as a brand-new session. - **Crash leftovers:** a HOME left behind by a hard crash (the runner killed before `close()`) stays under `homes/` until the session directory is removed. It is still excluded from backups. Homes under `homes/` are not swept on open, to avoid touching a HOME that a concurrently running runtime might still use. - **pnpm store:** dropping the pnpm store from ACPX backups means the next `pnpm install` in a replacement sandbox downloads packages again. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, and running the real OpenCode binary to check what state it reads on resume). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-25 08:51:39 -07:00
DottaandPaperclip	aec86c794b	fix(runner): preserve native work through controller outages (#17594) ... Keep remote native work under its authenticated connection lease through controller outages. Hold durable ACPX tools through reconnection, align the native Claude HTTP MCP client budget, and wait for the recovered handler before dispatch. Preserve Stop, cancellation, exact identity and artifact checks. Verified with real-process regressions and a real Claude Opus 5.5 Daytona workflow: 125 seconds offline during useful work, original process and lease retained, no failed native tool calls, queued follow-up completed, independently verified ZIP and committed workspace receipts. All current-head CI gates pass and Greptile is 5/5 with all findings resolved. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-25 09:01:24 -05:00
DottaandPaperclip	2a9d527c68	fix: preserve native startup during controller restart (#17511) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runtime keeps a durable runner alive across controller restarts. > - A restart can begin while a provider session is still opening. > - The old controller fenced read-only run admission before it could publish that prepared session. > - Admission failure then closed the runner that shutdown needed to detach. > - This pull request lets only the existing startup finish admission within the bounded shutdown wait. > - The controller detaches that exact session before it can submit a turn. New work and state changes stay fenced. ## Linked Issues or Issue Description **What happened?** A controller restart during native provider startup could close the prepared runner with `native control-plane run admission failed`. Recovery then found no running process and placed the task in a reconciliation hold. The same admission path follows both fresh startup and replacement after a failed provider resume. **Expected behavior** Shutdown must detach the existing prepared session so the next controller can recover it through the normal identity checks. Shutdown must not admit a new executor, submit a turn, or override an explicit Stop. **Steps to reproduce** 1. Delay a native backend while it opens its provider session. 2. Request controller restart detachment for that run. 3. Let backend startup finish before the 30-second deadline. 4. Observe that read-only admission fails and shutdown reports the run as inactive instead of detached. **Additional context** Related work: #17073 handles remote process-monitor classification; #17459 handles macOS process fingerprints; #17097 covers broader hot upgrade recovery. This change fixes the startup admission fence. It does not relax process identity or provider launch-profile checks. ## What Changed - Separate the read-only admission guard from the guard on control-plane state changes. - Let shutdown await only the startup already registered for the run. Close its admission window when the wait ends. - Reject a new execution after its run is fenced for shutdown. - Detach the prepared session and exit before runtime-question attachment or provider turn submission. - If Stop is pending before or after publication, retain its exact cancellation acknowledgment and cleanup owner across authority reads and runtime-question attachment. - Share one 30-second admission/Stop-settlement deadline across the restart sweep. Report an unsettled run as pending while healthy runs continue hot-restart adoption. - Cover the real runtime and admission path, deadline expiry, new-start rejection, persisted binding checks, and write fences. - Document the bounded startup handoff in the restart ownership contract. ## Verification - Red: the new actual-runtime/admission regression failed on the previous code. Shutdown reported the run as inactive and did not detach it. - Red: review regressions reproduced detach before a post-publication Stop acknowledgment and an actual hot-restart preparation error when startup Stop cleanup timed out. - Green on `e10d0cb238`: 18 startup/restart tests passed, including Stop during authority reads and runtime-question attachment, the real recovery preparation and shutdown coordinator with a stalled Stop plus healthy peer, and the shared sweep deadline. The earlier 36-case Stop/restart selection also passed before this follow-up. - Both complete changed test files previously passed on head `8e9c2610fe`: 857 tests. They used real PostgreSQL 18 with a fresh database per fixture. An earlier local harness attempt reused one database and hit duplicate fixture keys; that harness was corrected before this complete pass. - Real PostgreSQL: 7 admission and revocation tests passed. These cover wrong runner identity, persisted binding mismatch, checkpoint/result rejection, and in-flight write revocation. The local test harness used a fresh Docker PostgreSQL 18 instance because macOS embedded Postgres exhausted the host semaphore limit. Only database provisioning was replaced; assertions and production port code were unchanged. - Transport safety acceptance: the current source transport controlled a real Linux ARM runner and a deterministic Codex test provider in a disposable Debian container. After startup, detach and generic cleanup preserved the process. A new controller authenticated the same PID and provider session. The provider log had exactly one session start and no turn start. Duplicate launch was forbidden, the retained workspace was unchanged, and no stop or suspend command was issued. - The Linux acceptance check tests the shared native transport safety contract. It is not a live Claude response, a hosted Daytona run, or recovery of the original task. Those remain unverified. No original task, sandbox, credentials, or permissions were changed. - `pnpm -r typecheck` and `pnpm build` passed before the review follow-up. Server typecheck passed on current head `e10d0cb238`; current-head CI also passed the full typecheck and build gates. - The full local `pnpm test:run` command was attempted, then stopped after repeated embedded PostgreSQL bootstrap setup failures. That broad local run is incomplete. The current-head four-file Docker PostgreSQL run passed 879 tests, with one existing warm-copy `claim-uncertain` case timing out and recovery failing to import because the temporary provisioning mock omitted a support export. The harness is corrected and affected cases are being rechecked. The timeout is unexplained; it is not being labeled an infrastructure failure. No assertions are skipped by the database provisioning replacement. - All 55 clean-host CI checks passed on current head `e10d0cb238` (2 conditional skips), including all general test shards and native runner verification. Greptile reviewed that exact head at 5/5 with no findings. The previous Paperclip review findings are addressed in [the review response](https://github.com/paperclipai/paperclip/pull/17511#issuecomment-6391520242); its new advisory review remains queued. ## Risks - Admission must stay read-only. All writes retain the original synchronous controller fence, and both admission checks still validate the exact persisted binding. - Startup that exceeds the 30-second shutdown wait still fails closed. It cannot claim a successful detach or start a turn. - No schema migration or provider configuration change is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and test execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 21:05:45 -07:00
DottaandPaperclip	5c3dce09b5	fix(runner): preserve verified remote artifacts across controller releases (#17512) ... ## Thinking Path > - Paperclip manages AI agents and owns their task authority. > - Remote native agents can keep working while their controller restarts. > - The controller must authenticate the same runner and verify its provider files before recovery. > - The current path compares a retained provider pack with the new controller release. A release change can reject intact files. > - The new controller binary digest can also reject the original daemon during PRP authentication. > - This PR records the admitted release authority in private controller state and uses it for exact process reattachment. ## Linked Issues or Issue Description **What happened?** A remote Claude session started and used tools. After controller shutdown, recovery failed with `runner_remote_provider_artifact_incompatible`. The transport combined stderr into stdout, so the error omitted the verifier reason. The historical rejected manifest is unavailable. This PR reproduces controller release drift as a cause; it does not claim that the historical mismatch is proven. Reaching pack verification implies the earlier exact-process/controller-identity checks passed. That supports a preserved-state recovery path but is not a direct inspection of the deployed volume. **Expected behavior** A compatible, unchanged remote runner can reattach after a controller release changes. Full package hashes, exact process scope, protocol capabilities, and authenticated runner identity must still pass. Fresh launches use the current release. **Steps to reproduce** 1. Verify and admit a remote native session with release A. 2. Preserve its runner process and replace the controller with release B. 3. Reattach the original process. The old path compares its pack with B and fails at `manifest mismatch`. 4. With this fix, the controller reads its saved A authority and verifies all A hashes. Changed files still fail. **Paperclip version or commit** Reproduced from master `470cb34c0e42cb9d0b232d0bcf5d45cc7b0a81ca`; rebased onto `a3675ddcdf`. **Deployment mode** Native remote runner on a Daytona-backed Paperclip Computer. Local acceptance uses isolated Linux containers and actual packaged artifacts. Related work: #17361 and #17448 handle runner upgrade and qualification permissions. #17097 and #17439 add durable controller checkpoints and server adoption. That stack can skip remote artifact preparation with a saved checkpoint. This PR fixes the current file-backed path and retains full provider-pack verification; it has no checkpoint schema or empty-disk controller recovery. ## What Changed - Save the complete verified pack manifest and authenticated runner version/digest after successful session admission. - Bind the receipt to company, agent, workspace, session, runner, environment lease, remote directory, and provider descriptor. Keep it outside guest backup directories. - Reuse the saved authority for exact reattachment. Keep current release selection for fresh launches and explicit session replacement. - Fail closed for mismatched or corrupt receipts. Historical sessions without a receipt retain the current release integrity gate. - Report bounded, redacted verifier output from both transport streams. Warn on receipt write failure while returning the already admitted session to its owner. - Add scope, persistence, replacement, storage failure, and backend wiring regressions. Update recovery documentation and the feature map. ## Verification - Test-first red: the original selection chose release B instead of A after controller recreation. The new receipt regression passes. A separate red test showed the missing stdout diagnostic; it passes after the diagnostic fix. - Authority and diagnostic suites: 49 tests pass. Backend pack, process adoption, and harness persistence selection: 47 tests pass. - Actual Linux provider image: complete A manifest/artifact verification passed; controller source-revision drift failed on the old path; saved A authority passed; a modified ACPX sidecar failed its SHA-256 check. - Actual native Linux runner: authenticated a session, detached the controller, changed the controller manifest and binary, then authenticated the original PID and provider session with saved authority. There were zero duplicate spawns, zero new provider turns, no stop/suspend, and the workspace file remained. This probe used the real runner with a deterministic Codex fixture; it is separate from the real ACPX package integrity proof. - Full workspace typecheck and build passed. The full native executor suite first hit host PostgreSQL semaphore exhaustion. The complete changed files then passed 864/864 tests with separate real PostgreSQL databases in a disposable Linux container. The temporary local fixture changed only database provisioning and used normal repository migrations. Current-head CI also passed the full required test, Runner, browser, typecheck, and build lanes. - `node scripts/check-maps.mjs` and `git diff --check` passed. These structural checks do not prove live recovery. - Current head `48fca2ae33d99f7f6d2fe12f431cf76d25185c70`: 55 successful checks, two conditional skips, no pending or failed checks; fresh Greptile 5/5 with zero findings and zero unresolved review threads. Commitperclip quality review passed. - No original task was retried or changed. Existing Daytona access could not resolve the original sandbox or its image; a fresh staging Claude recovery is not claimed. ## Risks This prevents recurrence for sessions admitted with the new receipt. It does not retroactively recover the historical failed run. Saved authority exists only after this change has admitted a session. It cannot repair an ambiguous historical launch or restore a receipt on an empty replacement controller. A stale protocol or modified pack remains a hard failure. Receipt write errors warn without losing the live session handle, but later recovery still needs matching authority. The broader checkpoint stack changes the same server preparation and runner-artifact option and will require a deliberate integration. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 16:44:50 -07:00
Nicky Leach	410e5c6eed	fix(native): use current authority for warm plugin completion (#17444)	2026-10-24 10:54:40 -07:00
Nicky Leach	62f60bf158	Measure first deliverable text in native run traces (#17434)	2026-10-24 09:40:08 -07:00
DottaandPaperclip	71c04c0bfb	fix: stop idle environment cleanup polling (#17145) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent environments can leave pending teardown or unverified provider resources. > - Recovery must preserve those resources until ownership and workspace safety allow cleanup. > - Empty maintenance scans keep an idle instance and its database active. > - The existing rows, transaction signals, and scheduler can represent the actual recovery obligation. > - This pull request schedules pending cleanup and historical verification only while work remains. > - An empty or fully verified instance stops these scans without losing cleanup recovery. ## Linked Issues or Issue Description **What happened?** The server scanned pending environment cleanup on its maintenance tick. The run reaper also scanned it even when it recovered no runs or active leases. Historical resource verification scanned every five minutes, including when all history already had valid cleanup proof. **What did you expect?** Writes should wake cleanup. Pending rows, deferred orphan buffers, and unverified history should retain recovery deadlines. When none remain, these workers should make no periodic database or provider calls. **Steps to reproduce** Start a configured instance with no pending leases and fully verified history. Leave it idle. Observe repeated pending-cleanup and historical-verification reads from the maintenance paths. Related: #17106 supplies the preceding login-cleanup migration. #13523 improves lease-reaper query indexes. #16809 addresses reusable-sandbox recovery authority. This PR keeps those authority checks in their existing services. ## What Changed - Register two workers with the existing delivery coordinator and shared scheduler. Remove their unconditional maintenance-tick calls. Run the reaper's inline pending cleanup only after it recovers actual work. - Add commit-aware signals to the audited lease and custom-image writers. Existing transactions notify directly; other writers use one small transaction helper. No tables or migrations are added. - Derive pending cleanup deadlines from the existing row backoff and retry or claim metadata. Retain a historical retry only while an unverified candidate exists. - Register unwritten orphan buffers across runtimes that share the database handle. Buffer registration wakes recovery without a database connection. Load the existing spool at startup and retry failed or capacity-limited loads while needed. - Isolate spool failures per runtime. Attempt all buffers and persisted pending rows, then retain a retry for any failed spool. - Document producer coverage, idle behavior, and remaining blockers. Update the sleep inventory. - Repair the inherited CSS-recovery E2E test: the current build no longer emits task-page CSS. Exercise the actual lazy environment stylesheet, preserving real network failure, recovery, and bounded reload assertions. ## Verification October 24 integration update: merged current master and resolved the runtime/spool conflicts. Preserve serialized ingestion of records appended after startup. Plugin replacement failures now register deferred recovery with an idle hold; concurrent callers share one persistence attempt. The producer audit includes this new writer. - Focused cleanup selection: 270 tests now pass across five suites. The initial run lacked a built plugin SDK and executed no tests; building the SDK fixed setup. The new concurrent-flush assertion then reproduced duplicate rows; a single-flight guard fixed it and all 18 plugin integration tests passed on rerun. - Server typecheck passed. After review fixes, all 211 tests across the cleanup worker, real orphan spool, environment runtime, and plugin integration suites passed. These overlap the earlier focused selection. The rate-limit test initially expected a write after twenty idle minutes to wait; corrected it to assert that subsequent writes cannot accelerate the five-minute cadence. - Review fixes preserve retries for unreadable spool directories/records, keep historical provider checks off the startup wait while joining them at shutdown, and restore their independent five-minute minimum spacing. The complete [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/40108905033) passed on final commit `79ae1589c3c7b924db4bedb337d0c7ed3d95020b`, including all E2E shards, server/workspace tests, runner checks, typecheck, build/public-install verification, and security checks. - CI found a stale E2E asset matcher: `issue-detail-*.css` is no longer emitted. Both corrected browser tests passed against a fresh production UI build and disposable local instance. Fresh Greptile review is 5/5 on that same final commit, with all review threads resolved. Previous implementation verification: - `pnpm -r typecheck` and `pnpm build` passed. - New PostgreSQL worker suite: 12 tests passed. Covers quiet configured state, verified history, writes, outer commit and rollback, retry deadlines, restart, drain, deferred orphan buffers, spool failure, pagination, provider uncertainty, shutdown, and the idle run-reaper path. - Worker and environment runtime suites after review fix: 183 tests passed. A new assertion reproduced buffer starvation before the fix. Updated transaction fault injection and isolated simulated-instance database handles. Server typecheck also passed after the fix. - Run-recovery suites: 58 tests passed. Pending-cleanup suite: 38 tests passed after updating the logging fixture to seed real orphan recovery; an idle reaper no longer invokes cleanup. - Stop and workspace-recovery integration selection: 268 tests passed; two live-provider tests skipped by their environment gate. This overlaps the worker suite and is not an additional unique-test count. - Started the full local `pnpm test:run`, then stopped that older duplicate run after CI found the fixture mismatch. The corrected 38-test suite passed locally. The full sharded [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39678174479) passed on final commit `fc0ff3c4554ca4190a8b0b81ec06ef7065b8ad2c`, including build, typecheck, runner, server, workspace, and E2E checks. This is not a claim that the local full run completed. ## Risks - Missing a future producer signal could delay discovery until restart. The producer audit is documented in `doc/idle-environment-cleanup.md`. Independent SQL writers and another server process are outside the existing process-local notification contract. - Historical candidates blocked by ownership or provider uncertainty keep a five-minute retry until they can be verified. Provider destruction, claim fencing, Stop retention, and workspace safety remain controlled by their existing services. - This does not remove orphan-active discovery or legacy shutdown settlement from run recovery. Active custom-image expiry is also still periodic. External wake coverage does not change. - Local tests prove these workers become quiet and recover. They do not establish deployed instance suspension or fleet cost savings. ## Model Used OpenAI Codex (GPT-6), with code execution and reasoning. The exact serving model ID and context size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 09:04:49 -07:00
Nicky Leach	6b4bdc515b	Fix GitHub ownership checks for warm plugin tasks (#17417)	2026-10-24 08:26:28 -07:00
Nicky Leach	aa0037a137	Support Codex warm plugin task reuse and recovery (#17390)	2026-10-23 19:14:14 -07:00
DottaandPaperclip	bf223596ab	fix(runner): bootstrap fresh sessions in replacement sandboxes (#17320) ... Reuse the untouched-session proof and atomic directory claim when a sandbox replacement carries a provider session that has never run. Retain verified restore for saved backups and all prior or ambiguous session activity. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-23 15:35:24 -07:00
Devin FoleyandPaperclip	b5bf86ea09	Clear process identity when settled native runs stop (#17374) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each heartbeat run stores the process identity of its runner in `heartbeat_runs.process_pid`, `process_group_id` and `process_started_at` > - Only cancellations (`release_run_resources`), restart recovery and heartbeat recovery cleared these fields > - Failed native runs take the `schedule_retry` path, so they kept a PID after the run finished, for weeks > - Liveness checks for old runs then depend on PID-reuse defenses, and code that reads a set PID as "running" is misled > - This pull request releases the process identity of settled native runs when the process is confirmed gone, and backfills old sandbox rows > - The benefit is that a finished run no longer looks like it owns a live process, while live, warm and retrying processes keep their identity ## Linked Issues or Issue Description Refs #17358. Bug report described in-PR (bug report template). **What happened?** On one instance, 16 of 23 old native workspace sync folders belonged to failed or cancelled runs that finished more than a week earlier and still had `process_pid` set. The PID is written when the run's process starts (for remote targets it is the sandbox PID). The status arbiter clears it only through `release_run_resources`, which only cancellations emit. Failed native runs take `schedule_retry`, and other terminal paths (finalization reconciler, restart recovery terminal failures, watchdog timeouts) do not clear it either. **Expected behavior** When a run is terminal and its process is confirmed gone (or no longer owned), the process identity is cleared. A process that may still be alive keeps its identity. **Steps to reproduce** 1. Run a native task whose provider turn fails, so the arbiter decides `run_failed_partial_evidence_preserved` with a `schedule_retry` effect. 2. Wait for the run to finish and the runner to exit. 3. Read the run row: `status = 'failed'` and `process_pid` / `process_group_id` / `process_started_at` are still set. **Paperclip version or commit:** `master` at 75fcb0b3fc. **Deployment mode:** any. **Agent adapter(s) involved:** native runtime (`paperclip_runner`); local and sandbox targets. ## What Changed - `releaseSettledNativeRunProcess` (in `native-local-process-stop.ts`) clears `process_pid`, `process_group_id` and `process_started_at` of a native run under a row lock, only when all of these are true: - the status is `failed`, `cancelled`, `timed_out` or `interrupted`, and `finished_at` is set; - the finalization coordinator is absent, `committed` or `terminal_failure` (a `retryable_failure` same-run retry keeps the existing heartbeat recovery path; unfinished finalization keeps its identity); - the run has a session checkpoint or provider evidence, so native session resume never mistakes it for an unused bootstrap; - local target: every lease is released without failed cleanup, the latest `native.process_identity_recorded` event names this host and the same PID/group, and the PID and group probe absent. The release writes the existing `native.local_process_stopped` receipt in the same transaction, so `hasNativeLocalProcessStop`, `readNativeLocalProcessStop`, explicit continuation and acknowledged Stop keep their proof; - sandbox target: every lease has a remote termination receipt (sandbox PIDs are never probed on this host). - Succeeded runs keep their identity: a warm runner can serve the next turn, and chat retry matches warm predecessors by PID. - The heartbeat executor calls the release after lease and scratch cleanup at a terminal boundary (not when a same-run resume, workspace finalization or held ownership is pending). - Background work stays idle (see `rules/sleep-aware-background-work.mdc`): - Trigger: the executor settlement calls `settleNativeRunProcess`. If the release is not ready yet (process still alive, local lease not released, remote termination receipt missing), the run goes into an in-memory pending set. - Startup: the first `reapOrphanedRuns` pass after boot runs `discoverSettledNativeRunProcesses` once, in pages of 50, over the durable rows. This reconstructs work after a restart and covers terminal transitions outside an executor (reconciler, restart recovery, watchdog, timeouts). - Retry: later passes call `retryPendingSettledNativeRunProcesses` only for pending runs. It makes no database call when the set is empty. A run leaves the set when it is released, becomes ineligible, or reaches 20 attempts. - Runs that this process still executes are skipped. - Stop receipts now also record `processStartedAt`. `withReleasedNativeRunProcess` restores a released identity from the latest stop receipt. Retained session cleanup (`reconcileRetainedNativeSessionCleanup`) and warm transition restart checks use it. Quiescent runnerd state recovery reads the stop receipt when the prior run's identity was released. Without a receipt, all three still treat the identity as unproven. - Migration `0377_clear_settled_native_run_process_identity` backfills sandbox runs (without bumping `updated_at`) with the same rules: native, settled status, `finished_at` older than 1 day, settled coordinator phase, checkpoint or provider evidence, and at least one lease where every lease has a valid remote termination receipt. The snapshot is carried forward as `0377_snapshot.json`. - Not changed: legacy (non-native) runs. Case handoff release, scratch cleanup and conversation continuation read the terminal row's local PID/group as process-tree absence evidence, so clearing them needs separate work. `release_run_resources` cancellation behavior and `native_run_finalizations.controller_*` fields (they track the server controller, not the run process) are unchanged. ### Migration margin The 1-day margin is defense in depth. The real proof is the remote termination receipt: the provider confirmed that the sandbox stopped or was destroyed. One day is far longer than lease release, sandbox stop, restart recovery takeover and the finalization reconciler sweep, so no in-flight teardown or same-run recovery can still read these rows. Local rows are not backfilled in SQL: only the host that recorded the process can verify that it stopped and write the stop receipt. The runtime sweep releases them on that host. ## Verification - `pnpm exec vitest run src/__tests__/native-settled-process-release.test.ts` (new, 20 tests): release with a stop receipt and restore of the identity from it; `cancelled`/`timed_out`/`interrupted`; live process kept, then released after exit; kept for succeeded, `retryable_failure`, `workspace_finalizing`, unreleased local lease, another host, no evidence, unconfirmed sandbox, mixed leases; sandbox release with receipts; startup discovery paging and skip; pending retry; retry cap; no database access while nothing is pending; `reapOrphanedRuns` discovery, then retry after the process exits; executor release for a failed native run and no release for a succeeded run. - `native-provider-failure.integration.test.ts`: a real `schedule_retry` decision (`run_failed_partial_evidence_preserved`) leaves the failed run with its PID after finalization, and the release then clears it and the stop receipt reads back. - `native-session-executor.test.ts`: quiescent recovery and retained session cleanup accept a released identity with a stop receipt and reject one without a receipt. - `packages/db` `client.test.ts` migration rule: the backfill does not set `updated_at`. - `native-finalization-migration.test.ts`: migration 0377 clears only old settled native sandbox rows with confirmed termination (15 fixtures, including recent, succeeded, running, legacy, unsettled phases, no evidence, unconfirmed or local leases, no lease). - Related suites, all passing: `pnpm exec vitest run` on heartbeat-process-recovery, native-workspace-sync-retention(-config), native-finalization-{migration,settled-candidates,recovery}, chat-failed-run-retry-boundary, acknowledged-native-stop, explicit-native-continuation, conversation-continuation, native-session-resume, native-run-finalizer-telemetry, native-session-executor, native-safe-replacement, native-finalization-reconciler-timing, native-restart-recovery, native-finalization-recovery-state, run-lifecycle, agent-instruction/directory working copies, heartbeat-case-handoff (1920 tests across 23 files). - `pnpm run typecheck` in `server/`: passes. `pnpm run check:migrations` and the snapshot drift test in `packages/db`: pass. - Mutation checks (each reverted alone; a test failed every time): executor call, startup discovery, pending retry in `reapOrphanedRuns`, retry cap, pending-set cleanup on retry, start time in the stop receipt, retained cleanup receipt fallback, coordinator phase guard, succeeded exclusion, liveness probe, host check, remote receipt check, provider-evidence guard, local lease check, quiescent recovery receipt fallback, migration margin, migration NULL-safe receipt check, migration phase guard. The NULL-safe check came from a real bug the migration test found. The warm transition restart call site has no direct test: that path has no existing fixture. It uses the same tested helper. - Not run: the full server suite and e2e tests. ## Risks - Behavior change: more native runs now have a null `process_pid` after they settle. Every native reader was checked: explicit continuation, acknowledged Stop and agent stop use the stop receipt for null identities; cancelled-startup detection also requires no process events, so a released run is not mistaken for a cancelled startup; native session resume selection is protected by the evidence guard; chat retry reads its warm-predecessor PIDs from succeeded runs, which are not released. - A recycled local PID keeps a row unreleased (conservative), and sandbox runs from providers without termination receipts keep their identity. - Background cost: one paged discovery at startup. After that, only pending runs are checked, at most 20 times each. With nothing pending there is no recurring query. - A transition outside an executor that happens after startup, with no executor settlement, is released at the next startup discovery. - Migration: a data-only `UPDATE` on `heartbeat_runs` with a selective `WHERE`; it passes the migration safety check. It is safe to re-run. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 15:09:43 -07:00
Nicky Leach	412914cfa7	Read native plugin task output files on the execution host (#17285)	2026-10-22 23:38:09 -07:00
Nicky Leach	2cd6eb27c7	fix(runner): record the plugin-prepared working directory (#17243)	2026-10-22 20:42:26 -07:00
3 people	6dda8cc2b7	fix: preserve Stop authority and qualify complete recovery journeys (#17105) ... ## Thinking Path > - Paperclip lets people manage agents that do useful work. > - Users need complete tasks, clear questions, retained context, and dependable recovery. > - Separate run tests do not prove that the complete user experience works. > - Successful retries can hide earlier failures and missing evidence. > - This PR extends the existing Product E2E harness with three explicit journeys. > - It adds a qualification checkpoint over the full retained result pool. > - Independent artifacts, state, rendered conversations, and delivery receipts establish each outcome. ## Linked Issues or Issue Description Refs #17088, #17089, and #17046. This PR repairs two Stop lifecycle defects and combines the qualification changes previously reviewed in #17095, #17090, and #17099. Daily operations remain with #17051. It adds no second scheduler or recovery engine. **What happened?** A completed run can still leave a user without useful output, an understandable question, or a working continuation. Existing isolated tests and best-attempt reports do not establish the combined behavior. **Expected behavior** A necessary question includes its explanation and resumes once after an answer. Stop prevents automatic continuation until the user resumes. Restart during finalization preserves the accepted provider result and delivers it once. Qualification requires independent evidence for six journeys on both agents through Slack and Paperclip. **Steps to reproduce** List the necessary-question, followup-stop-resume, and finalization-recovery suites using the documented selectors. Run one exact cell with the configured model and target. Supply a private qualification manifest to the existing report command to grade all retained attempts. Missing proof and incomplete coverage must fail. ## What Changed - Wake existing accounting in the same committed native Stop closure. A receipt that arrives before closure must still settle once; incomplete receipts remain held. - Require a later user message before native Stop can promote earlier saved input. Preserve those earlier messages for the authorized resumption. - Add explicit necessary-question cells. Verify the actual prompt and context, answer, one bound continuation, and the delivered document link and revision. - Add explicit follow-up, Stop, and later resumption cells. Verify a live owned provider turn, cancellation acknowledgement, at least five seconds without an automatic path, retained direction and workspace context, and an actual browser download. - Add explicit Daytona finalization-recovery cells. Stop the owned controller after public result acceptance and during workspace finalization. Require the original result, restored file hash, resumed operation, one provider turn, and one receipt-bound delivery. - Share the receipt-based delivery observer across both recovery journeys. Keep model prose separate from the authoritative delivered comment identity. - Add a 24-cell qualification checkpoint over every retained attempt. Require three fresh candidate passes per cell and later staging proof with exact source, runtime, model, settings, instructions, and independent evidence. - Reject absent or unsafe manifest evidence. Preserve failed attempts and incomplete observations in the normal reports. - Provide four explicit rotating daily selections. Refuse daily selection until all required live drivers exist. Keep scheduling and spending with the existing operations owner. - Document the exact selectors, evidence contract, limitations, and remaining staging work. ## Verification - Local Product E2E support: 3,554 passed with two existing skips; source prerequisites: 128 passed. E2E typecheck passes. The actual file-chip and Download-control browser regression passes. - Two real-database regressions fail before the production repair. The stopped coordinator leaves accounting asleep after a complete receipt; the saved queue can restart from pre-Stop input. The repaired settlement and cancellation cohort passes 66 checks. The complete 256-case continuation suite passes with stronger two-phase Stop expectations, and 66 native executor accounting/Stop safety cases pass. Server typecheck passes. - All required CI, repository build/typecheck/tests, and fresh Greptile 5/5 pass at the final head 232d3c52d0, with no unresolved findings. - Both actual-model necessary-question development cells passed on the earlier recorded c4e409d819 source. This is supporting evidence for that revision, not current qualification. - Both fresh Stop development cells at 6036cee2f4 stopped, remained idle, resumed once, reached Done, and settled original and successor accounting. Their original machine grades remain failed: the observer counted the generated file-preparation carrier as another final reply. The corrected observer independently correlates the applied preparation receipt with the actual attachment/comment binding. Retained-wire diagnostic replay finds one final delivery for each model without changing either grade. Those original failures remain retained: https://github.com/paperclipai/paperclip/actions/runs/39644220458. - Both fresh actual-model Stop/resume cells pass all five checks at 232d3c52d0: https://github.com/paperclipai/paperclip/actions/runs/39647010372. Independent inspection confirms exactly one cancelled original and one committed successor on the same task, 5.091/5.513-second quiet intervals before the user resumes, retained workspace/instructions/tool identities, and one receipt-bound final response. Each visible JSON chip opens in a tab and its Download control yields verified bytes. The rendered reference and changed label independently reconstruct the registered file hash and the original context hash. Both cleanups pass. These remain local API-key/per-turn development results, not the required staging/Slack qualification. - Other retained Stop failures include missed active boundaries, the real accounting admission failure, and a preparation observer that required nativeIssueId before runtime selection. They remain recorded at https://github.com/paperclipai/paperclip/actions/runs/39636796645 and https://github.com/paperclipai/paperclip/actions/runs/39642060165. The preparation observer now validates all present task IDs and still requires selected native execution at the active checkpoint. - The remote finalization attempt failed during image export with no disk space before a provider started. Remote crash acceptance remains unverified. - Negative calibration rejects wrong or absent ownership/evidence, stale settings, unsafe files, repeated admissions, unacknowledged Stop, premature resumption, incomplete receipts, lost context, genuine duplicate final replies, and missing delivery. A prepared-looking body cannot suppress a comment; absent, rejected, foreign, or mismatched file receipts and attachment bindings also cannot suppress it. - Core staging/Slack drivers, 72 fresh candidate passes, deployed staging proof, and recovery of the original user-visible task remain incomplete. No measured reliability percentage or completed qualification is claimed. ## Risks These are explicit development cells. Their API-key and per-turn runs do not qualify deployed subscription authentication, warm reuse, or Slack. A fast provider can miss the Stop boundary; that attempt fails. The remote crash boundary depends on independently observed public state and an owned controller. The checkpoint is opt-in through a private manifest and cannot replace the missing live observers. The two production changes affect native Stop settlement and saved-input promotion. They use the existing transaction signals and admission path. A complete receipt can settle without another provider turn. Partial accounting and all ownership, cleanup, pause, and budget gates remain authoritative. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose its exact deployment ID or context-window size. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing>	2026-10-21 13:02:27 -05:00
Nicky LeachandPaperclip	00331e0254	Connect native Runner admission to plugin-owned workspaces (#16978) ... ## Thinking Path > - Paperclip manages work for AI agents. > - A plugin can own an execution environment and its files. > - #17029 supplies the task driver and bounded setup verification. > - Heartbeat still prepares workspace files on Core and expects a shell transport. > - This PR gives the selected plugin ownership of workspace preparation and native task execution. > - Core keeps admission, tool authority, and recovery checks. ## Linked Issues or Issue Description **Subsystem affected** Heartbeat workspace admission and native Runner execution. **Problem or motivation** A task-only plugin cannot run through Core filesystem staging or shell execution. Native admission and recovery must use the same lease and workspace ownership. **Proposed solution** Use the task driver from #17029. Resolve workspace and project identities without Core file operations. Connect native dispatch and restart recovery to that driver. **Alternatives considered** Keep workspace admission and dispatch in one change. A workspace change alone would leave dispatch on the previous transport. An execution change alone would leave Core staging files for the wrong host. ## What Changed - Enable plugin environment selection for native workspace sessions. - Resolve workspace metadata and authorized project references without Core checkout, copy, or provision operations. - Preserve agent-default and primary workspace policies. Use the pinned workspace provider for plugin-backed project storage. Reject unsupported Core service commands and cross-owner reuse. - Connect native execution to the prepared task driver. Keep assigned MCP authority on Core. - Save exact checkpoint identity, transfer selected account refreshes, and require termination proof before replacement or cleanup. - Complete managed plugin workspaces through their storage provider. Use the same check after a restart. Keep branch validation and leave task-owned files on the plugin. - Include default instruction content in runtime synchronization when no instruction directory exists. - Defer new-agent Copilot verification until the saved-agent lifecycle can acquire the agent-owned computer. ## Verification - Rebased onto `origin/master` at `898144f4b3`. Master advanced once during verification. Both manual conflicts joined import blocks: one in `native-session-executor.ts`, and one in `heartbeat/runtime-execution.ts`. The seven patches retain their logic. The first rebased head passed repository typecheck, build, all 862 focused execution/finalizer cases, and all CI gates. The latest head `93e5a72892` passes server typecheck and all 880 focused execution, finalization, and execution-control notification tests. All 54 CI/review checks pass, with two expected skips. Greptile ran automatically and scored this head 5/5 with no unresolved findings. No new review or Apex request was made for these import-only resolutions. - The successful CI workflow initially left three stale job-check reports. Targeted policy and summary reruns cleared the reports. No source change or test fix was needed. - Repository typecheck and build pass: `pnpm -r typecheck` and `pnpm build`. - Focused native execution and restart tests pass: 795 cases. The account and harness suites pass: 159 cases. Workspace admission passes: 48 cases. The SDK task lifecycle, workspace API, runtime-hook, and instruction checks also pass. - The heartbeat integration case uses the merged workspace provider contract and reaches native dispatch with no project files on Core. The new-agent case confirms that creation leaves the agent in `preparing` until lifecycle verification. - Map validation and the Copilot source-identity check pass. No Runner implementation or source-identity bump is in this PR. - The first rebased workspace run found two stale path assertions and a recovery guard that did not recognize task-owned targets. These are fixed. The storage integration fixture also needed its persisted source binding; the production authorization check stays in place. - Two prior review findings required rejection of Core service startup and cross-owner workspace reuse. The existing admission fixtures now check both failures. The installed-account authentication finding was fixed in merged #17029. - Tests reuse the current fixtures. No new provider test matrix or live Runner process is added for these fixes. - The prior head `2db273ec12` passed CI on its first attempt and reached standard Greptile 5/5. Apex scored 4/5 and found local-folder checks during plugin completion and recovery. - The completion fix reproduces both failures. It passes all 110 cases in the runtime execution, workspace finalizer, and workspace provider integration suites. Server typecheck and map checks pass. CI passed on `62d8335b4e` on its first attempt, and standard Greptile scored 5/5. The next Apex review scored 4/5. It found that a failed plugin inspection could abort restart recovery. - The restart fix records a failed workspace receipt and uses the current run retry policy. A regression first reproduced the aborted pass. It now checks retry state, retained results, and completion of another workspace in the same pass. All 68 finalizer cases and server typecheck pass. CI passed on `7d4016323b` on its first attempt. Standard Greptile and Apex both scored 5/5. Apex left one P2 about lost failure codes. - The diagnostic fix keeps the known branch and owner mismatch codes in receipts and run recovery state. It gives repair guidance for those codes and keeps unknown plugin errors generic. All 96 affected finalizer cases and server typecheck pass. Before the rebase, required CI passed on `c2fa61c6c8` on its first attempt. Standard Greptile and Apex both scored 5/5 on that head, with no unresolved findings. The new head retains these reviewed patches. - The earlier Superagent scan on `c2fa61c6c8` was inconclusive after a pinned merge-base checkout failed or timed out. The current scan on `93e5a72892` passes. - The exact rebased head passes all 48 workspace cases and server typecheck. The first local test launch could not bind PostgreSQL inside the sandbox. The authorized local-server run passed. - The prior head reached standard Greptile 5/5. Its first CI run hit an unchanged Runner cleanup timeout and an external build-runner shutdown. Both jobs passed on the single retry. The timeout cause is not established. - A local regression check found that a retained Core workspace could also enter a plugin task without file transfer. Core now rejects this move before provider work. The existing ledger test covers both directions and confirms that the source files stay intact. - The first active standard review scored 4/5. It found that a fresh plugin retry could enter Core Git preparation. This mode now uses the existing unsupported-storage error before workspace resolution or acquisition. Both unsupported-mode cases pass, and server typecheck passes. - No live Cloud or provider validation was run. ## Risks - The prerequisite #17029 is merged. This PR targets master. - Native plugin tasks support per-turn workspace sessions. Core service commands and fresh retries that import retained files remain unsupported. A workspace storage plugin can use the modes and strategies it declares. - Core-managed credentials support Codex and direct Claude accounts. Other managed credential routes fail closed. Other harnesses can use an installed account. - A provider must preserve its prepared configuration and report process termination correctly. Uncertain termination keeps the task pending cleanup. - Cloud deployment and live provider validation remain separate. ## Model Used OpenAI Codex, GPT-6. The exact model ID and context window are not exposed in this session. Tool use includes code editing, local tests, and the GitHub CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-21 08:22:48 -07:00
DottaandPaperclip	898144f4b3	fix: stop idle execution-control recovery polling (#17094) ... Replace idle execution-control polling with workers driven by committed writes and persisted deadlines. Preserve startup recovery, bounded retries, and shutdown safety. Stabilize the Slack setup browser regression by waiting for the restored setup screen before simulating its webhook. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-21 09:54:28 -05:00
DottaandPaperclip	30b753f614	perf(budgets): scope accounting and deliver enforcement asynchronously (#17046) ... ## Thinking Path > - Paperclip manages AI agents and their paid work. > - Budgets must prevent new work and stop reported overruns. > - Cost settlement also ran serial lifecycle reconciliation for every eligible company agent. > - That made request latency grow with unrelated idle agents and delayed results on stop delivery. > - This change separates atomic admission and accounting from durable asynchronous enforcement. > - Scoped live usage detects overruns without writing provisional ledger charges. > - The result is constant query scope for ordinary settlement and recoverable stop delivery. ## Linked Issues or Issue Description **What happened?** Settlement and budget checks entered company-wide lifecycle reconciliation. A company with 1,000 unrelated idle agents performed 1,001 accounting transactions for one ordinary cost evaluation. User requests could also wait for provider termination. Running usage checkpoints did not enforce reported monetary exposure. **Expected behavior** Keep the final reservation gate authoritative. Commit charges, scope blocks, and enforcement intent before returning. Deliver stop requests after commit. Read only applicable policies and execution scopes for ordinary charges. Stop long-running work when supported reported or priced usage establishes an overrun. **Steps to reproduce** 1. Create a disposable company with one company budget and idle agents. 2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated idle agents. 3. Compare query counts, company accounting transactions, and duration. 4. Use the controlled partial-usage and delayed-stop fixtures to check enforcement and request responsiveness. **Paperclip version or commit** The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from `master`. Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853, #13075. These cover the affected service charge, cancellation, and accounting paths. This change removes their shared reconciliation overhead. ## What Changed - Batch applicable policy exposure under the existing company accounting lock. Preserve exact fractional money and zero-reservation dispatch fences. - Commit a durable final budget dispatch claim before provider I/O. Retain a separate final issue/run ownership transaction through synchronous handoff, without holding the company accounting lock or awaiting provider work. Preserve native recovery lease checks and duplicate handoff fencing. Browser handoff rechecks current policy; it reuses a consumed rate slot only for the identical policy generation. - Separate ledger spend, live observed spend, and effective held capacity. Use the maximum of reservation and live observation for each unfinished operation. - Deliver budget cancellation through the existing coordinator. Revalidate generations and persist queued/running stop fences before external termination. Bound stop concurrency to four. - Apply agent holds only to the affected agent. Project budgets leave other projects running. Project real company transitions in bounded lifecycle batches. - Replace periodic broad reconciliation with durable pending work, startup repair, receipt replay, and concrete retry/window deadlines. Retain drain and idle-safety checks. - Enforce accepted cumulative usage checkpoints without provisional charges. Final accounting atomically replaces live exposure and settles reservations. - Add operator-gated budget spans, regression coverage, map updates, and a real-PostgreSQL comparison harness. No schema or first-party telemetry change. ## Verification - Full workspace `pnpm -r typecheck` passes on the final ownership fix, rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head `4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks pass. - Focused real-PostgreSQL coverage includes the 292-case rebased accounting/dispatch run, 63 browser cases, 52 run-control/cancellation cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and the exact Muse confirmation regression. Native executor unit coverage passes all 768 cases. Final ownership coverage passes: 74 adapter/use-case cases, nine public regression cases, and the additional deleted-project race. These exercise competing ownership, budget changes and charges between phases, zero/nonzero duplicate claims, rollback, recovery contention, read-only capacity rechecks, and controlled HTTP startup after commit. Another 74 pure/startup/tracing cases pass. The complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its external security scan identified a stale-owner handoff race. The final fix restores synchronous issue/run ownership fencing after the accounting transaction commits. Final head `8e84f55ce8` has fresh Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing security scan. The complete latest-head [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808) passes, including budget, browser, Runner, build, and canary jobs. The unchanged Runner close fixture passed on its sole same-head job retry. - Broad local `pnpm test:run` invocations were stopped before required source updates (rebase and review fixes). They are not claimed as full passes. Final full-suite coverage comes from the complete latest-head CI matrix; local focused coverage and the incomplete broad invocations are reported separately. - Reproduce performance with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-enforcement.ts --baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with `--reverse`. The harness uses disposable real PostgreSQL, 20 warm samples per variant/count, and records raw timings, SQL/lock counts, agent reads, unchanged agent state, and query plans. Final production-source measurement revision: `eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is committed in `4c5f4de8b1`. | Unrelated idle agents | Baseline median / p95 | New median / p95 | Baseline SQL / locks | New SQL / locks | New agent reads | | --- | ---: | ---: | ---: | ---: | ---: | | 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 | | 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 | | 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1 | 0 | - These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64. They isolate the cost-evaluation core and complete admission path, without hosted-region or provider latency. Prior-revision reversed-order controls also preserved the constant query counts; the final measurements above use the final production source. - Complete reserve → final ownership gate → immediate offline provider result is measured separately, with one responsible agent and one active operation at each unrelated-agent count. Admission p95 baseline/new: 0 agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents **24.51 / 74.05 ms**. Baseline admission was already scoped; the stronger final fence adds bounded overhead. New counts stay constant at 53 Drizzle statements, three transactions, two company-accounting locks, and five agent-reading statements bound to the responsible agent. Zero unrelated-agent reads, exactly one provider callback per sample, and no leftover runs or holds. All admission scenarios stay below 250 ms. No external provider calls are included. - The partial-usage/delayed-termination fixture passed 20/20: receipt persistence p95 13 ms, receipt return to observed durable Stop fence p95 28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database request p95 1 ms while termination was blocked. After releasing the controlled termination fixture, acknowledgement p95 was 14 ms. These are separately measured phases; fixture acknowledgement is not live-provider termination latency. - Models and provider deadlines are unchanged. No deployment, live inference, or staging acceptance is claimed. A quiet 30-minute follow-up waits for the merged revision on staging, then repeats classifier and controlled budget-stop acceptance in staging, recording hosted bookkeeping, queue delay, and actual provider acknowledgement separately. - Development failures retained: finalizer notification regressions (fixed; 27/27 affected cases passed), invalid browser admission fixture assumptions (fixed; browser coverage now passes 63/63), and first-head lifecycle/dispatch CI regressions (reproduced and repaired). A prior-head Cases E2E test hit its unchanged announcement-dismiss handler race while opening a filter; the same journey passed on `1ac9b58`. The external security review then found a stale-owner race between accounting commit and provider entry; final verification covers a competing run in that gap while retaining provider I/O outside the accounting lock. Early PostgreSQL startup attempts hit local semaphore pressure; serial reruns succeeded. The final-head Runner lane had one failure among 1,823 tests: the unchanged `timeout-attachment` fixture failed at its first process close after 6.28 seconds. The sole same-head job retry passed all 1,819 active tests (four skipped), including that fixture in 1.85 seconds. Three isolated local probes could not run correctly because the Rust fake-provider fixture binary was absent; cleanup masked the original startup failure. CI builds that fixture through its normal lane. These probes are inconclusive and are not counted as passes. ## Risks - This changes accounting, lifecycle, and cancellation ordering. Durable generations, source/sequence checks, ownership leases, and company locking remain required for correctness. - Provider stop acknowledgement can be slow or fail. Stop intent remains pending until existing termination proof succeeds; cancelled row status alone does not acknowledge delivery. - Mid-run limits depend on provider usage reports or supported price calculations. Providers that report only at completion cannot provide exact mid-call detection. - Signals are process-local wake hints for the supported single-active-writer model. Durable state and startup recovery remain authoritative. - There are no migrations, new queue tables, or cached balances. Repeat the classifier and controlled budget-stop acceptance after normal rollout. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and parallel coding agents. The host does not expose the exact backend model revision or context-window size, so those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-21 07:23:00 -05:00
Devin FoleyandPaperclip	572bef4ba7	fix: omit re-downloadable Codex remote plugins from failover backups (#17026) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex sessions in remote sandboxes copy the sandbox's Codex home into a host-side failover backup after every turn > - Ephemeral sandboxes restore that backup at the start of every turn, so its size affects both disk use and turn latency > - Most of each backup is remote plugin data that Codex itself treats as a cache and re-downloads on startup > - This pull request leaves that data out of Codex checkpoints, keeping everything a resumed session needs > - The benefit is backups roughly 80–90% smaller, less data moved per turn, and much slower disk growth ## Linked Issues or Issue Description Bug report described in-PR (bug report template). Related: #16866 (expires failover backups after their issues close). **What happened?** Failover backups grow quickly because each Codex checkpoint includes `$CODEX_HOME` remote plugin data. On one instance, 204 backups held 22.1 GB: - `codex-home/plugins/cache/openai-curated-remote`: 8.3 GB (about 29 MB per backup); - `codex-home/cache/remote_plugin_catalog`: 6.1 GB (one 38 MB catalog file per backup); - `codex-home/sessions` (the conversation history): 0.31 GB. About 80 new backups (9.8 GB) were written per day. Ephemeral sandboxes also copy all of this into a fresh sandbox before each turn. **Expected behavior** Checkpoints keep what a resumed session needs and omit data that Codex re-downloads itself. **Steps to reproduce** 1. Run a Codex agent with ChatGPT auth in a remote sandbox environment. 2. Inspect `<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current/codex-home`. 3. Most bytes are under `plugins/cache/openai-curated-remote` and `cache/remote_plugin_catalog`. **Paperclip version or commit:** `master` at f89c37791c. **Deployment mode:** server with remote sandbox environments. **Agent adapter(s) involved:** Codex (native app-server and ACPX). ## What Changed - **`server/src/services/native-runtime/native-session-executor.ts`:** Codex checkpoints (direct and under ACPX) now also exclude: - `plugins/cache/<marketplace>` for Codex's six remote marketplaces (`openai-curated-remote`, `created-by-me-remote`, `workspace-directory`, `workspace-shared-with-me`, `workspace-shared-with-me-private`, `workspace-shared-with-me-unlisted`); - `cache/remote_plugin_catalog`; - `plugins/.remote-plugin-install-staging`, and `.remote-plugin-install-staging` (the location older Codex versions use). - **Still kept:** local and bundled plugin caches, `plugins/data`, the Apps tool cache (`cache/codex_apps_tools`, which avoids waiting up to 30 s for the Apps MCP server), session rollouts, and the state databases. - **Tests:** a real-tar checkpoint test covers kept and omitted entries; the profile expectations are updated. - **`doc/recoverable-disk-cleanup.md`:** describes what Codex failover copies omit and the first-turn effect. - No code comments were added. **Why the omitted data is safe to drop** (Codex source at commit d4e03acb): - **Server-side source of truth:** the installed remote plugin list comes from the server (`/backend-api/ps/plugins/installed`) and is held in memory; these directories are not the source of truth. - **Self-healing startup:** at `app-server` startup, a background task reconciles missing bundles and force-refreshes the catalog (`core-plugins/src/manager.rs`, `remote_installed_plugin_sync.rs`). - **No path dependencies:** rollouts and state store plugin ids, not paths into these directories. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts` (in `server/`): 766 passed. - **Mutation check:** dropping one remote marketplace from the exclusion list fails both the new checkpoint test and the profile test. - `pnpm run typecheck` (in `server/`): pass. - **Manual resume with Codex CLI 0.162.1 and ChatGPT auth:** 1. Started a session in a fresh `CODEX_HOME`. This downloaded 28 MB of remote plugins. 2. Copied the home without the excluded entries: 34 MB became 4.9 MB. 3. Ran `codex exec resume --last`. It recalled the conversation in 3 s, and the remote plugins were re-downloaded in the background without errors. - **Compatibility:** restore compares only top-level directory names (`compatibleNativeHarnessBackupManifests`), so existing backups that still contain the plugins restore unchanged. ## Risks - **First turn after a restore:** remote plugin skills and tools can be missing until the background download finishes, typically seconds. Codex leaves an unavailable remote plugin out of a turn; it does not fail the turn. Later turns include it. - **Network and auth:** re-downloading requires ChatGPT auth and network access from the sandbox, which remote plugins require anyway, since Codex fetches the installed list from the server on every start. - **Per-turn download:** ephemeral sandboxes download the remote plugin bundles each turn, about 28 MB in the observed case, in the background. In exchange, about 90 MB less is copied from the host before each turn. - **Rollback:** revert. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, Codex source reading, a local Codex resume test, read-only measurements on a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 16:01:48 -07:00
DottaandPaperclip	505e1a5582	feat(cases): add persistent workflows with same-task agent handoffs (#16959) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - An agent workflow needs persistent facts and a safe way to transfer work. > - Creating a task for each phase splits the objective, discussion, workspace, and review policy. > - A case can retain the facts while one primary task moves between agents. > - Transfers must survive process failure and preserve private access, budgets, and completion gates. > - This pull request adds durable case handoffs, native case tools, and read-only case pages. > - Prompts describe the workflow. Existing task controls decide when work can run and finish. ## Linked Issues or Issue Description **Problem or motivation** Prompt-driven agent teams need a persistent record that can move through triage, implementation, verification, and correction. Ordinary reassignment does not provide a durable instruction, safe release barrier, or recoverable delivery receipt for an agent handing off its own running task. **Proposed solution** Extend standalone cases with one primary task, a descriptive phase, retained access sources, and a handoff ledger. Apply assignments after source execution and workspace settlement. Deliver each transfer through the existing durable wake queue. Provide `create_case`, `read_case`, `update_case`, and `handoff_case` in the native runner. Creation requires kickoff and explicit workflow text. An explicit handoff to `done` closes the case when normal task completion is accepted. **Alternatives considered** Do not add a workflow engine, filesystem queue, configured state graph, or one task per phase. The workflow document combines transition prose with an optional Mermaid `stateDiagram-v2` map. The server does not execute the diagram. The Pipelines experiment removed on master stays removed. **Roadmap alignment** This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run Recovery, and Activity Log attribution. Existing approval, review, budget, privacy, workspace, and responsible-user rules remain authoritative. Related work: [ordinary handoff wake recovery](https://github.com/paperclipai/paperclip/pull/15019). This feature adds a separate persisted case-handoff receipt and release barrier. ## What Changed - Add case routing fields, retained access sources, durable handoff history, delivery leases, and replay-safe migrations. - Keep the same primary task across agent phases. Preserve each returning agent's own session and the task's workspace. - Fence outgoing mutations, wait for provider and workspace release, and recover delivery without duplicate task comments or runs. Recheck current assignment permissions as the original caller before applying a delayed transfer. Scope a run Stop to that run’s own handoff intents, so repeated old Stop requests preserve newer work. Cancelling terminal intent without a source run also requires fenced current task ownership, including native retries and preparation with an old checkout. For Cloud callers, revalidate current stack membership with an uncached Cloud RPC and derive owner elevation from the current managed setting. - Keep case reads, documents, assets, Audit, and live events inside retained task/project permissions. - Add native case creation/read/update/handoff tools. Store required workflow text and initial documents on the first task. Page oversized case facts and referenced document bodies through the dedicated tool with bounded UTF-8 JSON, current routing versions, snapshot conflicts, and current source authorization on every page. - Add read-only case lists and detail pages, task-style list controls, compact dotted field tables, linked handoff tasks, and case filters in Audit. - Add native/legacy integration tests, migration/restart tests, browser coverage, and bounded live receipt fixtures. Extend the trusted sandbox condition to cover the new subscription fixture profile. - Hide obsolete paused-agent warnings on completed or cancelled task threads. - Synchronize the company-switch settings test with React effects after a CI failure. Preserve its security assertions and production remount behavior. ## Verification The branch directly includes master `9ef434fe79`. The required `ci / verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614) passes typecheck, build, tests, and all eight browser shards. Security checks pass. [Greptile gives this exact head 5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549), with no unresolved review threads. The optional Paperclip Review check remains queued because its assigned agent is paused. It is not a required branch check. No check result was overridden. Local workspace typecheck and build pass after the master merge, along with database migration checks, token gates, map checks, Storybook catalog checks, and the Copilot profile identity check. Standard builds refresh Runner vendor output. Focused checks pass 2,008 API/paging/context tests and 7 strict case protocol tests. The final commit changes one test fixture only; its ten tests, UI typecheck, and token gates pass. No production source changed in that commit. On current head `765fccd67c`, all **194 critical workflow tests** executed and passed with no skipped tests in those files: - 38 Stop/ownership tests, including board completion intent, native retries, stale Stop, and preparation with an old checkout. - 2 actual legacy process/HTTP handoff tests, including scratch failure, a closed-stdio child that continues writing, and recovery through a fresh worker after the process group stops. - 5 migration tests, including SQL column metadata, preserved null authority context, and all six prior-numbering receipts. - 3 process-settlement and 5 scratch tests. - 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool integration tests. - 7 bounded case/document-read projection tests and 7 strict protocol-contract tests. The current CI checkout was `fed65d49`, merging `765fccd67c` into master `9ef434fe79`. Its new document regressions reconstruct exact Unicode/escaped bodies and revisions through real native authority and authenticated HTTP with generic API tools disabled. They cover case-owned, primary-task, and linked-task documents; routing on every page; stale document snapshots; access loss and unlinking; unrelated/foreign/private sources; system documents; and quarantined content. Active-task document tools retain their existing behavior. These are current-head test results from the first CI attempt; the browser-only retry did not rerun these suites. The full `64fdd3403f` CI failed one separate settings test: after a company switch, it observed an unsettled API-key draft. The production component and test were unchanged from prior green source. Production switches remount both keyed settings layers; the test directly reused an unkeyed view and used a timer tick instead of waiting for React effects. The final commit replaces that fixture synchronization with React `act`. The exact secret-clearing assertions and all timeouts remain unchanged. The original assertion failure remains recorded; no unchanged retry was used. The narrow unchanged baseline and fixed fixture each pass 10 tests. The upstream result-paging merge preserves case capture namespaces and current asset privacy. Its API integration suite executed 39 of 41 tests successfully, with two skipped; the optional assigned-result PRP integration test was skipped. Neither skipped suite is counted in the 194 critical tests or as live provider proof. All 361 upstream migration entries and SQL bodies remain unchanged. Six case migrations follow master as 0363–0368 with identical SQL hashes. Official Drizzle generators produced the combined metadata and five retained snapshots. Generation reports no schema changes, and the snapshot drift test passes. The earlier `ce361ae5d0` CI passed on attempt 1, including its then-current 179 critical tests. Fresh review later identified unusable document references with generic API tools disabled; the bounded document reads and ten additional authenticated HTTP tests address that finding. The full local `pnpm test:run` attempt was cancelled after this Mac exhausted PostgreSQL semaphore capacity; it is **not** counted as a pass. Later database attempts also stopped before assertions or skipped, including one sandbox socket denial. Two original stale-Stop regressions did reproduce the bug against the old source. A REST serialization assertion passed after two preserved database-startup failures. No test timeout was increased. All 194 assertions listed above executed and passed in current-head CI. The complete 385-file, 3,963-test UI shard also passed after the fixture repair. A configured-provider receipt workflow passed 13 backend checks across triage → implementation → verification → correction → done, using one task and four work runs. The saved receipt is revision 2 with total 6 USD; case and task both closed. The original canonical attempt remains failed because its browser assertion confused the terminal destination with completed-task tooltips. The corrected full browser verifier passed on the same retained database and original runs, with no additional provider calls. The last clean-build browser readback is on `6a86c54a53`; later changes affect recovery, authorization, bounded reads, and settlement. Accounted work cost is a $1.010574 rate-card estimate; provider billing and setup-probe usage are unavailable. Copilot profile 49 passed the unchanged seven-fixture private protocol pack (46/46 checks, no retries) and the product file-edit canary after master's completion-replay change. All 60 bound implementation hashes remain unchanged on the current source. Historical profile-47/48 identities and results remain intact. See the [profile-49 compatibility record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json). This is bounded compatibility evidence, not a fresh full matrix, installed-package product test, or Daytona rollout. Preserved CI failures and retries: - On `20662930ac`, Slack managed setup failed a five-second success assertion after reload. Its test/application path was unchanged from a prior green head. One unchanged-source retry passed all 25 tests in that shard; no source or timeout changed. - On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19 UTC. Neither log had an assertion failure before interruption. One unchanged-source retry of each passed all Runner static checks and all 32 tests in browser shard 6; both aggregates passed. The next head, `7b4374edaf`, passed all CI without retries. - On `295f289b5a`, serialized shard 9 timed out during its first device-login test, then the next test found an active login lease. Direct login/test paths were unchanged from a prior green head. Cold import plus a timed-out async body using the next test's mutable actor/store is the supported fixture-race explanation; no request trace proves that sequence. The original failure remains recorded. One unchanged-source retry passed all 294 tests in 19 files, including all 41 device-login tests. No login source or timeout changed. The subsequent `ce361ae5d0` CI passed without retries. - On `64fdd3403f`, one company-switch settings assertion observed an unsettled draft. The final test-only commit uses React `act` without changing assertions, timeouts, or production code. On `765fccd67c`, all ten settings tests and the complete 385-file, 3,963-test UI shard passed. - On `765fccd67c`, browser shard 3 completed the split-handoff assertions, then cleanup received HTTP 500 from run cancellation after a one-second PostgreSQL row-lock timeout. The extra chat run was reporting completed work while teardown began. Cleanup stopped before restoring a global experiment flag, causing a later assertion to fail. Relevant production and test paths are unchanged from an earlier green commit. Stop's lock order is consistent, and no nested self-lock was found; the exact PostgreSQL blocker was not recorded. One unchanged-shard retry passed. The original failure, trace, and downstream flag failure remain recorded. No Stop implementation, cleanup assertion, or timeout changed. The retry supports intermittent cleanup contention; it does not identify the original blocking transaction. ## Risks - Local process transfers require host-owned PID/group absence. Missing or uncertain termination evidence keeps delivery pending, independently of scratch setup. - Handoff admission and task cancellation share execution ownership. New regression tests cover temporary holds, late source writes, and uncertain provider termination. - Retained private sources use an intersection of existing permissions. This intentionally fails closed when a source is missing or an existing task cannot preserve that boundary. - Migrations add tables and indexes. They preserve master history and can replay over prior feature installs. Migration 0368 adds nullable internal authorization provenance with no default or authority backfill. Historical transfers still awaiting release without this context fail closed; an operator must cancel them and request a new handoff. - Delayed Cloud handoffs depend on the current Cloud membership RPC. An unavailable, ambiguous, expired, or mismatched response holds the handoff for retry; it does not reuse cached authority or grant access. - Bounded reads reload and reauthorize the selected projection per page. Underlying work grows with the data size and number of pages. Changed data or routing requires restarting from offset zero without the old snapshot. - Phase names and workflow diagrams remain advisory. They do not satisfy review or approval gates. - This is a large, cross-cutting PR. The maintainer requested one PR for the whole feature; the commits separate data, runtime, UI, and qualification work. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated code review. The exact deployment model ID and context-window size were not exposed in the session. Live fixture agents used the explicitly configured provider model recorded in their test evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 12:46:14 -05:00
Devin FoleyandPaperclip	1f3695ac67	fix: expire native session failover backups after their issues close (#16866) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent sessions that run in a remote sandbox keep a host-side failover copy of the sandbox's harness state, so a replacement sandbox can resume the conversation > - Each session keeps that copy forever, and one copy can reach several gigabytes > - On a busy instance these copies fill the volume, and no existing cleanup touches them > - Once a session's work is closed, its copy matters only if someone reopens that work later > - This pull request expires the copy after the session's issues have been closed for a configurable window, and makes a reopened issue start a fresh session > - The benefit is bounded disk use without breaking active, recoverable, or leased sessions ## Linked Issues or Issue Description Bug report described in-PR (bug report template). **What happened?** After every remote turn, `checkpointRemoteRunner` writes the sandbox's runner state and provider session home (for example `codex-home`) to `durable-sessions/<scope>/failover-backups/current`. Only the same session's next checkpoint replaces it. No code path deletes a session's copy when the session ends, its issue closes, or its agent goes away, so the instance volume fills. On one instance: - 246 sessions held 17 GB of copies, 84 of them older than a week; - 24 new sessions added 5 GB in a single day, one copy alone 3.2 GB. **Expected behavior** Disk used by failover copies stays bounded. A copy is kept while its session can still resume (open issue, live run, pending recovery, or live lease), and expires once the session's issues have been closed for a while. A reopened issue then starts a fresh native session, instead of failing with `runner_harness_backup_unavailable` on a replacement sandbox. **Steps to reproduce** 1. Run agents with a native runtime in a remote sandbox environment. 2. Let them complete many issues. 3. Watch `<instance>/runtime/paperclip-runner/durable-sessions/*/failover-backups/current` grow without bound after the issues close. **Paperclip version or commit:** `master` at 9179d8dcc3. **Deployment mode:** server with remote sandbox environments. **Agent adapter(s) involved:** not adapter-specific (core bug); observed with Codex sessions. **Breaking changes** - Reopening an issue that has been closed for longer than the window (default 7 days) starts a fresh agent session instead of resuming the old conversation. - Disable with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or lengthen the window with `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS`. ## What Changed - **`server/src/services/native-runtime/native-harness-backup-expiration.ts` (new):** the sweep. It checks at most 200 sessions and expires at most 50 per batch, continuing from a cursor. A copy expires only when all of these hold: - every issue the session ran on is `done` or `cancelled`, closed before the window; - the copy was written before the window; - no run of the session is queued, running, or unfinished; - every native run finalization of the session is settled (`committed` or `terminal_failure`), with no recovery state or lease owner; - no `active`, `released`, `retained`, or `pending_cleanup` environment lease references the copy (by backup stamp, session run, or recorded lease); - the session is not executing, initializing, warm, or closing in this process, and its directory has no pending cleanup activation. - **One transaction:** the database checks share-lock the issues and delete the agent task session that points at the session, so a reopened issue cannot resume the expired session. - **Atomic, non-blocking removal:** the copy is renamed to `failover-backups/.retiring-<UUID>`, and a final check looks for a run of the session. If one appeared, or the check throws, the rename is undone. Only a confirmed copy becomes `.expired-<UUID>` and is deleted with asynchronous file operations, so a multi-gigabyte removal does not block the event loop. A later sweep restores an unconfirmed `.retiring-*` copy when `current` is missing, drops it when a newer `current` exists, and finishes an interrupted deletion. The rest of the session directory is kept. - **`native-session-executor.ts`:** exports `nativeSessionScopeDigestIsBusy`, which checks the in-process executing, initializing, warm, and closing session maps. - **`native-harness-backup-stamp.ts`:** exports the state base path the sweep scans. - **`config.ts` and `index.ts`:** three environment controls (`_ENABLED`, `_DRY_RUN`, `_DAYS`) and the scheduled sweep. It starts two minutes after boot, honors shutdown, standby, and idle drain, and logs candidates, reclaimed bytes, cleared task sessions, and skip reasons. - **Docs:** `doc/recoverable-disk-cleanup.md` describes the sweep; `doc/DEVELOPING.md` lists the variables. - No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/native-harness-backup-expiration.test.ts src/__tests__/native-harness-backup-expiration-config.test.ts` (in `server/`): 30 passed, 24 of them against embedded Postgres plus 6 config tests. Cases: - expiry for done and cancelled issues, including clearing the task session; - each keep condition: open issue, recently closed, live run, pending finalization, recovery in progress, live lease, recent copy, cleanup activation, in-process session; - settled finalizations and expired leases do not block; - dry run changes nothing; - bounded batches continue across ticks; - an interrupted deletion is finished, and an unconfirmed retiring copy is restored or dropped; - the configured clock controls the closed window. - **Mutation checks:** treating `in_progress` as closed, ignoring live leases, ignoring the configured clock, or not restoring a retiring copy each fail the corresponding tests. - `pnpm exec vitest run native-harness-backup config` (in `server/`): 19 files, 293 tests passed. - `pnpm run typecheck` (in `server/`): pass. - **Real server:** after building and bundling, `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` reached healthy. - **Manual equivalent on a live instance:** applying these checks by hand (with a 3-day window) to 242 copies freed 7.1 GB. Most of the remainder belonged to open issues or live leases. ## Risks - **Reopen after the window:** the agent starts a fresh session and loses the old conversation's provider history. Issue comments, documents, and work products are unaffected. - **Race with a wake:** the checks share-lock the issues, so a reopen waits for the transaction. A run created for the session between commit and rename is caught by the post-rename check, which restores the copy. Such a run would find no task session and start fresh. - **Missing owner run:** if the owning run row is gone (for example, after an agent was deleted), the copy is kept, because the sweep cannot prove the session is closed. - **Untouched:** quarantined session roots and `continuity-breaks` archives are not touched by this change. - **Rollback:** set `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false`, or revert. Expired copies cannot be restored. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, local builds, tests and boots, read-only measurements on a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 07:57:43 -07:00
DottaandPaperclip	92b238137b	Connect and configure GitHub Copilot through Apps (#15705) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps and AI Connections store credentials and govern access to agent runtimes. > - Users need a supported saved-token connection and config values that survive save and reopen. > - The release must preserve saved-token access and task lifecycle guarantees. > - This pull request will connect Copilot through the normal Apps and AI Connections path. > - The benefit is a reviewable part of the Copilot production release. ## Linked Issues or Issue Description Refs #15701 and #15703. Supersedes setup work in #15562. Base: #15703. Next: #15706. ## What Changed - Require configuration update authority before Copilot credential-bearing model discovery; default to the saved environment and preserve explicit selection for editors. - Keep Copilot discovery passive during reconnect reconciliation; refresh on explicit opening, account/environment changes and reconnect. - Add GitHub AI token connections with encrypted storage, ownership, grants and audit logging. - Keep repository access separate from Copilot runtime authentication. - Probe the selected environment without sending a model prompt and preserve authenticated model selection through save/reopen. - Keep fresh execution admission consistent with normal Copilot setup; persist task model choices without changing provider or permission policy. - Refresh authenticated models after reconnecting the same saved Copilot account, including recovery from a previous authentication error. - Apply the additive provider-default constraint migration through the normal generated migration workflow. ## Verification Runtime #15701 is merged at `d855e550ad8d2a267312983f26457aabf5393c3a`. History #15703 is merged at `df80616d98119cbf5abd40549794d9980a5fabf6`. Remaining source [`42bcbd61299655f7e37aabe6bb5f19bba770ad1d`](https://github.com/paperclipai/paperclip/commit/42bcbd61299655f7e37aabe6bb5f19bba770ad1d) contains the four-file authorization repair. The remaining PR trees preserve six nonoverlapping files from the newer public mainline managed-settings fix and the CI-owned lockfile. No unrelated provider is enabled. The security finding is repaired: existing-agent discovery requires `agent_config:update` before connection selection or token resolution. The saved environment is the default. Authorized editors can select a prospective environment. Both denied board-viewer and agent-reader regressions fail on the original source. All49 route tests and server typecheck pass with the fix. Denied requests perform no connection selection, credential resolution or probe. Current-head security scans pass on both remaining PRs; the repaired thread is resolved. Copilot CLI1.0.88 and profile47 retain digest `sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`. The [qualification record](https://github.com/paperclipai/paperclip/blob/42bcbd61299655f7e37aabe6bb5f19bba770ad1d/doc/plans/2026-10-20-copilot-qualification-v47.json) reports20/20 Product cells,154 assertions and7/7 protocol cases,46 assertions against frozen normal installed source `c2616370800f5f6df45246b0890ba35a7cd2faab`. The exact qualification model is `gpt-5.6-luna`. Every passing Product case confirms cleanup. Original failures remain recorded; automatic behavior retries are disabled. The route-only authorization repair changes no bound runtime/profile bytes and does not relabel the frozen campaign. The ordinary installed remote journey passed saved-token selection, model discovery, normal UI save/reopen, metadata-only Test, completed task, accepted receipt and exact19-byte browser download. An open model catalog caused no additional discovery requests during117seconds. Actual installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token browser entry was not exercised; encrypted connection creation used the normal API. The first Daytona file attempt timed out during delayed semantic dispatch; its precise stalled layer remains unresolved. Independent ordinary remote delivery and the other19 Product cells passed before one instrumented recheck passed8/8 with unchanged source,prompt,assertions and deadlines. No causal latency repair is claimed. Before the authorization repair, all four final PR heads passed required verify/E2E and fresh Greptile5/5 reviews with zero unresolved threads. Current-head CI/review is running for the two repaired heads. Local focused checks retain129 connection,59 runner,72 shared-eval and88 recovery/workspace passes, plus migration safety and token gates. The additive Copilot migration is361; released master history is preserved. [Cloud normal-package run39414239704](https://github.com/paperclipai/paperclip/actions/runs/39414239704) builds the pre-security source4f3f. All four native producers and Linux package assembly pass; the final clean installed consumers are finishing. All18 retained archives match their hashes;37,863 regular files contain zero selected credentials. That package does not contain the later route repair. The actual final published package must include it and pass the local/Daytona live canaries. The frozen Daytona image remains published and independently verified. No separate full-image scan is claimed. Docker builds run in the cloud. **Remaining:** current-head CI/review, setup and qualification merges, final publication, and ordinary local/Daytona canaries against the actual shipped package. The user explicitly authorizes the maintainer review override; CI remains a merge gate. No production deployment is claimed. Provider USD remains unknown and GitHub-attributed. ## Risks Bad credentials, missing entitlement, runtime installation and unavailable models must remain distinct failures. A connection must not adopt ambient credentials. The constraint migration changes accepted provider values, with no credential-storage migration. Current-master NewAgent fixture extraction is retained. Copilot cases use that shared support and test both passive discovery and explicit reconnect under the same setup lifecycle. The onboarding retry regression shares the component suite’s saved-connection, authenticated-model, draft, agent-list, hire and render lifecycle fixtures. Keeping it beside the existing onboarding failures makes adoption versus duplicate hire observable under the same lifecycle; this is the concrete cohesion reason for the existing large test file. ## Model Used OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this task. Used for code changes, review and tool-assisted verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 07:02:22 -05:00
3 people	b448710986	fix: derive startup warmth from per-layer reuse evidence (#16908) ... ## Thinking Path > - Paperclip helps people manage AI agents and inspect their work. > - Agent startup can reuse a lease, workspace, Runner, process, or session. > - A session ID can remain the same when other resources are new. > - The old cold/warm label did not show these separate decisions. > - This pull request records closed reuse receipts and derives labels from them. > - Reports now show partial reuse and missing evidence. ## Linked Issues or Issue Description Refs #11685. That issue tracks a specific ACPX fingerprint repair. This change records reuse evidence across startup resources. It does not repeat that repair. **What existing behavior does this improve?** Startup labels in the operator trace and Product E2E reports. **Current behavior** A cached runtime or a continuous session can suggest a warm startup even when workspace staging or provider launch work occurs. **Proposed behavior** Record each layer's reuse decision and a closed reason. Record observed provider launch counts. Use `warm`, `cold`, `partial_reuse`, or `unknown` from those receipts. Keep old runs unknown when they have no receipts. ## What Changed - Add local run-log receipts for the lease, workspace, Runner, provider process, and provider session. - Count physical ACPX launches, including respawns. Preserve the reason for a new process. - Reuse native workspace adoption and session lifecycle evidence. Keep native and bridged remote provider-process reuse and launch counts unknown where physical evidence is absent. - Show the per-run summary in retained results and the dashboard. Keep missing legacy values explicit. - Set the legacy trace boolean only for fully evidenced cold or warm startup. Keep the operator OTLP gate. Add no first-party Telemetry. - Test full reuse, workspace invalidation, two direct provider launches with session continuity, and report generation. - Start the final diagnostic write without waiting for persistence. Keep lease cleanup independent of a stalled event sink. - Keep remote provider evidence unknown after runtime setup fails or the host relay starts twice. - Align a stale Pi support-test expectation with current master profile 28 and its suite fingerprint. Keep the qualification rules unchanged. ## Verification - Passed: all 319 runtime, startup-timing, reuse-summary, and native workspace-history tests. Disposable local Git tests used the native Git binary. - Passed on the current commit: all 2,768 Runner E2E support tests across 128 files. This includes report and history tests for generated HTML and normalized JSON. - Passed: `pnpm --filter @paperclipai/adapter-utils typecheck`. - Passed on `df60f8dbdbfcb6d80c84cd3bf563f8d58b2f9f03`: all PR CI gates, including repository typecheck and build, all test shards, and browser tests. Both automated reviews are 5/5. GitHub reports 55 successful checks, one neutral review, and two skipped Storybook checks. - The first server shard run failed because an unchanged workspace-runtime fixture did not accept health connections within 10 seconds. Its isolated local test and the single failed-jobs rerun both pass. This establishes recovery; the cause of the first timeout remains unknown. - Passed in the hosted source run on this exact commit: recursive typecheck, token gates, the large Runner E2E typecheck, Runner E2E unit tests, and build. The retained `source.txt` matches this PR head. - The extra unsharded `pnpm test:run` reached its 60-minute limit (exit 124). It has no complete verdict. The broad workflow therefore reports failure. No assertion-failure marker appears in the retained partial log. Full-suite acceptance uses the passing canonical PR CI partitions. A read-only inventory check found no gaps across 1,023 general server files, 161 serialized suites, and 20 workspace projects. The separate chat/native Runner lanes cover the excluded server files. No tests or timeouts were weakened. The local environment has 4 GiB of memory. Use CI for the large Runner E2E typecheck. - The initial report tests failed because the harness could not resolve package imports from the repository root. Relative source imports fixed that failure. - The first broad local run had timeouts from the environment Git bridge. Native Git resolves the ancestry timeout and all runtime tests pass. The added report fixture lacked its required screenshot; the corrected report tests pass. A stale master assertion expected Pi profile 27, while the generated profile is 28. Updated its expectation and exact suite fingerprint; all 78 targeted support tests pass. The clean full support run passes all 2,768 tests. - The three review regression cases failed before repair. They showed remote counts of zero and two, and a completed run that waited for the final diagnostic sink. They now pass. The stalled-sink case keeps persistence pending while the run returns, both bridge stops run once, and the staging lock map becomes empty. - Hosted evidence for this commit: [source verification](https://github.com/paperclipai/paperclip/actions/runs/39335475416), [retained command logs and status table](https://github.com/paperclipai/paperclip/actions/runs/39335475416/artifacts/12228829282). All listed commands exited zero except the incomplete unsharded test command described above. - The physical-launch test drives the executor's spawn callback twice. The native workspace test uses real Git repositories and verifies reuse and invalidation. These are deterministic tests, not a paid provider qualification campaign. ## Risks - Consumers must handle `partial_reuse` and `unknown`. Partial evidence no longer produces a legacy cold/warm boolean. - Native and remote bridged provider process evidence is still unavailable on some paths. The report shows unknown values for those paths. - Receipt emission is best effort. Missing receipts reduce confidence in a label and must not change execution or settlement. - No schema migration, first-party Telemetry event, or endpoint default changes are included. ## Model Used OpenAI Codex, based on GPT-6, used reasoning, repository tools, and code execution. The exact provider model ID and context window are not exposed to this session. The existing implementation checkpoint was preserved from the earlier agent run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>	2026-10-19 21:52:57 -05:00
DottaandPaperclip	ac9459bae3	feat: configure company agent observability (#16874) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Companies use several agent harnesses and execution environments. > - Operators need one place to select where native agent logs and traces go. > - An inference proxy would add a dependency to the agent request path. > - This pull request supplies native OpenTelemetry settings to managed Codex and Claude Code processes. > - Company secrets protect destination credentials. Export failure does not change the task result. ## Linked Issues or Issue Description **Subsystem affected** Company settings, shared contracts, database, server, adapters, and runner. **Problem or motivation** An operator cannot configure a company destination for native agent logs and traces. Manual harness configuration is hard to apply across local and remote execution. **Proposed solution** Add one company destination: Braintrust or a custom OTLP HTTP/protobuf endpoint. Apply a transient snapshot when execution preparation starts. Keep credentials out of API reads, task environment manifests, recovery files, and command arguments. Replace a warm process when its export settings change and preserve its conversation. **Alternatives considered** Multiple destinations can use a user-operated collector for fan-out. This change does not add a collector, an inference proxy, or transcript scraping. **Roadmap alignment** This extends company settings, secret management, and support for multiple harnesses. The maintainer requested this implementation. Searches for observability and Braintrust found no duplicate company destination feature. Related infrastructure tracing work in #16236 has a separate content-free contract. ## What Changed - Add strict schemas, revision checks, encrypted secret storage, activity records, and board-only settings routes. - Add explicit synthetic logs and traces tests with a bounded HTTP request and sanitized results. - Supply native Codex configuration and Claude Code environment values through local and remote launch paths. - Preserve Codex login refresh through the existing managed credential link. Retain unsaved drafts across background reads. Require complete custom header re-entry when the endpoint origin changes. - Separate process export compatibility from conversation identity. Isolate generated Codex configuration and preserve resumable sessions, including remote home refresh after export changes. - Advance the immutable Pi/Copilot source attestations for shared environment and sidecar code changes. Keep their provider payloads unchanged and retain prior identities for decoding. - Redact destination credentials throughout launch diagnostics and HTTP logs. - Add the production settings form, Storybook states and journey, documentation, and feature/access map entries. ## Verification - Real managed local Codex 0.160.0 and Claude Code 2.1.286 tasks exported to Braintrust. API readback and the Braintrust UI showed the test records. A real remote Daytona Codex task exported through the legacy `exec` path. - Live tests exercised content capture off, export disabled, warm conversation reuse, and a destination that accepted connections but never replied. Agent work succeeded and export requests stopped at the configured three-second timeout. The UI connection test returned “Test payload accepted”; the saved API key stayed masked after reload. - All current-head CI test lanes, typechecks, packaging/build checks, and required aggregates passed on `71e994b2be9c8fb6d6e938a4ad7cf4c95fdafe31` ([CI run](https://github.com/paperclipai/paperclip/actions/runs/39319194318)). Greptile completed on that exact commit at 5/5, with zero unresolved review threads. The additional optional Paperclip Review check remains queued waiting for its assigned agent. - Local workspace typecheck and build passed. Focused verification included 239 combined runner tests, 789 API/native-session tests, 107 Codex home/ACP/sandbox-site tests, 204 feature/privacy tests, and the affected Rust suites. Migration, token, map, Storybook catalog, and Storybook build checks passed during acceptance. These checks ran at the relevant implementation or merge boundary; the current-head CI is the complete final gate. - Managed AI credential projections now receive enabled and disabled company export settings; genuine caller-owned Codex homes remain untouched. Two regressions reproduced the skip before the repair. All 276 engine/lifecycle tests and adapter-utils typecheck pass afterward. The launched TOML retains provider/MCP settings, credential continuity, and redaction. - Settings saves, removal, and the real company-deletion owner acquire secret mutation ownership before database locks. PostgreSQL NOWAIT probes cover all three paths; concurrent save/rotation/removal/deletion requests settle correctly. The affected suites passed 137 tests, followed by all 32 service/connection cases after expanded deletion coverage. Server and Codex adapter typechecks pass. - Codex source profiles use the effective CLI argument list. A spawned fixture CLI reproduced both ignored-argument failures; all three precedence/fallback cases now pass. - Real tar staging reproduced remote conversation loss in both API-key and subscription modes before the repair. Both now pass: changing managed settings preserves sessions, archives, native indexes, and refreshed subscription credentials. A corrupt upload preserves the prior config and conversation. - Preserved failure evidence: initial CI caught stale Pi/Copilot source attestations, a missing test-harness helper, and packaging fixture/probe mismatches; those were corrected and covered by focused checks. Superseded heads had intermittent attachment-composer timeouts despite successful uploads; [the original trace and screenshot remain available](https://github.com/paperclipai/paperclip/actions/runs/39309701089/artifacts/12215109203). The same attachment case passed on the final head: [25/25 tests in shard 3](https://github.com/paperclipai/paperclip/actions/runs/39319194318/job/118288637051). - A connection-fault fixture also exposed a stale pooled socket after its hang case; closing fixture responses isolated scenarios and the full affected suite passed. - The broad local test run was not clean: database resource failures, timeouts, an older cached module graph, and a stale optional-resource manifest assumption were recorded. Focused checks passed after the documented fixes. This does not establish a clean full local pass. - Native remote Codex, remote Claude, and the full ACPX Codex bridge remain unqualified. The direct Codex resume test does not establish full bridge acceptance. ## Risks - Native exporters perform network requests from each execution environment. A server connection test does not prove remote reachability. - Prompts, assistant responses, and tool content are enabled by default when supported. Native caps and harness-specific controls still apply. - Caller-owned Codex homes are unsupported for automatic configuration and remain untouched. - Native harnesses and their tool processes can inspect export credentials. The settings form and documentation advise using ingestion-scoped credentials. Retained headers remain bound to their saved scheme, host, and port; changing origin requires fresh values or removal. - Migration 0357 adds a company-scoped row and secret reference. It is safe to reapply. Secret removal and company deletion use the existing secret lifecycle. - Slow or unavailable destinations must not block task settlement. Export settings take effect at the next execution preparation boundary. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and browser verification. The exact serving model ID and context window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 15:36:04 -05:00
DottaandPaperclip	8193abb936	feat(claude): default qualified macOS and Linux targets to Paperclip Runner (#15997) ... ## Thinking Path > - Paperclip must run the selected agent on its selected host. > - Claude needs verified packages for each supported target. > - A Mac controller must select the correct Linux provider pack. > - New defaults must follow package and execution evidence. > - This PR adds Claude platform packaging and target-specific pack selection. > - Existing Codex Linux ARM64 behavior remains intact. ## Linked Issues or Issue Description Refs #15683. This PR depends on #15993 and #15994. Related scope: #15422 and #15224. ## What Changed - Reuse the merged four-target daemon producer/consumer matrix. Add target-owned Linux provider packs and carry their immutable identities through release assembly and vendoring. - Reuse master’s landed pinned Claude SDK ARM64 executable admission and retain package-integrity, architecture and no-follow admission. - Select target-specific remote packs so Mac controllers can use Linux hosts without a development pack override. - Preserve read-only runtime context on SSH startup and resume without accepting unverified old content. - Check ordinary installed Claude readiness without injected package authority. Retain Codex and Linux x64 consumer checks. - Extend Claude automatic target selection for the remaining qualified package targets. Retain unsupported-platform and external-override behavior. - Qualify managed Daytona snapshot names through provider IDs during Claude Connect. Pin native acquisition to the measured ID. Preserve a revocation marker after failed probes or target edits, and block native acquisition until requalification instead of falling back to a mutable name. Preserve immutable-image proof, company and account boundaries, and Legacy acquisition. ## Verification - Reconciled with master `9179d8dcc3e908c217fde4f63c9d08ccf8132eeb` through the prerequisite stack, retaining the merged startup fix and latest Codex Linux ARM64 behavior. - [Four-target exact-source package qualification](https://github.com/paperclipai/paperclip/actions/runs/39267321383) passed on source `04e0202ca81387b17cbe7fd094a8338648356fe9`: all four native builds, public-package assembly, Linux x64 verification and clean installed consumers on Apple Silicon, Intel Mac and Linux ARM64. The later master reconciliation adds shared OpenRouter attribution and changes runtime source. This historical result keeps its original source identity; [later qualification](https://github.com/paperclipai/paperclip/actions/runs/39281100929) passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. It predates the subsequent TypeScript startup fix. - Fresh local product acceptance on integrated source `b2c9fe43f9cd3bc5dd9d22552da0148845ba71a4` completed a useful file and follow-up through native Claude/Opus 5.5, High effort and the API-key connection. Both actual browser downloads were independently checked; the revision preserves the original body and session continuity. Total task estimates: $0.518432800. Ordinary web and API creation saved defaults with the native experiment disabled. - Local focused verification passed 993 startup/session tests, 243 UI tests, 12 workflow security tests, direct server/UI TypeScript checks, and token/palette gates before the last review changes. Initial DB-backed snapshot pin/revocation coverage passed 18 cases; the final serial route suites pass 122 credential/permission and 232 snapshot/environment cases after correcting a test hook timeout. Current-head cloud suites pass. - Preserved the failed pre-provider local fixture and its full pending reservation. The successful fixture has no unpriced or pending usage. Historical failed Cloud journeys, accounting holds, source identities and subscription limitations remain in the dated plan. - All Docker/Rust release production was performed in cloud CI. No local Docker image was built. An inadvertently invoked local Rust prerequisite was stopped. - Current-head cloud CI is green. All three stack PRs have fresh Greptile 5/5 reviews and zero unresolved review threads. Required typecheck, test and build gates ran in cloud CI; Docker work stayed remote. Earlier failed checks remain in their original runs. - The final master merge preserves task-loading, permission, skill-retention and OpenRouter changes. Pi uses a fresh combined profile 27, preserving both historical profile-24 declarations; Copilot remains profile 22. Focused protocol checks passed 155 cases (one skipped), the updated Pi attestation check passed 18 (one skipped), and 91 evaluation/catalog checks passed. - Latest-application manual continuation passed in 37.942 seconds on `695e5c005a85bddc8711d3fcddcaf6b7e1fb64d0`, with High effort, native Opus 5.5, Anthropic API attribution, the same resumed Claude session, an independently checked 635-byte file and persisted Done status. It uses the earlier verified Mac daemon; the preceding-source package qualification passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. That package run predates the small TypeScript startup fix and is not relabeled as qualification for it. - Upgrade limitation: changed compiled sidecar identity rejected old durable launch state and recovery hit session-open bounds before eventually resuming. That attempt was operator-cancelled and retained; the successful manual continuation does not qualify seamless upgrades. - Local latency profile, Opus 5.5/High unchanged: three alternating real-ACP loopback samples per variant reduce median provider-ready time from 5.081s to 4.034s. The matched real API task startup improves from 5.533s to 4.348s, but total reply time changes from 17.933s to 20.221s because agent work varies. This establishes a startup saving of about one second, not a consistent total-reply speedup. A real candidate follow-up passed in 20.146s and resumed the same saved session. The loopback fixture makes no external model requests; failed initial probes remain retained. - The startup change passes 107 focused runtime/effort/lease tests, 53 profile checks (one skipped), TypeScript checking and the TypeScript-only runtime build. Native hire fixture checks pass 70 tests; their first local run retained a socket-hangup failure and stale OpenCode expectation, both accounted for before the passing rerun. Broader required checks run in cloud CI. ## Risks - **Experience needs improvement:** successful connection setup still shows “Connection needs attention” because runtime checks are deferred. Cold provider-pack upload took 122–126 seconds per run while the task mainly showed “Thinking.” The tested Core-only preview excludes private Cloud UI plugins; this is not full Cloud distribution or subscription qualification. Desktop dark theme was tested; mobile and comprehensive accessibility were not. The portable report validates, but gallery interaction could not be inspected because host preview was queued. - The optional supply-chain scan returned neutral/inconclusive because GitHub Actions package coverage and infrastructure parsing are unavailable. It is not counted as a successful supply-chain qualification. Exact package, daemon and image integrity checks passed. - A stale, missing or wrong-architecture pack must fail admission. It must not change runner or target silently. - Provider-pack source changes invalidate affected earlier execution results. Existing unknown-cost and cleanup holds remain intact. - Live subscription execution remains unverified under the API-key qualification direction. Built-in authentication flows are retained. ARM64 cloud-image production is deferred. Merge and production deployment remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] Focused local tests and final cloud verification pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 14:17:24 -05:00
DottaandPaperclip	45b67f1d93	feat: default new Apple Silicon Claude agents to Paperclip Runner (#15993) ... ## Thinking Path > - Paperclip lets users choose an agent harness and account. > - The server resolves the runner for new agents. > - Claude still needs the same ordinary native-default journey as Codex. > - Existing agents must retain their runner, model and account. > - This PR defaults new Apple Silicon Claude agents to Paperclip Runner with Opus 5.5. > - Users can connect and work without a runner override. ## Linked Issues or Issue Description Refs #15683. This PR depends on #15994 and precedes #15997. Related scope: #15422, #13978 and #15224. ## What Changed - Resolve new Apple Silicon Claude agents to `paperclip_runner`, `provider: acpx`, `acpxAgent: claude`. Keep explicit Legacy, models, provider routing and saved execution. - Use `claude-opus-5-5` for ordinary Claude defaults and retain explicit overrides. - Reuse existing API-key/subscription connections, credential storage and checks. Generalize the shared Advanced runner control with Claude branding. - Carry configured effort into the shared v9 contract. Preserve compatible permissions, instructions, skills, timeouts and session behavior. Require Legacy for unsupported settings. - Qualify a new immutable managed Claude target at explicit Connect through the existing permission-gated probe. Use fresh scoped discovery for test and hire. Discard stale results and require an explicit Connect after settings change. - Retain merged Codex onboarding recovery, auth preparation, import/effort corrections and Grok Bot gates. - Check the Claude first-task wizard's saved default directly. Update configuration/runtime documentation and exact Opus 5.5 accounting. - Align hire fixtures with the merged host-environment restriction. Native agent-authenticated hires cannot inject host env settings; both creation endpoints verify that no extra agent is saved. ## Verification - Updated onto master `9179d8dcc3e908c217fde4f63c9d08ccf8132eeb` through #15994. Native input v9 carries effort while preserving v7 home modes and v8 Muse. Codex defaults remain unchanged. - Focused local verification passed 285 server tests, 323 UI tests, 124 setup/registry tests, and two v9 permission-pin transport cases. Direct server/UI TypeScript checks passed. An initial 30-second hook timeout was resolved by serial verification with a longer hook budget: all 122 credential/permission cases and 232 snapshot/environment cases pass. Direct server TypeScript checking also passes. - Ordinary web creation with the experiment disabled saved native Claude, Opus 5.5 and High effort without runner/model overrides. The actual product task and follow-up both succeeded on integrated source `b2c9fe43f9cd3bc5dd9d22552da0148845ba71a4` with the cloud-built daemon from `04e0202ca81387b17cbe7fd094a8338648356fe9`. The final master reconciliation also changes shared runtime source. Historical proof retains its source identity; a final-source cloud qualification is running before refreshing the preview daemon. - Both actual browser downloads were checked: v1 has three sections/six checkboxes (451 bytes); v2 preserves the original body and adds two Rollback checks (612 bytes). The original attachment stayed byte-identical. The follow-up records session resume and the same provider session. Total estimated API usage: $0.518432800, with no pending/unpriced usage in the successful fixture. - Fixed the live-discovered setup incompatibility: native API-key validation no longer depends on the global Legacy Claude CLI. The existing provider endpoint verifies the key without a paid prompt; subscription/custom routing probes remain intact. - Fixed security review's native-hire credential redirection: agent-supplied URLs, proxy/launch settings and other host environment controls are rejected before inheritance. Native credential references remain supported and partial edits use the saved harness. Added create/hire/update regression coverage. - Current-head cloud CI is green. All three stack PRs have fresh Greptile 5/5 reviews and zero unresolved review threads. Required typecheck, test and build gates ran in cloud CI; Docker work stayed remote. Earlier failed checks remain in their original runs. - Docker and native release builds ran in cloud infrastructure. An inadvertent local Rust prerequisite was stopped; no local Docker build was run. - The final master merge preserves task-loading, permission, skill-retention and OpenRouter changes. Pi uses a fresh combined profile 27, preserving both historical profile-24 declarations; Copilot remains profile 22. Focused protocol checks passed 155 cases (one skipped), the updated Pi attestation check passed 18 (one skipped), and 91 evaluation/catalog checks passed. - Latest-application manual continuation passed in 37.942 seconds on `695e5c005a85bddc8711d3fcddcaf6b7e1fb64d0`, with High effort, native Opus 5.5, Anthropic API attribution, the same resumed Claude session, an independently checked 635-byte file and persisted Done status. It uses the earlier verified Mac daemon; the preceding-source package qualification passed at `dca62c7aeb4b45f0dd91a8e18d386a16a2658ef8`. That package run predates the small TypeScript startup fix and is not relabeled as qualification for it. - Upgrade limitation: changed compiled sidecar identity rejected old durable launch state and recovery hit session-open bounds before eventually resuming. That attempt was operator-cancelled and retained; the successful manual continuation does not qualify seamless upgrades. - Local latency profile, Opus 5.5/High unchanged: three alternating real-ACP loopback samples per variant reduce median provider-ready time from 5.081s to 4.034s. The matched real API task startup improves from 5.533s to 4.348s, but total reply time changes from 17.933s to 20.221s because agent work varies. This establishes a startup saving of about one second, not a consistent total-reply speedup. A real candidate follow-up passed in 20.146s and resumed the same saved session. The loopback fixture makes no external model requests; failed initial probes remain retained. - The startup change passes 107 focused runtime/effort/lease tests, 53 profile checks (one skipped), TypeScript checking and the TypeScript-only runtime build. Native hire fixture checks pass 70 tests; their first local run retained a socket-hangup failure and stale OpenCode expectation, both accounted for before the passing rerun. Broader required checks run in cloud CI. ## Risks - Missing packages, authentication rejection, unsupported settings and unavailable models must fail explicitly. Existing agents retain saved execution; explicit runner changes use existing revision/session invalidation. - Subscription follow-up remains unverified; remaining live acceptance uses API keys per human steering. Historical accounting holds and the separate $50 ceiling remain intact. - The subscription acceptance limit remains explicit for review. No Claude database migration is introduced. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution and browser testing. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] Focused local tests and final cloud verification pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 13:58:52 -05:00
DottaandPaperclip	98ca6ff7c5	perf: reuse verified provider files during sandbox startup (#16493) ... Build provider-pack transport layers once, reuse verified image files in private directories, and upload only missing layers. Preserve archive integrity checks and directory fallback on unsupported hosts. Verified with 44 focused tests, all 48 main CI jobs, cloud Docker Runner checks, and Greptile 5/5 on the merged head. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-19 08:45:09 -05:00
DottaandPaperclip	f3cd3f4ea8	fix(runner): require confirmed exit before remote replacement (#16835) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote runs need one runner to own their process identity and listener. > - A health request can fail while that runner continues to work. > - The active monitor treated a failed request as a process exit. > - Recovery could then launch another runner against the occupied listener. > - This pull request retries health reads and requires proof of exit before replacement. > - A brief outage can recover on the same runner without replaying agent work. ## Linked Issues or Issue Description Refs: #16341 adds remote recovery diagnostics. Refs: #16686 retries verification when restoring a saved remote session. This change applies the shared probe policy to active process supervision. **What happened?** The active remote monitor settled its completion promise after a timed-out or rejected health request. The transport treated that result as an exit and started process recovery. An existing runner could still own the listener. Replacement then failed with `runner_ingress_bind_conflict` until the reconnect deadline expired. Missing or changed ownership markers also did not prove that the old process had exited. **Expected behavior** Retry transient read failures on the same runner. Require a matching ownership marker and proof of process exit before replacement. Preserve uncertainty when identity or liveness cannot be verified. **Steps to reproduce** 1. Start a remote runner and leave its listener alive. 2. Make one process-health request time out or reject. 3. Observe completion and the restart decision. Before this change, the monitor settles while the process is alive, and the restart wrapper permits another launch. **Paperclip version or commit** The defect was reproduced on master at `6e89919631`. The patch is based on current master at `c6a0cf4bce`. ## What Changed - Reuse the bounded read-only recovery probe for active supervision. Retry up to three reads within 30 seconds. - Check the saved process generation when available. Accept only the specific process-not-running receipt as exit proof. - Add an exit-confirmation contract to remote process handles. Require it for restart and shutdown completion. Preserve the cause of supervision errors. - Test transient failures, exhausted retries, conflicting identities, real listening processes, and an authenticated runner with a fake provider. Keep confirmed-exit replacement covered. - Document the recovery policy and coverage limits in the execution contract and maps. ## Verification - Red: the timeout and transport-error regressions failed before the patch because completion settled while the runner was alive. The restart-guard regression also failed because replacement was allowed. - Green: 14 server supervision cases passed, including a real listening process. All 120 shared control-plane tests passed. Focused Runner tests passed for authenticated live-runner fault injection and confirmed-exit replacement. - Review regression: a replacement-health test first failed because its error cause was lost. It passed after preserving the cause and detail. The four affected transport cases and Runner typecheck then passed. - `node scripts/check-maps.mjs` passed for 35 areas and 176 capability IDs. - `pnpm -r typecheck` and `pnpm build` passed locally. The sequential local full-suite run was stopped before completion; no local full-suite pass is claimed. Final commit `bcc1b1d339` passed the complete CI matrix: 55 checks passed and two optional Storybook checks were skipped. Greptile is 5/5 with no unresolved conversations. The PR has no merge conflicts. - The first real-listener attempt could not bind inside the execution sandbox. The same test passed with local socket access. The first package-test invocation used the root configuration and selected no tests; the package configuration ran all selected tests. - CI attempt 1: nine test jobs were cancelled together at 12:54:59 UTC. Their test steps report cancellation. The build and uninterrupted jobs passed. The interrupted jobs were retried once. That retry exposed one confirmed-exit message assertion failure in the Runner shard; the other jobs passed. The compatibility correction preserves the prior message. The failing case and four recovery cases then passed locally. The complete CI matrix passed after that correction. - No live sandbox restoration, deployment, or historical task recovery is claimed. ## Risks - A persistent outage now ends supervision with explicit exit uncertainty. It cannot authorize a competing process. Existing ownership and workspace recovery checks remain required. - Health reads can take up to 30 seconds before reporting an unavailable runner. Retries are read-only and bounded. - Local launchers keep their existing completion contract. No database migration or public API change is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 08:19:37 -05:00
DottaandPaperclip	3858d2341d	fix(recovery): default saved-monitor waits and keep task recovery quiet (#16686) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A task must keep a durable path to its next action. > - A provider can save a task monitor, then fail to yield because it omits internal continuation fields. > - An interrupted provider-pack check can then be reported as permanent artifact incompatibility. > - The task asks its user to certify recovered workspace files, even though this is an operator responsibility. > - This pull request derives the saved monitor continuation, bounds verification retries, and removes that task form. > - Normal recovery stays quiet. Unresolved holds retain inspection and the existing safety checks. ## Linked Issues or Issue Description **What happened?** A yielded completion without `continuation` is rejected even after the provider saves an eligible monitor on the task. Repeated attempts cannot finish. Remote verification also treats timeouts and missing exit status as artifact incompatibility. The task recovery notice exposes raw diagnostics and asks the task user to certify infrastructure files. **Expected behavior** Use current persisted task state to fill routine continuation fields. Retry interrupted verification reads within a fixed bound. Keep normal recovery in the working state. Show a compact notice only when work cannot safely continue. **Steps to reproduce** 1. Save an eligible monitor on the active task. 2. Call `paperclip_finish` with `reportedWorkDisposition: "yielded"` and omit `continuation`. 3. Return a timed-out remote provider-pack command with empty stderr during recovery. Inspect the resulting task notice. **Paperclip version or commit** Source baseline: `aaeff418cf` (master integrated with its current task visibility and saved-source reconstruction rules). Related changes: #15824, #16104, #16451, #16493, #16690. This fix complements saved-result recovery and verified pack reuse. It does not replace their workspace ownership checks. ## What Changed - Allow omitted or null continuation at the provider input boundary. Read current authorized task context and fill a monitor continuation only for the bound run and an eligible saved monitor. Keep accepted results strict. Apply this to Codex, OpenCode, both direct and production-sidecar ACPX paths, and external drivers. Reserve the semantic call identity during the saved-context read and reject cancellation or a changed run/turn. - Bind the shared completion change and continuation helper in a new Copilot v21 source declaration. Preserve v20 history and its decoder support. Qualification stays pending. - Expose the eligible continuation in task context. Completion admission still checks persisted monitor state. No schedule is created or renewed by the default. - Retry read-only provider verification at most twice per command. Hold exhausted reads with a distinct verification-unavailable cause. Reattachment cannot renew the read budget through an unchanged provider-attempt counter. Keep completed verification failures and wrong versions incompatible. - Remove the task file-certification form. Hide the notice only for a matching durable recovery path without an unknown-action replay hold. Keep missed retries, unsafe outcomes, and unresolved file holds visible. - Add production-component stories for automatic recovery, blocked and expired holds, reader access, persisted repair decisions, source reconstruction, queued continuation, and mobile. Preserve master’s guarded saved-source reconstruction choice after confirmed source destruction. Update the execution contract and access maps. ## Verification - Completion, helper, Codex, OpenCode, ACPX, and profile regressions pass. After the sidecar review fix, 271 tests across the production sidecar and three provider suites pass, plus 72 helper/profile tests. The real sidecar MCP bridge tests cover saved-monitor acceptance, no-monitor refusal, cancellation, duplicate call identity, and raw-versus-forwarded semantic receipt hashes. The OpenCode tests also cover eligible-monitor acceptance and refusal without one. - The fresh full native executor suite passes all 743 tests. Four profile, installation-integrity, and ACPX driver suites pass all 168 tests. The earlier CI fingerprint failure required a new immutable source declaration; its follow-up Docker failure required the matching context allowlist entry. Both fixes are verified locally. Targeted server monitor, tool authority, and provider verification suites pass. The probe tests include a real timed-out subprocess, an unrelated responsive timer, transport errors, and completed integrity rejections. - All 90 tests in the merged task notice, reconstruction choice, recovery display, and lineage suites pass. They cover reader access, live/missed/mismatched recovery, unknown outcomes, saved messages, retry refusal, and removal of the file form. Both review findings have failing-baseline/passing-treatment regressions: identical polling across retry expiry, and a failed refresh after cached running activity. - Browser walkthrough passed for automatic recovery, unresolved details, a 390px mobile story, and the merged saved-source reconstruction and queued states. The reconstruction control starts disabled and normal automatic recovery shows only Working. The stories use production components and fixture API data. They do not prove live cloud recovery. - Token gates, map checks, and Storybook catalog generation pass. The exact Docker context integrity command passes locally after adding the new build declaration to its allowlist. - Workspace typecheck and build pass on the final source, including the versioned identity update. The final server and UI guards pass their focused tests, typechecks, builds, and token gates. The final standalone Storybook builds. The saved-monitor default retains master’s stricter child-task visibility filter. Re-review caught an additional production-sidecar validator before forwarding; the fix now defaults at that boundary and has passing regression controls. After preserving master’s new reconstruction capability, 68 targeted reconstruction, tool-authority, and provider-verification server tests pass. The current-master integration at `2a80050a0d` has a fresh 5/5 review with zero unresolved threads. Its complete CI matrix, including Runner tests, is the merge acceptance record. - On the final master integration, local typecheck, build, and all 90 recovery UI tests pass. A concurrent runner run hit five 5-second sidecar startup timeouts; an isolated sidecar run passed 86 tests and timed out in the unrelated unsupported-bootstrap case. Those local deadline failures remain recorded; the canonical CI Runner lanes use their normal deadlines and pass. The unsharded local full-suite attempt did not complete. It spanned source edits; the fresh native executor run above passes. An unrelated workspace fixture timed out once and then failed to reserve 32 contiguous ports on this host. Its unchanged-base control and CI server suite pass. Full-suite acceptance uses the fresh latest-head CI matrix; this PR does not claim a passing unsharded local run. ## Risks The default applies only to an eligible persisted monitor. An explicit invalid continuation still fails validation. The added read occurs only when a yielded report omits its continuation. Admission rechecks current state after that read. An interrupted verification command can add one extra 30-second probe. Completed rejections stay closed. These probes do not change the pack, launch provider work, or replay external tools. Two inconclusive reads hold the original run. They do not schedule another reattachment with an unchanged provider-attempt counter. This PR’s pending Copilot v21 declaration binds the changed shared contract, resolver, direct driver, and production sidecar sources. The previous v20 declaration is immutable and remains decodable, but launch requires the new exact identity. Qualification remains pending. Task users lose the file-certification form. Board-authorized source-only repair remains available through the resolution API with the original evidence and stop-receipt requirements. This branch does not deploy or recover historical held runs. > This is a bug fix and UI polish within existing recovery behavior. ROADMAP.md was checked. ## Model Used OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with tool use and code execution. The host does not expose the context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (changed-area suites; the full local run limitation is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-18 21:43:15 -05:00
DottaandPaperclip	e6dcc6b095	Connect personal Muse through secure native execution and Vault OAuth (#16448) ... Add company-scoped Muse native control, OAuth, wake receiver and recovery. Recheck current authority after waits and before effects or receipt replay. Preserve task-read scope, assignment expiry, independent holds, nonblocking reassignment and truthful cleanup provenance. Retain source trust on progress comments and verify current Muse V8 authority through real signed-JWT HTTP requests. Preserve the upstream trimmed ordinary authentication lookup. Refs #16448. Preserve captured canonical queue recovery and concurrent environment preparation while retaining the scoped Muse V8 exclusion. Normal migration 0354 and existing Muse SQL remain unchanged. Fresh current-head Greptile review is 5/5 with the project-write finding addressed. Required CI and independent technical review gate the merge; maintainer override is reserved for formal approval rules. Recheck current Muse project-write authority after all receipt/task/run/admission waits, before creation or replay. Real PostgreSQL regressions reproduce four late effects/replays on the old route; the repaired three-file treatment passes 354 cases with no skips, and the scoped server typecheck passes. Ordinary/Dot behavior is preserved. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-17 22:44:36 -05:00