6325 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
DottaandPaperclip	86814c6550	fix(recovery): ready retained sandboxes before preparation (#18934) ... ## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 09:04:13 -06:00
DottaandPaperclip	32e5eada06	test(evals): pin reviewed attribution and secure question repairs (#18974) ... ## Thinking Path > - Paperclip must complete work through ordinary human and agent controls. > - The remote chat campaign exposed observer and setup defects as well as product discovery gaps. > - Case 466 rejected a valid handoff that used headings, pronouns and inline HTML. > - Case 487 seeded three daily runs although its declared journey needs six. > - Secure-entry replies used a real bound question card that the observer did not support. > - Team installation omitted its existing strict request schemas from the API search result. > - This PR pins the reviewed coherent repairs and keeps their metadata in sync. ## Linked Issues or Issue Description Refs #18921. Refs #18964. Refs #18967. Refs #18968. Refs #18970. Refs #18971. Refs #18972. The previous launch remains unchanged. Case 53 passed its latest native attempt. Cases 466, 487, 580 and 610 still retain their prior failures. The next representative run must test the repaired conditions. The company-secret case follows only after the personal-secret journey proves real secure entry and worker use. Its separate case-local follow-up is held for a later pin; case 578 is excluded from the next four-case selection and has no pass credit. ## What Changed - Pin immutable candidate `e757e447371c4781a3eff3a0ea9dc9c4eceb060a` and its existing descriptor snapshot. - Record the explicit oracle versions and source-file changes for timeline attribution, review fixture allowance, and bound personal/company secret questions. - Keep all 244 case identities, human requests, original run and action counts, time allowances, provider/model, private evidence limits and broader qualification holds unchanged. - Preserve the actual request-body schema affordance for team preview and installation, existing response links and prior help. ## Verification - The final integrated candidate passed 442 owning checks across 18 suites. All 242 declarations loaded with zero fixture callbacks; the existing manifest joins those declarations and two legacy tasks into 244 cases. - The team service fixture passed 29 tests. The frozen candidate API consumer passed 2045 tests after the three-line schema exposure change. These are component controls, not a new native pass. - The stale inventory expectation reproduced one failure with 16 passes. Its unchanged derivation traces five previously reviewed YAML corrections. Updating only the expected constant passed 17 controls, and the existing definition-drift and catalog-tamper negatives still passed. No scenario YAML changed in this candidate. - The existing SDK guard confirms all 506 inputs are unchanged and the reviewed registry remains `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No compiler build or registry renewal was needed. - The existing pin, TestOps contract, adapter, reconciliation and streaming controls passed locally. Current master also passed the existing reporting-lock guard with its actual current lock. - The final source includes the reviewed immediate-actor/consecutive-pronoun controls and checks every rendered question title, header and submit label for unsafe instructions. The remaining fixed scanner reasons are diagnostic only. The combined source passed 442 checks; all 131 pin/contract/API/streaming controls passed. - Fresh exact-head [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41847423336) passed: 48 successful jobs and one selector skip, including both required aggregates. Greptile reports 5/5 for this head, with no actionable findings or unresolved review threads. The current optional Paperclip review is queued and is not credited as an assessment pass; its prior-head 5/5 remains historical. The finite source peer review found no material issue and did not claim native execution. The prior CI attempt is retained. No new native pass is claimed by this PR. ## Risks The new source repairs observation and setup, but it does not prove the next native outcome. The secure-entry accounting records one real chat request and one authenticated UI answer as the two declared human actions; it keeps the raw comment count separate. The question path requires the bound pending card, actual resolver audit and delivery facts. Ordinary cases retain their existing accounting. Private scans and archive limits can still reject evidence. Prior failed and unknown attempts remain immutable. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, integration and local checks. The host does not expose the context-window limit. The evaluated model stays OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:40:01 -06:00
DottaandPaperclip	84e9c8f364	fix(retry): wait for admitted source cleanup before dispatch (#18898) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. **What happened?** A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. **Expected behavior** Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. **Steps to reproduce** 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. **Paperclip version or commit** Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. **Deployment mode** The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:33:39 -06:00
DottaandPaperclip	5d226ca08a	perf(chat): reuse bound-run authorization query templates (#18966) ... This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. **Current behavior** At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. **Proposed behavior** Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. **Reason and benefit** Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:32:31 -06:00
DottaandPaperclip	2cbfc35a07	perf(github): reuse the inherited review wait query template (#18965) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. **Subsystem affected** Server GitHub chat review waits. **Current behavior** Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. **Proposed behavior** Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. **Reason and benefit** Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:29:59 -06:00
DottaandPaperclip	76e43852ac	test(recovery): publish fixtures under the owned workspace path (#18969) ... ## Thinking Path > - Paperclip preserves agent workspace files before it retries stopped work. > - A saved workspace must use the directory owned by its workspace ID. > - The recovery test helper saves files under a generic `remote` directory. > - New tests from #18908 restore that fixture into a new allocation. > - The existing path validator rejects the fixture before the intended checks. > - This pull request gives the fixture the same published path as real workspaces. > - The tests can then verify native continuation and current repository denials. ## Linked Issues or Issue Description Refs #18908, which added the five native provider boundary cases. **What happened?** Five `workspace-recovery.test.ts` cases fail with `remote_workspace_published_path_invalid`. The fixture saves real files at `<allocation>/remote`. A cold restore requires `<allocation>/.paperclip-workspaces/<workspaceId>` so it can preserve the workspace-owned directory name. **Expected behavior** The fixture must satisfy the existing path contract. The complete case must start one native continuation. The four catalog cases must reach their repository access and warning checks. **Steps to reproduce** Run the selected five cases on master `0eaabe5c0b37c99d827f78a1a8598e7b3534078a`. They fail before their intended assertions. The same five failures reproduce on exact CI merge `dd508ded616ae0dd6062ddb03572f8b08a79c10c` with `chat-attachment-reuse.ts` restored to its pre-optimization `55d7df2bad` bytes. The failure is independent of the prepared-query change in #18966. **Paperclip version or commit** Master `0eaabe5c0b37c99d827f78a1a8598e7b3534078a`. The path validator predates the new cases; commit `b5b0dbb03f` introduced it. **Deployment mode** Local source tests with real PostgreSQL, temporary filesystem archives and a controlled native provider boundary. ## What Changed - Move the helper's real files into `.paperclip-workspaces/<workspaceId>` before recording the existing target, lease and workspace bindings. - Keep all production path validation and all native execution, archive and current-access assertions. ## Verification - RED: the five selected cases fail on current master without either prepared-query patch. - Causal RED: the five cases also fail on the exact CI merge with only the chat production module restored to its pre-optimization bytes. All five report the same invalid published path. - GREEN: the same five cases pass with this fixture-only change. The other 239 cases were skipped in this focused run. - Command from `server/`: `pnpm exec vitest run src/services/workspace-recovery.test.ts -t 'dispatches the recovered failed-preparation Retry once through the native provider boundary'`. - Node24.22.2 and real embedded PostgreSQL. The complete case reads restored `saved.txt`, writes `continued.txt` and completes through the real finalizer once. The controlled backend does not make a live provider call. - The first attempted baseline on the older `81b01bcc16` checkout skipped all 188 cases because the selected tests did not exist there. That attempt is not a baseline proof. - Greptile reviewed current head `4ca1487ddfafe868daeef5e1cf7a3b1c504e9b32` at 5/5, with no actionable findings. Current-head CI is green: 51 successful checks, two intentional skips and the successful Snyk status. [Workflow 41840916418](https://github.com/paperclipai/paperclip/actions/runs/41840916418) completed successfully on attempt 3. No duplicate full local build was run for this test-only change. - Initial CI attempt: the unrelated task-reload browser case timed out waiting five seconds for its heading after Reload page. Its assertion snapshot shows Loading conversation; the later retained screenshot shows the expected heading, saved comment and composer. The cause is unknown. The later Cloud-auth task case in shard 3 had the same five-second heading timeout: its assertion snapshot also shows Loading conversation, and its later screenshot shows the expected heading and composer. Both unchanged UI tests passed in the stacked #18966 run. Shard 7 passed its one same-source retry: 20 passed, one skipped; the previously failing task-reload case passed in 4.1 seconds. Its checkout remained `da8190914f2d5b77c127f6a8b0cdd2250f009611`. Shard 3 passed its one actual retry: 41/41 tests passed in 7.8 minutes; the previously failing Cloud-auth case passed in 6.9 seconds. Its checkout also remained `da8190914f2d5b77c127f6a8b0cdd2250f009611`. Passed jobs were preserved. These successful retries establish recovery on unchanged source; they do not establish the timeout causes. The original failure remains part of the verification record. ## Risks - Low risk. This changes only a test helper. The helper now relocates its files before it records all existing bindings. - Production path validation, workspace ownership and current repository access remain enforced. - The focused run covers five cases. Current-head CI must cover the rest of the shared helper's callers before review is complete. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Context-window size is not exposed. Used reasoning, tool calls, source editing, real PostgreSQL tests and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.9	2026-11-04 08:28:39 -06:00
DottaandPaperclip	396b7fc62a	fix(runner): scope session diagnostics to the active run recorder (#18944) ... ## Thinking Path > - Paperclip manages agents and their work. > - The native executor can retain a transport between runs. > - Each run recorder has a shorter lifetime than that transport. > - The transport kept its first recorder after warm reuse and completion. > - A later close could write to the wrong run or a sealed object log. > - This change binds diagnostics to the active run epoch and retires that binding at execution exit. > - The existing recorder still owns accepted-byte draining and honest incomplete finalization. ## Linked Issues or Issue Description **What happened?** A warm native transport retained the first run's diagnostic callback. Reusing the session for another run did not change that callback. A later idle close could append to a retired writer and emit a persistence warning. **Expected behavior** Active diagnostics belong to the current company, session and run. A retired transport callback must not use a completed recorder. Diagnostic metadata must not delay essential execution cleanup. **Steps to reproduce** Run the native warm-session regression with two sequential run owners. Close the retained transport after both runs finish. The original code calls the first recorder. The real completion regression also observes the resulting closed-writer warning after normal lease cleanup. This is a local source defect with behavioral red/green proof. Repeated hosted generic persistence warnings motivated the investigation, but their original exceptions were not retained. This PR does not claim to identify every such warning or repair an export failure. Related #18379 queues sanitized performance spans in heartbeat. It does not own transport diagnostic callback lifetime. ## What Changed - Bind transport diagnostics to an exact active company/session/run epoch, including warm reuse. - Activate a retained channel in the existing session-admission hook after identity validation and before attachment starts, including an attachment failure before `onSession`. - Forward active diagnostics synchronously. Detach their recorder at execution exit. Keep the original asynchronous metadata semantics. - Keep idle transport evidence inspectable. Log only a fixed category and trusted company/run identifiers to the process logger. - Preserve recorder-owned byte draining, redaction, source coverage and genuine persistence-failure warnings. - Cover actual native executor, completion and lease-close boundaries, including stalled local appends and exact object-log source/hash preservation. - Place the retained-workspace test fixture under the existing published-path contract after merging newer recovery tests from master. Production validation and native/catalog assertions remain unchanged. ## Verification - Current head: `c500e4933a85a4c453f66c872fa335108c661b15`. Normal master merges preserve upstream behavior. The one textual conflict was in the executor imports. - The first GitHub run tested merge tree `a4cb59448a3b5cc2047ac41c691720e6acec6e8d`, which included newer recovery tests from master. Five cases failed with `remote_workspace_published_path_invalid`. The same five failures reproduced on an identical local tree. The fixture now puts retained files under `.paperclip-workspaces/<workspaceId>` and keeps the target, lease and authority bound to that path. Production validation and the native-success/catalog-denial assertions are unchanged. - On unchanged `c500e493`, the full recovery file passed **244/244** tests. Existing cold-restore and occupied-root controls passed **4/4**. Server typecheck, server production build, module boundaries and whitespace checks passed. An intermediate missing-import setup failure was corrected and is separate from the five-case behavioral red. - Before that fixture correction, local `f2fdaddb` passed **1,221 tests** across the five diagnostic, executor, output, completion and object-store files. Recursive typecheck and full repository build also passed. Those results are tied to that earlier revision; [fresh current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41842043248) must qualify the final merge. - Behavioral regressions cover the original closed-writer warning, warm A-to-B ownership, and an actual Runner attachment exception before `onSession`. The admission hook gives the attachment diagnostic to B and leaves A untouched. The attachment fixture uses synthetic event persistence and outer failure projection; it does not certify a real database terminal projection. - The tests preserve synchronous diagnostic admission and asynchronous metadata recording. A stalled append reaches the existing bounded finalization result with unknown size/hash. The normal object-log control preserves canonical bytes, SHA256 and complete source coverage after lease cleanup. - An initial local run had **37 PostgreSQL startup failures** because the Darwin package lacked aliases to its included versioned libraries. Only isolated `node_modules` aliases were restored. No source or lockfile changed for this setup issue. A passing intermediate run crossed a source merge and is excluded from immutable-head proof. - An earlier combined test run passed 1,216 assertions but exited with a completion `afterAll` timeout. Its cause remains unproved. Later isolated 60/60 passes used the original timeout and do not erase that failure. An earlier repository-wide `pnpm test:run` was interrupted after its source became obsolete; it is not full-suite or current-head proof. Greptile reviewed the exact `c500e493` head at **5/5** with no open findings. Fresh CI must pass before merge. The native review is queued behind its separate saved-file recovery; no current-head native assessment is claimed. The provider execution loop is a fixture. These tests do not start a model or prove hosted recovery. Missing accounting receipts remain pending, not zero cost. Checkout-release invocation is observed; this is not a separate task-arbiter journey. The original native review published a valid finding before a separate workspace-export failure. The finding is fixed; this PR does not claim to fix that export failure or every hosted persistence warning. ## Risks Idle process logs intentionally omit raw provider text. The existing bounded transport evidence retains its own diagnostics. Accepted fs appends that do not settle retain the existing unknown-size/hash warning and cannot be certified complete. No new timeout, storage policy, schema or provider execution is added. ## Model Used OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis, implementation, tests and review. The exact context-window size is not exposed. Tool use and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local tests and recorded passing checks, pending checks and unresolved failures above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will require all Paperclip CI gates to be green before merge - [x] I will require Greptile 5/5 with no unresolved findings before merge - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:04:52 -06:00
DottaandPaperclip	0eaabe5c0b	refactor(accounting): remove company serialization from paid settlement (#18917) ... Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-04 07:29:28 -06:00
DottaandPaperclip	b891e69bd2	fix(native): correct unread wake comments before completion (#18937) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents report completion through a server-owned tool boundary. > - A truncated external-chat request requires a complete run-bound comment read. > - The terminal guard checks that receipt after the provider ends its turn. > - Early completion feedback omitted the same check and could accept an unread report. > - The provider then lost its chance to correct the report in the same turn. > - This pull request returns the existing read requirement through the completion tool before accepting the report. > - The final guard still protects the exact source and snapshot after provider settlement. ## Linked Issues or Issue Description Related: #16995 keeps run-scoped readers available to retained provider threads. #18913 preserves queued comment input. This change fixes a separate completion-feedback boundary. **What happened?** A native external-chat run could submit an accepted completion without reading all server-bound comments. The terminal guard then failed the run with `native_current_wake_comments_unread`. Early feedback also accepted partial, changed and foreign read receipts. The agent did not receive the required correction while it could still act in the same turn. **Expected behavior** The completion tool must reject the report with a `read_current_wake_comments` instruction and the cursor requirements. A complete current read must permit the corrected report in the same turn. The final snapshot check must remain effective. **Steps to reproduce** 1. Create a running native task with a truncated external-chat wake and exact accepted comment IDs. 2. Call completion feedback before reading the comments, or after only a partial or stale read. 3. Observe that the old code returns an accepted report instead of the read correction. ## What Changed - Check the existing run, company, task and agent binding before every completion-feedback acceptance return. - Reuse the authoritative current-wake reader and terminal receipt validator. - Share the two unchanged read-error codes from their owner across the validator, feedback and executor classification. - Convert only the two owned unread or changed-snapshot errors into a bounded tool correction. Other errors still propagate. - Keep the final validator, receipt parser, result persistence and admission policy unchanged. - Add real PostgreSQL regressions and an actual OpenCode MCP tool sequence with a controlled local provider process. - Document the early correction and final snapshot fence. ## Verification The source checkout uses Node 24.22.2 on macOS arm64 with a fresh owned embedded PostgreSQL database. Final published tip: `37bfb269432f86f98cab1a58c09b5519ed9477fd`. Its production code is commit `8c7297318c3a841fc1c72c846acfaabf58adabe9`; the last commit adapts three existing module mocks. The expanded affected group passes **1,284 cases** with **35 existing opt-in skips**. The skips require captured private-runner transition fixtures and a configured fresh database, which were not supplied. All nine real PostgreSQL/MCP cases execute and pass. Recursive typecheck and the full build pass for production commit `8c7297318c`; the final changed test layer also passes. - Untouched master `1a39dea9aa2bb9bea416439b0a574c788e88b9a0`: four causal failures. Unread, partial, changed and foreign complete receipts incorrectly receive accepted completion feedback. The ordinary inline-complete control passes. - Current source: all nine new cases pass. The actual authorized reader returns the full accepted comment. The same running task and run remain intact, with no replacement or accepted-result replay. - The OpenCode native backend sends a real MCP finish rejection to the controlled provider process. The process follows the returned reader instruction, reads every page, and resubmits finish. One provider turn produces one semantic result proposal. The terminal snapshot fence still rejects a later comment edit. - All 98 affected OpenCode driver and MCP bridge cases pass with the fixture-mode addition. - The first tool-sequence attempt failed on a fixture-only relative path before starting the provider. The test now uses its module URL. This setup failure is separate from the four causal failures. - The first recursive typecheck found missing local workspace links in the clone's excluded sandbox-provider directories. Existing dependency links were copied into the isolated clone. No lockfile or source change was made for this setup repair. - At `c57e532a87`, all 1255 cases in the six-file affected server family, recursive Node 24 typecheck and the full build pass. At production commit `1ae9f54107e1762bc8222572b365a013d4572d1a`, all 1,257 affected cases, recursive Node 24 typecheck and the full build pass. - Greptile found that the initial correction could restart a partial read after page two. On `c57e532a87`, following that correction fails with the actual reader cursor-order error. The follow-up continues the caller's most recent cursor and retains the changed-snapshot restart path. All nine new cases pass, including this regression, a stale-partial control, the foreign receipt refusal and the actual same-turn MCP sequence. - [Linux server3/12 job](https://github.com/paperclipai/paperclip/actions/runs/41788671370/job/126437378334) found five failures in the older format-feedback SQL fixture. Its flat heartbeat row did not match the real joined projection. The same five failures reproduced locally. The test-only follow-up returns a coherent run/agent/task projection and keeps the real resolver and formatter assertions. All 14 format and real-PG/MCP cases pass, and the generated server test layer typechecks. Production code and the earlier 1,257-case/typecheck/build proof are unchanged. - Qualification of the shared-code change found 49 failures because whole-module mocks omitted the new exports. The three mocks now pass through actual module exports and keep their controlled reader methods. Literal error assertions remain independent. The final 1,284-case group passes with the exact 35 opt-in skips described above. No production fallback or substitute code was added. - [One Linux Runner package attempt](https://github.com/paperclipai/paperclip/actions/runs/41792315273/job/126448500037) passed 4,913 cases and skipped 10, but its synthetic post-rename credential durability case failed with ENOENT. The complete Runner package tree and lockfile are identical to the [earlier Linux control](https://github.com/paperclipai/paperclip/actions/runs/41790199252/job/126441739681), where all 42 credential cases passed on Node 24.22.2. The exact case also passes once on both base and head on macOS with Node 24.22.2. The original failure cause remains unverified; OS/load and shared-test timing uncertainty remain. Credential code and assertions are unchanged. The single affected-job rerun passes, including all 42 credential tests. The original failure remains unexplained. - An ordinary full local test attempt was stopped before the review repair. It has no completion summary and is INCOMPLETE. Current-head full CI completed; no local full-suite pass is claimed. Greptile reviewed the final published tip at 5/5 with zero open threads. All current-tip checks pass: 54 successful check runs, one successful legacy commit status, 3 skipped and 2 neutral, with no pending or failing checks. Paperclip also reviewed final tip `37bfb26943` at 5/5 with no remaining findings. Its prior 4/5 code-ownership recommendation is fixed and resolved. No review nudge or hosted task retry was issued. The controlled OpenCode process uses the supported v1 factory envelope to exercise the common native feedback and MCP boundary. It does not call a live model or qualify a provider pack. PostgreSQL, authorization, pagination and receipt checks are real. No hosted task, model, permission or saved result was changed. Recovery of an already accepted historical run remains unverified. ## Risks - Each finish call adds an exact binding query. A mandatory read also validates the current accepted-comment snapshot through the existing transaction. Cost grows with those exact comment IDs, bodies and attachment metadata; it does not scan unrelated task history or add idle polling. - A valid read can become stale before the provider settles. The unchanged terminal guard still rejects that race. - Missing ownership or unexpected validation errors remain refusals. This change does not grant replay authority or turn an accepted historical result into fresh work. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window. The coordinator performed independent read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:19:12 -06:00
DottaandPaperclip	669538534c	fix(recovery): preserve failed remote reattach workspaces (#18936) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs can edit files in a remote execution workspace. > - A failed reattach can leave an observed coordinator and unpublished files. > - The failure handler recognizes the older workspace-sync descriptor but misses the modern workspace reference. > - The released task lock then prevents ordinary session recovery from claiming that old run. > - This pull request retains the files through the existing Stop, cleanup and workspace recovery owners. > - Exact ownership and Stop evidence permit file preservation while unknown provider effects remain under manual reconciliation. ## Linked Issues or Issue Description Related: #18877 protects native manual admission holds. #18879 handles scheduled native retry lock reclaim. #18918 fixes remote workspace row-lock order. Those changes do not record the missing modern failure-retention obligation. **What happened?** A native reattach failure can leave a failed run, a dirty modern remote workspace, an expired observed coordinator and a free task execution lock. The run has no accepted result. The failure handler skips file retention because it checks only the older descriptor. Recovery cannot export while the old coordinator remains owned. A mismatched workspace generation can also make orphan cleanup miss the existing file claim. **Expected behavior** Retain the exact original allocation and files. Stop the allocation through the existing lifecycle owner before retiring the stale coordinator. Keep checkpoints and unknown external effects available for manual reconciliation. Unsafe or changed source authority must prevent physical cleanup. **Steps to reproduce** 1. Prepare a modern remote workspace for a native run. 2. Record a provider checkpoint and an active interruption incident. 3. Fail reattach preparation and release the task execution lock. 4. Let the coordinator lease expire. 5. Run the existing orphan, environment cleanup and workspace recovery workers. 6. Verify the original files from a cold archive read and verify that no new native session starts. ## What Changed - Include modern prepared references in the existing heartbeat failure-retention producer. - Add a file-only historical intake for an exact failed observed run in the normal expired reattach state. - Require an untouched active ephemeral lease, a free blocked task, matching workspace writer and generation, no accepted result and complete single-owner allocation history. - Persist the existing StopAndRetain intent before provider I/O. Use nonblocking row locks and classify busy ownership only after rollback. - Use the existing native environment settlement owner after exact Stop and retention receipts arrive. Keep its single scoped discovery query. - Preserve source and coordinator fingerprints across the handoff. Keep original checkpoints, errors and manual unknown-effect incidents. - Keep stale or malformed modern workspace claims held when preservation intake refuses them. - Add actual PostgreSQL, heartbeat, allocator, worker restart, archive and safety controls. Document the narrow recovery contract. ## Verification Current published head `b1f49691187a398415cf9b16b723e3ab473f0f91`: all required CI checks pass (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with no open threads. The separate Paperclip assessment is queued and its current response says work is paused. That assessment is unavailable; no additional review nudge or hosted retry was issued. All local commands use Node 24.22.2 on macOS arm64. - Exact published head: `b1f49691187a398415cf9b16b723e3ab473f0f91`. Required CI passes (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with zero open threads. The optional Paperclip assessment remains queued/paused without a completed score. - Final recursive `pnpm -r typecheck`: passed. - Final `pnpm build`: passed. - Final three native producer, containment and scheduling files: **67/67 passed**. These include the real coordinator and worker restart, the production heartbeat recovery callback, exact Stop receipts, cold archive restore, and zero native backend, new allocation or destructive release calls. - The plugin allocator cases use the actual selector with `reuseLease: true`. Active and released ephemeral leases receive a fresh allocation. - The assembled reattach producer checks that the task and coordinator claim commit before physical resume. A historical pin refuses that current owner. A later claim behind a committed pin remains held. - The safety controls cover reusable allocations, manual and unknown states, cleanup in flight, accepted but unbound results, late allocation owners, wrong or malformed workspace authority, changed generation/checkpoint, lost Stop acknowledgement and a real cleanup-producer lock conflict. - Untouched base `e7549ab6131555f655736e6b544892a3b1e323cb`: four expected behavioral failures and one passing legacy-descriptor control. The failures cover the modern producer, original provider history, historical retention and wrong-epoch cleanup. Fixed production files were restored byte-for-byte after this comparison. - The first affected six-file run completed **562 passed / 1 failed**. The failure was an existing HTTP case with `socket hang up`. That exact case passes separately on both untouched base and this branch. The original connection failure remains unexplained. At `0e92de8c`, the complete workspace family passes **188/188** with unchanged assertions. The environment cleanup owner family also passes all 23 cases. The later guard changes are qualified by the complete 67-case native owner group. A full repository test run was not repeated locally; the requested full-ci workflow runs the complete repository suites. - Published `0e92de8c` fails both new explicit-null Stop/export marker controls because it creates a new retention pin. The follow-up uses the shared presence-aware Stop guard. Both controls now preserve the marker and record no new retention pin, Stop or destructive release. - Paperclip's review found that malformed workspace authority could throw and abort an orphan page. Published `5f6cd4f5` fails all three new controls: intake, the actual page and after-Stop settlement. The same strict schema now uses a safe parse, and the existing OBSERVED hold runs before the prepared-source lookup. The malformed source stays held while the next valid source receives its retention pin. - Two existing scheduling tests caught the draft's second global discovery query. The final single-query repair passes all 36 unchanged scheduling tests. - An earlier restarted archive assertion stopped after 20 seconds. Its recorded Stop was valid, and the coordinator had settled. The existing evidence recheck deadline was still 13 seconds ahead. The final test observes the unchanged 24–36 second policy within its existing 60 second case limit. The external fake provider Stop and file transport are controlled test boundaries. Database ownership, lifecycle workers, archive bytes, encrypted local storage and cold restore are real. This does not claim a live cloud sandbox, S3 or model run. No hosted task was changed for this PR. ## Risks - Historical intake is deliberately narrow. Reusable, already released, mismatched or competing allocations remain held. - File preservation does not settle unknown provider effects or authorize a repeated provider turn. The task can still require manual reconciliation. - A busy or changed owner delays preservation. The original files and claims remain retained. - This does not repair the underlying sandbox readiness failure or establish the cause of a separate archive-capture failure. - Deployment and recovery of original hosted tasks remain unverified. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window size. A separate Codex agent and the coordinator performed read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:19:03 -06:00
DottaandPaperclip	ecc253b73d	fix(workspaces): retain redacted archive capture causes (#18935) ... ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Remote workspaces need a verified saved archive. > - A failed capture must retain a safe cause for diagnosis. > - The codec and capture wrapper erased known validation and filesystem failures. > - The one-shot output fix in #18888 preserves only the existing disk and Git messages. > - This PR retains a closed set of capture failures without exposing workspace data. ## Linked Issues or Issue Description Refs #18888. **What happened?** Remote archive capture can still report only `remote_workspace_operation_failed`. This was observed after the public revision `f211d44c00fbc31d22e7ff687e22dfa05643da2a` included #18888. That observation identifies the capture stage. It does not identify the underlying failure. The real local codec gives a specific validation error for an escaping symlink. Node also gives a specific errno when an archive is missing. The capture wrapper erased both causes. The codec sent raw filesystem paths to its parent process before that wrapper discarded them. **Expected behavior** Retain exact, fixed capture validation and filesystem classes. Keep arbitrary messages, paths, filenames, file contents, and command output private. **Steps to reproduce** Call the real capture transport with an escaping symlink. Separately, create a canonical capture receipt whose archive is missing. Both fail with the generic transport message on unchanged master. The new tests use real Node processes and files. ## What Changed - Add a self-contained classifier for fixed capture validation messages and a finite Node errno set. - Redact unknown codec capture errors. Preserve recognized classes through helper installation, capture, and split or combined command output. - Keep the existing disk and unfinished-Git messages first. Keep other command behavior and all save, retry, ownership, Stop, and permission policy unchanged. - Document the diagnostic contract. ## Verification Node 24.22.2, pnpm 9.15.4, macOS arm64, isolated source checkout. - Six new real-process cases failed before production edits. The same six passed after the repair. - The complete archive and transport family passed **93/93**, with no skips. A new contract test covers every capture-side reason in the actual serialized codec; an unlisted-reason mutation fails that test. - These tests cover real symlink rejection, missing canonical archive, serialized-helper ENOENT/EEXIST redaction, unchanged source bytes and canonical receipt, unknown errno, malformed output, other command stages, and the successful capture/restore path. - A safety test caught a regression in the first patch: early redaction hid a Git child's existing disk-full signal. Unchanged master passed both split/combined cases; the first patch failed both. The final family passes both after the same closed disk recognition was restored. The earlier failure remains recorded. - Full recursive typecheck and `pnpm build` passed on the current head with Node 24.22.2. - A fresh probe of the current head's compiled `server/dist` artifact passed. It ran nine real Node commands with combined one-shot output. It checked both fixed failure classes, unchanged source/receipt bytes, and a successful capture, hash check, cold restore, and scoped cleanup. - The first head's full local `pnpm test:run` failed one unchanged GitHub fast-response streaming-quarantine acknowledgement assertion: an acknowledgement appeared where the test expected none. The first chat shard passed 319 cases; the second passed 323 and failed one. One bounded focused comparison passed on both that head and untouched master. It did not explain the original failure. No chat source, assertion, or timeout changed. No full local-suite pass is claimed. - All **57 current-head contexts are terminal and clean**: 53 success, two skipped, two neutral. Required [CI](https://github.com/paperclipai/paperclip/actions/runs/41781977986) gates passed. Fresh Greptile and Paperclip assessments both report **5/5** on `d7b2fcc22c75e3157e0c53bf4725e665810f2d19`, with no open review threads. The optional security scan is neutral, not an independent security assessment pass. ```sh pnpm exec vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts ``` This is a **diagnostic repair**. It does not establish the cause of the observed hosted failures, qualify a live provider capture, or recover the original files. No hosted resources were executed, started, changed, or retried for this PR. ## Risks The classifier can expose only a fixed message or a fixed class from its finite errno set. Unknown and altered messages keep the generic failure. Existing disk/Git priority and command-stage restrictions have explicit tests. The new classes may improve later diagnosis; they do not authorize another operation. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this agent session. The agent used reasoning, terminal tools, code edits, and real Node/filesystem tests. A separate Codex agent reviewed the production diff. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:55 -06:00
DottaandPaperclip	b6352578c5	fix(task-retry): retain chat Retry during identity contention (#18896) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry must preserve a person's request and enforce the current chat authority. > - Chat retry preparation takes nonblocking source locks before it can authorize the saved source. > - Sustained lock contention could discard the Retry request before it was saved. > - This pull request saves one unadmitted request in the existing chat retry queue. > - The worker checks the source and current access before normal admission. The task shows a scheduled retry while it waits. ## Linked Issues or Issue Description **What happened?** Task Retry returned HTTP 500 when another transaction held the saved chat principal's identity lock or the endpoint authorization row. The source preparation error rolled back the canonical Retry intent. The same gap existed when Retry had to restore a certified input batch that had never reached a provider. With the bounded synchronous retries in #18895, sustained contention returns 409 without a saved intent. **Expected behavior** HTTP 202 retains one Retry request. Acceptance does not grant source authority or mean that a provider started. The existing worker prepares the request if the source locks become available before its saved deadline. It must reject changed input, revoked access, Stop, or superseding work before input adoption, publication, or provider work. **Steps to reproduce** 1. Prepare a failed task with its original admitted GitHub chat source. 2. Hold that principal's `chat-identity` advisory lock or the endpoint row on a second PostgreSQL connection. 3. Send the normal task Retry HTTP request. Check the response and persisted intent. 4. Release the lock. Process the existing chat retry queue and check that one authorized successor is queued. **Paperclip version or commit** Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. **Deployment mode** Source checkout, real Express Retry route and embedded PostgreSQL. External chat transport is a test fixture. The live hosted fix has not been deployed or verified. Related: #18895 handles brief admission contention with bounded synchronous attempts and returns 409 if it remains busy. This change retains a durable request through sustained identity contention. It uses the existing background retry owner and adds real HTTP, restored-input, Stop, publication, restart, and task-presentation coverage. ## What Changed - Preserve a strict pending-source action only when the identity try-lock reports contention or the endpoint authorization query reports lock-unavailable through an explicit Retry callback. Bind it to the canonical task Retry, company, task, agent, endpoint, conversation, and principal. Other database and authorization errors keep their existing behavior. - Revalidate the canonical intent, current board permission, saved source, and latest execution before the existing worker can prepare that action. Ordinary authorization rejects an unprepared action. - Keep certified input restoration inside source preparation. Recheck the saved deadline before the prepared payload write. Expiry, a nested savepoint, or outer rollback cannot commit partial restoration. - Give source authorization contention one saved 60-second deadline and a separate counter. Keep its waiting phase until current source authorization succeeds or the action terminates. It does not spend the eight delivery attempts. Duplicate clicks, worker restart, and background sweeps do not renew either limit. Reuse the existing work notifications and outbox deadlines. - Persist completion of the authorization wait under the current claim before delivery. A restart revalidates current authority. Generic delivery failure preserves the stored phase even when a commit acknowledgment is lost; it does not grant a wake or renew the saved deadline, counters, or claim. - Project pending preparation and prepared but unsettled actions as the existing scheduled-retry state with the actual outbox deadline. Keep the exact task, intent, agent, and source binding. Active execution keeps its normal working state. No UI or API enum change is needed. - Add regressions and document the admission boundary. ## Verification - RED: the original real HTTP route returned 500 under a held PostgreSQL identity lock. The restored-input case also returned 500 while its 60 sibling source and budget cases passed. The current endpoint-row case returned 409 without a saved intent under the bounded synchronous fallback. - RED: the worker failed a valid Retry during an actual nine-second PostgreSQL lock. Removing only the latest-run guard makes the new queued-run negative fail. Source preparation could also commit restored input after its deadline, and an already prepared action could process after a late lock release. The prepared but unsettled task view lost its scheduled state. - GREEN: all 91 selected real PostgreSQL integration cases passed. Coverage includes the 62-case saved-input matrix; coalescing; restart; one real heartbeat successor; source, actor, Stop, newer-work and publication refusals; a real nine-second lock; fixed-deadline exhaustion; the separate eight delivery attempts; certified restoration rollback on expiry; prepared-action early and late release; a real conversation-row wait that crosses the deadline during current authorization; endpoint-row contention; unknown source; an already-revoked actor; unchanged unrelated ingress errors; durable-work idle settlement; actual outbox presentation deadlines; and recovery after authorization commits, with Stop and a lost commit acknowledgment. - GREEN: all 75 cases in `task-retry.test.ts`, `task-interruption-presentation.test.ts`, `chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts` passed without a name filter. - RED → GREEN: stopping after source authorization committed left a stale waiting flag, so a fresh worker failed a previously authorized Retry after the deadline. The fix saves that phase in the same transaction. A real PostgreSQL commit followed by an injected lost acknowledgment also reproduced generic settlement restoring the stale flag. The corrected settlement preserves the stored phase. No wake is sent on the ambiguous response. Recovery queues once after current checks; Stop still prevents a successor. The deadline, contention count, original input, failed run, and publications stay unchanged. - Module boundary checks pass. Full workspace typecheck and the full build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this exact head, both required checks (`ci / verify` and `ci / e2e`) pass. The inventory has 53 successful, three skipped, and one neutral check, plus the automatic Paperclip Review in progress. The legacy Snyk status passes. Greptile is 5/5 on this head, with no new comments or unresolved threads. A current Paperclip assessment has not been published yet. - The full local `pnpm test:run` did not pass: its second chat shard hit a 15-second timeout in the HTTP/restart test after 348 and 293 passing cases. One bounded diagnostic of the same 642-case shard pair passed, with that case completing in 606 ms. The matching Linux CI pair passed too. The original timeout remains unexplained and nonreproduced. No timeout changed. The earlier stopped run was incomplete, and both conflicting documentation additions are retained. - CI chat 3/3 exposed a separate fixture assumption: restoration can commit before a durable wake receipt exists. One known pre-receipt delivery failure reproduces the same missing-run error. The corrected fixture drives the existing worker until the exact canonical intent and wake agree. It proves one real wake, one successor, exact source/actor binding, and one restoration. The exact failed shard pair now passes locally: 635 cases. Workspace typecheck also passes at test-only head `94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is unchanged from the previous verified build. That head completed all registered checks: 53 successful, two neutral, three skipped, plus the successful legacy status. Greptile scored 5/5. The subsequent Paperclip 4/5 persistence finding is addressed by the source repair above; its new exact-head review is pending. - Compatibility: 111 selected cases passed with #18895 head `9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The branch is now synchronized with master `4a83ba4050`, which includes that change. - Preserve the old CI failure: at head `37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially failed the empty organization-audience replacement versus member-removal assertion in `tool-access-service.test.ts`. The exact race family passed on unchanged base and head; all 501 file tests and two 30-case controls passed on base. The one Linux replay passed 119 files and 2,833 cases, with three skipped. No guard or assertion changed. The initial failure remains unexplained and nonreproduced; this PR does not claim to fix it. - The prior external Superagent scan could not start because its Security Graph returned HTTP 503. Its neutral check was not a completed security assessment. - These tests exercise the real Express route, PostgreSQL transactions, and heartbeat queue. External chat transport is a fixture. They do not run a live model or prove recovery of an original hosted task. ## Risks - A still-pending source authorization wait fails after its fixed 60-second contention deadline. A completed wait is saved, but new contention keeps the original deadline. A healthy lock can exceed that limit. Other delivery errors retain the eight-attempt budget. Current authority checks remain required before admission. Inspection does not reset either limit. - The new pending phase must never be treated as an authorized chat source. Normal authorization rejects it until the worker prepares and binds the exact source. - The task projection reads the existing linked outbox deadline. It adds a scoped join and does not create a timer or change a business deadline. - No schema migration, provider identity change, or new delivery queue is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, tool use, and test execution. The host did not expose an exact deployment snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required Paperclip CI gates are green (`ci / verify` and `ci / e2e`; automatic review progress is recorded above) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:48 -06:00
DottaandPaperclip	c5251b4c0e	fix(chat): retry fully recorded setup failures before dispatch (#18882) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected tasks retain their exact external request across failures. > - Task Retry must preserve that request and prove that old execution cannot continue. > - A workspace setup failure can settle before any provider dispatch. > - The chat retry guard accepted one specific validation failure but rejected other fully recorded setup failures. > - This pull request reuses the same strict server settlement proof for those failures. > - The benefit is one safe continuation through the existing queue, with all source and execution gates intact. ## Linked Issues or Issue Description Refs #18803, #18845, #18852, #18853 and #18860. These changes preserve workspace writers, bind completed preservation, or repair other preparation and native continuation paths. This change repairs the separate connected-chat guard for fully recorded legacy setup failures. Search also found #18876, which repairs issue scope during deferred wake promotion and does not repair this guard. **What happened?** A connected task failed during workspace reservation before provider dispatch. Its source remained authorized. Task Retry saved an intent and chat action, but the chat worker rejected that request as `failed_chat_retry_not_authorized`. No successor was created. The old failure had a bootstrap marker, a dispatch-only Runner profile, the matching server error and durable measured settlement events, and no runtime, process, output, result or environment lease. **Expected behavior** A deliberate board Retry can continue the exact saved input after the complete never-dispatched proof passes. An unfinished earlier writer, changed source, accepted result or unknown execution state must still prevent execution. **Steps to reproduce** 1. Admit an authorized request through a restricted chat endpoint. 2. Settle setup before provider dispatch, for example while an earlier workspace writer is unfinished. 3. Keep the exact server error and durable measurement events, a dispatch-only Runner profile, and no execution resources. 4. Press task Retry after the independent writer and ordinary admission gates permit work. 5. The original guard saves the Retry action but refuses to create a successor. **Paperclip version or commit** Reproduced on master at `a00109aa35`. The branch is rebased on current master. **Deployment mode** Hosted task diagnosis. Local proof uses real Express HTTP routes, chat authorization, heartbeat queue admission and disposable embedded PostgreSQL on macOS arm64, Node 26.4.0. Provider work is deliberately prevented by occupied fixture capacity. ## What Changed - Extend only canonical task Retry's existing strict pre-provider preparation proof to generic server-settled setup failures. - Keep exact result shape, dispatch profile, controller, runtime, process, output, event and resource checks. No diagnostic message allowlist is added. - Exercise GitHub and Telegram HTTP Retry with the original sender and saved input. Keep the failed run unchanged and create at most one queued successor. - Cover provider events, retained native state/results, modified source, unknown result metadata, live controllers, process identity, source leases and an independent unfinished writer. - Document the guarded contract and correct the obsolete access-map sentence. No new endpoint or agent permission is added. ## Verification - Red before repair: both setup causes failed through the real HTTP/chat/queue path because the durable Retry had no successor. An initial retained-native negative fixture had an invalid foreign-key setup; that fixture was corrected before the final run. - Green: 37 focused integration cases pass, including 22 new GitHub/Telegram cases and 15 existing pre-provider source and execution guards. - Current head `1bf69d20e02a049248ec378f087505563daeb9a0`: all 37 focused integration cases and full repository `pnpm -r typecheck` pass after the master rebase. The full `pnpm build` also passes. - Map structure, module boundaries and `git diff --check` pass. - Full local `pnpm test:run` is running; all six chat shards passed and the remaining server suites are progressing. This is not a full-suite pass claim. - Greptile and Paperclip Review both reviewed this exact head at 5/5 with no actionable findings or review threads. - Initial CI passed all jobs except the unchanged Runner package suite. Its `runtime-host.test.ts` approve-reads case failed before its intended assertions because a managed credential home could not acquire its loopback-port quorum. The exact case passes locally. A wider 121-case check hit the same lease error in a different Pi case (120 passed); all 121 pass with one worker. The exact competing port owner is unknown. One unchanged retry of only failed CI jobs passed, including the Runner package suite and aggregate `ci / verify`. All current-head required CI gates are green; the original failure remains in workflow history. - The dependency setup initially missed the embedded PostgreSQL package's dylib symlinks. Running its own postinstall helper fixed the local setup before assertions. This was not a product test failure. - This change has not been deployed. The original hosted task has not been replayed with the fix. ## Risks - Never-dispatched proof must remain strict. A bootstrap marker or empty process fields alone cannot authorize continuation. The regression cases retain contrary-state holds. - The inherited session display hint remains history and supplies no resume authority. - Current connection, principal, source integrity, pause, budget, approval, dependency and workspace-owner gates remain in their existing owners. - No schema, provider profile, trust policy, model or credential permission changes are required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL/browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:40 -06:00
DottaandPaperclip	d8c1b9d4b5	fix(runtime): reclaim failed remote checkpoint retries (#18877) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must preserve the original execution and provider state. > - A failed remote run can retain its provider PID after heartbeat releases the task execution lock. > - Normal resumption excludes that PID, while restart recovery rejects the released lock. > - Recovery must reclaim that exact execution only after current admission and durable ownership checks pass. > - Failed retries and admission holds need separate finite budgets. Neither can prove a provider stopped. ## Linked Issues or Issue Description **What happened?** A failed remote native run remains in retryable failure without a task execution owner. Normal resumption and restart recovery both reject it before inspecting the preserved remote provider. The current remote workspace reference format also misses the due-retry routing path. **Expected behavior** A due exact-checkpoint retry reclaims its original task execution only when current ownership and safety gates permit it. Newer task input prevents an old claim. Lost or ambiguous provider state still prevents replay or replacement. Ineligible recovery eventually becomes a quiet manual reconciliation obligation. **Steps to reproduce** 1. Start a remote native run and persist its provider checkpoint and encrypted Runner authority. 2. Fail with exact-checkpoint recovery and a retained remote PID. 3. Release the task execution lock at heartbeat completion. 4. Run ordinary resumption and restart recovery after the retry deadline. 5. Observe that both reject the same eligible run before remote inspection. **Paperclip version or commit** Reproduced from source at `d59791828482`; integrated with upstream `db9be4c5a6`. **Deployment mode** Remote workspaces with PostgreSQL. Local regressions use disposable PostgreSQL and encrypted checkpoint persistence. Related public repairs #18846, #18854, #18860 and #18841 cover finished per-turn owners, pre-provider failures, unstarted follow-ups and storage migration. They do not reclaim this failed run's released execution lock. Merged #18879 also adds scheduled-resume lock retention and reclaim. This PR keeps that compatibility and requires encrypted Runner authority, canonical source ordering and a charged failed retry before reclaim. Its scheduled-resume regression now uses a real encrypted authority and retains a snapshot-only refusal control. ## What Changed - Share normal admission checks and one guarded execution-binding restore/event helper between normal and remote resumption. - Keep agent-name normalization in a module with no imports. Preserve heartbeat's existing export. Native admission no longer loads heartbeat orchestration for a string key. - Recheck company, assignment, agent eligibility, Stop, pause, approval, dependencies, execution and checkout ownership before reclaim. - Reject newer task input using PostgreSQL creation-time and ID ordering, preserving timestamp precision. - Require the recorded remote target, encrypted Runner authority, exact checkpoint identity and existing controller fence. - Commit task binding, controller lease, run status and recovery event atomically. Recognize both remote workspace reference formats. - Charge failed reattachments against the existing three-provider-attempt budget, including retained-lock crash windows. Active controller adoption preserves its charged attempt. - Use the existing five-retry/30-minute system-recovery policy for admission holds. Ended tasks and newer sources exhaust it immediately. Preserve PID, checkpoint and task bindings, suppress resource settlement, and assign manual reconciliation with executor stop/state explicitly unverified. - Register recovery intent before deadline writes; preserve live/future deadlines quietly and roll back zero-row writes. Record changed hold reasons once and terminal dispositions separately. - Keep historical provider-budget exhaustion behind current admission/source/controller/authority checks. Make an existing orphan-cleanup fixture explicitly due. - Include exact plugin task leases that have no workspace reference. Reuse every reclaim and retry-budget guard, with a bounded company/task/run lease projection. - Share strict manual ownership proof across stale-lock cleanup, adoption, queue release and wake admission. Require termination receipts for every exact remote lease before normal cleanup can release either binding. - Preserve governing Stop through the existing audited cancellation path. Its acknowledgement does not release an unverified provider. Ordinary Retry can wait for exact Stop and file preservation, then queue one continuation. ## Verification - Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its [current CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858) is terminal and clean: 54 successful, three skipped and two neutral contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on this exact head with no open findings. No fresh Paperclip 5/5 assessment is claimed. - The [prior complete CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388) passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and Paperclip both 5/5 and no open threads. - All 233 tests across the six affected heartbeat, restart-isolation, remote-admission, scheduled-resume and session-binding families pass on production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and plugin task leases, competing controllers, exact rollback, Stop, assignment/company boundaries, attempts, work-intent ordering, finite backoff and quiet held state. - The merge preserves #18931's per-run audit-failure isolation and this PR's unchanged-transition hold. Two initial composition cases failed because their setup or expected result described the older contract. The rollback test now expects the exact per-run deferral while retaining full coordinator equality and no work intent. The routine fixture writes a real encrypted checkpoint through the normal owner before reaching the actual routine unique constraint. No production admission guard or rollback assertion was weakened. A separate agent reviewed this exact merge and fixture delta without finding a blocker. - Full recursive `pnpm -r typecheck` and `pnpm build` pass on production merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits change only three test files. Both the two-file `a5104fa` and final `5d7baf` server test-layer typechecks pass. - Four production-identical `a5104fa` probes pass with the rebuilt actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL checkpoints under Node 24.22.2. Workspace-reference and plugin-lease admission each cover surviving and lost providers. Replacement preserves one provider launch, one original side effect, the charged attempt and restored task execution binding. Lost providers reject snapshot recovery without replacement. This is protocol-boundary evidence, not a live Claude or hosted provisioning claim. - Initial `a5104fa` CI found one additional routine integration fixture missing the persisted `exact_checkpoint_resume` mode. Its encrypted checkpoint was already present. The exact case fails locally before repair and passes after recording that mode and expecting the charged provider attempt. Every exact routine conflict, rollback, checkpoint, task binding, successor-count and sibling-startup assertion remains. Final head changes only that fixture. The correct native-server group passes all nine files: 175 executed cases and 14 container-only skips, matching the now-successful native-server CI gate. - The earlier canonical HTTP Retry acceptance passed on the prior reviewed head with real PostgreSQL, the recovery worker and real files. Pending Stop protects both bindings and starts no successor. Exact termination and copyback queue one preserved-workspace continuation. No browser or live-model outcome is inferred from that test. - The predecessor's full local `pnpm test:run` failed in an unchanged Discord Gateway fixture with a 200 ms lease and a 422 initial ownership error. Focused predecessor and untouched-base controls pass; that full-run cause remains unproved. The required command was also started on production-identical `a5104fa`. All six local chat shards passed, covering 1,850 cases. The duplicate local invocation was interrupted with SIGINT while the general-server group was still running after this head's complete CI and the combined repair qualification had passed. The wrapper exited 1 on cancellation, no process retained its test log, and no full local pass is claimed. An earlier attempt to select the native lane through the CI wrapper instead selected local package sharding: 2,631 cases passed and 13 failed. Those separate failures are retained without causal attribution or unrelated source changes; that run is not counted as native-server qualification. The direct native group and current complete CI matrix pass. - Live Claude Opus 5.5 continuation, hosted plugin provisioning, deployment and recovery of historical tasks remain unverified. No original task was retried or mutated for this verification. ## Risks - Restoring a task binding grants remote inspection; it does not prove that the provider survived. - Unknown effects, lost providers and missing semantic results retain their existing reconciliation requirements. - Exhausted admission holds preserve execution bindings and resource evidence for manual reconciliation. They do not authorize a fresh executor or provider/resource release. - The source guard is conservative: newer task input prevents recovery of an older failed source. - No database migration or provider profile change is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and shell execution. The runtime does not expose the exact model snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:07 -06:00
DottaandPaperclip	c39e076638	fix(runner): retain the cause of closed ACPX sessions (#18872) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Runner records provider failures so the task can show their cause. > - An ACPX control request can fail and close its session before the event pump runs. > - The next poll then reports only that the session is closed. > - A second cleanup error can also drop a valid startup cause. > - This pull request retains each failure at its existing reporting boundary. > - Operators can inspect the cause while the failed session stays closed. ## Linked Issues or Issue Description Related: Refs #18622, #18847, #18854. Those changes preserve provider causes and handle proven startup failures. This change covers a control failure after provider execution starts. **What happened?** A rejected sidecar control closes the ACPX session. A later poll replaces its error with `ACPX provider session is closed`. The native terminal loses the control classification. Separately, cleanup can discard an existing validated startup cause. **Expected behavior** Retain the first control classification in the failed terminal. Do not invent a provider explanation for that code. Keep a validated startup cause when cleanup also fails. Keep the session closed, uncertain tool outcomes pending, and the existing recovery checks effective. **Steps to reproduce** 1. Start an ACPX turn and receive an authorized tool call. 2. Return its result after the provider has retired that call. The sidecar rejects `tool.resolve`. 3. Poll the native backend. Before this change, the terminal reports only the closed-session guard. 4. For the separate startup case, combine a rejected `session.open` cause with a cleanup error. Before this change, the combined error drops its validated provider cause. **Paperclip version or commit** The two subprocess regressions and the cleanup-composition regression fail against unchanged production code at `683fa4725e`. **Deployment mode** Native Runner with ACPX. The process regressions use the real Rust subprocess transport and durable native backend. Their controlled stale-tool rejection matches the production sidecar response shape. The separate cleanup unit injects a cleanup error at the existing error-composition boundary. ## What Changed - Retain the first failure in the ACPX session and return it from later closed-state checks. - Keep the validated provider cause when cleanup also fails. - Match the stale-tool fixture to production. It carries the closed code and no invented provider cause. - Cover a rejected tool result, normal shutdown, raw sidecar-text exclusion, pending tool preservation, refusal of another turn, and terminal replay after controller restart. - Document the existing failure and recovery boundary. ## Verification - Red: both process regressions fail on the base code. Polling replaces the named control failure with the generic closed-session error. The cleanup unit also fails because the combined error loses its existing startup cause. - Green: all 12 ACPX session tests, all 36 native backend tests, and all four session unit tests pass. The stale-tool cases require the classification and absence of an invented provider cause. - Commands: `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_provider_session` and the same command with `--test native_provider_backend`. - Map structure and whitespace checks pass. - At head `981faa4`, the full Runner core Rust suite passes: 847 tests passed and three existing tests are ignored. Workspace typecheck and build pass. The existing TypeScript provider-failure and durable-settlement boundary suites pass all 30 tests. - An earlier concurrent local Rust attempt missed an existing Codex diagnostic deadline. That case passes alone on baseline and head, and the final full suite passes with the local socket support its tests require. - Complete current-head CI passes: 55 successful checks, five intentional skips and two neutral checks; no failed or pending checks. Both `ci / verify` and `ci / e2e` pass. - Failed CI attempts were investigated before rerunning affected jobs. Logs showed runner shutdowns/lost communication, an unchanged Windows WebSocket authentication deadline, and an unchanged Cursor Cloud Stop-fixture assertion. The Cursor Cloud case passes on baseline and head. No test assertion or CI guard was disabled. - The local full `pnpm test:run` invocation passed all six chat partitions, covering 1,838 expanded cases. It was stopped as a duplicate while still in general-server after complete sharded CI passed. A completed local full Vitest run is not claimed. - Greptile and Paperclip reviewed the corrected head at 5/5 with no actionable findings. There are no unresolved review threads. - The process test exercises control rejection, process cleanup, durable terminal persistence, and controller restart. The provider boundary is a deterministic sidecar fixture. It is not a live Claude or hosted recovery test. - A genuine provider refusal or uncertain external effect remains unresolved. This change preserves its evidence and does not make it safe to replay. ## Risks - The first control classification becomes visible where a generic closed-session error appeared before. Raw sidecar error text stays excluded. Startup details use the existing validated and redacted `ProviderFailure` contract. - A stored failure must not grant execution authority. The failed session still rejects calls and new turns. Normal shutdown has no fabricated failure cause. - No provider profile, digest, retry budget, process ownership rule, migration, credential, or permission change is required. ## Model Used OpenAI Codex, based on GPT-6, with source inspection, code editing, command execution and regression tests. The exact runtime snapshot and context-window capacity are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and available capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the bug template fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the local Rust, typecheck, build and boundary checks; complete sharded CI passes and the partial local Vitest result is reported above - [x] I have added behavioral regression tests - [x] I have updated the relevant execution and run-log documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:17:58 -06:00
DottaandPaperclip	2518ac85ab	fix(recovery): continue verified failed native preparation (#18908) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry keeps a durable request while the system stops old work and saves its files. > - Native preparation can fail after it allocates a workspace, but before it starts a provider. > - Completed preservation can resolve the action while the original Retry still waits. > - The old classifier rejected every allocated failed preparation. The source check also returned before its canonical Retry fallback. > - This pull request accepts only complete server proof and moves the existing exact-source fallback before that return. > - The same Retry can then start one normal continuation. Current access and unknown execution still hold work. > - Saving files and admitting that continuation must use the same parent-row lock order; the composed head includes the supporting preservation fix. ## Linked Issues or Issue Description **What happened?** Task Retry accepted a failed native preparation and saved its request. The workspace had complete preservation and stop receipts. The request still waited for execution recovery and created no continuation. **Expected behavior** The same deliberate Retry must continue after the system verifies all saved files and source leases. It must preserve current task, actor, repository, Stop and execution checks. **Steps to reproduce** 1. Record a native-adapter claim that fails during unresolved legacy preparation, before provider execution. 2. Finish real workspace preservation and exact lease cleanup. 3. Submit normal HTTP task Retry and reconcile its canonical request. 4. On the base commit, no run is linked to that request. With this change, one continuation is linked to it. **Paperclip version or commit** Reproduced on master `d7283e875772ca8c684dc86e471e6b48d543b7b1`. **Deployment mode** Source checkout with real local PostgreSQL and filesystem archives. Object storage, sandbox transport and the native provider backend are controlled in the final execution test. This is not a live model or hosted object-storage claim. **Prerequisite: #18918** at `4c1d7769e6cfa9536a67c4e3b536cef155f19966`. This head includes that parent through normal history-preserving merges while retaining `master` as its target. The parent owns remote workspace save, prepare and provision lock order, legacy import reservation and publication, their concurrency regressions, and the database lock-order documentation. Merge the parent before this PR; no deployment is implied. Related: Refs #18892 for interrupted preparation. Refs #18882 for the separate connected-chat setup path. Refs #18898 for already-admitted cleanup waits. This change does not depend on #18882 or #18898. ## What Changed - Accept allocated failed preparation only with the server error and durable settlement records, no execution/session/checkpoint/output/result/coordinator evidence, all exact remote source leases stopped, and verified saved workspace proof. - Recognize only the host's closed accounting scope and typed pre-provider unsupported-harness warning. Unknown accounting, provider metrics and arbitrary diagnostics remain holds. - Check the canonical failed-preparation source before an empty resolved-action list can return. Reuse the normal locked continuation checks. - Add HTTP, preservation-worker and native execution coverage, including source failures produced by real heartbeat workspace preparation. Keep one canonical request and one provider turn. Preserve the original failure and real archive files. - Restore storage, environment and experimental settings even if native test setup or owned async cleanup fails. - Cover changed Stop, source, actor, controller, provider evidence, snapshot generation, writer and lease receipts. Keep current unavailable-catalog denial before provider startup. - Share one bounded host-settlement receipt reader between failed-native preparation and existing-directory readiness. Keep each caller's accounting, profile and additional warning restrictions. - Include parent #18918's company/task/run-before-workspace ordering for save, prepare and provision. Keep legacy import publication out of reverse native-owner waits with exact NOWAIT refusal. Retain foreign keys, ownership checks, uploaded work and transaction-aware notifications. - Document the failed-preparation continuation and parent lock-order contracts. ## Verification - Causal base comparison: the normal HTTP/preservation regression fails on the canonical request's missing successor. Both existing interrupted-preparation controls pass; the same cases pass with the classifier/fallback repair. - Actual heartbeat-produced catalog failures reproduce RED before the host-accounting fix, both with and without the real typed unsupported-harness warning. No source failure records are patched in those cases. - The native execution case keeps `paperclip_runner`, native runtime selection and the requested Codex model. A controlled native session reads real restored `saved.txt`, writes `continued.txt`, emits an accepted result and finishes through the real finalizer once. Only the external native backend is controlled. - Current unavailable-catalog admission and suspended authenticated membership remain denials before provider work. Extra provider/session/accounting material, Stop, newer source, writer/generation and lease-receipt mismatches remain held. - Both settlement consumers pass 53 focused real PostgreSQL/filesystem/native-boundary cases on Node24.22.2. Existing-directory warning presentation/provenance and durable-measurement restrictions are preserved. - Parent #18918 proves real PostgreSQL lock cycles through actual save, prepare, provision, reservation and import publication owners. Its native status peer test proves a busy refusal without a deadlock. Immutable task binding, writer/epoch fences, same-journal retry and unexpected-error rollback controls retain uploaded objects and original files. The original spontaneous peer identity is not claimed. - Historical head `6204f565` passed all 54 CI checks, but its local affected run passed only 572/573 with a real PostgreSQL preservation deadlock and its Paperclip review was 3/5. Those results remain historical evidence; this composed head includes the explicit parent repair. - Production-composed head `93a02d4cf74f4d8d2cc6c1b62884bc58280500bf`, Node24.22.2 / pnpm9.15.4: full `pnpm build` and recursive `pnpm -r typecheck` passed. The six affected recovery, continuation, Retry, migration and concurrency files passed 724 tests. Opt-in remote workspace integration passed 65/65 and ownership lifecycle passed 16/16, with no skips. - Final head `65dcd6c6a601448b5a436b7a7f2c84b8cfde761b` adds only the parent's unsupported-host test guard. Its 18 concurrency cases and generated server test-layer typecheck passed on the exact head. Production bytes match the qualified head above. - Exact-head CI passed on `65dcd6c6`: 54 success, 3 skipped, 1 neutral; no failed or pending jobs. Greptile reviewed this exact commit at 5/5 with no actionable issue and zero open threads. - The current-head Paperclip assessment is registered and queued with no run at the latest observation. It has not delivered a current-head score. The earlier 3/5 assessment and its resolved findings remain history. No extra review request or hosted task replay was used to force delivery. ## Risks - The new allocated-source proof is conservative. Extra or missing execution material keeps work held and needs inspection. - This changes admission after completed preservation. It does not certify unknown provider effects or restore repository access. - Administrative import can now return a named busy hold while an execution row is owned. It retains the archive and journal for a later administrative retry. - There is no schema migration or new API. Stop, task ownership, current actor access, budget and approval checks retain their normal owners. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime model ID and context window are not exposed in this agent session. The agent used reasoning, tool calls, code edits, real local PostgreSQL and filesystem tests. A separate Codex agent reviewed the admission and execution boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:17:49 -06:00
DottaandPaperclip	de9f263454	fix(workspaces): prevent preservation lock cycles (#18918) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces save files before a later task turn can use them. > - Admission, preparation and preservation share task, run and workspace rows. > - Snapshot inserts also lock their foreign-key parents. > - Different acquisition orders could form a PostgreSQL deadlock and delay saved-file recovery. > - This pull request orders those runtime parents and refuses busy legacy publication owners. > - Files and ownership evidence remain available without replaying provider work. ## Linked Issues or Issue Description **What happened?** Snapshot reservation and publication could hold a workspace while waiting for a task or run. Other workspace owners could hold that parent while waiting for the workspace. Review also found reverse preparation, provisioning and legacy-import paths. Real PostgreSQL owner-pair tests reproduce `40P01`. **Expected behavior** Concurrent owners must complete or retain their normal ownership refusal. A lock cycle must not interrupt file preservation. **Steps to reproduce** 1. Prepare a remote workspace with retained files in disposable PostgreSQL. 2. Schedule actual save and reserve, preserve, prepare or provision functions on opposing rows. 3. Release the scheduling barrier. 4. Unfixed source deadlocks. Fixed source saves the files or returns its normal ownership hold. The tests also schedule an actual legacy import beside an arriving reservation and a stale native status commit. **Paperclip version or commit** The original three save regressions fail on unchanged master `ce7766dc11c4515190d48d395f26a90bba900f8a`. Reverse preparation, provisioning and imported no-op controls fail on published `2d13b3f`. The additional import-publication and native-status owner pairs also have causal PostgreSQL RED/GREEN proof. **Deployment mode** Source checkout with PostgreSQL and real filesystem commands. Snapshot encryption and reload use local object storage. No hosted storage, live model or original-task recovery claim. Related: Refs #18908, whose preservation acceptance exposed this separate fault. This repair has no code dependency on that PR. Refs #17846 for separate legacy terminalization locks. Refs #18746 for the documented admission order. Repository search found no existing repair for this owner graph. ## What Changed - Lock company, current task and run before the runtime workspace. Recheck the run binding and pin save-operation attribution. - Apply that order to reserve, prepare, provision and both save transactions. New shared-workspace provisioning receipts use the current task. - Lock import source parents before reservation and publication. Recheck the source binding and retain the publication fingerprint. - Use full-strength `NOWAIT` locks for legacy coordinator, run and allocation rows. Only PostgreSQL `55P03` becomes `remote_workspace_migration_blocked:execution_rows_busy`. The publication transaction rolls back. Other errors propagate. - Preserve writer, epoch, snapshot, foreign-key, Stop and owner checks. Keep command, capture and upload I/O outside row-lock transactions. - Add actual-owner concurrency, unknown-effect, eventual retry and rollback controls. Document each owning transaction and its foreign-key edges. ## Verification - Node 24.22.2 and pnpm 9.15.4: all 107 save-concurrency, migration-concurrency and migration tests pass. - All 81 enabled disposable-PostgreSQL integration and lifecycle controls pass on the final head. Both opt-in database variables point to a dedicated local test database; no skipped case is counted as qualification. - Actual preparation/provision/import no-op peers reproduce three deadlocks on the prior source. The fixed owners preserve and reload files. The real provisioning command runs once; a lost response remains an unknown effect and is never replayed. - Actual legacy publication versus reserve and stale native status commit also reproduce deadlocks before repair. Busy run/lease controls preserve the journal, archive and files, then finish from the same journal after release. - A real publication constraint failure rolls back earlier coordinator, profile and snapshot writes. Its original database error propagates. - Immutable same-company and foreign-task guards, writer/epoch loss and uploaded-object retention pass. Four real shared/inherited reuse cases fail old receipt attribution and pass current-task attribution. - The original spontaneous issue-foreign-key failure did not retain its peer statement. These controlled owner pairs prove supported defects; they do not identify that historical peer. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-save-concurrency.test.ts server/src/services/remote-workspace-migration-concurrency.test.ts server/src/services/remote-workspace-migration.test.ts`. Enable the separate lifecycle suites only with their documented disposable test database variables. - Full recursive typecheck and `pnpm build` pass on production commit `dacf6278abd3257c407a297c6db8872e76d85f61`, using Node 24.22.2 and pnpm 9.15.4. Current head `4c1d7769e6cfa9536a67c4e3b536cef155f19966` changes only the canonical PostgreSQL host-support guards. All 18 changed concurrency cases executed and passed with zero skips; their server test-layer typecheck passes. All 57 current-head check contexts are terminal and clean: 53 passed, two skipped and two neutral. Required `ci / verify` and `ci / e2e` pass. Greptile and Paperclip both reviewed exact current head at 5/5 with no actionable findings. There are no open review threads. - Composition with #18908 includes this exact parent history. Its production-composed head passed 724 affected tests, all 81 enabled integration/lifecycle controls, recursive typecheck and build on Node 24. Its final test-only guard head passed the 18 changed concurrency cases and their test-layer typecheck. This is local server/PostgreSQL/native-backend-boundary evidence, not a hosted model or storage claim. - The required broad local Vitest attempt stopped at an unchanged chat publication-batch test with `socket hang up`. The earlier CI workspace shard timed out in the unchanged CLI chunked-ZIP test. Both failures remain recorded; neither is reported as a pass or hidden by timeout changes. The chat case passes one bounded focused head/base control, so the original full-suite socket failure was not reproduced by that control. ## Risks - Short transactions take parent-row locks earlier. Competing writers can wait for those transactions. - Administrative import now refuses a busy native owner. It retains its journal and archive for a later explicit retry. - Native owners have different lock orders. The import's nonblocking requests avoid a new global native lock policy. - Missing bindings and uncertain command effects remain holds. The repair does not infer termination, release an executor or replay external work. - No schema, foreign-key or timeout change. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this agent session. The agent used reasoning, terminal tools, code edits, PostgreSQL and filesystem tests. A separate Codex agent reviewed the production ownership graph. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:17:40 -06:00
Dotta	55d7df2bad	fix(runtime): bound checkpoint data and preserve durable failed sessions (#18959) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner checkpoints preserve controller state across server replacements. > - Enrollment imports legacy controller state into those durable checkpoints. > - The importer passed extra source fields into each encrypted record, duplicating the whole source and exhausting memory. > - Some failed runs already had durable checkpoints but no session snapshot. Re-importing them incorrectly required resumable state. > - This PR stores only each record's binding and contents, and preserves matching failed checkpoints without changing their recovery status. ## Linked Issues or Issue Description Refs #18957 and #18943. **What happened?** A 538-record migrated checkpoint stored 1,108,851,510 bytes of encrypted material. Every record included the full 1.4 MB legacy source. Large writes disconnected and later reads exhausted memory. **Expected behavior** Each encrypted record contains only its company, issue, run, part and contents. **Steps to reproduce** Run offline controller migration with a retained controller that has many commands. Decrypt a saved envelope and inspect its fields. The old writer also includes the entire offline migration input. **Paperclip version or commit** Reproduced on `a1f3637c96dfadaf3a15ae9c83aa8227f4fed20e`. ## What Changed - Serialize only company, issue, run, part and contents in encrypted checkpoint envelopes. - Preserve an already-durable failed checkpoint when its run, runner and session match and the legacy session snapshot is absent. Keep its failure, recovery blockers, owner and pointers unchanged. - Report the number of failed checkpoints preserved during maintenance. - Test exact encrypted contents, idempotent preservation after removing the old home, and rejection of successful or foreign-session sources. - Document the migration findings and verification boundaries. ## Verification - The oversized-envelope regression failed before the serializer fix. - The already-durable failed-source regression failed before the preservation fix. - Four focused test files: 44 tests passed. The migration suite passed all 5 tests again after adding the foreign-session control. - Server typecheck passed. Greptile reviewed `c4f02a064c` at 5/5 with no findings. The exact-head PR workflow passed, including typecheck, build and all required tests. - Live staging repair preserved all 16 affected checkpoint sets: 2,982 records, 6,400,144,676 bytes reduced to 15,419,528 bytes. Each record matched its legacy source and passed re-encryption verification. Original encrypted rows and source files remain on the retained volume. - Full hosted enrollment and active-agent deployment proof remain pending. Deployment will follow the normal staging release pipeline. ## Risks The serializer affects normal and migrated checkpoint writes. The decoder already accepts the same five fields and continues to accept older envelopes. Preservation applies only to a failed run with an exact, validated durable checkpoint and no session snapshot; it grants no new recovery authority. This change does not rewrite existing database records automatically. Existing failures and missing provider files remain failures until their owning recovery path resolves them. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-04 07:14:58 -06:00
github-actions[bot]andlockfile-bot	81b01bcc16	chore(lockfile): refresh pnpm-lock.yaml (#18961) ... ## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>	2026-11-04 13:10:22 +00:00
DottaandPaperclip	5996f25122	fix(ui): replace selected text on ordinary paste (#18796) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People edit task documents and comments in the Markdown editor. > - Its link validator accepts safe text that is not a URL. > - Lexical uses that validator to turn pasted text into a link around the selection. > - Pasting JSON or prose can keep the old text and put the new text in its link destination. > - This pull request handles plain selected-text replacement in the existing paste plugin. > - The benefit is standard text replacement with existing manual link controls intact. ## Linked Issues or Issue Description Fixes #2930. Related: #8506 by `joegalbert-ai` identifies the same validator cause. Thanks for that analysis. This change adds current Lexical command coverage and preserves the separate manual and decorator link paths. The contributor PR remains unchanged. Reproduce by selecting an existing paragraph and pasting `{"status":"ready"}`. The old code keeps the previous paragraph as a link label and uses the JSON as its destination. The expected result is the pasted JSON as text. ## What Changed - Replace a selected text range, including soft line breaks, when the clipboard text is not an unambiguous link target. - Keep selected-label link wrapping for valid scheme, network, web and email targets, including international addresses. Treat ambiguous relative paths and fragments as ordinary pasted text. - Leave the manual link validator unchanged. Keep manual relative, internal, fragment, email and decorator links, with existing executable-scheme guards. - Use the same native Lexical rich-paste insertion API for selected HTML payloads. Preserve HTML formatting and the paste undo boundary. Keep files, Markdown, code and other selection types with their existing handlers. - Declare the clipboard API and test rich-text API at the existing Lexical 0.48.0 version. The scheduled repository bot owns the lockfile refresh; this PR has no lockfile delta. - Add real Lexical paste and link-command regressions through the production editor policy and paste plugin. This restores expected editing behavior; it adds no setting, endpoint or new feature-map entry. ## Verification - Before the fix, real selected-text paste tests fail for JSON, prose and multiline text. They retain the old label and create a link with the replacement as its URL. The valid-URL control passes. - A test-only experiment with the exact recognition predicate from #8506 fails four current decorator relative/fragment cases. The chosen paste-handler fix keeps the manual validator intact. The experiment was removed after preserving its result. - `pnpm --filter @paperclipai/ui exec vitest run src/components/MarkdownEditor.test.tsx src/lib/markdownPaste.test.ts src/lib/normalize-markdown.test.ts src/lib/mention-aware-link-node.test.ts src/lib/mention-deletion.test.ts`: 119 editor-focused tests passed. After the baseline update, the same editor suites plus App and ShellConfigurationGate passed 146 tests. - The React MDXEditor shell is mocked. The configured production predicate, actual paste-normalization plugin, native rich-text paste handler and real Lexical paste/link commands are not mocked. Tests cover URL wrapping, plain replacement, manual/decorator links, executable-scheme handling and paste-owner handoff. - UI typecheck and production build pass. UI token gates, module boundaries and diff checks pass. Initial test event-type errors were fixed. Greptile found that JSON with an email still formed a link. Its real-command regression failed before narrowing whole-address recognition and passes after the fix. An isolated frozen install and SDK build resolved local workspace prerequisite errors. - No live task edit or deployed replay was run. Full repository checks run in current-head CI. The initial quality gate rejected generated lockfile edits; the policy correction removes them without changing tested source. The earlier current-head workspaces-a CI failure was the independently fixed invitation-access fixture in App.test.tsx. Merge #18799 supplied that baseline fixture fix. Its local regression and the editor suites now pass together; all final-head CI checks now pass. Final head `5156085b83c908e1439927bbebf25ab0aa42f81b` has 60 terminal contexts: 56 success, three skipped, and the neutral Paperclip review at 5/5. Greptile is 5/5 with no unresolved findings. - Review regressions also failed before repair for a selected soft line break, formatted HTML with JSON, and international email addresses. These cases now pass. The HTML test checks that bold formatting survives. ## Risks - URL recognition must not turn ordinary text into a link. Ambiguous relative paths use text replacement on paste; the manual link controls still accept them. - The selected HTML path must preserve formatting. It delegates insertion to the native rich-paste API with the original clipboard. File payloads and non-simple selection types still yield. - Existing Markdown normalization still applies. Script/data scheme checks and rendered-link sanitization remain unchanged. - No visual tokens, schema, public API, provider or deployment configuration changes are required. No additional user documentation is needed for this expected editing behavior. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:09:37 -06:00
DottaandPaperclip	e54f9fbf82	fix(secrets): expose secure human entry in native discovery (#18927) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Some tasks need a company credential or the current person's credential. > - Paperclip already has secure human entry forms for these values. > - Native connection guidance did not distinguish storing a credential from connecting an external service. > - Native API search also omitted the existing secure entry route from secret operation help. > - This change makes that route discoverable while preserving human and owner authority. ## Linked Issues or Issue Description Refs #18780. Related #18921 corrects a separate observer false positive. Neither PR owns this production guidance change. **What happened?** A request to store an existing company or personal credential could lead to an unnecessary external provider question. Native API search did not explain the existing human entry surface. **Expected behavior** An agent can find the existing secure form, direct the human to the matching entry, and verify the result through its authorized interfaces. The agent must not ask for values in chat or act as the human. **Steps to reproduce** Search the native API for a company credential or personal publishing credential. The result should name the supported Secrets settings route and its human authority boundary. Open a known personal definition link and confirm that the existing Set my value action appears. ## What Changed - Distinguish Paperclip credential storage from external connection discovery in shared native guidance. - Describe the existing Secrets settings route, secure entry actions and current-person boundary in OpenAPI help. Qualify definition detail links as admin-only; other members use My secrets and Set my value. - Exercise discovery through the actual OpenCode MCP bridge and native API catalog. - Retain agent denials for personal definition and value routes. Exercise the real Secrets UI definition link. ## Verification - Focused local checks: four suites, 78 tests passed. These cover native discovery, secret-route permissions, shared guidance and the Secrets UI. - The new discovery test fails against unchanged master OpenAPI help. It passes with this change. - Token gates and diff checks passed. - Full workspace typecheck passed. The first attempt found missing SDK links after script-free installation; the existing root postinstall helper restored them. An unprivileged retry also hit the local tsx IPC sandbox restriction. The corrected owned setup passed without source changes. - Full local workspace build passed. The full local test run stopped in chat shard 2: one Discord receiver-lease test failed during initial configuration. That shard passed 263 tests; the first shard passed 310. The test and owning lease/configuration sources are unchanged from master. One focused isolation passed without code changes. Its cause remains unknown; this is not a complete local test pass. No unrelated Discord code was changed. [Normal hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41773266687) passed on exact head `0abb8b722779368109b9639f68122d7df02a8ab1`. - Fresh Greptile review reports 5/5 on the exact current head; all review threads are resolved. Paperclip Review remains in progress. The optional Superagent scan is neutral/inconclusive: Security Graph context retrieval returned HTTP 503 and assessment did not start. It supplies no security PASS credit. - A retained representative native attempt with this guidance produced the correct definition-scoped secure-entry link. Its observer then rejected an explicit warning against chat entry. That failed result remains unchanged. Human entry, saving the value and worker delivery were not exercised. This PR claims no complete product-eval pass. ## Risks This changes agent guidance and search ranking. It grants no new permission and changes no secret resolution or owner checks. Broader service-connection discovery still applies when a task needs a connection. The secure human forms are unchanged. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with repository tools, code execution and test analysis. The host does not expose the context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — the 78 owning controls pass; the broad local failure remains disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — normal CI passed; optional review states are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:01:57 -06:00
DottaandPaperclip	5b50c6b32c	fix(teams): return saved resource links after catalog installation (#18923) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Catalog installation creates agents, projects, and routines in one company. > - The install result includes saved IDs but no links to those resources. > - A chat agent can describe a partial installation without giving the operator a way to inspect it. > - This change returns canonical links for saved resources and documents them for API discovery. > - The operator can use the returned links to inspect the actual installation. ## Linked Issues or Issue Description **What happened?** A catalog install can create the intended resources and report skill warnings, while its response leaves callers to construct board URLs from IDs. A native chat attempt described the resources and warned about an incomplete skill, but omitted resource links in both its initial and repeat replies. **Expected behavior** The installation response should provide canonical company-scoped links for its saved agents, projects, and routines. It must preserve warnings and omit links for rows without saved IDs. **Steps to reproduce** Install a catalog team through the API. Inspect its saved resource rows and response. Before this change, the response has IDs and names but no resource links. **Paperclip version or commit** Parent `e7549ab613` from master. Related #16495 changes comment authorization metadata in the same OpenAPI file; it does not address team installation responses. No duplicate team-link PR was found. ## What Changed - Add typed `resourceLinks` entries with resource kind, saved ID, name, and canonical company-scoped href. - Read the company prefix with one bounded primary-key query before import. Encode URL segments and use saved IDs, including IDs returned for reused resources. - Preserve the portability import result, skill preparations, warnings, permissions, and collision behavior. - Document the response and add service, native API wire, OpenAPI, and real database regression coverage. ## Verification - Parent service regression: 1 failed, 28 passed. The new failure was the missing resource-link response. - Owning service/native wire, route authorization, and OpenAPI suites: 72 passed. - Real PostgreSQL catalog-install suite: 4 passed. Returned links match persisted agent IDs and the company prefix. - Shared and server TypeScript checks passed. Fresh shared/SDK builds and the owning Runner TypeScript prerequisite passed. - UI typecheck, Storybook catalog check, 30 catalog/map controls, and token gates passed. - Actual CI found an omitted required `resourceLinks` field in the existing Storybook install fixture; that fixture is now synchronized. The earlier response-status review correction is also retained: install schema and wire control use actual HTTP 201. Final head `0bca72cf1f7641fd842a04c5a75cae4bcc274f05` passed all 48 normal CI jobs, including `ci / verify`: [run 41773413940](https://github.com/paperclipai/paperclip/actions/runs/41773413940). Greptile is 5/5 with zero unresolved review threads. The optional Paperclip review remains queued and its comment says work is paused. The Superagent security scan is neutral; neither supplies completed assessment credit. - `git diff --check` passed. - Setup failures were retained: incorrect test working directory, sandbox listener denial, missing built SDK/Runner declarations, and native library aliases not hydrated from the package directory. Corrected setup used the owning build and shipped hydration script. No assertions or deadlines were relaxed. - Repository-wide tests/build were not run locally; the full normal CI suite passed. A new native chat attempt has not run for this change. The earlier chat attempt remains failed for missing links. Local response coverage does not prove that a model will include the links in its next reply. ## Risks - This is an additive JSON response field. Manually constructed TypeScript install results must include the new typed field. - The query adds one company-prefix lookup per installation. It selects one scalar by company primary key and runs before mutation. - Supplying links reduces URL construction work but does not guarantee model use. Native qualification is still required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing, and tool execution. The host did not expose the model context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All required normal CI gates are green; optional incomplete/neutral assessments are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:01:08 -06:00
DottaandPaperclip	d12de135a7	fix(runtime): preserve legacy identities and bound checkpoint imports (#18957) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers must preserve native Runner sessions before removing local storage. > - Legacy session imports validate the workspace and saved provider authority. > - A protocol lease can retain a logical workspace ID while copy-back uses a separate physical sandbox lease ID. > - Comparing these different IDs blocks valid legacy sessions during enrollment. > - This pull request checks the workspace binding, retains the full saved protocol identity check, and bounds checkpoint inserts inside the owner transaction. > - Existing sessions can migrate without changing their identity or discarding files. ## Linked Issues or Issue Description Refs #18943 and #18841. **What happened?** The first full staging enrollment stopped with `native_controller_migration_artifact_binding`. A successful native session had valid controller and provider-pack identities, but its physical sandbox lease had a different ID. **Expected behavior** Import the original session when its workspace and saved provider authority match. Reject foreign workspaces and protocol identities. **Steps to reproduce** Create a retained session whose sync record points to the execution workspace and a separate physical lease. Run the offline controller migration. The prior comparison rejects this valid record. **Paperclip version or commit** Reproduced on `84aa989bf9d08fc005ac273c37f9e1d860a4db25`. **Deployment mode** Docker with external PostgreSQL and shared object storage. ## What Changed - Compare the sync record's workspace ID with the frozen execution workspace. - Keep the full saved provider-pack check for company, agent, workspace, session, Runner, protocol lease, remote path, and provider digest. - Use distinct physical and protocol IDs in the database-backed migration fixture. Reject a foreign workspace and a changed protocol lease before any checkpoint import. - Limit migration inserts to 64 records and about 512 KiB of serialized rows per statement. Send a larger individual record alone. Keep all statements in the same transaction. - Verify that a conflict in a later batch rolls back earlier batches, then import and decrypt the complete checkpoint. - Record both live failures and the comparison evidence in the preservation plan. ## Verification - The updated fixture failed on the original comparison and passes with this fix. - Targeted migration, controller recovery, and provider authority tests: 39 passed. - Full repository typecheck and the latest server typecheck passed locally. The complete PR workflow passed on `a1f3637c96` (run 41827558125), including build and test gates. The Docker Compose setup download failed once before the test step; its infrastructure retry passed. A separate full local test/build run is still in progress. - A read-only staging preflight inspected 807 selected legacy sessions. It found zero mismatches under the corrected workspace and full saved-provider checks. - A single 538-record, 2.2 MB checkpoint insert repeatedly closed the live database connection on pooled and direct endpoints. A forced-rollback comparison imported the identical checkpoint with batches of 64. An unbatched control failed again after cache warm-up. The exact server-side disconnect cause remains unknown; all failed receipts are retained. - The original failed enrollment and volume are retained. The corrected importer committed 32 additional sessions and 2,982 checkpoint records. Full import and deploy handoff remain pending while memory use is investigated. ## Risks The import checks durable session authority. A weaker identity check could import the wrong session. This change retains the saved provider authority validation and adds an explicit foreign-protocol rejection test. Batching adds statements for large imports, but a late failure still rolls back the complete owner transaction. It does not rewrite session identity, change provider lifecycle, or detach any volume. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 06:36:31 -06:00
DottaandPaperclip	84aa989bf9	Preserve native sessions through legacy workspace enrollment (#18943) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 04:43:56 -06:00
DottaandPaperclip	cf51936587	fix(ui): open focused orphaned document annotation threads (#18950) ... ## Thinking Path > - Paperclip manages tasks and their saved documents. > - Review threads keep their comments when a document revision removes their selection. > - A supported document/thread link opens the annotation panel and selects the thread. > - The panel then removed every orphaned thread, including that selected target. > - This change keeps the exact focused orphan visible so its history can be read. > - Other unselected orphaned threads keep their existing display policy. ## Linked Issues or Issue Description **What happened?** A direct link to a resolved document annotation opened the panel but omitted its thread after the selection became orphaned. The server still retained the complete review comments. **Expected behavior** The selected thread and complete comments remain readable through the existing deep link. Viewing the history must not reopen, reply to or reanchor the thread. **Steps to reproduce** Create and resolve a review thread. Save a revision that removes its selected text. Open its supported document/thread link in a fresh view. The original panel shows other threads but filters out the requested one. Related #14648 fixes duplicate annotation positions. It does not repair this focused-thread filter. ## What Changed - Allow the exact focused orphaned thread through the existing panel filter. - Recompute that list when focus changes. Preserve the display policy for unrelated orphans. - Add a rendered hash-to-wrapper-to-panel regression with a long first comment, a later resolution note, focus changes and a fresh remount/query client. ## Verification - Untouched-production rendered regression failed because the focused target card was missing:1 failed,18 excluded by selection. - Five related annotation/hash/consumer files passed49/49 tests on this worktree's own locked graph. - UI typecheck, UI build, token gates, module boundaries and whitespace passed. - The rendered regression preserves resolved/orphaned state and confirms no create, reply or status mutation. It reads the full first comment after a fresh remount and hides the old orphan after focus moves away. - API results, Markdown rendering and selection overlay use the existing component-test fixtures. This is a rendered consumer proof, not a full browser/server rebase journey. - Initial dependency/build-prerequisite failures are retained locally. Normal install/shared/SDK preparation resolved them; no unrelated source change was made. - Repository-wide tests/typecheck/build beyond the affected UI were not repeated locally for this two-line presentation fix. Current-head required CI and substantive review remain pending. No map entry or command changes are needed. This restores the existing document-link/read-history contract without adding syntax, access or reattachment behavior. ## Risks A focused orphan now exposes its existing comment card and controls. The existing server authorization and mutation rules still apply. This change does not move the anchor, change resolution state, reveal an unselected orphan, or fetch a thread outside the already-authorized list. No style values or dependency changes are added. ## Model Used OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis, implementation and testing. The exact context-window size is not exposed. Tool use and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will require all Paperclip CI gates to be green before merge (currently pending) - [x] I will require Greptile 5/5 with no unresolved findings before merge (currently pending) - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 02:39:50 -06:00
DottaandPaperclip	612d9931a6	Support per-turn Runner sessions on ephemeral web servers (#18932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The e9aae9cf revision passed all 57 successful checks, with 5 skips and 2 neutral checks, and exact-head Greptile 5/5. The latest 9cbec30a4f follow-up changes only E2E catalog/admission controls; its 291 affected tests pass, all 57 check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile is 5/5 with no open review threads. - Migration numbering and safety checks pass. - A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload bytes across 89,100 archive entries after deleting the source directory. Restored SHA-256 matched; upload/readback took 56.5 seconds, restore 74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources were removed. - Archives preserve contained relative links, reject link chains/traversal, and enforce independent byte/entry/object limits. Offline verification streams hashes. Orphaned deleted-run backups stay on the original volume. Obsolete object retention preserves the selected backup, live lease references, and pending uploads with owners. - Isolated API-key Codex deployment: original active runner/session transferred to the replacement and committed. All 567 reads, 57 writes, and 57 readbacks succeeded. A second replacement completed a same-session continuation with all 287 reads and 29 writes succeeding. The strict cell failed two agent-authored shell commands; this result is retained as failed. - Isolated API-key Claude deployment: original active runner/session transferred and committed. All 584 reads, 59 writes, and 59 readbacks succeeded. A second replacement completed a same-provider-session continuation, with 211 reads and 22 writes succeeding. The strict cell failed an initial completion call that omitted a registered attachment citation; the agent corrected it. This result is retained as failed. - Isolated API-key OpenCode deployment: the same active runner/session transferred and committed. All 481 reads, 49 writes, and 49 readbacks succeeded. A second replacement retained the provider session and completed useful work, with 156 reads and 16 writes succeeding. The strict cell failed agent tool calls: a denied external-directory write, malformed artifact hashes, and a second completion after the first result was committed. There was one submitted/completed turn and one committed result; reconnect notification replay was identical. This result is retained as failed. - Grok's real first turn now completes with settled usage and provider cost ($0.41397); the next turn is admitted normally. That attempt then exposed a shared workspace-path identity regression before deployment. Three cold-restore cases reproduced it before the fix; 106 workspace/transport tests and four focused cold-restore cases pass afterward, including destroyed allocation and occupied-directory preservation. Server typecheck and build pass. At b5b0dbb03f both controller switches passed the read/write probe without failure, and the active runner retained its process while the controller generation advanced. The third turn exposed an invalid reasoning-token subset check; ad07e681ba fixes it with the actual receipt regression. Two later live attempts failed with an upstream Grok rate limit before deployment. A tiny preflight prompt succeeded before the second attempt, but the full workload was still rate-limited. The final live gate remains incomplete. No accounting or identity guard was bypassed. Earlier failed attempts and their exact resource-cleanup receipts remain recorded. - Earlier Codex/Claude/OpenCode receipts predate #18922; they establish their tested revisions. Current-head live qualification remains pending; #18943 separately preserves pre-enrollment logical/provider workspace addresses and adds actual legacy migration proof. Its first Codex migration attempt stopped safely at skill-version verification. - Earlier attempts remain recorded: missing local provider-pack setup blocked one Claude launch, and the observer rejected identical reconnect replay in another. Setup and the observer were corrected before the latest run. - Named staging enrollment has not started. Its serving revision and post-deployment journeys remain separate delivery gates. - Broader E2E harness validation exposed deployment-catalog and installed-lane coverage drift; the affected 291 controls now pass. Twelve other failures reproduce on unchanged base 1a39dea9aa. They concern cleanup call expectations, dated operating-overhead fixtures, a Pi profile-version expectation and workflow-security expectations. No baseline failure is reported as passing. ## Risks - Provider archives can contain credentials. Objects use authenticated encryption with scoped associated data and a wrapped data key. The original stable master key remains required. - Storage failure prevents publication and sandbox cleanup. Operators must keep shared storage available and preserve the previous volume during enrollment. - Legacy state without verified identity cannot become executable recovery authority. Archival preserves evidence and existing failures; it does not recover already-lost provider files or clear blocked tasks. - Local overlap probes establish the tested application behavior. They do not establish Railway routing behavior or live managed-subscription qualification. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and live integration tests. The exact runtime variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.7	2026-11-04 02:12:51 -06:00
Devin FoleyandPaperclip	1a39dea9aa	fix(heartbeat): keep a routine claim queued while its slot is held (#18933) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A routine creates one task per tick. Only one open task of a routine can hold an execution lock (`issues_open_routine_execution_uq`). > - Since #18879, a failed native run that waits for a scheduled resume keeps its task's lock. Routine dispatch counts only queued, running or scheduled runs as live, so it can queue a newer task of the same routine meanwhile. > - When that newer run is claimed, stamping its execution lock violates the unique index. The raw database error is not a deferrable claim rejection, so it escapes the queue loop. The queue code notes that this can stop the agent's queue and startup recovery. > - This pull request turns that exact violation into a 409 claim rejection, so the run stays queued until the slot is free. > - The benefit is that a busy routine slot delays one run instead of stopping the queue or startup. ## Linked Issues or Issue Description Refs #18879 (keeps a failed run's lock while its resume is pending), #18916 (defers the same unique-index conflict in restart recovery's lock reclaim) and #18931 (open; keeps one run's restart-recovery failure from stopping startup recovery). This is the same index conflict on a different path: the heartbeat queue claim, not restart recovery. It does not change #18916 or #18931. Its doc paragraph is separate from the one #18931 edits. **What happened?** While auditing the lock paths from #18879 for this index, I found that a queued run of a newer routine task can be claimed while an older task of the same routine still holds the slot through a failed run with a pending resume. The claim's lock stamp then raises `duplicate key value violates unique constraint "issues_open_routine_execution_uq"`. `isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error escapes `startNextQueuedRunForAgent`. **Expected behavior** The claim is rejected as a conflict and rolls back. The run stays queued and is claimed after the older task releases the slot. Other errors behave as before. **Steps to reproduce** 1. Create two open tasks of the same routine (same origin and fingerprint). 2. Give the older task an execution lock held by a failed run. 3. Queue a run for the newer task and claim it. On `master` the claim throws the unique violation. The new queue test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `0dedbf23c35`. **Deployment mode** Any server with routines whose tasks run on native runners. ## What Changed - `bindClaimedIssueExecution` in `server/src/services/heartbeat/queue.ts`: if stamping the execution lock violates `issues_open_routine_execution_uq`, it throws `conflict("Another execution for this routine is already in progress")`. The claim transaction rolls back, and the queue loop keeps the run queued as it does for other 4xx claim rejections. Every other error is rethrown unchanged. The issue checkout route already maps this violation to the same 409. - `doc/execution-semantics.md`: one paragraph on the behavior. ## Verification - Failing first: `queue.test.ts` "keeps a claim queued while another task of its routine holds the execution slot". On `master` the claim raised the `issues_open_routine_execution_uq` violation. Now it rejects with 409, the run stays `queued`, neither task's lock changes, and nothing executes. - Failing first, review round 1: `heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run queued while another task holds the routine slot, then runs it once the slot is free" goes through `resumeQueuedRuns`. On `master` it rejected with the unique violation. Now the routine run stays queued with no error code while another queued run for the same agent executes. After the slot is released, the routine run executes exactly once (executions are counted per run, because the finished task can queue its own follow-up run). - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed. - `npx vitest run src/services/heartbeat/recovery.test.ts src/__tests__/routines*.test.ts`: 4 files, 205 passed. - `pnpm --filter @paperclipai/server typecheck`: pass. - Not verified: the queue loop end to end on a real instance. The loop's handling of a 4xx claim rejection is existing behavior. ## Risks - While the older task holds the slot, the newer run stays queued and the loop logs "queued heartbeat run claim was rejected" on each pass. It is claimed once the older task's resume finishes or its lock is released. - Only this one constraint is mapped. Other database errors still escape as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:33:00 -08:00
DottaandPaperclip	b4c69ded9f	test(e2e): attribute native OpenCode skill loads (#18929) ... ## Thinking Path > - Paperclip lets people manage agents and assign saved company skills. > - Daily QA must prove that an agent loads the saved procedure it uses. > - The saved-skill journey accepts several native source-read paths. > - OpenCode's native proxy changes its built-in skill loader's normalized operation to unknown. > - The grader expected execute and rejected a real completed skill load. > - This PR accepts that one observed native shape with strict source and identity checks. > - Genuine loads can qualify while missing, foreign or incomplete evidence still fails. ## Linked Issues or Issue Description Refs: #18783. **What happened?** A combined QA run on `0884e73b46ee064b3d242e467035dde6c32e5cb5` failed `saved-skill-native-read-attributed`. The agent's successful native skill load exposed the complete saved procedure. The exact immutable assignment and source checks passed. Core recorded the built-in loader as `operation: unknown` and `readOnly: false`. The grader accepted this loader only with `operation: execute`. **Expected behavior** A complete native OpenCode skill load can qualify when its actual run, session, immutable source and paired receipt all match. A generic unknown operation cannot qualify. **Steps to reproduce** Run the saved-skill journey through Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` with OpenCode. Let the agent use its native `skill` loader. The direct TypeScript converter emits execute, but the real proxy forwards a `builtinToolCall` without that operation. Core's native converter records unknown. The former TypeScript-only fixture missed this boundary. ## What Changed - Add one closed native OpenCode skill-loader branch. Require the actual run, runner, native session, item, turn and paired call. - Bind the API assignment, admitted native skill, immutable version map and materialized runtime directory to the same canonical skill. - Require the exact wrapper name, heading and complete saved procedure. Require an absolute local base directory and an empty file inventory. Reject traversal, URI encodings, backslashes, query or fragment markers and all ASCII controls. Keep valid spaces. - Require a successful, complete and untruncated native receipt with the exact output byte count. Reject missing, duplicate, reordered, foreign, failed, dynamic or MCP carriers. - Add 65 loader and boundary controls. Drive the real TypeScript-to-proxy input and use the observed Core native carrier shape. The JavaScript test does not claim to run the Rust converter. - Document the native loader path. Preserve every existing accepted execute, filesystem, first-party and Codex parser path byte-for-byte. Keep all seven checks, one-run limit, 500-cent guard, task prompt, model and ordinary Beta-off Latest assignment unchanged. ## Verification - The sealed original Core receipt reproduces RED: the former grader returns zero reads. Changing only its normalized operation to execute reaches the old accepted branch. The narrow correction returns one attributed read in separate local analysis. The original QA grade remains failed. - The actual TypeScript, proxy and native normalization source regions match between Core091 and the tested baseline0884. The proxy emits `builtinToolCall` without an operation. The native converter supplies unknown. - 334 focused tests pass across nine files, including 65 new loader controls and all existing saved-skill evidence, flow, contract and catalog controls. Run `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/product-journeys/saved-skill-use-native-loader.test.ts` for the narrow regression. - TAB, ESC and DEL controls fail before the review hardening and pass after it. The valid-space positive remains green. - Canonical Runner TypeScript prerequisites, own server declarations and E2E types pass. The final E2E-only check reuses those same-commit declarations after only the two test-scope files changed. The checked candidate blobs match this committed head. - Credential-free discovery lists 25 catalog journeys, including saved-skill use. Registration and selection do not change. - `git diff --check` passes. No Rust build or product-model call ran locally. Full CI [41773295359](https://github.com/paperclipai/paperclip/actions/runs/41773295359) passed on `fb537b598da40176fa1752b49a4f676f468de1f7`: 48 successful jobs and one skip. Fresh Greptile is 5/5 with zero unresolved threads. The earlier same-head canceled run remains retained; no retry was requested. The optional native review ended neutral with a 5/5 source assessment and explicit execution/qualification coverage limits. Fresh staging qualification [41777787338](https://pages.paperclip.ing/evals-qa-smoke-41777787338-1/) passed on this exact head: all seven checks and all five phases, exactly one native OpenCode run, independent saved-document reopen/source preservation, and normal cleanup complete. TestOps recorded a new passed result and the automatic Slack report succeeded. - The failed original stopped before independent document, reload and source-after checks. Those checks remain unqualified for that run. The original failed grade remains unchanged. A separate genuine fresh QA pass now verifies the complete journey before daily re-admission. ## Risks This path depends on OpenCode's observed native wrapper and materialized directory contract. Missing or changed evidence fails closed. The fix adds no runtime service, schema or API behavior. The native-loader branch is covered by strict local regression controls and the original sealed Core receipt; a separate fresh staging run also passed the complete saved-skill journey. ## Model Used OpenAI GPT-6 through Codex with reasoning, tool use and code execution. The runtime does not expose a more specific deployment ID or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.6	2026-11-03 23:21:17 -06:00
DottaandPaperclip	652cd76026	fix(ui): expose Stop for failed native retries (#18899) ... ## Thinking Path > - Paperclip manages agents and their tasks. > - A native run can fail while it still owns a same-run retry. > - The server supports Stop for that pending execution. > - Run details show Cancel only for queued or running records. > - A failed pending retry therefore has no ordinary cleanup control. > - This pull request exposes the existing Stop eligibility and uses the existing cancellation endpoint. ## Linked Issues or Issue Description Refs #14743 and #14733 for correlated native Stop. Those changes cover backend cancellation and protocol qualification. They do not add the failed-run control to run details. Searches of current master and open PRs found no matching UI repair. **What happened?** A native run can show failed while its coordinator still owns a retryable failure with no accepted result. The backend can cancel that retry. Both run-detail implementations hide the cancellation button because the record is terminal. **Expected behavior** The selected run offers Stop when the server confirms that pending native retry. Unknown, legacy, terminal and accepted-result states do not gain this control. Existing queued and running cancellation stays available. **Steps to reproduce** Render either run-detail implementation for a failed native run with an eligible retry. The original UI has no Stop button. The original batched execution projection has no eligibility field. The regression tests exercise both real components and the actual PostgreSQL projection query. **Paperclip version or commit** The missing control and projection field were reproduced on 44a4ca1c61. Fresh master 8fd917aedf has identical affected production blobs before this repair. **Deployment mode** Board run details for a native Runner task. Local verification uses rendered React components and PostgreSQL. No hosted task mutation or provider retry was performed. ## What Changed - Add optional `execution.canStop` to the existing batched projection. Read the matching retryable coordinator and absence of an accepted result. The cancellation endpoint remains the authority owner. Add no database queries. - Exclude Grok failed retries from the new hint and both consumers. The existing Grok Stop endpoint cannot accept a failed retry before assignment. Keep this limitation explicit and preserve its active-work authority checks. - Show Stop for eligible failed runs in both run-detail consumers. Keep the existing cancellation endpoint and its company, actor and row-lock checks. - Use the existing request-correlation contract for failed-native Stop. Persist one scoped identity before sending and recover it after remount or reload. Read a matching canonical claim only for the same board actor, company and run. Use the existing UUID fallback on ordinary HTTP. Never retry Stop automatically. Refresh the selected run and history after either outcome. - Report a stale conflict or a true no-op200 response. Accept a matching durable pending cancellation intent without requiring terminal acknowledgement. Keep running, queued and Grok calls unchanged. - Add actual rendered interaction tests, real batched-query controls and a short execution contract. Keep maps unchanged for this bug fix. ## Verification - Untouched UI and real Postgres producer reds cover missing Stop and missing eligibility. A valid Grok review found an unsupported failed-run action; actual broker, projection and rendered controls reproduce it before exclusion. No backend authority was expanded. - The original60-case focused suite passed, including26 rendered cases, the batched producer, and actual Grok broker/assignment fences. The related57-case scope passed before the Grok correction; it is not presented as final-head proof. - A fresh native4/5 review found a stale200 no-op. Both actual rendered consumers failed that new assertion. The real authenticated HTTP/Postgres fixture proves `canStop=true`, coordinator revocation, uncorrelated unchanged200 with a cancellation activity, and existing correlated409 with no new activity or run mutation. - The correlated repair passed82 affected cases before master reconciliation. Exact `74c60c9ffe11c05bc85f3d0fa5a136f7e9d445cd` passed80 rendered/API, HTTP and native-correlation cases. The render scope verifies request bodies, stable identity after an unknown outcome, no-op200 error, pending intent acceptance, wrong-run/request denial and existing active/Grok behavior. - Subsequent current-head reviews found remount identity loss and unavailable `crypto.randomUUID` on ordinary HTTP. The corrected helper passed 52 rendered cases across both consumers, including an unknown response, initial scoped storage persistence, a prior browser record with a unique never-mounted run identity, exact ID reuse, canonical claim recovery, actor switch, wrong run and HTTP UUID fallback. A real local-board identity mismatch was reproduced with two additional reds before using the actual server-projected actor ID. - Real authenticated HTTP and existing native-correlation suites passed 45 cases for canonical claim recovery and actor fences. Production UI, server and vendored Runner builds and all server/UI type layers passed at the prior reviewed helper head; exact final `e0f723b1c86f1529029bf141943323764de28c22` UI types, token gates and production build passed. - Fixture failures are retained separately: a missing local SDK build dependency, and a missing coordinator FK teardown in the new HTTP fixture. Neither is a behavior red. Both fixtures were corrected and the affected tests passed. - Previous-head3d CI required one diagnostic rerun after a Runner shutdown and a real account-list migration assertion. Two untouched157-test runs did not reproduce the assertion. Complete original row equality is unavailable. The assertion remains unresolved; no test was sorted, skipped or changed to green CI. - Previous-headf12 CI lost eight child runners. Its one authorized failed-job retry also lost e2e5 to an explicit shutdown. No second retry was issued. Those failures remain recorded. Current-head CI and both substantive reviews are required; older checks and grades are not credited. - Previous-head74c CI found a real monitor scheduler timeout. An untouched local replay and diagnostic-only replay reproduced it: the original manual-monitor run was still running at the unchanged three-second deadline. The cause remains unresolved; temporary diagnostics were removed, and no assertion, timeout or production monitor policy was changed. No CI retry was issued. - Exact local `906af5c942ee0963eb4b64552160b57a18bcd2d1` integrates normal master after #18930 merged. All reviewed Stop production and test blobs remain unchanged from `e0f723b1c86f1529029bf141943323764de28c22`. The four rendered, projection, HTTP and correlation files passed 99 cases after integration. All UI/server type layers, token gates and production UI/server/Runner builds passed at that exact head; the server build stamp records it. - Previous-head e0 CI reported a Discord ownership error. That original failure is preserved. Separate #18930 proves and repairs receiver construction/renewal races with the same error; the original CI scheduling order remains unproved. This branch now contains that ordinary master merge and requires fresh CI rather than rerunning the old failed attempt. - Full repository checks were not run locally for this narrow slice. No hosted task, provider execution or Stop receipt was created by this work. ## Risks Grok failed retries remain unsupported by this UI change. The existing Grok running and queued Stop action, revoke path and assignment fences are unchanged. A valid initial review caught the overbroad hint; actual broker, projection and rendered regressions reproduce it before correction. No backend cancellation authority was widened. The optional hint can become stale. It grants no permission. The existing cancellation path rechecks current authority and can reject the request. Run details show that rejection. Older servers omit the hint, so failed runs keep their existing behavior until the server updates. No retry, task hold, accepted-result decision or provider receipt is changed. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 23:18:08 -06:00
Devin FoleyandPaperclip	6c7b136b16	fix(native): keep startup recovery running when one run's claim fails (#18931) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - At startup, the server runs native restart recovery before it reports ready. Recovery claims each interrupted or retryable native run in its own transaction. > - On one instance, one run's claim hit the routine execution unique index. The error escaped restart recovery, startup recovery failed closed, health stayed "still starting", and two deploys rolled back. > - #18916 fixed that exact conflict: it defers the run when its claim hits `issues_open_routine_execution_uq`. Any other error in one run's claim still aborts the whole pass, and so does a failed per-run audit write. > - This pull request handles every per-run claim failure the way #18916 handles the routine conflict, and keeps a failed audit write from stopping committed claims. > - The benefit is that one bad row cannot keep the server from becoming ready, and no run is claimed or changed when its claim fails. ## Linked Issues or Issue Description Refs #18916. This change generalizes its catch: #18916 keeps the routine-conflict disposition (`routine_execution_busy`) exactly as it is, and this change adds a separate reason for any other per-run failure. It does not change how the routine conflict is detected or deferred. Refs #18879, whose lock reclaim exposed the startup failure. Refs #18877 (open), which edits other parts of the same files. This change touches only the claim's error path and the audit loop, so a second merge needs at most a small rebase. **What happened?** Two deploys never became healthy. About 25 seconds after the listener bound, the boot log showed `startup native runner restart recovery failed closed` and `startup heartbeat recovery failed`, both from a failed `update "issues" set "execution_run_id" = ...` that violated `issues_open_routine_execution_uq`. #18916 now defers that violation. Any other error in one run's claim (another constraint, a trigger, a timeout) still escapes `claimNativeRestartRecoveries` and fails startup the same way. #18916's own test asserts this: "Other database failures still fail the recovery pass." **Expected behavior** One run's recovery failure is logged and recorded on that run, and the rest of startup recovery continues. The server becomes ready. Only a failure of the candidate scan itself fails the pass. **Steps to reproduce** 1. Create a native run that is due for restart recovery, and make its claim fail (the new test uses a trigger on `native_run_finalizations`). 2. Add a healthy run that is due for restart recovery. 3. Run restart recovery for both. On `master` the batch throws and the healthy run is not claimed. **Paperclip version or commit** `master` at `4a83ba4050f`. **Deployment mode** Any server that runs startup native restart recovery. ## What Changed - `claimNativeRestartRecoveries`: #18916's `.catch` on each candidate's claim transaction no longer rethrows other errors. The whole claim has rolled back, so the run's state is unchanged and nothing is claimed. The run gets `awaiting_evidence` with the same 30-second retry deadline. The reason stays `routine_execution_busy` for the routine slot conflict, and is the new `restart_recovery_failed` for any other error, which is logged at error level. The batch continues. - `recoverNativeRunsAfterRestart`: records a `restart_recovery_failed` deferral as a warning transition on the run ("Native restart recovery failed for this run and will retry"). A failed `getRun` or transition write is logged and no longer stops the dispatch of claims that already committed. - `doc/execution-semantics.md`: replaces "Other database failures still fail the recovery pass" with the per-run rule. ## Verification - Failing first: `native-restart-recovery-isolation.test.ts` (new, PostgreSQL). It seeds the reported shape (two open tasks of one routine: the older with a due scheduled resume and a cleared lock, the newer holding the lock), a run whose claim fails through a trigger, and a healthy run. On `master` the batch threw `injected recovery claim failure`. Now the routine run is deferred with `routine_execution_busy` and the failing run with `restart_recovery_failed`, and both keep their coordinator, attempt and locks. The healthy run is claimed. `recoverNativeRunsAfterRestart` completes and records an info transition for the routine run and a warning transition for the failing run. - Two existing tests expected the pass to reject on one run's failure: #18916's unrelated-constraint test, and failed-workspace retention's rollback test. They now expect the per-run deferral. Their rollback checks (no claim, unchanged coordinator, lease, lock and recovery actions) are unchanged, and the unrelated failure still does not become a routine wait. - `cd server && npx vitest run` over restart recovery (isolation, integration, diskless, preparation), controller recovery, heartbeat recovery, scheduled resume lock, failed-workspace recovery, startup cleanup, native resumption, live recovery work, startup feedback export, recovery deadlines, runtime selection, stop accounting, warm transition restart, workspace reconstruction, remote workspace migration and heartbeat process recovery: all pass except one. `heartbeat-process-recovery` "redacts opaque environment-bound credentials from Sentry diagnostics: result" fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: two `heartbeat/recovery.test.ts` cases make `getRun` and then `appendRunEvent` fail for a deferred run while a committed claim is in the same batch. Recovery completes and dispatches the claim exactly once; the `appendRunEvent` case also checks the warning transition. Both fail when the catches are removed. In one of four full runs of that file, the unrelated orphan-cleanup case "reconstructs missing native preservation during orphan cleanup without replay: reuse_by_environment" failed; it passed alone and in the other three full runs. - After the rebase onto `4a83ba4050f`: the isolation and failed-workspace recovery tests and `pnpm --filter @paperclipai/server typecheck` pass. - Not verified: a deploy to a real instance. ## Risks - A persistent per-run error now retries every 30 seconds and logs an error each time, instead of stopping startup. The run stays unclaimed, so ownership does not change. - If the database fails for every candidate's claim, startup recovery now completes with every run deferred, instead of failing closed. The candidate scan still fails closed, and the deferred runs retry on their deadlines. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:05:27 -08:00
DottaandPaperclip	0dedbf23c3	fix(chat): retain receiver ownership during runtime construction (#18930) ... ## Thinking Path > - Paperclip manages agents and their work. > - Chat receivers hold a durable lease before they accept messages. > - Runtime construction can outlast a lease renewal interval. > - Renewal previously required an attached runtime before construction had returned. > - That check could revoke the same receiver that was still starting. > - This change uses the exact pending initialization as proof during construction. > - The existing durable token and endpoint checks still fence every renewal. ## Linked Issues or Issue Description Related: #18899 exposed the existing receiver failure in its chat CI shard. This is a separate lifecycle repair. #18795 provides the relevant production and sleep fixture context. Searches found no overlapping open initialization repair. **What happened?** The receiver acquired its lease and started its renewal timer. The service then awaited credentials, command preparation and runtime construction before attaching the runtime to its context. A renewal during that interval treated the missing attachment as lost ownership. Configuration later failed with a Gateway ownership error, although its original durable token was still current. **Expected behavior** The exact pending initialization can retain its own lease while construction runs. Missing attachment alone must grant no ownership. A changed token, paused or removed endpoint, changed generation or credentials, and shutdown must still fence the receiver. **Steps to reproduce** 1. Use the real configure and PostgreSQL ownership producers with a 60-second lease and 50-millisecond renewal. 2. Hold the actual runtime construction call for 200 milliseconds. 3. Release construction. Untouched code returns the Gateway ownership error. 4. With the same construction delay and a five-second renewal control, configuration succeeds. **Paperclip version or commit** The causal regression fails on unchanged production source at `e7549ab613`. The original CI error was observed in the chat shard of #18899. That CI run did not record the precise setup and renewal order, so this local proof does not establish its historical scheduling. ## What Changed - Store the runtime context identity in the existing pending initialization entry. - Allow pre-attachment renewal only for that exact tracked initialization and current local ownership. - Recheck local ownership inside the existing renewal transaction. Preserve company, endpoint, generation, credential and original-token checks. - Before SDK listener initialization, wait for any older renewal, recheck local ownership, then obtain fresh durable proof. Preserve the existing checks after initialization. - Keep attached runtime identity checks, lease deadlines, renewal intervals, shutdown and listener retirement unchanged. - Add actual configure and database race controls. Document the shared Discord and Photon receiver contract. ## Verification - Untouched production red: the actual construction gate crosses a renewal and returns the same Gateway ownership error. The otherwise identical slower-renewal control passes. No lease, exception or provider outcome is injected in these positive cases. - A subsequent review found a second real race: final runtime removal did not prove that SDK initialization had never started. Four actual initialization-hook controls failed before the pre-initialize fence. The final matrix asserts zero hook calls for lost or changed authority, including a foreign-token control with no periodic renewal due. An attempted cross-company endpoint move is rejected by the existing database foreign key; the valid original initialization still succeeds. - A third actual-producer red holds completion of a real committed renewal, changes the durable token, and proves that joining that old promise could incorrectly accept configuration. The corrected startup check waits for that renewal and then obtains fresh proof; it cannot credit the earlier result alone. - Final relevant full Discord registration group, takeover/callback cases, buffered host-pause recovery, failed construction, credential-fenced pause and shared Photon consumers: 34 passed across two files; 1,840 other cases were deselected. Four executed cases cover Photon transport recovery, failed authentication, duplicate allocation ownership and checkpoint fencing. The tests use the real configure/database producers with fake provider transports and observe SDK initialization; they do not prove physical network behavior. - All server type layers, the module-boundary gate, and production server/vendored Runner build pass at exact final `38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp records this exact commit. - Earlier fixture mistakes are preserved separately: a test initially changed every lease key, including the credential lease, and another attempted an invalid cross-company row move. Neither is counted as a production behavior red. The corrected controls target the Gateway key and assert the existing company constraint. - An initial isolated dependency graph resolved an older shared telemetry build. Correct workspace links restored the all-layer typecheck without changing telemetry source. - Full repository checks were not run locally for this narrow slice. Fresh current-head CI and substantive reviews remain required. No hosted task, provider work, fleet action or deployment was performed. ## Risks The ownership helper is shared by Discord and Photon. The selected tests cover both consumers. The pending entry grants only permission to renew the original token; it grants no message, callback or publication authority and adopts no listener. Missing or changed initialization identity fails closed. There is no schema change, historical backfill, new worker or additional idle scan. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection, source editing, shell execution and test tools. The host did not report the context-window size. A separate coordinator reviews the source before delivery. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:57:39 -06:00
DottaandPaperclip	c6fe9b143c	test(evals): pin reviewed chat repairs and restore UI test setup (#18921) ... ## Thinking Path > - Paperclip agents must describe recorded work and respect human review decisions. > - The catalog must test the user's request without adding an unrequested format rule. > - Timeline case 466 wrongly required date and timezone tokens in every valid answer. > - Missing private evidence and stalled review admission also need bounded failure facts. > - The reviewed changes correct that contract and add diagnostics without changing product behavior. > - A new retained secure-entry reply also exposed an observer false positive for an explicitly excluded chat destination. > - A retained team-install reply also omitted every resource link, so the reviewed service now returns canonical hrefs. > - This PR pins the coherent corrections and fixes the observed UI-test interruption helper. ## Linked Issues or Issue Description Refs #18914. Refs #18923. Refs #18927. Refs #18780. This prepares the next diagnostic selection. It does not change the three-case run already in progress. The prior case-466 failure remains a failure because its answer omitted the required handoff. The prior case-53 original grade remains unknown. ## What Changed - Pin immutable candidate `25cd5e4ebb6b955701778d2d1d702f86a573cfaa`. - Update the existing descriptor snapshot and matching timeline oracle declaration to v3. Only case 466's expected outcome is clarified: respect a timezone when explicitly requested. - Update case 487's existing source-file declaration to include its diagnostic helper. Its oracle, request, run counts, and clocks stay unchanged. - The candidate adds fixed carrier/reason counts for case-53 evidence failures. It also records bounded, issue-bound delivery/wake/run facts only when case 487's existing settled check fails, then rethrows that original error. - Correct the shared chat-credential matcher for secure-form instructions that explicitly exclude chat. Mixed or later chat requests still fail. Give case 580 an explicit personal-secret v2 oracle identity and update its existing snapshot and manifest declaration. - Include the reviewed additive team-install resourceLinks response, its shared type and actual 201 OpenAPI schema. Preserve the full existing partial-install/repeat help and synchronize the existing typed Storybook result. The core PR remains unmerged. - Fix the UI test announcement handler that timed out during the two-tab attachment check. Only a detached target with no current replacement is ignored; real remaining-target errors still fail. The original functional assertions and deadlines remain. This changes no catalog case 550 behavior. - Keep all 244 case identities, original requests, model, deadlines, private scanner limits, and broader runtime qualification holds unchanged. ## Verification - All 47 existing TestOps contract, adapter, and manifest controls passed on this pin. - Integrated timeline and private-diagnostic controls passed: 162 timeline checks, 14 evidence checks, and full 242-declaration admission with zero fixture callbacks. - The source author ran 53 real PostgreSQL/review controls and the declaration-backed runner typecheck on the exact accepted review diagnostic. Those passed. - The integrated replay passed 179 tests. Its three PostgreSQL observer cases initially could not start because the local native package lacked a shared library. After the existing same-version runtime was resolved through owned dependency links, those three real PostgreSQL cases passed. The setup failures are retained; no source, test, timeout, scanner, or assertion was weakened. - The retained secure-entry consumer replay accepts the truthful secure-form/not-chat handoff with the correction. All 25 shared consumer checks and the fresh full 242-declaration admission passed (28 tests total). The earlier failed result is unchanged. This does not yet prove human entry or worker delivery. - A cheap prefreeze check on actual master passed the existing reporting-lock guard with unchanged reviewed lock bytes. - The resource-link type changes exactly one of 506 SDK inputs. The existing owning helper freshly compiled shared and plugin SDK outputs. Measured SDK runtime JavaScript is unchanged; the type declaration, its map and two build receipts changed. The emitted resource-link consumer compiles and a foreign resource kind fails with TS2322. The old registry/default proof rejected the change; the existing generator updated only that input and the new default proof passed. All 51 SDK and 20 input/build custody controls passed. Compiler, parser, Zod and canary certificate pins are unchanged; held runtime cases remain held. - Core service/wire/OpenAPI controls passed (72 tests), plus four real PostgreSQL checks. The typed story synchronization passed UI typecheck, catalog checks and 30 controls. Candidate token gates passed. - The prior CI two-tab check failed in the announcement locator handler. A controlled real-browser disappearance reproduces it before the fix and passes after it; a genuine remaining-target error still propagates. That prior failed check is retained. Fresh normal CI and review run on this actual correction. - The review-requested real detached-ElementHandle control now covers the catch branch and asserts the target is absent. Removing that predicate fails with the real browser detachment error; the restored three controls pass. This is test coverage only; helper behavior and deadlines are unchanged. - Fresh exact-head normal [PR CI](https://github.com/paperclipai/paperclip/actions/runs/41774814211) and [Windows Runner](https://github.com/paperclipai/paperclip/actions/runs/41774814064) passed. Greptile and the completed Paperclip review both report 5/5 with no actionable feedback; all review threads are resolved. The optional security scan is neutral, not a security assessment PASS. No new catalog runtime PASS is claimed for this candidate. ## Risks The timeline contract correction accepts valid answers without extra date labels. It still rejects explicit dates outside the requested window, missing attribution, failed-run misstatements, hidden work, and unsupported claims. Balanced owning controls retain those negatives. The diagnostics add evidence rather than changing delivery behavior. Case 53 can still fail closed at the existing archive limit. Case 487 can still fail its original settled check. The additive resource links are a product affordance and still need a real agent rerun. Earlier failed grades and attempts remain unchanged. The full 244 runtime catalog is not merged into master; its existing 196/244 gate remains. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, code integration, and local validation. The host does not expose the context-window limit. The model under evaluation remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:56:20 -06:00
DottaandPaperclip	4a83ba4050	fix(workspaces): isolate unpublished preparation directories (#18922) ... Bind new remote preparation directories to the writer epoch and environment lease. Preserve earlier unpublished files and the recorded path for published warm workspaces. Run the interrupted reused-allocation and saved-file lifecycle regressions with owned Postgres by default, and retain the warm-path recovery checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.5	2026-11-03 22:30:48 -06:00
DottaandPaperclip	f211d44c00	fix(heartbeat): preserve queued coalesced comments at claim (#18913) ... ## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:09:58 -06:00
DottaandPaperclip	e7549ab613	fix(runtime): defer native routine recovery collisions (#18916) ... Treat the exact routine execution unique conflict as a pending recovery obligation after the complete losing transaction rolls back. Preserve the current routine owner, provider attempt and checkpoint, and continue the startup batch. Cover real constraint failures, row preservation, one reclaim after release and unrelated database failures. Co-Authored-By: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.4	2026-11-03 21:47:56 -06:00
DottaandPaperclip	cb75c9510c	test(evals): restore chat setup and pin secure-entry guidance (#18914) ... ## Thinking Path > - Paperclip manages agent work and keeps credentials under human control. > - Agents discover supported operations through existing API and connection help. > - The measured secret-entry case chose external provider setup instead of native secure entry. > - The reviewed help distinguishes these actions and keeps the same permissions. > - Two related help changes explain recorded progress and partial team installation. > - The first remote setup found stale reporting-lock metadata on trusted master. > - This PR pins the candidate and renews that existing lock correction for narrow remote checks. ## Linked Issues or Issue Description Refs #18909. Refs #18780. The previous measurement did not complete the requested native secret-entry handoff. The next representative is case 580 on the existing TestOps plan 102 and job 100. Case 610 can use the same source for a separate representative. The progress-help change does not claim that case 466 passes. ## What Changed - Pin immutable candidate `b1ea76b286b870bb62a80eebf3fe00feed6f12c1`, which retains the approval-link fix from #18909. - The candidate adds only reviewed secret-entry guidance, three API help changes, and the matching SDK source registry entry. Its delta from the approval-link candidate covers three files. - Update the launcher snapshot target and expected target test. - Record the actual new SDK compilation and its one changed runtime JavaScript file. Preserve the previous qualification as history. - Refresh only the existing reporting-lock policy and patch from the prior reviewed lock to current master. The helper, actual lockfile, credential rules, and candidate source stay unchanged. - Keep all 244 case rows, inputs, clocks, model, private evidence controls, and held runtime cases unchanged. ## Verification - The owning secret help checks passed: 47 server/shared tests and 31 Secrets UI tests. The real discovery search failed before the help change and passed after it. - The owning progress and team-install search regressions each passed with their full 2057-test runner API suite. - In the candidate, the existing `scripts/ensure-plugin-build-deps.mjs` freshly compiled shared and plugin SDK outputs and exited zero. The emitted guidance consumer passed. Measured output changes contain only the guidance JavaScript, its declarations and maps, and two build receipts. - The old registry and default source proof rejected the changed input. After the existing registry generator updated that one entry, the 506-input check and default source proof passed. Runtime, compiler, parser, Zod, scanner, and canary certificate pins did not change. - All 51 affected SDK tests and all 20 input/build custody controls passed. One initial suite failed to load a missing local dependency; that setup failure is retained. The run with the existing package dependency links passed. - The three existing TestOps contract, adapter, and manifest test files passed: 47 tests, zero failures. - The first case-564 attempt stopped before credentials or model execution: the trusted controller lock had moved through normal dependency updates while its reporting-lock metadata remained stale. That attempt has no original native grade and is not a product failure or a PASS. - The existing lock-refresh command generated the exact reviewed correction from lock `54a81b2e` to current master lock `7540075e`. All seven correction, tamper, and refresh tests pass. The actual current-controller frozen install with lifecycle scripts disabled and both reporting/private-transfer imports pass. The frozen candidate still accepts its unchanged own lock. An initial offline install found an uncached package; the regular frozen install passed. - Fresh CI and review for the renewed head are pending. Local help checks do not establish a native agent outcome. No new TestOps PASS is claimed. ## Risks Help can improve discovery without changing the agent's final choice. The representative runs must test that remaining gap. The candidate retains the prior catalog application base plus the reviewed fixes. It does not refresh or qualify the full catalog. Automatic model retries remain zero and the authorized concurrency cap remains eight. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, code integration, compilation, and local validation. The host does not expose the context-window limit. The model under evaluation remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:28:49 -06:00
DottaandPaperclip	0884e73b46	test(e2e): verify saved assigned skill use (#18783) ... ## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.3	2026-11-03 21:13:40 -06:00
DottaandPaperclip	ce7766dc11	fix(recovery): resume retries with remote workspace storage (#18910) ... ## Thinking Path > - Paperclip manages work performed by AI agents. > - Native recovery must resume an interrupted run under its saved ownership. > - Remote workspace storage records that ownership in `runnerProfileJson.remoteWorkspace`. > - Recovery recognized only the older workspace record and plugin task leases. > - A retained remote PID could therefore leave a due retry without a dispatch. > - This PR routes the new record through the existing remote recovery verifier. > - Checkpoint, lease, process and single-executor checks remain authoritative. ## Linked Issues or Issue Description **What happened?** After a native sandbox turn returned without a semantic result, same-run recovery was saved with a retry deadline. The run retained its sandbox PID and the new remote workspace reference. The recovery selector skipped it. Later task attempts could then fail on the unfinished workspace writer. **What did you expect to happen?** The due retry must enter existing remote checkpoint and ownership verification. Recovery must resume the original run once or expose the real verification blocker. **Steps to reproduce** Persist a retryable native failure with a remote PID, a prepared `paperclip.remote-workspace-run.v1` reference, an existing lease and checkpoint, then run heartbeat recovery. The added PostgreSQL cases reproduce this shape. **Deployment mode** Native Paperclip Runner in a sandbox with remote workspace storage. Related: #18841 handles legacy workspace recovery during storage migration. #18908 handles failed native preparation. This repair covers a later, provider-started retry. ## What Changed - Recognize the current remote workspace record when classifying process location and selecting remote retries. - Extend the existing database and real-process fixtures to cover the new record, including a retained PID without a process group. - Reuse existing restart recovery. Add no new retry loop, ownership bypass, migration or workflow change. ## Verification - Before the repair: three new behavioral cases failed. Two retries never dispatched; a missing checkpoint never reached its visible recovery hold. - After the repair: all 82 heartbeat recovery tests pass. - All 57 neighboring resumption-binding, restart-recovery and remote-probe tests pass. - Safety cases retain retry deadlines, live controller ownership, active execution, missing-checkpoint rejection, single dispatch and zero host signaling of the sandbox PID. - Local platform: macOS arm64, Node 24.22.2, PostgreSQL 18.1. The tests use disposable databases and an owned live process as the remote PID collision control. Provider execution is outside this routing regression. - Full `pnpm -r typecheck` and `pnpm build` pass at `34aa2d08d38e8afec7f8d98ce984a74551caeda6`. - Current-head GitHub checks: 53 successful; `ci / verify` passed. Paperclip Review completed neutral with a 5/5 assessment and no findings. Two Storybook jobs were skipped by scope; the external security scan was neutral. No failures, pending checks, or merge conflicts remain. - Greptile: 5/5 at the same commit, with no review threads. - The duplicate full local `pnpm test:run` was interrupted after current-head CI passed. Its first two chat shards passed (574 cases); it has no complete local full-suite verdict. This is separate from the passing targeted recovery suites and required CI test lanes. - Staging diagnosis used read-only run, finalization, workspace and lease evidence. The fixed revision has not been deployed. The original task's useful continuation remains unverified. - Initial local attempts stopped before test execution because generated SDK output and PostgreSQL library aliases were missing. Those setup failures are retained separately from the behavioral red/green evidence. ## Risks A wrong remote classification could send a run to the wrong ownership path. The existing verifier still checks the saved checkpoint, controller lease and target. Tests require one dispatch, preserve process identifiers, and prove that recovery never inspects or signals a coincident host PID. Legacy workspace and plugin task coverage remains green. This repairs missed continuation; it does not establish the cause of the earlier missing semantic result. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool execution. The exact model snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:06:50 -06:00
3 people	27a825da2f	feat(telemetry): track the server OS with the registered contract (#18897) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its first-party telemetry uses a generated event contract and typed helpers. > - The existing events do not identify the server's OS family. > - A proposed event is dropped until the generated contract registers it. > - This PR syncs the registered contract and emits the OS family when an enabled server telemetry client starts. > - The benefit is a way to compare Windows, Linux, and macOS installations while keeping machine details out of the event. ## Linked Issues or Issue Description **Problem or motivation** We cannot compare Paperclip installations by host OS. Counting task or install events does not provide this dimension. **Proposed solution** Send one `server.started` event with Node's raw `process.platform` as `os_platform` each time the server initializes its enabled telemetry client. Compare distinct install identifiers per platform, rather than restart event totals. **Alternatives considered** A proposal-only emitter would send no data. A separate later contract-sync PR would leave the feature incomplete. This PR includes the canonical generated SDK and promotes the emitter together. CPU architecture and OS versions are outside this feature's scope. **Roadmap alignment** This is a narrow telemetry addition to existing server startup, with no new user flow. The full contract copy also adopts the three existing connector proposals described in #13578; their schemas are already registered. The generated file stays an exact copy, so their helpers and tests must be promoted in the same change. ## What Changed - Export `trackServerStarted` and call it once when server telemetry initializes. Repeated initialization returns the same client and emits no duplicate event. - Copy the canonical generated telemetry SDK byte for byte from schema source commit `5369e6b`. Remove the startup proposal marker. Normal opt-out and CI suppression apply before initialization. - Promote the three connector events included in the canonical SDK. Derive enum types from the generated map and preserve raw values. Update their real-client transport and disabled-state tests. - Assign startup to the existing `operational_enum_count` retention class. Document OS privacy, container/WSL interpretation, and distinct-install counting. - Add real-client startup transport tests and server initialization tests, including file opt-out, environment opt-out, and CI suppression. - Keep startup running when telemetry state storage fails. Publish the client and start periodic flushing only after the startup event succeeds. On failure, clean up the client, disable telemetry for the process, and log a generic warning without the storage path or underlying error. ## Verification - `pnpm --filter @paperclipai/shared typecheck`: passed. - `pnpm --filter @paperclipai/shared build`: passed. - `pnpm --filter @paperclipai/shared exec vitest run`: 98 files, 1,055 tests passed. - `pnpm --filter @paperclipai/shared exec vitest run src/telemetry`: 6 files, 39 tests passed. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/telemetry-startup.test.ts src/__tests__/shared-telemetry-events.test.ts`: 2 files, 19 tests passed. - Exact generated artifact comparison, `git diff --check`, and local secret/PII review passed. - Proposal inventory contains no startup or connector proposals. The repository extractor fails with TypeScript 7's missing compiler API; rerunning it with the installed TypeScript 5.9 compiler API passed. No extractor source change is included. - Initial local server typecheck attempts found missing workspace dependencies and a missing plugin SDK link in this fresh checkout. Dependencies and the local SDK link were repaired; `pnpm --filter @paperclipai/server typecheck` then passed. Full repository typecheck, tests, and build are covered by the PR CI rather than claimed as local passes. Greptile found that a telemetry state-file error could abort startup. New regression tests reproduced this on the previous head with both an injected permission error and the real state store blocked by a file. Both now pass, including no published client, no periodic flushing, no send, and no retry after failure. Shared telemetry tests and server typecheck were rerun after the fix. Full PR CI passed on `a41540b7899bc8a0aedbd78d72f8b69ad1a4b318`, including typecheck, build, all selected test shards, and the final verify gate. [Successful workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752039595/attempts/2). Attempt 1 was interrupted when the server (1/12) runner received a shutdown signal; its incomplete result was retained in the PR discussion. The same-commit shard retry and dependent gate passed. Greptile reviewed this exact head at 5/5 with all threads resolved. ## Risks This change starts sending one OS family value per server startup when telemetry is enabled. It sends no OS version, architecture, hostname, path, prompt, or machine identifier beyond the existing envelope. Containers and WSL report the runtime's OS. Event totals reflect restart frequency, so use distinct install identifiers for OS adoption analysis. The complete generated SDK also enables the previously registered connector proposals. Those events keep their existing approved payloads; no connection names, credentials, tool arguments, or results are added. Privacy review is requested for both newly active paths. Deploy the registered schema before merging this app change. Live production delivery has not been tested. No schema migration is needed. Revert this PR to stop the new startup emitter and restore the prior contract. ## Model Used - Original implementation: Anthropic Claude Opus 5.5, as recorded in the original commit. - Completion: OpenAI `gpt-6.1-sol`, xhigh reasoning, through Codex in T3 Code; shell, Git, and automated tests. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] The final change and description contain no private issue links or proposal markers - [x] The existing PR branch is preserved to retain its review history - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:00:47 -06:00
DottaandPaperclip	9e61fdebf9	test(evals): pin approval-link fix for case 564 (#18909) ... ## Thinking Path > - Paperclip manages work through agents and explicit approval gates. > - The hiring revision case requires a usable link to the saved pending proposal. > - A rendered unprefixed approval link currently misses the company redirect. > - The reviewed fix adds one existing redirect route. It changes no approval authority. > - The TestOps workflow reads its candidate source from a protected master pin. > - This PR pins that one-fix candidate so one remote case can test the original journey. ## Linked Issues or Issue Description Refs #17858. Refs #18780. The source candidate applies only the approval-detail redirect from the existing fix. The core fix PR remains unmerged. This PR changes the evaluation source pin. ## What Changed - Pin candidate `abbde061c7ec9181021858f7b1af21a48637f97f`, a direct child of the prior `fd5e3b1e` candidate. - Update the matching launch snapshot target and the expected-pin test. - Keep all 244 case rows, original inputs, clocks, model, and private evidence rules unchanged. ## Verification - The candidate differs from its parent in one line in `ui/src/App.tsx`. - The owning browser regression follows a real rendered link to a saved pending proposal. It fails before the route fix and passes after it. The 18 App tests, UI build, typecheck, and token gates passed. - The candidate token gates passed. The existing source SDK input check reports 506 unchanged inputs and unchanged runtime pins. No SDK registry or certificate was renewed. - The three existing TestOps contract, adapter, and manifest test files passed: 47 tests, zero failures. - The first local pin invocation lacked the YAML dependency. It is retained as a setup failure. The invocation with the existing locked dependencies passed without assertion changes. - Exact-head normal PR CI [41763017485](https://github.com/paperclipai/paperclip/actions/runs/41763017485) completed successfully: 48 successful jobs and one selector skip. Fresh Greptile and Paperclip reviews both report 5/5 with no findings; all review threads are resolved. The optional Superagent security check is neutral, so no security assessment PASS is claimed. No new provider run or TestOps PASS is claimed. The intended next check is only `questions-and-approvals.formal-hire-revision` on existing plan 102 and job 100. ## Risks The route regression does not prove the full agent journey passes. The first remote run will test that remaining gap. The candidate keeps the prior catalog application base plus this one route fix. It does not refresh the whole runtime or qualify all 244 cases. The existing broader qualification holds remain. Automatic model retries stay zero and the authorized concurrency cap stays eight. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, code integration, and local validation. The host does not expose the context-window limit. The model under evaluation remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:00:36 -06:00
DottaandPaperclip	630ed6624c	test(e2e): add verified dependency continuation QA (#18901) ... ## Thinking Path > - Paperclip lets people manage work assigned to AI agents. > - Task dependencies must hold assigned work until its blockers are complete. > - The reusable Product E2E set needs a browser journey for that full path. > - The journey must verify saved records, automatic wake identity, and the result after reload. > - The original implementation passed staging QA, but its automated review workspace has an export hold. > - This replacement integrates current master and earns a separate normal review while preserving the original attempt. ## Linked Issues or Issue Description Refs #18789, the original dependency QA pull request. Related: #7542 and #17524 for dependency wake and explicit resume behavior; #18782 for the merged lifecycle registration layout. **What existing behavior does this improve?** The Product E2E catalog can verify the complete blocked-task continuation path against a configured native Cloud target. **Current behavior** The main catalog lacks this full browser journey. The original PR's automated review task has a retained workspace export hold (`native_workspace_sync_out_retry_exhausted`), with neither Retry nor Continue available. Its task, files, checks, and failed export attempts remain preserved. **Proposed behavior** Add the same independently checked `task-dependency-resume` journey through this replacement. Its new commit normally merges the actual fetched master `4e29f001396c9fc76075850165ccfa3a6743e3ef` into verified source `e53dd51407b901b14bd582eeb2071fabe6955d9d`. That integration contains 39 real upstream path updates. The replacement must pass fresh CI and automated review on its own head. **Reason and benefit** Daily QA can detect premature work, a missing or duplicate continuation, and a result from the wrong run. A separate review workspace lets the normal review process inspect the current integration while the original held workspace stays intact. **Breaking changes** The change adds an explicit journey. Existing journeys and their registrations remain present. ## What Changed - Add a browser journey that creates a human-owned blocker and dependent task, saves their relation, selects Blocked, and assigns the configured native agent. - Require a measured ten-second blocked wait, a board-owned blocker completion, and exactly one automatic successful native run. - Independently verify the saved wake source and blocker identity, configured provider/model/environment, committed native phase, document revision authorship, and reloaded result. - Return `unsupported` before observations or task creation when the native provider or model identity is missing or blank. - Attempt both evidence writes, preserve an existing exact error object and outcome, and fail otherwise successful work when required evidence cannot be saved. - Grade runtime identity, document authorship, reload visibility, and retained history from their own evidence bound to the same company, task, agent, and run. All eight checks remain required. - Keep the random review reference in the blocker description. Save allowlisted private observations and read failures through the existing evidence pipeline. - Register the case at the distinct dependency anchors in the catalog, types, checks, source hash, dispatcher, features, and documentation. The delta against fetched master is exactly the original 15 QA paths. The fixture, contract tests, shared registrations, and dispatch remain unchanged. The browser actions and observations are byte-identical to `d41558b5`; only evidence finalization changes in that flow. Six shared-file AST comparisons still match fetched master after removing only the dependency registrations. All upstream integration changes remain preserved. ## Verification Prior acceptance remains bound to its actual source: - [Staging QA run 41731870131](https://pages.paperclip.ing/evals-qa-smoke-41731870131-1/) passed all eight checks and all five phases, with closed cleanup, on `dc9ea42d46d3015c010d0532b48e0da973d3611a`. - [Original full CI run 41745149524](https://github.com/paperclipai/paperclip/actions/runs/41745149524) passed on `e53dd51407b901b14bd582eeb2071fabe6955d9d`: 48 successful jobs and one skip on attempt 1. - The original failed staging result and the original held review remain intact. These prior passes are not new checks for this replacement. Passed locally on the current replacement source with Node 24.22.2 and pinned pnpm 9.15.4: - One complete `install --frozen-lockfile --ignore-scripts`, followed by the maintained `node scripts/link-plugin-dev-sdk.mjs` for links inside this checkout. - `node scripts/ensure-plugin-build-deps.mjs` and `pnpm --filter @paperclipai/db run build`. - `node scripts/typecheck-test-projects.mjs --low-memory tests/runner-e2e`, including the current Runner TypeScript build, fresh server declarations, and E2E typecheck. - 239 focused controls across eight files. These cover missing/null/empty/blank execution identity, real filesystem write failures, synchronous writer failures, frozen original errors, and independent defect attribution. - 5,302 comparisons against the prior grader, with zero aggregate acceptance changes or new throws. The sealed actual staging observations from run 41731870131 still produce the same eight passing checks. This is finite calibration of retained evidence, not a new staging run. - Credential-free discovery: 22 unique cases, including dependency and lifecycle. - Source-preservation checks and whitespace checks for both the normal integration and the replacement delta. The focused command is: ```sh node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/product-journeys/task-dependency-resume-evidence.test.ts tests/runner-e2e/product-journeys/task-dependency-resume.test.ts tests/runner-e2e/product-journeys/task-dependency-resume-contract.test.ts tests/runner-e2e/product-journeys/contracts.test.ts tests/runner-e2e/product-journeys/observation-read.test.ts tests/runner-e2e/catalog.test.ts tests/runner-e2e/journey-features.test.ts ``` The old generated cache failed current-source equality and was not copied. Canonical current outputs were built and sealed with their input hashes, including CLI, UI, and provider inputs. No Rust build, QA dispatch, model call, or action on the held workspace was performed for this replacement. Review follow-up: the first replacement head `b4408c5732e1800f7ccf66c639cf382b5ff2f01a` passed full CI run [41749940687](https://github.com/paperclipai/paperclip/actions/runs/41749940687), with 48 successful jobs and one skip. The native review then found the missing execution-identity preflight. The preflight now rejects that setup before work. Completed current-head checks are recorded below. The current E2E typecheck reused the same canonical declarations only after all 11,273 build inputs, two installed toolchain files, and 6,747 generated files matched their saved hashes. The five changed case files are outside that build-input set. One sandbox test run passed 216 controls and lacked the report subprocess output. A direct `tsx` diagnostic confirmed a temporary pipe `listen EPERM`. The unchanged 217 controls then passed with ordinary local IPC permissions. Both results remain recorded. The `d41558b5807e609032bbadddb103d6b0a7db4a12` attempt [41753398775](https://github.com/paperclipai/paperclip/actions/runs/41753398775) is retained: 45 successful jobs, one skip, two failed server shards, and a failed aggregate. The runtime readiness case refused connections within its ten-second window; the issue-deadline and large repository cases hit their existing test limits. Their causes remain unknown. A source-matched Linux CI run executed and passed all three cases, and one unchanged local runtime case passed. No timeout or assertion changed and no old-head diagnostic recheck was issued. The native review then identified error masking during evidence persistence and cumulative diagnostic flags. The current head fixes those two owned-case issues. Its completed current-head checks are recorded below. Seventeen of twenty new regression controls failed before these fixes; all now pass. Two additional controls exercise synchronous writer failures. Both original review and CI evidence remain intact. The eight required checks, one-run limit, 500-cent configured budget, and bounded ten-second observation are unchanged. Fresh full CI, current-head Greptile 5/5 with no open findings, and every actual review context must complete before merge. Prior statuses are not copied or overridden. Current-head verification is complete for `d9bc055f1f6609463679dc78fdcb96034d2c3727`: - [Full CI run 41758005698](https://github.com/paperclipai/paperclip/actions/runs/41758005698), attempt 2, completed successfully at 02:26:19 UTC on November 4: 48 successful jobs and one skip. - Attempt 1 lost one exact Spot runner. AWS confirmed `Server.SpotInstanceTermination` and `instance-terminated-no-capacity`; its log recorded the shutdown. That failed job and the failed aggregate remain preserved. - One normal failed-jobs recheck ran only the interrupted static checks (02:19:13–02:25:54 UTC) and aggregate (02:26:13–02:26:18 UTC). The other 46 successful start/finish pairs remained unchanged. The skipped selector was not an execution. - Greptile and Paperclip Review both scored this exact head 5/5. There are no unresolved review findings. ## Risks - The staging pass remains attributed to `dc9ea42`; final admission must retain that source provenance and the current preservation proof. - The case uses one paid work run when executed. Agent setup usage remains separate from task-run accounting. - The ten-second blocked observation proves a finite wait. Missing, incomplete, foreign, premature, or duplicate evidence fails the independent checks. - The original held workspace still needs its own disposition. This PR leaves it preserved for its owner. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, and code execution. The exact model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 20:35:43 -06:00
DottaandPaperclip	a194166017	perf(ci): repair dependency caches and parallelize transport fixtures (#18883) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect changes to its native Runner and control plane. > - Recent Runner jobs restored a Rust cache but still rebuilt dependencies for 74–92 seconds. > - New workspace install hooks disabled the materialized JavaScript dependency cache. > - A single transport file also kept 326 test cases on one worker. > - This pull request corrects the cache inputs, replays known install hooks, and separates independent transport test groups. > - The same checks can finish sooner without removing test cases or widening runner access. ## Linked Issues or Issue Description Refs #18794 and #18544. Related dependency-action version update: #17698; this change fixes the cache input contract used by the currently pinned action. **What happened?** Recent PR jobs such as #18783 and #18825 restored an exact Rust dependency cache from October 16. The virtual workspace's development profile changed on October 30. Rust Cache 2.9.2 hashes member manifests but omits that virtual manifest. Rebuilding cannot replace an immutable exact cache entry. The JavaScript cache also rejected the new server and Runner Copilot postinstall hooks. **Expected behavior** Changing a Cargo profile selects a new dependency cache. An exact JavaScript dependency restore still performs required workspace install hooks. Independent transport fixtures use the existing file workers while retaining all assertions and real deadlines. **Steps to reproduce** Inspect the Rust dependency cache hit and following `build:rust` in the recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs` on the previous revision to see `cacheable: false`. Run the original transport file to measure the serial test work. ## What Changed - Add the virtual workspace manifest and ancestor Cargo configuration to all shared Rust dependency keys. Keep reader/writer identities aligned and preserve master-only cache writes and CODEOWNERS-only AWS routing. - Recognize only the exact reviewed root, server, and Runner postinstall commands. Include the server/Runner installer scripts in the dependency key. Replay workspace postinstall hooks after a materialized restore. Unknown lifecycle hooks still disable reuse. - Move attachment recovery, warm attachment, live adoption, and suspension fixtures into separate files. Keep the large event-backlog cases in the original file to avoid starting another expensive fixture group at the same time. Share five stateless test helpers outside production source. - Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache producer and a fresh-host consumer. Require exact dependency/workspace hits and reject any Cargo unit that is rebuilt. Retain the JSON build evidence. Existing setup qualification checks real JavaScript cache restores. - Schedule the ten measured files over 50 seconds first, in approximate duration order. Package CI uses up to four workers, bounded by available CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15% memory. Preserve Vitest selection and shard membership and default ordering for other files. Job count and AWS authorization are unchanged. - Scope the optional paired server benchmark to changes in its measured inputs. Cache-only changes qualify exact restores and run the shared package tests using restored dependencies. - Update the cache and test documentation and the existing native cache contract guard. ## Verification - All 586 `.github/scripts/tests/*.test.mjs` policy tests passed, including cache trust and runner routing guards. Actionlint passed for all three changed workflows. - All 169 original top-level transport test definitions remain byte-identical. The original file passed all 326 cases in 317.99 seconds on this macOS host. A clean preliminary split passed the exact same named cases in 183.01 seconds with the same four-worker cap. - Preserve the first failed experiment: its parallel run overlapped the policy suite and timed out in the 1,024-event settled backlog case at 60 seconds; 325 cases passed. A clean repeat passed, but that does not establish the earlier failure's sole cause. The final layout keeps the backlog cases in the original file. The final layout passed the same 326 cases in 186.35 seconds (41.4% faster than the 317.99-second baseline). The original test bodies and all timeouts remain unchanged. - Fresh-host Rust qualification passed: all 308 compilation units were fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)). JavaScript setup dropped from 46 seconds cold to 23 seconds with exact dependency/build restores; all 1,052 shared-package cases passed against the restored tree ([final-head job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)). The previous qualification also passed (43 to 19 seconds) and confirmed all three postinstall hooks replayed. These are qualification results, not an end-to-end PR timing claim. - Initial full AWS package run passed 4,907 cases with 10 existing skips. The original-file remainder fell to 118.7 seconds, but the 100-second deadline file started last, leaving the full package at 417.43 seconds. Starting only three long files first passed 4,911 cases (including four new sequencer guards) in 422.54 seconds, so reordering alone did not improve full-package time. Summed file execution was 1,202 seconds across three busy workers. The final four-worker configuration passed **4,912 cases, with the same 10 existing skips**, in **324.99 seconds**, a **23.1% reduction** from 422.54 seconds. The five added cases guard ordering, selection, sharding, and path handling. Peak memory was 17.95% of the 16 GiB runner; average CPU was 49.14% ([final-head package job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)). - Local workspace typecheck and build passed. The complete local `pnpm test:run` passed all six chat partitions but reported failures in one existing OAuth redirect case and one gateway authority case in the general-server group. Both focused repetitions passed (three OAuth cases and one gateway case), and a complete repeat of both files passed all 593 cases in 42.89 seconds; that does not explain the original failures. The broad local run was stopped after 61 minutes once full final-head CI had passed; its remaining general/workspace/serialized work was not completed locally, and it is not reported as a pass. Original failure logs were retained. Full final-head CI provides the complete suite validation. Sequencer selection/shard guards and the 11-case transport suspension smoke passed. The corrected release workflow guard passed all 71 tests. - The first CI run failed only the policy guard that still expected the old native cache key; downstream aggregate checks also failed. That guard now checks the new profile/config prefix and existing host/target/protocol suffix. A second policy run found that the new sequencer test imported Vitest in the dependency-free policy lane; it now runs in the Runner package lane, where all four cases pass. Greptile reviewed final head `8011f84189` at 5/5 with no actionable findings and confirmed this issue fixed. Final-head policy, typecheck, build, and review passed. At 23:52:46 UTC on November 3, several AWS jobs were terminated together: the Runner package process received SIGTERM and browser logs recorded a runner shutdown signal. The failed/cancelled jobs passed on the same-head retry, including required `ci / verify`; no assertion or timeout was changed in response. The final concurrency adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable findings. Final-head required CI, including `ci / verify`, passed ([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)). Local timings exclude install, build, cache transfer, and CI queue time. ## Risks A new Rust key needs one successful canonical master build before ordinary PRs can hit it. The fresh-host qualification uses separate PR-scoped entries. Parallel test files can increase I/O contention; case execution inside each file, timeouts, and assertions are unchanged. Two extracted suites exceed 1,000 lines because they retain existing multi-stage recovery fixtures unchanged; one parameterized fixture alone is 1,059 lines. No production execution logic, lockfile, credentials, runner trust gates, or required check is removed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, command execution, and GitHub inspection. The host does not expose a more precise model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks passed; the broad-run failures and repetitions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 20:17:08 -06:00
Devin FoleyandClaude Opus 5.5	925965a09b	fix(task-retry): wait out brief chat authorization contention (#18895) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A user can click Retry on a task that is connected to an external chat conversation. > - That Retry prepares the chat retry inside one admission transaction. The chat step takes a per-principal authorization lock with `pg_try_advisory_xact_lock`. > - When another chat transaction holds that lock for a short time, the chat step throws `NativeChatReviewPresentationContentionError`. This error is not an `HttpError`. The route returns HTTP 500, and the error handler reports a server crash. > - The contention is routine and short. The admission transaction does not contact a provider, and it rolls back completely. The owning service can safely try it again. > - This pull request retries the admission a bounded number of times. If the lock stays busy, the service returns a retryable 409 instead of a 500. > - The benefit: the user's click usually succeeds without an error, and Paperclip stops reporting expected lock contention as a crash. ## Linked Issues or Issue Description **What happened?** A hosted staging server reported `NativeChatReviewPresentationContentionError: Chat response authorization is temporarily busy; no provider delivery was attempted` from `POST /api/issues/:id/retry`. The stack is `requestTaskRetry` → `writeExecutionControl` → `prepareFailedChatRunRetry` → `failedChatRetrySource`. The error came from the `chat-identity:<company>:<principal>` advisory try-lock. The route returned HTTP 500 for each occurrence. **Expected behavior** Short lock contention must not fail a deliberate Retry click. The service must wait it out. If the lock stays busy, the service must return a clear, retryable client error and must not report a server crash. **Steps to reproduce** 1. Bind a task to a restricted chat endpoint and conversation. Make the latest task run a failed run. 2. In a different database transaction, hold `pg_try_advisory_xact_lock(hashtextextended('chat-identity:<company>:<principal>', 0))`. 3. Call `POST /api/issues/:id/retry`. Before this change, the chat retry step throws `NativeChatReviewPresentationContentionError` and the route returns 500. **Paperclip version or commit** Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The server release that reported it was `d59791828482f213713ca55000c40e6c9c6c03e5`. **Deployment mode** Hosted staging, Node 24, PostgreSQL. Related: #18845 and #18621 changed nearby Retry admission. They do not handle this contention. ## What Changed - `server/src/services/task-retry.ts`: `requestTaskRetry` now calls the unchanged admission transaction (`admitTaskRetry`) in a bounded loop: 10 attempts, 100 ms apart. It retries only `NativeChatReviewPresentationContentionError` and lock-not-available contention (`isExternalChatWaitAuthorizationContention`, which matches `55P03`). Every other error, including 403/404/409 decisions, propagates unchanged on the first attempt. - If the lock stays busy for all attempts, the service throws `conflict(..., { code: "task_retry_busy", retryable: true })`. The UI's existing 4xx handling resets the Retry button, and the error handler does not report a 4xx as a crash. - `server/src/services/task-retry.test.ts`: three PostgreSQL cases: - Real advisory-lock contention. A second transaction holds the same `chat-identity` lock that production checks, then releases it. Retry succeeds and persists exactly one wake request. A replay with the same idempotency key returns the same request. - Persistent contention returns 409 `task_retry_busy` after exactly 10 attempts and persists no wake request. - A 403 from the chat step does not retry. The transaction body is unchanged. It was moved into `admitTaskRetry` without edits. ## Verification - RED: with the old `task-retry.ts` and the new tests, the contention case fails with the exact production error (`NativeChatReviewPresentationContentionError: Chat response authorization is temporarily busy…`). The persistent case fails because the error is not an `HttpError`. - GREEN: `pnpm exec vitest run src/services/task-retry.test.ts` passes all 39 cases against embedded PostgreSQL. - `pnpm --filter @paperclipai/server typecheck` passes (all layers, including tests). - `src/services/heartbeat/workspace-preparation.test.ts` (another `requestTaskRetry` caller): 165 of 167 cases pass. The 2 failures (`remote_workspace_migration_source_changed` in "resumes an imported path/file-url Git source…") also fail on unchanged master in the same environment, so this change does not cause them. - Runtime assumption (rule `runtime-dependency-boundaries`): the retry is safe only if a failed attempt leaves no persisted state. The contention test uses a real PostgreSQL advisory lock held by a separate connection and checks that only one wake row exists after the failed attempts. Each attempt is a separate transaction, so the loop holds no row lock or advisory lock while it waits. - The full repository `pnpm test:run` and `pnpm build` were not run locally. I rely on CI for those. ## Risks - Low risk. The worst case adds about 1 second of request time when the lock stays busy, and then returns 409 instead of 500. - An attempt can fail after an earlier step in the same transaction (for example, workspace preservation registration). The whole transaction rolls back, so the next attempt repeats those steps from a clean state. Those steps run in the same transaction and do not contact a provider. - If `requestTaskRetry` is called inside an outer transaction, each attempt rolls back only to its savepoint. Locks that the outer transaction holds stay held. In that case the loop ends with the same 409 after its bounded budget. - No schema, API shape, or UI change. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), through Claude Code with tool use (shell, file editing, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; see Verification for the two pre-existing failures) - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no behavior or command documentation changes are needed) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 18:05:40 -08:00
DottaandPaperclip	057898530c	fix(server): handle wrapped recovery action conflicts (#18885) ... ## Thinking Path > - Paperclip coordinates durable agent work and recovery. > - Independent workers can find the same task needs recovery. > - The database permits one active recovery action for that task. > - A losing insert must read the action that already committed. > - Drizzle wraps the PostgreSQL unique violation in its cause. > - The current classifier reads only the outer error and misses that conflict. > - This PR recognizes the same allowed conflict through a bounded cause chain. > - Both workers can then return the one durable action through the existing retry path. ## Linked Issues or Issue Description Refs: #18783. **What happened?** The saved-disposition-debt recovery regression failed in CI `41733864320`. Two recovery workers collided on `issue_recovery_actions_active_source_uq`. The fixture captured an unhandled PostgreSQL `23505` error. A separate canceled shard in that run lost its AWS Spot instance; that interruption is a different failure. **Expected behavior** Two service instances that race to create the same active action must return that one action. Unrelated database failures must still propagate. Existing transaction and ownership rules must stay in force. **Steps to reproduce** Hold a PostgreSQL SHARE lock on the recovery-action table. Start the same source-scoped upsert on two independent service instances. Confirm both real INSERTs are waiting in `pg_stat_activity`, then release the lock. The original code persists one action but rejects one caller with a `DrizzleQueryError`. Its outer code is absent; its `PostgresError` cause has SQLSTATE `23505` and the expected active-source constraint. **Paperclip version or commit** Supporting branch starts at master `54782706e1d83d92295246e499627930df03662d`. It now contains a normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The classifier correction stays unchanged. The PR diff against that master still contains only the recovery service and its tests. ## What Changed - Inspect up to eight error causes with cycle detection. - Require SQLSTATE `23505` and one of the same two recovery-action constraint names on the same error. Support the PostgreSQL driver's `constraint_name` field. - Reuse the existing source-scoped retry path and its unchanged three-retry limit. - Add a deterministic real-database collision regression and no-retry controls for unrelated uniqueness, other SQLSTATEs and cyclic errors. - Keep the original recovery regression, assertions and timeouts unchanged. Keep the saved-skill journey separate in #18783. ## Verification - Review follow-up: the real collision test now checks the intended retry merge. The surviving action has attempt count 2, active status, the default unlimited budget, the expected owner, supplied evidence and supplied attempt timestamp. Its wake and monitor fields remain null. Existing identity and convergence assertions run first. - A disposable counterfactual returned the winner unchanged after the lost insert. It passed the original barrier, one-row, no-rejection and same-ID assertions, then failed the new attempt-count check with 1 instead of 2. The counterfactual was not committed. The production service was restored byte-for-byte. The actual service passed all seven focused race, no-retry and sequential controls. The affected canonical test-tree-2 typecheck passed against verified unchanged local declarations. - Parent `4f8abfe4f2244649219ef6ced3534634193f0b53` full CI `41746314291` passed all 48 selected jobs with one intended skip. Its 5/5 summary also had two duplicate unresolved reliability comments about the missing state assertions. This test-only follow-up addresses both. Fresh full CI and current-head review are required again. - The deterministic real-database regression failed before the correction. Both inserts reached the lock and exactly one active action persisted. The losing error was a `DrizzleQueryError` with a `PostgresError` cause, SQLSTATE `23505` and constraint `issue_recovery_actions_active_source_uq`. - **11 focused checks passed across two files on the integrated head** (181.72 seconds). This includes the real-database collision, no-retry controls and all four unchanged recovery producer/dependency regressions, including the original failed real-clock saved-debt case. Both callers return the one persisted action. - Current-head canonical server typecheck passed on Node24.22.2 with all seven layers: below, cycle, above, tests and test-tree-0 through test-tree-2. Canonical shared, SDK, DB and Runner TypeScript prerequisites passed. The first filtered installs lacked CLI and UI imports required by the new test layers. Those module-resolution failures remain recorded as setup failures. A full frozen workspace install with lifecycle scripts disabled supplied the same locked dependencies. No source or type-configuration workaround was used. No Rust or broad workspace build ran locally; full repository typecheck, tests and build remain hosted-CI gates. - The actual supporting master baseline reproduced the same wrapped23505 failure before the correction. Earlier configuration-resolution, local IPC and native-library prerequisites remain separate setup failure evidence. No test failure was reclassified as a pass. - First supporting CI `41739489620` was canceled before its build/test jobs completed. Its later automatic full-CI run `41739747947` failed. The recovery server shard passed. The Runner failure was the Cursor fixture case repaired upstream in #18782, now included by the normal master merge. That CI also observed an attachment-browser timeout, a repository-rehome timeout and a lost-runner connection. Their underlying causes remain unknown. A later pass will establish recovery, not explain those earlier failures. These original results remain retained. - Current `fbd392b9b65dbcf4ac078825096037ed0d490a0b` full CI [41751706634 attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41751706634/attempts/2) passed: 48 successful jobs and one intended skip. Attempt 1 remains retained: ten primary jobs lost their bound Spot capacity and the verify job failed as a dependent. A single ordinary failed-job recheck produced eleven fresh successful executions (ten primaries and verify); timestamps prove 37 successful executions were retained. The skip is not counted as execution. Current Greptile is 5/5 with both addressed reliability threads resolved and no unresolved findings. All 57 current check runs are successful, neutral or skipped, and Snyk is successful. This is CI recovery; it does not explain the separately retained older attachment and repository timeouts. - The original CI failure and both failed saved-skill QA attempts remain unchanged. This PR does not establish a live QA pass or deploy a production change. ## Risks Conflict classification must not turn unrelated errors into retries. The new controls require those errors to propagate after one insert attempt. The traversal is bounded and stops at cycles. Transaction, company scope, ownership, notifications, retry count and database constraints are unchanged. No schema or public API change is required. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model ID or context-window size. These checks make no model or provider calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 19:51:37 -06:00
Michael NguyenandPaperclip	58d1b21098	fix: separate fresh setup from archived personal connections (#18902) ... ## Thinking Path > - Paperclip helps people manage AI agents and their service access. > - Each personal connection belongs to one user. > - Setup can recover an archived connection by its name and provider. > - A fresh setup can therefore select another user's old identity and fail its owner check. > - This change creates a separate connection for fresh setup after personal archival. > - Explicit reconnect still requires the existing owner and preserves its history. ## Linked Issues or Issue Description **What happened?** A fresh GitHub setup can return 403 after a personal connection with the same name was archived. The request does not include a reconnect target. Setup still selects the archived identity by name and provider. **Expected behavior** Fresh setup creates a separate personal identity. It leaves the archived connection, credentials, grants, and activity unchanged. Explicit reconnect rejects a different user. **Steps to reproduce** 1. Create a personal GitHub connection in a test company. 2. Archive that connection and its application. 3. As another authenticated member, start fresh personal setup with the same name and provider. 4. Observe the owner-check failure before this change. **Paperclip version or commit** Base: db9be4c5a. Tests use isolated database and HTTP fixtures. **Deployment mode** Authenticated. The legacy owner can be a local board identity. ## What Changed - Exclude implicitly recovered archived personal connections from fresh setup. - Use the existing company-scoped name allocator to create a separate application and connection. - Test three owners: a legacy board identity, another user, and the current user. - Check credential isolation, unchanged archived records and activity, missing-key rejection, and explicit reconnect denial. - Keep managed revival and generic MCP rollback/reconnect fixtures on explicit reconnect targets. - Verify generic personal URL setup creates an independent identity and preserves the archived records. - Document fresh setup and reconnect behavior. ## Verification - Final head: `96184e3179aab2eeb88df87d442691741a4875a2`. Full CI recovered and completed successfully: [run 41752164958, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752164958/attempts/2). PR rollup: 53 successful checks, 3 skipped, none failing or pending. Typecheck, build, all server shards, Runner lanes, and E2E gates pass. - Exact-head Greptile: 5/5, zero inline comments or review threads. Independent engineering/security reviews approved the production implementation; engineering approved the test-only follow-up. Browser acceptance review approved with the timing qualification below. - Final-head local generic MCP suite: 84/84 passed (23.26 s); server typecheck passed. Reviewer independently reran the three changed scenarios: 3/3 passed (15.74 s). - Production head `587631230`: local tool-access suite 504/504, recursive typecheck, and full build passed. Broad local checks were not repeated for the test-only delta. Full local `pnpm test:run` remains incomplete after interruption during the first chat shard; no full-local-suite pass is claimed. - Initial full CI at `587631230` failed three generic MCP fixtures that still relied on implicit archived personal recovery. All three reproduced locally (15.01 s). The test-only follow-up gives owner reconnect/rollback scenarios explicit targets and verifies independent fresh URL setup without relaxing their assertions. Corrected server shard 12/12 passes on final head. The unfinished earlier run was cancelled after the follow-up push. - Initial current-head E2E shard 3/8 failed a 5000 ms task-heading wait at `tests/e2e/cloud-auth.spec.ts:46` (40 passed, one failed). Its screenshot/accessibility snapshot contains the expected heading. The single authorized no-code rerun passed 41/41 with zero test retries, including all URL, handoff, and login-form checks. The first-heading assertion took 3774 ms on rerun. Both attempts tested the same merge commit for this PR head. These observations support a timing concern; the original cause remains unknown. The unchanged Cloud-auth path does not invoke archived-connection setup. Recovery is established; an infrastructure cause is not. If it recurs, capture an instrumented first-navigation trace and compare an unchanged baseline under comparable conditions. - PR remains draft. Merge and deployment are outside this request. Synthetic provider/browser fixtures do not prove live GitHub authorization or a deployed fix. ## Risks - Fresh setup after personal archival now creates a new ID and a suffixed name, including for the same owner. Clients must supply an explicit target to reconnect the old identity. - The guard applies to all personal gallery connections, not only GitHub. - No schema migration is required. Explicit owner checks remain in place. - Security review approved the unchanged production implementation. Merge and deployment are outside this change request. ## Model Used OpenAI GPT-6 through Codex. The runtime does not expose a more specific model snapshot or context window. Used reasoning, code editing, and shell tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 17:51:29 -08:00
DottaandPaperclip	e2dc2cc0be	fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893) ... ## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. **What happened?** A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. **Expected behavior** A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. **Steps to reproduce** Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. **Paperclip version or commit** Reproduced on ab2d2098a2. The branch includes current master 44a4ca1c61 before this PR was opened. **Deployment mode** Native Runner with provider ingress on a reusable sandbox workspace. Local regression uses real PostgreSQL and local TCP with a controlled provider-process boundary. No hosted provider call was made. ## What Changed - Keep physical ingress identity separate from logical task/session ownership. Never attach the parent to the child's session. - Reserve a physical allocation during local admission. Retire a distinct warm owner only after canonical company, agent, workspace, provider, allocation, lease, task and coordinator checks pass. - Require the current remote workspace writer and location to match the admitted run. Require the prior run to be committed and accounted, with its original controller generation and a clean released or expired lease. - Use the existing close, checkpoint and stopped-file collection path. Wait for close before parent dispatch, then recheck canonical authority before staging files or calling the parent backend. Retain the entry when close is uncertain. - Add actual executor/PostgreSQL/TCP safety cases and document the contract. Keep maps unchanged for this bug fix. ## Verification - Untouched production at ab2d2098: actual executor + PostgreSQL + TCP fails EADDRINUSE. Same positive case passes after the change. - Final focused suite: 20 passed. It covers distinct parent session identity; foreign company, allocation and workspace; active or unaccounted owner; changed controller; reassignment; competing lease/writer; stale allocation metadata; Stop before and after claim; workspace-lock race; failed close that leaves the listener and accepted result intact; and controller transfer, Stop or writer change during awaited close. These last cases reject the stale parent before backend invocation after the old listener closes. - Untouched head3425acbe: transferring the incoming coordinator during close still reaches the parent backend. The same actual executor/PostgreSQL/TCP case passes after the post-close authority check. - Broader `native warm session supervision` suite: 160 passed. This run preceded the final two-line logical-session collision guard; the final 20-case suite includes that guard. - `pnpm --filter @paperclipai/server typecheck`: all source and test layers passed, including `src/__tests__`. - `pnpm --filter @paperclipai/server build`: production server and Runner build passed at current head572acfef79. The exact-head build stamp is recorded. - The provider/close boundary is controlled. The fixture seeds canonical committed/accounted finalization data in PostgreSQL. These tests do not claim a live model result, operator ACK, hosted PID-to-listener identity, or recovery of a historical task. - Full repository pnpm checks were not run locally for this server-only slice. Required current-head CI and reviews are pending. ## Risks The new retirement path can close a process from another task. It requires the same canonical company, agent, allocation and workspace, a current exclusive writer, and a completed idle owner. Missing or changed evidence leaves admission blocked. Active owners, changed generations, reassignment and cancellation remain fenced. A failed close retains the owner and saved result. This covers current-controller warm entries with recorded physical identity. Historical processes outside the registry keep the existing exact-owner recovery requirements. No port shuffle, port kill, task capability expansion, new polling loop or invented receipt is added. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 19:44:12 -06:00
DottaandPaperclip	d7283e8757	test(e2e): cover chat reload and fresh sessions (#18784) ... ## Thinking Path > - Paperclip keeps agent work attached to saved company records. > - Agent Chat keeps one conversation for each person and agent. > - Reload must preserve context without starting another turn. > - The ordinary `/new` command must create fresh provider context and preserve history. > - Existing local chat journeys cover these behaviors, but the deployment-targeted daily catalog does not. > - This PR adds one reusable journey with independent saved-state and rendered-reply checks. ## Linked Issues or Issue Description Refs: #2242, #18655. **What existing behavior does this improve?** Product E2E coverage for Agent Chat persistence and session reset on an attached deployment. **Current behavior** The deployment-targeted catalog has no bounded check for recall after reopening and a fresh session in the same saved conversation. **Proposed behavior** Remember a random phrase, reopen and reload the chat, then ask for the phrase without supplying it. Send `/new` through the ordinary composer. Require a new native session and a fresh-context answer while the original chat and history remain saved. ## What Changed - Add the `chat-continuity` flow and independent evidence grader. - Reuse the existing chat sender and run-bound native reply reader. - Require exact board-user message attribution, three successful native provider turns and one synthetic reset row. Count all four persisted run IDs. A reset with provider input or a native session cannot pass as synthetic work. - Use the original bound issue-run summary to prove reset completion when the detail result is null or compact. Preserve the raw detail and require its reset context, company, agent, issue and wake comment to agree. Native provider turns require committed native state and independent output evidence; an optional duplicate result field is not required. - Require an empty agent-filtered run history before the first send. Reused agents are unsupported for this case. Validate every agent-scoped activity row and reject mixed or truncated responses; do not filter global history into a pass. - Require the fresh native detail to contain an explicit null `sessionIdBefore`. Missing or empty admission evidence cannot prove a fresh provider session. Keep optional result blobs optional. - Check provider, model, environment, native session, finalization and reset generation. Reject stale replies, duplicate sends, extra runs, missing history and unexpected work. - Permit authenticated chat screenshots through the existing same-origin and mounted-Layout guard. Authentication and unrelated settings pages remain excluded. - Register the case, exact checks, source hash inputs, discovery and documentation. Feature-map files are unchanged. ## Verification - Normal merge commit `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` has exactly two real parents: the staging-qualified chat source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` and lifecycle squash `8771606fd87f69ed4057779e3dce3a9bca8088e6` from #18782. Thirteen dedicated case, support and reused runtime/oracle files are byte-for-byte preserved. The eight shared registry placements match the finite integration forecast. The nine required checks and four-run limit are unchanged. - The integrated source passes 99 focused tests across three files and 925 related Product E2E/catalog/native-reply tests across 42 files. Credential-free discovery lists 22 unique journeys, including chat and lifecycle exactly once. The PR diff against the actual lifecycle squash passes its whitespace check and remains the original 17 chat paths. - The current canonical `node scripts/typecheck-test-projects.mjs --low-memory tests/runner-e2e` passes, including Runner TypeScript and server declaration prerequisites. The DB migration checks and build pass against the actual squash's execution-admission schema and migration 0428. No Rust build was run locally. Canonical generated outputs are bound to current tracked inputs and were rechecked after tests. - The first direct typecheck found a stale generated Runner export, `taskDocumentKeySchema`; the current canonical prerequisites resolved it. A frozen offline install lacked a cached locked tarball, and an initial child command reached the global pnpm version switch and failed signature retrieval. Fetching only locked packages and pinning child pnpm to 9.15.4 corrected the setup. The original diagnostics remain retained; no tracked source was changed to bypass them. - The unchanged chat implementation passed a genuine staging campaign, workflow 41725181387, at source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1`: all nine checks, all five phases, three provider turns plus one synthetic reset, all four usage IDs, and closed cleanup. This evidence remains bound to that original source. The new integration head requires its own current gates and coordinator qualification decision; this PR does not enroll anything in the daily plan. - The earlier staging workflow 41701828968 remains failed. Its nine behavior checks passed, but private collection rejected the valid reset's capture of the old session and cleanup failed. The private adapter fix and evidence-first recovery were handled separately. The original failed grade and archive are preserved. Reset prior-session capture is distinct from the third native turn's explicit-null fresh admission. - Three missing/undefined/empty fresh-admission controls reproduced false passes on `247827a066af01fa583ff030df79b61cb2c87730`, then passed after the strict null fix. Four earlier controls reproduced compact-detail, mixed-activity, missing fresh-agent receipt and unscoped-activity problems. Their original failures remain retained. Positive and wrong outcomes still enforce bound reset summaries, committed native state, configured identity and independent rendered output. Board provisioning uses the acting-agent filter; it does not grandfather earlier work by the test agent. - The first broad support attempt passed 861 tests and failed one report-generator check. A separate reproduction showed `listen EPERM` for the tsx local IPC socket before report output; the unchanged support suite passed with normal local IPC permission. An early flow fixture also returned an object where the events API returns an array; the real event reader rejected it, and only that fixture was corrected. Both original failures remain recorded. - Original commit `d6226b30acaf5a2dad28030f5b295ac1dd119565` failed full CI [41653171240](https://github.com/paperclipai/paperclip/actions/runs/41653171240): a CLI child-server runtime-file wait and two 15-second attention-retention timeouts, followed by database setup/cleanup errors. Their exact causes remain unresolved; the failed jobs and logs were retained and not retried. A later pass does not explain those timeouts. - Strict-admission source `055a7fa99e7cb8aad12b85fccf23880f0dc98a3f` failed full CI [41669527762](https://github.com/paperclipai/paperclip/actions/runs/41669527762): private-invitation and pending-shell access fixtures. The retained assertions match the account-identity fixture repair in #18799, now included through master, together with #18798 and #18650. No old-head job was retried. Subsequent source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` passed full CI 41680411215 and current-source review at 5/5. - Current full CI [41743580793, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41743580793/attempts/2) completed successfully on unchanged `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` at 2026-11-04 00:34:52 UTC: 48 successful jobs and one intentional selector skip. Attempt 1 remains failed: three run-bound runners were independently confirmed to have lost Spot capacity, and the two aggregate failures reflected those interrupted lanes. The build and boot health smoke had passed before interruption. Exactly one failed-job recheck was used; no source change or further retry was made. Fresh exact-head Greptile review is 5/5 with no new actionable issues and both review threads resolved. The separate Ponytail assessment says Ship with no must-fix or should-fix item; its optional direct board-row grader control remains documented. Whole-workspace recursive typecheck, full tests and full build were covered by hosted verification and were not repeated locally; the bounded local checks above passed. These gates do not claim that a later master merge tree has been live-qualified. ## Risks The attached target must supply a fresh disposable native agent and durable pre-send ownership for its chat. Its private adapter must collect all four run IDs, distinguish the synthetic reset from the three provider turns, and stop and clean up owned resources after evidence is saved. This case verifies browser reload, reopen and explicit session reset. It does not verify controller restart. Unknown setup or provider cost remains unknown. No shared QA source agent, model setting, production behavior or budget policy is changed by this public PR. ## Model Used OpenAI GPT-6 through Codex with reasoning, code execution and repository tools. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> canary/v2026.1104.0-canary.1	2026-11-03 19:20:34 -06:00
DottaandPaperclip	fbb7055b90	fix(runner): keep activity FIFOs out of Codex writable roots (#18886) ... ## Thinking Path > - Paperclip manages agents and their work. > - Runner owns each provider process and its activity channel. > - Codex Plan mode uses a restricted filesystem policy. > - Runner adds its activity FIFO as a writable filesystem root. > - Codex adds protected metadata paths beneath each writable root. > - A FIFO cannot contain those paths, so the Linux helper fails before useful work starts. > - This pull request grants access through the private activity directory and keeps discovery files read-only. ## Linked Issues or Issue Description Refs #18050. The activity channel came from that change. No existing FIFO-mount repair was found. #18853 repairs interrupted preparation recovery; it does not change this provider filesystem policy. **What happened?** A native Codex Plan session can fail while loading AGENTS.md instructions. The filesystem helper tries to inspect `activity.fifo/.aws` and fails with `Not a directory`. This happens before the provider can produce the requested plan. **Expected behavior** The sandbox must load permitted instructions and allow activity-channel writes. Plan mode must still reject workspace writes and private state access. **Steps to reproduce** Use the ignored `read_only_codex_opens_with_runtime_activity` regression in the Runner Rust suite with the source-pinned Linux Codex executable and its matching bubblewrap resource. The fixture calls the actual activity channel producer and policy projection. On the original producer it exits 101 with the FIFO metadata-mount panic. With this change it exits 0. **Paperclip version or commit** Original producer reproduced at 208bda1a0e. The local native artifact is official Codex 0.160.0 for Linux arm64, which matches the source pin. The failing hosted executable version was not independently available from its normal run records. **Deployment mode** Native Runner with Linux Codex. Verification used an isolated local Linux container with no network, credentials, or model call. ## What Changed - Grant writes to the Runner-created private activity-channel directory instead of using its FIFO as a filesystem root. - Keep the lock, descriptor, and helper read-only. Keep the existing network, workspace, environment, and active-run authority policies. - Add actual channel isolation coverage and a pinned executable regression. The latter also checks FIFO bytes, instruction reads, workspace write denial, private-state and synthetic credential-home denial, descriptor and lock write denial, lock removal denial, and ambient-variable filtering. - Document the native sandbox check and the separate fixture-export path. Export alone does not verify the sandbox. ## Verification - Actual producer plus official Linux Codex 0.160.0 and matching bubblewrap: original producer exits 101 with the FIFO `.aws` panic; changed producer exits 0 with all permission checks and exact FIFO output verified. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml runtime_notification --lib --offline`: 8 passed, 2 manual sandbox fixtures ignored. - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml codex_provider::tests --lib --offline`: 30 passed. - `cargo build --manifest-path packages/paperclip-runner/runner/Cargo.toml --bin paperclip-runnerd --offline`: passed. - Cargo format and diff checks passed. - A broader optional local Rust library sweep did not pass in the restricted macOS host. Three filesystem-monitor tests failed and another did not finish; the owned sweep was bounded and stopped. A socket-identity fixture also failed with EPERM. A representative monitor failure and the socket failure were repeated with the unchanged original producer. Their failures are retained and are not claimed fixed by this PR. - The existing ignored `publisher_crosses_real_codex_tool_boundary_without_runner_state_access` fixture was not executed successfully on the final head. Its stale channel-directory denial was corrected after Greptile review. The separate actual-producer Linux sandbox fixture verifies exact FIFO bytes and permission denials; it does not call `record-activity`. - Full local pnpm typecheck/test/build was not run for this Rust-only change. Required technical CI passes on fd4da0bf5e. Greptile is 5/5. The automatic Paperclip Review remains queued; merge is held for that review. No hosted task recovery or provider turn is claimed. ## Risks The writable grant now covers the process-private channel directory. The descriptor remains outside that grant. The lock has an explicit read-only mount inside it. Native sandbox checks verify the lock cannot be written or removed. Existing tests retain authority rotation, stale-context rejection, malformed input bounds, and fixed-path cleanup after endpoint replacement. The retained startup error also mentions a loopback permission error. This change fixes the independently reproduced FIFO-root defect. It does not prove that every hosted namespace capability is healthy. Preconfigured-home and custom policies retain their existing explicit deny rules. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution, and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 18:58:31 -06:00
Devin FoleyandClaude Opus 5.5	ad5cd785b1	fix(wake-queue): keep promoted Retry runs scoped to their issue (#18876) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The wake queue starts the next run for a task when the task's current run releases its execution lock. > - A board Retry saves a receipt with only `{ issueId, taskRetry }`. This receipt has no saved wake context. > - When the release drain promotes that receipt, the promoted context gets `taskKey` from the payload, but it does not get `issueId` or `taskId`. > - The new run takes the task's execution lock, but the database saves it as a company-scoped run with no issue. > - Low-trust workspace preparation then has no issue reference. The run fails before provider work with "Low-trust execution issue is outside the active trust boundary." > - This pull request restores the issue scope of a payload-only receipt during promotion. > - The benefit is that a Retry on a low-trust task runs in its task boundary and does not fail during setup. ## Linked Issues or Issue Description **What happened?** A user clicked Retry on a low-trust task while a run was busy. When the busy run ended, the release drain promoted the saved Retry receipt. The promoted run had `issue_id = NULL`, `scope_kind = company`, and a context snapshot with only `taskKey`, `wakeReason: task_retry`, `wakeSource`, and `wakeTriggerDetail`. Workspace preparation failed with `low_trust_boundary_mismatch` (HTTP 422) in the setup phase. **Expected behavior** A promoted run for a task must keep that task's issue scope. This is true also when the deferred receipt has no saved wake context. Workspace preparation must then evaluate the task's own trust boundary. **Steps to reproduce** 1. Use a task with a low-trust execution policy whose assignee is an agent. 2. While a run on that task is busy, click Retry on the task. 3. Let the busy run end. The release drain promotes the deferred Retry receipt. 4. The promoted run is company-scoped with no `issueId`. It fails in setup with `low_trust_boundary_mismatch`. **Paperclip version or commit** `master` at 51d1d4ce07. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `enrichPromotedWakeContext` (`server/src/modules/wake-queue/domain/context.ts`) now restores `issueId` and `taskId` from the payload when the deferred context does not have them. This is the same rule that the original wake enrichment uses. The function does not overwrite a value that the queued context already has. - I added a domain test for the payload-only receipt and for the no-overwrite rule. - I added a Postgres adapter test (`adapters/promoted-retry-scope.test.ts`). It runs the real release drain with a Retry receipt that has no saved wake context. It checks that the promoted run is issue-scoped and holds the task's execution lock. The database trigger `set_heartbeat_run_scope_kind` derives `issue_id` and `scope_kind` from the restored context `issueId`. ## Verification - Red before the fix: both new tests fail on unchanged `master`. The adapter test shows the incident shape: `issueId: null`, `scopeKind: "company"`, and `taskKey` equal to the issue UUID. - Green after the fix: - `pnpm vitest run server/src/modules/wake-queue/` passes (10 files, 289 tests). - The related suites pass (11 files, 468 tests): `heartbeat/queue`, `heartbeat/run-control`, `heartbeat/run-lifecycle`, `deferred-execution-receipt-work`, `execution-lock-orphan-cleanup`, `heartbeat-comment-wake-batching`, `issue-stale-execution-lock-routes`, `chat-failed-run-retry-boundary`, `task-retry`, `execution-control-work`, and `workspace-recovery`. - Server typecheck (`node server/scripts/typecheck-layers.mjs`) passes. - I did not run a live low-trust Retry on a deployed stack. The adapter test uses the real release drain, the real database trigger, and the persisted receipt shape that `requestTaskRetry` writes. ## Risks - Low risk. The change only adds `issueId` and `taskId` when they are missing and the payload has them. Deferred wakes that already have a queued context do not change. - The release drain still promotes a board Retry receipt directly. It does not go through `resumeTaskRetry`. This PR does not change that path. A promoted Retry therefore does not get `forceFreshSession` or `taskRetryRequestId`. This behavior existed before this PR. It is a possible follow-up, not part of this fix. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (code search, shell, and test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 16:57:44 -08:00
DottaandPaperclip	985b175e83	test(e2e): verify saved agent instructions on the current QA harness (#18891) ... ## Thinking Path > - Paperclip lets users configure agents and verify their work. > - Daily QA must exercise saved instructions on the QA staging instance. > - A fresh owned agent can test an instruction change without changing a shared agent. > - The test must prove that saved instructions caused the next run's output. > - It must also prove that the original instructions were restored. > - This pull request brings the qualified settings journey onto the current shared registry layout. > - The benefit is a reusable daily QA candidate with independent evidence and fresh current-source checks. ## Linked Issues or Issue Description Refs #18788. This is the replacement review PR for the same settings journey after integration with the merged lifecycle registry layout in #18782. **What happened?** The settings journey passed its original staging qualification at source898ae8ab4e49d2408688088ce82e3316b91cc09c. The earlier PR's review workspace remains retained for file preservation. That workspace and its recovery evidence remain intact. **Expected behavior** Review the integrated settings source through this PR's own normal CI and fresh reviews. The journey saves instructions through ordinary UI controls, verifies their effect on one current native run, and restores the original bytes. **Steps to reproduce** Use the documented dedicated agent-settings-persistence campaign. The provider must supply a fresh owned agent and the agent-settings-write capability. Save the instruction edit, reload it, complete the ordinary assigned task, verify its independently saved result and current-run attribution, then restore and verify the original instruction entry. ## What Changed - Add the settings browser journey, its independent evidence grader, fixtures, retained evidence contract and focused controls from #18788. - Require one fresh dedicated agent. Select its exact configured instruction entry through the file tree. Confirm both board saves through persisted source and audit evidence. - Require one committed native run on the configured provider, model and environment. Keep the synthetic instruction reference out of the task request. - Restore the original instruction bytes through the same ordinary UI and verify the saved result after reload. - Normally merge actual lifecycle squash8771606fd87f69ed4057779e3dce3a9bca8088e6 once. Place only settings registrations at the documented distinct anchors. Preserve all eight dedicated source/oracle/fixture/docs blobs from qualified898 exactly. ## Verification - Current integrated head: 330 focused checks pass across eight settings, catalog, contract, observation, result/report and operations test files. - Current E2E types pass: `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations tests/runner-e2e`. - Declaration reuse was bound to all11091 matching package/script/server/UI/root inputs, installed lock and TypeScript, plus all source and target generated-output hashes. No source or module-resolution links were copied. All copied output hashes remain unchanged after local checks. - Maintained product-journey discovery passes and includes both lifecycle and settings. The integrated catalog has22 entries, including the separate fresh-cloud onboarding case. - Static preservation checks pass for seven shared registry ASTs after only declared order changes. The settings dispatch body, required seven checks, one-run bound and all dedicated implementation/oracle blobs remain intact. - Historical staging evidence: [QA workflow41689186453](https://github.com/paperclipai/paperclip-cloud/actions/runs/41689186453) passed all seven settings checks and all five phases at source898ae8ab4e49d2408688088ce82e3316b91cc09c. This is the original qualification receipt. It is not a new run of this integrated head. - This PR must earn its own current-head full CI, fresh5/5 review, resolved threads and terminal passing normal check contexts. Those checks are pending. No prior review or check status was copied. - Full workspace checks and a new live QA/model run were not run locally. The coordinator owns current-source acceptance and daily-plan admission. ## Risks - The journey has one paid provider run and needs a fresh dedicated agent. It rejects a campaign that shares this agent with another paid journey. - Browser saves and restoration must persist before navigation. Wrong, missing, foreign or incomplete evidence fails the independent grader. Failed or active work is preserved for normal owned cleanup. - Registration and hash-input order changed during integration, so the definition/source identity changes. The historical QA receipt remains bound to its original source. - The retained earlier review workspace is still held for file preservation. This PR does not recover, reset, replace or destroy it, and submits no turn to it. - No product runtime, permission, model, shared agent, configuration, feature map, QA pin or TestOps-plan changes are included. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. The host does not expose the exact deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub issue/PR references; historical QA workflow is linked as qualification evidence) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 18:55:49 -06:00