2314 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	6364fa7327	fix(heartbeat): wait for retry predecessor cleanup (#18989) ... ## Thinking Path > - Paperclip manages agent work and preserves its task ownership. > - Heartbeat admission controls when a queued run can start. > - A failed run can still own workspace saving and environment cleanup. > - Automatic recovery can transfer the task execution lock to its queued retry before that cleanup finishes. > - The old queue check then sees the retry itself and misses the predecessor's remaining work. > - This pull request checks the exact retry predecessor before claiming the queued run. > - Existing cleanup notifications can then resume that same run through normal admission. ## Linked Issues or Issue Description Refs #18898 and #18691. The first repair covers canonical connected-task retries with a native source. The second covers live remote workspace writers. This change covers ordinary automatic continuation when the predecessor failed before native runtime selection. The search found no open repair for that path. **What happened?** A preparation failure saved a terminal legacy run row while its remote workspace save and environment cleanup were still active. Generic continuation created one queued successor and moved the task execution lock to it. The queue checked that current lock but did not check the successor's predecessor. The new run started and failed at the existing unfinished-writer guard. **Expected behavior** Keep that same successor queued until its exact predecessor finishes execution and cleanup. Check company, agent and task identity before using retry provenance as a wait condition. Preserve Stop, reassignment, failed cleanup and independent recovery gates. **Steps to reproduce** 1. Save a failed legacy preparation run with an active environment lease. 2. Queue its ordinary automatic continuation and bind the task execution lock to the successor. 3. Claim the queued run before the predecessor finishes cleanup. 4. The old code starts the successor. The fixed code retains it in the queue. 5. Commit successful cleanup through the existing cleanup owner. The ordinary dispatcher starts that same run once. **Paperclip version or commit** The causal regression fails on master `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** Source checkout on macOS arm64 with disposable PostgreSQL, the assembled heartbeat service and the production work coordinator. The adapter boundary is controlled. No hosted provider replay or deployment is claimed. ## What Changed - Check the current task execution owner and the retry's exact predecessor under the existing task claim lock. - Use the saved native task binding, ordinary task binding and historical context fallback in that order. Ignore unrelated retry provenance. - Reuse the existing live-executor and pending-cleanup checks. Reuse committed cleanup notifications and the existing dispatcher. - Add real PostgreSQL claim controls and assembled cleanup-event acceptance. Keep the original source failure, saved files and current task ownership intact. - Document the queued predecessor-cleanup rule. ## Verification - RED: the unchanged queue claimed the successor and wrote a new controller lease while its legacy predecessor retained an active remote environment lease. The regression expected the successor to remain queued. - GREEN: the same regression passes after the change. The complete queue suite passes all 71 tests. - The assembled heartbeat suite passes all six tests. The added cases cover one continuation after committed cleanup, saved-file reuse, final task lock release, board Stop, reassignment and failed cleanup. - Scope controls cover a deleted source after scan, another company, another task, another agent, canonical native task binding and the existing current-owner hold. A failed cleanup and a still-live local executor retain the wait. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. - Initial fixture attempts failed on database constraints and observer timing. The final fixtures use valid immutable bindings, the board Stop attribution and a completed real dispatcher call after reading the exact committed lease state. Production ownership guards were retained throughout. - Full low-memory repository typecheck passed at `5cddb49322`. The sync to master `04a24f4e06` adds only the independent retained-allocation test fixture from #18984; production code is unchanged. - Full `pnpm build` passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`. - The post-sync run executes all 80 behavior assertions successfully across the queue, assembled cleanup and updated upstream retention suites. Its queue `afterAll` then exceeds the existing 30-second database shutdown hook. That failed suite result is retained. The controlled post-build queue rerun passes all 71 tests with normal teardown. The earlier shutdown cause remains undetermined. Its owned queue-test PostgreSQL process has exited. The separate local default `pnpm test:run` was intentionally stopped after full unfiltered CI passed on the exact head. Four local chat partitions completed before that stop; this is not a local full-suite pass. The verified owned process group and its PostgreSQL fixture have exited. Logs and retained fixture data remain available. The timeout has not been increased. - Full unfiltered CI passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all six internal chat partitions, all nine serialized server shards, the workspace groups, native Runner lanes, typecheck, build and browser E2E. Both required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146) pass. The file selector is skipped, so this is the full CI test matrix. The head has 55 successful checks and three expected skipped checks. The ready-for-review contributor trust and security scan also pass. - Current-head Greptile is 5/5, its check is successful, and no inline review threads remain. The ordinary review check also passes. No merge, deployment or original task retry was performed. ## Risks - A predecessor with unfinished cleanup keeps its successor queued. Its existing cleanup owner remains responsible for settlement. - A task claim reads at most one additional predecessor row. A matching source also uses the existing ownership and cleanup queries. The change adds no timer, schema, credential, provider or model policy. - A committed cleanup signal only requests normal admission. Current ownership and independent execution gates still apply. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL verification. The session does not expose a more specific serving model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused suites pass; the separate local full-suite attempt is explicitly incomplete above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 11:16:54 -06:00
DottaandPaperclip	04a24f4e06	test(recovery): align retained allocation fixture with readiness (#18984) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Restart recovery must preserve the original remote allocation and its files. > - #18934 readies that same retained allocation before preparation. > - The older #18936 failure fixture rejects every readiness call. > - It stops before the intended preparation-failure boundary. > - This change updates only that fixture and checks the original allocation identity. ## Linked Issues or Issue Description **What happened?** The three remote preparation retention cases fail after #18934. Their driver stub throws `Reattachment must not resume the allocation`. The production code now calls readiness on the same validated lease before preparation, so the test never reaches its intended execution failure. **Expected behavior** The fixture should allow readiness of the original allocation. It must reject replacement acquisition and retain its preparation-error, ownership, saved-file and retention checks. **Steps to reproduce** Run `src/__tests__/heartbeat-native-remote-failure-retention.test.ts` on master `86814c6550`. All three cases stop at the stale readiness stub. Refs #18934 and #18936. This one-file baseline correction is split from #18970 so it can unblock the shared failure also observed on #18977. It carries no core API change or evaluation source pin. ## What Changed - Return the original lease from the fixture readiness stub. - Assert exactly one readiness call for the original company, environment, run, workspace, provider allocation and remote path. - Assert that no new allocation is acquired. Keep all existing preparation failure, ownership and retained-work assertions. ## Verification - Before #18934, the three existing cases passed locally. On the actual merged #18934 contract, they reproduced the three CI failures in 18.06 seconds. - With this fixture correction, all three passed in 11.83 seconds. Production runtime code and the original 60-second case deadline are unchanged. - The corrected test blob and its relevant production/helper blobs match the already-tested merged-source proof. Current master still has the same stale test blob; this isolated PR applies only the identical fixture correction. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/heartbeat-native-remote-failure-retention.test.ts`. - `git diff --check` passes. Full repository typecheck/build were not rerun for this one-file fixture split; fresh normal CI and review are pending. - [Original failing CI job](https://github.com/paperclipai/paperclip/actions/runs/41855440307/job/126665077718). It remains failed; no remote retry was used as proof. ## Risks - Low scope: test fixture only. A readiness call for a replacement or wrong allocation fails the identity assertions. - This does not alter production recovery, permissions, cleanup policy, evaluation deadlines or grading. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing and tool execution. The host did not expose the model context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 10:15:31 -06:00
DottaandPaperclip	86814c6550	fix(recovery): ready retained sandboxes before preparation (#18934) ... ## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 09:04:13 -06:00
DottaandPaperclip	84e9c8f364	fix(retry): wait for admitted source cleanup before dispatch (#18898) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. **What happened?** A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. **Expected behavior** Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. **Steps to reproduce** 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. **Paperclip version or commit** Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. **Deployment mode** The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:33:39 -06:00
DottaandPaperclip	5d226ca08a	perf(chat): reuse bound-run authorization query templates (#18966) ... This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. **Current behavior** At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. **Proposed behavior** Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. **Reason and benefit** Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:32:31 -06:00
DottaandPaperclip	2cbfc35a07	perf(github): reuse the inherited review wait query template (#18965) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. **Subsystem affected** Server GitHub chat review waits. **Current behavior** Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. **Proposed behavior** Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. **Reason and benefit** Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 08:29:59 -06:00
DottaandPaperclip	0eaabe5c0b	refactor(accounting): remove company serialization from paid settlement (#18917) ... Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-04 07:29:28 -06:00
DottaandPaperclip	669538534c	fix(recovery): preserve failed remote reattach workspaces (#18936) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs can edit files in a remote execution workspace. > - A failed reattach can leave an observed coordinator and unpublished files. > - The failure handler recognizes the older workspace-sync descriptor but misses the modern workspace reference. > - The released task lock then prevents ordinary session recovery from claiming that old run. > - This pull request retains the files through the existing Stop, cleanup and workspace recovery owners. > - Exact ownership and Stop evidence permit file preservation while unknown provider effects remain under manual reconciliation. ## Linked Issues or Issue Description Related: #18877 protects native manual admission holds. #18879 handles scheduled native retry lock reclaim. #18918 fixes remote workspace row-lock order. Those changes do not record the missing modern failure-retention obligation. **What happened?** A native reattach failure can leave a failed run, a dirty modern remote workspace, an expired observed coordinator and a free task execution lock. The run has no accepted result. The failure handler skips file retention because it checks only the older descriptor. Recovery cannot export while the old coordinator remains owned. A mismatched workspace generation can also make orphan cleanup miss the existing file claim. **Expected behavior** Retain the exact original allocation and files. Stop the allocation through the existing lifecycle owner before retiring the stale coordinator. Keep checkpoints and unknown external effects available for manual reconciliation. Unsafe or changed source authority must prevent physical cleanup. **Steps to reproduce** 1. Prepare a modern remote workspace for a native run. 2. Record a provider checkpoint and an active interruption incident. 3. Fail reattach preparation and release the task execution lock. 4. Let the coordinator lease expire. 5. Run the existing orphan, environment cleanup and workspace recovery workers. 6. Verify the original files from a cold archive read and verify that no new native session starts. ## What Changed - Include modern prepared references in the existing heartbeat failure-retention producer. - Add a file-only historical intake for an exact failed observed run in the normal expired reattach state. - Require an untouched active ephemeral lease, a free blocked task, matching workspace writer and generation, no accepted result and complete single-owner allocation history. - Persist the existing StopAndRetain intent before provider I/O. Use nonblocking row locks and classify busy ownership only after rollback. - Use the existing native environment settlement owner after exact Stop and retention receipts arrive. Keep its single scoped discovery query. - Preserve source and coordinator fingerprints across the handoff. Keep original checkpoints, errors and manual unknown-effect incidents. - Keep stale or malformed modern workspace claims held when preservation intake refuses them. - Add actual PostgreSQL, heartbeat, allocator, worker restart, archive and safety controls. Document the narrow recovery contract. ## Verification Current published head `b1f49691187a398415cf9b16b723e3ab473f0f91`: all required CI checks pass (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with no open threads. The separate Paperclip assessment is queued and its current response says work is paused. That assessment is unavailable; no additional review nudge or hosted retry was issued. All local commands use Node 24.22.2 on macOS arm64. - Exact published head: `b1f49691187a398415cf9b16b723e3ab473f0f91`. Required CI passes (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with zero open threads. The optional Paperclip assessment remains queued/paused without a completed score. - Final recursive `pnpm -r typecheck`: passed. - Final `pnpm build`: passed. - Final three native producer, containment and scheduling files: **67/67 passed**. These include the real coordinator and worker restart, the production heartbeat recovery callback, exact Stop receipts, cold archive restore, and zero native backend, new allocation or destructive release calls. - The plugin allocator cases use the actual selector with `reuseLease: true`. Active and released ephemeral leases receive a fresh allocation. - The assembled reattach producer checks that the task and coordinator claim commit before physical resume. A historical pin refuses that current owner. A later claim behind a committed pin remains held. - The safety controls cover reusable allocations, manual and unknown states, cleanup in flight, accepted but unbound results, late allocation owners, wrong or malformed workspace authority, changed generation/checkpoint, lost Stop acknowledgement and a real cleanup-producer lock conflict. - Untouched base `e7549ab6131555f655736e6b544892a3b1e323cb`: four expected behavioral failures and one passing legacy-descriptor control. The failures cover the modern producer, original provider history, historical retention and wrong-epoch cleanup. Fixed production files were restored byte-for-byte after this comparison. - The first affected six-file run completed **562 passed / 1 failed**. The failure was an existing HTTP case with `socket hang up`. That exact case passes separately on both untouched base and this branch. The original connection failure remains unexplained. At `0e92de8c`, the complete workspace family passes **188/188** with unchanged assertions. The environment cleanup owner family also passes all 23 cases. The later guard changes are qualified by the complete 67-case native owner group. A full repository test run was not repeated locally; the requested full-ci workflow runs the complete repository suites. - Published `0e92de8c` fails both new explicit-null Stop/export marker controls because it creates a new retention pin. The follow-up uses the shared presence-aware Stop guard. Both controls now preserve the marker and record no new retention pin, Stop or destructive release. - Paperclip's review found that malformed workspace authority could throw and abort an orphan page. Published `5f6cd4f5` fails all three new controls: intake, the actual page and after-Stop settlement. The same strict schema now uses a safe parse, and the existing OBSERVED hold runs before the prepared-source lookup. The malformed source stays held while the next valid source receives its retention pin. - Two existing scheduling tests caught the draft's second global discovery query. The final single-query repair passes all 36 unchanged scheduling tests. - An earlier restarted archive assertion stopped after 20 seconds. Its recorded Stop was valid, and the coordinator had settled. The existing evidence recheck deadline was still 13 seconds ahead. The final test observes the unchanged 24–36 second policy within its existing 60 second case limit. The external fake provider Stop and file transport are controlled test boundaries. Database ownership, lifecycle workers, archive bytes, encrypted local storage and cold restore are real. This does not claim a live cloud sandbox, S3 or model run. No hosted task was changed for this PR. ## Risks - Historical intake is deliberately narrow. Reusable, already released, mismatched or competing allocations remain held. - File preservation does not settle unknown provider effects or authorize a repeated provider turn. The task can still require manual reconciliation. - A busy or changed owner delays preservation. The original files and claims remain retained. - This does not repair the underlying sandbox readiness failure or establish the cause of a separate archive-capture failure. - Deployment and recovery of original hosted tasks remain unverified. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window size. A separate Codex agent and the coordinator performed read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:19:03 -06:00
DottaandPaperclip	b6352578c5	fix(task-retry): retain chat Retry during identity contention (#18896) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry must preserve a person's request and enforce the current chat authority. > - Chat retry preparation takes nonblocking source locks before it can authorize the saved source. > - Sustained lock contention could discard the Retry request before it was saved. > - This pull request saves one unadmitted request in the existing chat retry queue. > - The worker checks the source and current access before normal admission. The task shows a scheduled retry while it waits. ## Linked Issues or Issue Description **What happened?** Task Retry returned HTTP 500 when another transaction held the saved chat principal's identity lock or the endpoint authorization row. The source preparation error rolled back the canonical Retry intent. The same gap existed when Retry had to restore a certified input batch that had never reached a provider. With the bounded synchronous retries in #18895, sustained contention returns 409 without a saved intent. **Expected behavior** HTTP 202 retains one Retry request. Acceptance does not grant source authority or mean that a provider started. The existing worker prepares the request if the source locks become available before its saved deadline. It must reject changed input, revoked access, Stop, or superseding work before input adoption, publication, or provider work. **Steps to reproduce** 1. Prepare a failed task with its original admitted GitHub chat source. 2. Hold that principal's `chat-identity` advisory lock or the endpoint row on a second PostgreSQL connection. 3. Send the normal task Retry HTTP request. Check the response and persisted intent. 4. Release the lock. Process the existing chat retry queue and check that one authorized successor is queued. **Paperclip version or commit** Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. **Deployment mode** Source checkout, real Express Retry route and embedded PostgreSQL. External chat transport is a test fixture. The live hosted fix has not been deployed or verified. Related: #18895 handles brief admission contention with bounded synchronous attempts and returns 409 if it remains busy. This change retains a durable request through sustained identity contention. It uses the existing background retry owner and adds real HTTP, restored-input, Stop, publication, restart, and task-presentation coverage. ## What Changed - Preserve a strict pending-source action only when the identity try-lock reports contention or the endpoint authorization query reports lock-unavailable through an explicit Retry callback. Bind it to the canonical task Retry, company, task, agent, endpoint, conversation, and principal. Other database and authorization errors keep their existing behavior. - Revalidate the canonical intent, current board permission, saved source, and latest execution before the existing worker can prepare that action. Ordinary authorization rejects an unprepared action. - Keep certified input restoration inside source preparation. Recheck the saved deadline before the prepared payload write. Expiry, a nested savepoint, or outer rollback cannot commit partial restoration. - Give source authorization contention one saved 60-second deadline and a separate counter. Keep its waiting phase until current source authorization succeeds or the action terminates. It does not spend the eight delivery attempts. Duplicate clicks, worker restart, and background sweeps do not renew either limit. Reuse the existing work notifications and outbox deadlines. - Persist completion of the authorization wait under the current claim before delivery. A restart revalidates current authority. Generic delivery failure preserves the stored phase even when a commit acknowledgment is lost; it does not grant a wake or renew the saved deadline, counters, or claim. - Project pending preparation and prepared but unsettled actions as the existing scheduled-retry state with the actual outbox deadline. Keep the exact task, intent, agent, and source binding. Active execution keeps its normal working state. No UI or API enum change is needed. - Add regressions and document the admission boundary. ## Verification - RED: the original real HTTP route returned 500 under a held PostgreSQL identity lock. The restored-input case also returned 500 while its 60 sibling source and budget cases passed. The current endpoint-row case returned 409 without a saved intent under the bounded synchronous fallback. - RED: the worker failed a valid Retry during an actual nine-second PostgreSQL lock. Removing only the latest-run guard makes the new queued-run negative fail. Source preparation could also commit restored input after its deadline, and an already prepared action could process after a late lock release. The prepared but unsettled task view lost its scheduled state. - GREEN: all 91 selected real PostgreSQL integration cases passed. Coverage includes the 62-case saved-input matrix; coalescing; restart; one real heartbeat successor; source, actor, Stop, newer-work and publication refusals; a real nine-second lock; fixed-deadline exhaustion; the separate eight delivery attempts; certified restoration rollback on expiry; prepared-action early and late release; a real conversation-row wait that crosses the deadline during current authorization; endpoint-row contention; unknown source; an already-revoked actor; unchanged unrelated ingress errors; durable-work idle settlement; actual outbox presentation deadlines; and recovery after authorization commits, with Stop and a lost commit acknowledgment. - GREEN: all 75 cases in `task-retry.test.ts`, `task-interruption-presentation.test.ts`, `chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts` passed without a name filter. - RED → GREEN: stopping after source authorization committed left a stale waiting flag, so a fresh worker failed a previously authorized Retry after the deadline. The fix saves that phase in the same transaction. A real PostgreSQL commit followed by an injected lost acknowledgment also reproduced generic settlement restoring the stale flag. The corrected settlement preserves the stored phase. No wake is sent on the ambiguous response. Recovery queues once after current checks; Stop still prevents a successor. The deadline, contention count, original input, failed run, and publications stay unchanged. - Module boundary checks pass. Full workspace typecheck and the full build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this exact head, both required checks (`ci / verify` and `ci / e2e`) pass. The inventory has 53 successful, three skipped, and one neutral check, plus the automatic Paperclip Review in progress. The legacy Snyk status passes. Greptile is 5/5 on this head, with no new comments or unresolved threads. A current Paperclip assessment has not been published yet. - The full local `pnpm test:run` did not pass: its second chat shard hit a 15-second timeout in the HTTP/restart test after 348 and 293 passing cases. One bounded diagnostic of the same 642-case shard pair passed, with that case completing in 606 ms. The matching Linux CI pair passed too. The original timeout remains unexplained and nonreproduced. No timeout changed. The earlier stopped run was incomplete, and both conflicting documentation additions are retained. - CI chat 3/3 exposed a separate fixture assumption: restoration can commit before a durable wake receipt exists. One known pre-receipt delivery failure reproduces the same missing-run error. The corrected fixture drives the existing worker until the exact canonical intent and wake agree. It proves one real wake, one successor, exact source/actor binding, and one restoration. The exact failed shard pair now passes locally: 635 cases. Workspace typecheck also passes at test-only head `94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is unchanged from the previous verified build. That head completed all registered checks: 53 successful, two neutral, three skipped, plus the successful legacy status. Greptile scored 5/5. The subsequent Paperclip 4/5 persistence finding is addressed by the source repair above; its new exact-head review is pending. - Compatibility: 111 selected cases passed with #18895 head `9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The branch is now synchronized with master `4a83ba4050`, which includes that change. - Preserve the old CI failure: at head `37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially failed the empty organization-audience replacement versus member-removal assertion in `tool-access-service.test.ts`. The exact race family passed on unchanged base and head; all 501 file tests and two 30-case controls passed on base. The one Linux replay passed 119 files and 2,833 cases, with three skipped. No guard or assertion changed. The initial failure remains unexplained and nonreproduced; this PR does not claim to fix it. - The prior external Superagent scan could not start because its Security Graph returned HTTP 503. Its neutral check was not a completed security assessment. - These tests exercise the real Express route, PostgreSQL transactions, and heartbeat queue. External chat transport is a fixture. They do not run a live model or prove recovery of an original hosted task. ## Risks - A still-pending source authorization wait fails after its fixed 60-second contention deadline. A completed wait is saved, but new contention keeps the original deadline. A healthy lock can exceed that limit. Other delivery errors retain the eight-attempt budget. Current authority checks remain required before admission. Inspection does not reset either limit. - The new pending phase must never be treated as an authorized chat source. Normal authorization rejects it until the worker prepares and binds the exact source. - The task projection reads the existing linked outbox deadline. It adds a scoped join and does not create a timer or change a business deadline. - No schema migration, provider identity change, or new delivery queue is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, tool use, and test execution. The host did not expose an exact deployment snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required Paperclip CI gates are green (`ci / verify` and `ci / e2e`; automatic review progress is recorded above) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:48 -06:00
DottaandPaperclip	c5251b4c0e	fix(chat): retry fully recorded setup failures before dispatch (#18882) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected tasks retain their exact external request across failures. > - Task Retry must preserve that request and prove that old execution cannot continue. > - A workspace setup failure can settle before any provider dispatch. > - The chat retry guard accepted one specific validation failure but rejected other fully recorded setup failures. > - This pull request reuses the same strict server settlement proof for those failures. > - The benefit is one safe continuation through the existing queue, with all source and execution gates intact. ## Linked Issues or Issue Description Refs #18803, #18845, #18852, #18853 and #18860. These changes preserve workspace writers, bind completed preservation, or repair other preparation and native continuation paths. This change repairs the separate connected-chat guard for fully recorded legacy setup failures. Search also found #18876, which repairs issue scope during deferred wake promotion and does not repair this guard. **What happened?** A connected task failed during workspace reservation before provider dispatch. Its source remained authorized. Task Retry saved an intent and chat action, but the chat worker rejected that request as `failed_chat_retry_not_authorized`. No successor was created. The old failure had a bootstrap marker, a dispatch-only Runner profile, the matching server error and durable measured settlement events, and no runtime, process, output, result or environment lease. **Expected behavior** A deliberate board Retry can continue the exact saved input after the complete never-dispatched proof passes. An unfinished earlier writer, changed source, accepted result or unknown execution state must still prevent execution. **Steps to reproduce** 1. Admit an authorized request through a restricted chat endpoint. 2. Settle setup before provider dispatch, for example while an earlier workspace writer is unfinished. 3. Keep the exact server error and durable measurement events, a dispatch-only Runner profile, and no execution resources. 4. Press task Retry after the independent writer and ordinary admission gates permit work. 5. The original guard saves the Retry action but refuses to create a successor. **Paperclip version or commit** Reproduced on master at `a00109aa35`. The branch is rebased on current master. **Deployment mode** Hosted task diagnosis. Local proof uses real Express HTTP routes, chat authorization, heartbeat queue admission and disposable embedded PostgreSQL on macOS arm64, Node 26.4.0. Provider work is deliberately prevented by occupied fixture capacity. ## What Changed - Extend only canonical task Retry's existing strict pre-provider preparation proof to generic server-settled setup failures. - Keep exact result shape, dispatch profile, controller, runtime, process, output, event and resource checks. No diagnostic message allowlist is added. - Exercise GitHub and Telegram HTTP Retry with the original sender and saved input. Keep the failed run unchanged and create at most one queued successor. - Cover provider events, retained native state/results, modified source, unknown result metadata, live controllers, process identity, source leases and an independent unfinished writer. - Document the guarded contract and correct the obsolete access-map sentence. No new endpoint or agent permission is added. ## Verification - Red before repair: both setup causes failed through the real HTTP/chat/queue path because the durable Retry had no successor. An initial retained-native negative fixture had an invalid foreign-key setup; that fixture was corrected before the final run. - Green: 37 focused integration cases pass, including 22 new GitHub/Telegram cases and 15 existing pre-provider source and execution guards. - Current head `1bf69d20e02a049248ec378f087505563daeb9a0`: all 37 focused integration cases and full repository `pnpm -r typecheck` pass after the master rebase. The full `pnpm build` also passes. - Map structure, module boundaries and `git diff --check` pass. - Full local `pnpm test:run` is running; all six chat shards passed and the remaining server suites are progressing. This is not a full-suite pass claim. - Greptile and Paperclip Review both reviewed this exact head at 5/5 with no actionable findings or review threads. - Initial CI passed all jobs except the unchanged Runner package suite. Its `runtime-host.test.ts` approve-reads case failed before its intended assertions because a managed credential home could not acquire its loopback-port quorum. The exact case passes locally. A wider 121-case check hit the same lease error in a different Pi case (120 passed); all 121 pass with one worker. The exact competing port owner is unknown. One unchanged retry of only failed CI jobs passed, including the Runner package suite and aggregate `ci / verify`. All current-head required CI gates are green; the original failure remains in workflow history. - The dependency setup initially missed the embedded PostgreSQL package's dylib symlinks. Running its own postinstall helper fixed the local setup before assertions. This was not a product test failure. - This change has not been deployed. The original hosted task has not been replayed with the fix. ## Risks - Never-dispatched proof must remain strict. A bootstrap marker or empty process fields alone cannot authorize continuation. The regression cases retain contrary-state holds. - The inherited session display hint remains history and supplies no resume authority. - Current connection, principal, source integrity, pause, budget, approval, dependency and workspace-owner gates remain in their existing owners. - No schema, provider profile, trust policy, model or credential permission changes are required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL/browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:18:40 -06:00
DottaandPaperclip	e54f9fbf82	fix(secrets): expose secure human entry in native discovery (#18927) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Some tasks need a company credential or the current person's credential. > - Paperclip already has secure human entry forms for these values. > - Native connection guidance did not distinguish storing a credential from connecting an external service. > - Native API search also omitted the existing secure entry route from secret operation help. > - This change makes that route discoverable while preserving human and owner authority. ## Linked Issues or Issue Description Refs #18780. Related #18921 corrects a separate observer false positive. Neither PR owns this production guidance change. **What happened?** A request to store an existing company or personal credential could lead to an unnecessary external provider question. Native API search did not explain the existing human entry surface. **Expected behavior** An agent can find the existing secure form, direct the human to the matching entry, and verify the result through its authorized interfaces. The agent must not ask for values in chat or act as the human. **Steps to reproduce** Search the native API for a company credential or personal publishing credential. The result should name the supported Secrets settings route and its human authority boundary. Open a known personal definition link and confirm that the existing Set my value action appears. ## What Changed - Distinguish Paperclip credential storage from external connection discovery in shared native guidance. - Describe the existing Secrets settings route, secure entry actions and current-person boundary in OpenAPI help. Qualify definition detail links as admin-only; other members use My secrets and Set my value. - Exercise discovery through the actual OpenCode MCP bridge and native API catalog. - Retain agent denials for personal definition and value routes. Exercise the real Secrets UI definition link. ## Verification - Focused local checks: four suites, 78 tests passed. These cover native discovery, secret-route permissions, shared guidance and the Secrets UI. - The new discovery test fails against unchanged master OpenAPI help. It passes with this change. - Token gates and diff checks passed. - Full workspace typecheck passed. The first attempt found missing SDK links after script-free installation; the existing root postinstall helper restored them. An unprivileged retry also hit the local tsx IPC sandbox restriction. The corrected owned setup passed without source changes. - Full local workspace build passed. The full local test run stopped in chat shard 2: one Discord receiver-lease test failed during initial configuration. That shard passed 263 tests; the first shard passed 310. The test and owning lease/configuration sources are unchanged from master. One focused isolation passed without code changes. Its cause remains unknown; this is not a complete local test pass. No unrelated Discord code was changed. [Normal hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41773266687) passed on exact head `0abb8b722779368109b9639f68122d7df02a8ab1`. - Fresh Greptile review reports 5/5 on the exact current head; all review threads are resolved. Paperclip Review remains in progress. The optional Superagent scan is neutral/inconclusive: Security Graph context retrieval returned HTTP 503 and assessment did not start. It supplies no security PASS credit. - A retained representative native attempt with this guidance produced the correct definition-scoped secure-entry link. Its observer then rejected an explicit warning against chat entry. That failed result remains unchanged. Human entry, saving the value and worker delivery were not exercised. This PR claims no complete product-eval pass. ## Risks This changes agent guidance and search ranking. It grants no new permission and changes no secret resolution or owner checks. Broader service-connection discovery still applies when a task needs a connection. The secure human forms are unchanged. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with repository tools, code execution and test analysis. The host does not expose the context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — the 78 owning controls pass; the broad local failure remains disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — normal CI passed; optional review states are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:01:57 -06:00
DottaandPaperclip	5b50c6b32c	fix(teams): return saved resource links after catalog installation (#18923) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Catalog installation creates agents, projects, and routines in one company. > - The install result includes saved IDs but no links to those resources. > - A chat agent can describe a partial installation without giving the operator a way to inspect it. > - This change returns canonical links for saved resources and documents them for API discovery. > - The operator can use the returned links to inspect the actual installation. ## Linked Issues or Issue Description **What happened?** A catalog install can create the intended resources and report skill warnings, while its response leaves callers to construct board URLs from IDs. A native chat attempt described the resources and warned about an incomplete skill, but omitted resource links in both its initial and repeat replies. **Expected behavior** The installation response should provide canonical company-scoped links for its saved agents, projects, and routines. It must preserve warnings and omit links for rows without saved IDs. **Steps to reproduce** Install a catalog team through the API. Inspect its saved resource rows and response. Before this change, the response has IDs and names but no resource links. **Paperclip version or commit** Parent `e7549ab613` from master. Related #16495 changes comment authorization metadata in the same OpenAPI file; it does not address team installation responses. No duplicate team-link PR was found. ## What Changed - Add typed `resourceLinks` entries with resource kind, saved ID, name, and canonical company-scoped href. - Read the company prefix with one bounded primary-key query before import. Encode URL segments and use saved IDs, including IDs returned for reused resources. - Preserve the portability import result, skill preparations, warnings, permissions, and collision behavior. - Document the response and add service, native API wire, OpenAPI, and real database regression coverage. ## Verification - Parent service regression: 1 failed, 28 passed. The new failure was the missing resource-link response. - Owning service/native wire, route authorization, and OpenAPI suites: 72 passed. - Real PostgreSQL catalog-install suite: 4 passed. Returned links match persisted agent IDs and the company prefix. - Shared and server TypeScript checks passed. Fresh shared/SDK builds and the owning Runner TypeScript prerequisite passed. - UI typecheck, Storybook catalog check, 30 catalog/map controls, and token gates passed. - Actual CI found an omitted required `resourceLinks` field in the existing Storybook install fixture; that fixture is now synchronized. The earlier response-status review correction is also retained: install schema and wire control use actual HTTP 201. Final head `0bca72cf1f7641fd842a04c5a75cae4bcc274f05` passed all 48 normal CI jobs, including `ci / verify`: [run 41773413940](https://github.com/paperclipai/paperclip/actions/runs/41773413940). Greptile is 5/5 with zero unresolved review threads. The optional Paperclip review remains queued and its comment says work is paused. The Superagent security scan is neutral; neither supplies completed assessment credit. - `git diff --check` passed. - Setup failures were retained: incorrect test working directory, sandbox listener denial, missing built SDK/Runner declarations, and native library aliases not hydrated from the package directory. Corrected setup used the owning build and shipped hydration script. No assertions or deadlines were relaxed. - Repository-wide tests/build were not run locally; the full normal CI suite passed. A new native chat attempt has not run for this change. The earlier chat attempt remains failed for missing links. Local response coverage does not prove that a model will include the links in its next reply. ## Risks - This is an additive JSON response field. Manually constructed TypeScript install results must include the new typed field. - The query adds one company-prefix lookup per installation. It selects one scalar by company primary key and runs before mutation. - Supplying links reduces URL construction work but does not guarantee model use. Native qualification is still required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing, and tool execution. The host did not expose the model context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All required normal CI gates are green; optional incomplete/neutral assessments are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 07:01:08 -06:00
DottaandPaperclip	84aa989bf9	Preserve native sessions through legacy workspace enrollment (#18943) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 04:43:56 -06:00
DottaandPaperclip	612d9931a6	Support per-turn Runner sessions on ephemeral web servers (#18932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The e9aae9cf revision passed all 57 successful checks, with 5 skips and 2 neutral checks, and exact-head Greptile 5/5. The latest 9cbec30a4f follow-up changes only E2E catalog/admission controls; its 291 affected tests pass, all 57 check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile is 5/5 with no open review threads. - Migration numbering and safety checks pass. - A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload bytes across 89,100 archive entries after deleting the source directory. Restored SHA-256 matched; upload/readback took 56.5 seconds, restore 74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources were removed. - Archives preserve contained relative links, reject link chains/traversal, and enforce independent byte/entry/object limits. Offline verification streams hashes. Orphaned deleted-run backups stay on the original volume. Obsolete object retention preserves the selected backup, live lease references, and pending uploads with owners. - Isolated API-key Codex deployment: original active runner/session transferred to the replacement and committed. All 567 reads, 57 writes, and 57 readbacks succeeded. A second replacement completed a same-session continuation with all 287 reads and 29 writes succeeding. The strict cell failed two agent-authored shell commands; this result is retained as failed. - Isolated API-key Claude deployment: original active runner/session transferred and committed. All 584 reads, 59 writes, and 59 readbacks succeeded. A second replacement completed a same-provider-session continuation, with 211 reads and 22 writes succeeding. The strict cell failed an initial completion call that omitted a registered attachment citation; the agent corrected it. This result is retained as failed. - Isolated API-key OpenCode deployment: the same active runner/session transferred and committed. All 481 reads, 49 writes, and 49 readbacks succeeded. A second replacement retained the provider session and completed useful work, with 156 reads and 16 writes succeeding. The strict cell failed agent tool calls: a denied external-directory write, malformed artifact hashes, and a second completion after the first result was committed. There was one submitted/completed turn and one committed result; reconnect notification replay was identical. This result is retained as failed. - Grok's real first turn now completes with settled usage and provider cost ($0.41397); the next turn is admitted normally. That attempt then exposed a shared workspace-path identity regression before deployment. Three cold-restore cases reproduced it before the fix; 106 workspace/transport tests and four focused cold-restore cases pass afterward, including destroyed allocation and occupied-directory preservation. Server typecheck and build pass. At b5b0dbb03f both controller switches passed the read/write probe without failure, and the active runner retained its process while the controller generation advanced. The third turn exposed an invalid reasoning-token subset check; ad07e681ba fixes it with the actual receipt regression. Two later live attempts failed with an upstream Grok rate limit before deployment. A tiny preflight prompt succeeded before the second attempt, but the full workload was still rate-limited. The final live gate remains incomplete. No accounting or identity guard was bypassed. Earlier failed attempts and their exact resource-cleanup receipts remain recorded. - Earlier Codex/Claude/OpenCode receipts predate #18922; they establish their tested revisions. Current-head live qualification remains pending; #18943 separately preserves pre-enrollment logical/provider workspace addresses and adds actual legacy migration proof. Its first Codex migration attempt stopped safely at skill-version verification. - Earlier attempts remain recorded: missing local provider-pack setup blocked one Claude launch, and the observer rejected identical reconnect replay in another. Setup and the observer were corrected before the latest run. - Named staging enrollment has not started. Its serving revision and post-deployment journeys remain separate delivery gates. - Broader E2E harness validation exposed deployment-catalog and installed-lane coverage drift; the affected 291 controls now pass. Twelve other failures reproduce on unchanged base 1a39dea9aa. They concern cleanup call expectations, dated operating-overhead fixtures, a Pi profile-version expectation and workflow-security expectations. No baseline failure is reported as passing. ## Risks - Provider archives can contain credentials. Objects use authenticated encryption with scoped associated data and a wrapped data key. The original stable master key remains required. - Storage failure prevents publication and sandbox cleanup. Operators must keep shared storage available and preserve the previous volume during enrollment. - Legacy state without verified identity cannot become executable recovery authority. Archival preserves evidence and existing failures; it does not recover already-lost provider files or clear blocked tasks. - Local overlap probes establish the tested application behavior. They do not establish Railway routing behavior or live managed-subscription qualification. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution, and live integration tests. The exact runtime variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-04 02:12:51 -06:00
Devin FoleyandPaperclip	1a39dea9aa	fix(heartbeat): keep a routine claim queued while its slot is held (#18933) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A routine creates one task per tick. Only one open task of a routine can hold an execution lock (`issues_open_routine_execution_uq`). > - Since #18879, a failed native run that waits for a scheduled resume keeps its task's lock. Routine dispatch counts only queued, running or scheduled runs as live, so it can queue a newer task of the same routine meanwhile. > - When that newer run is claimed, stamping its execution lock violates the unique index. The raw database error is not a deferrable claim rejection, so it escapes the queue loop. The queue code notes that this can stop the agent's queue and startup recovery. > - This pull request turns that exact violation into a 409 claim rejection, so the run stays queued until the slot is free. > - The benefit is that a busy routine slot delays one run instead of stopping the queue or startup. ## Linked Issues or Issue Description Refs #18879 (keeps a failed run's lock while its resume is pending), #18916 (defers the same unique-index conflict in restart recovery's lock reclaim) and #18931 (open; keeps one run's restart-recovery failure from stopping startup recovery). This is the same index conflict on a different path: the heartbeat queue claim, not restart recovery. It does not change #18916 or #18931. Its doc paragraph is separate from the one #18931 edits. **What happened?** While auditing the lock paths from #18879 for this index, I found that a queued run of a newer routine task can be claimed while an older task of the same routine still holds the slot through a failed run with a pending resume. The claim's lock stamp then raises `duplicate key value violates unique constraint "issues_open_routine_execution_uq"`. `isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error escapes `startNextQueuedRunForAgent`. **Expected behavior** The claim is rejected as a conflict and rolls back. The run stays queued and is claimed after the older task releases the slot. Other errors behave as before. **Steps to reproduce** 1. Create two open tasks of the same routine (same origin and fingerprint). 2. Give the older task an execution lock held by a failed run. 3. Queue a run for the newer task and claim it. On `master` the claim throws the unique violation. The new queue test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `0dedbf23c35`. **Deployment mode** Any server with routines whose tasks run on native runners. ## What Changed - `bindClaimedIssueExecution` in `server/src/services/heartbeat/queue.ts`: if stamping the execution lock violates `issues_open_routine_execution_uq`, it throws `conflict("Another execution for this routine is already in progress")`. The claim transaction rolls back, and the queue loop keeps the run queued as it does for other 4xx claim rejections. Every other error is rethrown unchanged. The issue checkout route already maps this violation to the same 409. - `doc/execution-semantics.md`: one paragraph on the behavior. ## Verification - Failing first: `queue.test.ts` "keeps a claim queued while another task of its routine holds the execution slot". On `master` the claim raised the `issues_open_routine_execution_uq` violation. Now it rejects with 409, the run stays `queued`, neither task's lock changes, and nothing executes. - Failing first, review round 1: `heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run queued while another task holds the routine slot, then runs it once the slot is free" goes through `resumeQueuedRuns`. On `master` it rejected with the unique violation. Now the routine run stays queued with no error code while another queued run for the same agent executes. After the slot is released, the routine run executes exactly once (executions are counted per run, because the finished task can queue its own follow-up run). - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed. - `npx vitest run src/services/heartbeat/recovery.test.ts src/__tests__/routines*.test.ts`: 4 files, 205 passed. - `pnpm --filter @paperclipai/server typecheck`: pass. - Not verified: the queue loop end to end on a real instance. The loop's handling of a 4xx claim rejection is existing behavior. ## Risks - While the older task holds the slot, the newer run stays queued and the loop logs "queued heartbeat run claim was rejected" on each pass. It is claimed once the older task's resume finishes or its lock is released. - Only this one constraint is mapped. Other database errors still escape as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:33:00 -08:00
DottaandPaperclip	652cd76026	fix(ui): expose Stop for failed native retries (#18899) ... ## Thinking Path > - Paperclip manages agents and their tasks. > - A native run can fail while it still owns a same-run retry. > - The server supports Stop for that pending execution. > - Run details show Cancel only for queued or running records. > - A failed pending retry therefore has no ordinary cleanup control. > - This pull request exposes the existing Stop eligibility and uses the existing cancellation endpoint. ## Linked Issues or Issue Description Refs #14743 and #14733 for correlated native Stop. Those changes cover backend cancellation and protocol qualification. They do not add the failed-run control to run details. Searches of current master and open PRs found no matching UI repair. **What happened?** A native run can show failed while its coordinator still owns a retryable failure with no accepted result. The backend can cancel that retry. Both run-detail implementations hide the cancellation button because the record is terminal. **Expected behavior** The selected run offers Stop when the server confirms that pending native retry. Unknown, legacy, terminal and accepted-result states do not gain this control. Existing queued and running cancellation stays available. **Steps to reproduce** Render either run-detail implementation for a failed native run with an eligible retry. The original UI has no Stop button. The original batched execution projection has no eligibility field. The regression tests exercise both real components and the actual PostgreSQL projection query. **Paperclip version or commit** The missing control and projection field were reproduced on 44a4ca1c61. Fresh master 8fd917aedf has identical affected production blobs before this repair. **Deployment mode** Board run details for a native Runner task. Local verification uses rendered React components and PostgreSQL. No hosted task mutation or provider retry was performed. ## What Changed - Add optional `execution.canStop` to the existing batched projection. Read the matching retryable coordinator and absence of an accepted result. The cancellation endpoint remains the authority owner. Add no database queries. - Exclude Grok failed retries from the new hint and both consumers. The existing Grok Stop endpoint cannot accept a failed retry before assignment. Keep this limitation explicit and preserve its active-work authority checks. - Show Stop for eligible failed runs in both run-detail consumers. Keep the existing cancellation endpoint and its company, actor and row-lock checks. - Use the existing request-correlation contract for failed-native Stop. Persist one scoped identity before sending and recover it after remount or reload. Read a matching canonical claim only for the same board actor, company and run. Use the existing UUID fallback on ordinary HTTP. Never retry Stop automatically. Refresh the selected run and history after either outcome. - Report a stale conflict or a true no-op200 response. Accept a matching durable pending cancellation intent without requiring terminal acknowledgement. Keep running, queued and Grok calls unchanged. - Add actual rendered interaction tests, real batched-query controls and a short execution contract. Keep maps unchanged for this bug fix. ## Verification - Untouched UI and real Postgres producer reds cover missing Stop and missing eligibility. A valid Grok review found an unsupported failed-run action; actual broker, projection and rendered controls reproduce it before exclusion. No backend authority was expanded. - The original60-case focused suite passed, including26 rendered cases, the batched producer, and actual Grok broker/assignment fences. The related57-case scope passed before the Grok correction; it is not presented as final-head proof. - A fresh native4/5 review found a stale200 no-op. Both actual rendered consumers failed that new assertion. The real authenticated HTTP/Postgres fixture proves `canStop=true`, coordinator revocation, uncorrelated unchanged200 with a cancellation activity, and existing correlated409 with no new activity or run mutation. - The correlated repair passed82 affected cases before master reconciliation. Exact `74c60c9ffe11c05bc85f3d0fa5a136f7e9d445cd` passed80 rendered/API, HTTP and native-correlation cases. The render scope verifies request bodies, stable identity after an unknown outcome, no-op200 error, pending intent acceptance, wrong-run/request denial and existing active/Grok behavior. - Subsequent current-head reviews found remount identity loss and unavailable `crypto.randomUUID` on ordinary HTTP. The corrected helper passed 52 rendered cases across both consumers, including an unknown response, initial scoped storage persistence, a prior browser record with a unique never-mounted run identity, exact ID reuse, canonical claim recovery, actor switch, wrong run and HTTP UUID fallback. A real local-board identity mismatch was reproduced with two additional reds before using the actual server-projected actor ID. - Real authenticated HTTP and existing native-correlation suites passed 45 cases for canonical claim recovery and actor fences. Production UI, server and vendored Runner builds and all server/UI type layers passed at the prior reviewed helper head; exact final `e0f723b1c86f1529029bf141943323764de28c22` UI types, token gates and production build passed. - Fixture failures are retained separately: a missing local SDK build dependency, and a missing coordinator FK teardown in the new HTTP fixture. Neither is a behavior red. Both fixtures were corrected and the affected tests passed. - Previous-head3d CI required one diagnostic rerun after a Runner shutdown and a real account-list migration assertion. Two untouched157-test runs did not reproduce the assertion. Complete original row equality is unavailable. The assertion remains unresolved; no test was sorted, skipped or changed to green CI. - Previous-headf12 CI lost eight child runners. Its one authorized failed-job retry also lost e2e5 to an explicit shutdown. No second retry was issued. Those failures remain recorded. Current-head CI and both substantive reviews are required; older checks and grades are not credited. - Previous-head74c CI found a real monitor scheduler timeout. An untouched local replay and diagnostic-only replay reproduced it: the original manual-monitor run was still running at the unchanged three-second deadline. The cause remains unresolved; temporary diagnostics were removed, and no assertion, timeout or production monitor policy was changed. No CI retry was issued. - Exact local `906af5c942ee0963eb4b64552160b57a18bcd2d1` integrates normal master after #18930 merged. All reviewed Stop production and test blobs remain unchanged from `e0f723b1c86f1529029bf141943323764de28c22`. The four rendered, projection, HTTP and correlation files passed 99 cases after integration. All UI/server type layers, token gates and production UI/server/Runner builds passed at that exact head; the server build stamp records it. - Previous-head e0 CI reported a Discord ownership error. That original failure is preserved. Separate #18930 proves and repairs receiver construction/renewal races with the same error; the original CI scheduling order remains unproved. This branch now contains that ordinary master merge and requires fresh CI rather than rerunning the old failed attempt. - Full repository checks were not run locally for this narrow slice. No hosted task, provider execution or Stop receipt was created by this work. ## Risks Grok failed retries remain unsupported by this UI change. The existing Grok running and queued Stop action, revoke path and assignment fences are unchanged. A valid initial review caught the overbroad hint; actual broker, projection and rendered regressions reproduce it before correction. No backend cancellation authority was widened. The optional hint can become stale. It grants no permission. The existing cancellation path rechecks current authority and can reject the request. Run details show that rejection. Older servers omit the hint, so failed runs keep their existing behavior until the server updates. No retry, task hold, accepted-result decision or provider receipt is changed. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 23:18:08 -06:00
DottaandPaperclip	0dedbf23c3	fix(chat): retain receiver ownership during runtime construction (#18930) ... ## Thinking Path > - Paperclip manages agents and their work. > - Chat receivers hold a durable lease before they accept messages. > - Runtime construction can outlast a lease renewal interval. > - Renewal previously required an attached runtime before construction had returned. > - That check could revoke the same receiver that was still starting. > - This change uses the exact pending initialization as proof during construction. > - The existing durable token and endpoint checks still fence every renewal. ## Linked Issues or Issue Description Related: #18899 exposed the existing receiver failure in its chat CI shard. This is a separate lifecycle repair. #18795 provides the relevant production and sleep fixture context. Searches found no overlapping open initialization repair. **What happened?** The receiver acquired its lease and started its renewal timer. The service then awaited credentials, command preparation and runtime construction before attaching the runtime to its context. A renewal during that interval treated the missing attachment as lost ownership. Configuration later failed with a Gateway ownership error, although its original durable token was still current. **Expected behavior** The exact pending initialization can retain its own lease while construction runs. Missing attachment alone must grant no ownership. A changed token, paused or removed endpoint, changed generation or credentials, and shutdown must still fence the receiver. **Steps to reproduce** 1. Use the real configure and PostgreSQL ownership producers with a 60-second lease and 50-millisecond renewal. 2. Hold the actual runtime construction call for 200 milliseconds. 3. Release construction. Untouched code returns the Gateway ownership error. 4. With the same construction delay and a five-second renewal control, configuration succeeds. **Paperclip version or commit** The causal regression fails on unchanged production source at `e7549ab613`. The original CI error was observed in the chat shard of #18899. That CI run did not record the precise setup and renewal order, so this local proof does not establish its historical scheduling. ## What Changed - Store the runtime context identity in the existing pending initialization entry. - Allow pre-attachment renewal only for that exact tracked initialization and current local ownership. - Recheck local ownership inside the existing renewal transaction. Preserve company, endpoint, generation, credential and original-token checks. - Before SDK listener initialization, wait for any older renewal, recheck local ownership, then obtain fresh durable proof. Preserve the existing checks after initialization. - Keep attached runtime identity checks, lease deadlines, renewal intervals, shutdown and listener retirement unchanged. - Add actual configure and database race controls. Document the shared Discord and Photon receiver contract. ## Verification - Untouched production red: the actual construction gate crosses a renewal and returns the same Gateway ownership error. The otherwise identical slower-renewal control passes. No lease, exception or provider outcome is injected in these positive cases. - A subsequent review found a second real race: final runtime removal did not prove that SDK initialization had never started. Four actual initialization-hook controls failed before the pre-initialize fence. The final matrix asserts zero hook calls for lost or changed authority, including a foreign-token control with no periodic renewal due. An attempted cross-company endpoint move is rejected by the existing database foreign key; the valid original initialization still succeeds. - A third actual-producer red holds completion of a real committed renewal, changes the durable token, and proves that joining that old promise could incorrectly accept configuration. The corrected startup check waits for that renewal and then obtains fresh proof; it cannot credit the earlier result alone. - Final relevant full Discord registration group, takeover/callback cases, buffered host-pause recovery, failed construction, credential-fenced pause and shared Photon consumers: 34 passed across two files; 1,840 other cases were deselected. Four executed cases cover Photon transport recovery, failed authentication, duplicate allocation ownership and checkpoint fencing. The tests use the real configure/database producers with fake provider transports and observe SDK initialization; they do not prove physical network behavior. - All server type layers, the module-boundary gate, and production server/vendored Runner build pass at exact final `38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp records this exact commit. - Earlier fixture mistakes are preserved separately: a test initially changed every lease key, including the credential lease, and another attempted an invalid cross-company row move. Neither is counted as a production behavior red. The corrected controls target the Gateway key and assert the existing company constraint. - An initial isolated dependency graph resolved an older shared telemetry build. Correct workspace links restored the all-layer typecheck without changing telemetry source. - Full repository checks were not run locally for this narrow slice. Fresh current-head CI and substantive reviews remain required. No hosted task, provider work, fleet action or deployment was performed. ## Risks The ownership helper is shared by Discord and Photon. The selected tests cover both consumers. The pending entry grants only permission to renew the original token; it grants no message, callback or publication authority and adopts no listener. Missing or changed initialization identity fails closed. There is no schema change, historical backfill, new worker or additional idle scan. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection, source editing, shell execution and test tools. The host did not report the context-window size. A separate coordinator reviews the source before delivery. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:57:39 -06:00
DottaandPaperclip	f211d44c00	fix(heartbeat): preserve queued coalesced comments at claim (#18913) ... ## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 22:09:58 -06:00
DottaandPaperclip	0884e73b46	test(e2e): verify saved assigned skill use (#18783) ... ## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:13:40 -06:00
3 people	27a825da2f	feat(telemetry): track the server OS with the registered contract (#18897) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its first-party telemetry uses a generated event contract and typed helpers. > - The existing events do not identify the server's OS family. > - A proposed event is dropped until the generated contract registers it. > - This PR syncs the registered contract and emits the OS family when an enabled server telemetry client starts. > - The benefit is a way to compare Windows, Linux, and macOS installations while keeping machine details out of the event. ## Linked Issues or Issue Description **Problem or motivation** We cannot compare Paperclip installations by host OS. Counting task or install events does not provide this dimension. **Proposed solution** Send one `server.started` event with Node's raw `process.platform` as `os_platform` each time the server initializes its enabled telemetry client. Compare distinct install identifiers per platform, rather than restart event totals. **Alternatives considered** A proposal-only emitter would send no data. A separate later contract-sync PR would leave the feature incomplete. This PR includes the canonical generated SDK and promotes the emitter together. CPU architecture and OS versions are outside this feature's scope. **Roadmap alignment** This is a narrow telemetry addition to existing server startup, with no new user flow. The full contract copy also adopts the three existing connector proposals described in #13578; their schemas are already registered. The generated file stays an exact copy, so their helpers and tests must be promoted in the same change. ## What Changed - Export `trackServerStarted` and call it once when server telemetry initializes. Repeated initialization returns the same client and emits no duplicate event. - Copy the canonical generated telemetry SDK byte for byte from schema source commit `5369e6b`. Remove the startup proposal marker. Normal opt-out and CI suppression apply before initialization. - Promote the three connector events included in the canonical SDK. Derive enum types from the generated map and preserve raw values. Update their real-client transport and disabled-state tests. - Assign startup to the existing `operational_enum_count` retention class. Document OS privacy, container/WSL interpretation, and distinct-install counting. - Add real-client startup transport tests and server initialization tests, including file opt-out, environment opt-out, and CI suppression. - Keep startup running when telemetry state storage fails. Publish the client and start periodic flushing only after the startup event succeeds. On failure, clean up the client, disable telemetry for the process, and log a generic warning without the storage path or underlying error. ## Verification - `pnpm --filter @paperclipai/shared typecheck`: passed. - `pnpm --filter @paperclipai/shared build`: passed. - `pnpm --filter @paperclipai/shared exec vitest run`: 98 files, 1,055 tests passed. - `pnpm --filter @paperclipai/shared exec vitest run src/telemetry`: 6 files, 39 tests passed. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/telemetry-startup.test.ts src/__tests__/shared-telemetry-events.test.ts`: 2 files, 19 tests passed. - Exact generated artifact comparison, `git diff --check`, and local secret/PII review passed. - Proposal inventory contains no startup or connector proposals. The repository extractor fails with TypeScript 7's missing compiler API; rerunning it with the installed TypeScript 5.9 compiler API passed. No extractor source change is included. - Initial local server typecheck attempts found missing workspace dependencies and a missing plugin SDK link in this fresh checkout. Dependencies and the local SDK link were repaired; `pnpm --filter @paperclipai/server typecheck` then passed. Full repository typecheck, tests, and build are covered by the PR CI rather than claimed as local passes. Greptile found that a telemetry state-file error could abort startup. New regression tests reproduced this on the previous head with both an injected permission error and the real state store blocked by a file. Both now pass, including no published client, no periodic flushing, no send, and no retry after failure. Shared telemetry tests and server typecheck were rerun after the fix. Full PR CI passed on `a41540b7899bc8a0aedbd78d72f8b69ad1a4b318`, including typecheck, build, all selected test shards, and the final verify gate. [Successful workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752039595/attempts/2). Attempt 1 was interrupted when the server (1/12) runner received a shutdown signal; its incomplete result was retained in the PR discussion. The same-commit shard retry and dependent gate passed. Greptile reviewed this exact head at 5/5 with all threads resolved. ## Risks This change starts sending one OS family value per server startup when telemetry is enabled. It sends no OS version, architecture, hostname, path, prompt, or machine identifier beyond the existing envelope. Containers and WSL report the runtime's OS. Event totals reflect restart frequency, so use distinct install identifiers for OS adoption analysis. The complete generated SDK also enables the previously registered connector proposals. Those events keep their existing approved payloads; no connection names, credentials, tool arguments, or results are added. Privacy review is requested for both newly active paths. Deploy the registered schema before merging this app change. Live production delivery has not been tested. No schema migration is needed. Revert this PR to stop the new startup emitter and restore the prior contract. ## Model Used - Original implementation: Anthropic Claude Opus 5.5, as recorded in the original commit. - Completion: OpenAI `gpt-6.1-sol`, xhigh reasoning, through Codex in T3 Code; shell, Git, and automated tests. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] The final change and description contain no private issue links or proposal markers - [x] The existing PR branch is preserved to retain its review history - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 21:00:47 -06:00
DottaandPaperclip	057898530c	fix(server): handle wrapped recovery action conflicts (#18885) ... ## Thinking Path > - Paperclip coordinates durable agent work and recovery. > - Independent workers can find the same task needs recovery. > - The database permits one active recovery action for that task. > - A losing insert must read the action that already committed. > - Drizzle wraps the PostgreSQL unique violation in its cause. > - The current classifier reads only the outer error and misses that conflict. > - This PR recognizes the same allowed conflict through a bounded cause chain. > - Both workers can then return the one durable action through the existing retry path. ## Linked Issues or Issue Description Refs: #18783. **What happened?** The saved-disposition-debt recovery regression failed in CI `41733864320`. Two recovery workers collided on `issue_recovery_actions_active_source_uq`. The fixture captured an unhandled PostgreSQL `23505` error. A separate canceled shard in that run lost its AWS Spot instance; that interruption is a different failure. **Expected behavior** Two service instances that race to create the same active action must return that one action. Unrelated database failures must still propagate. Existing transaction and ownership rules must stay in force. **Steps to reproduce** Hold a PostgreSQL SHARE lock on the recovery-action table. Start the same source-scoped upsert on two independent service instances. Confirm both real INSERTs are waiting in `pg_stat_activity`, then release the lock. The original code persists one action but rejects one caller with a `DrizzleQueryError`. Its outer code is absent; its `PostgresError` cause has SQLSTATE `23505` and the expected active-source constraint. **Paperclip version or commit** Supporting branch starts at master `54782706e1d83d92295246e499627930df03662d`. It now contains a normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The classifier correction stays unchanged. The PR diff against that master still contains only the recovery service and its tests. ## What Changed - Inspect up to eight error causes with cycle detection. - Require SQLSTATE `23505` and one of the same two recovery-action constraint names on the same error. Support the PostgreSQL driver's `constraint_name` field. - Reuse the existing source-scoped retry path and its unchanged three-retry limit. - Add a deterministic real-database collision regression and no-retry controls for unrelated uniqueness, other SQLSTATEs and cyclic errors. - Keep the original recovery regression, assertions and timeouts unchanged. Keep the saved-skill journey separate in #18783. ## Verification - Review follow-up: the real collision test now checks the intended retry merge. The surviving action has attempt count 2, active status, the default unlimited budget, the expected owner, supplied evidence and supplied attempt timestamp. Its wake and monitor fields remain null. Existing identity and convergence assertions run first. - A disposable counterfactual returned the winner unchanged after the lost insert. It passed the original barrier, one-row, no-rejection and same-ID assertions, then failed the new attempt-count check with 1 instead of 2. The counterfactual was not committed. The production service was restored byte-for-byte. The actual service passed all seven focused race, no-retry and sequential controls. The affected canonical test-tree-2 typecheck passed against verified unchanged local declarations. - Parent `4f8abfe4f2244649219ef6ced3534634193f0b53` full CI `41746314291` passed all 48 selected jobs with one intended skip. Its 5/5 summary also had two duplicate unresolved reliability comments about the missing state assertions. This test-only follow-up addresses both. Fresh full CI and current-head review are required again. - The deterministic real-database regression failed before the correction. Both inserts reached the lock and exactly one active action persisted. The losing error was a `DrizzleQueryError` with a `PostgresError` cause, SQLSTATE `23505` and constraint `issue_recovery_actions_active_source_uq`. - **11 focused checks passed across two files on the integrated head** (181.72 seconds). This includes the real-database collision, no-retry controls and all four unchanged recovery producer/dependency regressions, including the original failed real-clock saved-debt case. Both callers return the one persisted action. - Current-head canonical server typecheck passed on Node24.22.2 with all seven layers: below, cycle, above, tests and test-tree-0 through test-tree-2. Canonical shared, SDK, DB and Runner TypeScript prerequisites passed. The first filtered installs lacked CLI and UI imports required by the new test layers. Those module-resolution failures remain recorded as setup failures. A full frozen workspace install with lifecycle scripts disabled supplied the same locked dependencies. No source or type-configuration workaround was used. No Rust or broad workspace build ran locally; full repository typecheck, tests and build remain hosted-CI gates. - The actual supporting master baseline reproduced the same wrapped23505 failure before the correction. Earlier configuration-resolution, local IPC and native-library prerequisites remain separate setup failure evidence. No test failure was reclassified as a pass. - First supporting CI `41739489620` was canceled before its build/test jobs completed. Its later automatic full-CI run `41739747947` failed. The recovery server shard passed. The Runner failure was the Cursor fixture case repaired upstream in #18782, now included by the normal master merge. That CI also observed an attachment-browser timeout, a repository-rehome timeout and a lost-runner connection. Their underlying causes remain unknown. A later pass will establish recovery, not explain those earlier failures. These original results remain retained. - Current `fbd392b9b65dbcf4ac078825096037ed0d490a0b` full CI [41751706634 attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41751706634/attempts/2) passed: 48 successful jobs and one intended skip. Attempt 1 remains retained: ten primary jobs lost their bound Spot capacity and the verify job failed as a dependent. A single ordinary failed-job recheck produced eleven fresh successful executions (ten primaries and verify); timestamps prove 37 successful executions were retained. The skip is not counted as execution. Current Greptile is 5/5 with both addressed reliability threads resolved and no unresolved findings. All 57 current check runs are successful, neutral or skipped, and Snyk is successful. This is CI recovery; it does not explain the separately retained older attachment and repository timeouts. - The original CI failure and both failed saved-skill QA attempts remain unchanged. This PR does not establish a live QA pass or deploy a production change. ## Risks Conflict classification must not turn unrelated errors into retries. The new controls require those errors to propagate after one insert attempt. The traversal is bounded and stops at cycles. Transaction, company scope, ownership, notifications, retry count and database constraints are unchanged. No schema or public API change is required. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model ID or context-window size. These checks make no model or provider calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 19:51:37 -06:00
Michael NguyenandPaperclip	58d1b21098	fix: separate fresh setup from archived personal connections (#18902) ... ## Thinking Path > - Paperclip helps people manage AI agents and their service access. > - Each personal connection belongs to one user. > - Setup can recover an archived connection by its name and provider. > - A fresh setup can therefore select another user's old identity and fail its owner check. > - This change creates a separate connection for fresh setup after personal archival. > - Explicit reconnect still requires the existing owner and preserves its history. ## Linked Issues or Issue Description **What happened?** A fresh GitHub setup can return 403 after a personal connection with the same name was archived. The request does not include a reconnect target. Setup still selects the archived identity by name and provider. **Expected behavior** Fresh setup creates a separate personal identity. It leaves the archived connection, credentials, grants, and activity unchanged. Explicit reconnect rejects a different user. **Steps to reproduce** 1. Create a personal GitHub connection in a test company. 2. Archive that connection and its application. 3. As another authenticated member, start fresh personal setup with the same name and provider. 4. Observe the owner-check failure before this change. **Paperclip version or commit** Base: db9be4c5a. Tests use isolated database and HTTP fixtures. **Deployment mode** Authenticated. The legacy owner can be a local board identity. ## What Changed - Exclude implicitly recovered archived personal connections from fresh setup. - Use the existing company-scoped name allocator to create a separate application and connection. - Test three owners: a legacy board identity, another user, and the current user. - Check credential isolation, unchanged archived records and activity, missing-key rejection, and explicit reconnect denial. - Keep managed revival and generic MCP rollback/reconnect fixtures on explicit reconnect targets. - Verify generic personal URL setup creates an independent identity and preserves the archived records. - Document fresh setup and reconnect behavior. ## Verification - Final head: `96184e3179aab2eeb88df87d442691741a4875a2`. Full CI recovered and completed successfully: [run 41752164958, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752164958/attempts/2). PR rollup: 53 successful checks, 3 skipped, none failing or pending. Typecheck, build, all server shards, Runner lanes, and E2E gates pass. - Exact-head Greptile: 5/5, zero inline comments or review threads. Independent engineering/security reviews approved the production implementation; engineering approved the test-only follow-up. Browser acceptance review approved with the timing qualification below. - Final-head local generic MCP suite: 84/84 passed (23.26 s); server typecheck passed. Reviewer independently reran the three changed scenarios: 3/3 passed (15.74 s). - Production head `587631230`: local tool-access suite 504/504, recursive typecheck, and full build passed. Broad local checks were not repeated for the test-only delta. Full local `pnpm test:run` remains incomplete after interruption during the first chat shard; no full-local-suite pass is claimed. - Initial full CI at `587631230` failed three generic MCP fixtures that still relied on implicit archived personal recovery. All three reproduced locally (15.01 s). The test-only follow-up gives owner reconnect/rollback scenarios explicit targets and verifies independent fresh URL setup without relaxing their assertions. Corrected server shard 12/12 passes on final head. The unfinished earlier run was cancelled after the follow-up push. - Initial current-head E2E shard 3/8 failed a 5000 ms task-heading wait at `tests/e2e/cloud-auth.spec.ts:46` (40 passed, one failed). Its screenshot/accessibility snapshot contains the expected heading. The single authorized no-code rerun passed 41/41 with zero test retries, including all URL, handoff, and login-form checks. The first-heading assertion took 3774 ms on rerun. Both attempts tested the same merge commit for this PR head. These observations support a timing concern; the original cause remains unknown. The unchanged Cloud-auth path does not invoke archived-connection setup. Recovery is established; an infrastructure cause is not. If it recurs, capture an instrumented first-navigation trace and compare an unchanged baseline under comparable conditions. - PR remains draft. Merge and deployment are outside this request. Synthetic provider/browser fixtures do not prove live GitHub authorization or a deployed fix. ## Risks - Fresh setup after personal archival now creates a new ID and a suffixed name, including for the same owner. Clients must supply an explicit target to reconnect the old identity. - The guard applies to all personal gallery connections, not only GitHub. - No schema migration is required. Explicit owner checks remain in place. - Security review approved the unchanged production implementation. Merge and deployment are outside this change request. ## Model Used OpenAI GPT-6 through Codex. The runtime does not expose a more specific model snapshot or context window. Used reasoning, code editing, and shell tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 17:51:29 -08:00
DottaandPaperclip	e2dc2cc0be	fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893) ... ## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. **What happened?** A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. **Expected behavior** A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. **Steps to reproduce** Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. **Paperclip version or commit** Reproduced on ab2d2098a2. The branch includes current master 44a4ca1c61 before this PR was opened. **Deployment mode** Native Runner with provider ingress on a reusable sandbox workspace. Local regression uses real PostgreSQL and local TCP with a controlled provider-process boundary. No hosted provider call was made. ## What Changed - Keep physical ingress identity separate from logical task/session ownership. Never attach the parent to the child's session. - Reserve a physical allocation during local admission. Retire a distinct warm owner only after canonical company, agent, workspace, provider, allocation, lease, task and coordinator checks pass. - Require the current remote workspace writer and location to match the admitted run. Require the prior run to be committed and accounted, with its original controller generation and a clean released or expired lease. - Use the existing close, checkpoint and stopped-file collection path. Wait for close before parent dispatch, then recheck canonical authority before staging files or calling the parent backend. Retain the entry when close is uncertain. - Add actual executor/PostgreSQL/TCP safety cases and document the contract. Keep maps unchanged for this bug fix. ## Verification - Untouched production at ab2d2098: actual executor + PostgreSQL + TCP fails EADDRINUSE. Same positive case passes after the change. - Final focused suite: 20 passed. It covers distinct parent session identity; foreign company, allocation and workspace; active or unaccounted owner; changed controller; reassignment; competing lease/writer; stale allocation metadata; Stop before and after claim; workspace-lock race; failed close that leaves the listener and accepted result intact; and controller transfer, Stop or writer change during awaited close. These last cases reject the stale parent before backend invocation after the old listener closes. - Untouched head3425acbe: transferring the incoming coordinator during close still reaches the parent backend. The same actual executor/PostgreSQL/TCP case passes after the post-close authority check. - Broader `native warm session supervision` suite: 160 passed. This run preceded the final two-line logical-session collision guard; the final 20-case suite includes that guard. - `pnpm --filter @paperclipai/server typecheck`: all source and test layers passed, including `src/__tests__`. - `pnpm --filter @paperclipai/server build`: production server and Runner build passed at current head572acfef79. The exact-head build stamp is recorded. - The provider/close boundary is controlled. The fixture seeds canonical committed/accounted finalization data in PostgreSQL. These tests do not claim a live model result, operator ACK, hosted PID-to-listener identity, or recovery of a historical task. - Full repository pnpm checks were not run locally for this server-only slice. Required current-head CI and reviews are pending. ## Risks The new retirement path can close a process from another task. It requires the same canonical company, agent, allocation and workspace, a current exclusive writer, and a completed idle owner. Missing or changed evidence leaves admission blocked. Active owners, changed generations, reassignment and cancellation remain fenced. A failed close retains the owner and saved result. This covers current-controller warm entries with recorded physical identity. Historical processes outside the registry keep the existing exact-owner recovery requirements. No port shuffle, port kill, task capability expansion, new polling loop or invented receipt is added. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) in Codex, with source editing, terminal execution and tool use. The host did not report its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 19:44:12 -06:00
DottaandPaperclip	7b1c53f8f1	fix(recovery): continue verified preparation after lease interruption (#18892) ... Use recorded system interruption and the existing execution, controller, stopped-lease, and saved-file proof instead of a shutdown-only error label. Preserve bounded retries, user Stops, task authority, and exactly one continuation. Keep the chat recovery fixture faithful to the preparation stage and terminal time written by the real lifecycle. Retain missing-field safety controls. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 18:54:16 -06:00
Dotta	82dd3bb702	Preserve unstarted queued turns during offline enrollment (#18887) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-03 18:33:41 -06:00
DottaandPaperclip	ab2d2098a2	test: verify sleep producer readiness and CI contracts (#18795) ... Reviewed and verified at 9f87b490d1. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 17:41:15 -06:00
Nicky LeachandClaude Opus 5.5	11ac90e31f	fix(build): typecheck server/src/__tests__ and fix every type error (#18739) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. **What happened?** No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. **Expected behavior** The server typecheck checks these files, and a type error fails the check. **Steps to reproduce** 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. **Paperclip version or commit** I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. **Deployment mode** Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. | Step | Total errors | Change | Cause | | --- | --- | --- | --- | | Config as committed | 6,536 | | | | Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request types were missing. | | Set `rootDir` to the repository root | 2,436 | -2,011 | Config. | | Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest accepts `.ts` imports. | | Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has `Promise.withResolvers`. | | Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8 `TS7016` errors and shows new real errors. | | Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing. | | Count only files in `server/src` | 2,058 | -84 | Other packages: 53 Daytona, 29 Runner test, 2 UI. | So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. | Kind of real error | Errors | | --- | --- | | Wrong argument or assignment type | 1,023 | | Stale or partial fixture objects | 525 | | Property that the type does not have | 317 | | Missing narrowing of null or unknown | 101 | | Implicit any | 60 | | Missing name or module | 16 | | Other | 16 | ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - **A test that never ran.** `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - **Assertions that could not fail.** - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - **Postgres processes that did not stop.** 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - **Arguments that the service ignored.** - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - **Problems in tests that master added during this work.** - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: | Check | Commit | Result | | --- | --- | --- | | Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers | `6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB | | `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e` | Exit 0, 0 errors, 42 seconds, peak 3.25 GiB | | The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0 errors | | `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` | `01db5634e4` | 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. | | `tsconfig.declarations.json` after a full server typecheck | `0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. | | Owned runtime test in `idle-completed-history` | `e8f2f448ac` | Passed, with the fix that master later replaced | | `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test | `6e45a08573` | 30 passed | | Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed | | The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. | The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. **Runtime check of the fixes.** Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: | Result | Tests | Detail | | --- | --- | --- | | Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. | | Failed | 7 | The same 7 tests also fail on the commit before the fixes. | | Skipped | 9 | These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. | The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. **Audit of the diff.** I counted these patterns in added and removed lines, to find fixes that only hide an error: | Pattern | Added | Removed | | --- | --- | --- | | `@ts-ignore` | 0 | 0 | | `@ts-expect-error` | 0 | 1 | | `.skip` | 0 | 0 | | `as any` | 9 | 9 | | `: any` | 11 | 2 | | `as never` | 8 | 2 | | `as unknown as` | 35 | 4 | | `Promise<unknown>` | 16 | 5 | | `expect(` | 128 | 113 | No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. **Accuracy of the check.** One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - **A large diff in test files.** 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - **A higher cost for changes to source types.** A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - **Changed test behavior.** Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - **The Daytona stub can go stale.** The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - **The release path.** `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - **The tsconfig change.** The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 15:27:38 -08:00
DottaandPaperclip	dcc0878b32	perf(agents): save healthy account settings without a runtime probe (#18690) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The agent runtime page saves account, model, and environment settings. > - The save handler starts a computer and runs a live account test before it writes the edit. > - A slow computer or provider can make a healthy account save take tens of seconds. > - Saved account health, access, installation, and compatibility can be checked without a runtime test. > - This pull request uses those checks for ordinary configuration saves. > - Legacy adoption, task adoption, and Copilot model checks keep live verification. > - The benefit is a fast save while Test and execution retain runtime checks. ## Linked Issues or Issue Description **What happened?** Saving a new AI account selection on an existing agent can take 20–30 seconds. Model edits can take the same path. The configuration PATCH acquires a computer, prepares account credentials, runs a provider/runtime probe, and releases the computer before it saves. **Expected behavior** A healthy installed account can be saved after current access and compatibility checks. A save confirms persistence. Test verifies the runtime. An unverified legacy account must still pass live adoption checks. **Steps to reproduce** Run the PostgreSQL regression with a 20-second controlled delay at computer allocation. On the original route, a successful save takes 20,191 ms and calls allocation and the runtime probe once. With the first patch, the same fixture takes 51 ms; with the environment-guard correction, it takes 37 ms and calls neither operation. Each timing has one sample. This is a local fault experiment with real HTTP and PostgreSQL. It is not a hosted latency measurement. Related: #17760 addresses binding comparison and prospective probe environment resolution. This change deliberately removes the live probe from ordinary healthy-account saves. It keeps live checks for legacy and task adoption. #17965 fixes the result of an explicit managed Runner test. Neither change removes this save-time wait. #18687 optimizes a different Apps configuration read. ## What Changed - Replace the internal probe boolean with explicit none, legacy, and always modes. - Use legacy mode for ordinary configuration saves. Keep current account health, access, installation, compatibility, environment, audit, and revision checks. - Keep the environment company check before the live-probe branch. Keep the live task-adoption transaction and Copilot model checks. - Add twelve real PostgreSQL save-route regressions, including company-bound environment checks. Keep the existing inherited-sandbox adoption fixture for legacy and task adoption. - Cover healthy and legacy settings changes separately. Keep provider rejection and reconnect tests on the legacy save path. - Update the specification and feature/access maps to distinguish saving from runtime verification. The configuration route owns this policy. The account service still owns account access and health. The existing environment test service still owns live probes. No new endpoint, migration, dependency, background job, or provider protocol is added. ## Verification Base: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`. - Original route with initial regressions: three failures and two passes. Healthy saves reached computer allocation. The controlled 20-second baseline saved successfully but failed the no-allocation assertion. - Initial save/adoption/Copilot selection: 21 passed. The name filter excluded 141 unrelated tests. The seven new cases cover persistence, revision and activity records, model edits, roster edits and stale retries, missing installation, revoked grants, legacy verification, and controlled allocation latency. - Explicit Test-route suite: 55 passed. The first attempt reached the 30-second import hook limit. The same suite passed with a 90-second hook limit. - Saved harness preflight suite: six passed. Account revocation, compatibility, probe failure and target release checks remain. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Review revision: both full save/account suites passed, 96 tests total. The controlled save took 37.08 ms with zero computer allocations and zero provider probes. The five new environment checks passed after the three foreign-environment cases failed against the first patch. - Local full `pnpm -r typecheck`: unavailable because cargo is missing. - Local full `pnpm build`: unavailable at the verified Copilot runtime prerequisite. Server typecheck preparation also reaches missing cargo. - Local full `pnpm test:run`: stopped after concurrent broad checks exceeded the container memory limit. No full-suite pass is claimed. Direct server TypeScript checking and its bounded retry were also killed by the container memory limit. The first remote Typecheck + Release Registry check passed; the next head still requires fresh CI. Reproduce the controlled timing: ```sh cd server PAPERCLIP_CONFIG_SAVE_PROFILE_DELAY_MS=20000 pnpm exec vitest run src/__tests__/agent-configuration-save.test.ts --hookTimeout 90000 ``` The fixture keeps real Express, authorization, account selection, PostgreSQL persistence, revisions and audit writes. Only remote allocation and the provider probe are controlled. The default tests do not contact a provider. Successful configuration saves no longer depend on either controlled operation. A later explicit Test or task must still verify the actual runtime. Review follow-up: the first Greptile review found that the new healthy-account path skipped the environment company guard. The corrected route checks the effective environment before it decides whether to probe. Five real PostgreSQL cases cover foreign, same-company and global environments, inherited defaults, and an existing account binding. Against the first patch, the three foreign-environment cases returned 200 and failed their expected 403 assertions; the two allowed cases passed. Initial fixture attempts hit unique environment constraints; the fixture now uses distinct sandbox rows and names. The first CI server shard found 11 tests that required a live probe for every save. Those expectations are changed to the documented healthy/legacy split, with additional healthy-selection cases. The build and Runner jobs were cancelled after this test failure. Their logs show no separate compile or test verdict. No cancelled check is counted as a pass. Latest commit `6257e9e67f289e1c650a8b4c2b818fcca37e2800`: [CI run 41542980091](https://github.com/paperclipai/paperclip/actions/runs/41542980091) passed typecheck, build, all test shards, Runner checks, and E2E. All 55 check runs are complete: 52 passed and 3 were skipped by their workflow conditions. The commit status check also passed. [Greptile review](https://github.com/paperclipai/paperclip/pull/18690#issuecomment-6590555746) is 5/5 for this commit with no new findings. The earlier environment ownership thread is resolved. These remote results complete the checks that the local tool and memory limits prevented. The earlier local failures remain recorded above. ## Risks A healthy saved account can become invalid at the provider before the next runtime use. Ordinary saves no longer detect that through a live test. They still reject unhealthy, revoked, incompatible, or uninstalled accounts in current storage. Explicit Test and execution retain provider authentication and project-authentication checks. Legacy-account adoption, task connection adoption, and Copilot account-specific model checks keep their existing live validation. There is no migration. Reverting the route change restores the earlier synchronous probe behavior. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. This runtime does not expose the exact model ID, context-window size, or reasoning setting. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full-suite limits are above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 16:56:11 -06:00
Dotta	049c1eacf5	Preserve historical native logs during ephemeral enrollment (#18864) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers need retained run history in shared storage. > - Some old native runs have surviving local logs but no completion digest or verifiable native source coverage. > - Enrollment must preserve those bytes without calling them a complete native transcript. > - This pull request adds a read-only archive path to offline enrollment. > - Users can still read old output after the original server filesystem is removed. ## Linked Issues or Issue Description **What happened?** Offline enrollment blocks on interrupted runs and unfinalized historical native logs. The current importer requires canonical native-event coverage even when the original local file survives. **Expected behavior** Preserve the exact surviving file in shared storage. Keep the original run status, completion metadata, and source events. Do not allow writers or native repair to modify archived history. **Steps to reproduce** Use an S3-configured instance with a terminal native run whose local log lacks a final size or digest. Stop all app deployments and run ephemeral maintenance. Before this change, interrupted status or unproven native coverage prevents enrollment. Related preservation work: #18826, #18835, #18841. This is a bounded migration fix, not a new live logging protocol. ## What Changed - The run-evidence migration owner can archive unfinalized native local files during offline enrollment. - Require the offline migration environment, zero active or queued runs across the instance, and expired controller/finalizer leases. Check global quiescence at each page boundary. Recheck each run's ownership and metadata before publication. - Verify stable local file identity and metadata, any recorded size/digest, uploaded prefixes, and remote SHA-256. - Store raw history in a separate `<run>.archive` journal with source cursor zero and backend `s3_archive`. Leave completion fields and native events unchanged. - The existing log store reads archives but rejects begin, append, enqueue, flush, and finalize. Native source repair does not run for archives. - Keep the default background importer's coverage checks. Continue to reject incomplete mirrors and corrupt files. - Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a pre-existing large-chunk fixture: the current verifier can prove exact compactor output, so the negative case now uses a different canonical source. ## Verification - Passed on final head b96e80dbb3: 13 migration/storage/worker tests against disposable local PostgreSQL and S3. - Passed: 76 log-store, cancellation, object-store, native watermark, and maintenance-service tests. - Passed: server typecheck and server build. - The archive journey removes original log files, reads exact saved bytes through the normal recovery-aware reader, checks unchanged run/source metadata, and verifies read-only behavior. - Negative cases cover ordinary-process use, another company's running execution, live controller/finalizer leases, an invalid recorded digest, an incomplete mirror, and a changed local source. - Review follow-up: move the global run-table scan from every item to page boundaries. A real PostgreSQL trigger admits new execution during publication; the page then fails, retains verified copies, and resumes after quiescence. All 5 migration cases passed. After rebasing onto merged #18841, all 13 migration/storage/worker cases and all 7 packaged maintenance cases passed, along with server typecheck, build, and runtime bundle. - Preserved failure: the old large-chunk expectation failed with both this change and the unmodified base importer. The updated fixture checks the verifier's current compaction contract and still blocks mismatched source output. - Passed: all 7 real PostgreSQL/S3 enrollment cases, including the bundled maintenance process under plain Node. Its interrupted native log is archived, the original log directory is removed, and the saved bytes remain readable. - Passed: packaged runtime build and full repository CI (54 successful checks). Four CI jobs lost their runners; a separate voice-recovery timing assertion failed. The unchanged voice case passed locally, and all failed/interrupted CI jobs passed on retry. Greptile reviewed the final head at 5/5 with no unresolved threads. The separate Paperclip Review app remains queued/paused. No live tenant migration or volume detachment is claimed by this PR. ## Risks - Rollback after enrollment requires a reader that supports `s3_archive`; old binaries cannot read the new backend. - Offline mode relies on the deployment owner stopping all old app processes. Database and file checks are additional fences, not a substitute for that prerequisite. - Archives may be incomplete transcripts. The implementation preserves that uncertainty and does not fabricate a successful run or completion digest. - No schema change, new background polling, or telemetry change. ## Model Used OpenAI Codex (GPT-6 family), with reasoning, repository tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-03 16:52:30 -06:00
DottaandPaperclip	683fa4725e	fix(evals): refresh native runtime and pin reviewed catalog repairs (#18780) ... ## Thinking Path > - Paperclip lets people inspect and manage their agents’ work. > - The timeline must report canonical ownership history, and OpenCode tool failures need bounded diagnostics. > - The measured candidate also had independently reproduced question-navigation and oracle defects. > - This change promotes the reviewed production fixes and pins one immutable, qualified source composition. > - Original queries, model, counts, clocks, privacy controls and qualification holds remain intact. ## Linked Issues or Issue Description Refs #18775, #18777, #18779, #18711, #18713, #18718, #18699, #18798, #18799. **What happened?** The measured candidate predates merged native-runtime fixes and contains fixture defects. The timeline omitted recorded reassignment/removal history. Finish-input errors lacked a closed correlation and shape witness. **Expected behavior:** project recorded ownership transitions without inventing facts, distinguish successful native execution from failed verification steps, and retain content-free finish diagnostics. **Steps to reproduce:** repeat an unchanged assignee while editing another field; then reassign or unassign it. Only recorded transitions should appear. Invalid OpenCode finish input should produce a correlated rejection witness without values or schema text. Related #18699 adds gated company OTLP diagnostics. This change uses the existing local callback; it adds no exporter or capture policy. ## What Changed - Promote canonical timeline creation, reassignment and removal projection from #18777. Initial child ownership requires a recorded fresh-creation receipt; delegation requires a readable recorded parent. Unknown legacy facts fail closed. User actors/owners require active company membership; agent actors/targets must exist in that company. Spans, events and edges share that admission filter. Remove the introduced all-history absence query and use guarded indexed UUID lookups. - Add the shared/OpenAPI `unassigned` event kind, preserving newer main contracts and issue-route/service code through reviewed semantic joins. Existing UI consumers remain generic; no new visual marker is claimed. - Add bounded OpenCode finish diagnostics: schema digest, hashed call identity, closed shape enums and validation category. A throwing sink does not change admission or cancellation. - Pin immutable candidate `fd5e3b1e5a379524c0d1651ed8273e75ffd86541`, composed from combined #18777 source `b734393eff0d7ef85b7ad8abd556d2a66ca3a4e6` plus the reviewed MCP diagnostic delta. It preserves actually merged #18775 `97c07d149b5670088faf15317dcf603c75e2af87` and #18779 `a352c057d475067b4fe67375f06785287dfbe0ce`, and merged runtime fixes #18711, #18713 and #18718. - Qualify narrative handling for adjacent repair/verification sentences, intervening native-failure wording, and negated repair objects. Questions, denials and hypotheticals retain their controls. Preserve Previous question beside the qualified Entity type import; separately restore the already-called `internalResultLink` goal helper binding. That import is a runtime fixture repair, not a type-only change. - Include the narrow older-baseline CI typecheck closure and causal heartbeat test observers in the candidate. All four default typecheck projects execute. Main already includes the protected upstream tooling and newer heartbeat fixture, so neither is overwritten with an older candidate file. - Preserve all 244 identities, original queries, model, native/human counts and clocks, including the explicit prior Studio accounting exception. The full catalog stays on its candidate branch; the 196/244 gate and historical SDK runtime holds remain. ## Verification The primary’s f478 retry repeated the real composer-hidden five-second assertion (1 failure,18 passes,1 skip). Investigation established a setup readiness gap: an outer route response can delay mounting the chat thread beyond the assertion. The successor adds the existing real-thread `aria-busy="false"` readiness barrier before the unchanged composer assertion. The original120-second scenario and digest, unread, source-link, history, and reply checks are unchanged. The full owning browser scenario passed naturally in10.2s and with its first real chat response held6000ms in12.0s. This synchronizes fixture setup; the underlying AWS late-mount latency remains unproved. **Current main gates passed:** exact head `823dab49fcd8470140c178771e5c0c6b78725e97` has [successful normal CI](https://github.com/paperclipai/paperclip/actions/runs/41724620555), including both required aggregates, fresh Greptile 5/5 and zero unresolved review threads. Paperclip Review is neutral with a completed substantive 5/5 and no actionable defect. The current optional Superagent Security Scan is neutral/inconclusive: Security Graph context retrieval returned HTTP503, so its assessment did not start. That result supplies no security PASS credit. The primary source PR retains its separate current-head gate before Root sequences normal protected merges. - Reviewed MCP controls: 130 on main and 129 on the older candidate. Meaningful parent comparisons failed before implementation/correlation correction. Controls cover actual offered schema identity, null/omitted admission, bounded rejection categories and throwing-sink cancellation. - Exact unchanged production postimages retain 63 Postgres/public-route controls, nine plan controls and the focused index-plan proof. Owner OpenAPI passes 32 controls; main’s newer contract passes 33. Earlier setup/sandbox failures remain recorded; unchanged database controls were not replayed. - Narrative owner controls pass 216, retaining all prior 207 controls. The independent peer passes 144 balanced cases; earlier 108 and 39-case matrices remain retained. Actual consumer parent failures reproduced the review findings before repair. - Main’s server NoEmit passed on its protected baseline; only two pin files changed afterward. The candidate’s prior full server and four-project checks passed. All server/package/script/root-lock inputs remain byte-identical; the changed Runner project’s guard passes on the current source. Compatibility driver controls pass 47 and six owning suites pass 82. The one explicitly reviewed Docker-only unit omission concerns an unimported image-build feature; all four intended project guards still execute. - Pin/adapter/contract controls pass 47. All 242 expanded declarations, the two legacy dispatch rows, and all 244 singleton private policies retain their authenticated identities. Descriptor, launcher and private-protocol bytes remain exact; declaration loading invoked zero fixture callbacks. - All 506 SDK input blobs match the meaningfully compiled `6abe5739` source vector. Prior real shared/SDK builds measured declaration/map changes and zero runtime-JavaScript changes; a compiled consumer accepts the new kind and rejects an unknown kind. The reviewed registry remains unchanged. Default source proof, 51 SDK controls, 20 custody controls and 16 UI consumer controls remain qualified; runtime/compiler/parser/Zod/canary pins and held runtime grades are unchanged. - The full Mac heartbeat run on prior observer source retained **58 passes / six failures in 802.78s**. Two nonassignee controls subsequently pass focused checks. Four late timeout causes remain unresolved locally; no flakiness/infrastructure cause or complete local pass is claimed. [Actual AWS CI](https://github.com/paperclipai/paperclip/actions/runs/41712023737/job/126182620050) ran this exact heartbeat file with **64 passes in 107.209s**. Its server/package inputs are unchanged in the current narrative-only successor. The source-identical f478 heartbeat file also passed all64 AWS tests in105.812s; source inputs remain unchanged. Current main ordinary CI passed; the primary source PR must independently close its latest-head ordinary CI gate. - Historical main [run 41713049555](https://github.com/paperclipai/paperclip/actions/runs/41713049555) recorded six synchronized runner-shutdown failures and cancellations; the shutdown cause is unknown. A separate chat passive-control assertion observed one `db.transaction` call where zero was expected. Finite local probes passed 2+2 but did not establish a cause or late stack. No external publication, cleanup cause, source fix or flakiness is inferred. The primary’s separate composer-visibility CI failure also remains an inconclusive historical result; no guessed fix is credited. No old-head retry was used. - Earlier UI/inbox fixture failures and the CLI startup timeout remain historical. UI/inbox repairs are now upstream in #18799/#18798 and main preserves those files verbatim. The CLI timeout cause remains unknown. Broader local repository/Rust qualification is not claimed; fresh hosted CI supplies broader coverage. - All four actual repair merges, an exact merged-tree/diagnostic source join, and fresh controller/source/private-policy/active-run guards precede the ten-case TestOps measurement. No paid or model eval was launched for this PR; the original three passes/seven failures remain unchanged. ## Risks The next ten-case model result is unproved. Earlier MCP timeout and continuation-shape causes remain unknown; diagnostics provide future evidence without relaxing validation or rewriting grades. Prior local/CI failures are retained, including the unresolved chat assertion and local heartbeat timeouts. Initial ownership requires canonical in-window audit facts. Missing legacy facts remain unknown. The additive `unassigned` contract is synchronized across shared/OpenAPI/server consumers, with no new UI marker. SDK runtime holds and the broader 196/244 qualification gate remain. Normal protected merges require fresh checks and resolved feedback; no CODEOWNER or branch-protection bypass is authorized here. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: repository analysis, code changes, source validation, and local test execution. The context-window size is not exposed by this harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — current focused/type/pin controls pass; the prior full Mac heartbeat failures remain disclosed and receive no local PASS credit - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — current main normal CI and required aggregates passed; optional neutral reviews and separate primary gate are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact current main head; full thread pagination verified - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 16:17:51 -06:00
DottaandPaperclip	451d06e2fc	fix: resume interrupted Runner preparation and confirm failure report delivery (#18853) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 16:10:49 -06:00
DottaandPaperclip	2920d88034	fix: preserve cancelled runs through native finalization (#18857) ... ## Thinking Path > - Paperclip helps people manage AI agents that do work. > - Native runs can accept a result before workspace file finalization ends. > - A user can Stop the run while file finalization is still pending. > - The direct finalizer replaced that cancelled run with the earlier provider status. > - The restart projection already preserves a cancelled run. > - This pull request preserves cancellation in the direct projection too. > - Saved results and normal task decisions remain available. ## Linked Issues or Issue Description **What happened?** An accepted successful result can wait for workspace finalization. If run-only Stop completes first, the run and its source wake become cancelled. Late finalization then changed the run to succeeded, cleared the cancellation error, and replaced its finish time. The wake remained cancelled. **Expected behavior** Preserve the winning run cancellation, finish time, errors and audited acknowledgement. Keep the accepted provider result and complete file finalization. The server task arbiter still decides task status from the accepted result. A valid result can finish the task even though the execution was stopped. **Steps to reproduce** Accept a native result through the control-plane port. Hold an actual workspace-finalization operation. Stop the run through the heartbeat service with a board request identity. Release the operation and call the direct finalizer. The new database regression exercises this sequence. Related work: #18804 settles exact claimed native wakes. This change preserves a wake that Stop already cancelled. #15212 and #12301 address the opposite race in legacy cancellation. They do not change this native finalizer projection. ## What Changed - Evaluate run status and errors against the current locked database row. Preserve a cancelled run when a late accepted result finishes finalization or when workspace/status-decision retries exhaust. Merge finalization failure metadata into the current row so concurrent receipts survive. - Keep finalization receipts and accepted results. Preserve existing server task-arbiter behavior and accounting fields. - Add real database tests for successful and failed accepted results, delayed workspace work, and Stop after the task decision but before run projection. - Check cancelled wakes, audited acknowledgement, no successor, restart projection and idle recovery. Update execution documentation. ## Verification - Before the production change, the actual producer sequence failed: the cancelled run became succeeded, its error code became null, and its finish time advanced. This failure remains saved privately. - The focused projection, cancellation-request and Stop-accounting suites pass all 67 tests. These include the same delayed-finalization sequence, both task status orderings, and complete and late accounting receipts without another provider turn. - Independent review found a second terminal writer in retry exhaustion. On the unchanged previous head, two real database cases changed a cancelled run to failed or succeeded after workspace or status-decision retries. A third case let actual board Stop win after the permanent workspace failure admission snapshot and still changed cancelled to failed. All three now pass. Existing uncancelled exhaustion controls still pass. No mock supplied the canonical cancellation state. - Initial review found that routine regression notes do not belong in the feature maps. Both map additions were removed. Production and test code are unchanged by that correction. - Runner TypeScript build and all server typecheck layers pass. Map, module-boundary and diff checks pass. - An additional direct typecheck of the normally excluded projection fixture reports two existing notification mocks whose transaction parameter is typed as `object`. The same check with the previous test file reports the same two errors. The new regression cases add no type errors. The standard server typecheck remains green. - The previous head passed required CI after one targeted retry of jobs that received the same runner shutdown signal. Those original failures remain recorded. The exhaustion correction passes fresh required CI on exact head `84404d34b9d982d8bca24b63ae2046051f5169b8` (workflow 41726120483, attempt 1). Both required aggregate gates pass. Greptile reviewed this head at 5/5 with no new comments; all review threads are resolved. An independent source review found no blocking issue in the correction. Runner TypeScript and all server typecheck layers pass locally; the exact production server and Runner build passes with the server stamp bound to `84404d34b9d982d8bca24b63ae2046051f5169b8`. The full required CI workflow passes. - This is local fixture evidence. It does not establish hosted recovery, a remote provider stop, or a deployed version. ## Risks - Accepted result finalization can still finish the task after run-only Stop. This is existing server policy; cancelled run API write authority stays revoked. - The fix protects cancelled execution history. It does not invent provider termination, saved-file or accounting proof. - No schema, endpoint, retry budget, provider call or new background worker changes. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used code execution, repository tools and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 16:10:21 -06:00
DottaandPaperclip	f187de99aa	Preserve domain deadlines across owned idle sleep (#18656) ... Reviewed and verified at 9b4acc456d. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 15:40:56 -06:00
DottaandPaperclip	0a1e4c3d0f	fix(workspaces): retry pristine remote setup timeouts (#18852) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. **What happened?** A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. **Expected behavior** A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. **Steps to reproduce** 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. **Paperclip version or commit** Base: `e0276e0166`, including #18827. **Deployment mode** Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate **Paperclip Review** check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 15:36:07 -06:00
Devin FoleyandPaperclip	909d86b59a	fix(native): retry runs that fail before provider work (#18854) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start the provider through Runner. `session.open` starts the provider session before any prompt. > - When `session.open` fails, the server records the generic `native_session_interrupted`. Recovery evidence then reports `ambiguous_state`, so the first failure blocks the task with no automatic retry. > - The failure is often temporary. On one instance an agent's provider account hit its usage limit. For about 14 hours every later run failed at `session.open`, and 47 tasks stayed blocked. Outside limit windows, the same error also occurs once and a later retry succeeds. > - The server already has a bounded same-run bootstrap retry for failures that are proven to come before provider work. It did not recognize this failure, and it did not consider a known provider limit. > - This pull request classifies a definite `session.open` failure as a pre-session failure. Bootstrap retries back off and wait out a known provider limit. The task is blocked only after the shared budget is used, with a clear reason. > - The benefit is that a temporary startup failure or a provider limit no longer strands tasks on the first failure, and every existing replay guard stays in force. ## Linked Issues or Issue Description Refs #18847. That pull request keeps the provider cause on `session.open` rejections and names typed categories (for example `classification=provider_rate_limit`). This change does not depend on it. With #18847, a typed limit at `session.open` uses the same cooldown path, because the server reads the cause with `providerFailureFromError`. Refs #18622, which added the `ProviderFailure` cause. Refs #13993, an open change about legacy adapter startup backoff; it does not touch the native recovery path. **What happened?** A native run (`paperclip_runner`) failed at `session.open` before any prompt: `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. The server recorded `native_session_interrupted`. Recovery evidence had no checkpoint and no provider event, but the cause was not in the pre-session list, so the mode was `ambiguous_state`. The coordinator went to `terminal_failure`, a board-owned `active_run_watchdog` action was created, and the task was set to `blocked` on the first failure. During a provider usage limit, every later run of the agent failed the same way, so each task was blocked. **Expected behavior** A run that provably failed before provider work retries automatically, with backoff and a bounded budget. If the agent has a known provider limit, the retry waits for the cooldown. The task is blocked only when the budget is used, and the recovery action explains why. A run where provider work may have started is never replayed. **Steps to reproduce** 1. Make Runner reject `session.open` for a native run (for example a provider at its usage limit). 2. Observe the coordinator: `terminal_failure`, `recoveryMode: "ambiguous_state"`, `providerSessionEstablished: false`. The task is `blocked`. 3. The new `native-startup-retry.integration.test.ts` reproduces this with a real database and the executor. **Paperclip version or commit** `master` at `5877d751c0e`. **Deployment mode** Native Runner with an ACPX Claude provider. The fix is in the server and is not specific to one deployment mode. ## What Changed - `nativeSessionFailureSourceCode` returns a new cause, `provider_session_open_rejected`, when Runner reports that `session.open` failed or was rejected. Those outcomes are definite, and the command runs before any prompt. An `indeterminate` outcome stays `native_session_interrupted`. Earlier, more specific causes (for example a missing runtime package) keep their classification. - `nativeProviderRecoveryEvidence` treats the new cause as pre-session. The durable evidence still decides. A bootstrap retry happens only with no provider session, no provider event and no checkpoint. Otherwise the mode stays `ambiguous_state` and the task is blocked as before. - New module `native-runtime/native-startup-retry.ts` plans each bootstrap retry. It is called only for `bootstrap_retry` mode. - Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then about 60 s, plus or minus 20 %). The jitter derives from the run and attempt, and it is persisted with `nextAttemptAt`. - Limit cooldown: evidence is a `rate_limit` or `quota_exhausted` provider failure on the current failure (the #18847 shape), or on an agent run that started in the last 24 hours. A `native_provider_usage_limit` or `provider_quota` error code also counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`), or one hour after the failure when no reset time is known. A later successful run of the agent clears older evidence. The cooldown is capped at 24 hours. The retry runs after the cooldown plus the backoff delay. - A failed evidence read is logged and falls back to plain backoff, so it cannot stop the failure from settling. - The budget does not change: three provider attempts, shared with checkpoint resumes and replacements. When it is used, the coordinator records `native_session_retry_exhausted`. The task is blocked, and the board-owned action says: "Provider startup failed on 3 attempts before any provider work began. Automatic retries are stopped." When a limit is known, the message includes the expected reset time. - While a retry waits, the coordinator detail, the recovery action evidence and the scheduled wake record `providerCooldownAt`. The next action names the reset time, and the task projection shows "Retry scheduled". - `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to `system-recovery-state.ts`. The legacy provider-quota recovery default now refers to it. The value does not change. - A chat retry of an exhausted `provider_session_open_rejected` run is accepted like the other pre-provider causes. The existing proofs still apply. - `doc/execution-semantics.md` describes the backoff, the cooldown and the exhausted state. ## Verification - Red: before the fix, the first new integration case failed. The coordinator was `terminal_failure` with `recoveryMode: "ambiguous_state"`, `originalFailureCode: "native_session_interrupted"` and `providerSessionEstablished: false`, and the task was `blocked`. Four more cases failed the same way. The two never-retry cases passed before and after the fix. - Green: `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` in `server` (embedded PostgreSQL, the real executor, and the real resume claim). 8 cases pass: - a rejected `session.open` is retried with backoff (attempt 1 in 24 to 36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an agent-owned action and a `resume_native_run` wake; - after three attempts the task is blocked, the action is board-owned with the clear reason, and the resume claim refuses another attempt; - a recent limit failure holds the retry until one hour after that failure, and the claim refuses it before the cooldown; - a provider reset time from an earlier run, or from the startup failure itself, sets the cooldown; - a distant reset time holds the retry for at most one day; - a later success, another agent's limit, and an expired limit do not delay the retry; - a durable provider event, or an indeterminate `session.open` outcome, still blocks the task with `ambiguous_state` and no retry. - `native-session-executor.test.ts`: all 1,087 tests pass, including a new classification case for `failed`, `rejected`, `indeterminate` and a different command. - `chat-channels.integration.test.ts -t "requires terminal-safe native ownership for an exact chat retry"`: 23 pass, including the new `session_open` case. That case fails without the list change. - Related suites pass: native session resumption, native provider failure, capacity and overload integration tests, provider failure classification, native status arbiter corpus, native restart recovery. - `npx tsc --noEmit -p server` passes. `git diff --check` passes. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964) is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`, typecheck, build, all server, chat and workspace groups, all serialized server suites, and all Runner checks, which run the native-runtime server tests. In the first attempt, eleven jobs stopped because their runners received a shutdown signal; no test reported a failure. A rerun of those jobs passed. - A local run of the whole `server/src/services/native-runtime` directory was stopped before completion because it runs serially for a long time. CI ran those files. - Runtime dependency: the new work is one indexed query (`company_id`, `agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap retry is planned or exhausted, not on the normal path. No new timer or polling: the wait uses the existing persisted `nextAttemptAt` deadline and the existing resume dispatch. - Not verified: a live provider account at its usage limit, and a live remote sandbox that waits through a full cooldown. The full repository suite was not run locally; CI ran it. ## Risks - A task that waits for a cooldown stays `in_progress` with its execution lock, for up to one day. This is the existing same-run retry state. Stop, reassignment, pause and an explicit Retry keep their current behavior through the existing claim checks. - During the wait the failed run keeps its environment lease in the same way as any same-run retry. A remote lease with an unexported prepared workspace is retained by the orphaned-lease sweep until the retry runs. Other active leases of the failed run are released after the usual five minutes, and the retry acquires a new one. - A `rate_limit` failure without a reset time (for example a short HTTP 429) delays only startup-failure retries of that agent, by up to one hour. Normal runs are not delayed, and a later success clears the evidence. - A startup failure that needs user action (for example a sign-in failure) now uses two automatic retries, about 90 seconds in total, before the task is blocked. Before, it was blocked at once with an ambiguous reason. - The agent's other queued runs are not held during a cooldown. Each one fails at `session.open` without provider work and then waits for the same cooldown. An agent-wide admission hold has no clean extension point in the current queue, so it is not part of this change. - Re-arming consumed scheduled monitors is out of scope. This change makes it matter less: a run that consumed a monitor wake now retries itself on the same run. A monitor that a run consumed before its retry budget was used stays consumed. - No schema or API change. The new error code appears on runs and in run error codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with extended reasoning and tool use (shell, file editing, test execution). The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:24:59 -08:00
3 people	4bdcd1cffc	fix(recovery): release holds for unstarted dispatch cancellations (#18838) ... Release false recovery holds for legacy dispatch cancellations whose retained evidence proves execution never started. Keep audit history, current admission gates, review-wait Retry, and retained workspace obligations. Share the execution-absence proof across cancellation policies and make the dispatcher error-code policy exhaustive. Add PostgreSQL coverage and stabilize the browser history fixture by waiting for pause cleanup before seeding it. Fixes #15483 Refs #18085 Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 14:54:59 -06:00
DottaandPaperclip	2a16dcf629	fix(workspaces): preserve shared directory trees during enrollment (#18835) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server needs existing task files in durable object storage. > - Historical workspace records can share a physical folder or contain other task folders. > - A task-level migration hold cannot fence every alias, and capturing a parent recursively duplicates nested task data. > - Offline enrollment imports children first and verifies their durable coverage before omitting them from a parent archive. > - Snapshot retention preserves those child versions for as long as a covering parent depends on them. > - Existing files, task bindings, and archived states remain intact through the storage migration. ## Linked Issues or Issue Description Builds on merged #18826. **What happened?** The per-workspace importer could capture a shared project folder while another workspace record still owned the same path. It also recursively included runtime-created nested task workspaces in each parent import. Packaged enrollment rejected existing workspace data instead of invoking its migration owner. **Expected behavior** Require offline maintenance for overlapping paths. Import nested tasks before their parents, verify every excluded directory against its saved snapshot, preserve dependencies through retention, and resume interrupted imports without replacing existing remote authority. **Steps to reproduce** Create a project workspace and another task workspace below its `.paperclip/worktrees/` directory. Try importing either while ordinary writers are running, then run offline enrollment. Interrupt a parent import after upload and resume it. ## What Changed - Add physical-overlap admission checks and an offline entry-point guard, including global running and queued executions. - Import pending records in leaf-first pages through the existing migration owner. - Verify nested snapshots by downloading, restoring, and comparing the original source before and after parent upload; check transitive dependencies. - Publish parent dependency receipts under child snapshot row locks. Retention pins those children and revisits them when the parent is deleted. - Index imported-child dependency lookups with a partial GIN index; a PostgreSQL query-plan regression covers 2,500 unrelated snapshots. - Propagate cancellation through capture and object storage operations and preserve existing retry journals. - Wire workspace migration into packaged maintenance and document its preservation contract. ## Verification - The original overlap and duplicate-parent regressions failed before implementation. - 73 migration and retention tests pass against real PostgreSQL and files, including interrupted uploads, resumed batches, equal-path aliases, changed or missing child data, nested descendants, and active or queued executions. - Six S3-compatible storage integration tests pass, including migration through the maintenance owner and restoration of both parent and child after deleting the original filesystem. - The standalone process case was initially skipped because the worktree had no bundle. After rebuilding the server and runtime bundle, that test passed against PostgreSQL and S3: authenticated ready/export receipts, preserved decision secret, blocked application routes, and clean SIGTERM shutdown. - Server typecheck, server build, runtime bundle build, and module-boundary checks passed. Full repository CI passed on 97136a20b9 after targeted retries of infrastructure-interrupted jobs. - Review fixes resolve physical paths through symlinked ancestors and persist each parent journal’s original capture scope, including pre-change journals. All three new regressions failed before their fixes. An initial abort/retry test exposed JSON object-key ordering in the saved scope; deep equality fixes that case, and the full migration suite passed. - Three new fixtures initially omitted required workspace fields; one also omitted its task binding. The fixtures now satisfy the existing contract; production validation was not relaxed. - The parent PR's full local suite stopped on a chat retry-notice timeout and a following fake-timer failure. All four relevant cases passed separately without source changes; this is not a claim that the full local suite passed. - No staging enrollment or live rolling-deploy result is claimed by this PR. ## Risks The offline flag is an internal maintenance contract: Cloud must actually stop every predecessor deployment before starting the process. Deep or large trees require additional object downloads and source comparisons and may exceed one maintenance window; import journals support retry. Dependency lookups are company-scoped and event-driven, with no added idle polling. Unavailable archived sources and unresolved recovery/copyback still block migration and require separate owner work. Unsupported local providers and cleanup_failed workspaces deliberately remain enrollment blockers: filtering them from both the batch and eligibility would abandon local files when the volume is detached. Two regression tests prove that rejected records keep their files and do not acquire remote authority. Original folders and the original volume remain retained. The additional review request to ignore parallel-write policy, stale task ownership, or a missing task binding in offline mode is intentionally not applied: stopping deployments does not establish those semantics. #18841 documents the separate owner actions and adds three offline regressions preserving their files and blocking detach. The batch reports the first failed workspace and supports journaled retry; it does not promise aggregate diagnostics. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change and contains no internal task ID - [x] I have run the focused tests locally and they pass - [x] I have added tests for the changed behavior - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address reviewer findings before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 14:50:14 -06:00
DottaandPaperclip	5877d751c0	feat(server): bind idle credential eligibility to incoming wake proof (#18639) ... Reviewed and verified at d1e3afa4af. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 14:30:07 -06:00
DottaandPaperclip	0b87308d5d	fix(native): defer warm continuation during predecessor finalization (#18811) ... ## Thinking Path > - Paperclip manages agent work and preserves its saved results. > - Native warm sessions can continue on a retained remote Runner. > - A completed predecessor can still have a live finalization lease. > - The continuation must wait for that owner before it can start work. > - Two admission checks treated that temporary lease as ambiguous state. > - This pull request uses the existing deadline-based wait for both checks. > - The same business run can continue once, with its saved context and gates. ## Linked Issues or Issue Description **What happened?** A compatible remote warm continuation can fail before provider work starts. Its predecessor has completed, but a finalizer still holds a live lease. The warm-session check reports `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The earlier retained-sandbox check has the same problem on the command transport. **Expected behavior** Keep the unstarted continuation on its existing run. Wait until the recorded owner deadline. Recheck ordinary ownership, Stop, credentials and checkpoint evidence before execution. **Steps to reproduce** 1. Complete a remote warm turn and retain its controller records and provider session. 2. Give that completed predecessor a live finalization lease. 3. Admit a compatible follow-up with full local controller projections. 4. Before this change, admission fails permanently. It must instead persist a bounded pre-provider wait. **Paperclip version or commit** The PostgreSQL regression failed before production edits on `adb49a4434`. The current head is `5762dfbe4e`, based on master `e491b23eb5`. **Deployment mode** Native remote sandbox execution. No database migration or provider artifact change is required. Related: #18718 refreshes stale warm-controller projections. #18343 adds controller replacement recovery. #18554 restores suspended remote sessions. This change handles a live predecessor finalizer after those source checks. The shared historical error does not identify its failed predicate. This regression proves the finalizer race; it does not prove that every historical instance of that error had this cause. ## What Changed - Defer a compatible warm continuation while its completed predecessor has a live finalization lease. - Use the same wait in the preceding retained-sandbox read check. - Preserve the current run, provider attempt count and checkpoint. Reuse the existing resumption scheduler. - Add PostgreSQL cases for both finalization phases, lease expiry, company scope, incompatible configuration, unfinished commands, Stop and duplicate resumption claims. - Add an opt-in three-turn Linux Runner case. Update the child-controller fixture to own and register a local API listener. - Document the waiting behavior and acceptance case. - Repair two readiness barriers in the refresh-worker integration fixture after a required CI shard failed twice: wait for paginated startup and for active worker promises to settle before checking idle SQL. ## Verification - **Native regression RED:** before production edits on `adb49a4434`, the PostgreSQL boundary case received the permanent identity error instead of `NativeControllerRecoveryWaitingError`. - **Regression GREEN on the current head:** all **71** focused recovery, settlement, heartbeat safety and refresh-worker cases pass on `5762dfbe4e` with Node 24.21.0 and PostgreSQL 18. They cover incompatible identities, company/configuration mismatches, expired or missing deadlines, unfinished commands, Stop, retained files and duplicate resumption claims. - **Real process acceptance on the current head:** the three-turn `finalizer-wait` case passes in 58.01 seconds. PostgreSQL, TLS, a rebuilt Linux Runner and the deterministic Codex app-server fixture confirm zero attempts and no provider turn during the wait, one claim resuming the same run, retained history, a healthy third warm turn and final suspension. The sibling `warm-suspended` case from master also passes, in a fresh container, in 46.02 seconds. Both runs use `5762dfbe4e`; the artifacts were rebuilt from that checkout with the repository's pinned Rust 1.97.1 toolchain. - Acceptance used Node 24.21.0 and glibc 2.41. Earlier probes exposed the missing fixture API listener and the preceding retained-sandbox guard; both are corrected. A stale disposable listener and an incompatible libc image failed setup before the successful fresh-container run. These setup failures are not the original regression's RED. - Repository typecheck/build passed locally before the rebase. Server typecheck passes after the rebase. **Full CI passes on the current head:** 48 successful jobs and one expected scope skip, including typecheck, build, Runner checks, all serialized-server shards, general tests, E2E and the final aggregate. Module-boundary and whitespace checks also pass. - Broad local inventories are incomplete: six chat shards passed (1,834 tests), but untouched server/UI/adapter cases hit timeouts. The exact shared-base expiration case passes alone; a full-file rerun passed all 89 assertions but timed out in teardown. A Node 24 rerun timed out in another lifecycle case and teardown. A later PostgreSQL recheck could not allocate shared memory; this task's redundant broad controllers were stopped and their logs retained. - CI attempt 1 failed a Cursor Cloud cancellation deadline, an E2E cleanup cancellation with a PostgreSQL lock timeout, and the refresh-worker startup settling limit. One failed-only retry passed Runner and E2E; the same refresh startup case failed again. Two aggregate jobs had previously stalled in the hosted-runner queue and blocked the retry; that obsolete attempt was cancelled. No repeated blind rerun is being used. - **Readiness repair RED/GREEN:** real PostgreSQL 18 behind a loopback proxy with 20 ms delay per direction reproduces the exact startup error before the test change; the same probe passes afterward. An initial controlled probe hit the dedicated-database naming guard before tests ran and is excluded from RED evidence. The full-file check then exposed a separate false-idle barrier: a worker could be running while its scheduler had no timer. The fixture now also checks the existing active-work tracker. All **24 cases pass** with both repairs. The at-most-six scoped SQL reads and zero recurring idle-SQL assertions are unchanged; startup alone has an explicit bounded condition wait. No production behavior or timeout changes. - Greptile and Paperclip Review are both **5/5 on the current head**, with no open findings. Both reviews used source inspection; the runtime evidence above was obtained separately. The previous review workspace was recovered on the first production attempt after repairing its partial-clone export. Its preserved result and real follow-up both finalized successfully, saved snapshots and released their writers and finalization leases. Three current-head CI shards received hosted-runner shutdown signals and were cancelled. All three passed their one justified retry. The queued aggregate had to be cancelled to schedule that retry; the final aggregate also passed. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41706617909). - This acceptance does not qualify a live Codex model or Daytona API run of the native repair. No deployment or recovery of the original task is claimed. ## Risks - A wait must not grant a second executor. The current attempt stays at zero until the existing scheduler claims it. Expired or missing deadlines retain their existing hold. - Saved state must survive the wait. Admission still requires the same company, task, session, configuration and physical sandbox evidence. - Current Stop, budget, credential and permission gates remain with ordinary resumption. No polling loop is added. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. Capabilities: reasoning, repository inspection, code editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 14:18:18 -06:00
DottaandPaperclip	1f55073f96	fix(cursor-cloud): support API-only controllers on managed stacks (#18310) ... ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:55:39 -06:00
DottaandPaperclip	4d31588dea	test(server): load model route setup before test deadlines (#18828) ... ## Thinking Path > - Paperclip uses native agents and local provider adapters to do work. > - The server model route must keep provider catalogs separate. > - Its tests use the real HTTP route with controlled service and adapter fixtures. > - Each test reset the module graph and imported the route again inside short test deadlines. > - Slow imports timed out. Unfinished work could then change mock call counts in a later case. > - This pull request loads the stable route graph during collection and keeps each case's fixture resets. ## Linked Issues or Issue Description Refs #18816 (affected required CI run). **What happened?** The pending Kimi model-route test exceeded its 15-second deadline in required Linux CI. The unchanged test file also failed locally: the first setup hook exceeded 30 seconds, later tests exceeded 15 seconds, and the last OpenCode case saw two calls instead of one after unfinished work reached the shared mock. **Expected behavior** Route import and transformation must finish before the case deadline starts. Each case must still test the real HTTP route with its own mock state, environment and profile values, and adapter registry cleanup. **Steps to reproduce** Run `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts` with a cold module graph. On a loaded worker, repeated module resets and imports can exceed the hook and test deadlines. ## What Changed - Register stable service mocks once and load the real route and middleware during test collection. - Remove per-case module invalidation. Keep all five original HTTP assertions, fixture resets, profile and environment restoration, and adapter unregister operations. - Keep the existing hook and test deadlines. Product source code does not change. No user-facing documentation changes are needed. ## Verification - The unchanged file reproduced the hook timeout, test timeouts, and leaked call-count failure locally. - All five original cases pass with the fix. The cold graph takes about 47 seconds during collection; the cases complete within the unchanged deadlines. - All five cases also pass at the same head in a shuffled order with seed `20261103`. - Module-boundary and diff checks pass. - Full repository typecheck, tests, and build run in required CI. No hosted product behavior is claimed from this test-only change. - Current-head CI attempt 2 failed in the typecheck harness because the trusted master workflow called `scripts/typecheck-test-projects.mjs`, which the older source head did not contain. The log is retained. A normal merge of master supplies the helper from #18738. All five original HTTP cases pass again on that merged baseline. Fresh CI and exact-head reviews passed at `86f0bf27267fe3f44ab813fb0a57ebbb94bae7fe`. - The refreshed run passed all server checks but one browser shard failed: `task-reload.spec.ts:63` waited five seconds for a task heading after the intentional bundle failure. The trace showed the saved task breadcrumb and properties, with the conversation still loading. The exact failure log and trace were retained. One targeted retry of that shard and its aggregate passed without source changes. This establishes recovery of the check; the original browser loading delay remains unexplained. - Final exact-head checks: 52 success, two skipped, and two neutral. Both Greptile and Paperclip reviews are 5/5. No review threads remain. ## Risks - Stable imports share module state across cases. The test retains explicit mock resets, adapter unregister, and profile/environment restoration. - The affected model route reads profile qualification and environment fixtures when it handles the request. These cases do not require a new module instance for each environment change. - No runtime or schema changes. No test deadline increase. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used repository tools, code execution, and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:50:15 -06:00
DottaandPaperclip	f836fec59e	perf(budgets): avoid company lock for routine eligibility checks (#18830) ... ## Thinking Path > - Paperclip helps people manage AI agents at work. > - The budget service checks whether an agent can start work. > - Preliminary checks acquired the company accounting lock even with no budget policy. > - Independent checks then waited behind accounting writes for the same company. > - This pull request reads one consistent snapshot before it decides whether reconciliation is needed. > - Healthy checks avoid the writer lock. State changes and final paid admission retain their existing authority. > - The benefit is less company-wide contention during routine eligibility checks. ## Linked Issues or Issue Description Refs #17697 (the preliminary-check portion). Refs #17817 (related serialization in agent configuration). Related approach: #17966. That PR adds a precheck for policies that cannot block. This PR also avoids the lock for healthy active hard-stop policies and manual pause reads. It uses a repeatable-read snapshot and leaves public transactional admission locked. It does not add the inactive-policy spend filtering proposed in #17966. **Current behavior** Every public `getInvocationBlock` call takes the company writer lock. Healthy checks can wait behind an unrelated receipt or settlement operation. **Proposed behavior** Read current scope state, policies, exposure and relevant incidents in a read-only snapshot. Return directly when no transition is needed. Otherwise discard the snapshot and reconcile current state under the company lock. **Reason and benefit** Independent preliminary checks can finish while another transaction owns the company row. Final reservations and paid-work admission still serialize against current accounting state. ## What Changed - Separate preliminary eligibility inspection from mutating reconciliation in the budget service. - Share scope loading, pause decisions and soft-incident resolution conditions between the two paths. - Keep no-policy checks free of exposure scans and incident reads. - Add inspection and reconciliation spans to the existing operator-gated Observability path. - Add PostgreSQL contention tests and regressions for pauses, holds, incidents, recovering native accounting and company boundaries. - Use the shared fixture startup timeout. Per-test deadlines and the 500 ms lock timeout stay intact. - Update the implementation contract and tracing documentation. Add dated investigation and future-design notes. The future accounting architecture remains a proposal. ## Verification - 73 targeted tests passed across eight files on implementation commit `244db565`. No tests were skipped. The files cover budget service, enforcement scoping and delivery, live receipt attempts, budget history, run settlement, heartbeat accounting and budget tracing. - Real PostgreSQL tests hold the company accounting lock while routine checks finish. The same reader's authoritative admission instead fails with PostgreSQL lock timeout `55P03`. - Full workspace typecheck passed before the rebase: `node scripts/typecheck-low-memory.mjs`. - Full workspace build passed before the rebase: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r --workspace-concurrency=1 build`. The Runner release binary was rebuilt from source. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Initial local setup failed before assertions: cold fixture setup exceeded 20 seconds, then a missing SDK build was exposed. The fixture now uses the shared 90-second startup allowance. The SDK was built before the successful test run. An incomplete local Copilot artifact also blocked the first build; the pinned installer repaired it before the successful canonical retry. - The branch rebased cleanly onto master `83b5f7d486`. All 73 targeted tests then passed again on clean head `244db565ae206502b100c3cdf5cd86a0ae91b372` (eight files, no skips). The full local Vitest suite was not run; remote CI must verify the wider suite. - Initial remote CI failed three monitor-scheduler cases at the suite's three-second heartbeat-idle wait. The unchanged suite then passed all 21 tests locally (`pnpm exec vitest run src/__tests__/issue-monitor-scheduler.test.ts`, 70.13 seconds). The one failed-job rerun was cancelled after the review follow-up was pushed; it produced no pass/fail result. No scheduler assertions or timeouts were changed. On updated head `ba2ab2a0`, the unchanged monitor suite passed all 21 tests in 38.381 seconds in [server shard 10/12](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126151059714). That shard passed 115 files and 2,677 tests, with no failures (one file and 20 tests skipped by existing suite configuration). - The independent review identified missing recovering-run coverage. The added contention case requires the locked fallback while another accounting writer holds the company row, then verifies the recovery block without pausing or cancelling recovery. The regression is in follow-up commit `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`. All 19 budget-service tests passed locally (no skips, 48.89 seconds), and `pnpm --filter @paperclipai/server typecheck` passed after the follow-up. - The updated CI run then hit a separate plugin-refresh fixture timeout at its final re-enable settle. The unchanged `external-object-refresh-work.test.ts` suite passed all 24 tests locally (52.02 seconds). Inspection identified a possible fake-clock race: its quiet helper can observe no scheduled deadline while reconstruction is still active, then encounter the existing 30-second reconstruction cooldown. The one same-head failed-job retry [passed](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126158227876): refresh suite 24/24, shard 115 files and 4,125 tests passed (one file and four tests skipped by existing configuration). No assertions, timeouts, or product code were changed. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41702687492) passed on exact head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: build, typecheck, Runner checks, all server/workspace suites and E2E. Both required aggregates are successful. All 59 checks are terminal: 54 successful, two advisory neutral and three expected skips. The PR is mergeable with a clean merge state. - Deployment latency and CPU effects have not been measured. - Greptile reviewed updated head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: 5/5, no actionable findings or unresolved Greptile threads. - Independent Paperclip review on the updated head: advisory 4/5, no blocker, recovery gap confirmed resolved. One canonical maintainability advisory remains visible and is documented below. ## Risks - Preliminary inspection can precede a concurrent policy or pause change. It grants no paid dispatch authority. The final locked admission and reservation checks remain required. - Healthy configured policies add an incident lookup. An unheld blocking scope can compute exposure during both inspection and locked revalidation. Already-held scopes skip the first exposure computation. - Holds and incidents must retain their existing transition effects. The fallback reads them again under the company lock, rather than mutating from the earlier snapshot. - Eligibility inspection and reconciliation still maintain separate transition lists. Future reconciliation changes must update the inspection predicate and its contention coverage together; consolidating the effects model remains a nonblocking review follow-up. - No schema or API response change is required. This change does not relax the current budget enforcement contract. ## Model Used OpenAI Codex with model `gpt-6.1-sol`. Used reasoning, repository inspection, code execution and delegated review. The session did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:48:56 -06:00
DottaandPaperclip	e491b23eb5	fix(runner): retire sandbox suspension receipts after admission (#18829) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Native Runner sessions preserve a conversation across task turns. > - Remote suspension records a local receipt for the stopped Runner identity. > - The built-in sandbox path now writes the same receipt as plugin tasks. > - Admission cleared that receipt only for plugin tasks. > - A later warm turn could reject a clean journal because the old receipt named an earlier run. > - This pull request clears the receipt after successful admission for both sandbox paths. ## Linked Issues or Issue Description Refs #18343 and #18554 for the shared suspension receipt producer. Refs #18718 for the related controller projection refresh. **What happened?** After a suspended built-in sandbox session resumed, its next warm turn could fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Its journal named the completed resumed run. Its local suspension receipt still named the older run. **Expected behavior** Successful admission must retire the prior suspension receipt. The next compatible warm turn must reuse the admitted session and provider history once. A failed admission must retain recovery evidence. **Steps to reproduce** 1. Complete and suspend remote turn A. 2. Resume turn B through the built-in sandbox transport. 3. Keep the controller warm, with full Runner state owned by the sandbox. 4. Admit turn C. The old local receipt conflicts with B's journal identity. **Paperclip version or commit** Reproduced in the executor regression on base `367f29478d`. **Deployment mode** Native Runner with the built-in remote sandbox transport and a warm lifecycle. ## What Changed - Clear the local suspension receipt after a remote sandbox session passes provider recovery and durable admission. Apply this to built-in runners and plugin tasks. - Extend the existing three-turn warm admission regression to cover a suspended receipt and remote-only Runner state. - Keep an active-owner control that rejects admission and retains its receipt. Keep the existing foreign-owner, retirement, configuration, plugin, and unchanged-session controls. - Add a real Linux Runner continuity case that suspends A, resumes B, and keeps B warm for C. Register the test fixture's owned local API listener. - Document receipt ownership and the new continuity case. ## Verification - The new behavioral regression failed before the production change. Turn C raised the exact identity mismatch above. The active-owner control passed and retained its receipt. - All 1,087 native executor tests pass after the fix. - Both opt-in Linux process cases pass at the final commit: ordinary warm reuse and suspension followed by warm reuse. They use PostgreSQL, TLS, current-source Rust binaries, and a deterministic Codex fixture. Each preserves one conversation, starts exactly three turns, and reuses the warm Runner and provider on turn C. - Full repository typecheck passes with `node scripts/typecheck-low-memory.mjs`. Full production build passes with `pnpm build`. - All current-head CI checks pass at `a95a6fdfe8`, including the full test shards, browser shards, Rust checks, build, and typecheck. The policy job needed one retry after repository checkout timed out before checks ran. - Greptile and Ponytail each give 5/5 at the final head. There are no open review threads or findings. - The local full `pnpm test:run` attempt passed 351 cases in its first chat shard. The session interruption stopped it during the second shard, so it has no complete local verdict. The complete CI test run passes. - Module boundary checks and `git diff --check` pass. ## Risks - A receipt removed before admission could discard recovery evidence. Removal stays in the successful `onSession` callback. The negative admission control checks retention. - A stale owner must not gain authority. Journal, run, lease, and scope checks remain in place before the callback. - Historical contradictory or quarantined state still requires the existing verified recovery path. This change does not delete those records. - No migration or provider configuration change is required. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific model ID or context limit. Used reasoning, repository tools, code execution, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:42:21 -06:00
DottaandPaperclip	d8a9b3eb0c	fix(ui): show task identifiers in activity toasts (#18802) ... Project the company-scoped identifier for authorized task events and use it in toast titles, fallback bodies, and links. Retain authorized delivery when the optional identifier lookup fails. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 13:28:15 -06:00
DottaandPaperclip	fb6554ac62	fix(native): settle source wake with terminal projection (#18804) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each agent wake request can bind to one heartbeat run. > - Native recovery can finish and project a run after its live executor has stopped. > - The legacy completion callback settles the wake, but native terminal projection did not. > - A succeeded run could therefore leave its exact wake claimed with no finish time. > - This pull request settles that claim in each native terminal projection transaction and reuses durable projection recovery for old committed rows. > - The benefit is consistent run and wake receipts without a provider replay or a new sweeper. ## Linked Issues or Issue Description Refs #13856 and #10109. These open PRs describe orphan claimed wakes and cleanup after terminal persistence. This change fixes the native projection owner and uses its existing durable worker. It does not modify those contributor branches. **What happened?** A native run can be accepted and committed, then reach a terminal state through recovery. Its source wake stays claimed because the live completion callback does not return to settle it. **Expected behavior** Terminal projection must settle only the same company's exact source wake bound to that agent and run. Completed, failed and cancelled outcomes must retain their meaning. Other live, queued, cancelled, unbound or newly rebound wake records must remain unchanged. **Steps to reproduce** Commit an accepted native result in the real database fixture. Leave its source wake claimed. Call native finalization with run-status projection enabled, without the live completion callback. Before this change, the run becomes succeeded and finished while the exact wake remains claimed. ## What Changed - Settle the exact source claim inside native finalization, committed recovery, and exhausted-finalization projection transactions. Fence it by wake ID, company, agent, run and current claimed state. - Lock the matching claim and register settlement notifications before the write. A failed transaction rolls back both run projection and wake settlement. - Let the existing durable projection worker discover a terminal run with an unfinished exact claim. A repaired committed row leaves the pending set. No new timer, sweep or table is added. - Add real database regressions for native recovery, all three terminal outcomes, old committed receipts, idempotence, scoped denial controls, notification ordering and rollback. Cover provider/workspace finalization exhaustion and denial when the caller does not own terminal projection. - Preserve the existing provider, accounting, workspace, permissions and result contracts. No API, schema, user command, access-map entry or documentation contract changes are needed. ## Verification - The new native coordinator/finalizer regression failed before the production change: the run was succeeded and committed, but the bound wake was still claimed. - The initial focused database run did not reach the assertion because the SDK was unbuilt. A second attempt hit sandbox loopback restrictions. A third exposed skipped package library hydration. These setup failures were corrected before the meaningful red was captured. - Real PostgreSQL projection and response suites passed 89 tests before the independent review correction. The reviewer requested notification intent registration before mutation. The correction adds a notification-order and rollback case. All 90 tests now pass across both suites. - Server typecheck passed after building the isolated checkout's Runner and SDK prerequisites. Fresh corrected-source server typecheck and production server build also pass. Full repository checks run in current-head CI. The earlier serialized-9 CI failure was the independently repaired inbox snapshot fixture in #18798 (expected nine reader calls, observed eight). A normal merge of the containing master supplies that fix; the native suites plus the repaired inbox suite now pass 96 tests together at merge head `456c92ef8d60c3fcbeaf66ff8010a7ebd3d1109d`. Server typecheck passes. That head was superseded by the reviewed retry-exhaustion correction below. The subsequent review found a third terminal write in retry exhaustion. Real database provider and workspace exhaustion regressions failed with claimed/null-finished wakes before the fix. The same fenced helper now runs inside that transaction. At final source head `9718d4c2ec6066accf549dc6fea795891f6b90ec`, all 100 focused tests, server typecheck and production server build pass, including projection-disabled denial and rollback of run/coordinator/wake on notification failure. - Final current-head CI passed: 54 success, 2 skipped, and the neutral Paperclip Review at 5/5. Greptile also reports 5/5. Required ci / verify and ci / e2e passed. All review threads are resolved. - No live task write, provider work, accounting repair, deployed replay or manual rollout was performed. ## Risks - A stale wake binding must never be settled. The update requires the exact wake ID, company, agent, run and claimed state. Tests cover unbound and newer-run bindings, foreign company and agent, queued and cancelled records, and retained live ownership. - Old claimed debt now participates in the existing projection predicate. It uses the source wake primary key and stops matching after settlement. Existing quiet-worker tests remain in the focused suite. - Wake settlement must share the projection transaction. Notification failure and outer rollback tests preserve both original rows. - Interruption, timeout, retained historical terminal-failure coordinators, and delayed response presentation keep their current owners. This change does not infer or replay provider work. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation where needed; existing contracts are restored - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:17:16 -06:00
DottaandPaperclip	e0c6992426	fix(heartbeat): prevent saved input starvation during queue reloads (#18819) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accepted messages and task Retry requests use durable wake receipts. > - The recovery worker loads receipts in pages of 100 and checks batches of 25. > - Company activity can reload those pages before the pending batch drains. > - Reloads reset ready receipts' dispatch times and can repeatedly favor receipts already checked. > - This pull request preserves the priority of unchecked ready receipts so saved input can reach normal admission. ## Linked Issues or Issue Description **What happened?** Saved input can remain deferred after the preceding run finishes. Repeated company eligibility changes reset the dispatch times of pending receipts during paginated reconstruction. A PostgreSQL regression checks only 75 of 178 receipts on the original code despite twelve eligibility changes. **Expected behavior** Each ready receipt must reach its admission check. Existing wait deadlines, retry backoffs, authorization, and execution gates must still apply. **Steps to reproduce** 1. Seed more than one page of deferred receipts. 2. Keep those receipts pending after their checks. 3. Send company eligibility notifications between scheduler passes. 4. Check that every receipt is reconsidered, including receipts later in the backlog. **Paperclip version or commit** Reproduced on `367f29478d`. **Deployment mode** Built from source with PostgreSQL. The defect occurs before provider execution. Related: #10195 reports a similar starvation symptom in a different promotion loop. #18621 added saved-input Retry after a successful run. This change fixes fairness in the deferred-receipt worker. ## What Changed - Add an optional entry merge callback to scoped deadline reconstruction. - Use it in the deferred-receipt worker to retain unread ready receipts' priority while refreshing their state. - Place receipts whose wait has expired into the ready queue during reconstruction. - Keep real wait, claim, and dispatch deadlines authoritative. - Add a paginated starvation regression, real process dispatch coverage, and GitHub lock-collision recovery coverage. - Document the recovery fairness rule. ## Verification - The PostgreSQL starvation regression checks only 75 of 178 receipts before the fix. It checks all 178 afterward, both with ordinary receipts and with expired execution waits. - Seven focused scheduler cases pass, including moved interrupt grace, moved native error deadline, fresh claim/wait deadlines, and unchanged-gate grace. - GitHub endpoint lock-collision recovery, retained board authority, HTTP Retry, and five current admission-gate cases pass. - Real process acceptance passes on the final code: task Retry crosses the repeatedly invalidated backlog, reconstructs its persisted queued run after a worker restart, saves `receipt-proof` in run output, and has one successor for the Retry receipt. The test holds executor capacity while advancing queue time, then starts fresh workers at PostgreSQL time before releasing a real process lease. The defect is in admission scheduling before provider execution; this acceptance uses PostgreSQL and a local process adapter. - The full workspace build and typecheck (`node scripts/typecheck-low-memory.mjs`) pass locally. The affected worker suites pass 52 of 53 cases together; the existing native multi-company batch case times out in that combined run and passes in isolation without code changes. All new regression and acceptance cases pass. - Local `pnpm test:run` stops in its first chat shard on three 15-second timeouts (Teams long native HTTP picture delivery, direct final-response materialization, and contended inbound notices). All three pass in isolation without code changes. Later full-suite groups did not run locally after that shard failure. - [The full GitHub Actions matrix passes](https://github.com/paperclipai/paperclip/actions/runs/41697489374) on `ac91df2cc6db6c77529d923177b375b8c92efd45`, including all general-test, serialized-server, browser, build, and typecheck jobs. Server shard 8 initially failed an upstream process-adoption test and passed its one rerun without code changes. - [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/18819#issuecomment-6612091535) on the same current head, with no actionable findings or unresolved review threads. The separate Paperclip Review check is neutral; its previous request was bound to an older commit. Focused command: ```sh pnpm exec vitest run server/src/__tests__/deferred-execution-receipt-work.test.ts server/src/__tests__/scoped-deadline-work.test.ts ``` ## Risks - Ready receipt order changes during repeated reconstruction. Admission still reads current durable state. - A ready receipt must not inherit an obsolete priority after it acquires a future deadline. The merge applies only when both pending and loaded entries are immediate. Deadline safety tests cover that boundary. - The merge callback is optional. Other scoped workers retain their current reconstruction behavior. ## Model Used OpenAI `gpt-6.1-sol` in Codex, max reasoning effort, with tool use and local code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused regressions and acceptance; full-run timing limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 13:17:00 -06:00
Nicky LeachandClaude Opus 5.5	8ab89f41e4	fix(build): typecheck test projects in 4 GiB (#18738) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents typecheck this repository, often on machines with limited memory > - paperclipai/paperclip#18737 made `pnpm -r typecheck` fit in 4 GiB because the server now checks in layers > - The test projects outside `pnpm -r` still import server source through relative paths, so each one checks most of the server as one program > - These projects peaked between 6.9 GiB and 14.9 GiB, so they fail on a 4 GiB machine > - This pull request makes these projects read the server layers as declarations, and adds a CI step that keeps them that way > - The benefit is that every TypeScript project in the repository can now be checked in 4 GiB, and a later change cannot silently undo this ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged. This pull request builds on the server layers that it added. No issue exists for this gap. **What happened?** `tsc -p tests/runner-e2e/tsconfig.json` peaked at 14.9 GiB at default settings. `tests/speko-native` peaked at 12.2 GiB, `server/src/__tests__/helpers/maintenance` at 9.6 GiB, and `tests/lifecycle-baseline` at 6.9 GiB. On one thread, `tests/runner-e2e` still peaked at 7.7 GiB. **Expected behavior** Each project passes with 4 GiB of memory. **Steps to reproduce** Run `tsc -p tests/runner-e2e/tsconfig.json` in a Linux container with `--memory=4g --memory-swap=4g`. The kernel stops the process. ## What Changed - Added `tsconfig.layers.json` to the four projects. Each file extends the project's `tsconfig.json` and references the generated server declarations. - Added `scripts/typecheck-test-projects.mjs`. It builds the server declarations and then checks the named projects, or all four. It takes `--low-memory`. - Changed `test:e2e:runner:typecheck` and `test:lifecycle-baseline:typecheck` to call that script. `docker-runner-check.yml` calls the first script by name, so that workflow needs no edit. - Added `--declarations-only` to `server/scripts/typecheck-layers.mjs`. The script now also generates `tsconfig.declarations.json`, so hand-written configs do not name individual layers. - Exported `runPnpm` from `scripts/typecheck-low-memory.mjs`, so the new script can use it. - Added a step to the `Typecheck + Release Registry` job. It runs straight after the server typecheck and reuses its declarations through `--reuse-declarations`. - The script now fails if a project loads server source, even when the types pass. Test support files, declaration files and JSON files do not count. This check does not enforce a memory limit. - Added a unit test that fails if a new project under `tests/` is neither in `PROJECTS` nor named as a single program. `tests/runner-acceptance` is the one named project. - Added `server/src/types/express.d.ts` to `tests/lifecycle-baseline/tsconfig.json`. That plain config failed alone with 11 TS2339 errors, and it now passes. - Exported `isTestFile` from `server/scripts/typecheck-layers.mjs`, so the source check uses the same rule as the layers. - Turned off `declaration` in the layered configs. These projects emit nothing, and declaration checks gave a false TS2883 error when `tests/runner-e2e` read the server as declarations. - Added tests, one of which runs the real compiler. Updated `doc/DEVELOPING.md` and `tests/lifecycle-baseline/README.md`. The references are not in each project's `tsconfig.json` because Vitest reads that file. A test showed that Vitest fails when a referenced config does not exist yet. The `tsconfig.json` files do not change, so editors and Vitest see the same configs as before. No new `package.json` scripts were added, as `.greptile/rules.md` requires. ## Verification I tested commit `9763880fab`, which is based on `1c9ca5829f`. I copied the commit into a named Docker volume and ran a fresh install. Each row ran in its own Linux arm64 container with `--memory=4g --memory-swap=4g` and 8 CPUs. The first row started with no build output. All rows used `--low-memory`. CI does not set a memory limit. I used the limit only for this measurement. | Project | Errors | Peak (cgroup) | Max process RSS | OOM kills | Time | | --- | --- | --- | --- | --- | --- | | `tests/runner-e2e` | 0 | 3.34 GiB | 3.06 GiB | 0 | 57 s | | `tests/speko-native` | 0 | 3.60 GiB | 3.27 GiB | 0 | 51 s | | `tests/lifecycle-baseline` | 0 | 3.43 GiB | 3.15 GiB | 0 | 50 s | | `server/src/__tests__/helpers/maintenance` | 0 | 3.39 GiB | 3.13 GiB | 0 | 52 s | `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:e2e:runner:typecheck` and `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:lifecycle-baseline:typecheck` also passed in the same limit, with peaks of 3.55 GiB and 3.52 GiB. I also ran the CI sequence in one container with the same limit: `pnpm --filter @paperclipai/server typecheck`, then `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations`. Both passed, with a peak of 3.46 GiB. The new step took 16 s. The server layer build sets the peak in each row. Against the declarations, the projects peaked between 0.9 GiB and 2.2 GiB at default settings on macOS. Server source files that each project loads, without test support, declaration or JSON files: | Project | Plain `tsconfig.json` | `tsconfig.layers.json` | | --- | --- | --- | | `tests/runner-e2e` | 1,345 | 0 | | `tests/speko-native` | 1,647 | 0 | | `tests/lifecycle-baseline` | 1,343 | 0 | | `server/src/__tests__/helpers/maintenance` | 1,716 | 0 | I counted these on `3f0f31b2f9`. The source check in the script confirmed 0 for all four projects on this commit. **Limits of this proof:** - Without `--low-memory`, the 4 GiB run does not pass. The kernel stopped the layer build for `tests/runner-e2e` on `3f0f31b2f9`, and the script exited with status 1. I did not repeat this run on the final commit. The script does not turn on low-memory mode by itself. - The Linux runs used arm64, not x64. - The new CI step has not run in CI. `pr-trusted.yml` runs from `@master`, so the step first runs after this pull request merges. Check its first run. **Other checks on this commit:** - `server/scripts/typecheck-layers.test.mjs`: 38 tests pass. One test uses the real compiler to show that the source check tells a layered config from a config with no references. - With the references removed from `tests/speko-native/tsconfig.layers.json`, the script exits with status 1 and reports 1,644 server source files. - With the declarations removed, `--reuse-declarations` exits with status 1 and reports TS6053. - A type error in a test project makes the script exit with status 1. - `tests/lifecycle-baseline/tsconfig.json` passes with 0 errors. - `tests/lifecycle-baseline` Vitest suite: 533 tests pass. - Workflow tests that read `pr-trusted.yml` and CI routing: 114 tests pass. **Earlier checks:** - `server/src/__tests__/server-package-build-script.test.ts` passed 10 tests on `3f0f31b2f9`. This test failed in `ci / General tests (server (9/12))` on an older head. It tested a build script split that the final #18737 removed. - A changed signature in `server/src/services/run-liveness.ts` failed `tests/lifecycle-baseline/authority.test.ts` through the rebuilt declarations, before the rebases. When I restored the signature, the check passed. **Not run locally:** - `pnpm test:run` and `pnpm build`. CI covers these. **Failures that this pull request does not cause:** `pnpm test:e2e:runner:unit` has 13 failures on this commit, in `cursor-cloud-replay-fault`, `first-task`, `operating-receipts`, `pi-controls-evidence` and `workflow-security`. The same 13 tests fail on `master` at `6ef37398c8`. A comparison of the test names shows no differences. PR CI does not run this suite. ## Risks - Low risk to runtime behavior. No shipped code changes. - **Slower check:** `pnpm test:e2e:runner:typecheck` now builds the server layers on each run. Before the rebase, it took 30 s on macOS at default settings, about the same as the old single program for that project. On this commit it took 57 s in low-memory mode in the 4 GiB container. The new CI step reuses the declarations, so it took 16 s. - **Narrower check:** Server source is now checked only by the server's own layers, not again under each project's compiler options. The projects also skip declaration-portability checks. These projects emit nothing, so those checks had no output to protect. - **Unchanged path:** `tests/runner-e2e/native-completion-checks.mjs` still runs `tsc -p tests/runner-e2e/tsconfig.json` as one program. It needs about 15 GiB at default settings. I did not change it because that file and its inputs are part of a fingerprinted source contract, and no workflow calls it. - **Layer growth:** The new checks do not measure memory. If the layers or the import cycle grow, the checks still pass and the peak rises. This is intentional. Crossing 4 GiB does not fail a build. - **Project list:** The CI step checks only the projects in `PROJECTS`. The unit test covers new projects under `tests/`, but not a new project in another directory. - **Reused declarations:** `--reuse-declarations` trusts the declarations on disk. CI uses it straight after the server typecheck on the same commit. If that typecheck fails, the job stops first. Stale declarations can pass locally, so `doc/DEVELOPING.md` warns against this. - **Job dependency:** A type error in a test project now fails `Typecheck + Release Registry`, and so `ci / verify`. - **Not covered:** The new step runs no tests. `node scripts/typecheck-low-memory.mjs` and the Docker image build also have no new guard. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:09:12 -08:00
DottaandPaperclip	4a38ff9c92	fix(ci): make Cloud readiness reruns and fixtures reliable (#18823) ... ## Thinking Path > - Paperclip manages agent work and preserves task state. > - Cloud readiness verifies an exact source revision before release. > - Its workspace tests must wait for execution and database cleanup to finish. > - Two attempts hit the same ten-second polling limit and read a running row. > - The second attempt also lost AWS runners and exceeded a fixture cleanup limit. > - This pull request waits for owned execution, bounds client shutdown, and gives verification reruns an independent runner pool. > - Assertions, source identity and required verification gates stay in place. ## Linked Issues or Issue Description **What happened?** [Cloud readiness run 41684102136](https://github.com/paperclipai/paperclip/actions/runs/41684102136) failed twice on master `367f29478d14003cc5c0f161b7ce8f3164993634`. Attempt 1 failed `heartbeat-workspace-busy.test.ts:473`: expected `succeeded`, received `running`. Attempt 2 failed the same helper in the inherited-workspace resume case at line 744. The helper polls for ten seconds, then returns the active row as if the wait finished. Attempt 2 also exceeded the 30-second `workspace-preparation.test.ts` teardown limit. Its client close had no deadline, and database plus directory cleanup shared the default hook budget despite the fixture's existing 90-second startup budget. Other attempt-2 failures show runner shutdown or lost communication. Several shutdown signals arrived at 18:05:17 UTC. Read-only AWS inspection confirms that all eight infrastructure-failed instances carry `runs-on-workflow-job-interrupted=true`. Five have explicit `Server.SpotInstanceTermination` reasons, including the server-1 runner that lost communication. Three were terminated through the fleet's shutdown path. CloudTrail records the Rust runner's interruption tag at 18:03:37 UTC and its termination by the RunsOn fleet role at 18:05:31 UTC, despite EC2 showing a user-initiated termination. Repeating the entire workflow reused the same Spot fleet. **Expected behavior** Wait for owned heartbeat execution before reading final state or scheduled retries. Bound database-client shutdown and allow the fixture's cleanup cost class to finish. Rerun verification on GitHub-hosted runners after an interrupted fleet attempt. Preserve all checks and report prior failures. **Steps to reproduce** 1. Delay the test adapter beyond the helper's ten-second polling window. 2. Invoke the ordinary local shared-workspace run or resume the deferred sandbox run. 3. Observe the old helper return `running` before the execution completes. 4. Evaluate the old runner expressions with `github.run_attempt = 2`; they still choose the interrupted AWS pool. **Paperclip version or commit** Base: `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions Cloud readiness, with disposable PostgreSQL fixtures. Related: #11293 fixed the earlier gap between cancellation and retry-row insertion. This change waits for the complete execution instead. Duplicate searches covered the workspace test, retention fixture and runner shutdown reports. ## What Changed - Replace both workspace-test polling helpers with the existing heartbeat completion barrier. Keep final status, retry and coordination assertions. - Bound workspace-preparation client shutdown to one second. Apply the existing embedded-PostgreSQL fixture budget to its teardown. - Use AWS only for the first exact-source attempt. Route later verification, chaos and cloud-migrator planning/build attempts to GitHub-hosted runners. The publisher keeps its existing routing. The final branch also incorporates upstream #18821's CODEOWNER routing fix. - Add runner-expression coverage for numeric, string, missing and later attempt values. Document retry routing. ## Verification - The original workflow's failed assertions and interrupted jobs remain linked above. A local two-suite run also passed all 157 assertions and failed only the original 30-second teardown hook. - All 341 workflow contracts pass. After the master synchronization, all 347 routing and CI-planner controls pass. Actionlint passes for the three changed workflows. - A matched delayed-adapter comparison uses disposable PostgreSQL 17 databases, the same 11-second adapter delay, and the same 90-second experimental outer budget. The old helper fails with `running`; the completion barrier passes. Production test deadlines stay unchanged. - Earlier experiments hit a 30-second experimental outer limit and the macOS host's shared-memory limit during native database startup. The completed comparison uses task-owned databases on an existing test server. Those earlier attempts are not counted as passes. - All 31 workspace-busy cases pass with normal test deadlines. The 126-case workspace fixture also passes with its 90-second teardown budget. - Final-head cleanup measurements: main client close 7 ms, owned database cleanup 20,938 ms, and workspace-home removal 826 ms. A measured fixture held 124,494 PostgreSQL files (208 MiB) after repeated table resets. The measured database cleanup alone consumes most of the old 30-second hook budget. The separate baseline reproduced a timeout at that hook; the fixture now uses its established 90-second cost class. - Repository typecheck and build pass on synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2`. - All 157 focused cases pass on the synchronized branch, including complete PostgreSQL 18 fixture shutdown. [Full hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41697804471) passes on the final head, including the required verification and E2E aggregates, all server/chat/workspace groups, Runner lanes and all eight browser shards. The separate GitHub quality-review workflow also passes. All 60 reported checks are terminal: 54 pass, three are informational neutral results, and three are expected skips. There are no failures or pending checks. The complete local `pnpm test:run` command was not run; targeted fixtures already encountered this host's shared-memory limit. Hosted CI supplies the full Vitest and browser matrix. - Greptile and Paperclip Review both give synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2` 5/5. There are no open review threads. - Module-boundary and whitespace checks pass. ## Risks - Hosted reruns can have different cache hit rates and queue times. They execute the same source and verification commands. - The first attempt retains the existing AWS Spot fleet policy. AWS evidence establishes several interruption-triggered failures; later verification attempts use an independent pool. This does not change the fleet's external capacity policy. - The completion barrier waits for owned background execution. A hung execution still fails the test's outer deadline. - The larger teardown budget remains finite. PostgreSQL's bounded stop protocol and cleanup errors still fail the hook. - No production execution, database schema or provider contract changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code edits and test execution. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 13:08:28 -06:00
DottaandPaperclip	ccc63ea458	fix(server): prove completed history before idle sleep (#18646) ... Reviewed and verified at 592bce69c6. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 12:37:25 -06:00
DottaandPaperclip	1c9ca5829f	perf(ci): overlap Runner workloads and cache debug workspace builds (#18794) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect its server and Runner contracts. > - Recent full runs waited on general server files and the final Runner lane. > - That Runner lane ran package tests before native server tests. > - PR jobs also rebuilt unchanged debug workspace binaries. > - This pull request overlaps those workloads and reuses exact trusted build inputs. > - The benefit is a shorter critical path with the same test coverage and AWS authorization. ## Linked Issues or Issue Description Refs #18544, #18773, #18774, #18798, #18799. **What existing behavior does this improve?** The wall time of required PR tests and the precision of affected-test selection. **Current behavior** General server and native files use two workers. The second Runner Vitest lane runs its package shard and then the native server group. Each Runner job compiles debug workspace units again. A Vite raw import also retains its consumer for unrelated source edits. **Proposed behavior** Use three isolated workers for general server files and four for native files. Reuse the two Runner slots for a complete package suite and a complete native server suite. Restore all debug binaries only on an exact input key. Record raw imports as explicit file inputs. ## What Changed - Separate the Runner package and native server workloads without adding CI jobs. Preserve the old command arguments for master-pinned workflows, local commands, and other shard counts. - Increase file workers while retaining fork isolation, serial cases, and the shared-database guard. - Add an exact debug workspace cache. Derive binary targets from Cargo. Include source, fixtures, generated inputs, manifests, compiler identity, flags, Cargo configuration, and checkout path in its key. Discard incomplete restores. Only verified master pushes can save it. - Keep older PR checkouts working when they lack the new cache helper. - Track Vite `?raw` imports as repository file inputs. Keep unknown loaders, untracked files, and opaque effects conservative. - Add cache and coverage guards and document the behavior. AWS runner authorization is unchanged. - Keep paired test benchmarking available when install hooks correctly disable materialized dependency reuse. The old benchmark asserted a cache hit in that safe fallback. - Wait for active dispatch work to finish before asserting idle reads. Analyze the two 16,400-row database fixtures before their queries so they have current planner statistics. Suppress only PostgreSQL notices inside the cleanup transaction to avoid logging every cascaded table; warnings/errors remain visible. Row counts, real queries, assertions, database reset, and timeouts remain unchanged. ## Verification - 196 affected CI tests passed, including routing, coverage, selection failure fallbacks, shared-database serialization, and existing/new cache guards. Final worker/cache adjustments also passed their focused checks. - `pnpm build` and `pnpm -r typecheck` passed locally. Full test coverage is being verified by the Linux CI profile; `pnpm test:run` was not repeated as a full serial local run. The paired CI benchmark passed. Both paired benchmarks passed, and full CI passed again after integrating newer master commits. The external Paperclip Review app is the only pending status. - Actual debug artifacts remained fresh after simulating checkout timestamps: Cargo completed in 0.82 seconds; the content key was unchanged. This is local reuse evidence, not a measured GitHub cache transfer. - Same real UI inventory for a retention-test-only change: 861 files; retained selection fell from 413 to 401. All 12 omitted files are raw-input consumers; edits to their inputs remain covered by regression tests. - The first local worker comparison is invalid. Development dependency links disappeared during its second run. The two-worker baseline also had two macOS path assertions in company-skills. Neither result is counted as a successful speedup. Dependencies were restored. A local repeat was interrupted before producing results. The completed Linux comparison below supplies the valid paired evidence. - [Paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41671244016): baseline and candidate ran the same 113 general-server files and 1,777 cases on one host. Wall time fell from 381.29s to 346.89s (9.0%). Both exited successfully and file/case inventories matched exactly. The fixture command took 31.77s versus 28.65s. This is one sequential pair, excluding setup, queueing and the full CI critical path. - [Repeat paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41675496862): the same 113 files and 1,974 cases took 485.06s in the baseline versus 368.54s in the candidate (24.0%). File and case inventories again matched exactly. The baseline includes newer master changes than the first pair, so these are two independent comparisons, not repeated timings of an identical tree. - Initial full AWS profile: package suite passed 4,891 tests (289 files, one skipped); native suite passed 172 tests (9 files, 12 pre-existing platform/feature skips). Native execution took 239.06 seconds; the older sample ran 165 native tests in 391.29 seconds. The package lane now takes 11m01 including setup and debug compilation. This is an indicative cross-run comparison, not a paired full-CI speedup claim. - The initial run failed two UI fixtures, one serialized inbox settlement assertion, and one dispatch idle assertion. The two UI failures reproduced locally. The fixes preserve authorization, admission, and zero-idle-read assertions. All four affected files passed locally: 48/48 tests. Token gates passed. All four also passed in the next AWS profile. That profile passed every other workload but timed out the 16,400-row keyed attention fixture at its existing 180s limit; this is accounted for by the explicit fixture-statistics change. Local focused comparisons pass both large cases before and after; their roughly 33s test execution is unchanged locally, so this is not claimed as an additional speed gain. A full local file run passed both large cases (7.57s / 18.05s) but hit the unchanged 30s teardown deadline on its first test while emitting thousands of cascade notices. Cleanup now suppresses those notices within its transaction; the complete file then passed 34/34 in 107.72s. The latest-head AWS run passed the previously failing shard: the keyed bulk case took 40.25s (previously exceeded 180s), the keyless case took 8.02s, and the complete attention file took 89.61s. The 16,400-row fixtures and original timeouts remain intact. - [Full CI profile before master integration](https://github.com/paperclipai/paperclip/actions/runs/41675497699) passed on commit `5858956d4ac2c99c911ad2447b63e1f8fc84e31e`, including all tests, build, typecheck and final verification. Greptile reviewed that exact commit at 5/5; the only actionable review thread is resolved. The separate Paperclip Review app is still queued. - [Final merged-head CI](https://github.com/paperclipai/paperclip/actions/runs/41678956312) passed on `499aa206a6e03bf8f3249692ea731367a000591d`: 48 successful jobs and one intentionally skipped file-selection job. Backup checkpoint qualification and legacy security status passed. Greptile reviewed this exact commit at 5/5. The PR is conflict-free; only the separate Paperclip Review app remains queued. - Integrated current master after #18798 and #18799 merged the inbox and UI fixture fixes separately. Preserved master's inbox settlement fix and verified its entire file (6/6 tests). Those three files are no longer part of this PR's diff. Removed the temporary `profile-ci` label after both successful benchmark pairs; the normal planner still schedules every job because the CI configuration changes are global inputs. - Second AWS profile: native tests took 229.97s with the same 172 passes / 12 skips; package tests took 427.44s with 4,891 passes / 10 skips. The complete package job took 10m44, including setup and 79s debug compilation. - The PR uses the trusted workflow from master. Candidate worker/layout code runs now; workflow-level debug restores begin after merge and the first successful master cache writer. ## Risks - Extra workers can expose load-sensitive tests. The final full CI profile passed, but one run is not a long-term flake-rate measurement. - The cache is an optimization only. Misses rebuild, and test commands still execute. A new Cargo workspace member requires a cache-contract review. - Compiler alternatives remain experiments. Required TypeScript checks are unchanged. Typechecker experiment: three sequential checks per tool on identical built projects, with TypeScript 7.0.2, Bun 1.4.4, and tsrs 0.20.1-ts7.1.0-beta. Median seconds (TypeScript / Bun / tsrs): shared 0.429 / 0.083 / 0.157; UI 2.554 / 0.390 / 0.740; server 9.949 / 1.194 / 1.782. TypeScript and Bun passed all three projects. All tools caught the same four deliberate assignment, property, nullability, and generic-constraint errors. tsrs reported one additional server TS2345 diagnostic in the run-dispatch adapter, so it did not pass the project comparison. These local compiler times exclude generation and build steps; they are not full CI savings. ## Model Used OpenAI GPT-6 Codex. Tool use and code execution assisted this change. The exact deployment identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:17:53 -06:00