mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
3a2249dd5768313f04b2aa3860dfd363492eda16
1760
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
3a2249dd57 |
fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. **What happened?** A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. **Expected behavior** Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. **Steps to reproduce** 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. **Paperclip version or commit** Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
6a45ea479e |
fix(heartbeat): retry failed legacy lease renewals until the deadline (#17851)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/52/changes/paperclip/52 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy agent run keeps a durable controller lease. The server renews the lease every 10 s. The lease lasts 60 s. > - The lease watchdog stops the run when it loses the lease, because another controller can then take the run. > - #17032 moved renewal to a private pool. It also retries a renewal that hits its statement timeout. But a renewal that fails with any other error still stops the run at once. > - Staging still shows this failure after #17032. A legacy run made 186 good renewals over 31 minutes. Then one renewal failed with an error 23.9 s after the last good renewal, and the watchdog stopped the run immediately (`controllerLeaseLossCause=renewal_error`). The lease still had about 36 s left. > - This pull request retries a failed renewal until the lease deadline, the same as a timed-out renewal. > - The benefit is that one dropped database connection no longer cancels a healthy run. The 60 s lease deadline still bounds the retries, so the ownership guarantee does not change. ## Linked Issues or Issue Description Refs #17032 Refs #16084 **What happened?** A legacy agent run on staging was cancelled mid-turn. The run was healthy. The cancellation evidence was `cancellationSource=controller_lease` and `controllerLeaseLossCause=renewal_error`, with `controllerLeaseRenewalCount=186` and `controllerLeaseSinceRenewalMs=23913`. The release already contained #17032. **Expected behavior** One failed lease renewal must not stop a run while the lease is still valid. The watchdog must try the renewal again. It must stop the run only when the lease deadline passes or when the database rejects the renewal. **Steps to reproduce** 1. Start a legacy run so that `watchLegacyControllerLease` watches it. 2. Make one renewal query fail with a connection error, for example terminate its Postgres backend with `pg_terminate_backend`. 3. See that the watchdog aborts the run immediately with `renewal_error`, although the lease has not expired. ## What Changed - `watchLegacyControllerLease` retries a renewal that fails with an error. It waits 1 s between attempts (`LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS`). The existing lease deadline stops the retries. - A retried renewal is safe. The UPDATE is idempotent, and it still requires this controller's boot ID and an unexpired lease. A rejected renewal (no row updated) still stops the run at once with `renewal_rejected`. - If the deadline passes while the attempts still fail with an error, the cause stays `renewal_error`. A statement timeout counts as a stall, so it stays `renewal_deadline`, as before. A good renewal clears the error state. - `deadlineLateMs` is now recorded whenever the deadline ends the lease, also for `renewal_error`. - Tests: - New embedded-Postgres test: it terminates the renewal's own backend while the renewal waits on the run row. The renewal retries on a new connection and keeps the lease. Without the fix, `assertOwned` rejects. - New unit tests: one failed attempt is retried and the lease is kept; a stall after a recovered error is still `renewal_deadline`. - The existing `renewal_error` test now checks that a renewal that always fails is retried and is reported as `renewal_error` at the deadline, with its timings. - `doc/run-log-events.md` describes the retry and the updated `renewal_error` and `deadlineLateMs` meaning. ## Verification - `server/src/services/legacy-controller-lease.test.ts` with embedded Postgres: 23 tests pass. - With the original `legacy-controller-lease.ts`, 4 of these tests fail. One of them is the real-Postgres test that terminates the renewal connection. - `server/src/services/heartbeat/run-completion.test.ts` and the `run-cancellation` tests: 35 tests pass. - A typecheck of the changed files reports 0 errors. CI runs the full typecheck. ## Risks - Low risk. The change is limited to the legacy lease watchdog. - A run with a permanently failing renewal now stops at the lease deadline (at most 60 s after the last good renewal), not at the first error. Another controller cannot take the run before the lease expires in the database, so ownership stays safe. - A non-transient error (for example a code bug in the query) is retried once per second until the deadline. The cause label is still `renewal_error`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
edb9d29942 |
fix(recovery): preserve Retry authority and reopen cross-task cases (#18997)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks keep execution identity and saved workspace files separate from control-plane updates. > - A scanner can have permission to reopen a case while its run cannot restore that case's files. > - Retry can also follow a preparation failure that has no recovery-action record. > - Both paths can accept valid work but leave the target unable to run. > - This change restores files under the target executor and saves admitted Retry authority on the canonical receipt. > - Existing workspace, user, source and successor checks still apply. ## Linked Issues or Issue Description **What happened?** An authorized agent updates another closed task. The route tries to restore the target workspace under the caller's run identity and returns `403 Task workspace run identity is unavailable`. A separate failure occurs when a normal board Retry follows a pre-provider preparation failure. Admission has no recovery-action row on which to save continuation authority. Dispatch then fails with `continuation_user_authorization_missing`. **Expected behavior** Accept the authorized task update. Restore its terminal workspace when the target's admitted executor starts. An admitted Retry must retain its exact dispatch authority even when the predecessor has no recovery-action record. **Steps to reproduce** 1. Complete a task with an isolated workspace. From a different running task, use an authorized comment or PATCH with `resume: true` on that closed task. 2. Separately, fail native preparation before provider work starts, with no recovery-action row. Submit board Retry, admit its successor, and build the execution continuation. 3. Before this change, the first path returns 403 and the second rejects its admitted authority. **Paperclip version or commit** Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased onto `afbc5a885b5`. **Deployment mode** Server-managed native task execution. Tests use real PostgreSQL and temporary workspace files. Related PRs: #16927 covers terminal workspace races. #17916 retains exact native export sources. Neither covers these route and Retry receipt failures. A GitHub search found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. ## What Changed - Defer workspace restoration only for authorized cross-task comments and updates from a live run. Checkout keeps its existing workspace identity checks. - Restore the target's selected terminal workspace under its own admitted run. Preserve its saved files and existing source checks. Leave manually archived workspaces unchanged. - Store server-created continuation authority on the canonical admitted Retry receipt when no source recovery-action record exists. Existing action records remain authoritative. Read the fallback only after matching the company, task, agent, user, source, generation and successor. - Add route and PostgreSQL regression tests. Document both paths. ## Verification - Red: both cross-task route tests returned the observed 403 before the route fix. - Red: the target executor could not reuse its archived terminal workspace before the restoration fix. - Red: an admitted preparation Retry with no recovery-action row failed with `continuation_user_authorization_missing` before the receipt fix. - Green: 419 recovery and continuation tests pass, plus 183 focused route, environment and Retry tests. The assembled heartbeat test starts the agent in the same terminal worktree and verifies both its unpushed commit and uncommitted file bytes. It fails before the restoration fix. - Negative cases reject wrong Retry runs, users, source runs, generations, unadmitted receipts and missing authority. The original failed run stays unchanged. Workspace tests verify saved file bytes. - Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries` and `git diff --check` pass. - The full suite found an existing assembled chat authority regression: clearing a recovery-action record could be bypassed by the new fallback. The fix now writes the fallback only when that source has no action record. That assembled regression passes with the original test unchanged. - Full `pnpm test:run` is running again. Its first attempt also hit an unrelated ancestor skill-root lookup for a missing Slack skill. The rerun uses a temporary untracked link to the repository's own bundled skills. This link is excluded from the PR. Full-suite completion and hosted CI are pending. - Hosted CI has multiple runner interruptions. E2e shard 1/8, server shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have GitHub annotations reporting lost runner communication; their logs return 404. The `ci / verify` assertion failed because the general-test matrix was cancelled. GitHub rejected job and failed-workflow reruns while this workflow is still running. Rerun the interrupted checks after it settles. - Greptile reviewed exact head `a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5, with no new blocking findings and no unresolved review threads. ## Risks - A deferred workspace restore can still fail at target admission. The run keeps the existing workspace failure path; this change does not bypass file recovery or provider stop checks. - Retry receipts now store server-created authority. Existing action receipts remain valid. No schema migration is required. - Local regression tests establish the software behavior. The new revision has not been deployed to a hosted instance. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, tool use and code execution. The context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b99dd57c82 |
test(evals): run Product regression from TestOps (#18983)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction |
||
|
DottaandPaperclip
|
6364fa7327 |
fix(heartbeat): wait for retry predecessor cleanup (#18989)
## Thinking Path > - Paperclip manages agent work and preserves its task ownership. > - Heartbeat admission controls when a queued run can start. > - A failed run can still own workspace saving and environment cleanup. > - Automatic recovery can transfer the task execution lock to its queued retry before that cleanup finishes. > - The old queue check then sees the retry itself and misses the predecessor's remaining work. > - This pull request checks the exact retry predecessor before claiming the queued run. > - Existing cleanup notifications can then resume that same run through normal admission. ## Linked Issues or Issue Description Refs #18898 and #18691. The first repair covers canonical connected-task retries with a native source. The second covers live remote workspace writers. This change covers ordinary automatic continuation when the predecessor failed before native runtime selection. The search found no open repair for that path. **What happened?** A preparation failure saved a terminal legacy run row while its remote workspace save and environment cleanup were still active. Generic continuation created one queued successor and moved the task execution lock to it. The queue checked that current lock but did not check the successor's predecessor. The new run started and failed at the existing unfinished-writer guard. **Expected behavior** Keep that same successor queued until its exact predecessor finishes execution and cleanup. Check company, agent and task identity before using retry provenance as a wait condition. Preserve Stop, reassignment, failed cleanup and independent recovery gates. **Steps to reproduce** 1. Save a failed legacy preparation run with an active environment lease. 2. Queue its ordinary automatic continuation and bind the task execution lock to the successor. 3. Claim the queued run before the predecessor finishes cleanup. 4. The old code starts the successor. The fixed code retains it in the queue. 5. Commit successful cleanup through the existing cleanup owner. The ordinary dispatcher starts that same run once. **Paperclip version or commit** The causal regression fails on master `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** Source checkout on macOS arm64 with disposable PostgreSQL, the assembled heartbeat service and the production work coordinator. The adapter boundary is controlled. No hosted provider replay or deployment is claimed. ## What Changed - Check the current task execution owner and the retry's exact predecessor under the existing task claim lock. - Use the saved native task binding, ordinary task binding and historical context fallback in that order. Ignore unrelated retry provenance. - Reuse the existing live-executor and pending-cleanup checks. Reuse committed cleanup notifications and the existing dispatcher. - Add real PostgreSQL claim controls and assembled cleanup-event acceptance. Keep the original source failure, saved files and current task ownership intact. - Document the queued predecessor-cleanup rule. ## Verification - RED: the unchanged queue claimed the successor and wrote a new controller lease while its legacy predecessor retained an active remote environment lease. The regression expected the successor to remain queued. - GREEN: the same regression passes after the change. The complete queue suite passes all 71 tests. - The assembled heartbeat suite passes all six tests. The added cases cover one continuation after committed cleanup, saved-file reuse, final task lock release, board Stop, reassignment and failed cleanup. - Scope controls cover a deleted source after scan, another company, another task, another agent, canonical native task binding and the existing current-owner hold. A failed cleanup and a still-live local executor retain the wait. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. - Initial fixture attempts failed on database constraints and observer timing. The final fixtures use valid immutable bindings, the board Stop attribution and a completed real dispatcher call after reading the exact committed lease state. Production ownership guards were retained throughout. - Full low-memory repository typecheck passed at `5cddb49322`. The sync to master `04a24f4e06` adds only the independent retained-allocation test fixture from #18984; production code is unchanged. - Full `pnpm build` passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`. - The post-sync run executes all 80 behavior assertions successfully across the queue, assembled cleanup and updated upstream retention suites. Its queue `afterAll` then exceeds the existing 30-second database shutdown hook. That failed suite result is retained. The controlled post-build queue rerun passes all 71 tests with normal teardown. The earlier shutdown cause remains undetermined. Its owned queue-test PostgreSQL process has exited. The separate local default `pnpm test:run` was intentionally stopped after full unfiltered CI passed on the exact head. Four local chat partitions completed before that stop; this is not a local full-suite pass. The verified owned process group and its PostgreSQL fixture have exited. Logs and retained fixture data remain available. The timeout has not been increased. - Full unfiltered CI passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all six internal chat partitions, all nine serialized server shards, the workspace groups, native Runner lanes, typecheck, build and browser E2E. Both required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146) pass. The file selector is skipped, so this is the full CI test matrix. The head has 55 successful checks and three expected skipped checks. The ready-for-review contributor trust and security scan also pass. - Current-head Greptile is 5/5, its check is successful, and no inline review threads remain. The ordinary review check also passes. No merge, deployment or original task retry was performed. ## Risks - A predecessor with unfinished cleanup keeps its successor queued. Its existing cleanup owner remains responsible for settlement. - A task claim reads at most one additional predecessor row. A matching source also uses the existing ownership and cleanup queries. The change adds no timer, schema, credential, provider or model policy. - A committed cleanup signal only requests normal admission. Current ownership and independent execution gates still apply. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL verification. The session does not expose a more specific serving model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused suites pass; the separate local full-suite attempt is explicitly incomplete above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aaea05e158 |
fix(native): preserve backups when completed warm sessions retire (#18986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps provider history when a warm session ends. > - Object storage must preserve that history before the sandbox is released. > - A completed turn releases its live controller lease. > - The backup owner lookup rejects that released lease and stops the next task preparation. > - This pull request uses the existing completed-run checkpoint fence for backup storage. > - The next turn can prepare after the previous provider state is saved. ## Linked Issues or Issue Description Refs: #18932, #18011, #18984. **What happened?** A warm Codex turn completed and kept its provider session alive. The next turn stopped during preparation with `native_harness_storage_owner_missing`. The backup lookup required a live lease after normal completion had released it. **Expected behavior** Save the completed session under its retained controller generation and committed result. Keep the existing owner and successor checks. Do not start another provider turn during the save. **Steps to reproduce** 1. Run a warm native session with object-backed provider backups. 2. Commit the first turn and release its controller lease. 3. Start the next task preparation and retire the previous session. 4. Observe the backup owner error before provider work starts. **Paperclip version or commit** The regression is present at `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** A hosted server with a remote native Runner and S3 storage. ## What Changed - Accept a released backup owner only when its exact controller record has a committed result. - Reuse the checkpoint owner policy for same-controller preparation and native successors. - Recheck the controller generation, committed result, and released state after upload. - Preserve live-owner rejection and the rule that a late upload cannot replace a newer conversation backup. - Add real PostgreSQL storage and restore regressions. Document the completed-run backup rule. ## Verification - On 2026-11-04, two new regressions failed on the unchanged source with `native_harness_storage_owner_missing`. These covered an idle completed session and a same-controller preparing successor. - After the fix, 54 selected tests passed: 30 durable backup tests and 24 existing early-retirement checkpoint tests. After the fixture-only upstream sync, all 58 selected storage, checkpoint, process, and affected fixture cases passed again. - The storage cases use real PostgreSQL, encrypted archives, object transfer, and restoration. They cover invalid completion evidence, foreign or expired successors, active predecessors, changed generations, changes during upload, and newer backup publication. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'released|backup|early retirement|durably retires an accepted warm Runner'`. - An independent source review found no outstanding code issue. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs` before the upstream fixture-only sync. Production source is unchanged across that sync. - An assembled process probe passed with the Runner binary built from this head and a fake Codex protocol peer. It started real processes, committed a result, released the source lease, claimed the next controller, and completed the source stop, drain, and suspend commands. It recorded one provider turn start and kept the task bound to its successor. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'durably retires an accepted warm Runner'`. - CI exposed the old allocation-readiness test setup in three cases. The branch now includes the merged #18984 fixture correction. The same three cases passed locally after that sync. - An earlier CI test shard was cancelled after its runner received a shutdown signal. Its log was retained before one rerun. Another old-head shard reported failure with no retrievable job log or annotations; its cause remains unknown. Current-head CI must pass independently. - Local `pnpm build` passed at this head. All 1,904 chat cases passed across the supported local shards; the earlier shards used byte-identical source before the fixture-only sync. The interrupted default command is retained and is not reported as a pass. - The local bulk server run had separate enrollment acknowledgment, environment-run status, monitor-settle, idle-history teardown, and repository-rehome failures. In isolation, idle-history passed all 75 tests and the unchanged repository cases passed with their original limits (4.9s large files, 6.4s refs). The remaining four local failures reproduced on unchanged master `04a24f4e06` under the same environment and limits. Their causes remain unproven. Logs and the baseline comparison were retained; the local full suite is not claimed as passed. - Current-head CI first hit the unchanged repository-rehome large-file case's 15s timeout. The same isolated case passed in 4.9s. After source/call-path comparison, one bounded failed-shard retry passed all 115 executed suites; no assertion or timeout was changed. The original timeout remains recorded. - Local verification uses macOS 26.5.2 arm64, Node 26.4.0, pnpm 9.15.4, PostgreSQL 18.1, and Cargo 1.97.1. The current tested head is `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`. - The full unfiltered [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41868587974) passed at `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`: all 12 general server, 3 chat, 9 serialized, workspace, native Runner, build/typecheck, E2E, and final verify lanes. The file selector was skipped, so this was the complete fallback matrix. - Greptile reports 5/5 on this exact head with no open review threads. Disposable local test processes were cleaned up and the source tree is clean. - The real hosted Codex and S3 workflow has not been rerun because the local verification environment has no remote provider credentials. The original task remains unchanged. ## Risks The main risk is a stale controller publishing provider history after a new owner starts. Both storage transactions keep the company, run, boot, and generation fence. A released owner must keep its committed result. Existing successor checks and the newer-backup publication guard remain in force. The live checkpoint writer still requires a non-null lease. No migration or API change is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; separate baseline broad-suite failures are disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
afbc5a885b |
fix(workspaces): save partial and shallow Git checkouts offline (#18990)
## Thinking Path > - Paperclip manages AI agents and their workspaces. > - A completed remote run must save its files before finalization can release its source. > - A shallow Git mirror uses upload-pack and can omit objects that exist only in the index. > - A filtered source can also fail when that mirror requests blobs the source never downloaded. > - Fetching missing history is not required to preserve an existing partial checkout. > - This change saves the available object stores and their existing shallow, sparse, and promisor state. > - The benefit is an offline save that preserves accepted work and remains readable by the existing archive reader. ## Linked Issues or Issue Description Refs #16727, #18705, and #18935. These changes cover managed clones, streaming capture, and safe diagnostics. This change repairs Git snapshot contents. **What happened?** Remote workspace capture invokes a mirror clone before it saves the archive. A shallow filtered sparse checkout with a missing promised blob fails that clone and surfaces `remote_workspace_operation_failed`. An ordinary shallow checkout can also lose an index-only object during the mirror transfer. **Expected behavior** Capture preserves the objects already available in the checkout, including staged work. It preserves partial, shallow, and sparse semantics without remote hydration. Corrupt or unsupported state must keep its source and receive no successful save receipt. **Steps to reproduce** 1. Create a real Git repository with two paths and commit history. 2. Clone it with `--filter=blob:none --depth=1 --sparse`, then select one path. 3. Stage a change, edit it again, and add an untracked file. 4. Remove access to the original repository and capture the workspace. 5. The base capture fails before transfer. The changed capture restores the available work after the original source is removed. **Paperclip version or commit** Reproduced on `86814c655`. The same transport/archive sources are present in `76e43852a`. ## What Changed - Preserve existing object files for shallow and partial repositories, including their original promisor pack markers. - Preserve shallow boundaries and the selected sparse settings and patterns. - Validate available object contents with full Git fsck. Reject unknown corruption instead of creating new promises. - Deny Git network protocols during capture. Keep the ordinary mirror's explicit local file transport. - Exclude ambient Git location, index, object-store, config, and template overrides. Disable templates for init and clone. - Keep the existing archive format, source fences, path guards, credential allowlist, and failure retention. - Add real command and reader regressions for full, shallow, filtered, sparse, corrupt, and unsupported repositories. ## Verification - Red: the compiled Node 24 transport completed helper installation, then failed capture with the same generic operation error and no transfer. Dirty source bytes remained intact. - Red: the new full/shallow/partial transport-to-reader cases exposed both filtered upload-pack failure and a missing staged-only object. Independent review then reproduced accepted corrupt object bytes and ambient template config/hook copying. - Green: all 110 transport/archive cases pass after those repairs. They cover staged, dirty, and untracked work; ordinary and forged-promisor corruption; malformed filters; alternate stores; unsafe links; credential redaction; and template isolation. An initial linked-worktree config regression was fixed before this complete passing run. - A compiled new capture restored through both the unchanged earlier reader and the new reader after source removal. HEAD, status, work bytes, sparse/shallow state, and the same missing promised object were preserved. Full Git fsck passed. No remote protocol was permitted. - Full repository `pnpm -r typecheck` and `pnpm build` pass at `91ca00a8327216def91303e19940e92ffe07c08f`. The build stamp names that exact head. - The [complete current-head CI run](https://github.com/paperclipai/paperclip/actions/runs/41865961403/attempts/2) is green: all 58 checks are terminal with no failure or pending check. Three expected skips and one neutral review remain those states. - Preserve the [first CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41865961403/attempts/1): one browser case reached its 5-second heading deadline while the conversation pane was loading; a chat worker reported shutdown; another browser worker lost communication and its log blob was absent. A failed-job-only retry passed at the same source head. No test deadline, assertion, or source change was made for that retry. - The complete local `pnpm test:run` passed all 1,904 chat cases across six partitions. Its later general-server phase timed out two unchanged tool-access cases at 15 seconds. That known-red command was stopped before completion. Both cases then passed unchanged in a fresh isolated process. The original timeout cause remains unknown; this is not a full local-suite pass. - [Greptile reports 5/5](https://github.com/paperclipai/paperclip/pull/18990#issuecomment-6635931211) at the current head with no actionable finding. The product review's loose-object cost advisory was [answered with local measurements and resolved](https://github.com/paperclipai/paperclip/pull/18990#discussion_r4470261833). There are no unresolved review threads. - Original service acceptance remains unverified. Recorded checkout text is mechanism-compatible narrative evidence, not proof of its physical Git state or original failure cause. This PR includes no deployment or additional live recovery. ## Risks - Full object validation and copying available metadata use CPU and temporary space proportional to the local object store. - The offline path rejects alternate object stores and unsupported partial filters. Such sources remain protected. - Promised objects remain missing. Later work still needs ordinary repository access to request them. - There is no archive format or schema change. Existing readers restore standard Git files, and existing full snapshots remain covered. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and local verification. The host does not expose a more specific serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
86814c6550 |
fix(recovery): ready retained sandboxes before preparation (#18934)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84e9c8f364 |
fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. **What happened?** A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. **Expected behavior** Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. **Steps to reproduce** 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. **Paperclip version or commit** Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. **Deployment mode** The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5d226ca08a |
perf(chat): reuse bound-run authorization query templates (#18966)
This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. **Current behavior** At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. **Proposed behavior** Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. **Reason and benefit** Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2cbfc35a07 |
perf(github): reuse the inherited review wait query template (#18965)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. **Subsystem affected** Server GitHub chat review waits. **Current behavior** Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. **Proposed behavior** Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. **Reason and benefit** Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
396b7fc62a |
fix(runner): scope session diagnostics to the active run recorder (#18944)
## Thinking Path > - Paperclip manages agents and their work. > - The native executor can retain a transport between runs. > - Each run recorder has a shorter lifetime than that transport. > - The transport kept its first recorder after warm reuse and completion. > - A later close could write to the wrong run or a sealed object log. > - This change binds diagnostics to the active run epoch and retires that binding at execution exit. > - The existing recorder still owns accepted-byte draining and honest incomplete finalization. ## Linked Issues or Issue Description **What happened?** A warm native transport retained the first run's diagnostic callback. Reusing the session for another run did not change that callback. A later idle close could append to a retired writer and emit a persistence warning. **Expected behavior** Active diagnostics belong to the current company, session and run. A retired transport callback must not use a completed recorder. Diagnostic metadata must not delay essential execution cleanup. **Steps to reproduce** Run the native warm-session regression with two sequential run owners. Close the retained transport after both runs finish. The original code calls the first recorder. The real completion regression also observes the resulting closed-writer warning after normal lease cleanup. This is a local source defect with behavioral red/green proof. Repeated hosted generic persistence warnings motivated the investigation, but their original exceptions were not retained. This PR does not claim to identify every such warning or repair an export failure. Related #18379 queues sanitized performance spans in heartbeat. It does not own transport diagnostic callback lifetime. ## What Changed - Bind transport diagnostics to an exact active company/session/run epoch, including warm reuse. - Activate a retained channel in the existing session-admission hook after identity validation and before attachment starts, including an attachment failure before `onSession`. - Forward active diagnostics synchronously. Detach their recorder at execution exit. Keep the original asynchronous metadata semantics. - Keep idle transport evidence inspectable. Log only a fixed category and trusted company/run identifiers to the process logger. - Preserve recorder-owned byte draining, redaction, source coverage and genuine persistence-failure warnings. - Cover actual native executor, completion and lease-close boundaries, including stalled local appends and exact object-log source/hash preservation. - Place the retained-workspace test fixture under the existing published-path contract after merging newer recovery tests from master. Production validation and native/catalog assertions remain unchanged. ## Verification - Current head: `c500e4933a85a4c453f66c872fa335108c661b15`. Normal master merges preserve upstream behavior. The one textual conflict was in the executor imports. - The first GitHub run tested merge tree `a4cb59448a3b5cc2047ac41c691720e6acec6e8d`, which included newer recovery tests from master. Five cases failed with `remote_workspace_published_path_invalid`. The same five failures reproduced on an identical local tree. The fixture now puts retained files under `.paperclip-workspaces/<workspaceId>` and keeps the target, lease and authority bound to that path. Production validation and the native-success/catalog-denial assertions are unchanged. - On unchanged `c500e493`, the full recovery file passed **244/244** tests. Existing cold-restore and occupied-root controls passed **4/4**. Server typecheck, server production build, module boundaries and whitespace checks passed. An intermediate missing-import setup failure was corrected and is separate from the five-case behavioral red. - Before that fixture correction, local `f2fdaddb` passed **1,221 tests** across the five diagnostic, executor, output, completion and object-store files. Recursive typecheck and full repository build also passed. Those results are tied to that earlier revision; [fresh current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41842043248) must qualify the final merge. - Behavioral regressions cover the original closed-writer warning, warm A-to-B ownership, and an actual Runner attachment exception before `onSession`. The admission hook gives the attachment diagnostic to B and leaves A untouched. The attachment fixture uses synthetic event persistence and outer failure projection; it does not certify a real database terminal projection. - The tests preserve synchronous diagnostic admission and asynchronous metadata recording. A stalled append reaches the existing bounded finalization result with unknown size/hash. The normal object-log control preserves canonical bytes, SHA256 and complete source coverage after lease cleanup. - An initial local run had **37 PostgreSQL startup failures** because the Darwin package lacked aliases to its included versioned libraries. Only isolated `node_modules` aliases were restored. No source or lockfile changed for this setup issue. A passing intermediate run crossed a source merge and is excluded from immutable-head proof. - An earlier combined test run passed 1,216 assertions but exited with a completion `afterAll` timeout. Its cause remains unproved. Later isolated 60/60 passes used the original timeout and do not erase that failure. An earlier repository-wide `pnpm test:run` was interrupted after its source became obsolete; it is not full-suite or current-head proof. Greptile reviewed the exact `c500e493` head at **5/5** with no open findings. Fresh CI must pass before merge. The native review is queued behind its separate saved-file recovery; no current-head native assessment is claimed. The provider execution loop is a fixture. These tests do not start a model or prove hosted recovery. Missing accounting receipts remain pending, not zero cost. Checkout-release invocation is observed; this is not a separate task-arbiter journey. The original native review published a valid finding before a separate workspace-export failure. The finding is fixed; this PR does not claim to fix that export failure or every hosted persistence warning. ## Risks Idle process logs intentionally omit raw provider text. The existing bounded transport evidence retains its own diagnostics. Accepted fs appends that do not settle retain the existing unknown-size/hash warning and cannot be certified complete. No new timeout, storage policy, schema or provider execution is added. ## Model Used OpenAI `gpt-6.1-sol` through Codex assisted with diagnosis, implementation, tests and review. The exact context-window size is not exposed. Tool use and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local tests and recorded passing checks, pending checks and unresolved failures above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will require all Paperclip CI gates to be green before merge - [x] I will require Greptile 5/5 with no unresolved findings before merge - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0eaabe5c0b |
refactor(accounting): remove company serialization from paid settlement (#18917)
Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b891e69bd2 |
fix(native): correct unread wake comments before completion (#18937)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents report completion through a server-owned tool boundary. > - A truncated external-chat request requires a complete run-bound comment read. > - The terminal guard checks that receipt after the provider ends its turn. > - Early completion feedback omitted the same check and could accept an unread report. > - The provider then lost its chance to correct the report in the same turn. > - This pull request returns the existing read requirement through the completion tool before accepting the report. > - The final guard still protects the exact source and snapshot after provider settlement. ## Linked Issues or Issue Description Related: #16995 keeps run-scoped readers available to retained provider threads. #18913 preserves queued comment input. This change fixes a separate completion-feedback boundary. **What happened?** A native external-chat run could submit an accepted completion without reading all server-bound comments. The terminal guard then failed the run with `native_current_wake_comments_unread`. Early feedback also accepted partial, changed and foreign read receipts. The agent did not receive the required correction while it could still act in the same turn. **Expected behavior** The completion tool must reject the report with a `read_current_wake_comments` instruction and the cursor requirements. A complete current read must permit the corrected report in the same turn. The final snapshot check must remain effective. **Steps to reproduce** 1. Create a running native task with a truncated external-chat wake and exact accepted comment IDs. 2. Call completion feedback before reading the comments, or after only a partial or stale read. 3. Observe that the old code returns an accepted report instead of the read correction. ## What Changed - Check the existing run, company, task and agent binding before every completion-feedback acceptance return. - Reuse the authoritative current-wake reader and terminal receipt validator. - Share the two unchanged read-error codes from their owner across the validator, feedback and executor classification. - Convert only the two owned unread or changed-snapshot errors into a bounded tool correction. Other errors still propagate. - Keep the final validator, receipt parser, result persistence and admission policy unchanged. - Add real PostgreSQL regressions and an actual OpenCode MCP tool sequence with a controlled local provider process. - Document the early correction and final snapshot fence. ## Verification The source checkout uses Node 24.22.2 on macOS arm64 with a fresh owned embedded PostgreSQL database. Final published tip: `37bfb269432f86f98cab1a58c09b5519ed9477fd`. Its production code is commit `8c7297318c3a841fc1c72c846acfaabf58adabe9`; the last commit adapts three existing module mocks. The expanded affected group passes **1,284 cases** with **35 existing opt-in skips**. The skips require captured private-runner transition fixtures and a configured fresh database, which were not supplied. All nine real PostgreSQL/MCP cases execute and pass. Recursive typecheck and the full build pass for production commit `8c7297318c`; the final changed test layer also passes. - Untouched master `1a39dea9aa2bb9bea416439b0a574c788e88b9a0`: four causal failures. Unread, partial, changed and foreign complete receipts incorrectly receive accepted completion feedback. The ordinary inline-complete control passes. - Current source: all nine new cases pass. The actual authorized reader returns the full accepted comment. The same running task and run remain intact, with no replacement or accepted-result replay. - The OpenCode native backend sends a real MCP finish rejection to the controlled provider process. The process follows the returned reader instruction, reads every page, and resubmits finish. One provider turn produces one semantic result proposal. The terminal snapshot fence still rejects a later comment edit. - All 98 affected OpenCode driver and MCP bridge cases pass with the fixture-mode addition. - The first tool-sequence attempt failed on a fixture-only relative path before starting the provider. The test now uses its module URL. This setup failure is separate from the four causal failures. - The first recursive typecheck found missing local workspace links in the clone's excluded sandbox-provider directories. Existing dependency links were copied into the isolated clone. No lockfile or source change was made for this setup repair. - At `c57e532a87`, all 1255 cases in the six-file affected server family, recursive Node 24 typecheck and the full build pass. At production commit `1ae9f54107e1762bc8222572b365a013d4572d1a`, all 1,257 affected cases, recursive Node 24 typecheck and the full build pass. - Greptile found that the initial correction could restart a partial read after page two. On `c57e532a87`, following that correction fails with the actual reader cursor-order error. The follow-up continues the caller's most recent cursor and retains the changed-snapshot restart path. All nine new cases pass, including this regression, a stale-partial control, the foreign receipt refusal and the actual same-turn MCP sequence. - [Linux server3/12 job](https://github.com/paperclipai/paperclip/actions/runs/41788671370/job/126437378334) found five failures in the older format-feedback SQL fixture. Its flat heartbeat row did not match the real joined projection. The same five failures reproduced locally. The test-only follow-up returns a coherent run/agent/task projection and keeps the real resolver and formatter assertions. All 14 format and real-PG/MCP cases pass, and the generated server test layer typechecks. Production code and the earlier 1,257-case/typecheck/build proof are unchanged. - Qualification of the shared-code change found 49 failures because whole-module mocks omitted the new exports. The three mocks now pass through actual module exports and keep their controlled reader methods. Literal error assertions remain independent. The final 1,284-case group passes with the exact 35 opt-in skips described above. No production fallback or substitute code was added. - [One Linux Runner package attempt](https://github.com/paperclipai/paperclip/actions/runs/41792315273/job/126448500037) passed 4,913 cases and skipped 10, but its synthetic post-rename credential durability case failed with ENOENT. The complete Runner package tree and lockfile are identical to the [earlier Linux control](https://github.com/paperclipai/paperclip/actions/runs/41790199252/job/126441739681), where all 42 credential cases passed on Node 24.22.2. The exact case also passes once on both base and head on macOS with Node 24.22.2. The original failure cause remains unverified; OS/load and shared-test timing uncertainty remain. Credential code and assertions are unchanged. The single affected-job rerun passes, including all 42 credential tests. The original failure remains unexplained. - An ordinary full local test attempt was stopped before the review repair. It has no completion summary and is INCOMPLETE. Current-head full CI completed; no local full-suite pass is claimed. Greptile reviewed the final published tip at 5/5 with zero open threads. All current-tip checks pass: 54 successful check runs, one successful legacy commit status, 3 skipped and 2 neutral, with no pending or failing checks. Paperclip also reviewed final tip `37bfb26943` at 5/5 with no remaining findings. Its prior 4/5 code-ownership recommendation is fixed and resolved. No review nudge or hosted task retry was issued. The controlled OpenCode process uses the supported v1 factory envelope to exercise the common native feedback and MCP boundary. It does not call a live model or qualify a provider pack. PostgreSQL, authorization, pagination and receipt checks are real. No hosted task, model, permission or saved result was changed. Recovery of an already accepted historical run remains unverified. ## Risks - Each finish call adds an exact binding query. A mandatory read also validates the current accepted-comment snapshot through the existing transaction. Cost grows with those exact comment IDs, bodies and attachment metadata; it does not scan unrelated task history or add idle polling. - A valid read can become stale before the provider settles. The unchanged terminal guard still rejects that race. - Missing ownership or unexpected validation errors remain refusals. This change does not grant replay authority or turn an accepted historical result into fresh work. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window. The coordinator performed independent read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
669538534c |
fix(recovery): preserve failed remote reattach workspaces (#18936)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native runs can edit files in a remote execution workspace. > - A failed reattach can leave an observed coordinator and unpublished files. > - The failure handler recognizes the older workspace-sync descriptor but misses the modern workspace reference. > - The released task lock then prevents ordinary session recovery from claiming that old run. > - This pull request retains the files through the existing Stop, cleanup and workspace recovery owners. > - Exact ownership and Stop evidence permit file preservation while unknown provider effects remain under manual reconciliation. ## Linked Issues or Issue Description Related: #18877 protects native manual admission holds. #18879 handles scheduled native retry lock reclaim. #18918 fixes remote workspace row-lock order. Those changes do not record the missing modern failure-retention obligation. **What happened?** A native reattach failure can leave a failed run, a dirty modern remote workspace, an expired observed coordinator and a free task execution lock. The run has no accepted result. The failure handler skips file retention because it checks only the older descriptor. Recovery cannot export while the old coordinator remains owned. A mismatched workspace generation can also make orphan cleanup miss the existing file claim. **Expected behavior** Retain the exact original allocation and files. Stop the allocation through the existing lifecycle owner before retiring the stale coordinator. Keep checkpoints and unknown external effects available for manual reconciliation. Unsafe or changed source authority must prevent physical cleanup. **Steps to reproduce** 1. Prepare a modern remote workspace for a native run. 2. Record a provider checkpoint and an active interruption incident. 3. Fail reattach preparation and release the task execution lock. 4. Let the coordinator lease expire. 5. Run the existing orphan, environment cleanup and workspace recovery workers. 6. Verify the original files from a cold archive read and verify that no new native session starts. ## What Changed - Include modern prepared references in the existing heartbeat failure-retention producer. - Add a file-only historical intake for an exact failed observed run in the normal expired reattach state. - Require an untouched active ephemeral lease, a free blocked task, matching workspace writer and generation, no accepted result and complete single-owner allocation history. - Persist the existing StopAndRetain intent before provider I/O. Use nonblocking row locks and classify busy ownership only after rollback. - Use the existing native environment settlement owner after exact Stop and retention receipts arrive. Keep its single scoped discovery query. - Preserve source and coordinator fingerprints across the handoff. Keep original checkpoints, errors and manual unknown-effect incidents. - Keep stale or malformed modern workspace claims held when preservation intake refuses them. - Add actual PostgreSQL, heartbeat, allocator, worker restart, archive and safety controls. Document the narrow recovery contract. ## Verification Current published head `b1f49691187a398415cf9b16b723e3ab473f0f91`: all required CI checks pass (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with no open threads. The separate Paperclip assessment is queued and its current response says work is paused. That assessment is unavailable; no additional review nudge or hosted retry was issued. All local commands use Node 24.22.2 on macOS arm64. - Exact published head: `b1f49691187a398415cf9b16b723e3ab473f0f91`. Required CI passes (54 success, 3 skipped, 1 neutral); Greptile is 5/5 with zero open threads. The optional Paperclip assessment remains queued/paused without a completed score. - Final recursive `pnpm -r typecheck`: passed. - Final `pnpm build`: passed. - Final three native producer, containment and scheduling files: **67/67 passed**. These include the real coordinator and worker restart, the production heartbeat recovery callback, exact Stop receipts, cold archive restore, and zero native backend, new allocation or destructive release calls. - The plugin allocator cases use the actual selector with `reuseLease: true`. Active and released ephemeral leases receive a fresh allocation. - The assembled reattach producer checks that the task and coordinator claim commit before physical resume. A historical pin refuses that current owner. A later claim behind a committed pin remains held. - The safety controls cover reusable allocations, manual and unknown states, cleanup in flight, accepted but unbound results, late allocation owners, wrong or malformed workspace authority, changed generation/checkpoint, lost Stop acknowledgement and a real cleanup-producer lock conflict. - Untouched base `e7549ab6131555f655736e6b544892a3b1e323cb`: four expected behavioral failures and one passing legacy-descriptor control. The failures cover the modern producer, original provider history, historical retention and wrong-epoch cleanup. Fixed production files were restored byte-for-byte after this comparison. - The first affected six-file run completed **562 passed / 1 failed**. The failure was an existing HTTP case with `socket hang up`. That exact case passes separately on both untouched base and this branch. The original connection failure remains unexplained. At `0e92de8c`, the complete workspace family passes **188/188** with unchanged assertions. The environment cleanup owner family also passes all 23 cases. The later guard changes are qualified by the complete 67-case native owner group. A full repository test run was not repeated locally; the requested full-ci workflow runs the complete repository suites. - Published `0e92de8c` fails both new explicit-null Stop/export marker controls because it creates a new retention pin. The follow-up uses the shared presence-aware Stop guard. Both controls now preserve the marker and record no new retention pin, Stop or destructive release. - Paperclip's review found that malformed workspace authority could throw and abort an orphan page. Published `5f6cd4f5` fails all three new controls: intake, the actual page and after-Stop settlement. The same strict schema now uses a safe parse, and the existing OBSERVED hold runs before the prepared-source lookup. The malformed source stays held while the next valid source receives its retention pin. - Two existing scheduling tests caught the draft's second global discovery query. The final single-query repair passes all 36 unchanged scheduling tests. - An earlier restarted archive assertion stopped after 20 seconds. Its recorded Stop was valid, and the coordinator had settled. The existing evidence recheck deadline was still 13 seconds ahead. The final test observes the unchanged 24–36 second policy within its existing 60 second case limit. The external fake provider Stop and file transport are controlled test boundaries. Database ownership, lifecycle workers, archive bytes, encrypted local storage and cold restore are real. This does not claim a live cloud sandbox, S3 or model run. No hosted task was changed for this PR. ## Risks - Historical intake is deliberately narrow. Reusable, already released, mismatched or competing allocations remain held. - File preservation does not settle unknown provider effects or authorize a repeated provider turn. The task can still require manual reconciliation. - A busy or changed owner delays preservation. The original files and claims remain retained. - This does not repair the underlying sandbox readiness failure or establish the cause of a separate archive-capture failure. - Deployment and recovery of original hosted tasks remain unverified. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and GitHub tools. The runtime does not expose the exact serving model ID or context window size. A separate Codex agent and the coordinator performed read-only source review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ecc253b73d |
fix(workspaces): retain redacted archive capture causes (#18935)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Remote workspaces need a verified saved archive. > - A failed capture must retain a safe cause for diagnosis. > - The codec and capture wrapper erased known validation and filesystem failures. > - The one-shot output fix in #18888 preserves only the existing disk and Git messages. > - This PR retains a closed set of capture failures without exposing workspace data. ## Linked Issues or Issue Description Refs #18888. **What happened?** Remote archive capture can still report only `remote_workspace_operation_failed`. This was observed after the public revision `f211d44c00fbc31d22e7ff687e22dfa05643da2a` included #18888. That observation identifies the capture stage. It does not identify the underlying failure. The real local codec gives a specific validation error for an escaping symlink. Node also gives a specific errno when an archive is missing. The capture wrapper erased both causes. The codec sent raw filesystem paths to its parent process before that wrapper discarded them. **Expected behavior** Retain exact, fixed capture validation and filesystem classes. Keep arbitrary messages, paths, filenames, file contents, and command output private. **Steps to reproduce** Call the real capture transport with an escaping symlink. Separately, create a canonical capture receipt whose archive is missing. Both fail with the generic transport message on unchanged master. The new tests use real Node processes and files. ## What Changed - Add a self-contained classifier for fixed capture validation messages and a finite Node errno set. - Redact unknown codec capture errors. Preserve recognized classes through helper installation, capture, and split or combined command output. - Keep the existing disk and unfinished-Git messages first. Keep other command behavior and all save, retry, ownership, Stop, and permission policy unchanged. - Document the diagnostic contract. ## Verification Node 24.22.2, pnpm 9.15.4, macOS arm64, isolated source checkout. - Six new real-process cases failed before production edits. The same six passed after the repair. - The complete archive and transport family passed **93/93**, with no skips. A new contract test covers every capture-side reason in the actual serialized codec; an unlisted-reason mutation fails that test. - These tests cover real symlink rejection, missing canonical archive, serialized-helper ENOENT/EEXIST redaction, unchanged source bytes and canonical receipt, unknown errno, malformed output, other command stages, and the successful capture/restore path. - A safety test caught a regression in the first patch: early redaction hid a Git child's existing disk-full signal. Unchanged master passed both split/combined cases; the first patch failed both. The final family passes both after the same closed disk recognition was restored. The earlier failure remains recorded. - Full recursive typecheck and `pnpm build` passed on the current head with Node 24.22.2. - A fresh probe of the current head's compiled `server/dist` artifact passed. It ran nine real Node commands with combined one-shot output. It checked both fixed failure classes, unchanged source/receipt bytes, and a successful capture, hash check, cold restore, and scoped cleanup. - The first head's full local `pnpm test:run` failed one unchanged GitHub fast-response streaming-quarantine acknowledgement assertion: an acknowledgement appeared where the test expected none. The first chat shard passed 319 cases; the second passed 323 and failed one. One bounded focused comparison passed on both that head and untouched master. It did not explain the original failure. No chat source, assertion, or timeout changed. No full local-suite pass is claimed. - All **57 current-head contexts are terminal and clean**: 53 success, two skipped, two neutral. Required [CI](https://github.com/paperclipai/paperclip/actions/runs/41781977986) gates passed. Fresh Greptile and Paperclip assessments both report **5/5** on `d7b2fcc22c75e3157e0c53bf4725e665810f2d19`, with no open review threads. The optional security scan is neutral, not an independent security assessment pass. ```sh pnpm exec vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts ``` This is a **diagnostic repair**. It does not establish the cause of the observed hosted failures, qualify a live provider capture, or recover the original files. No hosted resources were executed, started, changed, or retried for this PR. ## Risks The classifier can expose only a fixed message or a fixed class from its finite errno set. Unknown and altered messages keep the generic failure. Existing disk/Git priority and command-stage restrictions have explicit tests. The new classes may improve later diagnosis; they do not authorize another operation. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this agent session. The agent used reasoning, terminal tools, code edits, and real Node/filesystem tests. A separate Codex agent reviewed the production diff. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6352578c5 |
fix(task-retry): retain chat Retry during identity contention (#18896)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry must preserve a person's request and enforce the current chat authority. > - Chat retry preparation takes nonblocking source locks before it can authorize the saved source. > - Sustained lock contention could discard the Retry request before it was saved. > - This pull request saves one unadmitted request in the existing chat retry queue. > - The worker checks the source and current access before normal admission. The task shows a scheduled retry while it waits. ## Linked Issues or Issue Description **What happened?** Task Retry returned HTTP 500 when another transaction held the saved chat principal's identity lock or the endpoint authorization row. The source preparation error rolled back the canonical Retry intent. The same gap existed when Retry had to restore a certified input batch that had never reached a provider. With the bounded synchronous retries in #18895, sustained contention returns 409 without a saved intent. **Expected behavior** HTTP 202 retains one Retry request. Acceptance does not grant source authority or mean that a provider started. The existing worker prepares the request if the source locks become available before its saved deadline. It must reject changed input, revoked access, Stop, or superseding work before input adoption, publication, or provider work. **Steps to reproduce** 1. Prepare a failed task with its original admitted GitHub chat source. 2. Hold that principal's `chat-identity` advisory lock or the endpoint row on a second PostgreSQL connection. 3. Send the normal task Retry HTTP request. Check the response and persisted intent. 4. Release the lock. Process the existing chat retry queue and check that one authorized successor is queued. **Paperclip version or commit** Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. **Deployment mode** Source checkout, real Express Retry route and embedded PostgreSQL. External chat transport is a test fixture. The live hosted fix has not been deployed or verified. Related: #18895 handles brief admission contention with bounded synchronous attempts and returns 409 if it remains busy. This change retains a durable request through sustained identity contention. It uses the existing background retry owner and adds real HTTP, restored-input, Stop, publication, restart, and task-presentation coverage. ## What Changed - Preserve a strict pending-source action only when the identity try-lock reports contention or the endpoint authorization query reports lock-unavailable through an explicit Retry callback. Bind it to the canonical task Retry, company, task, agent, endpoint, conversation, and principal. Other database and authorization errors keep their existing behavior. - Revalidate the canonical intent, current board permission, saved source, and latest execution before the existing worker can prepare that action. Ordinary authorization rejects an unprepared action. - Keep certified input restoration inside source preparation. Recheck the saved deadline before the prepared payload write. Expiry, a nested savepoint, or outer rollback cannot commit partial restoration. - Give source authorization contention one saved 60-second deadline and a separate counter. Keep its waiting phase until current source authorization succeeds or the action terminates. It does not spend the eight delivery attempts. Duplicate clicks, worker restart, and background sweeps do not renew either limit. Reuse the existing work notifications and outbox deadlines. - Persist completion of the authorization wait under the current claim before delivery. A restart revalidates current authority. Generic delivery failure preserves the stored phase even when a commit acknowledgment is lost; it does not grant a wake or renew the saved deadline, counters, or claim. - Project pending preparation and prepared but unsettled actions as the existing scheduled-retry state with the actual outbox deadline. Keep the exact task, intent, agent, and source binding. Active execution keeps its normal working state. No UI or API enum change is needed. - Add regressions and document the admission boundary. ## Verification - RED: the original real HTTP route returned 500 under a held PostgreSQL identity lock. The restored-input case also returned 500 while its 60 sibling source and budget cases passed. The current endpoint-row case returned 409 without a saved intent under the bounded synchronous fallback. - RED: the worker failed a valid Retry during an actual nine-second PostgreSQL lock. Removing only the latest-run guard makes the new queued-run negative fail. Source preparation could also commit restored input after its deadline, and an already prepared action could process after a late lock release. The prepared but unsettled task view lost its scheduled state. - GREEN: all 91 selected real PostgreSQL integration cases passed. Coverage includes the 62-case saved-input matrix; coalescing; restart; one real heartbeat successor; source, actor, Stop, newer-work and publication refusals; a real nine-second lock; fixed-deadline exhaustion; the separate eight delivery attempts; certified restoration rollback on expiry; prepared-action early and late release; a real conversation-row wait that crosses the deadline during current authorization; endpoint-row contention; unknown source; an already-revoked actor; unchanged unrelated ingress errors; durable-work idle settlement; actual outbox presentation deadlines; and recovery after authorization commits, with Stop and a lost commit acknowledgment. - GREEN: all 75 cases in `task-retry.test.ts`, `task-interruption-presentation.test.ts`, `chat-failed-run-retry-boundary.test.ts`, and `chat-action-work.test.ts` passed without a name filter. - RED → GREEN: stopping after source authorization committed left a stale waiting flag, so a fresh worker failed a previously authorized Retry after the deadline. The fix saves that phase in the same transaction. A real PostgreSQL commit followed by an injected lost acknowledgment also reproduced generic settlement restoring the stale flag. The corrected settlement preserves the stored phase. No wake is sent on the ambiguous response. Recovery queues once after current checks; Stop still prevents a successor. The deadline, contention count, original input, failed run, and publications stay unchanged. - Module boundary checks pass. Full workspace typecheck and the full build passed at `d8a4a1a739aa6825714b04691d4166a00991808b`. At this exact head, both required checks (`ci / verify` and `ci / e2e`) pass. The inventory has 53 successful, three skipped, and one neutral check, plus the automatic Paperclip Review in progress. The legacy Snyk status passes. Greptile is 5/5 on this head, with no new comments or unresolved threads. A current Paperclip assessment has not been published yet. - The full local `pnpm test:run` did not pass: its second chat shard hit a 15-second timeout in the HTTP/restart test after 348 and 293 passing cases. One bounded diagnostic of the same 642-case shard pair passed, with that case completing in 606 ms. The matching Linux CI pair passed too. The original timeout remains unexplained and nonreproduced. No timeout changed. The earlier stopped run was incomplete, and both conflicting documentation additions are retained. - CI chat 3/3 exposed a separate fixture assumption: restoration can commit before a durable wake receipt exists. One known pre-receipt delivery failure reproduces the same missing-run error. The corrected fixture drives the existing worker until the exact canonical intent and wake agree. It proves one real wake, one successor, exact source/actor binding, and one restoration. The exact failed shard pair now passes locally: 635 cases. Workspace typecheck also passes at test-only head `94d691c9df61b05019b96900d5800a8f1c484e9a`. Production source is unchanged from the previous verified build. That head completed all registered checks: 53 successful, two neutral, three skipped, plus the successful legacy status. Greptile scored 5/5. The subsequent Paperclip 4/5 persistence finding is addressed by the source repair above; its new exact-head review is pending. - Compatibility: 111 selected cases passed with #18895 head `9b5d877161ffcbf16c0fb6920e8155a3620c85b0` in a disposable worktree. The branch is now synchronized with master `4a83ba4050`, which includes that change. - Preserve the old CI failure: at head `37954951fb66f3b50d0f2a3bec78d6993c479e87`, server shard 6/12 initially failed the empty organization-audience replacement versus member-removal assertion in `tool-access-service.test.ts`. The exact race family passed on unchanged base and head; all 501 file tests and two 30-case controls passed on base. The one Linux replay passed 119 files and 2,833 cases, with three skipped. No guard or assertion changed. The initial failure remains unexplained and nonreproduced; this PR does not claim to fix it. - The prior external Superagent scan could not start because its Security Graph returned HTTP 503. Its neutral check was not a completed security assessment. - These tests exercise the real Express route, PostgreSQL transactions, and heartbeat queue. External chat transport is a fixture. They do not run a live model or prove recovery of an original hosted task. ## Risks - A still-pending source authorization wait fails after its fixed 60-second contention deadline. A completed wait is saved, but new contention keeps the original deadline. A healthy lock can exceed that limit. Other delivery errors retain the eight-attempt budget. Current authority checks remain required before admission. Inspection does not reset either limit. - The new pending phase must never be treated as an authorized chat source. Normal authorization rejects it until the worker prepares and binds the exact source. - The task projection reads the existing linked outbox deadline. It adds a scoped join and does not create a timer or change a business deadline. - No schema migration, provider identity change, or new delivery queue is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, tool use, and test execution. The host did not expose an exact deployment snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] Required Paperclip CI gates are green (`ci / verify` and `ci / e2e`; automatic review progress is recorded above) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c5251b4c0e |
fix(chat): retry fully recorded setup failures before dispatch (#18882)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected tasks retain their exact external request across failures. > - Task Retry must preserve that request and prove that old execution cannot continue. > - A workspace setup failure can settle before any provider dispatch. > - The chat retry guard accepted one specific validation failure but rejected other fully recorded setup failures. > - This pull request reuses the same strict server settlement proof for those failures. > - The benefit is one safe continuation through the existing queue, with all source and execution gates intact. ## Linked Issues or Issue Description Refs #18803, #18845, #18852, #18853 and #18860. These changes preserve workspace writers, bind completed preservation, or repair other preparation and native continuation paths. This change repairs the separate connected-chat guard for fully recorded legacy setup failures. Search also found #18876, which repairs issue scope during deferred wake promotion and does not repair this guard. **What happened?** A connected task failed during workspace reservation before provider dispatch. Its source remained authorized. Task Retry saved an intent and chat action, but the chat worker rejected that request as `failed_chat_retry_not_authorized`. No successor was created. The old failure had a bootstrap marker, a dispatch-only Runner profile, the matching server error and durable measured settlement events, and no runtime, process, output, result or environment lease. **Expected behavior** A deliberate board Retry can continue the exact saved input after the complete never-dispatched proof passes. An unfinished earlier writer, changed source, accepted result or unknown execution state must still prevent execution. **Steps to reproduce** 1. Admit an authorized request through a restricted chat endpoint. 2. Settle setup before provider dispatch, for example while an earlier workspace writer is unfinished. 3. Keep the exact server error and durable measurement events, a dispatch-only Runner profile, and no execution resources. 4. Press task Retry after the independent writer and ordinary admission gates permit work. 5. The original guard saves the Retry action but refuses to create a successor. **Paperclip version or commit** Reproduced on master at `a00109aa35`. The branch is rebased on current master. **Deployment mode** Hosted task diagnosis. Local proof uses real Express HTTP routes, chat authorization, heartbeat queue admission and disposable embedded PostgreSQL on macOS arm64, Node 26.4.0. Provider work is deliberately prevented by occupied fixture capacity. ## What Changed - Extend only canonical task Retry's existing strict pre-provider preparation proof to generic server-settled setup failures. - Keep exact result shape, dispatch profile, controller, runtime, process, output, event and resource checks. No diagnostic message allowlist is added. - Exercise GitHub and Telegram HTTP Retry with the original sender and saved input. Keep the failed run unchanged and create at most one queued successor. - Cover provider events, retained native state/results, modified source, unknown result metadata, live controllers, process identity, source leases and an independent unfinished writer. - Document the guarded contract and correct the obsolete access-map sentence. No new endpoint or agent permission is added. ## Verification - Red before repair: both setup causes failed through the real HTTP/chat/queue path because the durable Retry had no successor. An initial retained-native negative fixture had an invalid foreign-key setup; that fixture was corrected before the final run. - Green: 37 focused integration cases pass, including 22 new GitHub/Telegram cases and 15 existing pre-provider source and execution guards. - Current head `1bf69d20e02a049248ec378f087505563daeb9a0`: all 37 focused integration cases and full repository `pnpm -r typecheck` pass after the master rebase. The full `pnpm build` also passes. - Map structure, module boundaries and `git diff --check` pass. - Full local `pnpm test:run` is running; all six chat shards passed and the remaining server suites are progressing. This is not a full-suite pass claim. - Greptile and Paperclip Review both reviewed this exact head at 5/5 with no actionable findings or review threads. - Initial CI passed all jobs except the unchanged Runner package suite. Its `runtime-host.test.ts` approve-reads case failed before its intended assertions because a managed credential home could not acquire its loopback-port quorum. The exact case passes locally. A wider 121-case check hit the same lease error in a different Pi case (120 passed); all 121 pass with one worker. The exact competing port owner is unknown. One unchanged retry of only failed CI jobs passed, including the Runner package suite and aggregate `ci / verify`. All current-head required CI gates are green; the original failure remains in workflow history. - The dependency setup initially missed the embedded PostgreSQL package's dylib symlinks. Running its own postinstall helper fixed the local setup before assertions. This was not a product test failure. - This change has not been deployed. The original hosted task has not been replayed with the fix. ## Risks - Never-dispatched proof must remain strict. A bootstrap marker or empty process fields alone cannot authorize continuation. The regression cases retain contrary-state holds. - The inherited session display hint remains history and supplies no resume authority. - Current connection, principal, source integrity, pause, budget, approval, dependency and workspace-owner gates remain in their existing owners. - No schema, provider profile, trust policy, model or credential permission changes are required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL/browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d8c1b9d4b5 |
fix(runtime): reclaim failed remote checkpoint retries (#18877)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must preserve the original execution and provider state. > - A failed remote run can retain its provider PID after heartbeat releases the task execution lock. > - Normal resumption excludes that PID, while restart recovery rejects the released lock. > - Recovery must reclaim that exact execution only after current admission and durable ownership checks pass. > - Failed retries and admission holds need separate finite budgets. Neither can prove a provider stopped. ## Linked Issues or Issue Description **What happened?** A failed remote native run remains in retryable failure without a task execution owner. Normal resumption and restart recovery both reject it before inspecting the preserved remote provider. The current remote workspace reference format also misses the due-retry routing path. **Expected behavior** A due exact-checkpoint retry reclaims its original task execution only when current ownership and safety gates permit it. Newer task input prevents an old claim. Lost or ambiguous provider state still prevents replay or replacement. Ineligible recovery eventually becomes a quiet manual reconciliation obligation. **Steps to reproduce** 1. Start a remote native run and persist its provider checkpoint and encrypted Runner authority. 2. Fail with exact-checkpoint recovery and a retained remote PID. 3. Release the task execution lock at heartbeat completion. 4. Run ordinary resumption and restart recovery after the retry deadline. 5. Observe that both reject the same eligible run before remote inspection. **Paperclip version or commit** Reproduced from source at `d59791828482`; integrated with upstream `db9be4c5a6`. **Deployment mode** Remote workspaces with PostgreSQL. Local regressions use disposable PostgreSQL and encrypted checkpoint persistence. Related public repairs #18846, #18854, #18860 and #18841 cover finished per-turn owners, pre-provider failures, unstarted follow-ups and storage migration. They do not reclaim this failed run's released execution lock. Merged #18879 also adds scheduled-resume lock retention and reclaim. This PR keeps that compatibility and requires encrypted Runner authority, canonical source ordering and a charged failed retry before reclaim. Its scheduled-resume regression now uses a real encrypted authority and retains a snapshot-only refusal control. ## What Changed - Share normal admission checks and one guarded execution-binding restore/event helper between normal and remote resumption. - Keep agent-name normalization in a module with no imports. Preserve heartbeat's existing export. Native admission no longer loads heartbeat orchestration for a string key. - Recheck company, assignment, agent eligibility, Stop, pause, approval, dependencies, execution and checkout ownership before reclaim. - Reject newer task input using PostgreSQL creation-time and ID ordering, preserving timestamp precision. - Require the recorded remote target, encrypted Runner authority, exact checkpoint identity and existing controller fence. - Commit task binding, controller lease, run status and recovery event atomically. Recognize both remote workspace reference formats. - Charge failed reattachments against the existing three-provider-attempt budget, including retained-lock crash windows. Active controller adoption preserves its charged attempt. - Use the existing five-retry/30-minute system-recovery policy for admission holds. Ended tasks and newer sources exhaust it immediately. Preserve PID, checkpoint and task bindings, suppress resource settlement, and assign manual reconciliation with executor stop/state explicitly unverified. - Register recovery intent before deadline writes; preserve live/future deadlines quietly and roll back zero-row writes. Record changed hold reasons once and terminal dispositions separately. - Keep historical provider-budget exhaustion behind current admission/source/controller/authority checks. Make an existing orphan-cleanup fixture explicitly due. - Include exact plugin task leases that have no workspace reference. Reuse every reclaim and retry-budget guard, with a bounded company/task/run lease projection. - Share strict manual ownership proof across stale-lock cleanup, adoption, queue release and wake admission. Require termination receipts for every exact remote lease before normal cleanup can release either binding. - Preserve governing Stop through the existing audited cancellation path. Its acknowledgement does not release an unverified provider. Ordinary Retry can wait for exact Stop and file preservation, then queue one continuation. ## Verification - Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its [current CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858) is terminal and clean: 54 successful, three skipped and two neutral contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on this exact head with no open findings. No fresh Paperclip 5/5 assessment is claimed. - The [prior complete CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388) passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and Paperclip both 5/5 and no open threads. - All 233 tests across the six affected heartbeat, restart-isolation, remote-admission, scheduled-resume and session-binding families pass on production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and plugin task leases, competing controllers, exact rollback, Stop, assignment/company boundaries, attempts, work-intent ordering, finite backoff and quiet held state. - The merge preserves #18931's per-run audit-failure isolation and this PR's unchanged-transition hold. Two initial composition cases failed because their setup or expected result described the older contract. The rollback test now expects the exact per-run deferral while retaining full coordinator equality and no work intent. The routine fixture writes a real encrypted checkpoint through the normal owner before reaching the actual routine unique constraint. No production admission guard or rollback assertion was weakened. A separate agent reviewed this exact merge and fixture delta without finding a blocker. - Full recursive `pnpm -r typecheck` and `pnpm build` pass on production merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits change only three test files. Both the two-file `a5104fa` and final `5d7baf` server test-layer typechecks pass. - Four production-identical `a5104fa` probes pass with the rebuilt actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL checkpoints under Node 24.22.2. Workspace-reference and plugin-lease admission each cover surviving and lost providers. Replacement preserves one provider launch, one original side effect, the charged attempt and restored task execution binding. Lost providers reject snapshot recovery without replacement. This is protocol-boundary evidence, not a live Claude or hosted provisioning claim. - Initial `a5104fa` CI found one additional routine integration fixture missing the persisted `exact_checkpoint_resume` mode. Its encrypted checkpoint was already present. The exact case fails locally before repair and passes after recording that mode and expecting the charged provider attempt. Every exact routine conflict, rollback, checkpoint, task binding, successor-count and sibling-startup assertion remains. Final head changes only that fixture. The correct native-server group passes all nine files: 175 executed cases and 14 container-only skips, matching the now-successful native-server CI gate. - The earlier canonical HTTP Retry acceptance passed on the prior reviewed head with real PostgreSQL, the recovery worker and real files. Pending Stop protects both bindings and starts no successor. Exact termination and copyback queue one preserved-workspace continuation. No browser or live-model outcome is inferred from that test. - The predecessor's full local `pnpm test:run` failed in an unchanged Discord Gateway fixture with a 200 ms lease and a 422 initial ownership error. Focused predecessor and untouched-base controls pass; that full-run cause remains unproved. The required command was also started on production-identical `a5104fa`. All six local chat shards passed, covering 1,850 cases. The duplicate local invocation was interrupted with SIGINT while the general-server group was still running after this head's complete CI and the combined repair qualification had passed. The wrapper exited 1 on cancellation, no process retained its test log, and no full local pass is claimed. An earlier attempt to select the native lane through the CI wrapper instead selected local package sharding: 2,631 cases passed and 13 failed. Those separate failures are retained without causal attribution or unrelated source changes; that run is not counted as native-server qualification. The direct native group and current complete CI matrix pass. - Live Claude Opus 5.5 continuation, hosted plugin provisioning, deployment and recovery of historical tasks remain unverified. No original task was retried or mutated for this verification. ## Risks - Restoring a task binding grants remote inspection; it does not prove that the provider survived. - Unknown effects, lost providers and missing semantic results retain their existing reconciliation requirements. - Exhausted admission holds preserve execution bindings and resource evidence for manual reconciliation. They do not authorize a fresh executor or provider/resource release. - The source guard is conservative: newer task input prevents recovery of an older failed source. - No database migration or provider profile change is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and shell execution. The runtime does not expose the exact model snapshot or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c39e076638 |
fix(runner): retain the cause of closed ACPX sessions (#18872)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Runner records provider failures so the task can show their cause. > - An ACPX control request can fail and close its session before the event pump runs. > - The next poll then reports only that the session is closed. > - A second cleanup error can also drop a valid startup cause. > - This pull request retains each failure at its existing reporting boundary. > - Operators can inspect the cause while the failed session stays closed. ## Linked Issues or Issue Description Related: Refs #18622, #18847, #18854. Those changes preserve provider causes and handle proven startup failures. This change covers a control failure after provider execution starts. **What happened?** A rejected sidecar control closes the ACPX session. A later poll replaces its error with `ACPX provider session is closed`. The native terminal loses the control classification. Separately, cleanup can discard an existing validated startup cause. **Expected behavior** Retain the first control classification in the failed terminal. Do not invent a provider explanation for that code. Keep a validated startup cause when cleanup also fails. Keep the session closed, uncertain tool outcomes pending, and the existing recovery checks effective. **Steps to reproduce** 1. Start an ACPX turn and receive an authorized tool call. 2. Return its result after the provider has retired that call. The sidecar rejects `tool.resolve`. 3. Poll the native backend. Before this change, the terminal reports only the closed-session guard. 4. For the separate startup case, combine a rejected `session.open` cause with a cleanup error. Before this change, the combined error drops its validated provider cause. **Paperclip version or commit** The two subprocess regressions and the cleanup-composition regression fail against unchanged production code at `683fa4725e`. **Deployment mode** Native Runner with ACPX. The process regressions use the real Rust subprocess transport and durable native backend. Their controlled stale-tool rejection matches the production sidecar response shape. The separate cleanup unit injects a cleanup error at the existing error-composition boundary. ## What Changed - Retain the first failure in the ACPX session and return it from later closed-state checks. - Keep the validated provider cause when cleanup also fails. - Match the stale-tool fixture to production. It carries the closed code and no invented provider cause. - Cover a rejected tool result, normal shutdown, raw sidecar-text exclusion, pending tool preservation, refusal of another turn, and terminal replay after controller restart. - Document the existing failure and recovery boundary. ## Verification - Red: both process regressions fail on the base code. Polling replaces the named control failure with the generic closed-session error. The cleanup unit also fails because the combined error loses its existing startup cause. - Green: all 12 ACPX session tests, all 36 native backend tests, and all four session unit tests pass. The stale-tool cases require the classification and absence of an invented provider cause. - Commands: `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml --locked -p paperclip-runner-core --test acpx_provider_session` and the same command with `--test native_provider_backend`. - Map structure and whitespace checks pass. - At head `981faa4`, the full Runner core Rust suite passes: 847 tests passed and three existing tests are ignored. Workspace typecheck and build pass. The existing TypeScript provider-failure and durable-settlement boundary suites pass all 30 tests. - An earlier concurrent local Rust attempt missed an existing Codex diagnostic deadline. That case passes alone on baseline and head, and the final full suite passes with the local socket support its tests require. - Complete current-head CI passes: 55 successful checks, five intentional skips and two neutral checks; no failed or pending checks. Both `ci / verify` and `ci / e2e` pass. - Failed CI attempts were investigated before rerunning affected jobs. Logs showed runner shutdowns/lost communication, an unchanged Windows WebSocket authentication deadline, and an unchanged Cursor Cloud Stop-fixture assertion. The Cursor Cloud case passes on baseline and head. No test assertion or CI guard was disabled. - The local full `pnpm test:run` invocation passed all six chat partitions, covering 1,838 expanded cases. It was stopped as a duplicate while still in general-server after complete sharded CI passed. A completed local full Vitest run is not claimed. - Greptile and Paperclip reviewed the corrected head at 5/5 with no actionable findings. There are no unresolved review threads. - The process test exercises control rejection, process cleanup, durable terminal persistence, and controller restart. The provider boundary is a deterministic sidecar fixture. It is not a live Claude or hosted recovery test. - A genuine provider refusal or uncertain external effect remains unresolved. This change preserves its evidence and does not make it safe to replay. ## Risks - The first control classification becomes visible where a generic closed-session error appeared before. Raw sidecar error text stays excluded. Startup details use the existing validated and redacted `ProviderFailure` contract. - A stored failure must not grant execution authority. The failed session still rejects calls and new turns. Normal shutdown has no fabricated failure cause. - No provider profile, digest, retry budget, process ownership rule, migration, credential, or permission change is required. ## Model Used OpenAI Codex, based on GPT-6, with source inspection, code editing, command execution and regression tests. The exact runtime snapshot and context-window capacity are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and available capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the bug template fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the local Rust, typecheck, build and boundary checks; complete sharded CI passes and the partial local Vitest result is reported above - [x] I have added behavioral regression tests - [x] I have updated the relevant execution and run-log documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2518ac85ab |
fix(recovery): continue verified failed native preparation (#18908)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task Retry keeps a durable request while the system stops old work and saves its files. > - Native preparation can fail after it allocates a workspace, but before it starts a provider. > - Completed preservation can resolve the action while the original Retry still waits. > - The old classifier rejected every allocated failed preparation. The source check also returned before its canonical Retry fallback. > - This pull request accepts only complete server proof and moves the existing exact-source fallback before that return. > - The same Retry can then start one normal continuation. Current access and unknown execution still hold work. > - Saving files and admitting that continuation must use the same parent-row lock order; the composed head includes the supporting preservation fix. ## Linked Issues or Issue Description **What happened?** Task Retry accepted a failed native preparation and saved its request. The workspace had complete preservation and stop receipts. The request still waited for execution recovery and created no continuation. **Expected behavior** The same deliberate Retry must continue after the system verifies all saved files and source leases. It must preserve current task, actor, repository, Stop and execution checks. **Steps to reproduce** 1. Record a native-adapter claim that fails during unresolved legacy preparation, before provider execution. 2. Finish real workspace preservation and exact lease cleanup. 3. Submit normal HTTP task Retry and reconcile its canonical request. 4. On the base commit, no run is linked to that request. With this change, one continuation is linked to it. **Paperclip version or commit** Reproduced on master `d7283e875772ca8c684dc86e471e6b48d543b7b1`. **Deployment mode** Source checkout with real local PostgreSQL and filesystem archives. Object storage, sandbox transport and the native provider backend are controlled in the final execution test. This is not a live model or hosted object-storage claim. **Prerequisite: #18918** at `4c1d7769e6cfa9536a67c4e3b536cef155f19966`. This head includes that parent through normal history-preserving merges while retaining `master` as its target. The parent owns remote workspace save, prepare and provision lock order, legacy import reservation and publication, their concurrency regressions, and the database lock-order documentation. Merge the parent before this PR; no deployment is implied. Related: Refs #18892 for interrupted preparation. Refs #18882 for the separate connected-chat setup path. Refs #18898 for already-admitted cleanup waits. This change does not depend on #18882 or #18898. ## What Changed - Accept allocated failed preparation only with the server error and durable settlement records, no execution/session/checkpoint/output/result/coordinator evidence, all exact remote source leases stopped, and verified saved workspace proof. - Recognize only the host's closed accounting scope and typed pre-provider unsupported-harness warning. Unknown accounting, provider metrics and arbitrary diagnostics remain holds. - Check the canonical failed-preparation source before an empty resolved-action list can return. Reuse the normal locked continuation checks. - Add HTTP, preservation-worker and native execution coverage, including source failures produced by real heartbeat workspace preparation. Keep one canonical request and one provider turn. Preserve the original failure and real archive files. - Restore storage, environment and experimental settings even if native test setup or owned async cleanup fails. - Cover changed Stop, source, actor, controller, provider evidence, snapshot generation, writer and lease receipts. Keep current unavailable-catalog denial before provider startup. - Share one bounded host-settlement receipt reader between failed-native preparation and existing-directory readiness. Keep each caller's accounting, profile and additional warning restrictions. - Include parent #18918's company/task/run-before-workspace ordering for save, prepare and provision. Keep legacy import publication out of reverse native-owner waits with exact NOWAIT refusal. Retain foreign keys, ownership checks, uploaded work and transaction-aware notifications. - Document the failed-preparation continuation and parent lock-order contracts. ## Verification - Causal base comparison: the normal HTTP/preservation regression fails on the canonical request's missing successor. Both existing interrupted-preparation controls pass; the same cases pass with the classifier/fallback repair. - Actual heartbeat-produced catalog failures reproduce RED before the host-accounting fix, both with and without the real typed unsupported-harness warning. No source failure records are patched in those cases. - The native execution case keeps `paperclip_runner`, native runtime selection and the requested Codex model. A controlled native session reads real restored `saved.txt`, writes `continued.txt`, emits an accepted result and finishes through the real finalizer once. Only the external native backend is controlled. - Current unavailable-catalog admission and suspended authenticated membership remain denials before provider work. Extra provider/session/accounting material, Stop, newer source, writer/generation and lease-receipt mismatches remain held. - Both settlement consumers pass 53 focused real PostgreSQL/filesystem/native-boundary cases on Node24.22.2. Existing-directory warning presentation/provenance and durable-measurement restrictions are preserved. - Parent #18918 proves real PostgreSQL lock cycles through actual save, prepare, provision, reservation and import publication owners. Its native status peer test proves a busy refusal without a deadlock. Immutable task binding, writer/epoch fences, same-journal retry and unexpected-error rollback controls retain uploaded objects and original files. The original spontaneous peer identity is not claimed. - Historical head `6204f565` passed all 54 CI checks, but its local affected run passed only 572/573 with a real PostgreSQL preservation deadlock and its Paperclip review was 3/5. Those results remain historical evidence; this composed head includes the explicit parent repair. - Production-composed head `93a02d4cf74f4d8d2cc6c1b62884bc58280500bf`, Node24.22.2 / pnpm9.15.4: full `pnpm build` and recursive `pnpm -r typecheck` passed. The six affected recovery, continuation, Retry, migration and concurrency files passed 724 tests. Opt-in remote workspace integration passed 65/65 and ownership lifecycle passed 16/16, with no skips. - Final head `65dcd6c6a601448b5a436b7a7f2c84b8cfde761b` adds only the parent's unsupported-host test guard. Its 18 concurrency cases and generated server test-layer typecheck passed on the exact head. Production bytes match the qualified head above. - Exact-head CI passed on `65dcd6c6`: 54 success, 3 skipped, 1 neutral; no failed or pending jobs. Greptile reviewed this exact commit at 5/5 with no actionable issue and zero open threads. - The current-head Paperclip assessment is registered and queued with no run at the latest observation. It has not delivered a current-head score. The earlier 3/5 assessment and its resolved findings remain history. No extra review request or hosted task replay was used to force delivery. ## Risks - The new allocated-source proof is conservative. Extra or missing execution material keeps work held and needs inspection. - This changes admission after completed preservation. It does not certify unknown provider effects or restore repository access. - Administrative import can now return a named busy hold while an execution row is owned. It retains the archive and journal for a later administrative retry. - There is no schema migration or new API. Stop, task ownership, current actor access, budget and approval checks retain their normal owners. ## Model Used OpenAI Codex, GPT-6 family. The exact runtime model ID and context window are not exposed in this agent session. The agent used reasoning, tool calls, code edits, real local PostgreSQL and filesystem tests. A separate Codex agent reviewed the admission and execution boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
de9f263454 |
fix(workspaces): prevent preservation lock cycles (#18918)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces save files before a later task turn can use them. > - Admission, preparation and preservation share task, run and workspace rows. > - Snapshot inserts also lock their foreign-key parents. > - Different acquisition orders could form a PostgreSQL deadlock and delay saved-file recovery. > - This pull request orders those runtime parents and refuses busy legacy publication owners. > - Files and ownership evidence remain available without replaying provider work. ## Linked Issues or Issue Description **What happened?** Snapshot reservation and publication could hold a workspace while waiting for a task or run. Other workspace owners could hold that parent while waiting for the workspace. Review also found reverse preparation, provisioning and legacy-import paths. Real PostgreSQL owner-pair tests reproduce `40P01`. **Expected behavior** Concurrent owners must complete or retain their normal ownership refusal. A lock cycle must not interrupt file preservation. **Steps to reproduce** 1. Prepare a remote workspace with retained files in disposable PostgreSQL. 2. Schedule actual save and reserve, preserve, prepare or provision functions on opposing rows. 3. Release the scheduling barrier. 4. Unfixed source deadlocks. Fixed source saves the files or returns its normal ownership hold. The tests also schedule an actual legacy import beside an arriving reservation and a stale native status commit. **Paperclip version or commit** The original three save regressions fail on unchanged master `ce7766dc11c4515190d48d395f26a90bba900f8a`. Reverse preparation, provisioning and imported no-op controls fail on published `2d13b3f`. The additional import-publication and native-status owner pairs also have causal PostgreSQL RED/GREEN proof. **Deployment mode** Source checkout with PostgreSQL and real filesystem commands. Snapshot encryption and reload use local object storage. No hosted storage, live model or original-task recovery claim. Related: Refs #18908, whose preservation acceptance exposed this separate fault. This repair has no code dependency on that PR. Refs #17846 for separate legacy terminalization locks. Refs #18746 for the documented admission order. Repository search found no existing repair for this owner graph. ## What Changed - Lock company, current task and run before the runtime workspace. Recheck the run binding and pin save-operation attribution. - Apply that order to reserve, prepare, provision and both save transactions. New shared-workspace provisioning receipts use the current task. - Lock import source parents before reservation and publication. Recheck the source binding and retain the publication fingerprint. - Use full-strength `NOWAIT` locks for legacy coordinator, run and allocation rows. Only PostgreSQL `55P03` becomes `remote_workspace_migration_blocked:execution_rows_busy`. The publication transaction rolls back. Other errors propagate. - Preserve writer, epoch, snapshot, foreign-key, Stop and owner checks. Keep command, capture and upload I/O outside row-lock transactions. - Add actual-owner concurrency, unknown-effect, eventual retry and rollback controls. Document each owning transaction and its foreign-key edges. ## Verification - Node 24.22.2 and pnpm 9.15.4: all 107 save-concurrency, migration-concurrency and migration tests pass. - All 81 enabled disposable-PostgreSQL integration and lifecycle controls pass on the final head. Both opt-in database variables point to a dedicated local test database; no skipped case is counted as qualification. - Actual preparation/provision/import no-op peers reproduce three deadlocks on the prior source. The fixed owners preserve and reload files. The real provisioning command runs once; a lost response remains an unknown effect and is never replayed. - Actual legacy publication versus reserve and stale native status commit also reproduce deadlocks before repair. Busy run/lease controls preserve the journal, archive and files, then finish from the same journal after release. - A real publication constraint failure rolls back earlier coordinator, profile and snapshot writes. Its original database error propagates. - Immutable same-company and foreign-task guards, writer/epoch loss and uploaded-object retention pass. Four real shared/inherited reuse cases fail old receipt attribution and pass current-task attribution. - The original spontaneous issue-foreign-key failure did not retain its peer statement. These controlled owner pairs prove supported defects; they do not identify that historical peer. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-save-concurrency.test.ts server/src/services/remote-workspace-migration-concurrency.test.ts server/src/services/remote-workspace-migration.test.ts`. Enable the separate lifecycle suites only with their documented disposable test database variables. - Full recursive typecheck and `pnpm build` pass on production commit `dacf6278abd3257c407a297c6db8872e76d85f61`, using Node 24.22.2 and pnpm 9.15.4. Current head `4c1d7769e6cfa9536a67c4e3b536cef155f19966` changes only the canonical PostgreSQL host-support guards. All 18 changed concurrency cases executed and passed with zero skips; their server test-layer typecheck passes. All 57 current-head check contexts are terminal and clean: 53 passed, two skipped and two neutral. Required `ci / verify` and `ci / e2e` pass. Greptile and Paperclip both reviewed exact current head at 5/5 with no actionable findings. There are no open review threads. - Composition with #18908 includes this exact parent history. Its production-composed head passed 724 affected tests, all 81 enabled integration/lifecycle controls, recursive typecheck and build on Node 24. Its final test-only guard head passed the 18 changed concurrency cases and their test-layer typecheck. This is local server/PostgreSQL/native-backend-boundary evidence, not a hosted model or storage claim. - The required broad local Vitest attempt stopped at an unchanged chat publication-batch test with `socket hang up`. The earlier CI workspace shard timed out in the unchanged CLI chunked-ZIP test. Both failures remain recorded; neither is reported as a pass or hidden by timeout changes. The chat case passes one bounded focused head/base control, so the original full-suite socket failure was not reproduced by that control. ## Risks - Short transactions take parent-row locks earlier. Competing writers can wait for those transactions. - Administrative import now refuses a busy native owner. It retains its journal and archive for a later explicit retry. - Native owners have different lock orders. The import's nonblocking requests avoid a new global native lock policy. - Missing bindings and uncertain command effects remain holds. The repair does not infer termination, release an executor or replay external work. - No schema, foreign-key or timeout change. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this agent session. The agent used reasoning, terminal tools, code edits, PostgreSQL and filesystem tests. A separate Codex agent reviewed the production ownership graph. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
55d7df2bad |
fix(runtime): bound checkpoint data and preserve durable failed sessions (#18959)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner checkpoints preserve controller state across server replacements. > - Enrollment imports legacy controller state into those durable checkpoints. > - The importer passed extra source fields into each encrypted record, duplicating the whole source and exhausting memory. > - Some failed runs already had durable checkpoints but no session snapshot. Re-importing them incorrectly required resumable state. > - This PR stores only each record's binding and contents, and preserves matching failed checkpoints without changing their recovery status. ## Linked Issues or Issue Description Refs #18957 and #18943. **What happened?** A 538-record migrated checkpoint stored 1,108,851,510 bytes of encrypted material. Every record included the full 1.4 MB legacy source. Large writes disconnected and later reads exhausted memory. **Expected behavior** Each encrypted record contains only its company, issue, run, part and contents. **Steps to reproduce** Run offline controller migration with a retained controller that has many commands. Decrypt a saved envelope and inspect its fields. The old writer also includes the entire offline migration input. **Paperclip version or commit** Reproduced on `a1f3637c96dfadaf3a15ae9c83aa8227f4fed20e`. ## What Changed - Serialize only company, issue, run, part and contents in encrypted checkpoint envelopes. - Preserve an already-durable failed checkpoint when its run, runner and session match and the legacy session snapshot is absent. Keep its failure, recovery blockers, owner and pointers unchanged. - Report the number of failed checkpoints preserved during maintenance. - Test exact encrypted contents, idempotent preservation after removing the old home, and rejection of successful or foreign-session sources. - Document the migration findings and verification boundaries. ## Verification - The oversized-envelope regression failed before the serializer fix. - The already-durable failed-source regression failed before the preservation fix. - Four focused test files: 44 tests passed. The migration suite passed all 5 tests again after adding the foreign-session control. - Server typecheck passed. Greptile reviewed `c4f02a064c` at 5/5 with no findings. The exact-head PR workflow passed, including typecheck, build and all required tests. - Live staging repair preserved all 16 affected checkpoint sets: 2,982 records, 6,400,144,676 bytes reduced to 15,419,528 bytes. Each record matched its legacy source and passed re-encryption verification. Original encrypted rows and source files remain on the retained volume. - Full hosted enrollment and active-agent deployment proof remain pending. Deployment will follow the normal staging release pipeline. ## Risks The serializer affects normal and migrated checkpoint writes. The decoder already accepts the same five fields and continues to accept older envelopes. Preservation applies only to a failed run with an exact, validated durable checkpoint and no session snapshot; it grants no new recovery authority. This change does not rewrite existing database records automatically. Existing failures and missing provider files remain failures until their owning recovery path resolves them. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
d12de135a7 |
fix(runtime): preserve legacy identities and bound checkpoint imports (#18957)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Ephemeral web servers must preserve native Runner sessions before removing local storage. > - Legacy session imports validate the workspace and saved provider authority. > - A protocol lease can retain a logical workspace ID while copy-back uses a separate physical sandbox lease ID. > - Comparing these different IDs blocks valid legacy sessions during enrollment. > - This pull request checks the workspace binding, retains the full saved protocol identity check, and bounds checkpoint inserts inside the owner transaction. > - Existing sessions can migrate without changing their identity or discarding files. ## Linked Issues or Issue Description Refs #18943 and #18841. **What happened?** The first full staging enrollment stopped with `native_controller_migration_artifact_binding`. A successful native session had valid controller and provider-pack identities, but its physical sandbox lease had a different ID. **Expected behavior** Import the original session when its workspace and saved provider authority match. Reject foreign workspaces and protocol identities. **Steps to reproduce** Create a retained session whose sync record points to the execution workspace and a separate physical lease. Run the offline controller migration. The prior comparison rejects this valid record. **Paperclip version or commit** Reproduced on `84aa989bf9d08fc005ac273c37f9e1d860a4db25`. **Deployment mode** Docker with external PostgreSQL and shared object storage. ## What Changed - Compare the sync record's workspace ID with the frozen execution workspace. - Keep the full saved provider-pack check for company, agent, workspace, session, Runner, protocol lease, remote path, and provider digest. - Use distinct physical and protocol IDs in the database-backed migration fixture. Reject a foreign workspace and a changed protocol lease before any checkpoint import. - Limit migration inserts to 64 records and about 512 KiB of serialized rows per statement. Send a larger individual record alone. Keep all statements in the same transaction. - Verify that a conflict in a later batch rolls back earlier batches, then import and decrypt the complete checkpoint. - Record both live failures and the comparison evidence in the preservation plan. ## Verification - The updated fixture failed on the original comparison and passes with this fix. - Targeted migration, controller recovery, and provider authority tests: 39 passed. - Full repository typecheck and the latest server typecheck passed locally. The complete PR workflow passed on `a1f3637c96` (run 41827558125), including build and test gates. The Docker Compose setup download failed once before the test step; its infrastructure retry passed. A separate full local test/build run is still in progress. - A read-only staging preflight inspected 807 selected legacy sessions. It found zero mismatches under the corrected workspace and full saved-provider checks. - A single 538-record, 2.2 MB checkpoint insert repeatedly closed the live database connection on pooled and direct endpoints. A forced-rollback comparison imported the identical checkpoint with batches of 64. An unbatched control failed again after cache warm-up. The exact server-side disconnect cause remains unknown; all failed receipts are retained. - The original failed enrollment and volume are retained. The corrected importer committed 32 additional sessions and 2,982 checkpoint records. Full import and deploy handoff remain pending while memory use is investigated. ## Risks The import checks durable session authority. A weaker identity check could import the wrong session. This change retains the saved provider authority validation and adds an explicit foreign-protocol rejection test. Batching adds statements for large imports, but a late failure still rolls back the complete owner transaction. It does not rewrite session identity, change provider lifecycle, or detach any volume. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84aa989bf9 |
Preserve native sessions through legacy workspace enrollment (#18943)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
612d9931a6 |
Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The |
||
|
Devin FoleyandPaperclip
|
1a39dea9aa |
fix(heartbeat): keep a routine claim queued while its slot is held (#18933)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A routine creates one task per tick. Only one open task of a routine can hold an execution lock (`issues_open_routine_execution_uq`). > - Since #18879, a failed native run that waits for a scheduled resume keeps its task's lock. Routine dispatch counts only queued, running or scheduled runs as live, so it can queue a newer task of the same routine meanwhile. > - When that newer run is claimed, stamping its execution lock violates the unique index. The raw database error is not a deferrable claim rejection, so it escapes the queue loop. The queue code notes that this can stop the agent's queue and startup recovery. > - This pull request turns that exact violation into a 409 claim rejection, so the run stays queued until the slot is free. > - The benefit is that a busy routine slot delays one run instead of stopping the queue or startup. ## Linked Issues or Issue Description Refs #18879 (keeps a failed run's lock while its resume is pending), #18916 (defers the same unique-index conflict in restart recovery's lock reclaim) and #18931 (open; keeps one run's restart-recovery failure from stopping startup recovery). This is the same index conflict on a different path: the heartbeat queue claim, not restart recovery. It does not change #18916 or #18931. Its doc paragraph is separate from the one #18931 edits. **What happened?** While auditing the lock paths from #18879 for this index, I found that a queued run of a newer routine task can be claimed while an older task of the same routine still holds the slot through a failed run with a pending resume. The claim's lock stamp then raises `duplicate key value violates unique constraint "issues_open_routine_execution_uq"`. `isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error escapes `startNextQueuedRunForAgent`. **Expected behavior** The claim is rejected as a conflict and rolls back. The run stays queued and is claimed after the older task releases the slot. Other errors behave as before. **Steps to reproduce** 1. Create two open tasks of the same routine (same origin and fingerprint). 2. Give the older task an execution lock held by a failed run. 3. Queue a run for the newer task and claim it. On `master` the claim throws the unique violation. The new queue test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `0dedbf23c35`. **Deployment mode** Any server with routines whose tasks run on native runners. ## What Changed - `bindClaimedIssueExecution` in `server/src/services/heartbeat/queue.ts`: if stamping the execution lock violates `issues_open_routine_execution_uq`, it throws `conflict("Another execution for this routine is already in progress")`. The claim transaction rolls back, and the queue loop keeps the run queued as it does for other 4xx claim rejections. Every other error is rethrown unchanged. The issue checkout route already maps this violation to the same 409. - `doc/execution-semantics.md`: one paragraph on the behavior. ## Verification - Failing first: `queue.test.ts` "keeps a claim queued while another task of its routine holds the execution slot". On `master` the claim raised the `issues_open_routine_execution_uq` violation. Now it rejects with 409, the run stays `queued`, neither task's lock changes, and nothing executes. - Failing first, review round 1: `heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run queued while another task holds the routine slot, then runs it once the slot is free" goes through `resumeQueuedRuns`. On `master` it rejected with the unique violation. Now the routine run stays queued with no error code while another queued run for the same agent executes. After the slot is released, the routine run executes exactly once (executions are counted per run, because the finished task can queue its own follow-up run). - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed. - `npx vitest run src/services/heartbeat/recovery.test.ts src/__tests__/routines*.test.ts`: 4 files, 205 passed. - `pnpm --filter @paperclipai/server typecheck`: pass. - Not verified: the queue loop end to end on a real instance. The loop's handling of a 4xx claim rejection is existing behavior. ## Risks - While the older task holds the slot, the newer run stays queued and the loop logs "queued heartbeat run claim was rejected" on each pass. It is claimed once the older task's resume finishes or its lock is released. - Only this one constraint is mapped. Other database errors still escape as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
652cd76026 |
fix(ui): expose Stop for failed native retries (#18899)
## Thinking Path > - Paperclip manages agents and their tasks. > - A native run can fail while it still owns a same-run retry. > - The server supports Stop for that pending execution. > - Run details show Cancel only for queued or running records. > - A failed pending retry therefore has no ordinary cleanup control. > - This pull request exposes the existing Stop eligibility and uses the existing cancellation endpoint. ## Linked Issues or Issue Description Refs #14743 and #14733 for correlated native Stop. Those changes cover backend cancellation and protocol qualification. They do not add the failed-run control to run details. Searches of current master and open PRs found no matching UI repair. **What happened?** A native run can show failed while its coordinator still owns a retryable failure with no accepted result. The backend can cancel that retry. Both run-detail implementations hide the cancellation button because the record is terminal. **Expected behavior** The selected run offers Stop when the server confirms that pending native retry. Unknown, legacy, terminal and accepted-result states do not gain this control. Existing queued and running cancellation stays available. **Steps to reproduce** Render either run-detail implementation for a failed native run with an eligible retry. The original UI has no Stop button. The original batched execution projection has no eligibility field. The regression tests exercise both real components and the actual PostgreSQL projection query. **Paperclip version or commit** The missing control and projection field were reproduced on |
||
|
Devin FoleyandPaperclip
|
6c7b136b16 |
fix(native): keep startup recovery running when one run's claim fails (#18931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - At startup, the server runs native restart recovery before it reports ready. Recovery claims each interrupted or retryable native run in its own transaction. > - On one instance, one run's claim hit the routine execution unique index. The error escaped restart recovery, startup recovery failed closed, health stayed "still starting", and two deploys rolled back. > - #18916 fixed that exact conflict: it defers the run when its claim hits `issues_open_routine_execution_uq`. Any other error in one run's claim still aborts the whole pass, and so does a failed per-run audit write. > - This pull request handles every per-run claim failure the way #18916 handles the routine conflict, and keeps a failed audit write from stopping committed claims. > - The benefit is that one bad row cannot keep the server from becoming ready, and no run is claimed or changed when its claim fails. ## Linked Issues or Issue Description Refs #18916. This change generalizes its catch: #18916 keeps the routine-conflict disposition (`routine_execution_busy`) exactly as it is, and this change adds a separate reason for any other per-run failure. It does not change how the routine conflict is detected or deferred. Refs #18879, whose lock reclaim exposed the startup failure. Refs #18877 (open), which edits other parts of the same files. This change touches only the claim's error path and the audit loop, so a second merge needs at most a small rebase. **What happened?** Two deploys never became healthy. About 25 seconds after the listener bound, the boot log showed `startup native runner restart recovery failed closed` and `startup heartbeat recovery failed`, both from a failed `update "issues" set "execution_run_id" = ...` that violated `issues_open_routine_execution_uq`. #18916 now defers that violation. Any other error in one run's claim (another constraint, a trigger, a timeout) still escapes `claimNativeRestartRecoveries` and fails startup the same way. #18916's own test asserts this: "Other database failures still fail the recovery pass." **Expected behavior** One run's recovery failure is logged and recorded on that run, and the rest of startup recovery continues. The server becomes ready. Only a failure of the candidate scan itself fails the pass. **Steps to reproduce** 1. Create a native run that is due for restart recovery, and make its claim fail (the new test uses a trigger on `native_run_finalizations`). 2. Add a healthy run that is due for restart recovery. 3. Run restart recovery for both. On `master` the batch throws and the healthy run is not claimed. **Paperclip version or commit** `master` at `4a83ba4050f`. **Deployment mode** Any server that runs startup native restart recovery. ## What Changed - `claimNativeRestartRecoveries`: #18916's `.catch` on each candidate's claim transaction no longer rethrows other errors. The whole claim has rolled back, so the run's state is unchanged and nothing is claimed. The run gets `awaiting_evidence` with the same 30-second retry deadline. The reason stays `routine_execution_busy` for the routine slot conflict, and is the new `restart_recovery_failed` for any other error, which is logged at error level. The batch continues. - `recoverNativeRunsAfterRestart`: records a `restart_recovery_failed` deferral as a warning transition on the run ("Native restart recovery failed for this run and will retry"). A failed `getRun` or transition write is logged and no longer stops the dispatch of claims that already committed. - `doc/execution-semantics.md`: replaces "Other database failures still fail the recovery pass" with the per-run rule. ## Verification - Failing first: `native-restart-recovery-isolation.test.ts` (new, PostgreSQL). It seeds the reported shape (two open tasks of one routine: the older with a due scheduled resume and a cleared lock, the newer holding the lock), a run whose claim fails through a trigger, and a healthy run. On `master` the batch threw `injected recovery claim failure`. Now the routine run is deferred with `routine_execution_busy` and the failing run with `restart_recovery_failed`, and both keep their coordinator, attempt and locks. The healthy run is claimed. `recoverNativeRunsAfterRestart` completes and records an info transition for the routine run and a warning transition for the failing run. - Two existing tests expected the pass to reject on one run's failure: #18916's unrelated-constraint test, and failed-workspace retention's rollback test. They now expect the per-run deferral. Their rollback checks (no claim, unchanged coordinator, lease, lock and recovery actions) are unchanged, and the unrelated failure still does not become a routine wait. - `cd server && npx vitest run` over restart recovery (isolation, integration, diskless, preparation), controller recovery, heartbeat recovery, scheduled resume lock, failed-workspace recovery, startup cleanup, native resumption, live recovery work, startup feedback export, recovery deadlines, runtime selection, stop accounting, warm transition restart, workspace reconstruction, remote workspace migration and heartbeat process recovery: all pass except one. `heartbeat-process-recovery` "redacts opaque environment-bound credentials from Sentry diagnostics: result" fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: two `heartbeat/recovery.test.ts` cases make `getRun` and then `appendRunEvent` fail for a deferred run while a committed claim is in the same batch. Recovery completes and dispatches the claim exactly once; the `appendRunEvent` case also checks the warning transition. Both fail when the catches are removed. In one of four full runs of that file, the unrelated orphan-cleanup case "reconstructs missing native preservation during orphan cleanup without replay: reuse_by_environment" failed; it passed alone and in the other three full runs. - After the rebase onto `4a83ba4050f`: the isolation and failed-workspace recovery tests and `pnpm --filter @paperclipai/server typecheck` pass. - Not verified: a deploy to a real instance. ## Risks - A persistent per-run error now retries every 30 seconds and logs an error each time, instead of stopping startup. The run stays unclaimed, so ownership does not change. - If the database fails for every candidate's claim, startup recovery now completes with every run deferred, instead of failing closed. The candidate scan still fails closed, and the deferred runs retry on their deadlines. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0dedbf23c3 |
fix(chat): retain receiver ownership during runtime construction (#18930)
## Thinking Path > - Paperclip manages agents and their work. > - Chat receivers hold a durable lease before they accept messages. > - Runtime construction can outlast a lease renewal interval. > - Renewal previously required an attached runtime before construction had returned. > - That check could revoke the same receiver that was still starting. > - This change uses the exact pending initialization as proof during construction. > - The existing durable token and endpoint checks still fence every renewal. ## Linked Issues or Issue Description Related: #18899 exposed the existing receiver failure in its chat CI shard. This is a separate lifecycle repair. #18795 provides the relevant production and sleep fixture context. Searches found no overlapping open initialization repair. **What happened?** The receiver acquired its lease and started its renewal timer. The service then awaited credentials, command preparation and runtime construction before attaching the runtime to its context. A renewal during that interval treated the missing attachment as lost ownership. Configuration later failed with a Gateway ownership error, although its original durable token was still current. **Expected behavior** The exact pending initialization can retain its own lease while construction runs. Missing attachment alone must grant no ownership. A changed token, paused or removed endpoint, changed generation or credentials, and shutdown must still fence the receiver. **Steps to reproduce** 1. Use the real configure and PostgreSQL ownership producers with a 60-second lease and 50-millisecond renewal. 2. Hold the actual runtime construction call for 200 milliseconds. 3. Release construction. Untouched code returns the Gateway ownership error. 4. With the same construction delay and a five-second renewal control, configuration succeeds. **Paperclip version or commit** The causal regression fails on unchanged production source at `e7549ab613`. The original CI error was observed in the chat shard of #18899. That CI run did not record the precise setup and renewal order, so this local proof does not establish its historical scheduling. ## What Changed - Store the runtime context identity in the existing pending initialization entry. - Allow pre-attachment renewal only for that exact tracked initialization and current local ownership. - Recheck local ownership inside the existing renewal transaction. Preserve company, endpoint, generation, credential and original-token checks. - Before SDK listener initialization, wait for any older renewal, recheck local ownership, then obtain fresh durable proof. Preserve the existing checks after initialization. - Keep attached runtime identity checks, lease deadlines, renewal intervals, shutdown and listener retirement unchanged. - Add actual configure and database race controls. Document the shared Discord and Photon receiver contract. ## Verification - Untouched production red: the actual construction gate crosses a renewal and returns the same Gateway ownership error. The otherwise identical slower-renewal control passes. No lease, exception or provider outcome is injected in these positive cases. - A subsequent review found a second real race: final runtime removal did not prove that SDK initialization had never started. Four actual initialization-hook controls failed before the pre-initialize fence. The final matrix asserts zero hook calls for lost or changed authority, including a foreign-token control with no periodic renewal due. An attempted cross-company endpoint move is rejected by the existing database foreign key; the valid original initialization still succeeds. - A third actual-producer red holds completion of a real committed renewal, changes the durable token, and proves that joining that old promise could incorrectly accept configuration. The corrected startup check waits for that renewal and then obtains fresh proof; it cannot credit the earlier result alone. - Final relevant full Discord registration group, takeover/callback cases, buffered host-pause recovery, failed construction, credential-fenced pause and shared Photon consumers: 34 passed across two files; 1,840 other cases were deselected. Four executed cases cover Photon transport recovery, failed authentication, duplicate allocation ownership and checkpoint fencing. The tests use the real configure/database producers with fake provider transports and observe SDK initialization; they do not prove physical network behavior. - All server type layers, the module-boundary gate, and production server/vendored Runner build pass at exact final `38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp records this exact commit. - Earlier fixture mistakes are preserved separately: a test initially changed every lease key, including the credential lease, and another attempted an invalid cross-company row move. Neither is counted as a production behavior red. The corrected controls target the Gateway key and assert the existing company constraint. - An initial isolated dependency graph resolved an older shared telemetry build. Correct workspace links restored the all-layer typecheck without changing telemetry source. - Full repository checks were not run locally for this narrow slice. Fresh current-head CI and substantive reviews remain required. No hosted task, provider work, fleet action or deployment was performed. ## Risks The ownership helper is shared by Discord and Photon. The selected tests cover both consumers. The pending entry grants only permission to renew the original token; it grants no message, callback or publication authority and adopts no listener. Missing or changed initialization identity fails closed. There is no schema change, historical backfill, new worker or additional idle scan. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection, source editing, shell execution and test tools. The host did not report the context-window size. A separate coordinator reviews the source before delivery. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4a83ba4050 |
fix(workspaces): isolate unpublished preparation directories (#18922)
Bind new remote preparation directories to the writer epoch and environment lease. Preserve earlier unpublished files and the recorded path for published warm workspaces. Run the interrupted reused-allocation and saved-file lifecycle regressions with owned Postgres by default, and retain the warm-path recovery checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f211d44c00 |
fix(heartbeat): preserve queued coalesced comments at claim (#18913)
## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e7549ab613 |
fix(runtime): defer native routine recovery collisions (#18916)
Treat the exact routine execution unique conflict as a pending recovery obligation after the complete losing transaction rolls back. Preserve the current routine owner, provider attempt and checkpoint, and continue the startup batch. Cover real constraint failures, row preservation, one reclaim after release and unrelated database failures. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0884e73b46 |
test(e2e): verify saved assigned skill use (#18783)
## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
630ed6624c |
test(e2e): add verified dependency continuation QA (#18901)
## Thinking Path > - Paperclip lets people manage work assigned to AI agents. > - Task dependencies must hold assigned work until its blockers are complete. > - The reusable Product E2E set needs a browser journey for that full path. > - The journey must verify saved records, automatic wake identity, and the result after reload. > - The original implementation passed staging QA, but its automated review workspace has an export hold. > - This replacement integrates current master and earns a separate normal review while preserving the original attempt. ## Linked Issues or Issue Description Refs #18789, the original dependency QA pull request. Related: #7542 and #17524 for dependency wake and explicit resume behavior; #18782 for the merged lifecycle registration layout. **What existing behavior does this improve?** The Product E2E catalog can verify the complete blocked-task continuation path against a configured native Cloud target. **Current behavior** The main catalog lacks this full browser journey. The original PR's automated review task has a retained workspace export hold (`native_workspace_sync_out_retry_exhausted`), with neither Retry nor Continue available. Its task, files, checks, and failed export attempts remain preserved. **Proposed behavior** Add the same independently checked `task-dependency-resume` journey through this replacement. Its new commit normally merges the actual fetched master `4e29f001396c9fc76075850165ccfa3a6743e3ef` into verified source `e53dd51407b901b14bd582eeb2071fabe6955d9d`. That integration contains 39 real upstream path updates. The replacement must pass fresh CI and automated review on its own head. **Reason and benefit** Daily QA can detect premature work, a missing or duplicate continuation, and a result from the wrong run. A separate review workspace lets the normal review process inspect the current integration while the original held workspace stays intact. **Breaking changes** The change adds an explicit journey. Existing journeys and their registrations remain present. ## What Changed - Add a browser journey that creates a human-owned blocker and dependent task, saves their relation, selects Blocked, and assigns the configured native agent. - Require a measured ten-second blocked wait, a board-owned blocker completion, and exactly one automatic successful native run. - Independently verify the saved wake source and blocker identity, configured provider/model/environment, committed native phase, document revision authorship, and reloaded result. - Return `unsupported` before observations or task creation when the native provider or model identity is missing or blank. - Attempt both evidence writes, preserve an existing exact error object and outcome, and fail otherwise successful work when required evidence cannot be saved. - Grade runtime identity, document authorship, reload visibility, and retained history from their own evidence bound to the same company, task, agent, and run. All eight checks remain required. - Keep the random review reference in the blocker description. Save allowlisted private observations and read failures through the existing evidence pipeline. - Register the case at the distinct dependency anchors in the catalog, types, checks, source hash, dispatcher, features, and documentation. The delta against fetched master is exactly the original 15 QA paths. The fixture, contract tests, shared registrations, and dispatch remain unchanged. The browser actions and observations are byte-identical to `d41558b5`; only evidence finalization changes in that flow. Six shared-file AST comparisons still match fetched master after removing only the dependency registrations. All upstream integration changes remain preserved. ## Verification Prior acceptance remains bound to its actual source: - [Staging QA run 41731870131](https://pages.paperclip.ing/evals-qa-smoke-41731870131-1/) passed all eight checks and all five phases, with closed cleanup, on `dc9ea42d46d3015c010d0532b48e0da973d3611a`. - [Original full CI run 41745149524](https://github.com/paperclipai/paperclip/actions/runs/41745149524) passed on `e53dd51407b901b14bd582eeb2071fabe6955d9d`: 48 successful jobs and one skip on attempt 1. - The original failed staging result and the original held review remain intact. These prior passes are not new checks for this replacement. Passed locally on the current replacement source with Node 24.22.2 and pinned pnpm 9.15.4: - One complete `install --frozen-lockfile --ignore-scripts`, followed by the maintained `node scripts/link-plugin-dev-sdk.mjs` for links inside this checkout. - `node scripts/ensure-plugin-build-deps.mjs` and `pnpm --filter @paperclipai/db run build`. - `node scripts/typecheck-test-projects.mjs --low-memory tests/runner-e2e`, including the current Runner TypeScript build, fresh server declarations, and E2E typecheck. - 239 focused controls across eight files. These cover missing/null/empty/blank execution identity, real filesystem write failures, synchronous writer failures, frozen original errors, and independent defect attribution. - 5,302 comparisons against the prior grader, with zero aggregate acceptance changes or new throws. The sealed actual staging observations from run 41731870131 still produce the same eight passing checks. This is finite calibration of retained evidence, not a new staging run. - Credential-free discovery: 22 unique cases, including dependency and lifecycle. - Source-preservation checks and whitespace checks for both the normal integration and the replacement delta. The focused command is: ```sh node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/product-journeys/task-dependency-resume-evidence.test.ts tests/runner-e2e/product-journeys/task-dependency-resume.test.ts tests/runner-e2e/product-journeys/task-dependency-resume-contract.test.ts tests/runner-e2e/product-journeys/contracts.test.ts tests/runner-e2e/product-journeys/observation-read.test.ts tests/runner-e2e/catalog.test.ts tests/runner-e2e/journey-features.test.ts ``` The old generated cache failed current-source equality and was not copied. Canonical current outputs were built and sealed with their input hashes, including CLI, UI, and provider inputs. No Rust build, QA dispatch, model call, or action on the held workspace was performed for this replacement. Review follow-up: the first replacement head `b4408c5732e1800f7ccf66c639cf382b5ff2f01a` passed full CI run [41749940687](https://github.com/paperclipai/paperclip/actions/runs/41749940687), with 48 successful jobs and one skip. The native review then found the missing execution-identity preflight. The preflight now rejects that setup before work. Completed current-head checks are recorded below. The current E2E typecheck reused the same canonical declarations only after all 11,273 build inputs, two installed toolchain files, and 6,747 generated files matched their saved hashes. The five changed case files are outside that build-input set. One sandbox test run passed 216 controls and lacked the report subprocess output. A direct `tsx` diagnostic confirmed a temporary pipe `listen EPERM`. The unchanged 217 controls then passed with ordinary local IPC permissions. Both results remain recorded. The `d41558b5807e609032bbadddb103d6b0a7db4a12` attempt [41753398775](https://github.com/paperclipai/paperclip/actions/runs/41753398775) is retained: 45 successful jobs, one skip, two failed server shards, and a failed aggregate. The runtime readiness case refused connections within its ten-second window; the issue-deadline and large repository cases hit their existing test limits. Their causes remain unknown. A source-matched Linux CI run executed and passed all three cases, and one unchanged local runtime case passed. No timeout or assertion changed and no old-head diagnostic recheck was issued. The native review then identified error masking during evidence persistence and cumulative diagnostic flags. The current head fixes those two owned-case issues. Its completed current-head checks are recorded below. Seventeen of twenty new regression controls failed before these fixes; all now pass. Two additional controls exercise synchronous writer failures. Both original review and CI evidence remain intact. The eight required checks, one-run limit, 500-cent configured budget, and bounded ten-second observation are unchanged. Fresh full CI, current-head Greptile 5/5 with no open findings, and every actual review context must complete before merge. Prior statuses are not copied or overridden. Current-head verification is complete for `d9bc055f1f6609463679dc78fdcb96034d2c3727`: - [Full CI run 41758005698](https://github.com/paperclipai/paperclip/actions/runs/41758005698), attempt 2, completed successfully at 02:26:19 UTC on November 4: 48 successful jobs and one skip. - Attempt 1 lost one exact Spot runner. AWS confirmed `Server.SpotInstanceTermination` and `instance-terminated-no-capacity`; its log recorded the shutdown. That failed job and the failed aggregate remain preserved. - One normal failed-jobs recheck ran only the interrupted static checks (02:19:13–02:25:54 UTC) and aggregate (02:26:13–02:26:18 UTC). The other 46 successful start/finish pairs remained unchanged. The skipped selector was not an execution. - Greptile and Paperclip Review both scored this exact head 5/5. There are no unresolved review findings. ## Risks - The staging pass remains attributed to `dc9ea42`; final admission must retain that source provenance and the current preservation proof. - The case uses one paid work run when executed. Agent setup usage remains separate from task-run accounting. - The ten-second blocked observation proves a finite wait. Missing, incomplete, foreign, premature, or duplicate evidence fails the independent checks. - The original held workspace still needs its own disposition. This PR leaves it preserved for its owner. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, and code execution. The exact model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a194166017 |
perf(ci): repair dependency caches and parallelize transport fixtures (#18883)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect changes to its native Runner and control plane. > - Recent Runner jobs restored a Rust cache but still rebuilt dependencies for 74–92 seconds. > - New workspace install hooks disabled the materialized JavaScript dependency cache. > - A single transport file also kept 326 test cases on one worker. > - This pull request corrects the cache inputs, replays known install hooks, and separates independent transport test groups. > - The same checks can finish sooner without removing test cases or widening runner access. ## Linked Issues or Issue Description Refs #18794 and #18544. Related dependency-action version update: #17698; this change fixes the cache input contract used by the currently pinned action. **What happened?** Recent PR jobs such as #18783 and #18825 restored an exact Rust dependency cache from October 16. The virtual workspace's development profile changed on October 30. Rust Cache 2.9.2 hashes member manifests but omits that virtual manifest. Rebuilding cannot replace an immutable exact cache entry. The JavaScript cache also rejected the new server and Runner Copilot postinstall hooks. **Expected behavior** Changing a Cargo profile selects a new dependency cache. An exact JavaScript dependency restore still performs required workspace install hooks. Independent transport fixtures use the existing file workers while retaining all assertions and real deadlines. **Steps to reproduce** Inspect the Rust dependency cache hit and following `build:rust` in the recent Runner package jobs. Run `node scripts/ci-dependency-cache.mjs` on the previous revision to see `cacheable: false`. Run the original transport file to measure the serial test work. ## What Changed - Add the virtual workspace manifest and ancestor Cargo configuration to all shared Rust dependency keys. Keep reader/writer identities aligned and preserve master-only cache writes and CODEOWNERS-only AWS routing. - Recognize only the exact reviewed root, server, and Runner postinstall commands. Include the server/Runner installer scripts in the dependency key. Replay workspace postinstall hooks after a materialized restore. Unknown lifecycle hooks still disable reuse. - Move attachment recovery, warm attachment, live adoption, and suspension fixtures into separate files. Keep the large event-backlog cases in the original file to avoid starting another expensive fixture group at the same time. Share five stateless test helpers outside production source. - Extend the opt-in `profile-ci` benchmark with a PR-scoped Rust cache producer and a fresh-host consumer. Require exact dependency/workspace hits and reject any Cargo unit that is rebuilt. Retain the JSON build evidence. Existing setup qualification checks real JavaScript cache restores. - Schedule the ten measured files over 50 seconds first, in approximate duration order. Package CI uses up to four workers, bounded by available CPUs; local runs retain Vitest defaults. The AWS runner has four CPUs and 16 GiB RAM; the three-worker run averaged 44% CPU and peaked at 15% memory. Preserve Vitest selection and shard membership and default ordering for other files. Job count and AWS authorization are unchanged. - Scope the optional paired server benchmark to changes in its measured inputs. Cache-only changes qualify exact restores and run the shared package tests using restored dependencies. - Update the cache and test documentation and the existing native cache contract guard. ## Verification - All 586 `.github/scripts/tests/*.test.mjs` policy tests passed, including cache trust and runner routing guards. Actionlint passed for all three changed workflows. - All 169 original top-level transport test definitions remain byte-identical. The original file passed all 326 cases in 317.99 seconds on this macOS host. A clean preliminary split passed the exact same named cases in 183.01 seconds with the same four-worker cap. - Preserve the first failed experiment: its parallel run overlapped the policy suite and timed out in the 1,024-event settled backlog case at 60 seconds; 325 cases passed. A clean repeat passed, but that does not establish the earlier failure's sole cause. The final layout keeps the backlog cases in the original file. The final layout passed the same 326 cases in 186.35 seconds (41.4% faster than the 317.99-second baseline). The original test bodies and all timeouts remain unchanged. - Fresh-host Rust qualification passed: all 308 compilation units were fresh, and Cargo completed in 1.59 seconds on a new runner ([final-head run](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126296908816)). JavaScript setup dropped from 46 seconds cold to 23 seconds with exact dependency/build restores; all 1,052 shared-package cases passed against the restored tree ([final-head job](https://github.com/paperclipai/paperclip/actions/runs/41745082837/job/126295937797)). The previous qualification also passed (43 to 19 seconds) and confirmed all three postinstall hooks replayed. These are qualification results, not an end-to-end PR timing claim. - Initial full AWS package run passed 4,907 cases with 10 existing skips. The original-file remainder fell to 118.7 seconds, but the 100-second deadline file started last, leaving the full package at 417.43 seconds. Starting only three long files first passed 4,911 cases (including four new sequencer guards) in 422.54 seconds, so reordering alone did not improve full-package time. Summed file execution was 1,202 seconds across three busy workers. The final four-worker configuration passed **4,912 cases, with the same 10 existing skips**, in **324.99 seconds**, a **23.1% reduction** from 422.54 seconds. The five added cases guard ordering, selection, sharding, and path handling. Peak memory was 17.95% of the 16 GiB runner; average CPU was 49.14% ([final-head package job](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126296339480)). - Local workspace typecheck and build passed. The complete local `pnpm test:run` passed all six chat partitions but reported failures in one existing OAuth redirect case and one gateway authority case in the general-server group. Both focused repetitions passed (three OAuth cases and one gateway case), and a complete repeat of both files passed all 593 cases in 42.89 seconds; that does not explain the original failures. The broad local run was stopped after 61 minutes once full final-head CI had passed; its remaining general/workspace/serialized work was not completed locally, and it is not reported as a pass. Original failure logs were retained. Full final-head CI provides the complete suite validation. Sequencer selection/shard guards and the 11-case transport suspension smoke passed. The corrected release workflow guard passed all 71 tests. - The first CI run failed only the policy guard that still expected the old native cache key; downstream aggregate checks also failed. That guard now checks the new profile/config prefix and existing host/target/protocol suffix. A second policy run found that the new sequencer test imported Vitest in the dependency-free policy lane; it now runs in the Runner package lane, where all four cases pass. Greptile reviewed final head `8011f84189` at 5/5 with no actionable findings and confirmed this issue fixed. Final-head policy, typecheck, build, and review passed. At 23:52:46 UTC on November 3, several AWS jobs were terminated together: the Runner package process received SIGTERM and browser logs recorded a runner shutdown signal. The failed/cancelled jobs passed on the same-head retry, including required `ci / verify`; no assertion or timeout was changed in response. The final concurrency adjustment at `fe5a69bb0b` also received Greptile 5/5 with no actionable findings. Final-head required CI, including `ci / verify`, passed ([aggregate](https://github.com/paperclipai/paperclip/actions/runs/41745082974/job/126300554711)). Local timings exclude install, build, cache transfer, and CI queue time. ## Risks A new Rust key needs one successful canonical master build before ordinary PRs can hit it. The fresh-host qualification uses separate PR-scoped entries. Parallel test files can increase I/O contention; case execution inside each file, timeouts, and assertions are unchanged. Two extracted suites exceed 1,000 lines because they retain existing multi-stage recovery fixtures unchanged; one parameterized fixture alone is 1,059 lines. No production execution logic, lockfile, credentials, runner trust gates, or required check is removed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository editing, command execution, and GitHub inspection. The host does not expose a more precise model snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted checks passed; the broad-run failures and repetitions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Michael NguyenandPaperclip
|
58d1b21098 |
fix: separate fresh setup from archived personal connections (#18902)
## Thinking Path
> - Paperclip helps people manage AI agents and their service access.
> - Each personal connection belongs to one user.
> - Setup can recover an archived connection by its name and provider.
> - A fresh setup can therefore select another user's old identity and
fail its owner check.
> - This change creates a separate connection for fresh setup after
personal archival.
> - Explicit reconnect still requires the existing owner and preserves
its history.
## Linked Issues or Issue Description
**What happened?**
A fresh GitHub setup can return 403 after a personal connection with the
same name was archived. The request does not include a reconnect target.
Setup still selects the archived identity by name and provider.
**Expected behavior**
Fresh setup creates a separate personal identity. It leaves the archived
connection, credentials, grants, and activity unchanged. Explicit
reconnect rejects a different user.
**Steps to reproduce**
1. Create a personal GitHub connection in a test company.
2. Archive that connection and its application.
3. As another authenticated member, start fresh personal setup with the
same name and provider.
4. Observe the owner-check failure before this change.
**Paperclip version or commit**
Base:
|
||
|
DottaandPaperclip
|
e2dc2cc0be |
fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893)
## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. **What happened?** A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. **Expected behavior** A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. **Steps to reproduce** Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. **Paperclip version or commit** Reproduced on |
||
|
DottaandPaperclip
|
d7283e8757 |
test(e2e): cover chat reload and fresh sessions (#18784)
## Thinking Path > - Paperclip keeps agent work attached to saved company records. > - Agent Chat keeps one conversation for each person and agent. > - Reload must preserve context without starting another turn. > - The ordinary `/new` command must create fresh provider context and preserve history. > - Existing local chat journeys cover these behaviors, but the deployment-targeted daily catalog does not. > - This PR adds one reusable journey with independent saved-state and rendered-reply checks. ## Linked Issues or Issue Description Refs: #2242, #18655. **What existing behavior does this improve?** Product E2E coverage for Agent Chat persistence and session reset on an attached deployment. **Current behavior** The deployment-targeted catalog has no bounded check for recall after reopening and a fresh session in the same saved conversation. **Proposed behavior** Remember a random phrase, reopen and reload the chat, then ask for the phrase without supplying it. Send `/new` through the ordinary composer. Require a new native session and a fresh-context answer while the original chat and history remain saved. ## What Changed - Add the `chat-continuity` flow and independent evidence grader. - Reuse the existing chat sender and run-bound native reply reader. - Require exact board-user message attribution, three successful native provider turns and one synthetic reset row. Count all four persisted run IDs. A reset with provider input or a native session cannot pass as synthetic work. - Use the original bound issue-run summary to prove reset completion when the detail result is null or compact. Preserve the raw detail and require its reset context, company, agent, issue and wake comment to agree. Native provider turns require committed native state and independent output evidence; an optional duplicate result field is not required. - Require an empty agent-filtered run history before the first send. Reused agents are unsupported for this case. Validate every agent-scoped activity row and reject mixed or truncated responses; do not filter global history into a pass. - Require the fresh native detail to contain an explicit null `sessionIdBefore`. Missing or empty admission evidence cannot prove a fresh provider session. Keep optional result blobs optional. - Check provider, model, environment, native session, finalization and reset generation. Reject stale replies, duplicate sends, extra runs, missing history and unexpected work. - Permit authenticated chat screenshots through the existing same-origin and mounted-Layout guard. Authentication and unrelated settings pages remain excluded. - Register the case, exact checks, source hash inputs, discovery and documentation. Feature-map files are unchanged. ## Verification - Normal merge commit `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` has exactly two real parents: the staging-qualified chat source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` and lifecycle squash `8771606fd87f69ed4057779e3dce3a9bca8088e6` from #18782. Thirteen dedicated case, support and reused runtime/oracle files are byte-for-byte preserved. The eight shared registry placements match the finite integration forecast. The nine required checks and four-run limit are unchanged. - The integrated source passes 99 focused tests across three files and 925 related Product E2E/catalog/native-reply tests across 42 files. Credential-free discovery lists 22 unique journeys, including chat and lifecycle exactly once. The PR diff against the actual lifecycle squash passes its whitespace check and remains the original 17 chat paths. - The current canonical `node scripts/typecheck-test-projects.mjs --low-memory tests/runner-e2e` passes, including Runner TypeScript and server declaration prerequisites. The DB migration checks and build pass against the actual squash's execution-admission schema and migration 0428. No Rust build was run locally. Canonical generated outputs are bound to current tracked inputs and were rechecked after tests. - The first direct typecheck found a stale generated Runner export, `taskDocumentKeySchema`; the current canonical prerequisites resolved it. A frozen offline install lacked a cached locked tarball, and an initial child command reached the global pnpm version switch and failed signature retrieval. Fetching only locked packages and pinning child pnpm to 9.15.4 corrected the setup. The original diagnostics remain retained; no tracked source was changed to bypass them. - The unchanged chat implementation passed a genuine staging campaign, workflow 41725181387, at source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1`: all nine checks, all five phases, three provider turns plus one synthetic reset, all four usage IDs, and closed cleanup. This evidence remains bound to that original source. The new integration head requires its own current gates and coordinator qualification decision; this PR does not enroll anything in the daily plan. - The earlier staging workflow 41701828968 remains failed. Its nine behavior checks passed, but private collection rejected the valid reset's capture of the old session and cleanup failed. The private adapter fix and evidence-first recovery were handled separately. The original failed grade and archive are preserved. Reset prior-session capture is distinct from the third native turn's explicit-null fresh admission. - Three missing/undefined/empty fresh-admission controls reproduced false passes on `247827a066af01fa583ff030df79b61cb2c87730`, then passed after the strict null fix. Four earlier controls reproduced compact-detail, mixed-activity, missing fresh-agent receipt and unscoped-activity problems. Their original failures remain retained. Positive and wrong outcomes still enforce bound reset summaries, committed native state, configured identity and independent rendered output. Board provisioning uses the acting-agent filter; it does not grandfather earlier work by the test agent. - The first broad support attempt passed 861 tests and failed one report-generator check. A separate reproduction showed `listen EPERM` for the tsx local IPC socket before report output; the unchanged support suite passed with normal local IPC permission. An early flow fixture also returned an object where the events API returns an array; the real event reader rejected it, and only that fixture was corrected. Both original failures remain recorded. - Original commit `d6226b30acaf5a2dad28030f5b295ac1dd119565` failed full CI [41653171240](https://github.com/paperclipai/paperclip/actions/runs/41653171240): a CLI child-server runtime-file wait and two 15-second attention-retention timeouts, followed by database setup/cleanup errors. Their exact causes remain unresolved; the failed jobs and logs were retained and not retried. A later pass does not explain those timeouts. - Strict-admission source `055a7fa99e7cb8aad12b85fccf23880f0dc98a3f` failed full CI [41669527762](https://github.com/paperclipai/paperclip/actions/runs/41669527762): private-invitation and pending-shell access fixtures. The retained assertions match the account-identity fixture repair in #18799, now included through master, together with #18798 and #18650. No old-head job was retried. Subsequent source `4c6cc9d67d2ae577e3a8ae7856bd6ae22a8489c1` passed full CI 41680411215 and current-source review at 5/5. - Current full CI [41743580793, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41743580793/attempts/2) completed successfully on unchanged `ddc37bb05b87cbc1fa449e4b2fc5a4677f0fe3b6` at 2026-11-04 00:34:52 UTC: 48 successful jobs and one intentional selector skip. Attempt 1 remains failed: three run-bound runners were independently confirmed to have lost Spot capacity, and the two aggregate failures reflected those interrupted lanes. The build and boot health smoke had passed before interruption. Exactly one failed-job recheck was used; no source change or further retry was made. Fresh exact-head Greptile review is 5/5 with no new actionable issues and both review threads resolved. The separate Ponytail assessment says Ship with no must-fix or should-fix item; its optional direct board-row grader control remains documented. Whole-workspace recursive typecheck, full tests and full build were covered by hosted verification and were not repeated locally; the bounded local checks above passed. These gates do not claim that a later master merge tree has been live-qualified. ## Risks The attached target must supply a fresh disposable native agent and durable pre-send ownership for its chat. Its private adapter must collect all four run IDs, distinguish the synthetic reset from the three provider turns, and stop and clean up owned resources after evidence is saved. This case verifies browser reload, reopen and explicit session reset. It does not verify controller restart. Unknown setup or provider cost remains unknown. No shared QA source agent, model setting, production behavior or budget policy is changed by this public PR. ## Model Used OpenAI GPT-6 through Codex with reasoning, code execution and repository tools. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
985b175e83 |
test(e2e): verify saved agent instructions on the current QA harness (#18891)
## Thinking Path > - Paperclip lets users configure agents and verify their work. > - Daily QA must exercise saved instructions on the QA staging instance. > - A fresh owned agent can test an instruction change without changing a shared agent. > - The test must prove that saved instructions caused the next run's output. > - It must also prove that the original instructions were restored. > - This pull request brings the qualified settings journey onto the current shared registry layout. > - The benefit is a reusable daily QA candidate with independent evidence and fresh current-source checks. ## Linked Issues or Issue Description Refs #18788. This is the replacement review PR for the same settings journey after integration with the merged lifecycle registry layout in #18782. **What happened?** The settings journey passed its original staging qualification at source898ae8ab4e49d2408688088ce82e3316b91cc09c. The earlier PR's review workspace remains retained for file preservation. That workspace and its recovery evidence remain intact. **Expected behavior** Review the integrated settings source through this PR's own normal CI and fresh reviews. The journey saves instructions through ordinary UI controls, verifies their effect on one current native run, and restores the original bytes. **Steps to reproduce** Use the documented dedicated agent-settings-persistence campaign. The provider must supply a fresh owned agent and the agent-settings-write capability. Save the instruction edit, reload it, complete the ordinary assigned task, verify its independently saved result and current-run attribution, then restore and verify the original instruction entry. ## What Changed - Add the settings browser journey, its independent evidence grader, fixtures, retained evidence contract and focused controls from #18788. - Require one fresh dedicated agent. Select its exact configured instruction entry through the file tree. Confirm both board saves through persisted source and audit evidence. - Require one committed native run on the configured provider, model and environment. Keep the synthetic instruction reference out of the task request. - Restore the original instruction bytes through the same ordinary UI and verify the saved result after reload. - Normally merge actual lifecycle squash8771606fd87f69ed4057779e3dce3a9bca8088e6 once. Place only settings registrations at the documented distinct anchors. Preserve all eight dedicated source/oracle/fixture/docs blobs from qualified898 exactly. ## Verification - Current integrated head: 330 focused checks pass across eight settings, catalog, contract, observation, result/report and operations test files. - Current E2E types pass: `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations tests/runner-e2e`. - Declaration reuse was bound to all11091 matching package/script/server/UI/root inputs, installed lock and TypeScript, plus all source and target generated-output hashes. No source or module-resolution links were copied. All copied output hashes remain unchanged after local checks. - Maintained product-journey discovery passes and includes both lifecycle and settings. The integrated catalog has22 entries, including the separate fresh-cloud onboarding case. - Static preservation checks pass for seven shared registry ASTs after only declared order changes. The settings dispatch body, required seven checks, one-run bound and all dedicated implementation/oracle blobs remain intact. - Historical staging evidence: [QA workflow41689186453](https://github.com/paperclipai/paperclip-cloud/actions/runs/41689186453) passed all seven settings checks and all five phases at source898ae8ab4e49d2408688088ce82e3316b91cc09c. This is the original qualification receipt. It is not a new run of this integrated head. - This PR must earn its own current-head full CI, fresh5/5 review, resolved threads and terminal passing normal check contexts. Those checks are pending. No prior review or check status was copied. - Full workspace checks and a new live QA/model run were not run locally. The coordinator owns current-source acceptance and daily-plan admission. ## Risks - The journey has one paid provider run and needs a fresh dedicated agent. It rejects a campaign that shares this agent with another paid journey. - Browser saves and restoration must persist before navigation. Wrong, missing, foreign or incomplete evidence fails the independent grader. Failed or active work is preserved for normal owned cleanup. - Registration and hash-input order changed during integration, so the definition/source identity changes. The historical QA receipt remains bound to its original source. - The retained earlier review workspace is still held for file preservation. This PR does not recover, reset, replace or destroy it, and submits no turn to it. - No product runtime, permission, model, shared agent, configuration, feature map, QA pin or TestOps-plan changes are included. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. The host does not expose the exact deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub issue/PR references; historical QA workflow is linked as qualification evidence) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7b1c53f8f1 |
fix(recovery): continue verified preparation after lease interruption (#18892)
Use recorded system interruption and the existing execution, controller, stopped-lease, and saved-file proof instead of a shutdown-only error label. Preserve bounded retries, user Stops, task authority, and exactly one continuation. Keep the chat recovery fixture faithful to the preparation stage and terminal time written by the real lifecycle. Retain missing-field safety controls. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
82dd3bb702 |
Preserve unstarted queued turns during offline enrollment (#18887)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin FoleyandPaperclip
|
4e29f00139 |
fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. **What happened?** A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. **Expected behavior** A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. **Steps to reproduce** 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. **Paperclip version or commit** `master` at `b0c4e345267`. **Deployment mode** Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - **Silent stall.** The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - **Candidate admission** carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - **The agent that loads the conversation** stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - **In-flight provider work** is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - **Launch shape.** A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - **Artifacts** are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press **Retry** once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f774992832 |
perf(ci): admit release candidates and isolate native image capacity (#18869)
Admit full source verification only for exact-source Docker or npm release work, while superseded pending sources complete metadata-only admission. Restart only strictly deferred runs; failed or cancelled checks require an explicit request. Route canonical master native images to dedicated 32-core Spot profiles and keep source checks on protected verification capacity. Use the four-slot GitHub-hosted publication pool for registry work and signing, with proof waits outside that pool and a fresh source-proof check before tagging. Trim historical checkout blobs and prepared-runner disk cleanup while preserving tag and fork publication behavior. Validation: release/source and source/artifact regressions, 665 workflow contracts, actionlint, complete latest-head PR CI, and Greptile 5/5 with all comments resolved. Existing CLI and large-file repository timing tests also passed independently; their CI shards received one scoped retry. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ab2d2098a2 |
test: verify sleep producer readiness and CI contracts (#18795)
Reviewed and verified at
|
||
|
|
ce332ecec0 |
fix(execution-admission): keep admitted project sets after settlement (#18871)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - #18746 added execution-admission storage: one admission row per attempt and one charge row per admitted project. > - Settlement deletes every charge row. The admission row keeps only the primary project and an opaque context revision. > - Thus, after settlement, the complete admitted project set is lost. > - A retained provider session still carries the references of its previous admission. Example: a task mentions A and B, the admission settles, and then a user removes B from the task text. The next admission must still count B, or start a fresh context. The primary project or the context token cannot supply B. > - This pull request keeps the complete admitted set on the admission row after settlement. Charge rows stay the only active capacity. > - The benefit is that later retained-session resolution can read the previous set. Settled history does not hold capacity or block project deletion. ## Linked Issues or Issue Description Refs #18746 (corrects its storage). Related: #2831, #14333. **Subsystem affected** Server execution admission (`server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** `settleAdmission` deletes all charges of an admission. No other storage records the complete project set. After settlement, the set cannot be recovered. Current task text is not a replacement, because it can change after admission. **Proposed solution** Add `execution_admissions.admitted_project_ids uuid[]`. It holds the sorted, distinct set. It is written once at admission and a trigger blocks changes. It has no project foreign key. `{}` is a known empty set. Null means unknown. Only settled rows can have null. Migration `0428` fills the set of unsettled `0426` rows from their charges, because those charges are still complete. Settled `0426` rows stay unknown. **Alternatives considered** - Keep charge rows after settlement with a `released_at` column and a partial active index. The project foreign key would then block project deletion permanently, or the key must be removed from active charges. One table would also hold two different facts. - A separate history table with one row per project. This is more rows, joins and constraints for data that is only read as a whole set by admission ID. - Encode the set in `context_revision`. That token is opaque and owned by the context owner. The database cannot validate it. - Use `heartbeat_runs.context_snapshot`. It is mutable and is per run, not per admission attempt. ## What Changed - Schema and migration `0428_retained_admitted_project_sets`: add `admitted_project_ids`, a check (null only when settled, no null elements, at most 100 entries) and an immutability trigger. The backfill fills unsettled rows from their charges and leaves settled rows null. The migration is safe to run again. Migration `0426` is not changed. - `recordAdmission` writes the set. A replay compares against the stored set. `settleAdmission` still deletes the whole charge set in one transaction and does not change the stored set. - Add `readAdmittedProjects` (company-scoped, by admission ID). It returns `known` with the set, or `unknown`. - Add a pure `retainedSessionProjects` rule: a retained session uses previous ∪ current. An unknown previous set returns `fresh_context_required`. No consumer uses it yet; C1b integrates retained sessions. - Update `doc/DATABASE.md` and `server/src/modules/README.md`. Not changed: no public cap writes, no claim or dispatch integration, no session integration, no provider activation, no `serialize` → cap 1 migration. Active counts still use only the indexed charge rows. ## Verification Red, on unchanged `master` code, with the new tests: - `server`: `npx vitest run src/modules/execution-admission/adapters/postgres.test.ts`. The new tests fail. With a temporary reader that uses the only storage `master` has (charges), the settled A+B admission reads `[]` instead of `[A, B]`. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts` fails, because no `0428` migration exists. Green, with this change: - `server`: `npx vitest run src/modules/execution-admission`: 27 passed. The new PostgreSQL test covers: A+B admission → settlement → set still readable, charges and counts zero, settlement replay keeps the set, a second attempt with a different set, primaryless and empty sets, company isolation, rollback, immutability, the check constraint, and deletion of a project that has only settled history (its ID stays in the set). It also checks that a project with an active charge still cannot be deleted. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts`: 2 passed. The upgrade test starts from the shipped `0426` schema with an unsettled A+B row, an unsettled empty row and a settled row. After the upgrade the sets are `[A, B]`, `{}` and null. A second run of `0428` changes nothing. - `pnpm --filter @paperclipai/db typecheck` and `check:migrations` passed. `node scripts/check-module-boundaries.mjs` passed. - Not run locally: the server layered typecheck. The local process was killed for memory (a 5.8 GiB limit shared with other jobs). CI runs it. This PR does not prove C2 claim races or C3 lifecycle authority. ## Risks - Migration: one `ADD COLUMN`, an `UPDATE` that only touches unsettled rows (none in production, because nothing writes admissions yet), a check and a trigger. It does not delete or rewrite data. - Settled rows created before `0428` stay unknown. A future consumer must start a fresh context for them. It must not treat them as empty or primary-only. - The stored set has no project foreign key. Company scope is checked at admission time, under the same project locks and the charge foreign keys. Size: about 70 production lines (schema 8, migration SQL 22, adapter 28, domain 15), 105 test lines and 24 doc lines. Generated metadata (`0428_snapshot.json`, `_journal.json`, removal of `0422_snapshot.json` by the standard pruning) is separate. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
11ac90e31f |
fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. **What happened?** No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. **Expected behavior** The server typecheck checks these files, and a type error fails the check. **Steps to reproduce** 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. **Paperclip version or commit** I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. **Deployment mode** Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. | Step | Total errors | Change | Cause | | --- | --- | --- | --- | | Config as committed | 6,536 | | | | Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request types were missing. | | Set `rootDir` to the repository root | 2,436 | -2,011 | Config. | | Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest accepts `.ts` imports. | | Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has `Promise.withResolvers`. | | Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8 `TS7016` errors and shows new real errors. | | Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing. | | Count only files in `server/src` | 2,058 | -84 | Other packages: 53 Daytona, 29 Runner test, 2 UI. | So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. | Kind of real error | Errors | | --- | --- | | Wrong argument or assignment type | 1,023 | | Stale or partial fixture objects | 525 | | Property that the type does not have | 317 | | Missing narrowing of null or unknown | 101 | | Implicit any | 60 | | Missing name or module | 16 | | Other | 16 | ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - **A test that never ran.** `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - **Assertions that could not fail.** - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - **Postgres processes that did not stop.** 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - **Arguments that the service ignored.** - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - **Problems in tests that master added during this work.** - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: | Check | Commit | Result | | --- | --- | --- | | Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers | `6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB | | `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e` | Exit 0, 0 errors, 42 seconds, peak 3.25 GiB | | The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0 errors | | `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` | `01db5634e4` | 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. | | `tsconfig.declarations.json` after a full server typecheck | `0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. | | Owned runtime test in `idle-completed-history` | `e8f2f448ac` | Passed, with the fix that master later replaced | | `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test | `6e45a08573` | 30 passed | | Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed | | The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. | The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. **Runtime check of the fixes.** Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: | Result | Tests | Detail | | --- | --- | --- | | Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. | | Failed | 7 | The same 7 tests also fail on the commit before the fixes. | | Skipped | 9 | These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. | The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. **Audit of the diff.** I counted these patterns in added and removed lines, to find fixes that only hide an error: | Pattern | Added | Removed | | --- | --- | --- | | `@ts-ignore` | 0 | 0 | | `@ts-expect-error` | 0 | 1 | | `.skip` | 0 | 0 | | `as any` | 9 | 9 | | `: any` | 11 | 2 | | `as never` | 8 | 2 | | `as unknown as` | 35 | 4 | | `Promise<unknown>` | 16 | 5 | | `expect(` | 128 | 113 | No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. **Accuracy of the check.** One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - **A large diff in test files.** 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - **A higher cost for changes to source types.** A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - **Changed test behavior.** Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - **The Daytona stub can go stale.** The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - **The release path.** `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - **The tsconfig change.** The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
f1fb1bcb8a |
fix(ui): let Done close completed Slack setup (#18878)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Slack connections let people talk to their assigned agents. > - Setup saves a completed state after the connection is ready. > - A saved setup link can still open Success after completion. > - Done submits completion again, which the server rejects. > - This pull request opens settings when the Slack setup is already complete. > - Users can close the saved setup without changing the bot's lifecycle. ## Linked Issues or Issue Description **What happened?** Reopening a completed Slack setup shows Success. Done then shows “Couldn't finish setup: This connection is not waiting for a setup test.” The user stays in the wizard. **Expected behavior** Done opens the saved connection settings. A paused bot stays paused. A setup that is still awaiting completion must use the existing verification request. **Steps to reproduce** 1. Finish a Slack connection. 2. Reopen its saved chat/connect link with provider=slack and its resume parameter. 3. Click Done on Success. 4. Observe the completion error instead of the settings page. **Paperclip version or commit** Reproduced live on Core |
||
|
DottaandPaperclip
|
dcc0878b32 |
perf(agents): save healthy account settings without a runtime probe (#18690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The agent runtime page saves account, model, and environment settings. > - The save handler starts a computer and runs a live account test before it writes the edit. > - A slow computer or provider can make a healthy account save take tens of seconds. > - Saved account health, access, installation, and compatibility can be checked without a runtime test. > - This pull request uses those checks for ordinary configuration saves. > - Legacy adoption, task adoption, and Copilot model checks keep live verification. > - The benefit is a fast save while Test and execution retain runtime checks. ## Linked Issues or Issue Description **What happened?** Saving a new AI account selection on an existing agent can take 20–30 seconds. Model edits can take the same path. The configuration PATCH acquires a computer, prepares account credentials, runs a provider/runtime probe, and releases the computer before it saves. **Expected behavior** A healthy installed account can be saved after current access and compatibility checks. A save confirms persistence. Test verifies the runtime. An unverified legacy account must still pass live adoption checks. **Steps to reproduce** Run the PostgreSQL regression with a 20-second controlled delay at computer allocation. On the original route, a successful save takes 20,191 ms and calls allocation and the runtime probe once. With the first patch, the same fixture takes 51 ms; with the environment-guard correction, it takes 37 ms and calls neither operation. Each timing has one sample. This is a local fault experiment with real HTTP and PostgreSQL. It is not a hosted latency measurement. Related: #17760 addresses binding comparison and prospective probe environment resolution. This change deliberately removes the live probe from ordinary healthy-account saves. It keeps live checks for legacy and task adoption. #17965 fixes the result of an explicit managed Runner test. Neither change removes this save-time wait. #18687 optimizes a different Apps configuration read. ## What Changed - Replace the internal probe boolean with explicit none, legacy, and always modes. - Use legacy mode for ordinary configuration saves. Keep current account health, access, installation, compatibility, environment, audit, and revision checks. - Keep the environment company check before the live-probe branch. Keep the live task-adoption transaction and Copilot model checks. - Add twelve real PostgreSQL save-route regressions, including company-bound environment checks. Keep the existing inherited-sandbox adoption fixture for legacy and task adoption. - Cover healthy and legacy settings changes separately. Keep provider rejection and reconnect tests on the legacy save path. - Update the specification and feature/access maps to distinguish saving from runtime verification. The configuration route owns this policy. The account service still owns account access and health. The existing environment test service still owns live probes. No new endpoint, migration, dependency, background job, or provider protocol is added. ## Verification Base: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`. - Original route with initial regressions: three failures and two passes. Healthy saves reached computer allocation. The controlled 20-second baseline saved successfully but failed the no-allocation assertion. - Initial save/adoption/Copilot selection: 21 passed. The name filter excluded 141 unrelated tests. The seven new cases cover persistence, revision and activity records, model edits, roster edits and stale retries, missing installation, revoked grants, legacy verification, and controlled allocation latency. - Explicit Test-route suite: 55 passed. The first attempt reached the 30-second import hook limit. The same suite passed with a 90-second hook limit. - Saved harness preflight suite: six passed. Account revocation, compatibility, probe failure and target release checks remain. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Review revision: both full save/account suites passed, 96 tests total. The controlled save took 37.08 ms with zero computer allocations and zero provider probes. The five new environment checks passed after the three foreign-environment cases failed against the first patch. - Local full `pnpm -r typecheck`: unavailable because cargo is missing. - Local full `pnpm build`: unavailable at the verified Copilot runtime prerequisite. Server typecheck preparation also reaches missing cargo. - Local full `pnpm test:run`: stopped after concurrent broad checks exceeded the container memory limit. No full-suite pass is claimed. Direct server TypeScript checking and its bounded retry were also killed by the container memory limit. The first remote Typecheck + Release Registry check passed; the next head still requires fresh CI. Reproduce the controlled timing: ```sh cd server PAPERCLIP_CONFIG_SAVE_PROFILE_DELAY_MS=20000 pnpm exec vitest run src/__tests__/agent-configuration-save.test.ts --hookTimeout 90000 ``` The fixture keeps real Express, authorization, account selection, PostgreSQL persistence, revisions and audit writes. Only remote allocation and the provider probe are controlled. The default tests do not contact a provider. Successful configuration saves no longer depend on either controlled operation. A later explicit Test or task must still verify the actual runtime. Review follow-up: the first Greptile review found that the new healthy-account path skipped the environment company guard. The corrected route checks the effective environment before it decides whether to probe. Five real PostgreSQL cases cover foreign, same-company and global environments, inherited defaults, and an existing account binding. Against the first patch, the three foreign-environment cases returned 200 and failed their expected 403 assertions; the two allowed cases passed. Initial fixture attempts hit unique environment constraints; the fixture now uses distinct sandbox rows and names. The first CI server shard found 11 tests that required a live probe for every save. Those expectations are changed to the documented healthy/legacy split, with additional healthy-selection cases. The build and Runner jobs were cancelled after this test failure. Their logs show no separate compile or test verdict. No cancelled check is counted as a pass. Latest commit `6257e9e67f289e1c650a8b4c2b818fcca37e2800`: [CI run 41542980091](https://github.com/paperclipai/paperclip/actions/runs/41542980091) passed typecheck, build, all test shards, Runner checks, and E2E. All 55 check runs are complete: 52 passed and 3 were skipped by their workflow conditions. The commit status check also passed. [Greptile review](https://github.com/paperclipai/paperclip/pull/18690#issuecomment-6590555746) is 5/5 for this commit with no new findings. The earlier environment ownership thread is resolved. These remote results complete the checks that the local tool and memory limits prevented. The earlier local failures remain recorded above. ## Risks A healthy saved account can become invalid at the provider before the next runtime use. Ordinary saves no longer detect that through a live test. They still reject unhealthy, revoked, incompatible, or uninstalled accounts in current storage. Explicit Test and execution retain provider authentication and project-authentication checks. Legacy-account adoption, task connection adoption, and Copilot account-specific model checks keep their existing live validation. There is no migration. Reverting the route change restores the earlier synchronous probe behavior. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. This runtime does not expose the exact model ID, context-window size, or reasoning setting. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full-suite limits are above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
049c1eacf5 |
Preserve historical native logs during ephemeral enrollment (#18864)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers need retained run history in shared storage.
> - Some old native runs have surviving local logs but no completion
digest or verifiable native source coverage.
> - Enrollment must preserve those bytes without calling them a complete
native transcript.
> - This pull request adds a read-only archive path to offline
enrollment.
> - Users can still read old output after the original server filesystem
is removed.
## Linked Issues or Issue Description
**What happened?**
Offline enrollment blocks on interrupted runs and unfinalized historical
native logs. The current importer requires canonical native-event
coverage even when the original local file survives.
**Expected behavior**
Preserve the exact surviving file in shared storage. Keep the original
run status, completion metadata, and source events. Do not allow writers
or native repair to modify archived history.
**Steps to reproduce**
Use an S3-configured instance with a terminal native run whose local log
lacks a final size or digest. Stop all app deployments and run ephemeral
maintenance. Before this change, interrupted status or unproven native
coverage prevents enrollment.
Related preservation work: #18826, #18835, #18841. This is a bounded
migration fix, not a new live logging protocol.
## What Changed
- The run-evidence migration owner can archive unfinalized native local
files during offline enrollment.
- Require the offline migration environment, zero active or queued runs
across the instance, and expired controller/finalizer leases. Check
global quiescence at each page boundary. Recheck each run's ownership
and metadata before publication.
- Verify stable local file identity and metadata, any recorded
size/digest, uploaded prefixes, and remote SHA-256.
- Store raw history in a separate `<run>.archive` journal with source
cursor zero and backend `s3_archive`. Leave completion fields and native
events unchanged.
- The existing log store reads archives but rejects begin, append,
enqueue, flush, and finalize. Native source repair does not run for
archives.
- Keep the default background importer's coverage checks. Continue to
reject incomplete mirrors and corrupt files.
- Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a
pre-existing large-chunk fixture: the current verifier can prove exact
compactor output, so the negative case now uses a different canonical
source.
## Verification
- Passed on final head
|