1626 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	97c4dc969e	test(evals): retire duplicate chat TestOps plans (#18625) ... ## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 11:07:32 -06:00
DottaandPaperclip	3f7811eef0	fix(runner): keep requests responsive during retired backup cleanup (#18619) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native runs keep durable harness backups for continuation and recovery. > - Publishing a new backup retires earlier backup directories in the server process. > - The synchronous permission walk and recursive removal block unrelated requests. > - This pull request makes that cleanup asynchronous and keeps publication waiting for completion. > - The benefit is continued request progress while large retired backups are removed. ## Linked Issues or Issue Description **What happened?** Publishing a backup with many small files blocks the Node event loop during directory permission repair and removal. A local comparison with 20,000 retired files reproduced 1.37–1.95 second event-loop stalls. This is a pause candidate. A short hosted profile has not established it as the dominant source of sustained CPU use. **Expected behavior** Unrelated requests can progress during cleanup. Publication still waits for complete removal, preserves rollback on a failed lease stamp, and leaves live files and symlink targets unchanged. **Steps to reproduce** Run the benchmark in `doc/performance/native-backup-publication.md`. It compiles the actual publication service from a pinned baseline and the working tree, prepares immutable retired backups, and sends foreground HTTP probes from a separate process. **Paperclip version or commit** Measured baseline: `774ddc72713d6aa7cb516691c71f1fc23bcfa5ff`. The measured treatment source is committed in this branch. Current head: `0a69cf2d43c2f9fcf032d96c1dceb56163bf21ba`. The runtime patch is unchanged from the measured source; later commits only strengthen the test and benchmark evidence paths. **Deployment mode** Local compiled service on macOS arm64 with Node v24.22.2. No hosted rollout has occurred. Related work: #18380 changed retention scheduling. #13676 included the existing immutable-backup cleanup behavior. Neither change makes publication cleanup asynchronous. The GitHub duplicate search also found #11929, which concerns database dumps rather than native harness backups. No matching open event-loop cleanup issue was found. ## What Changed - Use asynchronous filesystem operations for retired-backup cleanup. Visit directories serially and skip unnecessary metadata calls for file entries. - Await cleanup before publication proceeds, during rollback, after successful rotation, and in executor finalization. - Exercise immutable current and previous backups in the publication test. Preserve the existing symlink-isolation and rollback tests. Add a real separate-process HTTP probe that must complete while publication is still active. - Add a repeatable service benchmark and document its fixture, metrics, and acceptance limits. ## Verification Draft status: focused correctness tests, the local comparison, and repository typecheck pass. The complete local test gate is incomplete. The build passed. Complete CI, hosted acceptance, and final-head review remain pending. - Final focused Vitest run: 2 files, 4 tests passed in 649 ms with the normal server configuration. The regression warms a separate Node client, waits for its HTTP request during the stamp callback, queues the response for the next event-loop turn, and then allows final retired-tree removal to start. The async treatment serves that response before publication completes. The original synchronous cleanup fails the same final regression in a separate source fixture. That expected control failure is retained. - Syntax and `git diff --check` passed. - `pnpm -r typecheck` passed on head `332c57b88c` in 152.18 seconds, including the native Runner build. Validation uses Node v24.22.2 and the repository-pinned pnpm v9.15.4 with isolated dependencies. - `pnpm test:run` was intentionally stopped after 1,390.24 seconds (exit 143). All six chat shards passed, covering all 1,817 parameterized cases across their exact partitions. The general server phase was advancing but incomplete. The repository timing manifest estimates another roughly two hours for the serial remainder, so validation moved to final-head parallel CI. This is not a full local test pass. The command began on `332c57b88c`; later changes only refine the regression and benchmark, and the final regression has its own focused pass. - Benchmark failure/metadata checks pass in a tiny isolated Git fixture: an injected worker exit 17 is reported in 0.44 seconds without hanging, and the result includes a staged source change when the unstaged diff is empty. - `pnpm build` passed in 41.80 seconds on `9f7e4e669b`. The final commit only releases a pending test probe on failure; runtime and build sources are unchanged. An injected client exit verifies that final test failure path exits promptly in 0.97 seconds instead of hanging. - Three alternating baseline/treatment pairs used fresh fixtures with 100 directories × 100 files in each of two retired backups. Each publication removed 20,000 files. A separate process sent HTTP probes at fixed 5 ms intervals, including overdue arrivals. - Every run verified the replacement manifest, complete cleanup, and unchanged symlink-target contents and mode. All 2,395 HTTP requests completed without errors. The largest measured generator delay was 8.8 ms. | Metric | Baseline median (range), ms | Treatment median (range), ms | | --- | ---: | ---: | | Foreground request p95 | 1,357.7 (1,287.4–2,014.0) | 1.8 (1.7–25.9) | | Maximum event-loop delay | 1,439.7 (1,371.5–1,954.5) | 19.9 (17.4–40.7) | | Publication completion | 1,429.1 (1,360.2–1,952.9) | 1,439.5 (897.9–1,649.6) | | Process CPU during publication | 1,194.1 (1,041.8–1,225.5) | 2,423.1 (1,995.7–2,514.7) | The treatment served 180–329 requests while publication was active. The baseline served zero during that interval. Process CPU includes foreground work that the baseline cannot serve until cleanup completes. The result supports responsiveness, with a CPU tradeoff. It does not establish lower total CPU or a hosted latency gain. Reproduce with Node v24.22.2: ```sh pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts node server/scripts/benchmark-native-backup-publication.mjs --baseline-ref 774ddc7271 --output /tmp/native-backup-publication.json ``` The benchmark records the baseline SHA, treatment HEAD and source diff. Raw local evidence remains outside the repository. A first small smoke run hit a sandbox restriction on loopback listeners; the authorized retry resolved it. The smoke also exposed load-generator interval drift. Fixed-clock arrivals replaced that generator before the reported measurements. Pending gates: complete current-head CI, final-head Greptile review, and the real run/checkpoint/recovery journey. The full local test run remains explicitly incomplete. The four concrete review findings were fixed. The prior head received 5/5; final-head review remains pending. Local evidence remains outside Git: `/private/tmp/backup-publication-measurement-20261102.json`, `/private/tmp/native-backup-focused-final.log`, `/private/tmp/native-backup-typecheck.log`, `/private/tmp/native-backup-test.log`, `/private/tmp/native-backup-test-interruption.json`, `/private/tmp/native-backup-build.log`, and `/private/tmp/native-backup-harness-check.json`. The harness-check receipt points to its isolated metadata and worker-failure logs. The final client-failure and baseline-control logs are `/private/tmp/backup-benchmark-check-gprcfiiz/failed-client-cleanup.log` and `/private/tmp/backup-benchmark-check-gprcfiiz/final-probe-baseline-control.log`. These local paths are retained for the author; the commands above let reviewers reproduce the synthetic proof. ## Risks - Asynchronous removal adds filesystem-pool work and used more process CPU in the fixture. Check resource pressure as well as responsiveness before rollout. - Filesystem operations can now interleave with unrelated work. The existing session owner still controls publication. Cleanup remains awaited; no deletion is detached and no scheduler is added. - Errors must still fail publication or restore the prior backup as before. Corruption checks and destructive lease authorization are unchanged. - This synthetic macOS fixture does not represent every Linux volume or backup shape. It does not qualify a deployed fix. - Rollout acceptance should confirm the serving SHA, checkpoint completion, continuation/recovery, error rate, and loop delay under matched backup load. Revert if cleanup or recovery fails. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The exact served model ID and context-window size were not exposed to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 17:00:42 +00:00
DottaandPaperclip	0748b90062	fix(server): stop goal-less native runs before board task closure (#18616) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task owns its active run and its tool access. > - A board user can close a task while that run is still working. > - The task update route stopped native runs only when they had an active session goal. > - Providers without goals kept running after the route cleared task ownership. > - This pull request stops those native runs before a board closure releases ownership. > - The benefit is an orderly stop instead of tool denials and failed finalization. ## Linked Issues or Issue Description **What happened?** A board user marked a task Done while its native OpenCode run was active. The route cleared its execution lock without stopping the provider. Later tool calls failed with `paperclip_runner_tool_binding_not_authorized`. The final wake fence then failed with `native_current_wake_comments_binding_changed`, leaving workspace finalization in retry. Cancelled also released ownership before its later Stop call. **Expected behavior** A board closure must confirm native Stop before it releases task ownership. A failed Stop must leave the task open. The provider does not need to support session goals. **Steps to reproduce** 1. Start a native task with a provider that has no active session goal. 2. Mark the task Done while its run is active. 3. Observe that the old route updates the task without cancelling the run. ## What Changed - Reuse the existing audited cancellation path before a board user marks an active native task Done or Cancelled. - Keep the existing 409 response when Stop is unconfirmed. - Record the board user in the Stop metadata. The existing acknowledged-Stop policy then leaves queued input and recovery quiet until a later explicit wake. - Inspect additional runs only when a task has an execution owner; tasks without a run lock keep their existing completion path. - Keep native semantic completion, legacy completion, and goal control behavior. - Add route regressions for stop ordering, failed Stop, and legacy completion. - Update the execution contract and task maps. ## Verification - Red before the fix: three regressions failed. Done had no Stop; Cancelled stopped after the update; an unconfirmed Stop allowed Done. - A second red proved that a Stop receipt without the route's operator metadata was not recognized by the real recovery policy. Both cases pass after the metadata repair. - Final local result on macOS arm64 / Node.js 26.4.0: **348 tests pass across eight suites**. These cover the route, native self-completion authority, cancellation, acknowledged Stop, recovery queues, dependency wakeups, and completion telemetry. - CI found seven sibling route failures from an unnecessary active-run lookup on tasks without execution ownership. Those cases failed locally before the correction and pass afterward. - Earlier concurrent local runs hit a module-load hook timeout and one watchdog socket hang-up. The isolated watchdog case and the complete final eight-suite run pass with a 90-second startup budget. No assertion was removed. - `node scripts/check-maps.mjs` and `git diff --check`: pass. - `pnpm build` and `pnpm -r typecheck`: pass. Server build and typecheck also pass after the Stop metadata change; the final server build includes the lookup correction. - `pnpm test:run`: attempted; the unchanged chat shard selector failed before executing its selected tests (`No test found ... in lines ...`). This is separate from the PostgreSQL startup failure. CI supplies full-suite verification. - PostgreSQL integration and live-provider acceptance did not run locally. Embedded PostgreSQL failed before app startup with `could not create semaphores: No space left on device`. No usable isolated local database runtime was available. CI supplies the PostgreSQL regression environment. The full live OpenRouter workflow remains unverified. - The first final-head CI pass hit a 15-second timeout in the unchanged `attention-retention-work.test.ts` blocker-history fixture. Cleanup then deadlocked and later inserts reused IDs. This service test does not call the changed route. The original log is [retained here](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289220). The one allowed rerun also failed on the same commit: the `running` case exceeded 15 seconds again ([rerun log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125393915025)). The rerun otherwise passed 2,346 tests. No retention source or assertion was changed. This repeatable failure is the remaining CI gate; the PR is **not ready to merge**. The next step is to isolate that retention fixture/query in a usable PostgreSQL environment and repair its cause before rerunning CI. - Greptile reviewed final commit `b3cb41af2d911e9ced219e846cada3ecec835dbf` at **5/5**, with no unresolved review threads. Greptile is green. CI is blocked by the repeated retention test timeout described above. - CI native cancellation evidence: all 16 tests in `server/src/__tests__/native-cancellation-request.integration.test.ts` pass ([job log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289222)). - CI PostgreSQL wake-queue evidence: all 96 tests in `server/src/modules/wake-queue/adapters/postgres.test.ts` pass ([job log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289470)). ## Risks Board closure now waits for native Stop even when the provider has no goal. A failed or uncertain Stop keeps the task open. The change uses the existing cancellation and workspace-preservation path. It adds no retry loop, credential access, schema change, or publication authority. Direct service updates and agent semantic completion retain their existing paths. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, shell execution, and browser inspection. The exact serving snapshot ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 10:41:44 -06:00
DottaandPaperclip	412834a860	test: overlap long recovery and maintenance waits (#18544) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI verifies recovery, execution authority, and background work before changes merge. > - Diff selection saves work, but a selected file still runs its cases in sequence. > - The three longest files in a recent full master run each took over eight minutes. > - Many of these cases must observe real lease expiry or idle periods. > - This pull request gives independent cases separate fixtures so those periods can overlap. > - The benefit is a shorter test path with the same assertions and observation windows. ## Linked Issues or Issue Description Refs #18464. The latest measured successful master [Cloud readiness run](https://github.com/paperclipai/paperclip/actions/runs/41300933157) spent 581 seconds in issue recovery, 569 seconds in native restart recovery, and 512 seconds in maintenance server routes. File-level selection cannot divide those long serial files. Shortening their lease or idle windows would weaken the regressions they protect. ## What Changed - Split issue recovery into four files with separate database and worker fixtures. Keep all 18 cases. - Split native restart coverage into recovery, lease renewal, preparation, checkpoint cost, and disk-loss files. Keep all original 79 cases, including the existing container-qualified cases. The master update also retains its newly added recovery cases. - Close each native case's database clients after pending failure reports drain. The shortened file must not rely on idle timeouts to stay below PostgreSQL's connection cap. - Run the native Runner lane with two isolated fork workers. Register every new file in the lane and verify that the partition still includes every suite. - Give each maintenance case an owned database. Run at most two cases together with test-local assertion contexts. Keep both real 90-second quiet windows and all 17 cases. - Preserve serial execution when any explicit shared `PAPERCLIP_*_TEST_DATABASE_URL` is present. An admin URL that creates a separate database per fixture can run concurrently. - Refresh shard weights from the protected master run. Mark the initial split-file weights as conservative estimates until master measures them. - Anchor the attention-retention fake clock to PostgreSQL time. Its fixed clock started trailing real database timestamps after 12:00 UTC on November 1, causing an archive-deadline test to time out. Keep its deadline assertions and timeout unchanged. - Document the fixture and concurrency rules. Leave runner authorization and workflows unchanged. ## Verification - Final head `c9a4ff11b3ba81277341294c7721b34fb8705ede`: [full PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/41477848686), including typecheck, build, all test shards, E2E, and the final verification gate. The rollup has 55 successful checks, one neutral review check, four expected skips, and no pending or failed checks. Greptile reviewed this exact head at 5/5 with no actionable findings. - November 2 master update: merge commit `c9a4ff11b3ba81277341294c7721b34fb8705ede` resolves the native recovery import conflict while retaining the new busy-row and remote continuity cases from master. All 38 test-runner checks and the three new busy-row cases passed locally. - Retention verification after the master update: the first local full run passed 52 cases and timed out at 15 seconds in the paged-backlog case. That case passed in isolation. The unchanged master suite passed all 53 cases (82.95 seconds), and a full PR rerun also passed all 53 (76.65 seconds). The initial timeout cause remains unproven; no timeout or assertion was weakened. - Previously verified head `fd288a8e8c88df54dcdca7b7e42366b019182782`: [full PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/41312625439). There are 55 successful checks, one neutral review check, four expected skips, and no pending or failed checks. Greptile reviewed this head at 5/5 with no actionable findings. - Issue recovery: 18/18 passed before and after the split on the same local machine and PostgreSQL fixture path. Wall time changed from **630.6 seconds to 319.7 seconds** (49% less). All case titles match. - Maintenance also passed all 17 cases on AWS: structured file cost dropped from **511.8 to 282.6 seconds** across the linked master run and the first PR run. These are different CI runs, so the matched local comparison below is the stronger isolated measurement. - Maintenance: 17/17 passed before and after with the restored pinned dependencies. Wall time changed from **488.1 seconds to 317.2 seconds** (35% less). All 120 assertion lines remain. Both real 90-second windows also passed a final targeted run. - Native restart: 71 passed and eight existing container-only skips before and after the split. Wall time changed from **632.1 seconds to 346.8 seconds** (45% less). All 79 cases and 254 assertion lines remain. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Extra TypeScript checking of the moved recovery and maintenance tests passed. The standard server config excludes `src/__tests__`. A separate check of attention-retention fixtures reports existing type errors; the unchanged baseline has identical diagnostics, with none introduced by the clock fix. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection*.test.mjs`: 61 passed. - `pnpm test:run`: stopped in the second chat partition: 335 passed and four failed (two GitHub retry-state assertions and two Teams timeouts), after 339 passed in the first partition. These chat tests are unchanged. All four pass in a focused run (19.1 seconds). The full-suite ordering/load cause remains unproven. All three CI chat jobs passed. The previously failing cases passed there in 1.3, 1.3, 2.4, and 2.2 seconds; see [the chat job log](https://github.com/paperclipai/paperclip/actions/runs/41309691650/job/124857580714). The attention-retention timeout repeated in CI and in an isolated local run. Its fake clock was fixed at November 1 noon, which had fallen behind real PostgreSQL timestamps. Anchoring it to the database clock passes all 53 cases; the failed case now takes 425 ms. - The native CI lane passed all 162 runnable cases with eight existing skips after client cleanup: **650.7 seconds on the earlier master run to 391.0 seconds on this PR**. - CI found native connection exhaustion after the long waits moved out of the main file. The same failure reproduced locally at a 100-connection cap. Per-case client cleanup fixes the shortened file at that cap, and all five native files now pass together at that cap (71 passed, eight existing skips). - Early attempts exposed a missed fixture reference and a missed reporting setup hook during extraction; both were restored. The first CI policy run also required moving the new native fixture under test helpers; that boundary check now passes locally. A separate auxiliary command picked up the host pnpm version and removed dependencies. The original maintenance baseline reproduced the same missing-package failures, so that failed attempt is excluded from timing comparisons. A later unchanged-baseline run passed all 17 cases. The pinned install was restored before the passing maintenance run. The audit also covered heartbeat process recovery, deferred receipts, chat, workspace recovery, attention, and connection management. Their shared fake clocks/provider mocks or existing partitioning need different changes. The 16,400-row attention case and the large checkpoint stream remain intact because their size protects real limits. ## Risks - Two isolated processes use more memory and database connections than one. Concurrency is capped at two. Shared database overrides force serial execution. - Moving setup into helpers can change test lifecycle order. The split suites keep the same case assertions and use independent process state; focused runs and the full CI suite check the move. - Split-file weights are estimates based on case timings from the linked run plus repeated setup cost. The next protected-master refresh replaces them. Weights never permit tests to be skipped. - Local timing varies with host load. CI measurements after merge are needed to establish the overall PR wall-time improvement. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and code execution. The exact backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (changed suites and focused reruns; the full local-run caveat is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 10:19:24 -06:00
DottaandPaperclip	baa4a2e64e	fix(runner): restore Codex auth before source recovery (#18612) ... Restore disposable Codex authentication and safe provider configuration before cold source recovery. Preserve source receipts and conversation state until reconciliation, and rebuild assigned tools from the admitted launch binding. Validate remote launch destinations, clear stale authentication when no selected credential exists, and atomically publish SSH file transfers without following retained links. Red/green recovery regressions, a real two-turn Codex/MCP probe, and all current PR checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-02 10:14:24 -06:00
Nicky LeachandPaperclip	aab87ce707	Prove workspace delivery before inspecting cleanup candidates (#18493) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Completed tasks leave workspaces that need safe cleanup. > - Providers without ancestry inspection currently need a recorded merged PR before cleanup can start. > - A squash merge changes commit identity, so direct ancestry cannot prove that delivery. > - This change proves delivery from the finalized HEAD and captured destination before storage inspection. > - Fresh storage checks and a second delivery check still protect removal. > - This reuses the existing worker and receipts. It adds no table or timer. ## Linked Issues or Issue Description Refs #18410. Related: #13400 addresses workspace identity during inspection; this change adds destination-aware delivery discovery. ## What Changed - Resolve direct ancestry or discover an associated merged GitHub PR without a recorded work product. Require exact HEAD, destination repository and branch, and a reachable integration commit. - Use the latest settled finalization receipt as a candidate. Reject missing, failed, pending, conflicting, or changed evidence. - Inspect storage only after positive remote proof for providers that disable ancestry inspection. Recheck HEAD, branch, source identity and delivery before removal intent. - Add an optional provider capability for a specific PR recovery hint and an optional prepared destination receipt. Older providers keep their existing protocol. - Wake cleanup when finalization writes settle or gain a workspace binding. Keep remote-only retries for concrete unresolved delivery. Park missing or unsupported candidate identity. - Reuse the source project's explicit GitHub consent for delivery checks through the existing authorization service. Revalidate the bound repository and consent before removal; missing or revoked consent retains storage without switching identities. Public sources remain anonymous. - Update the contracts, documentation, access map, and targeted regressions. ## Verification - Initial-head GitHub CI passed at `5acd4c818be6f1764f8319f42891f9956c357c26`, including full build and typecheck. Standard Greptile is 5/5 with no open threads. The first Apex review scored 3/5 and found startup latency from remote proof lookups and removal retry identity changes across provider capability changes. Both corrections are pushed in `9d1a38df3d`, with the qualified destination-ref length fix. Red regressions reproduced each issue. Latest targeted checks passed: cleanup/worker 26, provider integration 41, SDK contract 8. Head `9d1a38df3d` passed standard and Apex reviews at 5/5 with no open threads, plus all CI including full build/typecheck. Canary Dry Run initially failed on an npm 404 fetching `rollup-4.67.3.tgz`; the exact tarball subsequently returned HTTP 200, and one targeted retry passed without source changes. Aggregate verification also passed. After #18403 merged, master introduced conflicts in shared workspace cleanup and metadata handling. Resolved them in `7700832536`, preserving remote snapshot cleanup and Git delivery-proof cleanup. Targeted merge verification: 93 tests passed in the first run (59 opt-in skips), then all 16 remote lifecycle tests passed with explicit database opt-in; map checks passed. The isolated database is stopped. The merge head received standard review 5/5. Its CI found two failures in unchanged test cleanup paths: native runtime temporary-directory ENOTEMPTY and agent-chat cancellation lock timeout. Those failures remain recorded and require successful final-head verification. The subsequent project-consent change has targeted coverage: authorization36, resolver33 and cleanup19 tests passed, including revoked consent, anonymous public access and source-binding races. The positive consent test failed before the fix. Current head `c772679e05` has green GitHub CI, including full build/typecheck and both previously failing test jobs, plus standard Greptile 5/5 with no open threads. Neither cleanup failure required changes to its test path. Apex also completed at 5/5 on this exact head (check `124651548532`, summary updated 2026-10-31 20:45 UTC), with no open review threads. The PR remains draft. Earlier failures remain recorded above. - Final focused run on the rebased branch: **182 tests passed across six files**, covering the resolver, delivery cleanup, provider bridge, finalization notifications, existing workspace service and SDK contracts. The disposable test database was stopped afterward. - Targeted resolver tests: 33 passed. These cover ancestry, discovered squash PRs, wrong HEAD/base/repository, removed integration commits, bounded responses, cancellation and credential isolation. - Targeted cleanup integration tests: 17 passed. Provider bridge integration: 36 passed. SDK contract tests: 7 passed. Finalization notification tests: 9 passed. - A negative control that removed the final delivery check failed the race regression as expected. The check was restored. - Read-only live GitHub probe: the new resolver discovered PR #18410 from HEAD `3f5ade4cd42b707040e9a82fdaed343e3d736a09`, without PR hints. It proved integration commit `6484f96c28bfb11a793ce4be6cc5ac15643a5a02` reachable from observed master `5bc029bf6c1db6647e96acea323f3723fd1d3eba`. GitHub also advertised the exact original HEAD at `refs/pull/18410/head`. - `node scripts/check-maps.mjs` passed. Structural checks do not prove deployment behavior. - Local tests are targeted by user request. Full build, typecheck and test suite run in GitHub CI. A standalone server typecheck could not complete because local Runner build declarations were absent; no changed-file diagnostics were reported. - Initial local execution hit a stalled pnpm wrapper, sandbox socket denial and stale generated SDK declarations. Direct installed binaries, the authorized isolated test database and rebuilt SDK resolved those setup failures. No deployment was performed. ## Risks - Destructive cleanup still requires fresh storage safety and lifecycle checks. Remote proof alone never authorizes removal. A persisted removal intent preserves existing recovery semantics. - Legacy workspaces without an unambiguous destination or completion HEAD remain retained. This does not guess the original default branch. - Private repositories use the explicit standing GitHub consent on the bound source project, with current membership, grant, repository access and trust checks. No separate company token is required when that consent is valid. Missing or revoked consent and unavailable or rate-limited access retain storage; the worker does not borrow a task or agent identity. - A known undelivered commit keeps the existing five-minute retry because external GitHub changes have no guaranteed ingress to this lane. This can keep cleanup scheduled. Missing evidence parks instead. - Providers need the new recovery capability to clean a squash merge whose original HEAD is reachable only through its PR ref. Providers without it remain conservative. - The resolver currently supports GitHub.com. Existing local and other-provider ancestry paths stay in place. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and parallel code agents. The exact runtime model ID and context-window size were not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 07:59:17 -08:00
DottaandPaperclip	71a20453e1	fix(workspaces): honor inherited remote workspace bindings (#18611) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Task creation can bind a child to its parent's execution workspace. > - Native sandbox runs reserve that workspace before the provider starts. > - Remote reservation rejected inherited isolated workspaces and operator branches because the original source task differed. > - This check stopped valid child tasks before their existing access and writer checks could run. > - This pull request uses the persisted binding and existing source-access checks to admit that continuation. > - The benefit is that authorized child tasks can continue saved work while the single-writer lock stays effective. ## Linked Issues or Issue Description **What happened?** A child inherited an active isolated remote workspace through normal task creation. Its next run failed during setup with `Workspace ownership changed.` No provider work started. The reservation guard treated the original `sourceIssueId` as exclusive ownership, although task creation had persisted a valid child binding. **Expected behavior** A bound child or follow-up can reserve the workspace after the prior writer saves. Current task/run access, source visibility, trust, company scope, closed-workspace state, and writer checks must still apply. The original source task remains provenance. **Steps to reproduce** 1. Create a task with an active saved remote isolated workspace or operator branch. 2. Create a child without an explicit workspace override. 3. Confirm that the child inherited the execution workspace ID. 4. Start its native sandbox run. The old reservation guard rejects it before provider execution. **Paperclip version or commit** Reproduced on source based on `06bc8db75c71142ed108e0f331394c5187015d6a`. The same guard is present at `1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. Related: #17156 proposes separate workspaces for child tasks. This repair preserves the current inheritance contract and enforces the remote writer lock. #16911 addresses child bindings across repositories. ## What Changed - Remove the original-task equality restriction for active remote workspaces. Keep the existing persisted-binding and retained-source authorization path. Explicit task-root selection follows the existing V1 filesystem contract; task-assignment grants do not define filesystem permissions. - Keep closed workspace rejection and unfinished-writer checks. - Add database-backed regressions for inheritance, replay, competing children, an unbound task, private source data, company isolation, and closed records. - Extend real save/restore tests for inherited isolated workspaces and operator branches. Keep shared-workspace discovery coverage. - Update the development guide and workspace maps. ## Verification - Red: before the repair, the inherited isolated/operator cases fail with `Workspace ownership changed.` The shared case passes. The unfixed suite has five failures and four passes. - Green: 11 reservation cases pass with real PostgreSQL 18 in an isolated Docker container. - An earlier run of the four affected workspace/preparation/trust suites passed 191 tests. The review follow-up passes all 15 selected reservation/save/restore cases. Inherited cases remove the original files, use a distinct allocation and target, restore the saved file, append child work, and independently read that work from the child's new snapshot. Shared-source discovery stays covered. The filter excludes 42 other integration cases. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` pass. A direct server typecheck also passes after the final test edit. - `pnpm test:run` stopped in its first chat shard: 338 passed, one Discord receiver-lease setup failure, 1,478 outside the shard. The failing test passes in isolation on both the unchanged baseline and the fix. Its suite-order failure is not explained by that retry and remains recorded. The unchanged baseline also fails three other Discord publication cases in that chat shard, while its next chat shard passes. This does not reproduce or explain the original lease failure; both results remain recorded. CI is complete on `e697f6e1b8`: 54 successful check runs, one successful legacy status, three skipped checks, and one neutral check. No check is pending or failed. Greptile gives this commit 5/5 with no remaining findings. The security reviewer accepted the V1-contract explanation; both review threads are resolved. - Live sandbox acceptance remains unverified. The available test credentials could not resolve the required sandbox snapshot. The provider rejected creation before workspace execution, and a scoped follow-up found no allocation. No live model turn, deployment, or original-task recovery is claimed. - Local environment: macOS arm64, Node 26.4.0, pnpm 9.15.4. Native PostgreSQL startup hit the host semaphore limit; tests use disposable Docker databases. No shared database was reset. ## Risks - A bound child can now write after the prior writer releases the workspace. Incorrect binding or access checks could expose source data. Regressions cover unbound, private-source, and foreign-company rejection. - Concurrent children must not become two writers. A real transaction race grants exactly one reservation; saved-file tests keep the second writer blocked until save completes. - No schema or migration changes. This PR does not change task-creation inheritance, requested models, credentials, trust, approval gates, or recovery budgets. ## Model Used - OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool use, code execution, and browser diagnostics. No sub-agents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — the targeted workspace suites pass; the full local chat-suite failure and baseline comparison are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 09:36:05 -06:00
DottaandPaperclip	774ddc7271	test(evals): add the full chat catalog to TestOps (#18542) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations check real user requests through the product. > - The YAML catalog has 244 scenarios, but the shipped TestOps completion route selects only three. > - The preserved catalog already supplies the execution mappings and independent oracles. > - A larger TestOps plan needs exact identities, bounded selection and source-bound private evidence. > - This pull request adds that plan route without merging the catalog app into master or replacing its grades. > - Users can run the full catalog or a selected subset and see the actual outcome and prerequisites. ## Linked Issues or Issue Description Refs: #18516 and #18515. This extends their completion pilot and reuses its trusted reporting path. **What existing behavior does this improve?** The existing three-case TestOps completion workflow. **Current behavior** The remaining catalog scenarios cannot be selected through that TestOps job. The one-case backlog plan has a separate strict singleton contract. **Proposed behavior** A separate candidate-scoped plan maps all 244 scenarios. Existing case aliases, jobs, plans and result history stay intact. Registration is distinct from a measured pass. ## What Changed - Add a source-derived 244-row manifest. It binds the 242 domain descriptors and two legacy tasks to the unchanged YAML queries, contexts, outcomes and original bounds. The JSON snapshot avoids importing the unmerged domain tree into master. - Reconcile exact case identities and a dedicated full-catalog plan. Reuse legacy aliases, including the existing backlog case, without renaming them. Preserve history and stop on conflicting aliases or uncertain writes. - Extend the shared completion adapter with a separate trusted full-catalog workflow context. Return a compact observed binding that fits one 48 KB protected variable. Keep the three-case pilot contract intact. - Add a fixed-source manual workflow. Small selections run serially. Larger selections allow eight independent workers and one serial external-input worker. Each worker and report reader keeps a singleton approval and exact-key AWS session. The eight-cell private authority limit is unchanged. - Validate and merge metadata-only singleton projections before one protected Allure upload and complete paginated readback. Preserve failed originals and cleanup failures beside passed results. Keep screenshots and raw evidence private. - Discard a changing full-catalog page-count snapshot and retry within the same observer bounds. Malformed page fields still fail at once. A pass still needs one consistent exact selection and all final evidence checks. - Keep workflow source and app source separate. The full lane builds candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are finite TestOps host admission and its offline test. This is not current-master app qualification. - Give full selections above three cases a bounded metadata-read allowance above the existing 120-second deadline ceiling. Readback has two upper bounds: at most twelve polls with five-second pauses, and a size-derived deadline of at most 374 seconds, whichever exhausts first. There is no guaranteed minimum wait or processing SLA. Small selections and pilot deadlines stay unchanged. Final identity, grade and evidence predicates stay intact. - Document the real readiness limits and setup path. Five coverage gaps, six evidence exceptions, eighteen additional-credential declarations and thirty-seven external-input declarations overlap. They are not new pass claims. ## Verification - Related Node controls passed: pilot API and dispatch, 244-row manifest, idempotent reconciliation, full selection, three-page result readback, source guards and partition/output-size bounds. The latest ten adapter controls include the actual registration receipt's variable-size limit and the real bootstrap facade's closed failure codes and privacy checks. - A deterministic 244-result replay with one second per HTTP response failed at the old deadline and now verifies after 127 virtual seconds. A three-second response replay still rejects at 375 virtual seconds. The full-lane allowance accounts for paginated reads and four-way result batches; it is not a guaranteed network SLA. All 55 related API/adapter controls pass. They also reproduce the changing-page-count failure before the fix, then verify the consistent snapshot. Permanently changing counts fail after twelve polls. A malformed numeric field still fails before a retry. - Owning export controls: 26 passed. They include all 244 synthetic carriers, mixed passed/failed originals, missing and foreign projections, privacy filtering and exact source binding. - Candidate host owner: 31 offline controls passed. The new finite-catalog case failed before admission changed. - Preserve two actual local regressions: canonical scenario provenance failed before its one-line fix; the full registration binding exceeded the variable limit before compact output. Both now pass. - Initial local harness setup attempts failed before test execution. The final narrow owner configuration avoids provider-bundle setup. - Full repository tests, build and typecheck were not run locally. Normal required PR CI provides the remote checks. - Owning TypeScript no-emit, actionlint, paid-workflow policy with unsafe controls, and diff checks passed. - Local tests used Node 26.4.0. These are synthetic source controls. No local app, database, native build or provider qualification ran. - CI [41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776) on `23e518be` recorded one 15-second timeout in `attention-retention-work.test.ts:153`. All other ordinary leaves passed. The test used a fixed JavaScript date but its retention deadline comes from PostgreSQL `updatedAt`. The exact failing await and original SQL timestamp were not captured. The corrected pre-sync head’s server shard (2/12) passed on AWS in CI 41309030928. Current master independently fixed the same clock assumption in #18543; conflict resolution adopts that whole upstream file. No local PostgreSQL run or timeout increase was used. - Current-head normal CI [41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187) passes all required checks on `fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all review threads resolved. The final diff has 23 catalog files. Current master’s whole retention test is adopted; the redundant earlier test change is absent. The valid changing-page snapshot regression passes with all 55 related API/adapter controls. No new TestOps import or live evaluation has run for this change. After delivery, use metadata-only bootstrap and a three-case normal TestOps selection to verify the new route. Do not claim a new 244-case measurement. ## Risks - The app target remains the preserved catalog candidate. Each result records its target and workflow revisions. No old campaign ordinal, acceptance gate or core fix is changed. - Some mapped cases need private account configuration or have known source/evidence gaps. Their original checks and evidence rules can still fail. A registration does not establish readiness or success. - Whole-plan execution can take hours and spend provider and AWS money. Original per-case limits and zero retries remain. Unknown setup usage is not free. This PR does not run all 244 cases. - The raw private `SOURCE_REF` field retains the prior integration-lineage label. It is not a verified checked-out branch. The actual candidate checkout and all authority checks use immutable `3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set `source.ref` to null. - TestOps has no atomic bulk upsert. Reconciliation retains partial progress and performs no automatic resend after an uncertain response. - The trusted inventory keeps a 12 MiB limit per public carrier file. The larger parser ceiling does not promise admission above that limit. The existing 244-case fixture produces a 1,359,465-byte normalized carrier; this is not a guarantee for future payload sizes. - Raw evidence remains under the existing private retention owner. Public GitHub artifacts and TestOps attachments contain only allowlisted metadata and availability receipts. - The new workflow needs the exact master entry in the existing runner group's selected-workflow list before a live run. No repository or runner-group visibility is broadened by this source change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and independent source review, using code and tools. The host does not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the relevant local controls and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 08:58:44 -06:00
3 people	06bc8db75c	fix(runner): install verified API artifacts in execution workspaces (#18494) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents use authorized API tools to read saved results. > - Copying a large response as model-generated text can lose bytes. > - The execution workspace needs an exact copy of the saved artifact. > - This change copies the authorized bytes without shell API credentials. > - The runtime verifies size and SHA-256 before it reports success. > - A live evaluation checks capture origin, installation and delivered bytes. ## Linked Issues or Issue Description **What happened?** A native agent lost one byte while copying a 24576-byte API text page. Existing capture and paging controls passed. The runtime had no verified saved-artifact-to-workspace copy tool. A separate historical screening refusal does not establish capture corruption. **Expected behavior** An authorized native run can install the complete saved artifact. The stored and installed bytes must match in size and SHA-256. A conflicting file must remain unchanged. A lost publication receipt must not cause an unsafe repeated write. **Steps to reproduce** Run `server/src/services/native-runtime/artifact-transfer.integration.test.ts` on baseline `c46b8f35add9255fc71be294d27f5a7865bcb47f`. Its capability assertion fails because `download_artifact` is absent. Apply this PR and run the same regression with the filesystem and API controls. It verifies 388131 synthetic bytes through the authority and installation boundary. Refs #18429. Related #17393, #17086 and #17285 cover paging, transfer foundations and remote output reads. This PR adds the inbound installation contract. ## What Changed - Add `download_artifact` for supported standard-mode native runs, with a 10 MiB limit. - Reuse current company, task, run, asset-content and responsible-user access checks. Check authority again before publication. - Bind installation to the current execution lease. Stage through confined Linux descriptors and publish with an atomic create-only link. Keep API credentials out of execution commands. - Verify bytes at storage and execution boundaries. Reject unsafe paths, symlinks, different existing files, truncation and corruption. Reconcile an unknown publication through verification only; an identical explicit retry remains safe. - Add API documentation, access-map entries and content-free audit records. - Preserve only the pinned Daytona SDK numeric HTTP status when failed creation cleanup is unconfirmed. Keep provider exception messages and headers private. - Keep only the finished runner binary and portable provider pack in their builder layers. Remove Rust targets/downloads and the Node install workspace/store before export. Report tree sizes. - Add one explicit Codex/Daytona Product E2E cell. Its independent oracle checks the server capture audit, redacted tool receipts, stored source, delivered attachment and browser link after reload. Uploaded substitutes and one-byte loss fail. ## Verification - Merge update at `6766b46553cab5b9787ade87a002eccaf50e68fc`: merge current master and preserve both explicit suites in the catalog. Focused catalog, image and oracle checks: 122 pass. Filesystem and authority integration: 20 pass. Maps, module boundaries and whitespace checks pass. New-head CI and Greptile are running. Full local build and typecheck remain unavailable because cargo is absent. - Behavioral red/green: the absent-tool regression failed before implementation and passed afterward. The initial missing-package attempt is recorded as setup failure. - Linux filesystem and real PostgreSQL/authority integration: 19 filesystem tests and 1 authority test pass. Existing API authority, capture and paging controls: 70 pass, 2 skipped. Provider transport in these integration controls is simulated. - Qualification calibration: 118 focused tests pass. The real persisted-event redaction case and a byte-identical uploaded substitute each failed before their oracle repair. The unchanged controls then passed. Harness typecheck passes. - Full recursive typecheck and build pass locally on the integrated production source at `086f7a2901`. Later commits change only qualification tests and documentation. - Structural map/module checks and whitespace checks pass. - Full local `pnpm test:run` was attempted by the implementation worker and stopped under its 8 GiB memory limit, with no full-suite verdict. Current-head CI must supply the full test result. - The complete local Product E2E support suite has 13 failures. The unchanged pre-qualification baseline reproduces all 13: host PostgreSQL semaphore capacity, stale timeout/profile/workflow inventory expectations, and expired fixed-date allowance fixtures. This is not a full local suite pass. - Earlier CI at `086f7a2901` failed a browser test during cleanup cancellation with a database lock timeout. It remains recorded. All browser shards passed on both later fully checked heads. - Two qualification campaigns were canceled during setup before provider execution while correcting the oracle. They provide no live product verdict. - Previous-head CI at `28fa001457f59df1c300c016dc569049e72d0061`: 53 checks pass, 3 intentional selection skips; no failed or pending checks. - Real image/export regression: campaign 41272351780 compiled the Linux runner and passed root/unprivileged CLI and transport checks, then failed BuildKit export with `no space left on device` (exit 102, filesystem 99.99%). No provider cell ran. Removing the Rust build trees alone was insufficient: campaign 41273604815 also failed during provider-pack export with exit 102. Commit `5bbb549c54` also consolidates dependency installation and pack construction so the disposable workspace/store can be removed in the same layer. All 122 image/oracle/catalog checks pass. Image and remote-pack publication then passed in campaign 41275196741 after cleanup changes plus a scoped 60-to-120 GiB expansion of only its delete-on-termination builder disk. The VM terminated and the expanded volume was deleted. This does not establish that cleanup alone makes the default 60 GiB builder sufficient. - The Daytona admission probe then failed before agent creation, with unconfirmed provider cleanup. The queued continuity case was canceled because it shares this prerequisite. No artifact-transfer or continuity success is claimed. - Status-diagnostic regression: the original generic wrapper failed the HTTP-status assertion. After the repair, the full plugin suite passes 501 tests, with 7 skips; its standalone TypeScript check passes. Controls reject invalid status values, avoid accessors, preserve the original create status and exclude credential canaries. - Previous-head Greptile: 5/5; its summary names `28fa001457f59df1c300c016dc569049e72d0061`. All review threads are resolved. - Live artifact transfer and three-turn warm continuity: NOT VERIFIED. The diagnostic case in https://github.com/paperclipai/paperclip/actions/runs/41277598289 confirmed Daytona HTTP 401 before agent creation. The configured `DAYTONA_API_KEY` in the existing `runner-e2e-paid` environment needs verification or refresh. The queued continuity control was not run after its shared admission prerequisite failed. No further provider attempt is planned until the credential changes. Unconfirmed provider cleanup remains recorded. ## Risks - Source-provenance changes can repeat the provider dependency install. The combined build layer trades install-cache granularity for lower retained disk use. - Linux descriptor semantics and Node are required on the execution host. Unsupported providers and restricted runtime profiles fail closed. - Storage reads buffer at most 10 MiB. A provider outage can leave a reserved staging file. Partial bytes never become the destination. - A lost publication receipt can leave a complete file with an unknown result. Recovery verifies the existing file without installation. Conflicts are never overwritten. - Artifact contents remain untrusted. The copy does not bypass screening or authorize downstream execution. - Live qualification uses the governed Codex API profile. It does not establish every model, subscription profile, platform or historical failure cause. ## Model Used OpenAI GPT-6 in Codex assisted with diagnosis, code, tests and qualification. The desktop runtime does not expose a more specific model deployment ID or context-window size. The model used reasoning, source inspection, tools and code execution. The native implementation agent is configured with OpenAI `gpt-6.1-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 08:27:09 -06:00
DottaandPaperclip	67948173f4	fix(sentry): report saved-message task attempt failures (#18541) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks retain user messages when an execution attempt cannot finish. > - The task interruption projection drives the saved-message failure notice. > - That state can occur without a new terminal run transition, which the existing Sentry reporter requires. > - This pull request reports that failed task state through the existing Sentry capture path. > - Operators can find the failure through the same tag and saved run links. ## Linked Issues or Issue Description Refs #13446 and #16300. The related PR search found no open implementation of this change. **What happened?** A task can show “Couldn’t finish this attempt. Your message is saved.” without a matching run-failure Sentry event. A stale saved-message wait can produce this state after the previous run succeeded. **Expected behavior** When run-failure reporting is enabled, this failed task state also reports with `event_kind=run_failure`. **Steps to reproduce** 1. Enable the backend Sentry DSN and `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` or `all`. 2. Leave a saved-message wait on an unfinished task after its last run has ended. 3. Read the task after the wait grace period expires. 4. The task shows the failure notice. Before this change, that read does not report it to Sentry. **Paperclip version or commit** Base commit: `e122478368`. ## What Changed - Report the system-owned failed presentation from the existing batch task reader. The pure projection remains unchanged. - Reuse `captureRunFailure()` with `event_kind=run_failure`, `failure_source=task_interruption`, and `error_code=task_attempt_failed`. - Preserve the actual saved run status and company, agent, task and run references. Load the run agent's adapter through a company-scoped join in the existing run query, including after reassignment. Add no database query or background timer. - Send fixed error text. Do not read chat bodies, prompts or raw run diagnostics for this event. - Suppress repeated reports for the same company/task/run/intent with a bounded process-local cache. Failed or disabled captures do not consume the report. - Document the enabled modes, event fields, duplicate behavior and cache limits. ## Verification - Regression before implementation: the two new Sentry tests failed because the capture entry point was absent; the existing 38 Sentry tests passed. - Focused verification: 332 tests passed across `run-failure-sentry.test.ts`, `task-interruption-query.test.ts`, `task-interruption-presentation.test.ts`, and `run-failure-report.test.ts`. The reassignment regression first reproduced the missing adapter, then passed after the query correction. - Command: `PAPERCLIP_TEST_POSTGRES_ADMIN_URL=<local-test-admin-url> pnpm exec vitest run server/src/__tests__/run-failure-sentry.test.ts server/src/__tests__/task-interruption-query.test.ts server/src/services/task-interruption-presentation.test.ts server/src/services/__tests__/run-failure-report.test.ts`. - PostgreSQL tests used isolated databases on a local PostgreSQL 17 server, with automatic cleanup. Initial embedded starts failed because installation scripts had not restored library links; after repair, the host semaphore limit prevented new clusters. The existing external-database helper supplied the working test path. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `git diff --check` passed. - `pnpm test:run` was attempted. It stopped in the unchanged chat integration shard collection step with “No test found” for line filters. The complete local suite has not passed. - Sentry capture assertions use the SDK test double. Live Sentry ingestion and a deployed browser journey were not tested. ## Risks - An enabled instance can send more Sentry events when task reads discover retained failures. It can emit a task-state event alongside the separate terminal run event. The source tag distinguishes them. - Deduplication is process-local and retains 1,000 reports. Restarts, multiple replicas or cache eviction can report a retained failure again. - No schema, permission, execution or UI changes. The existing opt-in reporting gate stays in place. ## Model Used OpenAI GPT-6 through Codex, with tool use, code editing and test execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 08:17:22 -06:00
DottaandPaperclip	1a6017d646	fix(server): resume suspended sessions after controller replacement (#18554) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve a completed conversation as well as active execution. > - A warm remote Runner can suspend before the replacement admits another turn. > - The replacement restores its controller checkpoint from PostgreSQL, but lacks the Runner suspension receipt. > - It therefore tries to adopt a live process and rejects the safely suspended source. > - This change restores the verified receipt and preserves retained launch files while the old authority is reconciled. > - Later turns can keep the same provider conversation after the old web server filesystem is deleted. ## Linked Issues or Issue Description Refs #18552, #18543, #18366. **What happened?** The real rolling-deployment journey passed both HTTP handoffs, with 765 reads and 78 verified saves. The first three agent turns kept their native and provider sessions. After the second replacement, the fourth continuation failed with `remote_warm_session_not_idle`. The remote Runner was suspended with an empty outbox and no pending terminal. A focused regression reproduced that exact failure. After restoring its suspension receipt, the same test exposed a stale launch-file read during source recovery. **Expected behavior** A replacement can resume the same provider conversation from a settled suspended Runner. It must validate the physical sandbox and all persisted identity fields. A live source must still pass the existing adoption checks. **Steps to reproduce** Enable the Linux Runner fixture for `native-runner-restart-recovery.integration.test.ts`. Run the `diskless-warm-suspended` cases. The fixture completes and gracefully suspends each source, deletes the controller projections, and admits a later turn from another process. It does not write a synthetic suspension receipt. ## What Changed - Reuse the existing per-turn suspended-receipt reader before warm admission chooses live adoption or suspended recovery. - Keep the receipt read retryable when a failed remote read leaves only the controller projection. - Leave every non-suspended warm state to the existing live-adoption checks. A preliminary backpressure observation cannot reject admission before that owner inspects the current Runner state. - Preserve current sandbox binding, settled-outbox, exact-identity, local-evidence and admission checks. - Pass the structured launch specification to remote asset preparation. Source reconciliation keeps the old authority's retained launch files; a current-authority launch still stages its current files. - Add two Linux Runner/PostgreSQL regressions for two controller-cache deletions and three turns on one provider thread, including an interrupted receipt read. - Update the deployment qualification record, including the failed live evidence and the successful staging database-idle observation. ## Verification - Before the fix, the focused regression fails with the exact live `remote_warm_session_not_idle` error. - Both new regressions pass with real PostgreSQL and Linux Runner. The provider is deterministic; this is not a live model qualification. - The first receipt-only fix failed on the deleted controller skills cache. The completed fix passes that same scenario without recreating fixture launch assets. - All 1,067 surrounding launch supervision, warm recovery, launch-asset and controller recovery tests pass. The exact `cc23019691` server build passes. All ten Linux continuity cases pass, including live adoption, graceful suspension, per-turn resume, changed configuration and failed-read retry. Each case checks provider-thread identity and exact turn counts. The live Codex/Daytona journey passes on the same clean source and exact build: 518 reads, 53 saves and 53 verified saves with zero probe failures; both predecessor homes deleted; the active execution completes once; all four turns preserve the provider conversation and native receipts; saved files and the final command output are independently checked. No automatic retries. Child credential and object-storage bucket cleanup pass. Browser reconnection remains a separate hosted acceptance check. - The review refinement has a failing-before regression: only the preliminary read reports transient backpressure; the subsequent adoption reads the real Runner state. It and both suspension cases pass on the final source (three cases), with server typecheck. The existing idle check remains unchanged. An initial retry hit a fixture port left by the intentional pre-fix failure; a fresh container resolved that infrastructure error. - Independent cleanup verification found two archived setup sandboxes left by the live harness. Both were deleted and confirmed absent. The failed automatic-cleanup check remains recorded. - No claim of the ten-deploy Railway campaign or production canary yet. ## Risks - A cold warm-session admission can now make one bounded remote state read when local lifecycle evidence is absent. Its failure stops admission and remains retryable. - This path cannot replace contradictory local evidence, transfer authority, or treat a suspended process as a live one. - Source recovery must keep the previous authority's launch files until reconciliation completes. The launch identity selects that behavior; ordinary launches still prepare current assets. - No schema migration or public API change. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge CI qualification note: first final-head run `41322376054` passed the native/runtime and remaining suites but failed the unrelated `automatic_message_on_reactivation` browser fixture at its pre-reactivation deferred-queue count (0 rather than 1). The unchanged test passed locally against isolated PostgreSQL (1 test, 43.6 s); inspection found an asynchronous company-pause/fixture-seeding timing window. Original CI logs and local evidence are retained. The single failed-job retry passed; all final-head CI gates are green. Both reviews are 5/5 and no review threads remain unresolved. The first local embedded-Postgres launch failed before tests during cluster initialization; the isolated external-Postgres test passed, and its disposable database was removed after correcting the private launcher cleanup call. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 10:08:51 -06:00
DottaandPaperclip	c0550e7010	fix(server): preserve sessions and defer busy overlap recovery (#18552) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve remote execution and later task continuation. > - The active-run handoff now survives deletion of the departing server home. > - The next admission still compared disposable instruction and skill paths as configuration. > - A native receipt could also settle without the old adapter returning its terminal metadata. > - A replacement can also collide with a run row locked by the serving controller. > - This change preserves durable session identity and terminal evidence, and defers locked recovery work so overlap can reach readiness. ## Linked Issues or Issue Description Refs #18543, #18366, #9977, #18118. **What happened?** A real Codex/Daytona journey replaced both control-plane servers and deleted their homes. Reads, saves, saved files and active execution passed. A later useful answer used a new provider session. All four run records lacked the native terminal contract. The whole qualification therefore failed. A later attempt also exposed startup failure: the replacement waited for a row lock held by the serving controller and aborted on its one-second lock timeout. **Expected behavior** An unchanged managed instruction entry and immutable skill version retain the session across server cache relocation. Actual content and configuration changes still reset it. A committed native receipt retains its terminal contract even when the originating adapter never returns. A busy recovery row defers through the existing scheduler while the replacement reaches readiness. **Steps to reproduce** Run the `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with separate controller homes and shared PostgreSQL/S3. The focused regressions reproduce the failures without a model call. ## What Changed - Read the managed instruction entry digest from the company-scoped durable file head. Do not restore or read the cache for this check. - Omit disposable instruction paths only when that durable authority exists. Fingerprint selected immutable skill versions in both runtime skill inputs. - Preserve the existing local and external instruction path checks, and unversioned skill identity. - Publish the accepted native terminal contract in the same transaction as the canonical result receipt. - Skip locked recovery rows and schedule the same run for retry. Avoid an audit write that would block on the same row. No ownership is transferred while busy. - Keep replacement readiness independent of an in-progress skill selection; migration and cache eviction still require exact pins. Missing durable skill versions still refuse startup. - Project only the instruction digest in SQL instead of loading the entire agent-file manifest per admission. - Add real PostgreSQL lock contention tests for the issue, run and coordinator, including the heartbeat startup caller and later claim. - Limit native test teardown to its owned database on a shared test host. The old behavior also fails on clean master. - Add cache-loss, scope, changed-content, changed-version and finalization regressions. Update the deployment qualification record. ## Verification - Both focused regressions failed before the source changes. - All 249 tests pass across `heartbeat-workspace-session`, `agent-file-durability`, and `paperclip-control-plane-port`. - Server typecheck and exact-revision build pass. - Assembled admission ran 59 passing cases but failed during native-suite teardown. The teardown failure reproduced on clean master. All 16 native resumption cases and teardown pass after the database-scope fix. - All three row-lock regressions fail with the pre-fix lock timeout and pass with deferred recovery. The full recovery suite passes 74 cases, including the startup caller; eight Docker-specific cases were not enabled. - The real attempt at `d2165dec5d` failed when B aborted on the run lock. Its credential and bucket were removed; all three provider sandboxes were independently confirmed absent. The next real attempt at `aae8a53d01` failed a different readiness gate while a run was preparing skill pins. That race now has a failing-before regression; all 33 readiness, durable-file and managed-skill tests pass after the fix. Its owned resources are cleaned. The `a15a12651c` live attempt passed both web replacements: 765 reads, 78 saves and 78 verified saves with zero HTTP probe failures. The first three turns preserved the provider session and native terminal contract. The fourth continuation failed because a safely suspended remote Runner was treated as live/idle; that distinct recovery gap is being handled in a follow-up. All three owned sandboxes were independently confirmed absent. No claim of full Railway qualification yet. ## Risks - The new fingerprint shape may reset a session once when upgrading from older fingerprint metadata. It is stable across subsequent cache replacements. - A versioned skill is identified by its immutable selected version; materialization and admission still enforce availability. Unversioned skills retain their existing path boundary. - A native terminal describes the accepted execution result. Workspace and issue settlement remain separate authorities. - This does not yet prove the ten-deploy staging campaign, natural database autosuspension or the production canary. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 09:14:34 -06:00
DottaandPaperclip	7751fc3107	fix(server): complete volume-free controller admission and recovery (#18543) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its volume-free web server must admit remote warm execution without depending on a local disk. > - Fresh run rows retain the initial legacy label during workspace preparation. > - The native runtime selector commits the execution mode later. > - The earlier guard rejected valid remote runs before that selection. > - This PR checks storage during preparation and the resolved lifecycle during runtime selection. > - Agent-file recovery also has to reconstruct disposable cache paths from the retained remote directory. > - The deployment test removes the complete predecessor home, including immutable caches, to enforce that contract. ## Linked Issues or Issue Description Refs #18518, #18529, #18533. **What happened?** A real Codex/Daytona deployment test failed during setup because the volume-free workspace guard checked the initial run label. After fixing admission, controller retirement first failed on an immutable skill cache. Removing that cache exposed a production recovery failure: B rejected the saved agent-directory path from A, even though the remote execution identity was unchanged. The next live attempt recovered that directory and transferred the same runner, but adoption then tried to recreate provider launch assets from the deleted controller home. **Expected behavior** Allow remote workspace preparation before native selection. Reject actual legacy and per-turn execution at the runtime boundary. Delete the entire stopped predecessor home, including immutable caches, without following links to shared dependencies. B must reconstruct its local agent-file projection from the exact registered remote lease while preserving the save baseline and unsaved remote edits. Reattaching an already-running provider must not prepare a new launch or require the old launch files. The full backend must recover the admitted instructions from the matching durable run checkpoint. **Steps to reproduce** Run the explicit `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and an owned Daytona fixture. The new boundary regressions also reproduce the premature guard and missing runtime rejection without a model call. **Paperclip version or commit** The failures were observed on `247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on `e122478368`. ## What Changed - Validate Runner and remote workspace authority before runtime selection without trusting the initial run label. - Reject resolved legacy execution and local-source fallback, preserving the native warm-policy check before admission. - Rebind remote agent-file caches on ephemeral controllers and retrieve their validated contents without uploading over the retained execution. Preserve pending saves when the remote collector is unavailable. - Restore over-quota remote directories without applying save quotas to run admission; ordinary saves still enforce quotas. - Skip provider launch asset materialization when adopting an existing Runner. A real-process regression deletes the old skill/instruction sources before reattachment. - Recover admitted system instructions through Runner-owned checkpoint validation for the same run, session and context digest. Fresh launches still compose current instructions. - Make the attention-retention test advance past its database-assigned deadline, fixing a clean-master clock mismatch. - Serialize remote collection and cleanup across controllers with an operation-scoped Postgres advisory lock; fence stale cache-path updates. - Keep synthetic deployment probe tasks in a separate disposable company so they cannot block the business completion oracle. - Share the launcher's directory cleanup with rolling retirement. Reject a symlink root and skip linked descendants. - Test admission with real PostgreSQL transactions and cleanup with real read-only directories and external symlinks. - Update developer documentation and the canonical qualification record. Broader deployment acceptance remains in progress. ## Verification - `pnpm -r typecheck` and `pnpm build` passed. - All 156 workspace preparation, runtime selection and ephemeral contract tests passed. Two new regressions failed before the source fix. - Seven focused Product E2E cleanup/replacement tests and `pnpm test:e2e:runner:typecheck` passed. - `pnpm test:run` stopped at an unrelated 15-second chat integration timeout after 338 passes. That exact test passed in isolation without increasing its timeout. The later full-suite shards did not run. - `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures. Clean master reproduced 11 failures in unchanged onboarding expectations, operating receipts, Pi catalog assertions, workflow bootstrap counts and embedded PostgreSQL startup. The other two workflow assertions were fixed by the intervening master commit. These failures are retained, not counted as passes. - All 122 workspace-preparation tests pass after the reviewed host-fallback fix. The first broader attempt exposed a test environment leak; restoring the ephemeral flag between cases resolved it. - Agent-directory and instruction regressions passed 179 tests. Five focused cache-loss cases pass: different home, same path on an empty filesystem, foreign lease, concurrent controller, and missing collector. - The live fixture at `a18f6b85a3` gracefully stopped A and deleted its entire home, then failed the 420-second remote reattachment deadline on the saved local agent-directory path. That failure is retained; exact owned sandbox cleanup was independently confirmed. The next live attempt at `e14ca5f3e9` restored the cache and transferred the same runner, but failed while rematerializing launch assets from A. Both failures are retained and their owned sandboxes confirmed absent. No active-overlap, ten-deploy, or production-canary acceptance is claimed. - All 71 related Runner adoption/recovery cases pass, including the new removed-context regression (failed before the source fix). - The revised full agent-directory suite passed 110 cases and failed three cleanup timing cases. All 20 cleanup/cache-loss cases then passed in isolation with unchanged timeouts, including over-quota recovery. - Server and Runner TypeScript checks pass after the latest corrections; the live retry remains pending. - The live test at `67714af9ff` adopted the same remote execution after A was deleted, then failed while the full backend read the retired instruction bundle. All owned sandboxes were independently confirmed absent. Its regression now passes with actual backend recovery and no new thread or turn. All 59 backend factory tests and focused checkpoint identity tests pass. Runner and server TypeScript checks pass; the exact committed server build passes. A fresh paid-model retry followed. - The attention-retention CI timeout reproduced on clean master. The deadline correction passes all 53 tests in that file with unchanged timeouts. - The real `cd690de4d3` run transferred the same runner/session after deleting A, completed, and completed the next revision. During handoff, 372 reads and 38 saves had zero failures; all 38 saves were verified. The full attempt was interrupted because the probe's open task contaminated the business company oracle. This remains incomplete evidence. The scope regression fails before the fixture fix, then passes; all nine deployment support tests and E2E typecheck pass. All three owned sandboxes were confirmed absent and temporary credentials/storage were cleaned up. A clean corrected attempt is running. ## Risks - The runtime guard must still reject legacy and per-turn execution before dispatch. Tests cover these refusals and a valid resolved warm policy. - Directory permission changes apply only during cleanup of stopped, test-owned homes. Traversal does not follow symlinks. This changes no production cache permissions. - Cache reconstruction must not change remote execution authority or rebase the original save baseline. Tests verify foreign-lease rejection, preserved receipts, and absence of local-to-remote upload. - Postgres locking is scoped to active remote file work and adds no idle polling; local/durable-server locking remains unchanged. - No schema or public API changes. External conversation exemptions remain unchanged. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest real qualification at f5915fdc33: both empty-home replacements completed with 523 reads, 53 saves and 53 verified saves, zero probe failures. The active remote execution survived and completed once; saved project bytes and the later useful answer passed. The full workflow remains FAILED: the later continuation reset its native/provider session and the public run row lacked the native terminal contract. These newly exposed compatibility/projection gaps are tracked in the next implementation slice; this PR is not a claim of full Railway qualification. All owned Daytona sandboxes were independently confirmed absent after cleanup. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 08:24:14 -06:00
DottaandPaperclip	247552fe07	test(server): avoid deep traversal in durable file assertions (#18533) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud web servers restore agent files from shared storage after replacement. > - The durability test checks restoration, corruption, cancellation and retry. > - Its generic equality assertions traverse every byte in many large buffers. > - The post-merge check for #18531 exceeded the 15-second test timeout. > - This change uses native buffer equality and keeps every byte check. ## Linked Issues or Issue Description Refs #18531. The post-merge Cloud-source verification run timed out in the new cold-folder test. Other shards passed. The unchanged test passed locally. This change removes avoidable assertion cost; it does not increase test or production timeouts. ## What Changed - Use `Buffer.equals` for the cold-folder content assertions. - Keep the concurrency, cancellation, cleanup, old-projection and retry assertions. - Record packaged-image proof, the hosted deployment state and the original CI failure in the existing plan. ## Verification - Eight durability tests passed against a real local PostgreSQL fixture. Object storage uses controlled asynchronous streams. - The same test file took 10.03 seconds before and 5.90 seconds after this change. The changed cold-folder case took 561 ms after the change. - The first local attempt could not start embedded PostgreSQL and ran no tests. The existing external test database enabled the comparison. - Original failed CI: https://github.com/paperclipai/paperclip/actions/runs/41296253723 . That failure remains in the qualification record. - Server typecheck and build passed. All latest-head PR checks passed, including the full test, typecheck, build and browser gates. One cancelled runner job was retried after its log showed an infrastructure shutdown; the retry passed. This PR changes no production code or database schema. ## Risks - A failed content check now reports a boolean mismatch instead of a large byte diff. Equality still covers buffer length and every byte. - Local timing is evidence for the fixture, not a general CI performance guarantee. - Hosted active-agent and rollout acceptance remain incomplete as recorded in the plan. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime snapshot and context-window size are not exposed. Used code editing, shell execution and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 05:13:30 -06:00
DottaandPaperclip	1d348bbef0	fix(server): restore durable agent folders with bounded concurrency (#18531) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A replacement web server restores agent files from shared storage. > - The old restore reads each object in sequence. > - Modest network latency makes a 1,024-file folder exceed its deadline. > - This pull request uses at most eight concurrent restore streams. > - Every file must pass its size and hash checks before the folder is published. ## Linked Issues or Issue Description Refs #18518 and #18526. Local deployment qualification found that a 64 MiB folder could not be restored after the old server filesystem was removed. ## What Changed - Restore at most eight files concurrently within the existing file authority service. - Cancel sibling transfers on failure. Await all workers before removing the temporary folder. - Keep the existing deadline, file-transfer database pool, authorization and publication rules. - Add regression coverage for bounded concurrency, corruption, settled cancellation, retained previous bytes and retry. - Document the behavior and update the deployment plan with local qualification evidence. ## Verification - Real PostgreSQL and S3-compatible storage: 1,024 files of 64 KiB each, plus instructions, with 25 ms added to each object GET. Delete the entire predecessor home and disable local adoption before reading through the successor. - Before: the read failed after 30.065 seconds. After: the same case passed in 4.180 seconds and fetched 1,025 objects. All owned test resources were cleaned. - The durable-file suite passed eight tests against PostgreSQL. Server TypeScript compilation passed. All 56 latest-head CI checks are complete and green or intentionally skipped, including full typecheck, build, unit and browser suites. Greptile is 5/5 with no review findings or open threads. - The plan separately records a released-container overlap proof: 130 reads, 26 saves, zero failures, two deleted predecessor containers and a retained authenticated session. That image predates this restore change. ## Risks - A cold folder can use eight simultaneous streams instead of one. Two file-transfer database connections bound the simultaneous restore callers. Streams verify bytes without loading entire objects into memory. - The 30-second deadline is unchanged. The measured size and latency do not qualify every folder at the maximum storage limits. - No schema, API or UI changes. Paid-model and hosted deployment acceptance remain separate work. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 04:09:47 -06:00
DottaandPaperclip	d941ed3879	feat(server): verify existing-volume ephemeral enrollment (#18529) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud web servers can now keep durable state in PostgreSQL and S3. > - Existing servers still have files and a decision signing key on their volume. > - The volume must stay attached until every data owner verifies its migration. > - This pull request adds a dedicated maintenance process for that transition. > - Cloud can preserve the original key and remove the volume only after a matching receipt. ## Linked Issues or Issue Description Refs #18518 and #18526. Existing-volume enrollment is a required step in the zero-downtime web deployment work. This is the Core maintenance side; the Cloud transition is a separate private change. ## What Changed - Package a standalone maintenance entry that starts no normal app routes or execution workers. - Coordinate the existing agent-file, run-evidence and skill migration owners. Follow every page and verify shared bytes before final eligibility. - Require stopped writers. Refuse active execution, missing local bytes, local workspaces and external instructions. - Read the existing decision key without generating a replacement. Preserve normal key creation for ordinary server startup. - Require an attempt token for status and export. Export the original key only after successful migration. Keep status and errors free of secret material. - Document the Cloud ownership boundary, deadline, retry and retention requirements. ## Verification - Full workspace typecheck and build passed before the review fixes. The final server TypeScript compilation passed after correcting Drizzle's optional predicate type. - Production runtime bundling passed: 2,702 inputs, 122 chunks, including the maintenance executable. - The focused suite passed 21 cases using PostgreSQL and real S3-compatible storage. It covers multi-page migration of 26 agents, exact reads after deleting the predecessor home, corrupt-object rejection, active/missing-data refusal, and the bundled plain-Node process with authenticated export and clean SIGTERM exit. - Regression cases refuse unsaved terminal working copies, require explicit stopped/cleaned receipts for unavailable copies, clean 101 expired traces through the owning service, and enforce the deadline after migration succeeds. - Initial integration attempts exposed fixture cleanup ordering problems; corrected runs pass. CI also caught a Drizzle predicate type error, now fixed and compiled locally. - All repository CI gates passed on a4db955403, including typecheck, build, browser suites and serialized tests. Greptile reviewed that head at 5/5; all four review threads are resolved. Hosted enrollment and active-agent continuity remain separate acceptance work. ## Risks - This is an explicit maintenance entry. The caller must confirm all prior deployments stopped and keep normal tenant routing closed through detachment. - A timeout blocks export but an owner may finish its current bounded operation before observing cancellation. Cloud must confirm process exit before detaching. SIGTERM has a ten-second hard exit deadline. - The authenticated export contains the original signing key. The Cloud caller must put it only in the stack vault and must not log the body. - Local-workspace and external-instruction migrations remain their existing owners' responsibility. Enrollment fails closed until those data are eligible. - No database schema changes. Ordinary server startup is unchanged unless this maintenance entry is explicitly launched. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 03:56:44 -06:00
DottaandPaperclip	73cd8eb9a5	fix(server): deliver answers promptly during web handoff (#18530) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web servers can overlap while Railway replaces one deployment. > - An answer can reach the replacement while the original server owns the native execution. > - A competing delivery claim can make the owner miss the first notification. > - The normal retry interval then delays the answer by 30 seconds. > - This pull request gives pending answers a one-second retry in ephemeral mode and verifies delivery between separate processes. ## Linked Issues or Issue Description Refs #18518 and #18526. This closes a measured native-question delivery delay during control-plane web deployment overlap. ## What Changed - Register question delivery through its owning service. Use a one-second retry for ephemeral servers and retain the configured cadence for ordinary servers. - Keep the existing durable claims, response identity, authorization and continuation behavior. - Add a two-process PostgreSQL regression for a claim held by the non-owning server. Verify one command, settlement, idle behavior and reactivation by another committed answer. - Keep the assembled startup fixture on the real registration function. Update the deployment plan with current evidence and remaining acceptance work. ## Verification - The same local two-process race took 30.157 seconds before the fix and 1.197 seconds after it. The command sink is simulated; PostgreSQL, notification transport, interaction response, delivery claim and worker are real. - The cross-process regression and assembled startup suite passed 122 tests. No further sweeps or pending inspections occur after settlement. A second answer wakes the owner through PostgreSQL without a test-owned sweep. - Server TypeScript compilation passed. All PR CI gates passed on d1d426c0dc, including full tests, typecheck, build, browser shards and serialized suites. Greptile reviewed that head at 5/5 with no findings. - Live provider continuation, hosted staging deployments and production acceptance are still separate work. ## Risks - An unresolved answer causes more frequent reconciliation in ephemeral mode. Settled history does not keep the worker active. Existing claim fencing and bounded error attempts are unchanged. - A 30-second abandoned claim still needs its existing stale-claim recovery. This change addresses released claims, not proof of provider liveness. - The test simulates the provider command sink. It cannot establish that a paid model consumed the answer or that a remote execution survived a deploy. - No schema, API or UI changes. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 03:39:48 -06:00
DottaandPaperclip	16787e4167	Qualify rolling web deployments and preserve Stop through handoff (#18526) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 02:53:27 -06:00
DottaandPaperclip	2b2b5a43b7	fix(daytona): restore the recorded directory for file recovery (#18527) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped sandbox can hold workspace files that still need a durable copy. > - File recovery must reopen the exact allocation and directory recorded by the run. > - Workspace preparation can place that directory below the allocation root. > - Daytona resume returned the allocation root even when recovery needed the recorded child directory. > - This pull request lets the file-recovery owner request that exact directory. > - Ordinary resume keeps its allocation-root behavior. The recovery source checks stay strict. ## Linked Issues or Issue Description **What happened?** A stopped native run had a prepared remote workspace. The recovery service stopped and retained its allocation. The next file-recovery step failed with `source_binding_invalid` and `reopened_cwd_mismatch`. Daytona resumed the correct allocation and matched its sentinel, but returned the allocation root instead of the workspace directory recorded on the lease. **Expected behavior** The file-recovery owner must reopen its exact recorded directory without creating an allocation, an empty directory, or an agent turn. Ordinary execution resume must keep returning the allocation root for normal workspace preparation. **Steps to reproduce** 1. Prepare a task workspace below a Daytona allocation root. 2. Stop and retain the allocation with unsaved workspace files. 3. Request source-bound file recovery. 4. Before this change, the provider returns the allocation root and the strict cwd check rejects it. **Paperclip version or commit** Reproduced from `050365e09d88411960fc3bacc1c08c46c0d9b6fa`. **Deployment mode** Docker with the bundled Daytona provider. Local tests use synthetic provider boundaries and disposable PostgreSQL. Refs: #18517, #15749. Related retained-export work: #17916. That work does not change the Daytona resume handler. ## What Changed - Add an optional driver-only `requestedCwd` resume field to the SDK contract and existing host forwarders. - Set the field only after the owned file-recovery path verifies the recorded source. - Require Daytona to match the requested directory to the recorded lease directory. Require a normalized absolute path without NUL bytes. Keep the allocation and sentinel checks. - Keep ordinary resume unchanged when the field is absent. - Add provider, host, and real workspace-preparation regressions. Document the separate directory contracts. ## Verification - The original provider regression failed for reusable and ephemeral leases. The scoped regression also failed to read a real retained file from the wrong root. Ordinary resume controls passed before the fix. - Current provider handler: **345/345 passed**. This includes exact-directory, malformed, missing, foreign, sentinel, reusable, ephemeral, ordinary-root, and real-file cases. - Removing only the owned producer's cwd request makes its real PostgreSQL regression fail with the original `reopened_cwd_mismatch`; the source file was restored exactly afterward. - Current host selection: **10/10 passed** against disposable PostgreSQL 17. It covers the inspected owner, sandbox forwarding, generic plugin forwarding, normal resume without the field, same and later task workspace preparation, and ownership rejection. The same-workspace control retains saved bytes and does not append another workspace path. - Current SDK and Daytona builds, server build, module boundaries, maps, and diff checks passed. The server used its normal unchanged-Runner cache. Its canonical input digest matched the previously verified build, and the canonical assessor accepted the copied original stamp and outputs. The changed SDK and server compiled from current source. - Local package-wide validation exposed two unchanged 30-second timeouts in the 145,000-member tar-listing cases. A single-file control reproduced both timeouts. That test mocks this provider module. The failures remain recorded; no timeout changed. The earlier package run had 497 passes, two failures, and seven skips. It is historical evidence from the abandoned broader resume experiment, not final-source package acceptance. - Initial host attempts timed out during database setup before any assertion. The final host selection used the existing supported pre-migrated disposable template. No test timeout changed. - Greptile reviewed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no actionable issues, zero review threads**. [Current review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555253315). - The official [Ponytail review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555237850) also assessed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no findings**. Its check is terminal neutral; its completed source assessment is not a test-execution claim. - [Primary CI](https://github.com/paperclipai/paperclip/actions/runs/41286802674) initially lost its spot runner in the UI/CLI workspace lane. The log records an explicit shutdown signal and no test assertion result. UI, CLI, lane scripts and dependencies are unchanged; the adjacent identical-source baseline lane passed. The single failed-lane retry passed at the same source. The initial failure is retained. All exact-head CI gates are now green. - No live repair or image deployment is claimed by this PR. Commands: ```sh node packages/plugins/sandbox-providers/daytona/node_modules/vitest/vitest.mjs run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/plugin.test.ts node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/workspace-recovery.test.ts server/src/__tests__/environment-runtime.test.ts -t 'recovers a protected retained source|owned export directory|prepares the (same|later) task workspace|does not reopen a retained source|resumes only the exact stopped sandbox lifecycle|explicit export directory' pnpm -C packages/plugins/sandbox-providers/daytona build pnpm --filter @paperclipai/server build node scripts/check-module-boundaries.mjs node scripts/check-maps.mjs git diff --check ``` ## Risks - A provider that does not support the optional field can still return a different directory. Recovery keeps failing closed on that mismatch. - Ordinary resumes omit the field. The tests cover the same workspace and a later task so that normal preparation does not gain an extra path prefix. - The field grants no source ownership, stop proof, file-persistence proof, or execution authority. Existing allocation, company, task, run, claim, and cwd checks remain in force. - No schema migration, retry-policy change, model replay, or historical-row repair is included. Existing failed attempts remain historical facts. Deployed file recovery still needs its normal admission and independent persistence and physical-stop receipts. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use, and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 02:47:30 -06:00
DottaandPaperclip	006d5691a6	feat(server): support volume-free web server handoff (#18518) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 01:58:54 -06:00
DottaandPaperclip	050365e09d	Preserve native workspace files after lifecycle Stop (#18517) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent termination must stop execution and preserve unfinished workspace files. > - A native run can acknowledge a lifecycle Stop, then finish without an accepted result. > - The file preservation worker required a board actor even when the exact native Stop receipt was valid. > - This left a prepared remote workspace allocated without a file recovery action. > - This pull request admits the exact audited receipt for file preservation only. > - The existing recovery service can then save the files and stop the same allocation without another agent turn. ## Linked Issues or Issue Description **What happened?** A cancelled native run can have a terminal, unowned coordinator and a prepared remote workspace. Agent lifecycle cancellation records a valid run-only Stop receipt without a board user. The preservation worker rejects that receipt because it uses the board Stop predicate. The source lease remains active. A resolved execution watchdog does not supply the missing file recovery action. **Steps to reproduce** Seed a cancelled native run with a terminal-failure coordinator, no result or owner, an acknowledged and audited run-only Stop, and a prepared remote workspace. Omit board attribution. Start the registered cancellation preservation worker. Before this change, it records no workspace recovery marker. **Expected behavior** Record file recovery for the exact bound allocation. Keep the cancelled result, accounting, task state and watchdog history. Use the normal recovery service to stop, save and clean up that allocation without another provider turn. **Paperclip version or commit** Reproduced on source `1e8e32290c657227857aeda2620a73db52bab4f2` with real local PostgreSQL 17. The observed deployment uses a sandbox provider and remote workspace storage. Related current workspace work: #18514. Public PR and issue searches found no duplicate repair. ## What Changed - Expose the existing exact run-only Stop receipt check. Reject empty audit identifiers. - Use that receipt only in cancelled workspace preservation. Keep board Stop and continuation checks unchanged. - Defer preservation while the run still holds a live controller lease. - Schedule that wait at the actual controller lease expiry. A lease renewal observed under the preservation lock also sets the next deadline. - Add a PostgreSQL regression for the remote-workspace-only lifecycle record, resolved watchdog history, normal recovery admission, ownership controls, rollback and idle behavior. - Document the separate file recovery obligation and its physical cleanup limits. ## Verification - Before the fix: the registered-worker PostgreSQL regression fails because the file recovery marker is absent. - After the original repair at `6fd1042efd`: 20/20 focused tests and all five sibling suites pass, 320/320 tests. The exact-head server build and typecheck pass. - The repository reviewer then found that a live controller caused a fixed 30-second retry. Both the existing-lease and real row-lock renewal cases fail against that source. The new `adb1d4eccdd48be4c86906d5bd111a6e39f20191` schedules the actual recorded deadline before locking and after the locked reread. The lock still enforces the ownership guard. - At `adb1d4eccd`: all 22 focused PostgreSQL controls pass. The directly affected retention and board receipt sibling suites also pass, 124/124 tests across three files. They prove no intermediate reads or transactions before expiry, preservation after expiry, and no extra execution. Server typecheck and the exact-head server build pass. - Map checks, module boundary checks and `git diff --check` pass. - Run `pnpm exec vitest run server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts server/src/services/native-runtime/native-failed-workspace-recovery.test.ts server/src/services/acknowledged-native-stop.test.ts` with the supported external test-database variable set to a disposable local PostgreSQL database. The earlier broader run also covered run control and workspace recovery. - Exact-head [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41274963476) completed successfully. All 59 current-head checks are terminal with zero failures; required `ci / verify` and `ci / e2e` pass. Both [Greptile](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553528545) and the [repository assessment](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553542139) reviewed exact `adb1d4eccd` at 5/5, with zero findings or unresolved threads. The repository assessment's check concludes `NEUTRAL`; the validated assessment itself is 5/5. Earlier reviews, including the valid 4/5 deadline finding at `6fd1042efd`, remain in review history. The [earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/41273965963) is also retained. No old result substitutes for a current-head gate. - Real sandbox recovery and physical cleanup remain unverified. Earlier failed QA outcomes remain failed. This PR changes no test grade or historical row. - Repo-wide local tests and typecheck were not run. The focused server checks cover this change; canonical CI supplies the remaining repository gates. ## Risks - The newly admitted receipt permits only file preservation. It cannot authorize a continuation, accept a native result or certify provider termination. - Recovery still requires exact company, task, run, lease, allocation and workspace bindings. Accepted results and live owners remain with their existing owners. - Provider or storage failure can leave recovery pending. The source must remain retained until saved-file and physical termination receipts exist. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 00:17:46 -05:00
DottaandPaperclip	1e8e32290c	Give metadata project tasks private remote workspaces (#18514) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task can belong to a project without using that project's files. > - The specification gives a task without a selected or configured source its own private folder. > - The remote source planner instead rejects such a task when its project has no configured workspace. > - This stops an ordinary delegated task before native execution starts. > - This pull request selects an empty, task-owned remote source for that narrow default case. > - Explicit source choices, existing workspace authority, and repository requirements retain their current checks. ## Linked Issues or Issue Description Refs #18504 and #18351. The earlier reference-project fix covered additional project mentions. This fix covers the task's own organizational project. **What happened?** A task in a project with no configured workspace fails with `remote_workspace_source_unavailable`. Its project has no workspace policy, its four workspace selection and binding fields are null, and it requests no repository, branch, or local folder. The failure occurs before native runtime selection. A derived project folder exists in the project facade even when no source is configured; that derived path does not grant file access. **Expected behavior** Use the task's private remote workspace. Keep the project on the task for organization. Do not adopt the project folder, agent home, or another repository. Preserve explicit workspace and source requirements. **Steps to reproduce** 1. Create a project without configured workspaces or a workspace policy. 2. Assign a normal task in that project to a native sandbox agent. Leave all workspace selections and bindings unset. 3. Start the task with remote workspace storage enabled. 4. On the prior source, observe the source-unavailable error before allocation. On this source, observe a company-owned remote workspace whose `sourceIssueId` is the task and whose filesystem project and source IDs are null. **Paperclip version or commit** Reproduced against the deployed source planner at `756599e4ea069c9338c320fe8846523d3dfb92a0`. This branch starts at `987e0932bf66e703750cd04698b3c7e8993580a7`; the relevant planner and preparation inputs at that parent match the deployed source. **Deployment mode** Native sandbox execution with remote workspace storage. The regression tests use real PostgreSQL with the current migrations. ## What Changed - Allow an implicit empty source only after a company-scoped query confirms the project has no workspace rows and no explicit workspace policy. - Keep existing bindings, pending selections, explicit source choices, Git requirements, and local folder choices outside this default. - Add real PostgreSQL source controls and an assembled heartbeat preparation control through workspace reservation and the environment acquisition boundary. - Clarify the existing task-owned folder contract in the implementation specification. ## Verification - Causal RED: the new source control and the assembled allocation control both fail against the unchanged deployed planner with the observed source-unavailable error. - GREEN: 19 focused PostgreSQL controls pass across the source planner and assembled heartbeat preparation suites. The allocation control checks the persisted company/task owner, null filesystem project IDs, private remote root, original organizational project, and unchanged project and host sentinel files. - Negative controls retain configured repository sources and reject a configured source without a URL, an explicit missing source, an unknown selection, a required project policy, a required branch, an explicit local folder, a foreign project, and unsupported read-only project sources. - Current-source transitive TypeScript checks and `node scripts/check-maps.mjs` pass. The map check is structural evidence only. - Fresh exact-head Greptile is 5/5 with zero unresolved threads at `680461e180232d0f4a057528617e39b64b4f0e3a`. [Review](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552946468). The official [Paperclip assessment](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552931726) is 4/5 and remains recorded unchanged. A separate non-author source review found no actionable issue after the full producer/consumer trace, with manual confidence 4/5. Neither static review ran the tests or proved deployed native execution. - Full repository tests, build, and type checks passed in [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41269733360). All 59 checks are terminal: 55 success, three skipped, and the official Paperclip check neutral; no failed or pending checks. The successful build checked out merge preview `03dce9f9bc708e778dd214ca85dfa1ba58e0a806`, with parents current main `9dbbfa4fb298448db83ace946b338bff321d8b2a` and this exact head. Its four helper blobs match this head; the four added mainline document-harness paths remain unchanged. Local tests use source aliases with the current DB, shared, and Runner source, rather than borrowed generated declarations. - Actual QA staging qualification is pending deployment. The previous failed journey and cleanup evidence remain failed; these local checks do not relabel them. ### Review follow-up - Fresh workspace retry: the normal producer sets `taskDirectory:true` for this empty-source case. [Heartbeat](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat.ts#L2288-L2296) separates the saved filesystem project from the organizational project. [Environment selection](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat/environment-selection.ts#L506) selects the fresh task directory. Preparation passes that flag, and the planner's existing task-directory return runs before the new binding guard. The proposed fresh failure omits that producer flag. The existing fresh-intent and binding checks stay strict. This is source evidence, not an extra runtime test claim. - Empty policy object: the [public project policy schema](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/packages/shared/src/validators/project.ts#L24-L43) requires `enabled:boolean`. The project PATCH schema makes outer fields optional and does not make nested policy fields optional. A clear uses `null`; `{}` is rejected. A defensive parser's treatment of unknown stored data does not establish a valid `{}` producer. Keep the narrow fallback closed for non-null policies. - Boolean grouping: JavaScript evaluates the existing predicate as `sourceId || (projectId && !implicitTaskDirectory)`. The explicit-source and metadata-default controls cover those branches. The optional parentheses suggestion does not identify a behavior change; this reviewed source remains frozen. - The official 4/5 assessment and its static verification limits remain visible. The independent manual review withdrew its initial fresh-retry concern after the full producer trace. Actual staging qualification still requires deployment and a new real journey result. ## Risks - The new query runs only for a source-less default admission. It reads one authorized project and uses `NOT EXISTS` for that project's source rows. It adds no recurring poll, write, provider call, or local folder access. - A project with an explicit workspace policy remains subject to the prior source requirement. This conservative rule does not reinterpret policy fields. - The allocation control uses the real planner and PostgreSQL reservation. It stops at environment acquisition, so it proves workspace ownership rather than provider filesystem execution. Live staging qualification is still required. - No API, database schema, permission, model, budget, run limit, or QA journey oracle changes are included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This session does not expose a more precise backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 23:09:30 -05:00
5 people	34f8d25089	fix(github): expose review finding categories (#18429) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - GitHub review bots submit structured assessments through task-bound tools. > - Each repository can set its own finding categories. > - The schema accepts a string, but the server rejects values outside the saved review policy. > - Review instructions did not list the accepted values. The error did not list them either. > - This change gives agents the values before submission and keeps server validation. ## Linked Issues or Issue Description **What happened?** Review submissions failed with an unknown finding category. The agent corrected the assessment before publication. The sampled trace has two rejections followed by a processed review. Historical tool inputs are null. Authorized diagnostic inspection found no saved raw provider trace for the representative run. The exact rejected values remain unknown. **Expected behavior** Agents can find the accepted categories before submission. Unknown values still fail validation. **Steps to reproduce** Set a custom finding category. Start a review. Submit a different category. Before this change, the instructions and error do not list the accepted value. ## What Changed - List configured categories in manual and automatic review instructions. - Return categories from the saved assessment policy in `begin_review`. - Describe the dynamic constraint in the shared discovery schema. Keep custom categories valid. - List accepted values in category errors. Keep authorization and publication checks. - Add contract and database integration tests. Update documentation and access maps. ### Hypothesis and thesis for a fix The missing category guidance can cause avoidable submission retries. A model can also ignore the policy returned by metadata reads; the historical inputs do not establish which occurred. The repair belongs to the GitHub review contract. Use the saved policy categories at review start, explain the constraint, and retain server rejection. A fixed global enum would reject valid custom categories. ## Verification - Baseline `f1d7808`: two new contract assertions failed for missing category guidance and error detail. The same assertions pass after the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts server/src/services/chat-github-tools.test.ts`: 41 passed. - `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'returns snapshot finding categories'`: 1 passed. Real PostgreSQL and service code; simulated GitHub HTTP. Checks policy changes, rejection before publication, accepted custom category persistence, and processed publication. - `node scripts/check-maps.mjs`: passed. Structural evidence only. - Shared TypeScript check passed. Server TypeScript checking was killed by the 8 GiB memory limit, including a separate retry after the other checks stopped. - Full typecheck and build could not complete: `cargo` is absent. Full tests were stopped after concurrent checks exhausted memory. CI must supply the full checks. - Follow-up compiler repair: the unchanged baseline reproduces TS2352 in the category discovery test. Runtime matchers replace the unsafe schema cast. They retain both guidance and custom-category assertions. The repair is commit `0dcb94c676`, with current master merged at `14d7052854`. The final head is `c46b8f35add9255fc71be294d27f5a7865bcb47f`; the remaining code-comment review finding was removed and resolved. - Final-head CI: all gates pass, with no failed or pending checks. Greptile gives this head 5/5 with no outstanding findings; the only inline thread is resolved. The workspace test shard passed on one bounded rerun after an earlier CLI child-server readiness timeout; no code change was made for that rerun. - The review host passed full recursive typecheck and build after the compiler repair and master integration. Fresh database rechecks hit the host PostgreSQL semaphore limit. The full local test command hit a pre-existing chat line-selection failure before assertions. These local limits do not replace the passing final-head CI evidence. - No live model or GitHub publication at the changed revision. No deployment or merge. ## Risks - Instructions add policy text. An agent can still ignore it. - Existing bot catalogs get the schema description when tools are synchronized. Trusted review instructions and `begin_review` also expose categories. - The service test uses simulated GitHub HTTP. It does not prove that deployed agents avoid retries. ## Model Used The original worker used OpenAI `gpt-6.1-sol` through Codex with reasoning, tool use, and code execution. A later Codex review repaired the test compiler error. That review runtime exposes GPT-6 but no exact snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com> Co-authored-by: cryppadotta <34892728+cryppadotta@users.noreply.github.com>	2026-10-31 22:06:32 -05:00
DottaandPaperclip	e09f40888a	fix(recovery): preserve native task access during resolution (#18489) ... ## Thinking Path > - Paperclip controls agent execution and task ownership. > - A native run needs its task binding until it reports its result. > - Direct task updates already reject self-status changes that remove this binding. > - Recovery resolution can also change the source task status. > - That route did not use the same protection. > - This change rejects the unsafe status change before the task or recovery receipt changes. > - The run can then use semantic completion while normal recovery permissions remain effective. ## Linked Issues or Issue Description Refs #18411. **What happened?** A running native owner can resolve an older recovery action with `sourceIssueStatus: "in_review"` or `"done"`. The request succeeds and changes its own task status. That can release the execution binding before the run reports. Later task tools fail with `paperclip_runner_tool_binding_not_authorized`. **Expected behavior** Return a conflict before the task status or action receipt changes. Tell the run to report through `paperclip_finish` or `paperclip_block`. Preserve unchanged-status resolution, Board recovery, legacy-agent behavior, and the existing conversation and handoff rules. **Steps to reproduce** 1. Give a running native owner the task execution lock and an older recovery action. 2. Call `POST /api/issues/{id}/recovery-actions/resolve` as that run. Use `outcome: "restored"` and `sourceIssueStatus: "in_review"`. 3. Before this fix, the route returns 200 and changes the status. After this fix, it returns 409 with `native_run_status_reported_by_finish` and makes no task or action mutation. ## What Changed - Reuse the existing native self-status guard inside recovery resolution's source-task transaction. - Let the guard receive the requested status from either route contract. - Add HTTP route regressions for rejected review/completion transitions and continued task editing after rejection. - Cover unchanged-status native resolution, Board recovery and legacy-agent resolution. - Update execution semantics and both recovery maps. ## Verification - Baseline `91fdf02`: the two new rejection cases fail because the real Express route returns 200. Three safety cases pass. - Fixed source: all 198 tests in `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` pass. - `node scripts/check-maps.mjs` passes. This is structural evidence. - Greptile and Paperclip Review both give the integrated head 5/5 with no actionable findings. - Full recursive typecheck, full build, and a repeat of all 198 route tests pass at integrated head `0fa104dfb240500acd01303cd446363ea39c6c75`. All current-head CI gates pass (55 successful checks/statuses; no failed or pending checks). - The full local `pnpm test:run` command fails in chat-test line selection with `No test found`, before assertions. This also occurs with the unchanged master chat test file. It is not a passing full local suite. - The route tests use the real route, validation and guard. They mock service and database boundaries. The observed hosted failure establishes the original mechanism. Fixed-source live provider acceptance remains unverified. No deployment or merge is part of this PR. ## Risks - A native owner now receives a conflict for a previously accepted recovery status change. It must use the existing semantic result path. - This prevents a new binding loss. It does not prove that an old remote executor stopped or recover missing provider state. - Board, legacy, conversation and handoff behavior retains existing authority. The regression includes representative allowed paths. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (198 focused route cases; the full local command has the recorded selection failure) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 22:06:08 -05:00
DottaandPaperclip	756599e4ea	fix(workspaces): keep project references without source folders (#18504) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can receive project references in normal task and chat messages. > - Remote workspace preparation checks those references before a native run starts. > - A project with no workspace source has no folder to attach. > - The current check rejects that project as an unsupported read-only folder. > - This pull request keeps the project reference and reports the missing source without attaching files. > - Projects with configured sources keep the current remote access restrictions. ## Linked Issues or Issue Description Refs #18403 and #18351. This is a product prerequisite for the hiring journey. It changes no journey code. **What happened?** A native chat run failed before execution when its message referenced an authorized project with zero configured workspaces. The error was `workspace_validation_failed`: “This remote runtime cannot safely provide read-only project folders.” The project had no repository or checkout. The failed QA run and original grade remain preserved. **Expected behavior** Keep the task workspace when a referenced project has no configured source. Report that no project folder was attached. Continue to reject configured cross-project sources that require an unsupported read-only mount. **Steps to reproduce** 1. Use a native remote task folder with multi-project references enabled. 2. Mention a same-company project that has no configured workspaces in a normal comment. 3. Prepare the run workspace. **Paperclip version or commit** Reproduced on `b889ac39b7fbed558c5f4cfbbc2b73c4ca916810` with the real workspace resolver and PostgreSQL migrations. The staging failure was recorded in workflow [41253428446](https://github.com/paperclipai/paperclip-cloud/actions/runs/41253428446). ## What Changed - Distinguish authorized project references with a known empty workspace list from configured sources. - Return a resolution warning and failure receipt for an absent source. Do not attach a folder or create a workspace. - Preserve project authorization, company filtering, the disabled remote-source gate, and the read-only restriction for configured sources. - Add real PostgreSQL regression cases and document the reference contract. ## Verification - Causal RED on unchanged `b889`: the metadata-only project case failed with the same read-only error. The real repository negative already passed. - GREEN: eight PostgreSQL cases pass. They cover an absent source with the remote-source switch both on and off, a saved existing workspace, a repository, a configured source without a repository URL, the disabled remote gate, mixed references, and a foreign company. - The existing assembled workspace test also passes: a real read-only project source is rejected before reservation or environment acquisition. Total: 9 selected tests pass; 119 unrelated tests were not selected. - Tests use PostgreSQL 17 and the current source migrations. The local Vitest configuration selects current DB, shared, and Runner sources because the borrowed dependency output predates the remote workspace feature. - Current-source TypeScript checking of both changed TypeScript files and their dependency graph passes. It uses the repository's real Express actor declaration. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Canonical reproduction: `pnpm exec vitest run --project server server/src/services/heartbeat/remote-workspace-source.test.ts` with the existing embedded PostgreSQL support or `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` set to a disposable test cluster. - Historical `0f2b` passed all 59 CI checks. A separate non-author manual source audit reported no actionable findings at confidence 4/5; it did not rerun tests or qualify live QA. The official 4/5 review requested a flag-scope test and saved-workspace control. This test-and-documentation update adds both, with production source bytes unchanged. - Current-head canonical [CI 41258109282](https://github.com/paperclipai/paperclip/actions/runs/41258109282) passed: all 59 checks are terminal (55 success, 3 skipped, 1 neutral), with no failures or pending checks. Build and typecheck passed. The eight-case regression file passed in the canonical Linux server 1/12 lane. Its checkout `6f69ccca70d10e78885b133fb54a80940bf175a5` has parents `b889` and this exact head; its tree equals this head. - Fresh Greptile and official Ponytail assessments each give 5/5 for exact `27660027d12ff31427d611019e90dabfb6d8503a`, with zero unresolved review threads. The separate manual audit remains 4/5. - Live QA remains pending on a reviewed app build. This regression proof does not claim that the hiring journey passed. ## Risks A project source could be added after project hydration. This change uses the same run-preparation source snapshot as the existing authorization and source check. It only permits a positively known empty workspace list. Configured sources, unknown shapes, foreign projects, and existing execution authority keep their current checks. No schema, credentials, provider, model, budget, or filesystem permissions change. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 20:19:28 -05:00
DottaandPaperclip	66e8fab944	fix: complete native chat responses and improve API discovery (#18364) ... ## Thinking Path > - Paperclip must perform work and report its outcome accurately. > - A chat response and its delegated or operator-owned work are separate stages. > - Native chat contracts instead required literal completion of the user's whole request. > - Agents kept searching or waiting even when a truthful handoff should finish the response. > - OpenCode also has separate acknowledgement backlog and empty-response failure paths. > - This PR aligns chat completion with its current stage and makes OpenCode completion truthful. ## Linked Issues or Issue Description Refs #18098 for retained OpenCode evidence and #18377 for separate eval setup, oracle and reporting repairs. **What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP timeout behind 800 text fragments. Pilot [41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525) separately exposed a fabricated successful reply after a reasoning-only response. Later restart attempts timed out without an answer. Their native criterion literally required restarting the instance, although the chat instructions and authority required an operator handoff. General authority guidance alone did not resolve it. **Expected behavior:** A standing chat completes its current authorized response. It performs and verifies permitted actions, verifies delegated handoffs, or explains the specific unavailable action and operator path. The worker's task remains separate. Ordinary execution tasks must still complete their requested work. Empty provider responses must not become invented success. **Version and reproduction:** Actual pinned OpenCode and native Rust Runner, with retained full-stack AWS attempts. Product baseline master `51ea624344`. This PR adds no historical catalog to master. ## What Changed - Build standing-chat completion contracts from the server-owned conversation identity. Scope each current request to the response stage; preserve request text, criterion IDs, order, human-response scope and source revisions. Require verified authorized actions and truthful handoffs. Ordinary task contracts stay strict. Existing admitted inputs retain their saved contract during recovery; new contract content uses the existing revision/hash mechanism. - Guide prospective defaults to their owning resource and preserve existing work unless backfill is requested. Guide unavailable board/operator operations to the supported handoff. These are general chat instructions with no eval identities or expected answers. - Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes. Preserve text, metadata and order; flush before non-text frames and shutdown. - Give an empty OpenCode response one correction within its existing turn, model, reasoning, context and deadline. Accept actual completed answer parts with incomplete headers. Preserve actual active tool work, bound idle evidence grace and fail truthfully if correction stays empty. - Reject empty OpenCode terminals in Rust without actual prose or an accepted semantic result. Never invent successful replies or evidence. Refresh generated project-discovery contracts. - Preserve accurate public API-discovery descriptions for existing project defaults and managed instance restart operations, including their field scope and authority limits. ## Verification **Qualified: [pilot 41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937) passes 3/3, with all 35 independent checks passing.** | Case | Checks | Full case duration | Evidence and cleanup | | --- | --- | --- | --- | | Instance restart | 9/9 | 140.1 s | complete | | Project defaults | 11/11 | 204.2 s | complete | | Project work | 15/15 | 204.0 s | complete | All three are original-attempt results from source `c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved conversation contracts were inspected, as were ordinary native worker contracts where present. Each protected archive includes the actual API state and final PNG; cleanup completes. Original queries, `openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks, concurrency one and zero eval retries are preserved. Earlier passes do not transfer into this qualification. Both current heads pass all checks: product **62** (56 success, one neutral, five skipped), catalog **56** (54 success, one neutral, one skipped). Both have exact-head Greptile **5/5**, zero unresolved review threads and clean mergeability. The three trusted private-evidence protocol files remain byte-identical to the dispatch workflow. Raw evidence stays protected. - **2,044 API discovery tests pass.** Red controls showed that “restart instance” selected unrelated drain/settings operations and “save project defaults” selected repository replacement. Accurate public summaries now rank the existing operations correctly; route registration preserves descriptions. No endpoint, permission or field behavior changes. - **112 real-database, authenticated HTTP, native Runner and OpenAPI consumer tests pass locally**, using disposable task-owned Docker PostgreSQL. It was removed afterward; shared IPC was untouched. These cover persisted contract selection, admitted native input and the model envelope together. The isolated CI suites also pass. - **77 focused contract/input/context tests pass.** Red controls catch the wrong conversation predicate; canonical identity requires both agent and user IDs, including the actual heartbeat projection. Ordinary tasks, agent-only issues and planner tasks remain strict. Coalesced requests preserve criterion identities, source hashes and human-response scope. - **144 OpenCode driver tests pass**; native core **441 pass, one ignored**; packaging **56 pass, six skipped**. Server typecheck and build pass. - Real pinned OpenCode/native Runner: a delayed 25-second acknowledgement completes within the original 30-second MCP deadline. The text-fragment control changes 817 notifications / 54.1 seconds / timeout to 45 / 19.0 seconds / accepted finish. Separate fast, six-minute and real authenticated API probes complete 32 tool calls plus finish. These do not reproduce every remaining historical MCP timeout, and no common cause is claimed for all such failures. Retained failures remain attributed to their original source. Earlier runs exposed fabricated empty-response success, prospective-default backfill, a literal whole-task chat criterion, fixture readiness and oracle/reporting defects. The immediately preceding [41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608), product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15; restart 4/9 with the repaired contract actually admitted but no answer; defaults timed out before settlement, then exceeded the unchanged 256 MiB original-evidence allowance. Defaults domain behavior on that attempt remains inconclusive. #18377 retains the full attempt ledger, including missing archives and pre-paid cancellations. The fresh 3/3 establishes these selected outcomes; it does not establish full-catalog reliability. ## Risks The shared contract change applies to standing native chats across providers. It changes model instructions and completion scope, while permission, approval and review gates remain enforced by their existing owners. Incorrect model claims remain possible; the product evals independently require actual mutations, native work and delivery. The correction adds at most one provider request within the existing deadline; text batching can delay adjacent text by 50 ms. Three qualified attempts do not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use, code execution and live eval tooling. Exact hosted model revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] Final same-source three-cell pilot passes 3/3 with required evidence and cleanup --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 19:52:42 -05:00
DottaandPaperclip	b889ac39b7	fix: serialize native completion replies and bounded retries (#18502) ... ## Thinking Path > - Paperclip manages agents and their work. > - A delegated task reports its result to the source conversation through a durable completion outbox. > - Native execution can succeed before the presentation worker saves the accepted final reply. > - The outbox treated that short interval as a missing answer and cleared the target run. > - The presentation worker then rejected the valid answer because its delivery no longer belonged to that run. > - This change retains the existing target for one fixed presentation grace while accepted presentation is pending. > - The existing presentation owner can save the reply without another model run or more budget. ## Linked Issues or Issue Description **What happened?** A completion-delivery retry can run immediately after a native callback succeeds. It clears the callback's delivery target before the presentation worker saves the accepted answer. The reply then fails with `chat_completion_superseded`. A daily run limit correctly blocks the unnecessary replacement run, leaving the conversation active without its saved answer. **Expected behavior** The outbox must retain its current target and attempt while that exact native run has accepted presentation work. The existing presentation worker must save the answer once, acknowledge the delivery, and settle the conversation. Existing actor, session, result, digest, budget and retry checks must remain enforced. **Steps to reproduce** 1. Accept a native DONE result and current-turn final message for a completion callback. 2. Run its due completion-outbox handler after terminal projection but before reply presentation. 3. Run the existing presentation worker. Before this fix it rejects the reply because the outbox has cleared the target. **Paperclip version or commit** Reproduced on `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f` through the real PostgreSQL outbox and presentation services. The reported failure remains preserved. A new QA run is required after deployment; local tests do not qualify the deployed fix. Related: #17696 schedules accepted-response recovery. #18118 preserves canonical completion receipts. #18364 handles provider output and response-stage contracts. Those changes do not protect the completion-outbox target during this interval. ## What Changed - Defer a new execution attempt when the existing selector finds pending presentation for the same company, conversation, run and agent. - Retain the queued delivery target and attempt with an atomic compare-and-set. Await the existing scoped worker notification before that write in the same transaction. - Serialize retry with reply insertion under the existing child-task-first lock. Recheck locked task Done state and the delivery's stable completion-event version, status, target, attempt and exact lease. A delivered, retargeted or newer-claim winner prevents another wake. Admit the next attempt only after the winning transaction commits and releases its locks. - Limit presentation grace to five existing one-minute delivery leases, measured from the original run `finishedAt`. Clamp both the delivery lease and the native retry deadline to that immutable limit. Repeated sweeps cannot extend it. After expiry, resume the existing retry, exhaustion and daily-limit checks. - Keep publication authority in the existing materializer. The candidate query only defers a retry. Read the freshly reloaded delivery status, target and saved receipt after acquiring its child-task lock. - Fence both hold and retry writes with the exact claimed event version, attempt and lease. A stalled old hold cannot overwrite a newer lease. - Keep completion-event identity separate from mutable task ownership versions. Ordinary Done reassignment preserves its existing completion event; a real reopen retains normal supersession. - Add real PostgreSQL coverage for the observed ordering, outer commit and rollback, one saved reply, conversation settlement, idle SQL, retry boundaries and superseded deliveries. - Document the completion-outbox and presentation contract. ## Verification - Before the fix, the real PostgreSQL regression fails with `chat_completion_superseded` in `existingChatCompletionReply` when the outbox handles the terminal run first. After the fix, the same test passes without a new model wake. - Focused command: `vitest run --project @paperclipai/server server/src/services/native-runtime/agent-chat-response-recovery.test.ts server/src/__tests__/chat-completion-delivery.test.ts` — **95 pass** on `45f53a8c32fcca0bbdacf717286d37e30c1ba308`, using a disposable database on an owned local PostgreSQL 17 container. - Controls preserve retries for failed, cancelled, missing-final, missing-accepted-result, old-generation, foreign-company and foreign-agent targets. Reset or superseded task versions retain supersession. Digest and external-context declines never authorize a reply, never extend the fixed window, and leave it at exact expiry. An external native deadline cannot extend the limit. A valid reply at the last millisecond saves once without another model run. - The first reviewed head `b6f9c484b9567bc60af6d684c04e00058c1048ed` received Greptile 5/5 but an independent 3/5 review correctly found an unbounded hold. Three new real PostgreSQL controls failed on that source. This head adds the fixed bound and makes them pass. The earlier review, controls and CI remain retained; its review score does not qualify the new head. - The bounded predecessor `7300d2a978a5c4b3c13d85899bed3e98ae3c49fa` received an independent 4/5 review that correctly found an expiry-versus-in-flight-reply race. A real PostgreSQL advisory gate reproduced it after reply authority checked the delivery but before insertion. The retry lock and exact claim CAS now make that control pass. Both orderings, outer rollback, delivered/retargeted/superseded winners and competing lease claims pass. The next-wake test can acquire both parent and child locks, proving dispatch occurs after release. The valid 4/5 review and failing control remain retained. - At `39dd86e8f51d937cf382b2757736e0477201d21d`, both reviews found valid remaining defects: the reply helper still used its pre-lock delivery snapshot, and ordinary Done reassignment could stall a delivery by changing the task version. Four real PostgreSQL controls fail on that source and pass here: concurrent retry-first, a newly saved receipt after waiting, and reassignment within grace or at expiry. A real reopen through the issue service still supersedes the old event. - A later coordinator review found the hold CAS still lacked the attempt and lease fences already present on retry. A real PostgreSQL producer barrier reproduced an old hold shortening a newer claim from 120 seconds to 60 seconds. The same exact claim fences now prevent that overwrite. This failed control is retained; no old run was blindly retried. - The real worker saves one exact run-owned reply and acknowledges the delivery only after outer commit. An outer rollback publishes no reply. After settlement, advancing the scheduler clock by 60 seconds causes no further SQL. - Module boundaries, map structure, `git diff --check` and the canonical server build (including TypeScript and Runner vendor preparation) pass. The first local build stopped in the host package-manager setup; the installed pinned pnpm completed the build. Full repository typecheck, test and build gates pass in current-head CI; they have not all been run locally. - Final local checks also caught a test cleanup type error (TS2677) and one 94/95 readiness failure: the worker saves its comment before post-commit conversation settlement finishes. The test-only follow-up removes an invalid cleanup type predicate and waits on the existing bounded worker-quiet check before reading final state. All assertions and the same five-second bound remain. Source `45f53a8c32fcca0bbdacf717286d37e30c1ba308` passes all 95 cases and the canonical server build. The earlier failures are retained. - The final policy cleanup removes only 15 newly added narrative comment lines (seven service, eight test), as required by CONTRIBUTING.md. Both TypeScript AST signatures and `removeComments=true` emitted JavaScript are identical to tested source `45f53a8c32`. This was checked with the TypeScript 6.0.3 API and the installed native TypeScript 7.0.2 compiler. Assertions, timeouts and behavior are unchanged; the 95 tests were not repeated for identical compiled code. - The superseded `45f53a8c32` CI run recorded two 15-second timeouts in attention-retention cases. That failure remains retained. The test and its ten inspected owner or support files are identical on the tested master revisions. The canonical base run passes all 53 attention cases, and the comparison PR passes the same server lane, including those two cases in 9.3 and 9.8 seconds. This does not establish the timeout cause. Current-head CI is a new run after the required comment cleanup, not a retry of the failed source. - The optional Paperclip review bot ended neutral without a completed assessment because its execution paused. That result is not an approval. A separate [independent read-only audit](https://github.com/paperclipai/paperclip/pull/18502#issuecomment-6550216227) reports 5/5 confidence on exact head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c`, with no findings. Its manual assessment does not replace or relabel the incomplete bot result. - Current head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c` has a fresh Greptile 5/5 review with no open threads. [CI run 41249007430](https://github.com/paperclipai/paperclip/actions/runs/41249007430) passes all primary jobs and both required gates (`ci / verify` and `ci / e2e`). All 59 PR contexts are terminal, with no failed contexts. The actual CI checkout is `18db9e2750d6a799f26b5b8ba31bcccbea254227`, which combines master `681f7b77262a9df6c1b4aad55291bf960061e491` with this exact head. Fresh QA verification on the deployed application remains required before claiming the live failure is fixed. ## Risks A candidate can retain a delivery whose stricter presentation checks still prevent publication, but only within the fixed five-minute window. The existing worker keeps publication authority. If a valid reply remains unavailable at expiry, the normal callback retry may require another admitted model run; the original result and completed child work remain preserved. Existing retry, exhaustion, model-run and spending limits still apply. Failed or cancelled execution and genuinely missing accepted output retain the existing retry path. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use and code execution. The runtime does not expose the exact hosted revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 18:07:41 -05:00
DottaandPaperclip	681f7b7726	perf(server): retire aggregate maintenance polling (#18425) ... Reviewed and verified at 901c4c4bdc. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 17:22:31 -05:00
DottaandPaperclip	9ca9af544a	Drive remaining live run and resource recovery from durable work (#18419) ... Reviewed and verified at 6a4c193906. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 16:00:35 -05:00
DottaandPaperclip	eb12cfc96b	perf(github): activate connection work from events and deadlines (#18395) ... Reviewed and verified at f77620b9a4. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 15:38:43 -05:00
DottaandPaperclip	5e4cc01073	feat(workspaces): keep remote files off the webserver disk (#18403) ... Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 14:54:01 -05:00
DottaandPaperclip	75fa37f30f	fix(server): expire native backups from retention deadlines (#18380) ... Reviewed and verified at 5b137b5cda. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 14:31:31 -05:00
DottaandPaperclip	453f5fbb78	perf(external-objects): schedule scoped refresh obligations (#18384) ... Reviewed and verified at 3e58e12f87. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 14:09:24 -05:00
DottaandPaperclip	c32ecf6657	fix(server): schedule only outstanding merge confirmations (#18382) ... Reviewed and verified at bc22051c56. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 13:44:46 -05:00
Nicky Leach	6484f96c28	fix(workspaces): inspect storage only when needed (#18410)	2026-10-31 11:43:18 -07:00
DottaandPaperclip	9a3f395737	fix(server): settle routine history without idle scans (#18399) ... Reviewed and verified at 0b65f490d9. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 13:19:31 -05:00
Dotta	3937215a05	Qualify explicit native written references in the basic question journey (#18459) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Basic task journeys must prove that people can answer an agent and receive saved work. > - The question journey asks for a choice, then a typed reference, then a saved note. > - A native choice card can offer a real written-answer control as well as predefined options. > - The old observer rejected that form before a person could type the reference. > - This pull request accepts that explicit native control and keeps exact answer and work evidence. > - The separate text-only and named-tool qualifications keep their stricter contracts. ## Linked Issues or Issue Description Refs #18346 and #18456. The integration inherits the reviewed fixture prerequisite #18387 and Agent Chat recovery #18465 through their actual merged parent. The producer fix in #18456 is separate from this basic journey contract correction. Public duplicate PR and issue searches found no other correction for this form equivalence. **What happened?** The basic question journey requires the reference card to have `answerMode: text`. It rejects a native single-select card even when its canonical form explicitly enables a written answer. This makes the observer test a widget shape in addition to the requested behavior. **Expected behavior** The person must make a real choice and type the complete reference. A native card with an explicit written-answer control can satisfy the second step. Selecting a predefined option cannot satisfy it. The agent must resume the bound work and save both actual answers in its own task document. The document must survive reload. **Steps to reproduce** Use the focused recording and real QuestionForm browser test in this PR. Keep the reference as a single-select native question with one predefined option and `customAnswer.enabled: true`. The previous grader rejects `question-resume.forms`. The previous browser helper rejects `single_select` before it uses the written-answer control. **Paperclip version / commit** Original source base `f2db909393de124caf6a5380a52d4e0e1d621661`. Current integration `a417fd827347ed6f7998774a9de6aee70b76d867` merges exactly `12b0cc630471ee151ec5bc8e41a34f09236d6723` into reviewed question source `faec4ce6d775a34d303d200987e4e84d6dc26caf`. The scheduled failure on `82d15946c5844d3ae41c6ade62fa224053f20a9a` remains a failure. Its saved card had no canonical written-answer capability and would still fail this change. The retained normalized events do not prove whether the original provider flag was omitted or false. **Deployment mode** Attached native task journeys, including QA Cloud. This PR changes the observer and grader. It does not change the app, provider prompt, model, budget, timeouts, cleanup, or runtime deployment. ## What Changed - Accept one provider-bound single-select reference card with explicit canonical custom answers and valid, distinct options. Keep pure-text reference support. - Make the browser select the actual written-answer control and fill the real rich input. Opt in only for the basic question journey. - Keep the exact no-option typed response, immutable card, board actor, native request and continuation, saved agent document, reload, model, environment, checkpoints, and existing one-to-three-run checks. - Reject missing or disabled custom capability, inferred legacy capability, malformed or extra questions, foreign native identities, changed cards, substituted options, and wrong or missing reference text. - Add a real production QuestionForm and TaskChatRichInput Chromium probe. It first selects a predefined option, then proves the helper clears it and submits only the exact custom text. - Document the contract correction and preserve the historical failed attempt and its separate cleanup outcome. Keep necessary-question and named-tool documentation qualification unchanged. ## Verification - Before the observer change, the focused grader regression failed only the new valid-written-reference case; the other 108 cases passed. - Before the browser helper change, real Chromium passed the text control and four rejection controls but failed the valid native written-answer control at the old pure-text assertion. - Initial focused Runner E2E unit selection at `af07f4de`: **256 passed across 9 files**. This includes question/resumption, clarification evidence/flow/collection/deadline, necessary question, documentation, contracts, and report catalog tests. - Initial real Chromium selection at `af07f4de`: **12 passed**. Six new production QuestionForm/control checks and all six existing document-readiness checks pass. No form or rich-input component is mocked. - Historical test-only head `faec4ce6d775a34d303d200987e4e84d6dc26caf` adds a direct canonical-absent legacy `freeText` grader negative and renders the actual legacy presentation in the browser. All four production helper blobs remain identical to the reviewed head. A temporary legacy inference mutant makes the new negative fail. After restoring production source, **179 focused unit tests and all 6 real QuestionForm checks pass**. The six document-readiness specs and their helper are unchanged. - Original [CI run 41205748589](https://github.com/paperclipai/paperclip/actions/runs/41205748589) failed the staged-listen Runner handoff test; that failure is preserved and was not retried. Its Runner source closure was identical to an earlier passing canonical lane. The symptom also recurred on other heads and is tracked independently; no infrastructure attribution or live pass is claimed. - Current integration `a417fd827347ed6f7998774a9de6aee70b76d867` preserves **all 12 question files byte-for-byte** from faec. Every other tracked path is identical to exact merged parent `12b0cc630471ee151ec5bc8e41a34f09236d6723`, which contains actual prerequisite `baf651c8d0122efc93f3ef03e891919eeb666215`. There is no manual conflict resolution or additional production edit. - On that integrated head: **257 focused Vitest tests across 9 files, 206 required Node source/reporter tests, and 12 real Chromium tests pass**. Map/module-boundary checks and catalog discovery pass. The actual `--preview` command passes with a synthetic question target and an intentionally unavailable adapter; it performs no credential read, adapter loading, provisioning or model call. It retains maxRuns3, a 600000ms campaign bound and zero retries. - Exact-head [CI run 41210806950](https://github.com/paperclipai/paperclip/actions/runs/41210806950) passes, including official typecheck, build, all eight browser shards, both Runner Vitest lanes and aggregate verify. All 59 current PR contexts are terminal with no failures. Fresh [Greptile summary](https://github.com/paperclipai/paperclip/pull/18459#issuecomment-6543878408) explicitly reviews `a417fd827347ed6f7998774a9de6aee70b76d867` at 5/5 with no findings; its exact-head check succeeds and all threads are resolved. No fresh deployed journey pass is claimed yet. - Catalog discovery lists the existing question journey. Map checks and module-boundary checks pass. Diff checks pass. - `tsc -p tests/runner-e2e/tsconfig.json` was attempted. The fresh integrated checkout still lacks compiled Runner, plugin SDK, and adapter declarations. The attempted local check reports 344 diagnostic lines from dependency/type closure and zero diagnostics in all 12 question-owned paths. Official exact-head CI establishes the full typecheck/build gates. Full repository build, typecheck, and test suites were not repeated locally for this observer change. - The first broad local report selection lacked the CLI loader and then hit sandbox IPC restrictions. Locked dependency installation and an authorized local IPC run resolved those setup errors; the full selected 256 tests pass. The initial localhost probe also needed the normal local-listen permission. - No fresh QA pass is claimed. The coordinator must qualify the final reviewed runtime and harness together. Prior failed results remain preserved. ## Risks The basic journey accepts one additional form with explicit native provenance. It still rejects the original closed card without a written-answer capability. The separate producer default fix must reach the selected remote provider pack before it can affect a deployed run. Local browser and grader tests do not prove that artifact served or that a model completed the live journey. Existing cleanup and physical-stop obligations remain separate. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, browser testing, and code execution. The host does not expose the exact underlying model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked related work above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run the selected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-31 13:16:40 -05:00
DottaandPaperclip	dce383c733	perf(server): stop recurring attention retention scans (#18388) ... Reviewed and verified at a61af9888e. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 12:28:40 -05:00
DottaandPaperclip	b6b57079b9	fix: evaluate status cards on committed changes and deadlines (#18401) ... Reviewed and verified at 21795e222a. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 11:41:59 -05:00
DottaandPaperclip	e40de0c9e8	perf(ci): refresh test weights and narrow graph dependencies (#18464) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, reliable CI feedback. > - Diff-aware CI selects test files and partitions them into runner jobs. > - Stale cost estimates put several slow suites on the same runner. Broad imports also couple small tests to large modules. > - This PR refreshes test costs, runs independent graph discovery in parallel, and narrows three dependency boundaries. > - Selection safeguards and CODEOWNER-only AWS routing stay in force. ## Linked Issues or Issue Description Refs #18398. This follows the audit of [PR #18427's verification run](https://github.com/paperclipai/paperclip/actions/runs/41169978784). The audit found 403 selected general-server files without timing weights. Three suites measured at 232s, 122s and 116s shared one shard because each received a roughly 2s fallback weight. Local graph discovery across four projects took 43.7s serially and 21.7s with two workers. A reverse-order repeat took 37.7s serially and 22.6s with two workers. Both comparisons produced identical selection results. This is a local measurement, not a claim about production CI speed. ## What Changed - Refresh general-server costs for 1,285 of 1,309 current files and serialized costs for all 163 files from two successful protected-master Cloud readiness runs. Retain prior costs for unmeasured files, including dedicated chat and native-runner suites. - Record passing server file costs, including collection, setup and environment preparation. A weekly hosted workflow validates trusted source runs and opens a timing-only PR. It tests shard coverage, pauses while a refresh PR is open, and never merges automatically. - Discover at most two project graphs concurrently in isolated, heap-limited Node processes. Cache repeated inventory reads within one selection. Worker errors and timeouts retain the full task plan. - Fetch only the merge commit and parents for the selector. A real shallow-clone regression proves the complete diff still includes more than 300 files, renames and deletions. - Share the identical log viewer and run details between both AgentDetail pages. The viewer test imports the shared implementation and runs each of its 13 distinct cases once. - Extract the pure API error class while preserving the API client's public exports. Import delivery work signals through their existing narrow DB export. ## Verification - PASS: `pnpm -r typecheck` and `pnpm build`. - PASS: 284 CI selection, timing, shallow-diff, sharding and routing tests. - PASS: 74 further cache, routing and graph-proof tests. - PASS: 94 targeted UI assertions across the viewer, run invocation, API errors/client and progress helpers. All 76 assertions in the 12 server suites made independently selectable also pass. - PASS: `pnpm check:token-gates`, module-boundary checks, YAML parsing and `git diff --check`. - PASS: [Full PR CI and required `ci / verify` aggregate](https://github.com/paperclipai/paperclip/actions/runs/41206683345), including all 12 general-server shards, all nine serialized shards, workspace groups, chat suites, Runner lanes and build checks. - PASS: [GitHub hosted real-workspace selector and cache/routing qualification](https://github.com/paperclipai/paperclip/actions/runs/41206683205). - PASS: Both Greptile and Paperclip reviews are 5/5 with no actionable findings or inline comments on `7dc65c51b8`. The PR has no merge conflicts. - PASS: The production refresh parser accepted all 107 structured file-cost records from a real successful server CI job, including setup/loading cost. - Local `pnpm test:run` completed all six chat shards successfully. The duplicate monolithic local run was stopped after the complete remote suite passed; it was not completed locally. The focused log-viewer command improved from 18.84s (26 duplicate variant cases) to 8.49s (all 13 distinct behavior cases). The dependency audit confirms 12 server files and the API-error test no longer need conservative always-run retention. Repartitioning the audited PR with the new costs changes the slowest shard from 810s to 591s of summed historical test execution (1,124 of 1,142 files measured). This is a workload simulation, not measured new CI wall time. ## Risks - Two graph processes use more peak memory. Each has a 2 GiB heap limit and a three-minute timeout. Any failure preserves the original task plan. Real workspace qualification runs in GitHub CI. - Historical logs contain test/hook costs only. New reporter records add loading/setup costs after merge. The manifests identify measurement coverage and source runs. These weights change placement, never permission to skip tests. - Runner load adds timing noise. The refresh uses medians and requires at least 90% measurement coverage in each lane. - GitHub may require approval to run checks on a PR created with `GITHUB_TOKEN`. Refreshes remain reviewable and do not merge themselves. - The shared viewer is a mechanical extraction. Both pages retain their existing exports, and the behavior assertions remain covered. AWS runner authorization is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, reasoning, code execution and GitHub tools. The exact deployment ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and six chat shards; complete suite passed in CI as detailed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 11:05:12 -05:00
DottaandPaperclip	b7f1c3769c	feat(native): record verified provider-pack session admission (#18469) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions can use an installed pack or a retained earlier pack. > - The current app revision does not identify the pack that a session admitted. > - Core already verifies the selected pack and binds its launch artifacts. > - Normal run records lack a closed receipt for that admitted selection. > - This pull request records that receipt after session admission and attachment. > - Operators can compare the actual Core admission identity with their expected pack. ## Linked Issues or Issue Description Refs #17749 for protected provider-pack selection and #18343 for retained controller recovery. The related OpenCode producer repair is #18456. Those changes do not publish the closed run receipt added here. A search found no existing `nativeProviderPackAdmission` PR. **What happened?** Core can admit a compatible installed provider pack with a source revision that differs from the current app. A retained session can also keep its original pack. The normal run record does not show the admitted pack digest, proxy digest, and source revision. An expected launch path or current app revision cannot fill that evidence gap. **Expected behavior** The run has a closed, server-owned receipt after actual session admission and attachment. The first receipt remains immutable history. A separate current-proof state verifies only the actual current admission epoch. Retained recovery needs positive original provenance, and missing legacy evidence stays unknown. **Steps to reproduce** 1. Select and verify an installed pack whose digest differs from the current app pack. 2. Open the native session successfully. 3. Inspect the normal run profile for a bound admission receipt. **Paperclip version or commit** The original selected base was `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The branch now includes exact actual prerequisite parent `3ee4c5b6138da82d0b1f51aa010540561bf43c22`; the PR diff against it is the five receipt, executor, test and documentation files. **Deployment mode** Core remote native execution with provider-pack selection and retained session attachment. ## What Changed - Retain the verified selected pack identity only after a backend returns the matching admitted session. - Publish `runnerProfileJson.nativeProviderPackAdmission` from the session callback after attachment succeeds. - Mark the receipt with `stage: session_admitted` and `authority: core_verified_selection`. - Store bound company, agent, run, session, runner, logical `runnerEnvironmentLeaseId`, pack digest, OpenCode proxy digest, and source revision fields. Retained attachments identify `inheritedFromRunId`. The logical Runner scope is distinct from the physical environment lease. - Lock the current coordinator claim before the profile write. Check owner, attempt, generation, controller boot, lease expiry, phase, and absence of an accepted result. Merge JSON fields without replacing other profile data. - Restore retained metadata only from the exact prior closed Core receipt, positively verified prior state and matching retained binding. Capture same-run prior provenance from the locked claim row before invalidation; cross-run recovery reads only the exact source. Legacy absence remains unknown while recovery continues. - Preserve the first receipt as history. Atomically mark a new opaque admission epoch pending in the existing mandatory claim transaction. Publish current verification only for the matching claim and epoch after actual attachment; changed or unsupported admission becomes unknown without blocking authorized execution. - Preserve synchronous session publication and quarantine ordering before the best-effort receipt write. Fresh replacement selects its newly verified pack; a captured retained-process flag cannot relabel the replacement with an older pack. Preserve the existing private pack authority even when a local dirty revision is ineligible for the closed receipt. - Add focused admission, recovery, identity, PostgreSQL ownership, lock, and JSON-preservation tests. Document the receipt contract. ## Verification - RED: the actual managed-backend path verified and admitted an installed pack but had no closed metadata. Failed open published no evidence. Additional controls fail when retained recovery invents missing history or dirty-revision eligibility gates the existing private authority. - RED calibration: removing mandatory pending invalidation fails the actual pre-admission control; reusing the retained flag for fresh replacement falsely restores the old pack. Removing the epoch fence fails the real PostgreSQL stale-callback case; reintroducing a receipt-specific execution error fails both authorized Runner/logical-lease replacement controls. - GREEN: 49 focused Core tests, all 33 admission identity/transaction tests (20 pure and 13 PostgreSQL), and all 23 existing private-authority tests passed on exact integrated head `165a9096cf3e0d34d2db1e6b52960ad4ac0ad132`. No all-skipped run is counted. - Final test-only head `11ac34baf00552a5d322b5f5f1a8e491859cacec` adds the positive executor-callback SQL-writer wiring control, exact session/identity/claim/epoch assertions, profile preservation, and null-session zero-write check. Removing only the production writer call makes it fail on a missing receipt. All70 focused identity/Core tests pass with this closure; production code, the13 PostgreSQL tests and the23 private-authority tests are byte-identical to the qualified165 source (106 distinct passing controls across the combined proof). - The PostgreSQL tests used an owned loopback-only cached Postgres 17 container with the supported `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` helper and aliases to current DB/shared source. The container was removed afterward. They cover immutable history, JSON preservation, stale claims and epochs, expired leases, terminal runs, changed/unsupported replacements, and a failed write retaining pending proof. - Historical pushed `f7f12fe` CI executed all 24 earlier admission tests, including nine PostgreSQL cases, in server5. Its warm-quarantine failure exposed the new hook-ordering issue fixed here. The same run also encountered a six-character UUID fixture-prefix collision and the independently repaired cached issue-link browser selector; those historical failures are not claimed as green. - Node 24.22.2 and Vitest 5.0.3. The credential-free normal QA runtime import preflight passes with networking disabled. Map contracts, token gates and `git diff --check` pass. - Local TypeScript validation still sees stale borrowed Runner/shared declaration mismatches outside the changed ranges. No diagnostics remain in the new helper or changed test/hook ranges; clean current-head CI is required. - Repository-wide typecheck, full tests and build were not completed locally. A full local executor attempt reached this host's unsupported native PostgreSQL listener and was interrupted; its result is not counted. Canonical run `41214512239`, attempt 2, is successful on exact final head `11ac34baf00552a5d322b5f5f1a8e491859cacec`. All 59 checks are terminal: 55 success, 3 skipped and 1 neutral; none failed or pending. The actual Linux suites execute all 33 admission tests (including 13 PostgreSQL cases) and all 1,022 executor tests successfully. The [original five failures and bounded unchanged-source controls](https://github.com/paperclipai/paperclip/pull/18469#issuecomment-6545593767) remain preserved. One failed-only request was made; GitHub expanded the reusable workflow call and dependents. No second request was made. Fresh exact-head Greptile review is 5/5, all review threads are resolved, and the PR is mergeable. - These local results claim no live provider admission, rollout or recovery. ## Risks - The authority is Core's verified selection plus successful session admission. It is not an independent provider statement or continuous runtime measurement. - Receipt fields cannot grant execution or replace the existing provider, process, lease, workspace, or recovery checks. - Legacy missing receipts and unavailable database reads remain unknown. The current release cannot label an older admitted pack. - The closed receipt requires a clean, 40-character hexadecimal source revision. Verified local `-dirty` packs still execute and retain their existing private recovery authority; their closed admission proof stays `unknown`, so it cannot certify an immutable source revision. - The first receipt is historical. Consumers must require current state `verified`; pending or unknown cannot certify the current pack. Changed Runner scope or pack evidence does not add execution policy. - A receipt save failure leaves the atomically invalidated state pending. The existing artifact authority, cancellation, workspace and process checks remain responsible for execution safety. - The receipt adds no manifest, environment, credential, or filesystem path to the run record. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 10:59:24 -05:00
DottaandPaperclip	3ee4c5b613	Make saved execution receipt recovery react to durable work (#18409) ... Reviewed and verified at 8f306e0a6e. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 10:03:59 -05:00
DottaandPaperclip	42fd568cf8	Drive issue recovery from committed changes and exact deadlines (#18402) ... Reviewed and verified at 11b49125db. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:41:27 -05:00
Dotta	12b0cc6304	fix(chat): recover committed native Agent Chat replies (#18465) ... ## Thinking Path > - Paperclip manages agent work and keeps its results available to users. > - Agent Chat needs a visible reply before the conversation can wait for more input. > - Native result commitment and reply presentation are separate obligations. > - A completed conversation can retain a valid final message without a saved reply. > - Existing reply recovery excludes these completed conversation turns. > - This pull request repairs that persistence gap through the existing recovery worker. ## Linked Issues or Issue Description **What happened?** A native Agent Chat turn completed successfully and saved its workspace. Its accepted result and final assistant message were durable, but no reply appeared in the conversation. The chat stayed active. The deployment hiring journey in #18351 then reached its deadline while waiting for the first reply. **Expected behavior** Recover the exact accepted turn's final reply once, then use normal conversation settlement. Keep later user input active. Do not run the agent again or replace a selected or deleted reply. **Steps to reproduce** 1. Create an active conversation with one admitted native turn. 2. Commit an accepted DONE result and a final-channel assistant message for that exact run, session and turn. 3. Finish workspace finalization before live heartbeat reply presentation. Leave no saved reply. 4. Run normal response recovery. Before this fix it finds no conversation candidate and leaves the chat active. **Paperclip version or commit** Reproduced with actual services at frozen master `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The production predicate also exists in the deployed revision that exposed the failure. Related work: #18351. The broader receipt and recovery work in #18118 and #18409 does not touch these source files. ## What Changed - Discover unanswered committed Agent Chat turns through the existing native response presentation worker. - Recheck company, agent, responsible user, current execution and conversation generation under the normal transaction locks. - Require the accepted result digest and an exact final message from the same native run, session and accepted turn. Keep external publication authority separate. - Commit the reply and presentation receipt before normal conversation settlement. Later input stays active. Recovery does not repeat agent work. - Preserve skip decisions and selected or deleted replies. Add database controls for foreign identity, stale sessions, partial commitment, duplicates and ordinary internal task behavior. - Document the recovery boundary in the developer guide. ## Verification - Real-service RED/GREEN uses all 418 canonical migrations in isolated PGlite and the exact new regression fixture. Before the fix: zero candidates, no reply, recovery returns false and the chat stays active. After the fix: one candidate, one exact reply, recovery returns true and the chat waits. A repeat returns false and creates no second reply. - All 20 canonical regression cases pass locally through a fixture-only PGlite bridge. Only local database provisioning and `createDb` selection differ from the normal embedded PostgreSQL fixture. Production services, SQL queries, transactions, protocol validation and assertions remain real. Linux embedded PostgreSQL CI is required before merge. - Focused current-source no-emit typing passes for both production modules and the new regression file, with normal Express augmentation and source aliases. The broader local presentation/ownership and conversation-settlement PGlite bridge run was interrupted before it reported results; it is not counted as a pass. No repository-wide local build is claimed from borrowed workspace dependencies. Clean-build CI and canonical PostgreSQL consumer tests remain required. - `git diff --check` and the scoped source/test secret scan pass. No captured run data, credentials or private target identities are committed. - No live journey retry, deployment, configuration change or test-plan admission was performed. The original failed journey remains failed; this production fix still needs deployed qualification through the unchanged journey. - Current-head [canonical Linux CI](https://github.com/paperclipai/paperclip/actions/runs/41207018345) passed all 57 terminal checks with no failure or pending check. The new 20-case database regression actually passed in [server shard 8](https://github.com/paperclipai/paperclip/actions/runs/41207018345/job/124555006090), at 13:59:25 UTC (2,643 ms). Typecheck, build and the existing consumer lanes passed. Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/18465#issuecomment-6544208002) on exact `f32a432e10b377ae6277d6f728995d5fedcf9d7b` is 5/5; the review-thread inventory is empty with no pagination. No live deployment or journey pass is claimed by these checks. ## Risks Eligible active conversations can gain a previously missing reply. Recovery accepts only the exact saved final message under its accepted result and current conversation identity. It does not run a model, repeat hiring or change a prior test grade. Reply persistence precedes waiting-state settlement; existing conversation recovery retains ownership of a later interrupted settlement. No schema or public API changes are included. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and real database test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-31 09:08:20 -05:00
DottaandPaperclip	baf651c8d0	perf(heartbeat): drive retry promotion from durable deadlines (#18387) ... Reviewed and verified at f1c48c924d. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-31 09:07:52 -05:00
DottaandPaperclip	12d8c3043e	fix(native): fence retries when replaying an audited Stop (#18457) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep Stop intent and retry state in durable records. > - A provider error can leave a failed retry with an earlier acknowledged Stop. > - The replay path returned the existing receipt before it closed that retry. > - This pull request applies the existing retry fence to the validated replay. > - The same Stop can settle without another provider cancellation or a new receipt. ## Linked Issues or Issue Description Refs #14733 for the related caller-bound Stop contract. This change covers a later retry that already has a Stop acknowledgement. The older CI-only #14743 is not a production delivery PR. **What happened?** A failed native run can have a retryable coordinator and an earlier audited Stop. A repeat Stop returns the old acknowledgement without moving the coordinator to `terminal_failure`. The final failed-to-cancelled transition requires that fence, so the Stop cannot settle through that path. **Expected behavior** A valid replay closes the same retry and keeps the original intent and acknowledgement. It does not send a second provider cancellation. Physical stop and workspace cleanup still require their own evidence. **Steps to reproduce** 1. Store a failed native run with a `retryable_failure` coordinator. 2. Store a matching run-scoped Stop with a dispatched acknowledgement and audit IDs. 3. Call `cancelNativeSession` for that same run and inspect the coordinator. **Paperclip version or commit** Reproduced on public master `f2db909393de124caf6a5380a52d4e0e1d621661`. **Deployment mode** Core native execution. The regression uses the real cancellation executor with controlled persistence and provider boundaries. ## What Changed - Reuse the existing guarded coordinator fence for a validated Stop replay. - Test repeated replay, unchanged receipts, one fence, and no second provider cancellation. - Reject foreign company, run, issue, scope, effect, and missing-audit receipts before any write. - Check that an acknowledged Stop with an unconfirmed remote lease still fails the physical-stop guard. - Document the separate retry fence and physical-cleanup requirements. ## Verification - RED on the selected public base: the new executor case returned success but left the coordinator in `retryable_failure` with a future retry. One case executed and failed for that assertion. - GREEN: 62 focused tests passed across `native-session-executor.test.ts` and `native-cancellation-request.test.ts`. These include ownership, caller conflict, changed-phase, and receipt controls. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-cancellation-request.test.ts -t 'native session cancellation|caller-bound native cancellation|failed native retry caller admission'`. - Canonical Linux CI passed all 1,018 executor cases and all 32 caller-cancellation cases. The exact file completion lines are in [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366271) and [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366219). - Node 24.22.2 and Vitest 5.0.3. Map contract checks and `git diff --check` pass. - Local dependencies are linked from an existing checkout. Full local TypeScript validation reports older Runner/shared declarations that lack current Kimi and continuity APIs. The changed ranges have no diagnostics. The canonical clean CI Typecheck + Release Registry gate passed. - A broader two-file run was stopped when an unchanged integration case tried to start embedded Postgres and hit the host listen restriction. It is not a passing full-suite result. No host settings or other services were changed. - Repository-wide `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were not completed locally. Current-head CI passed against clean source and dependencies. - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/18457#issuecomment-6543772487) on `dbc1c1a5b71450194d568725592969660864cb5a`: 5/5 with no actionable findings. The review-thread inventory has no threads and no pagination. Full [current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41203989386) passed: 59 terminal checks, 55 successful, 3 skipped, and 1 neutral. No failure or pending check remains. Merge status is CLEAN. No live recovery, rollout, or physical termination is claimed. ## Risks - A wrong or stale Stop must not close another retry. Existing run/coordinator locks, caller checks, receipt binding, and the final compare-and-set remain in place. - This fix changes logical retry settlement only. Provider termination, workspace preservation, lease cleanup, and continuation gates remain separate. - The regression controls persistence and provider boundaries. It does not establish the physical state of a remote provider. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 08:25:16 -05:00
DottaandPaperclip	c2971badde	perf(server): recover independent native replies concurrently (#18421) ... ## Thinking Path > - Paperclip manages work for AI agents. > - A native run can commit its result before its reply is saved. > - The response recovery worker saves replies from those committed results. > - It checks each candidate in series, so one task can delay unrelated tasks. > - This pull request checks up to four independent tasks at once. > - Each task keeps serial checks, and the existing coordinator owns retries and shutdown. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of committed native task replies. **Current behavior** The worker checks one candidate at a time in a batch of up to 25. A database lock or a slow read for one task delays replies for other tasks. **Proposed behavior** Check up to four company-and-task groups at once. Check candidates within each group in their existing order. Keep the current batch limit and company selection. **Reason and benefit** A ready task reply can be saved while another task waits for a database lock. The change reduces waiting under a reply backlog. **Breaking changes** Replies for different tasks can be saved in a different order. There are no API or schema changes. Other deadline workers keep serial dispatch. Related work: #17696 adds the recovery worker. #17501 reduces its read payloads. #18367 adds its timing spans. These changes are already merged. The PR search found no open change that adds this concurrency. ## What Changed - Add optional keyed concurrency to the shared scoped deadline worker. - Enable a limit of four for native response recovery, keyed by company and task. - Await all active checks before the pass ends. Retain per-candidate failure backoff, drain behavior and pending work. - Test concurrency limits, task order, company separation, drain/resume, shutdown, retries and idle wake behavior. - Add a PostgreSQL regression that holds one task's finalization row lock while a second task saves its exact accepted reply. - Document the worker's concurrency and sleep contract. ## Verification - The PostgreSQL lock regression fails with serial dispatch. It passes with this change. Both accepted replies are saved once after the lock is released. - `pnpm exec vitest run server/src/__tests__/scoped-deadline-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts server/src/__tests__/company-work-batch.test.ts server/src/__tests__/deadline-work.test.ts server/src/services/execution-preparation-timing.test.ts` — 55 tests pass. - `pnpm exec vitest run server/src/services/presentation-ownership-read.test.ts` with its documented dedicated PostgreSQL URL — 21 tests pass. The disposable database was dropped. - `pnpm -r typecheck` — passes. - `pnpm build` — passes. - One matched local replay with 25 older pending replies and one fresh reply used the real worker, a 10-connection PostgreSQL pool and 8 ms delay in each network direction. The saved fresh reply was first observed after 29.193 s with serial dispatch and 9.798 s with this change. All 26 replies were saved exactly once in both runs, with no worker errors. Both disposable databases were dropped. This is one synthetic workload pair, not a staging measurement. - `pnpm test:run` was attempted. Embedded PostgreSQL could not initialize its template after five attempts, so the broad run was stopped. The changed PostgreSQL tests pass on the separate local server. Remote CI is green. This PR does not claim a measured staging speedup. - The [first CI browser shard](https://github.com/paperclipai/paperclip/actions/runs/41162257463/job/124422204506) failed during company cleanup after its task assertions and empty-run check passed. Company deletion hit the `activity_log_company_id_companies_id_fk` constraint. The fixture uses paused process agents and reported no runs. The one failed-jobs rerun passed all 45 browser tests. All CI checks are now green. ## Risks - Four checks can use more database connections than serial recovery. The limit applies to this queue only; the batch limit stays at 25. - The shared helper has more dispatch logic. Its default remains serial. Tests cover task order, retry deadlines, drain and waiting for active checks during shutdown. - This improves reply recovery under backlog or lock contention. It does not establish a target for total agent-turn latency. ## Model Used OpenAI Codex, based on GPT-6, with code editing, tool use and local test execution. This session does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 07:39:53 -05:00
DottaandPaperclip	492d537340	fix(native): recover controllers across webserver replacement (#18343) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact fc68584, with the matching server build stamp. The final paired native feature-cost comparison explicitly uses the **same cap-three policy and common current DB factory in both variants**; its baseline is controlled, not the unmodified historical cap-two configuration. Original gates and workload counts remain unchanged. The one strict Linux/Daytona case started at 04:40:56 UTC only after both new profiles passed; it passed on the first attempt with all 49 workflow checks. Independent cleanup confirms all three owned sandboxes absent, owned Postgres/controllers/observer gone, VM terminated, root volume absent and ephemeral SSH key/agent removed. **Final controlled cap-three qualification: PASS.** Baseline native implementation `59da15066a` and candidate `fc68584ce6` use the same current DB factory and checkpoint policy (request/liveness/checkpoint = 10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000 historical tasks. All correctness, absolute and 10% relative gates pass; original cap-two failures remain unchanged. | Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline → candidate | DB-request p99 ms, baseline → candidate | Maximum drain | | --- | --- | --- | --- | --- | | Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) | 107.11 → 111.90 (+4.48%) | 26.35 ms | | Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) | 103.736 → 103.841 (+0.10%) | 930.14 ms | Every matched arm preserves operation/statement/transaction counts, including 3,900 PRP and 3,900 snapshot transactions. All three actual pools are included; there are zero lost owners, missed lease deadlines, dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses one Node process simulating 300 controllers; it does not qualify two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL contributes to measured database CPU. The original unmodified cap-two baseline is separately retained as failed; this cap-three comparison is explicitly controlled. The exact final-source 8/512/8,192-chunk test also passes: zero recovery SQL, transactions or changed rows in each text-only interval, zero idle SQL, seven lifecycle transactions and twelve changed rows per sample. Existing lease traffic remains separate. The final ordinary managed application replay passes active recovery, completion, native-cache removal and C continuation; its UNKNOWN local debug-daemon provenance and partial instruction-observer limits remain recorded. The live case uses the attested Linux producer instead. **Live qualification: PASS, one attempt, no retries.** Campaign `native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started 04:40:56 UTC on the exact final Linux webserver source; suite definition hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`. The existing immutable image and attested Bookworm Runner artifact are unchanged. - A was SIGKILLed and its native cache discarded. B started after 300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms and resumed at 317,682 ms within the original 420,000 ms grace. The same active run, Runner, provider PID/start fingerprint and one submitted/started/accepted turn were preserved; one final result/terminal and workspace finalization completed, with no retry/recovery action. - After active and queued completion, B was SIGKILLed and its native cache discarded again. C received exactly one ordinary task-comment API follow-up, completed a new tool and produced a useful reply while preserving native/provider/Runner identities. This closes the earlier failed post-C gate. A separate C OS-process PID field was not exported; the physical-PID proof is specifically A→B. - Four distinct results were accepted and published; all four finalizations committed at attempt 1, all four accounting-v2 projections settled, and the task-session pointer advanced to C. The coordinator independently checked these facts and all 49 canonical checks against the sanitized receipts. - Canonical cleanup passed and independent checks confirmed three exact sandbox 404s plus complete inventory/zero owned remnants, no owned PG or controller/observer processes, EC2 terminated, root volume absent, and the ephemeral SSH key/agent removed by 04:54:25 UTC. Live scope is classic remote Codex warm continuity with non-native storage dependencies available. Prepared-task completed B→C, full per-turn application disk loss, browser reconnection, two-server load and overall Railway/HTTP continuity were not live-qualified here. Earlier failed campaigns and the original cap-two burst failure remain failed historical evidence. A report-renderer cache setup failure was preserved; the normal sanitizer subsequently succeeded with an explicit cache path and a new output directory, without changing validators. The repair keeps retained events under their original six-field authority until persistence and accounting accept them. Unknown semantic effects stay unresolved. A compact staged target survives source bootstrap, cached-command delivery and authenticated handoff, including an uncertain commit response. Direct and same-guest-listener recovery require the verified original Runner and the exact endpoint. Target snapshot publication marks readiness in memory, with retirement coalesced into an existing command write. Headers contain only the staged descriptor/digest; a prequeue template is written once as a changed checkpoint part and then recovered from the existing command/receipt. No second session snapshot transaction, streaming SQL path, new pool or recurring idle scan is added. | Current repair gate | Evidence and scope | | --- | --- | | Source-tail regression | Real Runner protocol RED reproduced attachment rejection. Frozen core/transport pass five source-tail cases: source usage attribution, held persistence/no premature ACK, lost commit response plus cache loss, unknown semantic hold and accounting rejection hold. | | Handoff loss windows | Direct and listener modes pass queued/prepared/activated/completed boundaries (eight cells), preserving the actual Runner/provider process. Missing adoption proof and changed endpoints reject recovery. | | Compact storage | A 31 KiB attachment grant writes its separate part once, leaves it unchanged across source snapshots and removes it at queueing. Headers do not duplicate the grant. | | Accounting | 21 real-Postgres usage tests and 19 source-binding tests pass. Recovery hydrates immutable source receipts without resetting their producer or billing classification; conflicting/incomplete usage holds. | | Admission and malformed evidence | Driver 16, staged evidence 42, restart classifier 27 and source usage 19 focused cases pass. Four malformed-file regressions pass at d6a7, whose executor is byte-identical at 484, and server typecheck passes. Full executor file: 990 pass; 16 embedded-Postgres `initdb` setup failures before assertions, retained separately. | | Required checks | Exact fc68584 recursive typecheck and build pass locally, with matching server stamp. Forced full CI succeeds (48 jobs pass, one intended skip); all PR checks are clean. Maps/token gates pass for their unchanged source surfaces. Fresh exact-head Greptile is 5/5, all six threads resolved. | | Ordinary application path | Exact fc68584 managed component passes active recovery after natural expiry, completion, native-cache loss and ordinary C continuation. Production bytes match before/after; temporary fixtures restored and owned DB/services cleaned. UNKNOWN local debug-daemon/partial instruction-observer limits remain; the separate live case uses the attested Linux producer. | | Producer | Locked Bookworm build from exact 5b7 Rust tree d03e494217, equal to final fc68584 Rust source. SHA-256 65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf; 38,515,328 bytes. Actual Debian 12 execution and live admission pass. Controller/admission/DB changes do not relabel the producer source. | | Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate, 300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%); WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms (+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass relative gates, but absolute drains above fail. Identical operation/SQL counts, zero lost leases/drops/errors. One Node process simulates 300 controllers; this does not qualify two-webserver overlap or HTTP capacity. | | Live acceptance | Strict final-source Linux classic Codex case passes 1/1 on attempt 1, all 49 checks. Active recovery within grace and useful retained C continuation succeed; four accepted/published/committed/settled outcomes. Independent provider, PG/process, VM/root and SSH cleanup complete. Prepared/task and broader rollout limits are explicit above. | **Historical qualification snapshot through `9eb9ac6`.** The head at this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and Runner package sources are byte-identical to `283e05ffaa`; the two later commits change the explicitly scoped qualification and its documentation only. It includes master through the storage-owner fix `45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot, CI/review passed but the strict post-C native recovery gate failed; the PR stayed draft and required another native repair. Prior `39a4343` CI and review passed, but its full live workflow failed; that grade remains unchanged. The first repaired `435ccd2` live attempt failed before any injected outage. It exposed a legitimate terminal snapshot before result acceptance; no accepted result ID or publication receipt was observed. Its canonical report remains 0/1 `candidate_failure`, with cleanup passed and independent provider absence confirmed. `0862062` defers publication until result acceptance and fixes the fresh review's coordinator/issue lock-order finding. Both failures have deterministic real-Postgres regressions. The `0862062` live attempt then proved accepted-result publication but failed active recovery: a resumed-lease preparation guard demanded suspension after the exact live Runner was already verified for adoption. `64c0b746` preserves that verified active/idle-warm process, revalidates its proof/liveness, and keeps ordinary suspension checks and later PRP authentication. Its live run passed preparation but exposed a second active-warm boundary: all durable binding fields matched except opaque turn/item attachment IDs, which the server had regenerated. `7a203274` carries the complete verified six-field binding and keeps the strict Runner mismatch guard. Its cloud case then passed active adoption, completion, publication and accounting. The queued continuation exposed accepted-result retirement writes and an implicit-home fingerprint drift; both are fixed with deterministic regressions in `39a4343` before this fresh full qualification. The prior live case exposed a real publication gap. A goal-less native turn could persist its accepted result and terminal snapshot before provider cleanup, while the task-session pointer was still empty. Normal later admission then selected a fresh session. The fix publishes that pointer and a private bound receipt in the **existing post-acceptance terminal checkpoint transaction**, before cleanup. Earlier terminal observations persist without a handled receipt or continuation pointer. No result-acceptance or snapshot transaction was added. The late completion path recognizes the receipt. This also prevents a late callback from restoring a pointer after a user reset. Goal/control publication keeps its existing owner. The dedicated Daytona fixture now uses the existing warm/reusable environment. Its sandbox, Runner and provider storage must survive. The former generic per-turn fixture disposed the sandbox after completion and kept provider-history backup bytes under the discarded directory. Whole per-turn disk-loss acceptance therefore has a separate provider-backup storage dependency. It remains distinct from the passing native per-turn component checks with backup storage available. The 60-second lease, 300-second outage and 420-second grace are unchanged. | Gate | Current repair evidence | | --- | --- | | Assembled regression | Baseline fails with an accepted result and terminal snapshot but a NULL task pointer while real provider cleanup is held. The fix passes normal continuation and reset cases, with no pointer seeding. Exact merged `92b7447` passed 2/2 twice, including after normal package TypeScript output refresh. Production files are byte-identical at frozen `435ccd2`. | | Persistence and concurrency | 19 helper/transaction cases and all 59 existing control-port cases pass on `0862062`. They cover bound result IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay, missing stubs, foreign bindings and atomic 4 KiB rollback. Existing snapshot transaction count stays one. The new tests reproduce the exact live missing-result-ID error and the inverse replay/status lock order against `435ccd2`, then pass after the correction. Replayed results still reject stale generations. | | Completion | All 32 completion-boundary tests pass on merged `92b7447`; the source is unchanged at `435ccd2`. Native result-shaped session metadata is preserved. | | Qualification fixture | A valid review report reproduced three rejected correct replies. Exact token-boundary matching fixes them. 140 focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens, marker prefixes/suffixes and wrong authors remain rejected; retained-identity checks are unchanged. | | Required checks | Full recursive typecheck and standard build passed at `435ccd2`. Both exact commands also passed on `0862062`, with its exact server build stamp; both also pass on `7a203274`, with its exact server build stamp. Fresh `0862062` Greptile passed 5/5, with zero unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48 success, one expected skip) and Greptile passed 5/5 with zero unresolved findings. The `7a203274` normal CI also passed 48 jobs with one expected skip, with fresh Greptile 5/5 and zero unresolved findings. Current `39a4343` recursive typecheck and standard build both pass with persisted exit codes and its exact build stamp; fresh Greptile is 5/5 with zero unresolved findings. Its normal CI completed successfully: 48 jobs passed and one expected job skipped. The fresh live result remains separately failed. The `0862062` normal CI had 46 passing jobs, one expected skip, one worker communication failure and its aggregate failure; GitHub states the self-hosted runner lost communication. No failed assertion or specific OOM/spot cause was established, and no blind retry was used. | | PostgreSQL supplement | One corrected all-pool pair at the preceding repair passed 300-controller/10,000-history correctness and absolute gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613 each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL 70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a repeated improvement claim or an exact `0862062` CPU measurement. Its two later changes defer premature publication and reorder existing locks; they add no transaction, stream/idle writer, pool or timer. The first collapsed-pool attempt remains invalid and retained. This is not an exact final-head CPU/WAL measurement; later fixes change ordering, identity selection and in-memory skill replacement without adding snapshot transactions, pools, stream writes or idle SQL. | | Live publication proof (`0862062`) | Baseline run `06fcc806-f186-4190-9b57-087b61d75533` accepted result `58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names the same result, the task pointer has parameters and `lastRunId` matches, and the normalized native identity matches the following active run. Accounting v2 settled. | | Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at 23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms; natural generation-2 ownership at 312,906 ms. At about 319 s, preparation failed `runner_harness_state_mismatch: resumed_not_suspended`. No authenticated after-receipt passed within 420 s. Exact original Runner PID 668/start/fingerprint was independently still alive; provider auto-stop was 15 minutes. No retained C follow-up qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains failed; subsequent independent cleanup deleted all three owned resources with exact 404s and confirmed full intended-account inventory with zero owned resources. | | Verified adoption preparation (`64c0b746`) | 91 focused cases pass, including actual preparation for four active providers and a distinct prior/current warm proof. Unproved resumed state, dead process and stale warm owner remain rejected. The earlier options-only fixture missed this gate. No spawn, backup restore or PRP-authentication bypass was added. Server typecheck passes. | | Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2 at 312,995 ms. Preparation passed, then strict PRP recovery rejected the generated attachment IDs at about 319 s. The saved and desired Runner, lease, normalized session and run IDs matched; only `turn_attachment_…`/`item_attachment_…` differed from generated `turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains failed; independent cleanup confirmed deletion/404s and zero owned provider resources. | | Six-field identity repair (`7a203274`) | Actual server-factory regression RED4→GREEN38. A real Rust Runner test passes healthy first run → random-ID warm handoff → held active second run → restored controller. It verifies a new live authenticated connection, exact identity, same Runner/provider OS identities, no duplicate launch/turn, and no redispatch or invented receipt for an uncertain tool. The synthetic wrong pair still fails the unchanged guard. This protocol case does not claim business completion. Server and Runner typechecks pass. | | Active recovery/completion (`7a203274`) | **PASS for the active component.** Full five-minute SIGKILL/cache-loss hold. B ready at 308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated at 317,720 ms and resumed processing at 318,527 ms. Same run `1e1ccb00-0728-4935-97da-57562d81cb42`, Runner `246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start 23:48:27.470, provider lease/session and one submitted turn. Accepted result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization and accounting v2 all settled. | | Full workflow (`7a203274`) | **FAIL, preserved.** Queued late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed `runner did not durably suspend before checkpoint` before outage 2. Canonical 0/1 candidate failure, cleanup passed, evidence valid. C retention was not exercised. Independent provider absence, controllers/PG cleanup, EC2 termination and root-volume absence all confirmed; closed receipt SHA-256 `676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. | | Retirement and stable configuration (`39a4343`) | Real Runner + Postgres RED reproduced the exact suspension failure and nine owner rejections. `turn.stop` is now lifecycle checkpoint maintenance under unchanged owner/result/generation fences. GREEN25/25 proves actual persisted suspension, exact identity and completed stop/drain/suspend; unaccepted, foreign, stale and new-work negatives remain rejected. Two default-home fingerprint REDs become GREEN9/9; explicit mode changes still invalidate. No global matching exemption or historical fingerprint rewrite. | | Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.** Launched 00:15:29 UTC with automatic retries disabled. Active adoption passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms; natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed 330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner `4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and provider/session identities. Accepted result `08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict suspension error before outage 2. Its row selected a new native identity, but no new third provider process or invocation was observed. Recovery-time configuration still reset only `adapterConfig`; native provider, context and workspace subtrees were unchanged. Canonical 0/1 candidate failure, evidence valid, normal cleanup passed; all campaign provider/PG/process resources independently absent. Closed campaign receipt SHA-256 `5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`; EC2/SSH remain temporarily held for the concrete follow-up repair. Original 60/300/420-second limits and strict queued/second-replacement checks remain unchanged. Direct-process path; no new live prepared-task claim. | | Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade regression RED3→GREEN9. The successor now uses its existing production coordinator claim and prompt renewal before strict retirement. A real Runner/Postgres case starts in the actual native-observed/unclaimed phase, claims through production code, then closes and durably suspends the predecessor in 1.657 s. All 25 real-Postgres cases pass; unclaimed/foreign writers remain rejected. Shared attachment staging stays after retirement; Stop and denied claims neither close nor stage. No new claim or SQL path was added. | | Configuration continuity (`283e05ffaa`) | Real shared/project-primary environment selection → configuration with four skill versions reproduces `reset: true`, changed category exactly `adapterConfig`, after a Postgres JSONB read. Pinned restoration had reordered the generated preference while actual runtime skills stayed identical. Replacing existing pinned rows in place preserves the original fresh name/key order and fingerprint. The assembled regression passes, as do three existing pinned-version/rename/alias cases and server typecheck. Genuine model/skill-selection changes still invalidate. No fingerprint-format change or broad compatibility exemption. | | Final production repair (`283e05ffaa`) | Full recursive typecheck/build and [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106) PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved threads. Live baseline, recovered run and queued continuation all succeeded, each accepted/published/committed once with accounting v2 settled. Their native/provider identities and configuration fingerprint match; both recovery and continuation report no reset, no changed categories and task-session reuse. Active outage/authentication remains within original timing. | | Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache discarded, C became ready in 9,151.410 ms. `openParent()` then hit a 30-second `page.goto`/DOMContentLoaded timeout before posting the final request. No fourth native run or C continuation was exercised. Cause of navigation timeout is unknown. C task had no execution lock, queued comments, handoff or recovery blocker; all three prior runs were committed/accounted. Normal cleanup passed; independent provider absence, controller/PG cleanup, EC2 termination and exact root-volume absence confirmed. Sanitized report bundle SHA-256 `2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. | | Native API continuation qualification (`9eb9ac6f9e`, definition 7) | **FAIL, preserved.** The real post-C API request dispatched run `c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed `native_session_interrupted` / `native_session_recovery_failed` at 01:10:00.640 UTC. It selected the same normalized native identity, Runner and checkpoint-provider identity as the three succeeded turns. No C result was accepted; pending accounting remains explicit. Active recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019 ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms, resumed 316,612 ms. Queued continuation succeeded/accepted/published. B was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803 ms. Original classifier `transient_infrastructure`, whole-case failure and cleanup passed are unchanged; that classifier does not establish the root cause. All owned provider/PG/process resources, EC2, root volume and SSH agent independently absent. Closed receipt SHA-256 `7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`; sanitized bundle `71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`. Exact backend detail is unavailable in approved structured evidence. | | Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard build and exact build stamp PASS. [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417) PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited summary checked, zero unresolved threads. Exact8/512/8,192 stream probes again report zero recovery SQL/transactions/rows and settled-idle SQL; seven control transactions/twelve changed rows each, with 36 separate legitimate renewals in the longest case. CI success does not supersede the failed live C gate. | The completion supplement observed secondary request/liveness/checkpoint transaction p99 increases of 23.0%/45.7%/15.6% in its single pair. Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm and no lost leases. The named pool caps stayed 10+2+2. An aggregate activity sample peaked at 15 non-observer entries without backend-type classification, so no claim is made that all were client connections. No sampled lock episodes does not establish zero lock waits. Twelve pure reader/selector cases also pass with the new adjacent receipt. This is not another executed process rollback. Development reader `5cf7aead` can parse the recovery state, but its old run-detail redactor can expose receipt IDs/outcome to an authorized run reader. The first production release containing recovery readers, complete attachment-identity handling and both private-field redactions is the rollback floor. The implicit-home fix prevents new recovery-time fingerprint drift; it does not rewrite previously stored fingerprints from failed candidates. The development probe is not a published supported rollback target. Latest actual `0862062` Runner tests independently passed 8/512/8,192 text chunks with zero recovery SQL, transactions and changed rows, plus zero settled-idle SQL. All three observed seven setup/control transactions and twelve changed rows. The 8,192-chunk case separately recorded 31 legitimate lease-renewal statements, with no lost lease. These are component SQL observations, not a production CPU/WAL claim. The new adoption and identity branches affect recovery preparation only. The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed all three stream sizes with the same zero-SQL/idle result; each latest 8,192-chunk case recorded 36 separate renewal statements. Exact `39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its separate native group passed 150 tests plus eight skips. The first API fixture revision omitted the mandatory final screenshot; this was caught before paid launch and allocated no provider/model work. The correction captures the actual retained Chromium page with an explicit unqualified-browser label, without changing the standard report validator. Automatic approval review rejected exporting raw private API/workflow snapshots because of sensitive payloads and insufficient destination-specific authorization. Sanitized reports and bound/hash-only receipts are retained; raw snapshots were not copied through another channel. This does not change the failed grade. Repair characterization also preserves its failed attempts. An unconditional staging relocation broke three existing same-scope scripted cases; the final implementation defers staging only for an actual superseded warm scope, and all nine order/race cases pass. One baseline transform overlapped an edit and is explicitly invalid; the actual immutable `39a4343` loader then produced the three genuine REDs. The independent task-directory hypothesis produced a RED but was excluded by the live shared-workspace evidence; no directory policy change was landed. A shared-workspace fixture authorization omission was corrected separately from the four-skill semantic RED. Setup failures are retained separately from semantic outcomes. These include an old host port after the owned Postgres fixture restarted and a sandbox-denied TSX IPC pipe. The supported external-Postgres run passed, and the final exact recursive typecheck passes with normal process permissions. Seven stale declaration errors after the master merge cleared after compiling the actual merged SDK/Runner TypeScript sources. No declarations were edited by hand. <details> <summary>Earlier qualification measurements and failed attempts (historical, before this repair)</summary> **Historical blocked gate (before the repair below).** Final native implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later commits change only qualification fixtures, their matcher and documentation; `server/` and `packages/` are unchanged. It includes master through `fd008979f5794fda3d5f54151d36acd31d0de140`. The live Daytona measurement proves five-minute active recovery, once-only completion and accounting, but **fails retained-session identity after the second replacement**. The saved checkpoint and recovery metadata exist. The task-session pointer remains empty, and ordinary admission creates new native/provider sessions. The exact skipped publication boundary remains unproven. A useful saved-file result alone does not satisfy retained-session continuation. This application-path gap blocks merge independently of CI. A separate new-master CI failure is owned by the storage workstream (fix in #18357). No storage guard, session-identity oracle, lease/grace limit or provider result was relaxed. | Gate | Observed result and exact scope | | --- | --- | | Distinct source recovery and compatible rollback | PASS at A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B `5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four controller processes, natural 60-second lease expiry, absent native projections before replacements, three succeeded/committed unique results, one provider thread, unchanged Runner identity, and strict idle cleanup. 177.782 seconds. | | Immutable installed native component | PASS at installed A `5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B `eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each controller imports its contained compiled server and DB. Three unique committed results, stable Runner/thread, no extra provider restart on same-contract rollback, strict cleanup. 106.672 seconds. This is compiled component proof, separate from full CLI-native application acceptance. | | Normal public CLI startup | PASS for B `eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64: fresh external DB, 415 migrations, health ready, root HTML 200, company POST 201 plus GET readback. The installed tree remained unchanged after execution. | | Five-minute component outage | Actual controller absence 300,168 ms. Natural generation-2 claim at 407.196 seconds after crash, same provider attempt and Runner, original run succeeded/committed. The larger 540-second case timed out before its retained follow-ups, so it is an active-adoption pass, not a complete four-controller pass. The complete short source/installed cases above independently cover follow-up and rollback. | | Drain/resume review regression | Fixed in `226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases pass: overdue normal/restart waits, future timer cancellation, another facade resuming the original owner, and transient retry. Against exact prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail and one passes. The later head changes documentation only. | | Final corrected-fixture Daytona | **PARTIAL: active recovery PASS; retained identity FAIL**, exact live source `0b6afc499f`. A was SIGKILLed and native cache discarded; actual offline hold 300,000.375 ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at 312,059 ms, authenticated at 315,996 ms and resumed processing at 316,519 ms, within the original 420-second boundary. The same run, Runner PID/start, session, lease and event prefix survived; one turn submitted. Original execution succeeded and settled accounting. After completion, B was killed and its native cache discarded; C was ready in 9,077.779 ms. C executed the saved project and returned correct `queue_r`, with all four runs succeeded/accounted. However, native and true provider-session identities changed: the unchanged retained oracle fails. No whole-workflow pass is claimed. | | Prior matched-producer Daytona | FAILED at exact `abed921e12`. The matching current-source Runner passed normal admission, completed the first task and settled its accounting. During the active revision, A was SIGKILLed and its native cache discarded. Actual offline hold was 300,000.727 ms; B was ready at 308,304 ms and naturally claimed generation 2 at 309,651 ms. At about 318 seconds, the remote process check reported `process_not_running` for the original admitted Runner. No authenticated after-receipt or retained follow-up passed. The current failure is preserved; no automatic retry, lease/grace relaxation, debt edit or provider replay is used. Bounded forensic read found preserved exact-bound state with no recoverable failure, a transport-reset diagnostic and no terminal delivery; this supports external process loss without proving the exact cause. The sandbox had a five-minute auto-stop policy, conflicting with the required surviving-sandbox assumption. Owned cleanup precedes the separate corrected-fixture case. | **Live retained-session boundary.** Prior completed run `3806…` and C follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database session snapshots and `nativeControllerRecovery`. Their tool fingerprint, provider configuration, workspace descriptor, runtime-context digest, driver and work mode match. There is no recorded force-fresh/workspace-selection change, and this is an ordinary task rather than a conversation-generation reset. The same issue/agent task-session row has `session_params_json: null` and `last_run_id: null` from baseline time. Both native normalized UUID and actual provider session/thread hashes differ after C. Full application admission therefore did not select the retained source. No fallback to file execution or relaxed identity comparison is accepted as proof of continuity. **Independent cleanup completed.** Full current-account provider inventory confirms no resources owned by either final fixture, with exact-ID absence checks. Owned controllers and campaign Postgres are stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its delete-on-termination root volume returned `InvalidVolume.NotFound` at 22:14:08 UTC, and the temporary SSH key/agent were removed. The canonical failed report and immutable oracle input are retained. Closed qualification receipt SHA-256: `b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`. **Original measurement and grader repair remain separate.** The C reply contains the requested marker followed by a newline and the correct output. The old workflow waited for one literal space, so its normal receipt collection stalled; that original attempt was closed and its canonical `cleanup_failure`/interrupted result is preserved. Commit `af9e2d157d` repairs only this reply predicate, with 144 focused cases, E2E typecheck, discovery and diff checks passing. Ordinary string matching stays literal. The corrected predicate accepts the saved reply, but the unchanged retained-identity oracle still fails. Pure active-recovery grading of the captured measurement passes 7/7; this is not a rewritten passing campaign. The sandbox idle-window conflict was corrected only for this explicit fixture (15 minutes). Actual baseline and later sandbox metadata confirm that policy; the original active box was disposed before a post-gate policy read, so its 15-minute setting is established by the normal create/environment configuration rather than an after-read of that box. Production native lifecycle and the 60/300/420-second constants remain unchanged. The development compatibility reader A is a genuine backport commit based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required idle-goal reader fixes. It is not a previously published release. Its older executor differs from B. The rollback floor must include this reader capability. The source component cases admitted the Linux ARM64 Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the installed component case admitted the `5cf7aead` package. Their Runner bytes are identical: SHA-256 `b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each admitted artifact remained unchanged throughout its case. [Final public package build](https://github.com/paperclipai/paperclip/actions/runs/41121460635) and [reader-floor build](https://github.com/paperclipai/paperclip/actions/runs/41122170506) both passed all four daemon builds, package assembly and both Darwin clean consumers. Both Linux ARM consumers failed before provider work with the same existing local Codex permission-profile error: bubblewrap attempts `activity.fifo/.git` and reports “Not a directory.” This matched baseline failure remains open; it is not a native remote continuity pass. Full post-run tree and build-stamp revalidation also passed after the compiled native case. The installed component trees are A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`. **Controlled PostgreSQL qualification passed.** Baseline/candidate/candidate/baseline workloads used 300 controllers, 10,000 historical tasks, real snapshot/encrypted PRP writes and unchanged pools. PostgreSQL and client were each limited to two CPUs and 1536 MiB on an isolated eight-vCPU Linux host. | Workload | PG CPU per operation | WAL per operation | Database request p99 | | --- | ---: | ---: | ---: | | Steady | +1.18% | +1.81% | +1.91% | | Aligned burst | +3.40% | +2.56% | +3.81% | | Two logical servers | +3.11% | +2.56% | +9.68% | All arms passed absolute/correctness gates with unchanged control transaction counts, no dropped operations and no owner loss. Overlap p99 has little margin below the relative threshold. It uses two clients in one Node process, not two HTTP servers. [Closed measurements and limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json) and exact-source reproduction patches are retained. [Earlier contended-host failures](doc/plans/2026-10-30-native-controller-load-evidence.json) remain unchanged. The production DB-claim probe claimed 300 owners in 2.84 seconds and restored/committed controls and snapshots in 8.74 seconds, with zero owner loss or settled-idle SQL; its provider references were synthetic. The measured checkpoint/snapshot/renewal paths retain the same persistence behavior after integration; later accounting and business writes have separate owners. Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487` passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed rows during text intervals; zero settled-idle SQL. Each retained seven control transactions and twelve changed records. Existing renewal SQL was measured separately. The guard/scheduler fixes add no stream path, pool or recurring idle scan. The live fixture used current-source producer SHA-256 `f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a normal locked exact-head build with pinned Rust 1.97.1 in Bookworm. Actual network-isolated, read-only ABI execution passed (GLIBC requirement 2.34 against runtime 2.36). The immutable base image remained `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`, source `a0ef3ce7`. This was an unpublished development producer, explicitly distinct from the base image and admitted before the fresh fixture began. The earlier host build failed ABI preflight before admission. The preceding old-producer fixture completed its initial task but lacked the usage-basis marker required by current master; it remained pending/unpriced and correctly paused. It was independently cleaned without debt or budget changes. Two preceding local Linux setup attempts OOM-killed before provider allocation, and their cleanup passed. These attempts remain separate from the final recovery failure. **Failures retained and resolved separately.** The earlier `98f05fc` probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that reader and the fresh follow-up passed. Later component cleanup failures were traced by a scoped exception observer to the fixture adapter rejecting directory downloads. A bounded owned-directory transfer with exact-byte/mode preflight corrected the fixture; source and installed strict-cleanup cases then passed. No production guard was relaxed. The initial local integrated control-port run hit its 30-second embedded Postgres startup hook before executing 59 cases; 18 other focused cases passed. A bounded rerun used a fresh owned external Postgres fixture, applied normal migrations before Vitest, and passed all 59 cases with the original timeouts unchanged. Its container and data were removed; the failed attempt remains retained. The additional opt-in full Product E2E Vitest run is not green: 5,313 passed, seven failed and three skipped. An actual `abed921e12` archive with the same dependencies, Node and IPC permissions reproduces all seven failures with the same reasons. Five are unchanged workflow-policy/profile-pin assertions, including Pi profile version 37 versus the test pin 35. Two database suites failed during embedded Postgres startup before their assertions; semaphore exhaustion is confirmed in the observer log. The observer passes its supported external-Postgres rerun with original timeouts and source unchanged. The controller-database client-kill/postmaster assertions remain unexecuted after `initdb` bootstrap failure; no external override is supported. The unit launcher stopped before its Node-test phase, which was then run independently and passed 181/181. The original failed full run is preserved. These results remain distinct from the passing focused checks and normal repository CI. The first local Product E2E typecheck also encountered stale compiled Runner declarations; a genuine package-local TypeScript compile refreshed them and the unchanged check passed, without a dependency install or producer rebuild. Earlier [Daytona campaign 41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900) failed at sandbox creation before any agent, with the underlying provider error unavailable. Its [sanitized report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html) remains a `cleanup_failure`. Both old creation names are absent in the current authenticated account, but the old CI account identity was not retained, so account equivalence is unproven. The subsequent exact `eaef0d5` source Product E2E attempt passed image/agent setup and the initial project, then held the controller offline for 300,003.7 ms. Its Mac supervisor failed owned-process inspection before B readiness; no adoption passed. Full current-company lease/label checks, provider deletion, exact-ID 404s and complete inventory confirmed cleanup of that attempt. Its local server and owned Postgres stopped. The prior AWS campaigns and database qualification VM were terminated with root volumes removed. The old admitted test Runner lacks the newer usage-basis marker introduced on master. The integrated component B turns retained unpriced, incomplete accounting obligations with `costAccountingPending: true` and no settled charge. The compatibility-reader A turn used its legacy receipt interpretation but likewise retained pending accounting and no settlement timestamp. This proves obligation preservation, not a whole-run accounting migration. Runner/receiver release ordering remains governed by the usage owner's contract. At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`, Greptile was 5/5 with zero unresolved threads and CI completed with 54 successful checks and five expected skips. At `0b6afc499f`, Greptile again passed 5/5 with zero unresolved threads, but CI ended with 51 successful checks, five skips and three failures. Server 9/12 fails a deterministic managed-skills maintenance assertion newly inherited from master `198218bf` in merge `c80e00ad`; that test is absent from the PR head, and the storage owner has the explicit-StorageService fix in #18357. Runner 2/2 passed 2,283 unit cases but its runner received a shutdown signal during the long stream probe; its cancelled step and aggregate failure are preserved. No blind retry was used. Current `af9e2d157d` checks are pending; the matcher-only change passes 144 focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI failures and the interrupted unsharded local test run remain recorded in the dated plan. CI runs the repository's normal test partitions. Final verification does not claim Railway rollout, uninterrupted HTTP service or whole-webserver filesystem loss. </details> ## Risks - The task-pointer repair passes its assembled regression. The first live repair exposed pre-acceptance ordering and the fresh review exposed a replay lock-order issue; both are fixed and pass real Postgres tests at `0862062`. The following live attempt exposed the verified-running-versus-suspended preparation gate. Its narrow fix passes 91 focused cases at `64c0b746`. The next live case exposed synthetic attachment IDs; `7a203274` fixes that field selection and passes real Runner authentication. The subsequent queued continuation exposed strict retirement maintenance and a false default-home change; `39a4343` fixes both with real Runner/Postgres and fingerprint regressions. The `39a4343` live continuation failed again despite green CI. `283e05ffaa` repairs the actual native-observed admission ordering and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass targeted real-Postgres regressions and the live active plus queued paths. The definition 6 post-C browser navigation timed out before admission. Explicit definition 7 now verifies native continuation through the ordinary comment API; its strict post-C native gate failed after a real dispatch and remains historical failed evidence. The final fc68584 repair now passes its fresh strict 49-check Linux/Daytona campaign; the prior failed grades remain unchanged. Browser reconnection is handed to its own workstream with the original failed evidence. No prior failed campaign is reclassified as passing. - Incorrect checkpoint/source binding or stale ownership could duplicate external work. Missing evidence remains a hold; existing receipt and finalization owners keep uncertain effects unresolved. - Recovery must retain maintenance writes after accepted results without allowing a stale business turn to write. Real lock-wait and generation tests cover that boundary. - The current two-connection paired burst profile fails its absolute drain gate in both baseline and candidate, despite passing relative load gates and all correctness checks. A causal capacity diagnostic selected three slots; the final controlled cap-three profiles pass unchanged thresholds. Full application replacement and two-server overlap capacity remain separate. There is no new streaming writer, pool or recurring idle scan. - The remote sandbox and Runner must survive the recovery window. Provider auto-stop is a separate lifetime bound from reconnect grace; the explicit qualification fixture uses a longer idle window. - Workspace and provider backup availability remain external dependencies. This PR does not certify whole-webserver filesystem loss, uninterrupted HTTP service or a Railway rollout. ## Model Used Implementation and initial verification used OpenAI GPT-6 through Codex; the initial hosted snapshot was not exposed. The follow-up qualification used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with repository tools, code execution, CI and scoped test infrastructure. Context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and exact-source application/cost checks pass; full forced CI passes; earlier local embedded-PG setup failures remain recorded) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green (exact fc68584 full CI and auxiliary checks pass; forced complete test partitions) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (exact fc68584 check and edited summary pass; all six threads resolved) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 07:33:52 -05:00
DottaandPaperclip	322fa6b65c	fix(ui): restore task document authoring and add persistence journey (#18356) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Task documents keep versioned work beside each conversation. > - The default chat interface could read general documents but hid their editor. > - People also need unfinished edits to survive a tab switch or reload. > - This pull request restores the shared editor in Artifacts. > - A portable browser journey checks exact saved content and revision history. > - The benefit is a usable document flow with independent deployment checks. ## Linked Issues or Issue Description **What happened?** A person using the default task interface could read a general task document but could not create or edit one. The existing editor was available in Classic mode. A pending create could also leave newer text as a new form after its editor unmounted, causing a conflict against the person's own successful creation. **Expected behavior** A person can create and edit general documents through Artifacts. Titles, bodies and revision history persist after reopening and reloading the task. Unfinished text survives tab switches. A successful create makes retained newer text an edit of that confirmed revision. Later writes from another actor still cause a conflict. **Steps to reproduce** 1. Open an ordinary task in the default chat interface, then open Artifacts. 2. Try to create a general document. The previous interface has no authoring control. 3. With authoring restored, submit a new document, keep typing, and switch to Properties before the request finishes. 4. Reopen Artifacts. The previous draft path restores a new form despite the successful create. Related public work: #17598 adds document copy/download and #18157 adds image galleries. The selected actual merged parent is `f1d780838218f975064f38333946805e0ec103a5` from #18427. It includes the stable profile-entry ordering repair and the exact recovery-fixture prerequisite. The resulting PR diff contains 32 document and reporting files. ## What Changed - Reuse `IssueDocumentsSection` for general documents in default-chat Artifacts. Keep Classic authoring in its existing task body. Ordinary tasks, conversations and Coordinator mode share the same editor options and signed-in permission wrapper. - Preserve Plans, file-review documents, attachments, uploads and work products. Hide duplicate general-document cards. Use the same general-document key rules for visible rows, validation and refused writes. - Forward image uploads, mentions, references, author maps and feedback to both editor mounts. Keep empty mention/project values stable while queries load. - Back up unfinished forms within their signed-in account, task and editor scope. Preserve text entered before its key. Record a successful create on its matching retained newer buffer, including after unmount or an early reopen. Preserve newer title/body edits, real remote conflicts and matching cancellation cleanup. - Add the model-free `task-document-persistence` journey. Setup creates only an owned empty backlog task. The browser creates and edits the document; normal APIs independently check company, task, actor, exact content and both revision records. - Reopen the owned task through the explicit Backlog view, then reload it. Require exact UI readback and zero native runs. Add only the Doc registration, definition-hash inputs, report mapping and four closed checks to the shared harness. - Count explicit zero-run browser receipts after private journey metadata is removed for publication. Preserve missing billing evidence, positive usage and native/protocol accounting. - Update browser assertions for the restored inline editor while retaining exact download bytes, revision collisions, paused-state and no-run checks. Update the feature map, usage docs and shared editor stories. ## Verification - Latest focused checks use official Node 24.22.2: 568 journey, catalog, report, billing and attempt-accounting tests across 25 suites pass. All four admitted journeys remain registered. The seven relevant UI suites pass 529 tests, including the task-page container after the parent merge. Maps, token gates, the PR-scope diff check and the owned credential-pattern scan pass. The normal QA import preflight also passes with an empty environment and network access blocked. - Billing regression: before the repair, the explicit empty public receipt counted one run, and two positive-usage receipts were discarded by browser metadata. Those three cases fail before and pass after the repair. Two additional red regressions cover aggregate usage beside an empty run list. That ambiguous shape now keeps an unavailable receipt and a minimum count of one. Missing receipts retain unavailable billing, and native/protocol one-or-more run totals retain their measurements. - The previous fresh CI run failed its unchanged snapshot-only recovery fixture: one global sweep did not call its callback. The original failure remains recorded. An isolated PGlite probe with all canonical migrations exercises the real reconciler: it processes two earlier sources, then reaches the owned third source on the next sweep. It preserves source evidence, files and run IDs. This proves bounded pagination, not full native snapshot repair. - Local native recovery tests could not start PostgreSQL because the host semaphore limit was reached. That run executed zero cases. Canonical Linux CI later passed all 143 fixture cases, including the original snapshot-only case and the new competing-candidate control. The exact fixture repair is now inherited from #18427 and is absent from this PR diff. - The portable oracle rejects 19 stale, missing, foreign or incorrect outcomes. Result validation also rejects model/provider metadata and native runs for this case. - Five real Artifacts-to-Properties unmount regressions fail before the repair and pass after it. They cover both editor modes, reopening before and after completion, later typing, and a foreign revision that must still conflict. The edit-storage failure case also checks that Cancel clears the promoted fallback copy. - UI coverage includes default chat, Classic, conversations, Coordinator, Plan rendering, document-card deduplication, reserved keys, pending-query stability, permissions and rich editor options. - Earlier real browser checks passed five upload cases and two artifact-arrival cases. They retain exact CSV bytes, original/revised content, collision, paused-state and no-run assertions. Current-head normal CI runs the updated browser suites again. - The branch merges only the selected actual parent `f1d780838218f975064f38333946805e0ec103a5`. The merge has no conflicts or manual resolution. The SQL service and recovery fixture match that parent exactly. The own document editor, persistence flow, saved-state oracle and billing repair retain their previously qualified source blobs. Follow-up, clarification, question/resumption, screenshot helpers and the Steering map are byte-identical to the merged parent. Existing project/search/question/follow-up registrations remain present. - Required document source blobs remain exact: Doc UI `22e80663b053363e5f75e1432dd9043dc0e70243`, its unmount tests `16f3f4ed0dc9b11aaff129e06e6cf2bbf4522713`, readiness helper `800a747284db19b8484650b6759e8bd1f356839d`, and budget service `3a15d489f0aa55c22e8f7fe6a6bc6a0e14e5914e`. - Local typing is limited by borrowed declarations from another checkout. They omit current `skill_test` and sandbox creation-operation fields. No dependency declarations or repository config were changed. The authoritative clean CI Typecheck + Release Registry gate passes on exact head `2ffce29827ae170420d740940c1c069c7225a74c`. - Full monorepo typecheck, tests and build were not run locally. Borrowed build declarations from another checkout limit local typing. Authoritative [clean CI](https://github.com/paperclipai/paperclip/actions/runs/41171717008) passes on exact head `82d15946c5844d3ae41c6ade62fa224053f20a9a`: 55 successful checks, 3 skipped and 1 neutral, with zero pending or failed checks. Typecheck, build, Runner, server and browser lanes pass. Greptile gives that same head a fresh 5/5. All 15 review threads are resolved, with no further thread page. The PR is mergeable and clean. - Failure history remains recorded. The earlier recovery failure was repaired with a bounded, owned-source fixture. A later CI worker shut down after 2,375 Runner unit tests passed. Its one authorized narrow retry passed those tests and all five native integration suites. The separate AgentConfig browser failure was reproduced and fixed by the merged #18427 parent. These earlier passes do not replace current-head CI. - Final-source live qualification passed on exact harness and served application `82d15946c5844d3ae41c6ade62fa224053f20a9a`. QA run `41177696858` completed all four closed checks and all five phases with zero native runs. Cleanup passed; the durable manifest is complete with no pending evidence. The coordinator verified the original grade and manifest. - The [public qualification report](https://pages.paperclip.ing/evals-qa-smoke-41177696858-1/) records zero agent runs, empty run IDs and empty run usage. Pricing remains unavailable and billing remains incomplete; the report does not claim measured zero cost. - The earlier harness `36c8d9ddd76d0b7ddb28a9a6c674d9e538f6df88` pass against application `2ffce29827ae170420d740940c1c069c7225a74c` remains historical evidence. It is separate from the final-source measurement above. - The selected parent brings profile-entry ordering, proactive chat, a database migration, Runner updates, and task-page chat/read behavior. Its new task-page container is included in the final application measurement. The coordinator retains the original evidence and owns the primary merge and daily-plan admission. ## Risks Artifacts now mounts the existing shared editor. Its autosave, conflict, revision, annotation and server access contracts remain in place. Tests cover both interface modes, shared scopes, later remote writes and unavailable edit storage. Plans and generated file-review documents keep their existing flows. The PR adds no database schema or API change. The shared billing repair requires explicit captured empty run IDs and an unambiguous usage object containing only an empty run list for public browser-only receipts. Other fields beside an empty list keep conservative unavailable billing. Missing measurements remain unavailable. Contradicting positive usage remains counted. The inherited recovery fixture keeps the production sweep limit and its source, readback and no-agent-work assertions. The live journey must fail when authoring controls or saved outcomes are absent. It must not create the expected document through setup APIs or switch to another interface to pass. Final deployment verification must retain the actual app and harness revisions separately. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact runtime model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 01:49:24 -05:00
DottaandPaperclip	59da15066a	feat(runner): deliver native Kimi Code (#18391) ... Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-30 22:26:15 -05:00