1683 Commits

Include renames

Author	SHA1	Message	Date
Nicky LeachandClaude Opus 5.5	8ab89f41e4	fix(build): typecheck test projects in 4 GiB (#18738) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents typecheck this repository, often on machines with limited memory > - paperclipai/paperclip#18737 made `pnpm -r typecheck` fit in 4 GiB because the server now checks in layers > - The test projects outside `pnpm -r` still import server source through relative paths, so each one checks most of the server as one program > - These projects peaked between 6.9 GiB and 14.9 GiB, so they fail on a 4 GiB machine > - This pull request makes these projects read the server layers as declarations, and adds a CI step that keeps them that way > - The benefit is that every TypeScript project in the repository can now be checked in 4 GiB, and a later change cannot silently undo this ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged. This pull request builds on the server layers that it added. No issue exists for this gap. **What happened?** `tsc -p tests/runner-e2e/tsconfig.json` peaked at 14.9 GiB at default settings. `tests/speko-native` peaked at 12.2 GiB, `server/src/__tests__/helpers/maintenance` at 9.6 GiB, and `tests/lifecycle-baseline` at 6.9 GiB. On one thread, `tests/runner-e2e` still peaked at 7.7 GiB. **Expected behavior** Each project passes with 4 GiB of memory. **Steps to reproduce** Run `tsc -p tests/runner-e2e/tsconfig.json` in a Linux container with `--memory=4g --memory-swap=4g`. The kernel stops the process. ## What Changed - Added `tsconfig.layers.json` to the four projects. Each file extends the project's `tsconfig.json` and references the generated server declarations. - Added `scripts/typecheck-test-projects.mjs`. It builds the server declarations and then checks the named projects, or all four. It takes `--low-memory`. - Changed `test:e2e:runner:typecheck` and `test:lifecycle-baseline:typecheck` to call that script. `docker-runner-check.yml` calls the first script by name, so that workflow needs no edit. - Added `--declarations-only` to `server/scripts/typecheck-layers.mjs`. The script now also generates `tsconfig.declarations.json`, so hand-written configs do not name individual layers. - Exported `runPnpm` from `scripts/typecheck-low-memory.mjs`, so the new script can use it. - Added a step to the `Typecheck + Release Registry` job. It runs straight after the server typecheck and reuses its declarations through `--reuse-declarations`. - The script now fails if a project loads server source, even when the types pass. Test support files, declaration files and JSON files do not count. This check does not enforce a memory limit. - Added a unit test that fails if a new project under `tests/` is neither in `PROJECTS` nor named as a single program. `tests/runner-acceptance` is the one named project. - Added `server/src/types/express.d.ts` to `tests/lifecycle-baseline/tsconfig.json`. That plain config failed alone with 11 TS2339 errors, and it now passes. - Exported `isTestFile` from `server/scripts/typecheck-layers.mjs`, so the source check uses the same rule as the layers. - Turned off `declaration` in the layered configs. These projects emit nothing, and declaration checks gave a false TS2883 error when `tests/runner-e2e` read the server as declarations. - Added tests, one of which runs the real compiler. Updated `doc/DEVELOPING.md` and `tests/lifecycle-baseline/README.md`. The references are not in each project's `tsconfig.json` because Vitest reads that file. A test showed that Vitest fails when a referenced config does not exist yet. The `tsconfig.json` files do not change, so editors and Vitest see the same configs as before. No new `package.json` scripts were added, as `.greptile/rules.md` requires. ## Verification I tested commit `9763880fab`, which is based on `1c9ca5829f`. I copied the commit into a named Docker volume and ran a fresh install. Each row ran in its own Linux arm64 container with `--memory=4g --memory-swap=4g` and 8 CPUs. The first row started with no build output. All rows used `--low-memory`. CI does not set a memory limit. I used the limit only for this measurement. | Project | Errors | Peak (cgroup) | Max process RSS | OOM kills | Time | | --- | --- | --- | --- | --- | --- | | `tests/runner-e2e` | 0 | 3.34 GiB | 3.06 GiB | 0 | 57 s | | `tests/speko-native` | 0 | 3.60 GiB | 3.27 GiB | 0 | 51 s | | `tests/lifecycle-baseline` | 0 | 3.43 GiB | 3.15 GiB | 0 | 50 s | | `server/src/__tests__/helpers/maintenance` | 0 | 3.39 GiB | 3.13 GiB | 0 | 52 s | `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:e2e:runner:typecheck` and `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:lifecycle-baseline:typecheck` also passed in the same limit, with peaks of 3.55 GiB and 3.52 GiB. I also ran the CI sequence in one container with the same limit: `pnpm --filter @paperclipai/server typecheck`, then `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations`. Both passed, with a peak of 3.46 GiB. The new step took 16 s. The server layer build sets the peak in each row. Against the declarations, the projects peaked between 0.9 GiB and 2.2 GiB at default settings on macOS. Server source files that each project loads, without test support, declaration or JSON files: | Project | Plain `tsconfig.json` | `tsconfig.layers.json` | | --- | --- | --- | | `tests/runner-e2e` | 1,345 | 0 | | `tests/speko-native` | 1,647 | 0 | | `tests/lifecycle-baseline` | 1,343 | 0 | | `server/src/__tests__/helpers/maintenance` | 1,716 | 0 | I counted these on `3f0f31b2f9`. The source check in the script confirmed 0 for all four projects on this commit. **Limits of this proof:** - Without `--low-memory`, the 4 GiB run does not pass. The kernel stopped the layer build for `tests/runner-e2e` on `3f0f31b2f9`, and the script exited with status 1. I did not repeat this run on the final commit. The script does not turn on low-memory mode by itself. - The Linux runs used arm64, not x64. - The new CI step has not run in CI. `pr-trusted.yml` runs from `@master`, so the step first runs after this pull request merges. Check its first run. **Other checks on this commit:** - `server/scripts/typecheck-layers.test.mjs`: 38 tests pass. One test uses the real compiler to show that the source check tells a layered config from a config with no references. - With the references removed from `tests/speko-native/tsconfig.layers.json`, the script exits with status 1 and reports 1,644 server source files. - With the declarations removed, `--reuse-declarations` exits with status 1 and reports TS6053. - A type error in a test project makes the script exit with status 1. - `tests/lifecycle-baseline/tsconfig.json` passes with 0 errors. - `tests/lifecycle-baseline` Vitest suite: 533 tests pass. - Workflow tests that read `pr-trusted.yml` and CI routing: 114 tests pass. **Earlier checks:** - `server/src/__tests__/server-package-build-script.test.ts` passed 10 tests on `3f0f31b2f9`. This test failed in `ci / General tests (server (9/12))` on an older head. It tested a build script split that the final #18737 removed. - A changed signature in `server/src/services/run-liveness.ts` failed `tests/lifecycle-baseline/authority.test.ts` through the rebuilt declarations, before the rebases. When I restored the signature, the check passed. **Not run locally:** - `pnpm test:run` and `pnpm build`. CI covers these. **Failures that this pull request does not cause:** `pnpm test:e2e:runner:unit` has 13 failures on this commit, in `cursor-cloud-replay-fault`, `first-task`, `operating-receipts`, `pi-controls-evidence` and `workflow-security`. The same 13 tests fail on `master` at `6ef37398c8`. A comparison of the test names shows no differences. PR CI does not run this suite. ## Risks - Low risk to runtime behavior. No shipped code changes. - **Slower check:** `pnpm test:e2e:runner:typecheck` now builds the server layers on each run. Before the rebase, it took 30 s on macOS at default settings, about the same as the old single program for that project. On this commit it took 57 s in low-memory mode in the 4 GiB container. The new CI step reuses the declarations, so it took 16 s. - **Narrower check:** Server source is now checked only by the server's own layers, not again under each project's compiler options. The projects also skip declaration-portability checks. These projects emit nothing, so those checks had no output to protect. - **Unchanged path:** `tests/runner-e2e/native-completion-checks.mjs` still runs `tsc -p tests/runner-e2e/tsconfig.json` as one program. It needs about 15 GiB at default settings. I did not change it because that file and its inputs are part of a fingerprinted source contract, and no workflow calls it. - **Layer growth:** The new checks do not measure memory. If the layers or the import cycle grow, the checks still pass and the peak rises. This is intentional. Crossing 4 GiB does not fail a build. - **Project list:** The CI step checks only the projects in `PROJECTS`. The unit test covers new projects under `tests/`, but not a new project in another directory. - **Reused declarations:** `--reuse-declarations` trusts the declarations on disk. CI uses it straight after the server typecheck on the same commit. If that typecheck fails, the job stops first. Stale declarations can pass locally, so `doc/DEVELOPING.md` warns against this. - **Job dependency:** A type error in a test project now fails `Typecheck + Release Registry`, and so `ci / verify`. - **Not covered:** The new step runs no tests. `node scripts/typecheck-low-memory.mjs` and the Docker image build also have no new guard. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-03 11:09:12 -08:00
DottaandPaperclip	4a38ff9c92	fix(ci): make Cloud readiness reruns and fixtures reliable (#18823) ... ## Thinking Path > - Paperclip manages agent work and preserves task state. > - Cloud readiness verifies an exact source revision before release. > - Its workspace tests must wait for execution and database cleanup to finish. > - Two attempts hit the same ten-second polling limit and read a running row. > - The second attempt also lost AWS runners and exceeded a fixture cleanup limit. > - This pull request waits for owned execution, bounds client shutdown, and gives verification reruns an independent runner pool. > - Assertions, source identity and required verification gates stay in place. ## Linked Issues or Issue Description **What happened?** [Cloud readiness run 41684102136](https://github.com/paperclipai/paperclip/actions/runs/41684102136) failed twice on master `367f29478d14003cc5c0f161b7ce8f3164993634`. Attempt 1 failed `heartbeat-workspace-busy.test.ts:473`: expected `succeeded`, received `running`. Attempt 2 failed the same helper in the inherited-workspace resume case at line 744. The helper polls for ten seconds, then returns the active row as if the wait finished. Attempt 2 also exceeded the 30-second `workspace-preparation.test.ts` teardown limit. Its client close had no deadline, and database plus directory cleanup shared the default hook budget despite the fixture's existing 90-second startup budget. Other attempt-2 failures show runner shutdown or lost communication. Several shutdown signals arrived at 18:05:17 UTC. Read-only AWS inspection confirms that all eight infrastructure-failed instances carry `runs-on-workflow-job-interrupted=true`. Five have explicit `Server.SpotInstanceTermination` reasons, including the server-1 runner that lost communication. Three were terminated through the fleet's shutdown path. CloudTrail records the Rust runner's interruption tag at 18:03:37 UTC and its termination by the RunsOn fleet role at 18:05:31 UTC, despite EC2 showing a user-initiated termination. Repeating the entire workflow reused the same Spot fleet. **Expected behavior** Wait for owned heartbeat execution before reading final state or scheduled retries. Bound database-client shutdown and allow the fixture's cleanup cost class to finish. Rerun verification on GitHub-hosted runners after an interrupted fleet attempt. Preserve all checks and report prior failures. **Steps to reproduce** 1. Delay the test adapter beyond the helper's ten-second polling window. 2. Invoke the ordinary local shared-workspace run or resume the deferred sandbox run. 3. Observe the old helper return `running` before the execution completes. 4. Evaluate the old runner expressions with `github.run_attempt = 2`; they still choose the interrupted AWS pool. **Paperclip version or commit** Base: `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions Cloud readiness, with disposable PostgreSQL fixtures. Related: #11293 fixed the earlier gap between cancellation and retry-row insertion. This change waits for the complete execution instead. Duplicate searches covered the workspace test, retention fixture and runner shutdown reports. ## What Changed - Replace both workspace-test polling helpers with the existing heartbeat completion barrier. Keep final status, retry and coordination assertions. - Bound workspace-preparation client shutdown to one second. Apply the existing embedded-PostgreSQL fixture budget to its teardown. - Use AWS only for the first exact-source attempt. Route later verification, chaos and cloud-migrator planning/build attempts to GitHub-hosted runners. The publisher keeps its existing routing. The final branch also incorporates upstream #18821's CODEOWNER routing fix. - Add runner-expression coverage for numeric, string, missing and later attempt values. Document retry routing. ## Verification - The original workflow's failed assertions and interrupted jobs remain linked above. A local two-suite run also passed all 157 assertions and failed only the original 30-second teardown hook. - All 341 workflow contracts pass. After the master synchronization, all 347 routing and CI-planner controls pass. Actionlint passes for the three changed workflows. - A matched delayed-adapter comparison uses disposable PostgreSQL 17 databases, the same 11-second adapter delay, and the same 90-second experimental outer budget. The old helper fails with `running`; the completion barrier passes. Production test deadlines stay unchanged. - Earlier experiments hit a 30-second experimental outer limit and the macOS host's shared-memory limit during native database startup. The completed comparison uses task-owned databases on an existing test server. Those earlier attempts are not counted as passes. - All 31 workspace-busy cases pass with normal test deadlines. The 126-case workspace fixture also passes with its 90-second teardown budget. - Final-head cleanup measurements: main client close 7 ms, owned database cleanup 20,938 ms, and workspace-home removal 826 ms. A measured fixture held 124,494 PostgreSQL files (208 MiB) after repeated table resets. The measured database cleanup alone consumes most of the old 30-second hook budget. The separate baseline reproduced a timeout at that hook; the fixture now uses its established 90-second cost class. - Repository typecheck and build pass on synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2`. - All 157 focused cases pass on the synchronized branch, including complete PostgreSQL 18 fixture shutdown. [Full hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41697804471) passes on the final head, including the required verification and E2E aggregates, all server/chat/workspace groups, Runner lanes and all eight browser shards. The separate GitHub quality-review workflow also passes. All 60 reported checks are terminal: 54 pass, three are informational neutral results, and three are expected skips. There are no failures or pending checks. The complete local `pnpm test:run` command was not run; targeted fixtures already encountered this host's shared-memory limit. Hosted CI supplies the full Vitest and browser matrix. - Greptile and Paperclip Review both give synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2` 5/5. There are no open review threads. - Module-boundary and whitespace checks pass. ## Risks - Hosted reruns can have different cache hit rates and queue times. They execute the same source and verification commands. - The first attempt retains the existing AWS Spot fleet policy. AWS evidence establishes several interruption-triggered failures; later verification attempts use an independent pool. This does not change the fleet's external capacity policy. - The completion barrier waits for owned background execution. A hung execution still fails the test's outer deadline. - The larger teardown budget remains finite. PostgreSQL's bounded stop protocol and cleanup errors still fail the hook. - No production execution, database schema or provider contract changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code edits and test execution. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 13:08:28 -06:00
DottaandPaperclip	83b5f7d486	fix(ci): keep CODEOWNER PRs on AWS as master advances (#18821) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR checks use reserved AWS capacity for authorized maintainers. > - The routing gate waited for GitHub capacity and compared an old test merge with a newer base branch. > - Normal merges into master then sent trusted PR jobs to the GitHub queue. > - This change restores AWS admission and validates the immutable merge selected for the run. > - Current CODEOWNER and rerun identity checks still control PR execution. ## Linked Issues or Issue Description Related: #18376 moved the gate to GitHub-hosted capacity. #16042 introduced reserved AWS admission. **What happened?** In PR #18811, GitHub created test merge `608d862ff1650db108a3832a2c67f8795924f0e7` at 17:17 UTC. Master advanced at 17:25. The gate started at 17:30 and rejected the older merge parent because it did not contain the newer master tip. The authorized PR then selected GitHub-hosted runners. The gate itself also needed a GitHub-hosted slot. **Expected behavior** Authorized maintainers use reserved AWS admission and PR workers. Normal base advancement does not change runner placement. Current owner checks, PR head checks, run-attempt checks, and base-history validation remain required. **Steps to reproduce** 1. Start a PR run whose author and sender are current CI CODEOWNERS. 2. Advance master while its routing job waits. 3. Run the gate against the original event merge and the current base tip. 4. Before this change, the gate selects `ubuntu-latest`. After this change, it selects the public PR AWS fleet. **Paperclip version or commit** Reproduced on master `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions with RunsOn Fleet on AWS. ## What Changed - Restore the reserved routing fleet for the synchronized maintainer admission roster. - Check the immutable event merge and its PR head. Accept base snapshots between the event base and the current base tip. - Allow GitHub to regenerate its live merge ref while the original run waits. - Retain stacked-base validation and all current CODEOWNER, repository, author, sender, and rerun-attempt checks. - Add advancement, regenerated-merge, unrelated-history, and admission-placement regressions, including a stacked parent that descends from the event base but diverges from the live base. - Align the developing and CI profiling guides with reserved AWS admission and current CODEOWNER checks before PR execution. ## Verification - `node --test .github/scripts/tests/ci-routing.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 103 passed. - `node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-plan.test.mjs`: 31 passed. - `actionlint .github/workflows/ci-routing.yml`: passed. - `git diff --check`: passed. - Mutation check: removing the snapshot-to-live-base comparison in a temporary workflow copy makes the new side-branch regression fail, proving it protects that ancestry guard. - [Greptile: 5/5](https://github.com/paperclipai/paperclip/pull/18821#issuecomment-6612170970) with a successful check on `abd6a8ce347228219d8e5127f34f57151788830e`; both review threads are resolved. - A regression reproduction on the original workflow selected GitHub instead of AWS. The patched workflow passes that scenario. - A read-only execution of the patched routing step against the actual event for run [41682867664](https://github.com/paperclipai/paperclip/actions/runs/41682867664), using live GitHub metadata, selected `runs-on/fleet=paperclip-public-pr-x64/env=public-ci`. This did not allocate a runner or rerun CI. - Full repository typecheck, Vitest, and build were not run in this worktree, which has no installed dependencies. The change is limited to workflow routing, its tests, and documentation. Broader CI checks remain queued. ## Risks - A stale admission roster can allocate the five-minute metadata gate. The gate still checks current CODEOWNERS before it authorizes any PR code on AWS. - The routing runner group must authorize `ci-routing.yml@refs/heads/master`. The fleet already exists; live placement still needs confirmation after merge. - Rewritten base history, changed PR heads, and unavailable identity or ancestry evidence continue to deny AWS PR execution. - Existing runs retain their previously resolved workflow. New runs use this change after it reaches master. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code execution, and live GitHub API tools. The runtime does not expose the exact serving model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 12:47:16 -06:00
DottaandPaperclip	ccc63ea458	fix(server): prove completed history before idle sleep (#18646) ... Reviewed and verified at 592bce69c6. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 12:37:25 -06:00
DottaandPaperclip	367f29478d	fix(recovery): preserve the bound remote writer before continuation (#18803) ... ## Thinking Path > - Paperclip manages agent work and preserves task files. > - A remote workspace records the exact run that owns its unsaved files. > - A later setup failure can become the latest task run without owning those files. > - Retry only inspected that latest run, so the original writer had no file-recovery owner. > - A released reusable source also failed the legacy retention rule. > - This change registers the bound writer, saves its files, and then admits the saved user request once. > - Existing stop, ownership, approval and execution-evidence checks remain effective. ## Linked Issues or Issue Description **What happened?** A native run prepared a remote workspace and failed before finalization. Its sandbox stopped with a dedicated stop receipt. The workspace stayed dirty and retained that run as its writer. A later setup failure became the latest task run. Retry inspected the newer run and left the original file obligation unregistered. New attempts failed at the writer fence. Registration alone was insufficient. After file recovery, continuation still required the Retry source to be one of the older recovery sources. A newer pre-provider failure did not satisfy that check. Workspace selection could also choose a fresh directory instead of the recovered files. **Expected behavior** Find the writer through the task's current workspace binding. Register only its exact prepared source. Preserve the released lease and its stop receipts. Save the files before continuation. Independently prove that a newer setup failure never dispatched work. Select the verified saved workspace and admit the Retry once. **Steps to reproduce** 1. Fail a native run with a prepared remote workspace and a released reusable sandbox. 2. Keep matching dedicated stop receipts and the workspace's dirty writer authority. 3. Record a newer pre-provider setup failure for the same task. 4. Request Retry. Before the fix, no file-maintenance record owns the actual writer. **Paperclip version or commit** Reproduced before production edits on master `f7ad1dc7ca`. The branch was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`. **Deployment mode** Authenticated hosted execution with Paperclip Runner and remote workspace storage. Companion: #18798 (now merged upstream) fixes a pre-existing inbox test synchronization race. Its test-only commit `110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with attribution so this PR can run the complete CI matrix. Related: #17800 registers missing preservation for the selected source run. #18678 holds unfinished remote exports. #18718 repairs a prior warm-controller projection failure. #17916 addresses a separate legacy warm-copyback policy case. ## What Changed - Discover the failed writer through the company-scoped task/workspace binding during Retry and authenticated message admission. - Reuse the exact prepared-source validator for released remote workspaces. Registration changes neither the lease policy nor its stop receipts. Physical recovery retains its full allocation checks. - Check a newer failed preparation separately before allowing an older, repaired source to release continuation. Provider events, leases, results and execution ownership remain disqualifying evidence. - Select the bound saved workspace through the continuation's verified repair receipt. - Add PostgreSQL safety cases and an HTTP Retry workflow with real file capture and saved-copy reads. Document the execution contract. - Reuse #18798's completion barrier for the existing inbox cancellation test; preserve its eight-slot limit, rejection and coalescing assertions. ## Verification - RED before production edits: the exact stopped writer has no `workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass on the baseline. - GREEN: the expanded Retry/message/preview selection passes 17 cases. All 627 tests in the five affected recovery suites pass on recovery commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop, ownership, saved messages and duplicate Retry. - The HTTP workflow registers maintenance, keeps execution held before save, reads back the saved file bytes, and admits exactly one continuation with `resume_preserved_workspace`. A registration-only repair first failed after save because it did not reconcile the newer Retry source. The incident-shaped pair passes with an independent active stranded-task action present. - Greptile identified the path where the newer setup failure already owns a reconciliation hold. Its new regression failed before the correction (saved files, zero queued continuations); the contradictory-dispatch control passed. All four HTTP variants pass after the correction, covering both hold shapes and both dispatch-evidence controls. - Repository typecheck and build passed. Server typecheck and build also passed after the recovery changes. Module-boundary and whitespace checks pass. - The full local Vitest command failed in the GitHub fast-response acknowledgement test (`delivery_unknown`: one response write instead of zero). That isolated case passes on this branch and unchanged master; the original full-run cause remains unestablished. A sibling run passed 199 test cases but timed out in workspace-suite database cleanup. Unchanged master also passed all 126 workspace cases and timed out in the same cleanup hook; all five acknowledgement outcomes passed in that baseline comparison. These local failures remain recorded rather than being reported as suite passes. - The first hosted matrix failed the pre-existing inbox cancellation assertion (expected nine computes, saw eight). The same assertion failed in [master CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840) and reproduced locally before applying #18798's existing repair. All six inbox HTTP/database cases pass after that repair. Final head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed test change to the verified recovery code. - [Final hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377) passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci / verify`, `ci / e2e`, typecheck, build, every server/chat/workspace group, Runner checks and all eight browser shards. There are 54 successful check results, two informational neutral results and three skipped results, with no pending or failing results. - Greptile and the Paperclip reviewer both report 5/5 for that final head; no review threads remain open. The Paperclip review check is informational (`neutral`) and records its local-testing limitations. - The file workflow uses real PostgreSQL, subprocess file capture, encrypted saved objects and production recovery code. Provider resume/stop calls are simulated at the environment driver boundary. This does not prove a live Daytona or Codex recovery. No deployment or original-task retry is claimed. ## Risks - File-maintenance discovery must not authorize another executor. Existing writer, coordinator and physical allocation checks still own that decision. - A false bootstrap summary cannot prove that a newer run never executed. The new path also requires its immutable native-adapter claim, unresolved preparation state, absent coordinator/process/session/lease/result evidence, and no dispatch events. - Additional database reads run only during explicit task admission or Retry strategy selection. They are company/task scoped and add no polling loop. The saved-writer query returns one matching snapshot and omits prompt bodies. - No schema, credential, provider profile or deployment setting changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code execution and test tools. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:25:24 -06:00
DottaandPaperclip	1c9ca5829f	perf(ci): overlap Runner workloads and cache debug workspace builds (#18794) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect its server and Runner contracts. > - Recent full runs waited on general server files and the final Runner lane. > - That Runner lane ran package tests before native server tests. > - PR jobs also rebuilt unchanged debug workspace binaries. > - This pull request overlaps those workloads and reuses exact trusted build inputs. > - The benefit is a shorter critical path with the same test coverage and AWS authorization. ## Linked Issues or Issue Description Refs #18544, #18773, #18774, #18798, #18799. **What existing behavior does this improve?** The wall time of required PR tests and the precision of affected-test selection. **Current behavior** General server and native files use two workers. The second Runner Vitest lane runs its package shard and then the native server group. Each Runner job compiles debug workspace units again. A Vite raw import also retains its consumer for unrelated source edits. **Proposed behavior** Use three isolated workers for general server files and four for native files. Reuse the two Runner slots for a complete package suite and a complete native server suite. Restore all debug binaries only on an exact input key. Record raw imports as explicit file inputs. ## What Changed - Separate the Runner package and native server workloads without adding CI jobs. Preserve the old command arguments for master-pinned workflows, local commands, and other shard counts. - Increase file workers while retaining fork isolation, serial cases, and the shared-database guard. - Add an exact debug workspace cache. Derive binary targets from Cargo. Include source, fixtures, generated inputs, manifests, compiler identity, flags, Cargo configuration, and checkout path in its key. Discard incomplete restores. Only verified master pushes can save it. - Keep older PR checkouts working when they lack the new cache helper. - Track Vite `?raw` imports as repository file inputs. Keep unknown loaders, untracked files, and opaque effects conservative. - Add cache and coverage guards and document the behavior. AWS runner authorization is unchanged. - Keep paired test benchmarking available when install hooks correctly disable materialized dependency reuse. The old benchmark asserted a cache hit in that safe fallback. - Wait for active dispatch work to finish before asserting idle reads. Analyze the two 16,400-row database fixtures before their queries so they have current planner statistics. Suppress only PostgreSQL notices inside the cleanup transaction to avoid logging every cascaded table; warnings/errors remain visible. Row counts, real queries, assertions, database reset, and timeouts remain unchanged. ## Verification - 196 affected CI tests passed, including routing, coverage, selection failure fallbacks, shared-database serialization, and existing/new cache guards. Final worker/cache adjustments also passed their focused checks. - `pnpm build` and `pnpm -r typecheck` passed locally. Full test coverage is being verified by the Linux CI profile; `pnpm test:run` was not repeated as a full serial local run. The paired CI benchmark passed. Both paired benchmarks passed, and full CI passed again after integrating newer master commits. The external Paperclip Review app is the only pending status. - Actual debug artifacts remained fresh after simulating checkout timestamps: Cargo completed in 0.82 seconds; the content key was unchanged. This is local reuse evidence, not a measured GitHub cache transfer. - Same real UI inventory for a retention-test-only change: 861 files; retained selection fell from 413 to 401. All 12 omitted files are raw-input consumers; edits to their inputs remain covered by regression tests. - The first local worker comparison is invalid. Development dependency links disappeared during its second run. The two-worker baseline also had two macOS path assertions in company-skills. Neither result is counted as a successful speedup. Dependencies were restored. A local repeat was interrupted before producing results. The completed Linux comparison below supplies the valid paired evidence. - [Paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41671244016): baseline and candidate ran the same 113 general-server files and 1,777 cases on one host. Wall time fell from 381.29s to 346.89s (9.0%). Both exited successfully and file/case inventories matched exactly. The fixture command took 31.77s versus 28.65s. This is one sequential pair, excluding setup, queueing and the full CI critical path. - [Repeat paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41675496862): the same 113 files and 1,974 cases took 485.06s in the baseline versus 368.54s in the candidate (24.0%). File and case inventories again matched exactly. The baseline includes newer master changes than the first pair, so these are two independent comparisons, not repeated timings of an identical tree. - Initial full AWS profile: package suite passed 4,891 tests (289 files, one skipped); native suite passed 172 tests (9 files, 12 pre-existing platform/feature skips). Native execution took 239.06 seconds; the older sample ran 165 native tests in 391.29 seconds. The package lane now takes 11m01 including setup and debug compilation. This is an indicative cross-run comparison, not a paired full-CI speedup claim. - The initial run failed two UI fixtures, one serialized inbox settlement assertion, and one dispatch idle assertion. The two UI failures reproduced locally. The fixes preserve authorization, admission, and zero-idle-read assertions. All four affected files passed locally: 48/48 tests. Token gates passed. All four also passed in the next AWS profile. That profile passed every other workload but timed out the 16,400-row keyed attention fixture at its existing 180s limit; this is accounted for by the explicit fixture-statistics change. Local focused comparisons pass both large cases before and after; their roughly 33s test execution is unchanged locally, so this is not claimed as an additional speed gain. A full local file run passed both large cases (7.57s / 18.05s) but hit the unchanged 30s teardown deadline on its first test while emitting thousands of cascade notices. Cleanup now suppresses those notices within its transaction; the complete file then passed 34/34 in 107.72s. The latest-head AWS run passed the previously failing shard: the keyed bulk case took 40.25s (previously exceeded 180s), the keyless case took 8.02s, and the complete attention file took 89.61s. The 16,400-row fixtures and original timeouts remain intact. - [Full CI profile before master integration](https://github.com/paperclipai/paperclip/actions/runs/41675497699) passed on commit `5858956d4ac2c99c911ad2447b63e1f8fc84e31e`, including all tests, build, typecheck and final verification. Greptile reviewed that exact commit at 5/5; the only actionable review thread is resolved. The separate Paperclip Review app is still queued. - [Final merged-head CI](https://github.com/paperclipai/paperclip/actions/runs/41678956312) passed on `499aa206a6e03bf8f3249692ea731367a000591d`: 48 successful jobs and one intentionally skipped file-selection job. Backup checkpoint qualification and legacy security status passed. Greptile reviewed this exact commit at 5/5. The PR is conflict-free; only the separate Paperclip Review app remains queued. - Integrated current master after #18798 and #18799 merged the inbox and UI fixture fixes separately. Preserved master's inbox settlement fix and verified its entire file (6/6 tests). Those three files are no longer part of this PR's diff. Removed the temporary `profile-ci` label after both successful benchmark pairs; the normal planner still schedules every job because the CI configuration changes are global inputs. - Second AWS profile: native tests took 229.97s with the same 172 passes / 12 skips; package tests took 427.44s with 4,891 passes / 10 skips. The complete package job took 10m44, including setup and 79s debug compilation. - The PR uses the trusted workflow from master. Candidate worker/layout code runs now; workflow-level debug restores begin after merge and the first successful master cache writer. ## Risks - Extra workers can expose load-sensitive tests. The final full CI profile passed, but one run is not a long-term flake-rate measurement. - The cache is an optimization only. Misses rebuild, and test commands still execute. A new Cargo workspace member requires a cache-contract review. - Compiler alternatives remain experiments. Required TypeScript checks are unchanged. Typechecker experiment: three sequential checks per tool on identical built projects, with TypeScript 7.0.2, Bun 1.4.4, and tsrs 0.20.1-ts7.1.0-beta. Median seconds (TypeScript / Bun / tsrs): shared 0.429 / 0.083 / 0.157; UI 2.554 / 0.390 / 0.740; server 9.949 / 1.194 / 1.782. TypeScript and Bun passed all three projects. All tools caught the same four deliberate assignment, property, nullability, and generic-constraint errors. tsrs reported one additional server TS2345 diagnostic in the run-dispatch adapter, so it did not pass the project comparison. These local compiler times exclude generation and build steps; they are not full CI savings. ## Model Used OpenAI GPT-6 Codex. Tool use and code execution assisted this change. The exact deployment identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:17:53 -06:00
DottaandPaperclip	8c2502503d	fix(github): preserve repository access during PR webhooks (#18805) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Agents need current repository access before they can start work. > - Managed GitHub access checks run while webhook delivery can update the same grant. > - Pull request events update webhook health but do not change repository permissions. > - The refresh guard treated those health updates as access changes and could exhaust its retry. > - This change keeps repository preparation available during normal pull request activity. ## Linked Issues or Issue Description **What happened?** A pull request webhook can make a managed GitHub repository refresh fail with `github_access_changed`. Repository discovery then omits the failed connection. Workspace admission can report that repository access is no longer available, even after the checkout succeeds. **Expected behavior** Pull request activity must preserve valid repository access. Installation changes and newer access snapshots must still stop a stale refresh. **Steps to reproduce** 1. Start a managed GitHub access refresh. 2. Process a pull request edit while the provider request is in flight. 3. Repeat on the bounded retry. The old guard rejects both checks because `lastWebhookAt` changes. ## What Changed - Use `accessRevision` and `lastAccessRefreshAt` as the existing access fences. Stop using webhook delivery time as an access fence. - Allow the existing commit merge to retain the latest webhook timestamp and health after a pull request event. - Exercise real webhook processing and PostgreSQL during refresh. Cover a pull request event, an access conflict followed by that event, and installation suspension. - Document the distinction between access changes and delivery health. ## Verification - Red: both pull request race cases failed on the old code with `github_access_changed`. - Green: all seven focused refresh cases pass. These include the existing concurrent access-change checks and the new suspension case. - Command: `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts -t 'binds a managed GitHub identity'`. - Full affected group: 526 tests pass across the tool-access, repository, and webhook suites. - `pnpm -r typecheck` and `pnpm build`: pass. - Current-head CI: 54 checks pass. The duplicate Paperclip review is neutral. Two Storybook checks are skipped. Greptile and the completed Paperclip review both rate this commit 5/5 with no findings. There are no unresolved review threads. - The duplicate local `pnpm test:run` did not complete. Its first four chat shards passed. It was stopped after CI completed on this commit; the full CI test matrix passed. The complete affected local group passed separately. - Retained earlier failure: the first affected-group attempt passed 525 tests and had one socket hang-up in the profile attention feed test. That test passed alone without changes. The unchanged base tool-access suite also passed all 498 tests, and the complete fixed affected group then passed. The cause of the socket closure remains unproven. - Environment: source checkout on macOS arm64, Node 26.4.0, pnpm 9.15.4, disposable PostgreSQL. Provider responses and Cloud transport use controlled fixtures. The webhook handler, refresh logic, transactions, and saved grant reads are real. - Live acceptance remains unverified. The available CLI token cannot list GitHub App installations. The affected staging deployment still serves the old code. This PR does not deploy or recover the original task. ## Risks - Removing an access fence could admit stale permission data. The access revision and refresh timestamp checks remain. The real suspension webhook regression confirms that stale provider data cannot restore removed access. - This change adds no retry, timer, schema change, or credential forwarding. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, code editing, and terminal tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 11:15:49 -06:00
Nicky LeachandClaude	6ef37398c8	fix(build): fit typecheck and image build in 4 GiB (#18737) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors, agents, and image builds run `tsc` to check and compile the repository. > - The repository uses TypeScript 7, which is a native binary. Node heap flags do not limit it. > - The server is one large program, so one `tsc` process needs about 10 GiB. > - The operating system stops `tsc` under a 4 GiB memory limit. > - This pull request checks the server in layers, removes the server from the CLI program, and removes declarations and checks from the image emit. > - The benefit is that the full typecheck and the image `tsc` step now complete in 4 GiB. ## Linked Issues or Issue Description I found no duplicate pull request or issue. A search for `tsc`, memory, typecheck, and OOM found only runtime memory work, which is a different subject. **What happened?** `tsc` used too much memory, and the kernel stopped it with exit code 137 in resource-constrained environments. Measured peaks before this change: | Command | Peak | | --- | --- | | Server `tsc --noEmit` | About 10 GiB | | Server `tsc` emit, as in the Docker build | About 10.7 GiB | | CLI `tsc --noEmit` | About 7 GiB | | `pnpm -r typecheck`, whole process tree | About 10.2 GiB | **Expected behavior** The typecheck and the image `tsc` step complete under a 4 GiB memory limit. **Steps to reproduce** 1. Start a Linux container with `--memory=4g --memory-swap=4g --cpus=4`. 2. Install the dependencies on the base commit. 3. Run `pnpm -r typecheck`. The kernel stops the process with exit code 137. 4. Run `pnpm --filter @paperclipai/server build`. The `tsc` step needs more than 4 GiB. **Paperclip version or commit** Base commit `09162fe893`, with TypeScript 7.0.2. **Deployment mode** Not applicable. This issue applies to development, CI, and image builds. It does not change the server at runtime. ## What Changed - Add `server/scripts/typecheck-layers.mjs`. It reads the import graph on each run and runs one `tsc` process per layer. The layers are the files below the largest import cycle, the cycle, the files above the cycle, and the tests. Each layer reads the earlier layers as declarations. - Add `scripts/run-tsc.mjs`. It adds `--singleThreaded` when `PAPERCLIP_TYPECHECK_LOW_MEMORY=1` is set. The server layers and the UI typecheck use it. - Add `scripts/typecheck-low-memory.mjs`. It checks one package at a time in this mode. It has no `package.json` entry, as `.greptile/rules.md` requires. - Change the server `typecheck` script to build only the Runner TypeScript output. It does not build the Runner release binary now. `pnpm build` and `typecheck:build-gaps` still run the full Runner build and its generated-file checks. - Change the `tsc` step of the server `build` script to `scripts/run-tsc.mjs`. The `Dockerfile` sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, so the image emits JavaScript without checks or declarations. Without this variable, `build` still does a full check for npm packages and CI. - Use a non-literal specifier for the `@paperclipai/server` import in `cli/src/commands/run.ts`. The import result already had no useful type, and the literal specifier added the full server to the CLI program. - Add a layered server typecheck step to the `pr-trusted` typecheck job, because `typecheck:build-gaps` does not run this script. - Update `AGENTS.md` and `doc/DEVELOPING.md`. ## Verification Results on Linux use a clean copy of this branch, a new `pnpm install`, Node 24, Rust 1.97.1, 4 CPUs, and a 4 GiB memory limit with no swap. Results before the merge of `master`: | Command | Result | Peak non-reclaimable memory | | --- | --- | --- | | `node scripts/typecheck-low-memory.mjs` | Exit 0, 0 OOM kills | 3,492 MiB | | `PAPERCLIP_TSC_EMIT_ONLY=1 pnpm --filter @paperclipai/server build` | Exit 0 | 3,656 MiB, from the Rust release build | | Server single-program `tsc --noEmit --singleThreaded` | Exit 137, 1 OOM kill | Reached the limit | Results on macOS on the merged head, with peak memory for the full process tree: | Command | Before | After | | --- | --- | --- | | `pnpm -r typecheck` | 10.2 GiB, exit 0 | 5.47 GiB, exit 0 | | `node scripts/typecheck-low-memory.mjs` | Not available | 3.81 GiB, exit 0 | | CLI typecheck | 7.06 GiB | 1.68 GiB, before the merge | Other checks: - Compiler tests: new tests run the real compiler on a small fixture. They prove that the layers emit declarations and pass, that a renamed export fails the layer that imports it, and that a changed type fails the test layer. - Declaration imports: a new compiler test proves that types from `.d.mts` and JSON imports stay correct in later layers. On the real server, a wrong use of `AgentFileManifest` in the cycle layer failed with `TS2353`. - Coverage: a test on the real server source asserts that each source file is in one layer and that no import goes to a later layer. The test layer contains the remaining files. `src/__tests__` stays out of the root list, as in `server/tsconfig.json`. - Error detection: I added one type error to one file in each of the four layers. The typecheck reported all four errors and exited with 1. - Cross-layer errors: I renamed an export in the lower layer. The cycle and upper layers reported the error and the typecheck exited with 1. I reverted these temporary edits. - Emit: full `tsc` and `--noCheck --declaration false --declarationMap false` gave byte-identical JavaScript and source maps for 2,249 files on the base commit. - Runtime: `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` passed with a bundle from the emit-only build. - CLI: the esbuild bundle keeps the dynamic import, and `node --check` passed. - Tests on the merged head: `typecheck-layers`, `prepare-runner-vendor`, `cli-invocation-safety`, the CLI `runtime` tests, and all 503 `.github/scripts/tests` tests passed. Before the merge, `release-verify-workflow` and `runner-release-crate-cache` passed. - Full `pnpm --filter @paperclipai/server build` passed before the merge. Failures during this work: - The first low-memory run failed in the UI with `TS5023: Unknown compiler option '-b'`, because the flag came before `-b`. I corrected the argument order and added a test that runs build mode with the flag. - The first rename check did not rename the export, because BSD `sed` does not support `\b`. Its exit 0 was not evidence. I ran the check again with `perl`, and the result above is from that run. - The first version of the real-compiler fixture had a `TS2774` error in its own source. The test found this error, and I corrected the fixture. - CI failed `server-package-build-script.test.ts` on `1abc5e5a43`, because that test reads the steps of the server `build` script, and I had moved them to `build:prepare` and `build:assets`. My first search for readers of this script did not include `server/src`. I removed those scripts and `build:image`, restored one `build` script, and kept the test unchanged. - `GOMEMLIMIT` did not help enough, and I removed it from the plan. On these steps, `3GiB` saved at most about 300 MiB. With default checkers, the cycle layer was stopped with and without it, and the lower layer went from 7 seconds to 151 seconds. Not run or not proven: - I did not run the full `pnpm test:run`, a root `pnpm build`, or a full `docker build`. I expect CI to run the test and build gates. - I did not measure the Docker build steps other than the server build. - I did not run the 4 GiB checks on x86-64. - I did not measure the merged head in the 4 GiB container. The macOS result on the merged head is 3.81 GiB, and the merge added one source file to the lower layer. ## Risks - The margin is small. In the 4 GiB container, the cycle layer peaks about 600 MiB below the limit, and the Rust release build in the image build peaks about 440 MiB below it. Growth of the largest import cycle uses this margin first. - `node scripts/typecheck-low-memory.mjs` applies only to `pnpm -r` packages. These commands still check the server as one program and need more than 4 GiB: `pnpm build`, `test:e2e:runner:typecheck`, `test:lifecycle-baseline:typecheck`, and the `tests/speko-native` project. - A local flow that used the Runner binary from a previous typecheck must now run `pnpm --filter @paperclipai/paperclip-runner build`. - The image does not include server declarations now. I found no consumer in this repository. A downstream image that reads `server/dist/*.d.ts` must change. - An image build does not find type errors now. A typecheck in CI must find them before the image build. - The import scan uses a regular expression. A false edge can only make a layer larger. A missed edge causes a `TS6307` failure, so it cannot skip a file. - The new CI step adds about one minute to the typecheck job. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, and code execution. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-11-03 08:53:35 -08:00
DottaandPaperclip	5f1b59ba2c	fix(db): preserve binary data in checkpoint backups (#18650) ... Reviewed and verified at 69792ccbcf. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 10:47:19 -06:00
DottaandPaperclip	1423d745e8	feat(server): expose owned sleep blocker diagnostics (#18632) ... Reviewed and verified at 26e5a22b3c. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-03 10:45:10 -06:00
DottaandPaperclip	f79264d253	perf(recovery): project current wait facts during action reconciliation (#18793) ... ## Thinking Path > - Paperclip manages work and execution for AI agents. > - Recovery actions keep unresolved work visible and preserve its owner. > - Action reconciliation checks whether a durable wait or another live path has appeared. > - Its current-wait check collected a complete retry fingerprint through seven queries. > - This caller used only the durable-wait flag and reason. > - This pull request reads those facts in one fresh statement and shares the existing reason policy. > - The local service uses less CPU. Fresh mutation guards and full retry fingerprints stay in place. ## Linked Issues or Issue Description **What existing behavior does this improve?** Active recovery action reconciliation. **Subsystem affected** server/ — orchestration and recovery. **Current behavior** Each admitted source action collects blockers, children, interactions, approvals, work products, runs and wakes. It sorts and hashes these records. This caller uses only the durable-wait flag and reason. **Proposed behavior** Read three current existence facts in one company-bound statement. Use the same reason policy as the complete fingerprint collector. Keep child checks, outside-action checks and mutation services. **Reason and benefit** The actual local PostgreSQL service uses less CPU across 8, 32 and 100-action synthetic cohorts. This removes unused work on mostly empty results. The live workload mix and fleet benefit remain unknown. **Breaking changes** No API or schema changes. Query facts now share one statement snapshot. Scalar issue fields still come from the source cohort read. **Additional context** No duplicate projection change was found in the public PR search. The roadmap has no overlapping core feature. This is a narrow performance change. ## What Changed - Add a current durable-wait reader and share reason selection with the full fingerprint collector. - Use the reader only in active action reconciliation. Keep the full collector for retry and child policy. - Add PostgreSQL regressions for reason priority, scope, board waits, native holds, healthy children and concurrent decisions. - Add a manual reproducer and local paired results. Each invocation uses a unique evidence directory and exclusive output files. ## Verification - Focused PostgreSQL regressions: 11 passed. - Separate actual-service proof: 7 passed. Four CPU tests were excluded. Results, records, semantic relation edges and notifications match the pinned baseline. Late query facts and a concurrent action decision are covered. - Retained primary local paired CPU medians: 8 actions 7.091 → 2.886 ms; 32 actions 21.922 → 8.374 ms; 100 actions 68.384 → 32.718 ms. All nine CPU and wall pairs improve at these sizes. The data file retains all 72 primary samples and ranges. - Warm application counts: 81 → 33 queries at 8 actions; 321 → 129 at 32; 1001 → 401 at 100. Rows rise from N to 2N because each EXISTS statement returns one row. - See `doc/benchmarks/2026-11-03-active-recovery-projection.md` and its JSON data for commands, provenance and limits. - `pnpm -r typecheck` and `pnpm build` passed on head `f2f8c4a755ac67679d78c6005f420a21a101c1ad`. - Shipped manual helper smoke with `--proof-only`: 7 passed, 4 timing tests skipped. All count-proof sample arrays are empty. Unique output names and cleanup of the three temporary source files were verified. - Exact-head CI completed with 52 successful checks, three intentional skips and a successful Snyk status. Greptile completed successfully at 5/5 with no inline findings. The `review` workflow passed commitperclip dependency and PR quality gates. No separate Paperclip Review check or verdict was published. The local `pnpm test:run` attempt passed all six chat shards (1,834 tests), then reported 62 Muse failures and one gateway failure in the general-server batch. The initial cases each reached the unchanged 15-second timeout. The Muse case was followed by fast failures. After the full batch breadth and complete CI coverage were established, the owned known-failed run was interrupted once with SIGINT. Remaining general-server, serialized and workspace scopes were not completed by that local invocation. Its interrupted tool exit code is not a pass. Each initial timeout case then passed in one fresh isolated control. A single fresh full Muse file passed all 256 tests. Source review found no changed projection path in either initial case. The original timeout causes remain unresolved; later passes and green CI do not erase them. The first reserved CPU run lost its raw files after a harness path error caused an unintended repeat to overwrite them. Its summary and log remain historical evidence. The separately retained repeat is the primary inspectable evidence above. It had no renewed quiet reservation. The runs are not pooled. The corrected untimed proof and public helper prevent output replacement. No third CPU run was made. ## Risks - One statement observes its query facts together. This does not reproduce every race history of seven concurrent statements. - Existing action update predicates, issue mutation locks, board ownership, native reconciliation holds and company guards remain in place. - The evidence is a local source service benchmark on macOS and real PostgreSQL. It is not deployed CPU or request latency acceptance. The census was startup-adjacent. Conditional scale suggests only a modest fleet return. - The original local batch timeout causes remain unexplained. No hosted, Sentry or OTLP improvement is claimed. No merge or deployment is requested. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and isolated controls; the broad local failure/interruption is described above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 10:13:03 -06:00
DottaandPaperclip	f7ad1dc7ca	fix(workspaces): make attached project repositories writable (#18770) ... ## Thinking Path > - Paperclip lets people assign work to AI agents and inspect the result. > - A task can use repositories from more than one project. > - Remote workspace preparation treated every additional project as a read-only source. > - This rejected valid Git repositories before the provider started, even when no source was read-only. > - The task page then hid the preparation reason behind a generic failed-attempt notice. > - This pull request prepares authorized repositories as writable task copies and shows redacted workspace validation errors. > - People can work across both projects and see the reason if folder preparation fails. ## Linked Issues or Issue Description **What happened?** A remote task with a selected Git-backed project and another mentioned Git-backed project failed before provider execution. It reported `remote_workspace_readonly_sources_unsupported`. Neither project had a read-only setting. The task notice showed only that the attempt failed. **Expected behavior** Each attached repository should be writable inside the task workspace. This should preserve source access checks and saved work. A workspace validation failure should explain its cause on the task page. **Steps to reproduce** 1. Configure remote workspace storage and a native sandbox environment. 2. Select a Git-backed project for a task. 3. Mention another authorized Git-backed project in the task. 4. Start the task. Preparation rejects the additional source before it starts the provider. **Paperclip version or commit** Reproduced against `dc9f387edc`, before this change. **Deployment mode** Managed remote workspace storage. Local regression checks use disposable PostgreSQL and independent filesystem processes. Related merged work: Refs #18403 and #18504. Those changes established remote storage and metadata-only references. They still reject configured cross-project repositories. No duplicate writable-repository repair was found. Refs #18774 for the test-only retention-fixture repair reused to unblock CI. ## What Changed - Add every configured Git repository of an authorized referenced project to `.paperclip-repositories/` as an editable task copy. - Retain saved paths, refs, pins and files. Recheck retained project access and source trust. Keep explicit read-only, local-source import and remote feature gates. - Keep committed snapshot repository coverage separate from later requests. Honor configured default refs, resolve remote branch names, and publish secondary checkouts only after checkout succeeds. Share configured-ref resolution between root and attached-repository scripts. - Include redacted workspace validation messages in the task interruption projection and existing failure notice. Preserve Retry permissions and render messages as plain text. Explain retained-folder access and trust denials while preserving the dedicated low-trust project-access path. - Add source, authorization, redaction, UI and save/restore regressions. Update the implementation contract and workspace maps. - Reuse the reviewed test-only repair from #18774. Batch the shared-workspace history fixture, verify its complete privacy-source map, and drain owned test bodies before cleanup. Production retention behavior is unchanged. ## Verification - Red before repair: four source-resolution cases failed with the reported read-only error. The task-error projection test also failed because it omitted the cause. - Green: 49 source and transport tests, 16 task projection and redaction tests, and 10 UI tests. The assembled preparation check passes. All 48 remote workspace lifecycle tests pass. The full preparation suite passed all 126 assertions but its cleanup hook exceeded 30 seconds. The unchanged base reproduces the same cleanup timeout with all 126 assertions passing. The focused preparation check finishes cleanly. - The real PostgreSQL/Git lifecycle checks cover both first-turn and later project attachments. They verify the configured release branch, saved edits, replacement allocations, unchanged source folders, committed archive contents, and released writers. - Review regressions were red before repair: the default ref became null, later attachment failed during restore, non-default remote branch checkout failed, and invalid secondary refs left partial checkouts. They now pass. - The local browser shows the workspace validation explanation and Retry action. The fixture uses a saved failed run; this proves the error display, not a successful live sandbox run. - `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and token checks pass. - A local two-turn acceptance probe repeated at `17ed67eaa7` used real Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files in both task repositories. The services saved them, removed the first allocation and its Git sources, restored another allocation, and Codex read the saved files and wrote a follow-up in both folders. This uses the real source resolver and persistence services with a local command-runner stand-in for sandbox transport. It does not qualify Daytona, managed subscription forwarding, or the original task. The first probe waited for stdin; closing the test harness pipe fixed that setup error before provider work. - `pnpm test:run` passed the first chat shard, then failed one chat response concurrency assertion in the second shard. That case passes in isolation on this head. The same isolated chat case passes on the unchanged base. Both corresponding unchanged-base chat shards also pass (351 and 324 tests). The cause of the earlier wider-suite chat failure is not established. The preparation cleanup timeout and chat failure remain part of this record. - CI at `17ed67eaa7`: all other verification jobs passed. Browser shard 6 initially reported a request callback from the preceding test after teardown; its single unchanged-code retry passed. Server shard 10 timed out the 2,100-row retention confirmation fixture at 15 seconds on both attempts. That exact case passes locally, but the repeated CI failure required a repair. Its setup repeatedly rebuilt a shared-workspace privacy map. The logs and failures remain part of this record. - Reused the test-only repair from #18774 without changing its source file. At `ab774e9418`, all 54 retention tests and all 25 ordinary production privacy tests pass locally (79 total). Full CI [41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903) passed on that commit, and its exact-head Greptile review was 5/5. The fixture preserves its 2,100 rows, query-bound assertions, privacy-source map, and completed-history test deadlines. - The additional review identified retained-source access/trust denials that still bypassed the workspace explanation. Real-resolver regressions reproduced both plain-403 failures before the correction. Both now yield a safe workspace validation explanation, while the dedicated low-trust denial and saved workspace remain unchanged. The optional duplicate ref-resolution block is also consolidated. - Final review corrections: 65 source, transport and error-projection tests passed, followed by all 48 lifecycle cases against an owned disposable PostgreSQL database (113 total). The initial combined command skipped the explicitly gated lifecycle suite; the subsequent dedicated invocation enabled and passed all 48. Full workspace typecheck and build pass. The first standalone lifecycle launcher lacked the repository TypeScript loader; rerunning with that loader resolved its import setup error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full CI [41654422329, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2) passed, Greptile is 5/5 on that exact head, and all review threads are resolved. The first attempt timed out the unchanged voice-session report/history case at 15 seconds; 3,027 cases in its shard passed. That exact case passed locally, then its single unchanged-code shard retry passed. The wider-suite timeout cause remains unproven. The completed Paperclip advisory check is neutral; its earlier findings are corrected. No staging deployment or original-task recovery is claimed. - Master advanced while checks ran, producing conflicts in two adjacent test additions. Merged `27d35314be` into this branch as `d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both workspace-error tests and the audited Stop/receipt tests. Kept workspace maps to minimal corrections under the updated map rule. All 82 source, transport and interruption cases, all 48 real-database save/restore lifecycle cases, full typecheck, build, and map checks pass locally on the combined code. Current-head [CI 41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214) and Greptile are pending; the earlier verified head has not been merged. ## Risks - Additional remote project repositories are now editable task copies. Shared source folders and upstream push permissions do not change. - Snapshot coverage uses existing workspace metadata. Older workspaces capture their previous inventory at reservation; successful save publication advances coverage and snapshot identity in one transaction. No SQL migration is needed. - A retained copy can contain files from a project that is no longer mentioned. Preparation rechecks that project's current access and trust policy before reuse. - Explicit read-only descriptors still fail. Source-less projects remain metadata-only. Local folders still require import. No migration or provider-pack update is needed. - Error display reads only workspace validation errors. The existing run-secret registry and username censor run before display truncation. A redaction failure omits the explanation. - The retained fixture repair pauses its row trigger only within the isolated seed transaction, rebuilds the exact source map, and restores the trigger before commit. A rollback regression and 25 ordinary privacy cases pass. Its paged-case test budget honors the existing bounded inner waits; production deadlines are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving model ID and context-window size are not exposed by this session. The test-only fixture changes are reused from #18774, whose model record identifies OpenAI GPT-6.1 Sol. The optional live acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex CLI 0.165.0. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 15:33:44 +00:00
DottaandPaperclip	b40e6edaea	perf(inbox): combine startup reads and reduce refetch pressure (#18790) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Mine shows the work and failures that need the viewing user's attention. > - A reload waits for account access, task rows, parent captions and failure state. > - Those reads form several browser request stages before useful rows appear. > - Live invalidations can also restart the same task-list request more than once. > - This change supplies startup data together and removes redundant refresh work. > - It preserves current authority, failure presentation, archive state and row stability. ## Linked Issues or Issue Description Refs #18645. Related: #18642. This change implements the read-receipt behavior independently on current master. It does not use that PR's branch or patch. **What happened?** Mine starts its primary request after the session, company and board-access reads. It then requests parent captions and rich failure context. Four observed loads spent another 0.45–0.70 seconds after the primary rows arrived. One slower load waited 1.71 seconds for parent context. The same run observed two HTTP 429 responses. These timings identify request waits; they do not attribute the delay to SQL or Node CPU. Read receipts also invalidate general task collections and dashboards. Overlapping parent and child invalidations can start two requests and abort the first. **Expected behavior** The initial page should use one account-safe startup projection and one complete Mine response. A read receipt should refresh the affected viewer's inbox state. Each invalidation batch should refresh a covered query once. Required captions and current failure state must be ready before first reveal. Existing rows should remain usable during a transient refresh error. **Steps to reproduce** 1. Open a populated Mine inbox with parent tasks outside the visible index and a current failed run. 2. Reload with browser HTTP cache retained and application query state empty. 3. Inspect the session, company, board access, task-index and context request sequence. 4. Open or mark a task read and inspect the unrelated collection refetches. **Paperclip version or commit** Baseline: `44a9a043b80c18314d8dd89556f444dae696d3d6`. **Deployment mode** Authenticated board UI. Local proof uses the real APIs and database with a synthetic session resolver. No hosted candidate deployment is claimed. ## What Changed - Add optional, same-user `boardBootstrap` data to the existing session response. Seed account-keyed company and board-access reads from it. Keep ordinary endpoint reads for fallback, retry and live reconciliation. A returning account cannot reuse revoked cached access. Cache collection and later real refetches remain effective. - Add the board-only Mine snapshot endpoint. Reuse the canonical issue selector, private-task filtering, run-read decisions, secret redaction and interruption/recovery presentation. Assemble the fixed 100-task page, parent captions and bounded 200-run failure window on the server. Start independent reads together. Optional source errors keep successful task rows usable. - Keep array-shaped source caches for archive and undo. Key Mine source, context and reveal state by account. Validate the response's user stamp before committing it. Remove array-only cross-tab Mine broadcasts, which cannot carry its paired failure context. The server retains authenticated in-flight coalescing. - Scope read-receipt invalidation to the affected viewer's inbox, badges and chat unread state. Keep company and task audit views fresh. Preserve genuine task mutations, run updates and subscription-gap reconciliation. - Remove covered scalar-prefix invalidations within a batch. Preserve predicates, exact matches and different query/refetch constraints. A later event can still refresh again. - Retire Mine in-flight reuse after local company activity or run lifecycle changes. A post-archive read starts fresh. Old work keeps its admission slot until it settles, even if every HTTP client aborts. The eight-compute limit stays in place. - Add API, component and browser regressions. Extend the production timing harness with matching API/UI artifact receipts, retained-cache reloads, request status counts, layout checks and optional controlled network latency. Update API and verification documentation. ## Verification - Full `pnpm -r typecheck` passes. - Full `pnpm build` passes. - Token gates, map checks and diff whitespace checks pass. - 72 Inbox component tests pass. Startup-specific checks pass, including account switching, revoked cached access, malformed/absent bootstrap data and cache collection. The canonical authenticated bootstrap route also passes against real PostgreSQL. - Six real PostgreSQL Mine route cases pass, including full equivalence with legacy authorized projections, privacy, fixed bounds, membership revocation, coalescing, archive freshness and aborted-client admission. Four focused legacy coordinator controls pass. Six assembly tests pass. - Refetch and invalidation regressions pass. Meaningful pre-fix tests observed two request starts and one abort for overlapping invalidations; the fix produces one start and no abort. The archive race test observed the post-archive GET joining the old body before the fix. - All 11 existing startup browser checks pass, including Tasks/Mine code loading, desktop/mobile bounds, non-streamlined Coordinator, task navigation, failed preloading and Back position. The nine focused startup, fallback, partial-source error, rate-limit recovery, archive/undo and geometry cases also pass. The 429 cases use the inherited retry window. Archive and keyboard undo are corroborated through the real API and reload. Authentication and failure fault injection in these cases are explicitly simulated. - Matched production before/after timings are complete. Full aggregate Vitest coverage is left to CI. At 15:15 UTC, the final-head runner gate and `review` workflow pass. The 45 build/test/policy jobs are queued; the Build job has no `ubuntu-latest` runner assigned and has executed no steps, so remote aggregate results remain unverified. Paperclip Review and Superagent Security Scan are in progress. Superseded runs for this PR were cancelled. No current-head check has reported a failure; the CI checklist remains open. The remaining step is runner allocation and completion of those external checks; persistent queueing needs the repository CI owner's attention. - Greptile completed on final head `ecde295565309275c4f2fae903caf6892f1eaafd` at 5/5. The map-scope finding is fixed and resolved; there are no open review threads. ### Matched browser timings Five alternating baseline/candidate pairs per network scenario, measured serially in headful Chrome 156.0.8078.80 on Apple silicon with Node 24.22.2 and a 1440×900 viewport. Both production UIs use their matching real API revision against one fixed disposable PostgreSQL fixture: 34 tasks, 31 visible task rows, one paused agent and one terminal failure. No active agent/provider work. HTTP cache is retained; each initial sample reloads the document and clears application query state. Warm navigation and Back retain query state. CPU and throughput are native. The 100ms scenario uses CDP minimum request-to-response-header latency without request interception. It is a controlled request-dependency experiment, not a staging reproduction. Values are median milliseconds with min–max in parentheses. Paired saving is baseline minus candidate for each alternating pair; its median differs from the difference of arm medians. Positive saving favors the candidate. | Network / journey | Baseline | Candidate | Median paired saving | | --- | ---: | ---: | ---: | | Native / initial rows | 187.6 (128.9–208.5) | 191.7 (130.3–210.9) | −23.3 | | Native / Recent → Mine | 100.17 (99.90–101.87) | 100.07 (99.88–101.37) | −0.16 | | Native / task → Back | 70.45 (56.63–72.02) | 66.64 (56.52–67.71) | +4.94 | | 100ms / initial rows | 930.3 (914.1–1015.7) | 799.0 (697.1–826.0) | +211.8 | | 100ms / Recent → Mine | 100.09 (97.73–100.16) | 100.18 (96.65–100.98) | −0.02 | | 100ms / task → Back | 74.97 (62.77–75.82) | 75.47 (61.51–76.19) | +0.35 | The controlled-latency initial pairs all improve, saving 108.3–217.0ms. Native initial pairs have mixed signs (−67.5 to +78.1ms), so they establish no reliable native page-speed improvement. Warm and Back timings are effectively unchanged. Median primary-request start is 110.4→135.2ms natively and 497.8→480.9ms with 100ms latency; these runs do not establish a large startup-only timing gain. API requests through the two-second post-reveal window fall from median 32 (32–37) to 27 (25–28) natively, and 37 (33–38) to 30 (25–32) with 100ms latency. All 20 samples preserve the same ordered 31 task links, first-row DOM identity and geometry. All recorded API responses are HTTP 200; zero unfinished requests, 429s or JavaScript errors. Lower 429 incidence on staging remains unproven. Measured source: baseline `44a9a043b80c18314d8dd89556f444dae696d3d6`, candidate `4c12b2ef0ecf0754676e99bd6b9399e84408646c`. Final head `ecde295565309275c4f2fae903caf6892f1eaafd` changes only two feature-map files and existing browser test expectations since that measured candidate; production executable inputs are identical to the measured candidate. Asset hashes, API health revision receipts and every raw sample were retained. These results do not claim hosted-page improvement or deployment of this PR. ## Risks - Startup enrichment adds work to the session read. Failure of this optional projection falls back to the ordinary company and access endpoints. It does not invalidate a valid session or replace per-request server authorization. - Account changes, stale cache entries and same-company navigation can expose old presentation data. Account keys, principal stamps, reveal scopes and canonical bootstrap replacement have dedicated negative cases. Permission failures hide cached task rows; transient network, 429 and 5xx failures preserve them. - Current failure state can differ from the historical failed run. The snapshot uses the existing canonical rich projection and verified missing-ID coverage. Partial context failures retain an explicit retry path. - Mutation retirement follows events delivered to the current server process. This adds no cross-replica invalidation protocol or distributed read-after-write guarantee. - New UI assets require the matching API. Older UI/session consumers retain their existing response and endpoint behavior. There is no database migration, provider call or deployment change. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, browser automation and delegated source review. The exact runtime model ID and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 09:28:57 -06:00
Dotta	5dee4993f1	fix(server): distinguish aborted HTTP responses in access logs (#18781) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use HTTP logs to investigate task updates and database contention. > - Node starts each response with status 200 before it sends any headers. > - The custom HTTP formatter printed that status when the client disconnected before a response. > - This made an aborted request look like a successful task update. > - This pull request labels incomplete responses as aborted and preserves error severity. > - Operators can distinguish HTTP delivery from database commit evidence. ## Linked Issues or Issue Description - Refs: #18753. - Related: #14369 removes writes from legacy comment attribution reads. This change addresses a separate logging defect. The missing-write incident in #18753 remains unconfirmed. We reproduced a misleading `PATCH /issues/:id 200` log when the client received no HTTP response. We did not reproduce an HTTP 200 delivered before the mutation committed. ## What Changed - Label premature connection closure as `request aborted before response` or `request aborted during response` in both success and error log paths. - Log aborts as warnings, including on routes whose successful polling logs are normally silent. Keep explicit errors and 5xx responses at error severity. - Keep private webhook paths and query values out of aborted-request logs. - Add real HTTP disconnect coverage and real PostgreSQL task-update tests with synthetic data. - Document that HTTP completion logs do not establish database commit or rollback. ## Verification - Baseline: the new HTTP disconnect test failed because the log said `PATCH /api/issues/synthetic-task 200` at info level, while the client received no response. - `node node_modules/vitest/vitest.mjs run server/src/__tests__/http-log-aborted-response.test.ts server/src/__tests__/http-log-redaction.test.ts server/src/__tests__/issue-update-lock-contention-postgres.test.ts` passed: 81 tests in 3 files. - `pnpm -r typecheck` and `pnpm build` passed. The build used a second clean clone at the exact PR commit. - GitHub CI completed at `e3e96d836119df5fa1c7f22615484265de5aa700`: 52 checks passed and two Storybook checks were skipped. Greptile returned 5/5 with no findings. - Local `pnpm test:run` passed all six chat shards (1,834 tests). Its general server stage then reported four failures. We stopped the run after preserving these results and comparing the failing cases. The later workspace and serialized stages were not completed locally. The full local suite is not green. - The four observed failures were the two OAuth concurrency cases in `tool-access-service.test.ts`, the bounded workspace conflict case in `workspace-runtime.test.ts`, and the `enableNativeRunner`-disabled stop-control case in `muse-control.test.ts`. - Comparisons used the same sanitized environment on unchanged master `2b14ec03a48f17606db53fd7fc0edc53bc930409` and this PR. Both OAuth cases pass in isolation on both revisions. The full tool-access file passes all 498 tests on the PR; master has a different 15-second timeout, with 497 tests passing. The workspace case passes on the PR but fails port-reservation setup on master. The Muse case passes on both. These results do not explain or clear the original broad-suite OAuth and Muse failures. - Tested on macOS arm64, Node.js 26.4.0, pinned pnpm 9.15.4, and disposable embedded PostgreSQL. The API fixture uses a synthetic process-adapter agent. No model provider runs and no production database or real credentials participate. - The PostgreSQL fixture verifies an actual lock waiter with `pg_blocking_pids`. A blocked issue update does not return 200 before release. After release, status, monitor policy, inline comment, and audit receipt are persisted before the completed response. - Holding the company row with `FOR NO KEY UPDATE` does not block these fixture writes. They return 200 with persisted data while the compatible parent-row lock remains held. - An injected database trigger failure returns 500 and leaves the issue, comment, and mutation audit unchanged. Two influence-attempt receipts remain under the existing documented cap policy. - The combined lock/disconnect/failure test verifies no delivered response, an aborted log, and rollback after an injected database failure. The injected failure is not evidence of the reporter's production root cause. - Source inspection at release `v2026.1023.0` (`1b4c803816e82b0265c8422e1a804327a7ed8c54`) finds the same unconditional success formatter and awaited issue-update transaction. That release was not runtime-tested. ## Risks - Log consumers that parse the literal method/path/status message must handle the aborted-request message. - Client disconnects during streaming or SSE responses also use the aborted warning. Expected disconnects may therefore produce more warnings. - A disconnected request may still commit after the client leaves. This change reports transport completion and does not cancel or retry mutations. - This PR does not close #18753. The original missing-write cause and a received-200-without-commit scenario remain unconfirmed. - Local full-suite failures remain unresolved despite green CI. Keep this PR in draft until that verification gap is accepted or resolved. > This is a targeted bug fix. ROADMAP.md does not list overlapping feature work. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, tool use, code execution, and parallel subagent investigation. - The exact runtime model ID and context window size were not exposed to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-03 09:26:03 -06:00
DottaandPaperclip	27d35314be	feat: connect experimental Cursor Cloud to native Runner (#18069) ... ## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 09:01:39 -06:00
DottaandPaperclip	e16de641ba	fix(heartbeat): retain Stop and receipt debt after setup failure (#18776) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A restored execution can fail during preparation. > - A preparation failure does not prove that earlier provider work never started. > - Replacing its result metadata can remove an existing Stop and falsely release its accounting hold. > - This pull request preserves those obligations at the terminal write. > - The benefit is honest cleanup and receipt debt without new execution authority. ## Linked Issues or Issue Description **What happened?** A recovered native run can fail a configuration check before dispatch in the new controller. The setup failure writes bootstrap metadata with `providerWorkStarted: false`. It also replaces the previous result metadata. This can remove a pending native Stop. Accounting then accepts the new bootstrap flag and releases the original hold despite an incomplete receipt. **Expected behavior** Keep the current run-Stop and receipt debt when setup fails after earlier native execution. Preserve an acknowledgement that commits during the failure write. Fresh setup that never dispatched must remain eligible for pre-provider settlement. Keep account-identity checks effective. **Steps to reproduce** 1. Save admitted native execution, an active provider checkpoint, an original held reservation, and an audited pending run-only Stop. 2. Recover the run and fail preparation with an account configuration error before the new controller dispatches. 3. Run ordinary cost reconciliation with an incomplete receipt. 4. Observe that the base removes the Stop and releases the hold as if the run never started. Related: #18682 repairs exact-session Stop cleanup after restart. This change protects setup failure before that path. #18702 handles AI connection-request waits. #18035 proposes broader accounting policy. #18778 separately handles retained cleanup across account changes. Account, budget, and receipt authority are unchanged here. ## What Changed - Mark setup-failure status writes through an internal lifecycle option. At the atomic write, a prior coordinator attempt bound to the same run, company and task prevents false pre-provider evidence, even with a stale loaded profile. Lock the coordinator before the run so an in-flight claim cannot be missed. Selected execution input alone does not imply provider work. - Keep genuine pre-dispatch setup eligible for settlement after fresh native input is persisted, whether the loaded profile is current or stale. - Preserve only exactly bound native run-Stop fields from the current database row during failure writes, including the intentional handoff marker. Keep the latest acknowledgement when the update waits behind another writer. Normal Stop delivery still advances its own metadata. Replace unrelated stale result fields. - Add real PostgreSQL regressions for receipt debt, original holds, stale snapshots, late acknowledgement, fresh setup, dispatched work, and foreign or task-scoped metadata. - Update the execution contract. ## Verification - On untouched master, the recovered-setup regression fails six assertions: Stop removal, false bootstrap evidence, false accounting settlement, lost debt, and released hold. The fresh-setup control passes. - A second regression fails four assertions with the first repair present when the loaded profile lacks the current saved execution. The atomic current-row check fixes that case. - The focused safety selection passes. It includes real row-lock contention. A controlled concurrent writer seeds audited acknowledgement state while the terminal update waits. The product code retains that state and creates no acknowledgement or provider turn. This race test does not prove physical provider termination. - Fresh native and legacy setup can settle without a receipt. Dispatched native and legacy work keeps its debt. Foreign run, company, issue and task-scope metadata stays excluded. Unrelated old result fields are replaced. - After the final review fixes, all three completion, Stop and budget-cancellation suites pass: 121 tests. The exact `pnpm run typecheck:build-gaps` entry point passes after the final source and service-test edits. Module-boundary checks pass. An ordinary adapter pre-provider receipt control also passes. - Native review also found that fresh selected input could be mistaken for prior execution. Real-admission loaded/stale cases fail against the earlier head; a prior-attempt-only control passes. The correction uses the bound coordinator attempt and includes a foreign-task negative. Reservation `providerStartedAt` is stamped at reservation creation and is not used as provider-work evidence. - A row-lock regression also exposes a first execution claim committing during the terminal write: the prior statement snapshot misses it and falsely releases the hold. Locking the exact coordinator before the run fixes that race and preserves the existing claim lock order. Concurrent Stop acknowledgement tests remain green. - Review found two preservation gaps. New real-database reds show handoff classification loss and overwritten budget Stop delivery progress. Failure-only preservation and the bounded handoff marker address them. - Initial helper-import and reservation-order fixture errors are retained separately from causal failures. No assertion or deadline was relaxed to pass the regression. - Current-head Greptile is 5/5. The optional native review stopped before completing and returned NEUTRAL; this is not substantive approval. Its prior finding was reproduced and fixed. Coordinator and independent source/evidence reviews found no additional issue; neither is a hosted recovery test. - These are local database tests with saved provider state. No hosted recovery, physical provider termination, historical-state repair, or deployment is claimed. Full repository tests and build passed in current-head CI: 55 successful checks, three expected skips, and the explicitly incomplete optional native review returned NEUTRAL. Required verify and e2e aggregates passed. ## Risks - The current database row must retain canonical Stop state while replacing other result metadata. The preservation is limited to the same native run, company, issue and run-only Stop scope. - Setup-failure history checks keep uncertain work pending. They do not create a receipt, acknowledgement, reservation or charge. - The setup-only write option leaves ordinary adapter pre-provider receipts unchanged. No account-identity, budget, dispatch, or provider authorization gate is bypassed. - Already overwritten historical state needs separate evidence and repair. This change does not reconstruct it. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 08:49:28 -06:00
DottaandPaperclip	2b14ec03a4	fix(ai-connections): remove account execution locks (#18778) ... ## Thinking Path > - Paperclip manages AI agents and the accounts that supply their model credentials. > - Several runs can use the same AI connection at the same time. > - Runtime preparation treated unfinished credential cleanup as exclusive account ownership. > - Manual account tests also held a company credential mutex for the entire provider test. > - A failed cleanup could therefore cancel fresh runs before the provider started and keep them in automatic retries. > - This pull request removes that execution gate and those read and probe locks. > - Each invocation keeps its own credential home. Credential updates still use the existing freshness and authorization checks. ## Linked Issues or Issue Description **What happened?** Fresh runs stopped with `ai_connection_busy` and the message “The selected AI account is still completing credential cleanup. Retry shortly.” Pending cleanup from an earlier invocation blocked every new invocation of that account. Concurrent manual account tests also waited on a company mutex. **Expected behavior** AI connections support concurrent runs and account tests. Cleanup for one private runtime must not prevent another private runtime from using the same account. **Steps to reproduce** 1. Prepare a managed OpenAI account and retain an earlier runtime whose cleanup has not finished. 2. Prepare another run with that account. The old implementation rejects it before provider work starts. 3. Alternatively, keep one manual account test open and start a second test with the same account. The old implementation prevents the second provider callback from starting. **Paperclip version or commit** Reproduced with two PostgreSQL regressions on `17e8bd773543d62ab3f444c55255acfd82575efd`. Both fail before the fix and pass after it. **Deployment mode** The reported deployment used authenticated Cloud. Local reproduction uses disposable PostgreSQL and fixture credentials. No live provider credentials are required for these regressions. Related search found #15952, which concerns account removal and provider application archival. It does not remove this runtime cleanup gate. This fix does not introduce the account concurrency cap requested in #14564. ROADMAP.md was checked; this change restores concurrent account use. ## What Changed - Remove the account-wide cleanup assertion and cleanup mutex from run admission. - Read ordinary subscription credentials without acquiring the company file mutex or grant and secret write locks. Only an actual durable replacement recovery enters the existing write transaction. - Remove the mutex held across manual account tests. - Retry retained cleanup without making new runtime preparation wait for it. A home-specific cleanup check considers only that exact private home. - Keep cleanup attempts for one runtime idempotent. Keep physical retirement, credential freshness, reconnect, revocation and company checks. - Update concurrency, refresh and cleanup regressions, and the implementation contract. ## Verification - Before/after: the pending-cleanup and simultaneous-probe regressions both fail on the original code and pass with the fix. - Affected suites: 186 tests passed across `ai-connections.test.ts`, `antigravity-ai-connections.integration.test.ts`, and `retained-google-credential-cleanup.test.ts`. They use real PostgreSQL, encrypted fixture credentials and controlled provider callbacks. - `pnpm -r typecheck` passed. - `node scripts/check-maps.mjs` and `git diff --check` passed. - `pnpm build` passed on the rebased branch. The server typecheck and production server build both passed again on final commit `03bd3973bdf85108bbd82355abdb217f0e1c83bd` after the runtime-home ownership changes. - Caller coverage: all 166 cases pass across native credential handoff, managed Google subscription writeback, model discovery, heartbeat configuration and hired-agent execution. The full hiring file passed all 79 cases after its admission assertion was corrected. Together with the affected suites, 352 targeted cases pass. - The final home-binding check passed all 188 cases across the account, managed Google writeback, model discovery and retained-cleanup suites. - Review follow-up: two real-PostgreSQL cases pass for quota refresh and concurrent Google account tests. The quota case observes a credential row update in progress, verifies that the runtime takes no account file mutex, and confirms that the private home receives the committed replacement token. The Google case completes a second provider callback while the first callback remains open in a separate private home. - Greptile rated the runtime change 4/5 and found only misplaced map patch notes. Those additions were removed. The implementation documentation remains in `doc/connections/AI-CONNECTIONS.md` and the implementation contract. - `pnpm test:run` was attempted. Chat partition 1 passed 351 expanded cases. Partition 2 passed 324 cases and failed one Teams file-receipt API case at its 15-second deadline: `refuses generic resolution with stale version transfer preconditions without changing receipts`. That case exercises publication resolution, not AI account execution. Its isolated rerun passed. The remaining local partitions did not run after that failure. This is not a claim that the full local suite passed. - Wider caller checks initially identified old serialization expectations and two cleanup retry cases. Updated assertions now require concurrent account use. The partial-removal correction lets a retained home finish removal after a durable credential handoff. All affected caller cases pass, including the full hiring file. - Live provider concurrency and deployment recovery have not been qualified. This PR does not claim that the reported Cloud deployment contains the fix. ## Risks - A new invocation can use the currently saved credential while another runtime still owns a refreshed copy. The existing writeback checks decide which credential can replace the saved value. - Actual credential updates and durable replacement recovery retain atomic storage transactions. This change removes execution serialization, not the integrity checks on stored credentials. - Unfinished cleanup retains its private home and its existing idle hold. This change does not certify a previous provider process as stopped. - There is no database migration, credential format change, permission expansion or ambient-account fallback. ## Model Used - OpenAI GPT-6 through Codex desktop, with source editing, shell execution, PostgreSQL fixture tests and read-only browser diagnosis. The session does not expose the exact deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 07:56:37 -06:00
DottaandPaperclip	86b365f992	feat(auth): add independent agent configuration read permission (#18766) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents need to inspect configuration and instructions when they set up teammates. > - Configuration reads currently depend on configure or suggestion grants. > - Legacy standard agents did not receive those newer grants. Some self-read routes also lose the target agent identity. > - This pull request adds an independent read grant and applies only that grant to eligible existing agents. > - The benefit is that agents can inspect and verify configuration without new write authority. ## Linked Issues or Issue Description Refs #15283. Related earlier proposals: #8064 and #12050. Those proposals target older authorization code; this change uses the current configure/suggest model and adds a read-only rollout for existing agents. Related bounded peer metadata proposal: #14160. **What existing behavior does this improve?** Agent configuration, configured skill, instruction, and revision reads through the existing APIs and native instruction tools. **Current behavior** An agent needs `agents:configure` or `agents:suggest-changes` to inspect a peer. Older standard agents can hire a teammate but cannot verify the saved instructions. Configuration and revision routes also reject self-reads because they authorize a company resource instead of the target agent. **Proposed behavior** Add `agents:read` for redacted configuration and instruction reads. Keep `agents:configure` as read/write and preserve suggestion-grant compatibility. Standard new agents receive the read grant. A one-time migration grants only reads to eligible existing standard agents. Individual routes authorize the target agent and grant scope. **Reason and benefit** Read access can be granted without configuration changes, rollback, or environment probes. Legacy agents can verify teammates without receiving new write permissions. ## What Changed - Add `agents:read` to the shared permission contract and standard-agent defaults. - Accept read, configure, or suggestion grants for configuration reads, with target scope and the responsible-user ceiling. - Fix target-aware self and scoped configuration, skill, and revision reads. - Add an audited data migration. Skip low-trust, built-in, inactive, pending, terminated, setup, and scoped configuration/suggestion cases. Preserve existing read scopes and later revocation. - Update the specifications, default-permission documentation, and agent access map. ## Verification - Focused acceptance: 276 tests passed across seven suites on commit `a37314df88`: authorization, real-key HTTP access, route authorization, canonical instructions, default grants, migration replay, and lifecycle configuration. - The HTTP journey uses a real hashed agent API key and a disposable PostgreSQL database. It verifies redaction, scoped reads, denied edits and rollback, cross-company isolation, and both agent and responsible-user revocation. - New read-grant regressions failed before the authorization change. A realistic paused/pausing/resuming migration fixture also failed before the lifecycle filter correction and passed afterward. Setup-held paused agents stay excluded. - `pnpm -r typecheck` and `pnpm build` passed on the rebased branch; the final lifecycle migration correction also passed the database package typecheck and build. The latest commit changes only a test assertion. - `pnpm --filter @paperclipai/db check:migrations`, `node scripts/check-maps.mjs`, and `git diff --check` passed. - Full current-head [CI run](https://github.com/paperclipai/paperclip/actions/runs/41641531426) passed: typecheck, build, all general and serialized test lanes, Runner checks, all eight browser shards, canary dry run, policy, and aggregate gates. The local full-suite attempt was stopped after exposing the outdated cross-company 403 assertion; interrupted local full-suite attempts are not counted as passes. ## Risks The migration deliberately expands redacted configuration and instruction visibility for eligible existing standard agents in the same company. An absent older write grant does not deny the new read capability. It adds no write grant. Scoped configure/suggestion grants and all existing read-grant scopes stay intact. Administrators can remove the independent read grant after the one-time rollout; ordinary updates and startup do not restore it. Configure and suggestion grants still imply reads within their own scopes. Cross-company configuration/history reads now return 404 through the common target-aware helper. No schema change or credential-value disclosure is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The host does not expose a more specific runtime model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 07:26:27 -06:00
3 people	17e8bd7735	fix(runner): save native checkpoints when the sandbox disk is full (#18705) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox save two things after each turn: the provider checkpoint and the workspace export > - An agent can fill the sandbox disk during its own work, for example with a package install > - The turn still finishes, but both saves need free sandbox disk, so they fail and the task becomes blocked with no clear reason > - This pull request makes the checkpoint work on a full disk and makes a full disk visible in the export failure > - The benefit is fewer stranded checkpoints, and an operator sees at once why an export failed ## Linked Issues or Issue Description Related: #18715 added the shared-principal response-recovery control synchronized below. #18644 reduced the scratch disk that the workspace capture uses. #18172 added the disk-free streaming fallback to the Daytona directory sync-out. Both were deployed when this failure occurred. Neither changes the checkpoint copy, and neither can help when the disk has no free space. **What happened?** A native agent ran a package install for a large monorepo in a 10 GB Daytona sandbox. `df` in the run showed `10G 10G 4.6M 100%`. The agent committed its work and reported `done`. Then: 1. The provider checkpoint made a filtered copy of the provider directory inside the sandbox (`tar -cf archive; tar -xf archive -C snapshot`) to apply its exclusions. On the full disk, tar failed. The run recorded `runner_remote_checkpoint_snapshot_failed` without the reason, and the partial copy stayed on the disk. 2. Workspace finalization failed three times, and file preservation failed six times. The capture builds its archive inside the sandbox, so each attempt failed. Every failure was recorded as `remote_workspace_operation_failed` / `export_interrupted`, so nothing named the full disk. 3. The task went to `blocked`. The only copy of the commit was in the retained sandbox. **Expected behavior** The checkpoint does not need free sandbox disk when the provider can stream. A workspace export that cannot fit says that the sandbox disk is full. **Steps to reproduce** 1. Run a native agent with a provider whose checkpoint has exclusions (for example ACPX) in a small remote sandbox. 2. In the turn, fill the sandbox disk (for example `pnpm install` of a large repository, or `fallocate` of the free space) and finish the task. 3. The checkpoint fails with `runner_remote_checkpoint_snapshot_failed`, and every workspace export attempt fails with a generic code. **Paperclip version or commit** Observed on `410b0e27015`. This change is based on `5955dd4eb4c`. ## What Changed - Synchronize the shared-principal response-recovery regression with its saved comments and empty final deadline. Cover an in-flight discovery invalidation. Keep every authority lock, exact publication, error, and no-retry assertion. This changes test synchronization only. - `syncRemoteRunnerDirectoryOut`: when the in-sandbox checkpoint copy fails with a full-disk tar diagnostic, remove the partial copy and pass the exclusions to the provider sync-out (`exclude` on the directory mapping). The Daytona sync-out streams the directory without sandbox disk when it cannot write its archive. Other copy failures and timeouts still throw `runner_remote_checkpoint_snapshot_failed`. They now also remove the partial copy. - `remote-workspace-transport.ts`: the helper install and the capture scripts report `remote_workspace_sandbox_disk_full: ...` when Node reports `ENOSPC` or `EDQUOT`, or a Git or tar child reports "No space left on device". `command()` passes only that exact message through, like the existing unfinished-Git-operation message. - `native-workspace-finalizer.ts`: a remote save that fails with that message adds it to the failed operation's stderr. The failure code, the retry budget, and the recovery flow do not change. ## Verification - Current-head infrastructure interruption: the [Build job](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125923826158) completed the actual build and production bundle boot, then the runner received a shutdown signal during the clean public Grok npm-install check. GitHub marked that step cancelled. The exact host-stop cause was not retained. One bounded rerun of only the failed Build and dependent `ci / verify` jobs passed on the unchanged head. [Build attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41637668555/job/125931400745) completed successfully, including the clean Grok npm-install check. No product code, assertion or workflow changed for this interruption. - CI failure retained: commit `6c5937fcae` failed [chat shard 1/3](https://github.com/paperclipai/paperclip/actions/runs/41634638570/job/125913957217) because the shared-principal fixture saw zero saved comments after `worker.ready` and zero active work. The other chat child passed. The exact case passed once locally and then 30 times on the same Node 24.22.2 version as CI; those passes alone do not explain the CI interleaving. - Controlled RED/GREEN: invalidate the real database discovery once while its read is in flight. The worker correctly retains a future reconstruction deadline, but the old initial-pass/active-count barrier reproduces the same zero-versus-one comment assertion. Waiting for both saved comments and final `active=0`/`nextWakeAt=null` passes. Six real-PostgreSQL controls pass on final commit `a60e5629e8`, including endpoint independence, source revocation, and external lock failure/recovery. No sleep, timeout increase, weakened assertion, or product scheduling/authorization change was added. - A local server typecheck first lacked the checkout's built Runner declarations. After the declared Runner TypeScript build, the server typecheck passes on `a60e5629e8`. Final-head remote verification is complete: 54 successful checks, 3 expected skips, no failures or pending checks, and successful `ci / verify` and `ci / e2e` aggregates. Greptile is 5/5 on this exact commit with no unresolved review threads. - Merge preparation: rebased onto master `b22ef22d56` and kept both test-import additions. Added the checkpoint fallback and capacity-failure contract to `doc/execution-semantics.md`. On rebased commit `6c5937fcae`, 47 focused checkpoint, transport, finalization and retained warm-projection tests passed across three files. The name filter skipped 1,040 unrelated cases. Map and whitespace checks passed. The final-head CI and Greptile results are recorded above. - `npx vitest run server/src/services/native-runtime/native-session-executor.test.ts server/src/services/native-runtime/native-workspace-finalizer.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts` passes (1,152 tests). - `npx vitest run server/src/services/remote-workspace-transport.test.ts server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts` passes (30 tests). - New tests: - Checkpoint, with real files: a full-disk snapshot leaves a partial copy. The provider then gets the exclusions, the session files are kept, the package caches are left out, and the partial copy is removed. - Checkpoint: a permission failure and a timeout still fail, remove the partial copy, and do not call the sync-out. - Transport: the real capture scripts run with an `ENOSPC` preload, for the helper install and for the archive write. Both report the full-disk message and leave no partial attempt. A message with extra text still collapses to `remote_workspace_operation_failed`. - Finalizer: the operation stderr names the full disk and keeps `workspace_sync_out_failed`. Other failures keep the bare code. - With the source change removed, the 3 checkpoint cases and the 2 transport ENOSPC cases fail. - `npx tsc --noEmit -p server/tsconfig.json` is clean. - Suites that refer to the changed error codes (`explicit-native-continuation`, `native-recovery-notice-work`, `native-run-finalizer-telemetry`, `native-workspace-export-retry`, `remote-workspaces.integration`) pass. In `chat-channels.integration`, 2 cases fail locally on unmodified `master` too (a bundled skill path outside the checkout, and a chat run timing assertion). They are not related to this change. - Manual recovery of the observed task: the retained sandbox was resumed with free space, file preservation was requested again, and the saved workspace contained the agent's commit. ## Risks - The fallback runs only after the snapshot fails with a full-disk diagnostic. The normal path does not change. - Provider exclusions are not anchored to the directory root. In the fallback, a nested entry that has an excluded name (for example a nested `tmp` or `config.toml` in a Codex home) is also left out. The fallback never copies more than the snapshot, so credentials stay out of the checkpoint. - The fallback depends on the provider honoring `exclude`, which the sync protocol requires. The Daytona and Kubernetes providers honor it. - A workspace export on a full disk still fails. This change only names the cause. A full fix needs one of these: free-space reserve for the export, removal of re-creatable caches before the export, or a capture that streams its archive. Each of these changes behavior, so this pull request does not include one. - No migration. ## Model Used - OpenAI Codex, GPT-6 family, assisted with merge preparation, conflict resolution, documentation, test synchronization and verification. The harness does not expose the exact serving model ID or context-window size. - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). It did the root-cause analysis from run events, tool output, and sandbox inspection, wrote the code and tests, and ran local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com>	2026-11-03 07:09:52 -06:00
DottaandPaperclip	b22ef22d56	fix(runner): show provider failure details on tasks and runs (#18622) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts providers and records their run outcomes. > - Provider failures can contain useful details below a generic protocol error. > - Those details were lost at the sidecar, command receipt, and run projection boundaries. > - A blocked task therefore showed a generic failure even when the provider gave a clear reason. > - This pull request carries a bounded, sanitized provider failure through those boundaries. > - Operators can read the actual cause on the task and run pages. Existing recovery policy still controls retries. ## Linked Issues or Issue Description Refs #17543 for the related closed JSON-RPC classification work. Refs #17272 for provider recovery in the legacy OpenCode adapter. This change adds display evidence for native Runner providers and keeps recovery policy separate. **What happened?** A native Claude session returned an HTTP 429 message below an ACP `Internal error`. The task and run showed only a generic startup failure. Operators had to inspect the provider response to learn that the account was rate limited. **Expected behavior** The task and run show the sanitized provider message, provider name, startup or turn phase, and available HTTP status. Unknown categories keep the message. Display evidence does not grant a retry or replace a recovery decision. **Steps to reproduce** 1. Start a native Claude run whose account is rate limited during model selection, or return a structured provider failure from a Codex or OpenCode turn. 2. Let the run fail and open its task page. 3. Before this change, the page shows a generic failure. With this change, the page shows the provider explanation. The run page shows the same explanation after reload. **Paperclip version or commit** Source checkout based on `f8a888a072`. ## What Changed - Add one JSON Schema for `ProviderFailure` and generate matching Runner and shared TypeScript readers. Keep the Runner package independent of the app workspace at runtime. - Normalize known ACPX, Codex, and OpenCode error fields. Preserve unknown messages. Redact known credential values and credential patterns before applying the 4,000-character limit. Do not copy response bodies, headers, stacks, or prompts. - Carry startup failures through the sidecar, Rust Runner, durable command receipts, and private transport error details. Keep provider text out of generic transport error strings and Sentry failure messages. - Keep turn failures in authenticated run events. Recover the same evidence from persisted events during server finalization. Store the cause in the company-scoped run result. - Show one shared notice on failed tasks and run details. Keep existing retry permissions, scheduled retry state, and resolved-task behavior. - Reapply registered run-secret and current-user redaction to task display data. Batch secret lookups for only the visible provider causes. - Document the contract and update the capability maps. Add regressions for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode errors, redaction, durable receipt replay, restart finalization, and UI rendering. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions build, test, package, and validation jobs passed on `d49c79ab4a`. The security checks passed. Greptile gave this head 5/5 with no open review threads. Paperclip Review remains pending. Its review worker exhausted six workspace recovery attempts with export_interrupted. A fresh review request was accepted and is waiting for that saved workspace. The ordinary Retry action rejected reuse of the stopped request. This is an external review-worker blocker; no check was overridden. The duplicate local `pnpm test:run` was stopped after the CI test shards passed. - Focused server settlement, provider evidence, and restart tests: 93 passed. Runner admission and normalization tests: 103 passed. Focused Runner wire and terminal tests: 29 passed. The real Rust/sidecar command receipt replay test passed. - The new projection test failed before implementation because the task had no provider cause. The startup producer regression also failed before extraction moved ahead of the generic error-code replacement. Both pass with this change. - Live check: a rebuilt local sidecar received an actual Claude HTTP 429 during startup. The new cause kept the full provider message. A synthetic task in an isolated local app then reached the same provider error through the native Runner path. The dedicated run result, task API, task banner, and run detail all showed the same message after reload. The later reporting fix keeps run.error generic; a regression proves the display cause still survives restart and stays out of Sentry capture. - The first isolated app attempt failed earlier in `admission_credential`. An ordinary manual Retry reached the provider and produced the 429 above. This PR does not claim to fix that separate admission failure. An initial parallel local database test setup also failed; the serial restart suite passed. - Security review found that copying display text into run.error could forward it to Sentry. A regression failed with that behavior. The fix keeps the cause in its display field. All 38 settlement/restart tests and the server typecheck pass. - All 29 protocol contract tests pass, including bounds and extra-field rejection. An earlier full Rust core run passed. The later full run hit one 5-second receipt-limit timing assertion; its isolated rerun passed. The full Linux and Windows Rust CI jobs passed on this head. - Review regressions pass for explicit Codex driver identity, native Rust startup and resume receipt propagation, oversized OpenCode/ACPX errors, task read-time registered-secret redaction, and the standalone source closure. The focused Runner run passed 103 tests; projection/restart tests passed 16. - Read-time redaction edge cases passed: a missing decryption key omits only the unsafe optional explanation; expanded redacted messages remain visible within the Unicode character limit. All 14 task projection tests and server typecheck pass. - The package Build check found an old Pi assertion against Error.message. The probe now checks the exact closed classification in the authenticated receipt and requires the new provider cause. Both polling transport cases pass. A local credential-free Pi probe stopped at admission_installation because this checkout lacks the Pi distribution; the installed Linux package probe passed in CI on this head. - UI coverage includes startup and turn text, unknown categories, literal rendering of provider text, read-only retry behavior, and resolved-task suppression. Token gates and map checks passed. - The live check covers Claude startup on macOS. Codex, OpenCode, terminal failures, and restart paths have deterministic regression coverage. The change is not deployed. ## Risks - Provider messages are untrusted text. The boundary selects specific fields, redacts credential patterns and known environment secret values, bounds the message, and renders plain text. New provider-specific secret formats may need added redaction patterns. - Optional fields preserve compatibility with older producers and stored runs. Old runs without the field keep their current fallback text. No database migration is required. - The cause stays in company-scoped run data. This change does not send provider text to first-party Telemetry or add it to OpenTelemetry span attributes. - Normalized categories and retryability are display evidence. Existing recovery classifiers, retry limits, ownership checks, and approval gates keep their authority. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser verification. The host does not expose the exact serving variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:13:26 -06:00
DottaandPaperclip	718336eb04	fix(native): refresh prior warm controller projections (#18718) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native remote sessions keep their controller state in a database checkpoint. > - An admission can also leave a full local copy of a prior run's state. > - The prior owner can commit more state without changing its run ID. > - Warm admission skipped a refresh when the local and saved run IDs matched. > - This pull request refreshes that prior-run copy when no retained warm owner exists. > - The next turn can use the committed state while existing ownership checks remain effective. ## Linked Issues or Issue Description **What happened?** A prior warm run can complete after an earlier admission reads its controller checkpoint. The local copy can still contain a pending command or lack the final terminal event. A later admission sees the same run ID and skips restoration. It can then hold work against that old copy even though the database checkpoint has settled. **Expected behavior** Read the canonical prior-run checkpoint before reuse when no retained in-memory owner exists. Keep current-run state with its controller. Keep unresolved canonical work and invalid checkpoint bindings blocked. **Steps to reproduce** 1. Create a full prior-run local projection while a command is pending. 2. Complete that command and record the terminal in the canonical checkpoint under the same run ID. 3. Start an ordinary remote warm continuation without a retained in-memory owner. 4. Before this change, the local command remains pending. With this change, the completed command and terminal replace the stale copy. **Paperclip version or commit** Reproduced on master `36df8933f0` before changing production code. Related work: #18018 restored per-turn checkpoints; #18343 added controller replacement recovery; #18554 restored missing remote suspension receipts. This change handles a full prior-run projection whose run ID already matches the saved supervisor. It does not change the settlement guard or MCP compatibility policy. ## What Changed - Refresh a prior-run warm controller projection even when its run ID has not changed. - Keep current-run and retained warm owner authority on the existing path. - Add five executor cases for a settled source, a pending canonical command, a current run, legacy file-backed state, and an invalid checkpoint binding. - Add an opt-in Linux continuity case with a stale full local projection and two follow-up turns. - Update execution documentation and the paired session verification recipes. ## Verification - RED before the production edit: the executor fixture retained `pending` in its local command record when the canonical record was `completed`. - GREEN on the final commit: 48 focused tests passed across executor restoration, retained warm owner rotation, per-turn restoration, and remote settlement guards. - `pnpm --filter @paperclipai/server typecheck` passed. The server TypeScript check also passed after the integration fixture was added. - `node scripts/check-maps.mjs` and `git diff --check` passed. - The unit fixture uses the real durable controller serializer and the real executor entry. It stops at current tool authority before provider/backend creation. It proves restoration and guard behavior, not full task acceptance. - Actual protocol RED/GREEN: the same opt-in continuity fixture uses real PostgreSQL, TLS, the current Linux Rust Runner, and a deterministic fake Codex backend. With only the executor restored to its parent revision, continuation fails with `prior_owner_terminal_state_indeterminate` in the settlement guard after one provider turn (20.16 seconds). The PR source completes all three turns with one provider thread and final suspension (34.77 seconds). Hard recovery permits one provider-process replacement; the healthy final follow-up starts no additional process. Both runs use the same Runner binary and owned container. - The exact PR executor source was restored after the baseline comparison, its Git blob was verified, and the worktree is clean. The disposable local container was removed. This fixture proves the controlled stale-projection recovery path; it does not establish a live model or Daytona API result. - Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. Greptile is 5/5 on the current commit, with no unresolved review threads. No original task was retried and no staging sandbox lifecycle was changed. ## Risks - Admission without a retained owner performs one additional scoped checkpoint read when a full prior-run projection already exists. - The canonical checkpoint must still pass its company, task, session and record validation. A pending canonical command stays held. Current-run and retained owner state are not replaced. - The regression proves a stale local projection defect. It does not establish that a historical incident had those same local bytes. ## Model Used OpenAI Codex, GPT-6 family. The harness does not expose the exact serving model ID or context-window size. Capabilities: repository inspection, code generation, tools, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:12:22 -06:00
DottaandPaperclip	db8a273562	fix(native): publish fresh retry sessions for later turns (#18713) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native tasks save a provider session for later comments and monitor checks. > - The goal projection also records the run that owns the session event sequence. > - A fresh retry uses a new native session, but the projection accepted only the old session. > - The accepted checkpoint then rejected session publication because the goal source still named the old run. > - This pull request lets the current task executor advance that source after a terminal predecessor with no unfinished goal. > - Later turns can then use the fresh retry's saved session. ## Linked Issues or Issue Description Related: #12669, #18343. I searched open and closed PRs for source epochs and native task-session publication. I found no duplicate repair. **What happened?** A successful fresh native retry could retain its predecessor's task-session pointer. A later comment or monitor selected the old provider session. The goal source transition required both runs to have the same native session. The terminal checkpoint then saved a `suppressed_newer` publication result. **Expected behavior** The current task executor can publish its accepted fresh session. A later turn resumes that session. Delayed events cannot restore the old source or revive a cleared goal. **Steps to reproduce** 1. Complete a native turn and retain its task-session row. 2. Admit a fresh session on the same task without deleting the row. 3. Complete that turn, then send an ordinary follow-up. 4. Before this change, the saved pointer can still name the first session. **Paperclip version or commit** Reproduced on source commit `36df8933f0` with a PostgreSQL regression before the production change. **Deployment mode** Source checkout. The affected producer runs for local and remote native sessions. ## What Changed - Permit a different native session to advance the goal source only when its run holds the task execution lock and assignee, and the task is in progress. - Keep the terminal predecessor, native runtime, runner identity, absent or completed goal, source sequence, and conversation reset checks. - Test accepted session publication, a fresh heartbeat followed by an ordinary resumed turn, stale events, current-owner loss, and unfinished or unreadable goals. - Add an opt-in live Codex file-persistence check. Update execution documentation and both recovery maps. ## Verification - Red: the new PostgreSQL publication regression failed before the producer change. The saved session, display ID and goal source still named the predecessor. - Green: 72 distinct tests passed across the final focused runs of `runner-goals.test.ts`, `native-task-session-publication.test.ts`, `native-session-resumption.test.ts`, and `remote-warm-session-recovery.test.ts`. - The assembled heartbeat test admits three turns, opens two provider sessions, resumes the second session once, and stores three results. It uses a deterministic provider backend. - `node scripts/check-maps.mjs` and `git diff --check` passed. - Real local Codex acceptance passed on macOS arm64 with `gpt-6.1-sol` and explicit warm mode. Three heartbeat turns saved three exact file lines. The fresh second turn published its new normalized and provider session IDs. The ordinary third turn retained both IDs. The fixture closed its warm owners and removed its private credential copy and workspace. The production code was at `450e15e`; the final test-only commit selects explicit warm mode and uses the existing owned-home cleanup helper. The production change is identical in the final head. - The earlier per-turn fixture preserved all three file steps and selected the correct normalized session, but provider-thread reuse failed. `run.attach` reported that the Codex provider had pending events; the safe fallback used another provider thread. This remains a separate failure. The first fixture cleanup also failed on immutable instruction assets. The owned-home helper fixed cleanup, and the second per-turn attempt exposed the provider identity assertion directly. - Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. Greptile is 5/5 on the current commit, with no unresolved review threads. Broad local workspace checks were not duplicated; the focused regressions and actual provider acceptance above were run locally. No merge, deployment, or recovery of an original task is included. ## Risks - Source epoch changes affect saved goal state. The fresh-session path requires current task authority and refuses unfinished or unreadable saved goals. - The change starts no provider work and changes no prior-owner recovery guard. Unknown external outcomes retain their reconciliation hold. - A prior session remains held when it lacks the required authority or state proof. - Per-turn provider-thread reuse remains an open coverage gap described above. This change does not alter that attachment guard. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are unavailable in this session. The agent used code inspection, tools, and executable tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:12:00 -06:00
DottaandPaperclip	89e3c23062	fix(daytona): stream files when export copies cannot fit (#18712) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native task must save its files before finalization can release its writer. > - Remote workspace capture can produce an archive that fits on the sandbox disk. > - Daytona then makes another copy before it downloads that archive. > - The second copy can run out of space and leave completed work behind a save hold. > - This pull request uses the existing bounded stream when that copy cannot fit. > - The benefit is a saved archive without a second archive-sized sandbox allocation. ## Linked Issues or Issue Description Related: #18644 reduced capture scratch space. #18172 added the disk-free directory export. #18705 preserves provider checkpoints on a full disk and reports capture failures. This change handles the later single-file download copy. It does not depend on #18705 and does not replace its checkpoint fix. **What happened?** A native provider turn returned a successful result and captured its workspace archive. The Daytona outbound file guard then failed with exit 43 and `No space left on device` while it copied the archive to a reserved download file. Finalization and file-preservation retries could not pass this boundary. The workspace kept its writer hold. **Expected behavior** Download an existing archive through a confined sandbox process when its second copy cannot fit. Keep the source and all finalization holds until the normal save commits. **Steps to reproduce** 1. Capture a workspace archive in a Daytona sandbox. 2. Leave less free space than the archive needs for another copy. 3. Finalize the task. The outbound single-file guard fails before download. **Paperclip version or commit** Reproduced against `6d72243d8a`. The incident used the native sandbox runtime. This branch is based on current master. **Deployment mode** Cloud with native Paperclip Runner execution and remote workspace storage. ## What Changed - Remove partial outbound file copies after a failed snapshot command. Only copy capacity failures select the stream path. Permission, confinement, and transport errors still fail. - Use the existing acknowledged terminal stream to read each file as the sandbox user. Recheck canonical confinement and regular-file status. Use a no-follow open. Keep the sandbox disk limit, host free-space reserve, shared transfer deadline, and bounded teardown. - Keep host files in private temporary files until the complete batch has transfer receipts. Never pass a mutable live source to the privileged download API. - Add real-file and real-terminal regressions, a disposable Daytona acceptance case, and workspace verification notes. ## Verification - Red before repair: a real 5 MiB copy under a controlled file-size limit failed with exit 43. The same test passes after repair and compares every source and destination byte. - The ten capacity cases pass. They cover producer failure after output, source replacement, stream size limits, permission errors, timeout teardown, cleanup, and a successful retry. A two-file case completes the first stream and fails the second. Both old host files stay intact, both source files survive, and retry saves both files. An isolated early-publication mutation makes that test fail. - The initial complete Daytona package suite passes: 537 tests passed, 8 skipped. After the test-only review follow-up, all 60 focused transfer, stream, and diagnostic tests pass. The package typecheck also passes. - Real Daytona acceptance passes: an owned Ubuntu 24.04 sandbox stops and resumes, a copy to `/dev/full` reports ENOSPC, production export saves the exact archive hash, and a second save after reconnect passes. The test deletes only its disposable sandbox. This is provider-boundary acceptance, not a UI journey. - Assembled finalization acceptance passes with real Postgres and a live Linux x86_64 Daytona sandbox. The production control-plane port accepts one fixture result. The production workspace finalizer encounters the failed copy, saves the files, and commits the task as `done`. A new database client reads the committed result and exact file hash. The workspace has no writer or running save, and there is still one run and one result. No model turn runs. The target resolver is bound to the live provider fixture. Object storage uses the production encryption and local-disk service with an S3 admission label; this does not prove an S3 network transfer or a UI journey. - The live tests used the fixed transfer source with SHA-256 `857b3d8c9ad75170707d95277c17c051ce4582fa79a9c38331d51d2800b36af5`. That source is identical in this PR head. - `pnpm -r typecheck` passes locally. The duplicate full local test run passed all six chat shards before it was stopped during the server lane in favor of completed current-head CI; a duplicate local build was skipped. Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. - Greptile is 5/5 on the current commit with no unresolved review threads. - `node scripts/check-maps.mjs` and `git diff --check` pass. Map checks prove structure only. ## Risks - The fallback requires the Bash and GNU `dd`/`base64` tools already required by the existing stream path. - Terminal streaming is slower than the normal bulk download. It runs only after a copy capacity failure. - The workspace capture itself still needs sandbox space. This change does not fix a full disk before an archive exists. - The test injects a failed copy sink. It does not exhaust a shared host disk. No original task was retried, and no deployment or merge is part of this PR. - No schema, API permission, writer ownership, or retry-budget change. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell tools, browser diagnostics, and automated tests. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:11:41 -06:00
DottaandPaperclip	1bc3ed6dd5	fix(opencode): keep native tool calls pending until Runner settlement (#18711) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native OpenCode sends semantic tool calls through the durable Runner protocol. > - The Runner retains a call until its result is known. > - OpenCode and its HTTP bridge each stopped waiting after 30 seconds. > - A slow response could leave a call pending after the provider had moved on. > - A later run then failed the pending-call guard during attachment. > - This pull request keeps native calls under their Runner owner until settlement or cancellation. > - The benefit is one delivered result and a safe next run. ## Linked Issues or Issue Description Related: #18343 applies the durable wait policy to native ACPX calls. This change applies that policy to native OpenCode. **What happened?** An assigned tool call took more than 30 seconds. OpenCode reported an MCP timeout while the Runner still retained the request. The source turn could finish, but suspension failed. A later `run.attach` failed because provider tool calls were still pending. **Expected behavior** The native proxy must wait for the authoritative result under the Runner owner. Stop, bridge close, and lease expiry must still end the wait. Unknown effects must still block attachment. **Steps to reproduce** 1. Start a native OpenCode run with an assigned semantic tool. 2. Hold the controller's result for 31 seconds. 3. Observe the provider's MCP result and attach a later run to the session. **Paperclip version or commit** Reproduced against `6d72243d8af0362d73fbcb5331fd42a384800d4e`. The branch is based on `36df8933f0`. ## What Changed - The native OpenCode proxy enables a driver capability for durable tool requests. - That capability removes the bridge's separate request timer and sets OpenCode's fixed MCP client ceiling to one hour, consistent with the native ACPX sidecar. - Ordinary driver calls keep their 30-second deadline. Assigned external MCP connections keep their existing budget. - Cancellation, result replay, tool authorization, and the pending-call attachment guard keep their existing contracts. - The native regression holds a result for 31 seconds, then completes a second run with the same Runner process and no pending provider calls. - The deterministic provider fixture honors the emitted MCP budget and retains the initial completion contract across ordinary follow-ups. - Execution semantics and the session verification maps describe the wait and its safety checks. ## Verification - Red before production edits: the bridge aborted a durable request at the ordinary 30-second boundary. The driver emitted `30000` instead of the expected `3600000` client budget. - The final combined bridge, driver, proxy, and native protocol run passed: 95 tests in five files (71.17 seconds). - The native delayed-result probe passed with the current Rust Runner and current bundled proxy. It completed the later run with the same Runner PID and an empty pending-call ledger. - Actual provider check passed on macOS arm64 with `opencode-ai@1.18.34`, the changed source, and a local deterministic OpenAI-compatible model. One assigned write waited 31 seconds and saved one file. Both turns completed. The second turn retained the saved tool receipt. The write ran once across five model requests. - Bridge safety cases cover result replay without another action, ordinary timeout behavior, MCP cancellation, and teardown of an unanswered durable request. - Both runner TypeScript projects passed typecheck. The first attempt lacked the eval-kernel build; it passed after building that declared dependency. - Map alignment, module boundaries, and `git diff --check` passed. - The first native follow-up probe failed because its synthetic provider forgot the initial completion-contract revision. This caused a repair turn and another discovery call. The fixture now retains the contract, and the same probe passed. The failure is retained here. - Runtime acceptance used the implementation at `583d316ad4`. The later clarification commit changes comments, documentation, and a test name only. It does not change runtime behavior. - Full current-head CI passes, including typecheck, build, general and serialized tests, Rust/Runner checks, all eight E2E shards, and the `ci / e2e` and `ci / verify` aggregate gates. Greptile is 5/5 on this commit, with no unresolved review threads. No deployment or retry of the affected historical task was performed. The actual provider check used a local model; it does not qualify a deployed Linux sandbox or a live model service. ## Risks - A native call can wait longer while its Runner owner remains valid. Stop, teardown, and lease expiry still end the wait. The MCP client has a separate fixed one-hour ceiling. Runner ownership can renew beyond that ceiling. If the client deadline expires first, the operation outcome remains unknown and the existing hold must still block replay and run attachment. - This prevents new premature timeouts. It does not classify or replay an old request whose effect remains unknown. - No schema, migration, credential, or permission change is required. ## Model Used - OpenAI Codex, GPT-6 family. The harness does not expose the exact model ID or context window. Capabilities used: code generation, tool-assisted diagnosis, local execution, and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:11:24 -06:00
DottaandPaperclip	dd02595390	fix(db): retain hosted census construction callers (#18725) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use a bounded database census to find expensive read paths. > - The hosted server combines application code and capture wrappers in generated files. > - Generic HTTP owners suppressed caller capture. Generated paths also failed the caller allowlist. > - This PR retains trusted query construction locations without source maps. > - Operators can distinguish application query sites while exact owners, privacy limits and capture bounds stay intact. ## Linked Issues or Issue Description Refs #17522, #17564. **What happened?** A real map-free bundle with two application query sites produces one callerless census group. This affects unowned queries and queries under the generic `http.request` owner. **Expected behavior** The census should distinguish trusted query construction sites. It must keep unknown calls unknown and preserve specific semantic owners without stack capture. **Steps to reproduce** Run the committed bundle test and pinned comparison helper. The helper bundles the actual capture module and real Drizzle, with a fake postgres-js client. It does not assign a fake path to an isolated mock caller. **Paperclip version or commit** Control: `09162fe893b464dd2ccdafd14bd3ef1611202a86`. The branch is rebased onto `1e3ab690b072bbf6a1c1f1d0454ea49315b2074c`; neither intervening commit changes this owner. ## What Changed - Keep the exact generic HTTP owner and add bounded caller capture only during the active census window. - Exclude capture wrapper frames with a local V8 stack boundary. Keep one static generated construction anchor for trusted root builders. - Keep unscoped bundled driver and relational paths unknown. Their first frame can belong to the ORM. - Cover exact application lines, alternate parameters, prepared builders, transactions, nested savepoints, privacy and inactive windows in a real bundled scaffold. - Add a portable pinned comparison helper and retained measurements. Existing group, page, byte, expiry and output-rate caps remain in force. ## Verification - PASS: 54 focused unit, real-bundle, existing runtime bundler and real-PostgreSQL tests. - PASS: database TypeScript check. - PASS at final head `b270b92821`: full local `pnpm -r typecheck` and `pnpm build`. - The first local `pnpm test:run` was stopped after two chat partitions passed (668 cases), because source changed during the run. This is partial evidence only. The fresh final-head full local command passed all six chat partitions (1,829 cases). Its general phase reported one failure in the unchanged workspace port-conflict test. The exact case then passed alone (1 pass, 178 skipped), and the unchanged full file passed 179/179. The original failure did not reproduce; its cause is unknown. The general phase was stopped after its remaining serial queue was measured at roughly three hours. Cancellation did not print the original assertion diff. The full command is failed/incomplete, not passed. Remaining general files, workspace-project phases and 164 serialized files were not reached. Fresh final-head CI passes all partitions. - Final-head code CI passes. Greptile is 5/5 with its thread resolved. Paperclip Review is delayed by an external review workspace recovery failure; it has not completed on this head. - Five serial alternating rounds per mode, 1,000 queries per sample, 40 retained samples, Node v24.22.2. Generic and unowned attribution rises from 0/1,000 to 1,000/1,000 across two exact application anchors. Specific owners stay stack-free. - HTTP median wall cost is 10.528 → 15.169 µs/query; process CPU is 15.461 → 17.662 µs/query, with noisy overlapping ranges. This is census overhead, not a product speedup. - Review failure retained: the existing runtime bundler test first failed because the public parser accepted arbitrary generated frames. The fix restores default rejection and permits generated frames only from the trusted builder boundary. The unchanged existing test now passes. - Full ranges, raw samples, proof boundaries and reproduction commands: https://github.com/paperclipai/paperclip/blob/b270b92821ae47c456a1b61d78c1593c5348ede7/doc/performance/2026-11-03-db-capture-hosted-callers.md ## Risks - Active generic HTTP census now captures stacks. More groups can reach the existing 2,048-group cap sooner. - Anchors identify construction sites in one exact generated build. They do not identify later async consumers, TypeScript lines or CPU owners. - Unscoped driver calls can remain unknown. This is intentional. - The no-network local fixture excludes deployed Sentry/OTel stack formatting and traffic effects. It does not establish exact hosted overhead. - Wrappers remain installed after expiry. Disabled, future and expired windows do not capture stacks. No schema, authorization, provider, or query behavior changes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code execution and repository tools. The host does not expose the context-window size. Independent review used the same model family. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:04:42 -06:00
DottaandPaperclip	bf7f77d8cd	fix: avoid response recovery self-contention (#18715) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server must present accepted agent responses as comments under current chat authority. > - The recovery worker can check four tasks at the same time. > - Different tasks can share a chat endpoint or external principal. Their authorization transactions then compete for endpoint or identity locks. > - A failed transaction leaves the accepted response pending and repeats its source queries on a later retry. > - This pull request groups checks that share a task, endpoint or principal before dispatch. It keeps every transactional authority check and lock. > - The benefit is less failed recovery work and prompt, exactly-once presentation of valid accepted responses. ## Linked Issues or Issue Description Refs #17714 for a separate publication-commit lock-order fix. Refs #18686 for separate run-list JSON work. Neither changes this recovery admission path. **What happened?** A response-presentation worker reported `could not obtain lock on row in relation "chat_endpoints"`. Four real PostgreSQL recovery fixtures reproduced three such failures in one pass with no external ingress. The worker grouped by task, while authorization locks shared endpoints and principals. Review also exposed two valid Slack endpoints in one workspace that share one principal and compete for its try-advisory lock. **Expected behavior** Recovery checks that share an endpoint or principal should retain task order without competing with one another. Unrelated endpoints should make progress. The server must still reject stale authority and retry legitimate external contention. **Steps to reproduce** Run the shared-endpoint regression or the portable comparison helper linked below. The fixture creates four accepted historical responses for distinct tasks and principals on one actual endpoint. ## What Changed - Read optional endpoint and principal scheduling hints through a company-scoped join to the canonical wake action. The join does not change eligibility or grant authority. - Extend existing bounded check groups to merge intersecting resource keys. Preserve task order, transitive conflicts, independent-group progress, shutdown, drain and pending scopes. - Add real PostgreSQL controls for shared endpoints, a principal shared across two endpoints, an observed blocked transaction with independent endpoint progress, external locks, exact selected output, and endpoint revocation despite a forged hint. - Add a portable helper that materializes the pinned baseline with `git show`, runs the existing real fixture, retains results, and restores temporary files. ## Verification - Final principal-extension head `57047441b28765ed1cf49b8924efc69f5c03dcb4`: 34 focused scheduler/chat controls passed on Node 24.22.2 / pnpm 9.15.4. Independent exact-head source review is green. Full local `pnpm -r typecheck` and `pnpm build` pass on this head. Fresh Greptile and Paperclip reviews are both 5/5 with no findings. All 59 final-head contexts are complete: 55 success, 3 skipped, and the neutral Paperclip review at 5/5. There are no failures or pending checks. - Earlier endpoint-only head `eea437e7edbe7eb41eb880d4d8fd6d99a9c247da`: full local typecheck/build and all remote CI passed. The redundant local full suite was interrupted for this review fix after 956 passing chat cases; it is not a full local pass. - Final shared-principal counts-only comparison completed the same two exact responses: product calls 165 → 147, rows 114 → 100, outer contention failures 1 → 0. SQL query errors stayed zero; advisory-lock refusal is not a failed SQL query. - The earlier endpoint-only study completed five alternating pairs / ten measured runs: median calls 443 → 293, rows 314 → 200, query errors 6 → 0 for four exact responses. Its instrumented shared-host CPU data predates the principal extension and is not a final-head or hosted CPU claim. Retry clocks were accelerated; wall time is not hosted latency. - [Raw samples, final principal work counts, portable commands and failure accounting](https://github.com/paperclipai/paperclip/blob/57047441b28765ed1cf49b8924efc69f5c03dcb4/doc/performance/2026-11-03-response-presentation-contention.md). - Initial server4 CI reported three provider calls instead of two in the unchanged Anthropic identity fixture. The exact local case passed; the full 88-case file passed with observations showing the two expected identity-route calls. The third CI call's origin remains unknown. One bounded rerun of the failed job and its dependent CI aggregate passed. No assertion or source change was made for that failure. - The causal benchmark baseline worker/query is `82c033ed127d07862e6765aa4b69a3cf339bde81`. The PR base is `36df8933f0`. They are not claims about the currently serving revision. ## Risks - Connected resource groups can reduce parallelism when tasks link several endpoints. This preserves the existing task ordering contract. - Missing or stale endpoint/principal hints retain current transactional checks. Other workers and ingress can still cause legitimate NOWAIT retries. - There is no schema, API, permission or provider-transport change. Endpoint revocation and company isolation remain enforced by their existing owners. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), with tool use and code execution, produced and reviewed this change. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-03 06:04:25 -06:00
Dotta	f8b35b3318	fix(ui): retain Slack conversation handoff after setup (#18722) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Slack setup connects a person with an agent in Slack. > - Welcome delivery can complete setup before the person chooses Open Slack. > - Success and saved settings then offer no link to the conversation. > - This pull request retains Open Slack in both views. > - The benefit is that people can use the connected bot without finding its URL elsewhere. ## Linked Issues or Issue Description Refs: #18707. This is the next problem found during its deployed user walkthrough. **What happened?** Managed setup completed after welcome delivery. The Open Slack link disappeared. Success offered Done, and saved settings offered only Slack app configuration. The bot could answer, but the handoff to its conversation was lost. **Expected behavior** A person can open the connected bot from Success and from the saved connection. **Steps to reproduce** 1. Complete managed Slack setup with welcome delivery. 2. Wait for automatic Success before choosing Open Slack. 3. Choose Done and open the saved connection. No conversation link is available. ## What Changed - Retain Open Slack on Success while keeping the centered Done action. - Add Open Slack to completed Slack connection pages. Keep app configuration separate. - Use a saved DM only when its linked owner matches the current signed-in account. Other managers and unknown identities use the provider-link or Slack fallback. - Update the existing component regressions, three Storybook cases, setup guidance and feature recipe. The Slack UI owns the shared URL helper. This change adds no API or permission contract and leaves unfinished setup on its existing path. ## Verification - Focused Slack component and prompt suites: 40 passed. All related chat UI suites: 269 passed across 19 files. The settings regression switches from the installer to a second user and then to a signed-out state. - Repository typecheck and production build: passed. - Token gates, map structure and Storybook catalog checks: passed. The map/catalog Node suite passed all 30 tests; these are structural checks. - Storybook managed and own-app Success play assertions passed. The managed mobile view retained readable Open Slack and Done controls after scrolling. Saved settings play assertions passed after a fresh page load. Fixtures simulate Slack; no live provider actions were sent from Storybook. - The first local typecheck rejected an unsupported Storybook assertion option. It passed after using an exact-name regular expression. The settings story first hit a stale Vite dynamic-import cache; a normal reload fetched the current module and all settings play assertions passed. Restarting the owned review server later emitted Storybook manager state timeouts; all three production-page play cases passed again on the corrected head. - The preceding deployed fix produced real Slack DM and thread replies. This follow-up still needs deployed acceptance through its visible Open Slack link. - Greptile first found that a second manager could receive the installer's DM link. The correction checks the linked owner against the resolved account and falls back while identity is unknown. The two threads have replies and are resolved. Greptile reviewed corrected head 94a6946719 at 5/5 with no new findings. The optional Paperclip agent check ended neutral without a complete assessment; one fresh review was requested. No fresh secondary assessment was available at merge. - [All required CI suites passed](https://github.com/paperclipai/paperclip/actions/runs/41571276944) on corrected head 94a6946719. Full local Vitest was not repeated for this UI-only follow-up. ## Risks - Low risk. This adds navigation to existing Slack destinations. - Old connections without a known DM use their provider link or Slack. - No schema, credential, scope, identity-link or transport changes. The UI consumes the existing shared account identity query. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository editing, browser use and command execution. The runtime does not expose a more precise model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-02 21:33:59 -06:00
DottaandPaperclip	36df8933f0	fix(slack): resolve manager grants during bot setup (#18707) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Slack chat connections let people start and continue work with an agent. > - Managed setup reuses a personal Slack workspace authorization to create and install the bot. > - The provisioning route passed that Slack grant ID to the generic tool-grant lookup. > - A valid workspace therefore failed with a 404 before provisioning started. > - This pull request selects the existing Slack grant resolver for provisioning and validates the request-body ID. > - The benefit is that saved Slack setup can continue while company and personal ownership checks remain enforced. ## Linked Issues or Issue Description Refs: #15679, which introduced managed Slack setup. **What happened?** Continuing a saved managed Slack draft returned `Connection management resource not found`. The workspace list and endpoint read both succeeded. The provisioning POST returned 404. **Expected behavior** Continue should use the selected personal Slack manager grant and create or resume the bot once. **Steps to reproduce** 1. Authorize a Slack workspace for managed bot setup. 2. Create or resume a managed Slack chat connection. 3. Select the saved workspace and choose Continue. ## What Changed - Select the Slack manager-grant resolver for the managed provisioning route. - Accept a validated grant ID from the route parameter or request body. Enforce the current user's ownership for both board and agent requests. - Run the existing duplicate-install regression through the HTTP route. Add HTTP rejection cases for missing, malformed, foreign-company and other-user grants. - Document the grant boundary and update the matching feature and agent-access maps. The shared authorization guard owns resource resolution and company checks. Its existing Slack resolver previously handled only the revocation route parameter. This change extends that resolver to the provisioning body's grant ID. Generic tool-grant resolution and deployments without managed Slack keep their existing path. ## Verification - Before the fix, the HTTP installation regression failed with the same 404 as the browser walkthrough. - `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'managed Slack'` — 31 passed. Uses real PostgreSQL, production routes and provider fixtures. The board actor is a fixture; this does not qualify live Slack or agent authentication. - `node scripts/check-maps.mjs` and `git diff --check` — passed. - `pnpm -r typecheck` and `pnpm build` — passed. Full local Vitest is running. [All CI gates passed](https://github.com/paperclipai/paperclip/actions/runs/41561466277) for commit c04b426e60. Greptile and Paperclip reviews are 5/5 with no required changes. Deployed Slack acceptance remains pending. ## Risks - Low risk. The guard must select the correct grant registry. Tests cover valid ownership, duplicate requests and rejected IDs. - Board requests now receive the ownership denial in the guard. The Slack service already required the same owner. - No schema, credential format, provider scope or runtime transport changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository editing, browser use and command execution. The runtime does not expose a more precise model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 19:55:29 -06:00
Nicky LeachandPaperclip	6d72243d8a	fix(runtime): recover task creation after a lost acquire reply (#18692) ... ## Thinking Path > - Paperclip manages agent execution and the resources that execution owns. > - A retained environment task can need a replacement before the next turn. > - Core records the creation attempt before it asks the provider to acquire it. > - A lost reply can leave that journal without the resource ID needed for cleanup. > - This change lets an opted-in provider recover the identity from that recorded attempt. > - Core still requires the existing exact-task termination proof before it releases ownership. ## Linked Issues or Issue Description Refs #18405 and #18658. This is the separate lost-reply follow-up from their recovery audit. **Draft status:** Both prior review findings are fixed in `a66520c8d4e06e0eec6ccb5446190ace0ca3003f`. Fresh standard Greptile is 5/5 with no unresolved threads. Current-head CI is green (53 successful checks/statuses including Snyk, three expected skips). Apex completed 5/5 on this exact head at 2026-11-03T00:54:59Z (request 6592407130), with no unresolved threads. The review cycle is complete; the PR remains draft for human merge review. **What happened?** If a replacement task's acquire reply is never received, the creation journal can reach manual reconciliation without a resource ID. Startup cleanup cannot complete, and ordinary Stop cannot release the run owner. **Expected behavior** When the selected provider supports creation inspection, Core should retain enough binding information to inspect the recorded attempt and continue verified cleanup. An inspected identity must not count as termination proof. **Steps to reproduce** 1. Use a retained task whose provider supports creation inspection. 2. Lose the replacement acquire reply before Core records its resource ID. 3. Restart cleanup and inspect the existing creation journal. 4. Confirm that the exact task is stopped and its termination receipt is saved before cancellation releases the owner. **Paperclip version or commit** Current head `a66520c8d4e06e0eec6ccb5446190ace0ca3003f`, based on master `3a7bb4dac62dbd7a34a55deb665c55df310bdf64`. Tested with isolated PostgreSQL fixtures. **Deployment mode** Local source tests with controlled provider responses. No deployment or live model request. ## What Changed - Save the selected inspection capability and prior opaque binding in the existing creation journal. Acquire through that same worker so a worker replacement cannot change the recorded contract. - Make failed opted-in creations due through the existing cleanup notification and saved retry deadline. - Add optional SDK observation field `creationLease`. Use it only to recover the requested creation's identity and binding, then follow the existing Stop, status, completion and receipt checks. - Keep old entries and providers without the recorded opt-in under their existing reconciliation policy. - Add actual PostgreSQL recovery, cancellation and worker-replacement regressions. Document the provider contract. ## Verification - Current correction: 53 distinct targeted cases pass (20 real-worker IPC, 12 task-recovery PostgreSQL and 21 generic-sandbox cases), with server typechecking. The IPC selection was rerun after the final late-observation ordering refinement; the other tested sources were unchanged. - An assembled live-child IPC and PostgreSQL case proves cleanup and cancellation release the owner despite the original worker never replying and remaining alive. Accepted host work, unrelated acquisitions and late observations remain accounted for. Removing the recovery handoff and restoring the old drain snapshot produce distinct assertion failures. - The unsupported-host control skips all 12 PostgreSQL cases without setup; removing the guard fails the controlled setup. These tests do not constitute live VM/provider qualification. - Initial head `5db6882` passed an exact reference-tree comparison against master plus the reviewed follow-up patch, 97 targeted cases and server typechecking. Those checks predate the final inspection correction. - Previous head `59f87fa` passed 32 targeted cases (11 task-recovery and 21 generic-sandbox cases) and server typechecking. Both `pending` and `needs_reconciliation` observations without `creationLease` preserve ownership and get a future retry deadline. Restoring the previous guard makes both new cases fail. - The reproduced failure was a wrong generic cleanup shape that could resolve its journal while leaving the owner held—not a demonstrated false ownership release. The correction reuses the existing missing-identity guard before the generic decoder and leaves generic sandbox behavior unchanged. - Earlier affected selection: 72 cases passed on the original follow-up base. This overlaps the final selection and is not an additional count of unique final-head tests. - Two isolated negative controls restore the missing task inspection and redirected-worker behaviors. Both fail at the expected recovery and dispatch assertions without changing production source files. - The PostgreSQL cases verify one cleanup lease, both late-result orders, recovery after the old journal deadline, ordinary legacy behavior, retained ownership on missing or mismatched identity, and cancellation acknowledgement after exact termination proof. - The tests use actual database and run-control services with controlled worker callbacks. They do not prove live IPC or production recovery timing. - A compatible provider is needed for the new recovery path. No normal successful-startup database or network call is added. No ordinary TTFT improvement is claimed; the repair removes an indefinite ownership hold for supported lost replies. No full local CI suite was run; GitHub runs the full regression gates. ## Risks The provider must recover the identity for the recorded creation operation and preserve its selected binding. `creationLease` does not authorize a new turn or prove that a process stopped. Older journal entries do not infer a new identity after an upgrade. Unavailable original machine bindings or credentials retain the existing uncertainty behavior; this change does not repair machine or credential loss. No schema migration, deployment or ROM qualification is included. ## Model Used OpenAI Codex, GPT-6 family. Exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted tests and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 17:45:07 -08:00
DottaandPaperclip	40ea7a9b89	fix(heartbeat): wait for inherited remote workspace writers (#18691) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - A child task can inherit its parent's remote workspace. > - The remote workspace permits one writer until that writer saves its files. > - Reservation reports an active writer as a generic setup failure. > - The existing shared workspace wait does not cover inherited isolated workspaces or a race at reservation. > - This pull request sends verified active writer contention through the existing scheduled retry path. > - The child keeps a live execution path while unfinished files remain protected. ## Linked Issues or Issue Description **What happened?** A native sandbox task inherited an isolated remote workspace from its running parent. Setup failed with `The workspace still has an unfinished writer or save operation.` The child became blocked before provider execution and had no scheduled continuation after the parent saved. **Expected behavior** The child waits for the active writer through the existing workspace retry path. It keeps its workspace binding. A stale writer or a stopped writer with unfinished saves still requires reconciliation. **Steps to reproduce** 1. Start a native sandbox task with remote workspace storage. 2. Create a child that inherits its isolated workspace or operator branch while the parent is still writing. 3. Start the child. On the old code, atomic reservation raises a generic setup failure. **Paperclip version or commit** Observed on `410b0e27015495ab8951768b5b0b94727de33785`. Regression reproduced on source based on `554a770c6f8e48e327e19f5208b575a5ec5bce30`. **Deployment mode** Authenticated managed server, external PostgreSQL, native Codex with a Daytona sandbox. The failed attempt stopped before runtime selection or provider startup. Related: #18611 permits inherited remote bindings. #16371 adds waits for reused shared execution workspaces. This repair handles active writer contention at atomic remote reservation. #17156 proposes a separate change to task workspace inheritance. ## What Changed - Return a typed conflict with the exact active writer from the remote reservation owner. Keep company, binding, access, and writer checks in that transaction. - Translate a recent running writer into the existing heartbeat workspace wait. Keep stale, unknown, stopped, dirty, and uncertain save fences. The writer task lookup uses its primary key. - Preserve the admitted assignee role across preparation so reassignment cannot turn an old owner into a guest retry. - Add PostgreSQL regressions for isolated workspaces, operator branches, native writer identity, unchanged saved authority, admission after save, durable retry, Stop, and reassignment before contention reaches the wait handler. - Update the development guide and workspace verification recipes. ## Verification - Red: both inherited isolated/operator preparation cases fail on the old code. They receive `HttpError(remote_workspace_writer_pending)` instead of a workspace wait. No provider or allocation starts. - Green: those same cases pass after the repair. The assembled public heartbeat cases also pass: one scheduled retry owns the task lock, no sandbox lease is created, the original writer and saved authority remain unchanged, and Stop prevents retry promotion. - Ownership race red/green: changing the assignee during preparation produced `wasIssueAssignee: false` before the review fix. The same regression now preserves its original role. A real transaction fault boundary also reassigns the task after reservation rejects it and before the wait handler runs; promotion cancels that retry with `issue_reassigned`. - Test isolation: an initial combined placement cached the synthetic S3 run-log backend and failed twelve later preparation cases. The assembled native tests now have their own process. The ordered reproduction and all four complete affected suites pass after that change: 171 tests, zero skipped. - Four additional saved-file lifecycle cases pass against disposable Docker PostgreSQL 17 with independent filesystem execution and local object storage. They restore inherited shared, isolated, and operator-branch files on a new allocation and retain the second-writer fence. The other 42 cases in that file were outside the selection. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` pass on the final source. - The earlier `pnpm test:run` passed all six chat shards: 1,828 tests. Its remaining general-server phase was interrupted after final-head CI passed. That local command began before the review fix and is not a complete local full-suite pass. - Final head `096235d8fc80033f92d8de1fcfa9835795ba0456`: 54 successful checks/statuses, three skipped checks, and two neutral checks. None is pending or failed. All build, typecheck, general/serialized tests, Runner checks, and eight browser E2E shards pass in [CI](https://github.com/paperclipai/paperclip/actions/runs/41544310900). The PR is mergeable with no conflicts. - Greptile is 5/5 on that exact head. Its ownership finding is resolved and no review thread remains open. Paperclip Review also reports 5/5 with no findings; its check conclusion is neutral. Its earlier status-conflict recovery completed through the ordinary review request path. - Local environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, isolated native PostgreSQL and the disposable PostgreSQL 17 lifecycle fixture. The acquisition boundary is controlled in the focused preparation cases. No live Codex/Daytona replay is claimed: the local environment and test configuration lack the required test credentials. - No deployment, merge, or original task recovery is claimed. ## Risks - Incorrect contention classification could retry a stopped or unverified writer. Only a running writer with a same-company task and recent activity enters the wait path. Stale and terminal dirty cases retain the conflict. - Retries must not start another executor or release unsaved files. Reservation remains atomic. The tests check unchanged writer authority, no allocation before release, one durable retry, retained binding, Stop, and reassignment before the conflict is caught. - No API schema, database migration, provider configuration, model, credential, trust, or approval changes. ## Model Used - OpenAI `gpt-6.1-sol` through Codex. The context window size is not exposed in this session. Used reasoning, tool use, code execution, and browser diagnostics. No sub-agents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted and chat results are recorded above; the full local command was interrupted - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 19:26:59 -06:00
DottaandPaperclip	5955dd4eb4	fix(files): preserve saved reads during allocation cleanup (#18701) ... ## Thinking Path > - Paperclip saves remote agent work in committed workspace snapshots. > - Files can read that save after the writer finishes. > - Compute cleanup can clear the recorded location while a saved preview loads. > - The current fence treats that physical cleanup as a change to the saved bytes. > - Permit only that one-way location removal for an already verified saved copy. > - Keep all logical-authority checks and live-read fences. ## Linked Issues or Issue Description Refs: #18677. This fixes the remaining in-flight cleanup race after saved reads began preferring the current committed version. A Files preview can return `422 workspace_changed` when allocation cleanup clears the location during archive loading. The epoch, committed snapshot, source, clean state, and absent writer can all remain unchanged. The extracted copy remains valid. An original QA denial recorded this error eight milliseconds after location cleanup. Its later successful read does not regrade the failed attempt. The original audit does not reconstruct the full earlier authority record. A controlled production-composed regression isolates the same cleanup transition. ## What Changed - Allow the recorded location to become null during a verified saved read while every other parsed authority field stays identical. - Remember observed removal so a reintroduced or foreign location still rejects the read. - Keep live reads, dirty state, active writers, changed epochs, snapshots, and sources fenced. - Add a real `createApp` content-route regression with PostgreSQL, encrypted archive storage, and private-copy cleanup. - Add negative controls for logical changes, foreign and reintroduced locations, and live location removal. - Update developer guidance and the workspace feature/access maps. ## Verification - Red: the production `createApp` route returned exactly `422 workspace_changed` when normal archive loading overlapped location removal. - Green: all 108 Files route and authorization tests passed. The preview returns the saved bytes, and its private extraction is removed. - All 46 opt-in workspace lifecycle tests passed on one owned local PostgreSQL database with local filesystem transport. The database and server were cleaned up. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` passed. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41550908692) completed successfully on `45517b474b6866de2de6c618b7ff2193c93038ac` (attempt 2). All PR gates are clean: 55 successes, 3 skips, and 1 neutral review result. The full suite ran in CI; it was not duplicated as a long local full-suite invocation. - [Original CI failure](https://github.com/paperclipai/paperclip/actions/runs/41550908692/job/125638034035) is retained: an unchanged mobile frame-sampling check collected 5 frames where more than 5 are required; all sampled geometry was identical. The exact unchanged local Chrome case passed, then one failed-job-only retry passed on the same source. No assertion or time budget changed. - Fresh exact-head Greptile review is 5/5 with no findings, inline comments, or unresolved threads. The separate Paperclip static review is also 5/5 with no actionable findings. - No deployed acceptance is claimed by these local tests. The original failed QA attempt remains failed. ## Risks This permits one specific transition for an immutable saved copy: the original recorded location can disappear. The company/workspace query, epoch, snapshot, source, dirty state, writer, and location-reintroduction checks remain. A live read still rejects physical location changes. The change does not release compute, change snapshot admission, or alter extraction limits. No schema or API-shape change is required. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 18:33:49 -06:00
DottaandPaperclip	16643a6e03	fix(runner): recover pending Stop cleanup after controller restart (#18682) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description **What happened?** A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. **Expected behavior** Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. **Steps to reproduce** 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 18:22:40 -06:00
DottaandPaperclip	4814d1ab69	fix(ui): show composer images as rounded thumbnails (#18689) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use the new-task and thread composers to send text and files. > - Images currently enter the text editor or appear with other files. > - Large images interrupt text entry and make the two composers inconsistent. > - This pull request puts image previews above the text in both composers. > - People can check and remove images while they continue to edit the text. ## Linked Issues or Issue Description Refs: #13206. Related shared-composer work: #15091. **What existing behavior does this improve?** Image selection, paste, and drop in the new-task and thread composers. **Current behavior** Thread images enter the editor at the text cursor. New-task images can appear inline or with file attachments. Images do not have one preview size. **Proposed behavior** Show images above the text as 80 by 80 previews with rounded corners. Use the existing spacing and radius tokens. Keep file attachments in their current row. Add image references when the thread message is sent. **Reason and benefit** People can review images without moving or deleting message text. One shared image row keeps both composers consistent. **Breaking changes** New image uploads stay outside the editor until send. Old inline-image drafts remain readable. No server API or database changes are required. ## What Changed - Add `ComposerImageAttachments` for local previews, removal, upload status, and retry. - Let `TaskChatComposer` own the image row and all image input paths. Keep attachment IDs in the existing send flow. - Let `NewIssueDialog` stage images and upload retained files after task creation. Save file bytes in browser IndexedDB by account and company. Restore them after reload. Clear them after creation. Show storage errors. Merge saved file changes in one IndexedDB transaction. Delete only the files removed by that draft. This keeps files saved by other open tabs. - Keep rich editing for old image Markdown. Use the same MIME and filename checks for image previews in both composers. - Save optional MIME data with thread draft attachments. Existing draft records remain valid. - Add desktop and mobile Storybook examples. Update the task feature map and component tests. ## Verification - `pnpm exec playwright test --config tests/e2e/playwright.config.ts tests/e2e/board-attachment-receipts.spec.ts`: all 14 Chromium tests pass against the real local Board, API, storage, and disposable database. The suite proves image recovery in two open tabs after saves and task creation, new-task image recovery after reload, byte-for-byte upload content, cleanup after creation, rich editing of old image drafts, upload receipt recovery, and removal during pending uploads. - The first local browser attempt could not start because Chromium was absent. Chromium was installed. A later run passed 12 tests but the new legacy-draft fixture restored an empty body. That fixture wrote storage while an empty composer was mounted. It now seeds before React startup and uses a real uploaded image. The complete 13-test rerun passes. Early component runs also exposed the new restore timing. Submit and file controls now wait for restoration. The test helper now waits for React effects. - 316 focused tests pass across `TaskChatComposer`, `NewIssueDialog`, `MarkdownEditor`, `composer-draft`, and `useStagedIssueFiles`. The tests cover explicit file removal, file recovery, account and company changes, storage failures, MIME-empty images, upload retries, and removal. - The two-tab browser test failed on the previous commit: the second save erased the first image. The fix merges files in one read/write transaction and records explicit removals. The first fixed run reached the final create step but had no text request after the other tab cleared the shared text draft. The test now enters a fresh request before that step. The full 14-test rerun passes. - `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/storybook-catalog.mjs`, and `git diff --check` pass. - `pnpm --filter @paperclipai/ui typecheck` and `pnpm --filter @paperclipai/ui build` pass after building the local plugin SDK dependency. Repository typecheck and build stop because this workspace has no `cargo`. Full CI must pass before merge. The first CI run passed typecheck, build, and all unit-test shards. Its browser failure expected the old inline upload label. The assertion now uses the thumbnail upload status. - The prior browser review used the production components with simulated upload and submit calls. Desktop, mobile, and dark-theme captures were accepted before this PR. Browser checks confirmed equal-size rounded previews above the editor, no inline image nodes, no horizontal overflow, and removal that preserves text. These checks do not prove live backend persistence. - Earlier repository typecheck and build attempts also stopped without `cargo`. An earlier full test run was killed with a recorded out-of-memory event during a server integration shard. It had no full-suite verdict. One document-recovery test timed out during that concurrent run. Its isolated rerun and later complete focused runs passed; the initial timeout cause was not established. - An initial catalog command in this workspace ran before dependency installation finished. It could not load Storybook. The command passed after installation. The first UI typecheck could not find plugin SDK declarations while that package was still building. The UI typecheck passed after `pnpm --filter @paperclipai/plugin-sdk build`. To inspect the layout, open `composer-new-task--images`, `composer-new-task--mobile-images`, `composer-add-menu--images`, and `composer-add-menu--mobile-images` in Storybook. Select images with different aspect ratios. Check removal, mixed files, failed upload and retry, and draft restoration. - Greptile reports 5/5 for commit `9b9e93beeed80371f667ad2e1a9573e82332c05c`. There are no open review threads. [CI](https://github.com/paperclipai/paperclip/actions/runs/41544180330) passed on this commit. All 58 checks finished: 54 passed, one review is neutral, and three optional checks were skipped. The legacy Snyk status passed. Paperclip also reports 5/5. No review threads are open. ## Risks - Open tabs share the account and company draft. Atomic file writes keep changes from other tabs. A reload can restore those saved files. - Attachment state now controls when image references enter a thread message. Tests cover pending uploads, failures, retry, removed images, and retained upload IDs. - New-task images use the existing staged-file upload flow after task creation. A failed upload follows that flow's existing error behavior. Browser storage can be unavailable or full. The composer keeps the files in memory and shows a recovery message. - Legacy inline-image drafts retain their original text. No migration is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and shell execution. The exact runtime snapshot ID and context-window size are not exposed in this session. The earlier implementation's model snapshot is not recorded in the handoff. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My remote branch name describes the change and contains no internal Paperclip ticket ID - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 23:35:32 +00:00
Nicky Leach	3a7bb4dac6	perf(runner): activate shared retained startup context (#18405)	2026-11-02 15:12:57 -08:00
DottaandPaperclip	58308a3e63	perf(apps): scope sidebar review counts to one connection (#18688) ... ## Thinking Path > - Paperclip helps people manage AI agents and their governed app access. > - The Apps sidebar displays a Review badge for one connection. > - It polls the full company attention service every 30 seconds to get two counts. > - That service also loads every profile, rule and active catalog entry. > - This pull request adds an authorized connection read for only those counts. > - The same badge now avoids unrelated profile evaluation and database result decoding. ## Linked Issues or Issue Description **Current behavior** The contextual sidebar polls company-wide attention. It needs only one connection's signed pending action count and whether any catalog entries are quarantined. **Proposed behavior** Both sidebar variants poll a connection-scoped count endpoint with existing connection read authority. The full attention API stays unchanged. **Reason and benefit** The retained 200-profile fixture performs seven service queries and returns 80,115 rows for this badge. The new reader performs one query and returns one row. This reduces local service CPU and avoids unnecessary decoded profile/rule data. Related merged evaluator fixes are already in the baseline: Refs #18673, Refs #18681. A search found no existing PR for this scoped count reader. ROADMAP.md contains no duplicate planned work. ## What Changed - Add the narrow shared DTO, typed client, guarded REST/OpenAPI route and catalog-owned reader. - Count independent aggregates with both company boundaries and exact signed-argument semantics. Keep archived counts at zero. - Update both contextual sidebar variants. Keep the exact badge, 30 second interval and existing attention-prefix invalidation. - Add PostgreSQL differential tests, authenticated HTTP tests, sidebar tests, a repeatable benchmark and access documentation. ## Verification - Baseline: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`, including both merged evaluator fixes. Service treatment: `0827f7d0b3`; route/fixture refinement: `230c42ee15`; final local existence gate: `390a218254`. - Passed: focused PostgreSQL differential test; assembled HTTP test with board session and agent JWT; 14 sidebar component tests; token gates; map checks; diff check. - Fixture: 200 profiles, 76,000 rules, 1,900 active tools, 695 removed references, 12 agents, a sibling connection and 2,000 unrelated historical invocations. Complete normalized two-count results are equal on both paths. - Quiet local measurement: seven alternating warm pairs after two warmup pairs, Node 24.22.2, macOS, real disposable PostgreSQL. Legacy/new service CPU median: 213.891/1.680 ms; ranges: 209.329–228.552/0.515–6.137 ms. Wall median: 156.867/1.248 ms; ranges: 151.739–162.289/1.115–1.386 ms. Queries: 7/1. Returned rows: 80,115/1. - Run `cd server && node --expose-gc --import ./node_modules/tsx/dist/loader.mjs scripts/benchmark-connection-review-counts.ts`. See `doc/connection-review-counts.md` for scope and failure history. Byte-volume accounting runs outside timing; no HTTP payload reduction is claimed. - Initial raw scalar SQL failed with ambiguous columns. Typed subqueries fixed it and passed real PostgreSQL equality. Initial fixture agents lacked permissions and correctly received 403. The final fixture grants explicit authority and tests revocation. An initial benchmark included byte serialization in timing; the reported run excludes it. - Passed required local `pnpm -r typecheck` and `pnpm build`. The initial sandbox typecheck could not open the tsx IPC socket; the escalated rerun passed. Full monolithic `pnpm test:run` was not run locally: repeating a 15–25 minute suite serially across both performance PRs adds no distinct evidence. Broad suite acceptance uses final-head CI, which runs the same general/serialized test partitions and the native Runner lane. Final-head CI is in progress. - Passed built-static Chromium journey on an owned local-trusted server and disposable PostgreSQL: Review 3 appears, only scoped counts are requested by this sidebar, resolving requests/clearing quarantines then reloading removes the badge, and archived counts are zero. The first ad hoc launcher omitted required decision-service options and failed at startup; using the normal no-op wake callback fixed that harness. - Greptile identified an invalid resolved-request fixture label. Corrected `denied` to the declared `rejected`, then reran the real PostgreSQL differential test. The count SQL and non-pending exclusion are unchanged. - The first final-head CI failed the static cross-tenant existence-oracle guard because the new explicit resource company assertion had no nearby `hasCompanyAccess` gate. Existing management admission already rejects foreign resources with 404. The route now also follows the standard local two-step gate. The unchanged guard plus authenticated HTTP negative cases passed (3 tests). No allowlist or test was weakened. - These local service measurements do not prove hosted performance. No deployment is included. ## Risks Low risk. New read admission reuses existing current-authority checks. The old attention API is unchanged. Counts still require database work over relevant pending/catalog rows. No cache, schema migration or telemetry is added. Roll back this PR if the badge differs or scoped reads deny previously permitted users. ## Model Used OpenAI `gpt-6.1-sol` (Sol 6.1), with reasoning, tool use and code execution. The host does not expose the configured context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 17:12:22 -06:00
DottaandPaperclip	554a770c6f	perf(apps): scope connection profile configuration reads (#18687) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps Permissions page reads saved access profiles for one connection. > - The page fetched every company profile, rule, and binding. > - It used only the app profile and eligible task-granted profiles for that connection. > - This pull request adds a saved configuration read owned by the profile service. > - The route keeps current connection and per-profile management checks. > - The benefit is fewer database rows, less Node work, and a smaller response for the same displayed configuration. ## Linked Issues or Issue Description **Subsystem affected** Server profile configuration reads and the Apps Permissions and Review data caller. **Problem or motivation** One connection page loaded and serialized unrelated company profiles. In the local fixture, this produced a 32.35 MB response with 76,001 rule rows. **Proposed solution** Select the displayed connection's app profile and exact eligible connection-intent profiles before loading their entries and bindings. Return an explicit configuration DTO with complete selected arrays. Keep summary and pending-review work on the existing full API. **Alternatives considered** Filtering the full response in the browser retains the database and Node cost. Truncating the full API would break other clients. Filtering entries to the connection would lose selectors and bindings needed by saves. **Roadmap alignment** This improves the existing governed Apps capability. It does not add a new access policy. Related merged work: #18673 and #18681. Related harness cleanup: #18683. Public PR searches found no duplicate connection configuration projection. ## What Changed - Add `ToolProfileConfiguration` and `GET /api/tool-connections/{connectionId}/profile-configuration`. - Select `app:<connectionId>` at any status and exact active, correctly bound `connection_intent` profiles. Run the existing per-profile visibility filter before configuration queries. - Preserve complete ordered entries and bindings. Keep all save behavior, empty-binding install fallback, and the existing full profile API. - Switch AppDetail and AgentConnectionAccess to the configuration DTO. Use a distinct cache key below the existing profiles prefix so mutation and reconnect invalidations still refresh it. - Add real PostgreSQL differential and authenticated HTTP tests, UI caller coverage, a reproducible benchmark, and access-map documentation. ## Verification Baseline: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`, after both #18673 and #18681. Final head: `b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`. Current implementation and reproduction details are in [the performance note](https://github.com/paperclipai/paperclip/blob/b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22/doc/performance/2026-11-02-connection-profile-configuration.md). - AppDetail: 79 tests passed. Existing access/action saves, install fallback, grant removal, errors, and refresh cases use the new caller. - Real PostgreSQL differential: passed. Full ordered selected configuration equals the old full-list selection. Cases cover extra bindings, sibling selectors, wrong source/key/status/binding, null and non-string agent metadata, fresh mutations, empty app bindings, visibility filtering, and company scope. - Production HTTP authentication: API key and run JWT cases passed. Hidden personal accounts omit spanning profiles. Private connection reads fail. Restored ownership restores visibility. Revoked management grants deny fresh calls. Final UID canonicalization and foreign-company 404 assertions passed on the current head. - Map structural check, token gate, and `git diff --check`: passed. - Required local `pnpm -r typecheck` and `pnpm build`: passed on the current head. `pnpm test:run` was started once. Its first chat shard passed (337 tests, 151.87 seconds); the second shard was intentionally interrupted after the complete final-head CI suite passed. This partial local attempt is not a full local test pass. The same stable-runner general and serialized partitions and Runner lanes passed in [final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41538805231), including aggregate `ci / verify`. No local assertion failures were observed before interruption. - Final-head focused acceptance: four files, 85 tests passed. Full token gates, maps, and module boundaries passed. - Actual browser acceptance passed on the current checkout with isolated real PostgreSQL and a local trusted board. An action saved as Off persisted after reload. Both app agent bindings stayed intact. Removing an agent-only task grant persisted after reload. Requests used only the narrow profile endpoint. No browser exceptions occurred. No responses were mocked. No provider calls or hosted changes were made. The helper stopped the owned runtime and database after the check. - Final exact-head checks: 54 success, one clean neutral Paperclip Review, three skipped, zero pending or failures. Legacy Snyk status: success. No inline review comments. - Greptile reviewed exact head `b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`: 5/5, no findings, completed successful check. Five alternating unprofiled pairs ran with real PostgreSQL on macOS/arm64, Node v24.22.2, and forced GC. The fixture has 210 profiles (200 heavy profiles plus ten eligibility cases), 76,001 rules, 1,900 active tools, 695 removed tools, and 12 agents. The selected configuration has two profiles and 1,391 rules. Each pair asserts complete ordered selected DTO equality. | Measurement | Full list | Configuration read | | --- | --- | --- | | Service wall ms, median [min, max] | 200.927 [193.319, 213.390] | 6.414 [5.508, 6.973] | | Node CPU ms, median [min, max] | 262.751 [246.727, 271.573] | 27.525 [22.576, 29.308] | | Select queries / returned rows | 7 / 78,139 | 3 / 1,397 | | Actual response JSON bytes | 32,353,534 | 592,233 | | Serialization wall ms, median [min, max] | 90.714 [88.715, 99.304] | 1.644 [1.637, 1.745] | | Serialization CPU ms, median [min, max] | 90.644 [88.569, 96.778] | 1.654 [1.637, 1.807] | Reproduce: `node --expose-gc --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-connection-profile-configuration.ts 5`. The benchmark measures the service and actual response JSON separately. It excludes HTTP and authorization work through an identity visibility callback. Node CPU does not include PostgreSQL CPU. No hosted or foreground-under-load improvement is claimed. Preserved failures: offline install lacked a locked tarball; frozen online install passed. Sandbox localhost and tsx IPC restrictions blocked initial DB/typecheck attempts; permitted retries use the same checks. The first HTTP suite needed the unbuilt plugin SDK; its owning build helper fixed setup. Shared typecheck caught a missing barrel export, now fixed. A broad typecheck inherited the host's unpinned nested pnpm executable; the owned stalled process was stopped and restarted with pinned pnpm 9. The complete current-head CI suite and aggregate verify passed. The separate Paperclip agent review published 5/5 with no findings at this head and completed with a clean neutral conclusion. An added foreign-company assertion expected 403; the existing guard correctly returned 404. Only the test expectation changed. The first real browser attempt saved and reloaded the action correctly, but grant deletion returned 422: the stress fixture had a company binding, which makes that profile the company default. The UI showed the existing deletion error. The second browser seed used the production agent-only intent shape and passed removal and reload. This does not change the stress fixture or product deletion gates. Initial external browser script attempts also needed an ESM extension and the correct dependency path. Logs, first-failure screenshots, traces, and result files are retained in the local evidence directory. ## Risks - SQL selection must stay aligned with connection-intent eligibility. Differential fixtures cover the exact current predicate. - A selected connection can still have many rules. This change preserves all selected configuration. - Per-profile authorization still reads current selected resources and can add work. Connection visibility does not bypass that check. - No migration, default-allow, effective-access, policy-review, provider, or deployment behavior changes. The old endpoint remains available. Rollback restores the old AppDetail caller and removes the additive endpoint. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with code execution and repository tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:55:56 -06:00
DottaandPaperclip	00fea0da52	refactor(perf): reuse database test setup for profile benchmark (#18683) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool profiles govern the tools that agents can use. > - Profile listing and app attention shared nested selector scans that consumed the server CPU. > - #18681 and #18673 replaced those scans with indexes for each computation. > - Their performance benchmarks need consistent setup and reproducible per-service measurements. > - This pull request uses the shared helper entry point and retains the exact measurement harness and results. > - Reviewers can inspect both the CPU savings and the limits of the evidence. ## Linked Issues or Issue Description Refs #18681 and #18673. **What existing behavior does this improve?** The consistency of benchmark setup and the reproducibility of evidence for the two merged CPU fixes. **Current behavior** The profile-review benchmark imports the database starter directly. The matching benchmark uses the shared server test-helper entry point. On the current base, that helper re-exports the database starter, which already supports `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`. The previous committed benchmark measures both services together; the per-service measurements and separate HTTP/Inspector runs lacked a committed harness. **Proposed behavior** Use the same test-helper entry point in both benchmarks. Retain the compile, per-service timing, Inspector, and HTTP-probe programs with their exact commands and raw samples. Preserve the existing embedded and admin-server database behavior. **Reason and benefit** A shared entry point keeps benchmark setup consistent. The committed harness lets reviewers reproduce and compare each real owning service before and after the merged fixes. The failure-path checks verify that result-write errors still release the database and probe resources. **Breaking changes** None. This change affects benchmark setup and documentation. ## What Changed - Import the database starter from the repository test helper in `benchmark-profile-review-projection.ts`. - Record separate CPU and elapsed measurements for profile listing and app attention. - Retain the actual compile, per-service measurement, Inspector capture, and separate-process HTTP probe harness. - Always tear down the test database, HTTP server, and probe even when result writing fails. - Retain all timing and responsiveness samples as sanitized JSON. - State the runtime, fixture, source revisions, and limits of the local comparison. ## Verification - 24 focused tests passed across selector matching, pending projection, review-service freshness, and effective assignments. - Four selected real database-backed service/route tests passed for summaries, review changes, auto-allow behavior, and company isolation. The other 494 tests in that file were not selected. - Injected an `EISDIR` result-write failure into both profiling programs on an owned local admin server. Each child exited with the expected error and removed its disposable database. The responsiveness child also terminated its probe/server handles. - The changed benchmark completed with a configured local PostgreSQL admin URL. Its full responses matched the baseline, and query/row counts matched. - The exact documented compile, benchmark, and responsiveness commands completed from the committed harness at `ca458baddc6461fb6c888984246e329e2bb4bdfe`. The JSON records that build revision and the four owner source hashes. - Five alternating samples per service and implementation used Node 24.22.2 with compiled ES modules. No runtime TypeScript loader ran in those measurements. - Median process CPU from the committed harness: profile listing 3,164 ms to 385 ms; app attention 2,466 ms to 221 ms. Both versions returned identical ordered results and made seven queries per service. - Separate Inspector profiles confirmed that the repeated scope and summary scans were removed. Three separate-process HTTP probe trials per service showed less event-loop blocking. These are local synthetic service results, not hosted page-load acceptance. - Initial local setup failed on missing embedded PostgreSQL library aliases and an unbuilt plugin SDK. Restoring local aliases and building the SDK resolved those setup failures. The original failed setup logs were retained; the 28 selected tests then passed. - The merged summary fix CI run [41535304968](https://github.com/paperclipai/paperclip/actions/runs/41535304968) hit two 15-second timeouts in the unchanged attention-retention history cases. The server shard passed unchanged on one failed-jobs-only retry. Original and retry logs are retained. The compared runs used different runners and synthetic merge trees, so the original timeout cause is not established. No timeout or assertion was changed. - Full repository typecheck, test matrix, and build passed in CI on final head `2f484ed4284f3fa632d8e73d9be084827d1c987e`: 54 successful check runs, three expected skips, one non-blocking Paperclip review, and a successful Snyk status. No failures or pending checks remain. Those broad commands were not rerun locally for this benchmark-tooling follow-up; the 28 selected local regressions and the actual profiling/failure-path programs ran locally. - Fresh Greptile is 5/5 on that exact head. Both review P2 threads are resolved. The Paperclip review's current-base documentation note is corrected in this description; its optional tooling suggestions remain outside this bounded change. ## Risks - The test helper creates a unique disposable database on the configured admin server and removes it afterward. The operator must use a database server where those test operations are appropriate. - Synthetic data and shared-host activity affect absolute timings. The measurements do not establish a production latency percentage or capacity limit. - Production behavior is covered by the two linked merged fixes. This follow-up changes only benchmark setup and documentation. ## Model Used OpenAI GPT-6 through Codex, with code execution and repository tools. The exact runtime model revision and context window were not exposed in this session. The original linked performance fixes record their own model details. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:50:58 -06:00
DottaandPaperclip	d0ffa179b4	perf(tools): index profile selectors per resolution (#18673) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Core resolves tool profiles before it prepares runtime MCP assignments. > - A selected profile can contain a rule for each individual tool. > - The resolver and profile summary scan these rules for each catalog row. > - This repeats millions of exact comparisons for a large selected profile. > - This pull request indexes each selector kind once per computation. > - The benefit is less server CPU with the same current access decisions and result contract. ## Linked Issues or Issue Description **What existing behavior does this improve?** Effective tool assignment and profile summary computation for large selected profiles. **Current behavior** Each catalog row scans the profile's includes and excludes. The work grows with catalog size times rule count. The runtime assignment path also computes a profile summary with the same scans. **Proposed behavior** Build five selector sets for each include or exclude group. Match catalog rows with exact set membership. Retain all current database reads and the full result contract. **Reason and benefit** Reduce repeated JavaScript comparisons. On the synthetic PostgreSQL fixture below, median process CPU per resolution falls from 50.068 ms to 10.129 ms. This is independent of [#18371](https://github.com/paperclipai/paperclip/pull/18371), which removes unused administrative results and narrows instruction descriptors. This PR retains those results and does not include that change. ## What Changed - Index application, connection, catalog-entry, tool-name, and risk selectors in the profile owner. - Use indexed matching in effective access resolution and profile summaries. - Add a differential test against the original single-rule matcher. Cover null applications, exact special-character names, unknown selectors, and empty rule groups. - Add a PostgreSQL regression for per-profile exclusion precedence, the union of multiple effective profiles, and current rule removal. - Add a real PostgreSQL resolver benchmark and document reproduction and limits. ## Verification Baseline: `410b0e27015495ab8951768b5b0b94727de33785`. Treatment: `895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`. Node 24.22.2 on macOS arm64. Source modules run with `tsx`; these are local service measurements, not production-build or hosted acceptance. The real owning resolver ran against disposable PostgreSQL with 1,890 active catalog tools, 1,575 individual include/exclude rules, one deny-by-default effective profile, and 12 agents. The installation dependency returns an empty list because the fixture has no installed connections. There are no provider calls. Full returned assignments, profile summaries, rules, bindings, and names are identical to baseline. After two warmups per version, seven alternating samples ran ten resolutions each. Other workers paused CPU measurements and builds. Unrelated host work was not controlled. | Per resolution | Baseline median (range) | Change median (range) | | --- | --- | --- | | Process CPU | 50.068 ms (46.380–58.880) | 10.129 ms (8.089–11.072) | | Elapsed time | 50.362 ms (44.799–106.408) | 8.853 ms (7.719–18.590) | | Queries | 6 | 6 | | Returned database rows | 3,480 | 3,480 | Median process CPU falls 79.8%; median elapsed time falls 82.4%. The long baseline elapsed sample had no matching process CPU increase. Query and row counts do not decrease. Small or broadly allowed profiles can save less. This does not establish whole-run latency or hosted performance. Reproduce with `server/scripts/benchmark-tool-profile-matching.ts`; baseline module preparation is in `doc/mcp-discovery-performance.md`. Passed on the treatment source: ```sh node node_modules/vitest/vitest.mjs run server/src/__tests__/tool-profile-matching.test.ts server/src/__tests__/tool-effective-assignments.test.ts server/src/services/native-runtime/runtime-context.test.ts server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts ``` All 45 tests in four suites passed. `git diff --check` passed. The first PostgreSQL test attempt failed at setup because the sandbox blocked its loopback listener (`EPERM`); the authorized unsandboxed retry passed. No behavior failure was discarded. At measured head `895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`, CI typecheck, build, canary, Runner checks, and all eight browser shards passed. Server general shard 8/12 failed an unchanged background-quietness assertion in `heartbeat-run-dispatch-work.test.ts`: expected 235 select calls, observed 236 after advancing fake timers. Its cause remains unverified; no baseline reproduction was run. That head also failed the module-boundary policy because the benchmark inserted synthetic agents outside the test fixture tree. Follow-up head `e20165b6eb` moves the fixture body byte-for-byte into `server/src/__tests__/helpers/tool-profile-matching-fixture.ts`, following the existing listing benchmark pattern. The production matcher and benchmark measurement loop are unchanged. `node scripts/check-module-boundaries.mjs`, a Node 24/tsx fixture import check, and `git diff --check` pass. The benchmark was not rerun for this extraction. New-head CI and review are pending. Greptile gave the measured head 5/5 with no actionable issues. No deployment or live runtime mutation was performed. ## Risks Exact equality must remain identical to the original matcher, especially for null application selectors and exclusions. The differential and PostgreSQL tests cover these cases. Sets add linear temporary memory in rule count. They live only for the current computation. Current company-scoped state reads, profile precedence, revocation, and runtime assignment digest checks remain in place. Revert this commit to roll back. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with source inspection, code execution, tests, and local PostgreSQL benchmarking. The context window size is not available in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:13:18 -06:00
Nicky LeachandPaperclip	a9b18e1cbe	fix(runner): preserve cleanup ownership across startup failures (#18658) ... ## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 14:13:00 -08:00
DottaandPaperclip	b4315943e8	fix(heartbeat): keep remote retry workspace strategy stable (#18660) ... ## Thinking Path > - Paperclip preserves agent work across separate execution services. > - A remote retry can create a new isolated Git workspace. > - Later admissions compare its saved configuration with the current policy. > - Fresh preparation also creates a temporary controller-side strategy. > - Remote preparation saved that temporary strategy in its fingerprint. > - The next unchanged turn then reported a workspace strategy replacement. > - This change saves the admitted remote strategy so unchanged reuse stays stable. ## Linked Issues or Issue Description Related preparation work: #18543 and #17924. I searched open and merged workspace/session repairs and public workspace-strategy issues. None covers this fresh remote retry case. **What happened?** After Retry created an isolated remote Git workspace, each unchanged follow-up reported `workspace_config_freshness` with `action: replace` and `changedCategories: [strategy]`. It retained the same workspace. The stored fingerprint contained `project_primary`, which was only a temporary controller provisioning choice. **Expected behavior** Save the configured remote strategy. An unchanged follow-up must reuse the same workspace without a false replacement report. A real strategy, base-ref, mode or environment change must retain its replacement checks. **Steps to reproduce** 1. Prepare a fresh isolated remote Git retry with a `git_worktree` strategy. 2. Complete its runtime handoff. 3. Admit another turn with the same workspace policy. 4. Inspect the recorded workspace freshness decision. **Paperclip version or commit** Reproduced on `b9e94b85a69bf679818eeef9ecf4655554a89c6b`. The regression also fails on the unchanged preparation code in the current base. **Deployment mode** Authenticated server with remote workspace authority and a sandbox execution target. ## What Changed - Keep the admitted strategy when a fresh workspace uses remote authority. - Retain the temporary controller strategy for existing host-side seed preparation. - Extend the PostgreSQL preparation test through a fresh admission and its next ordinary admission. Assert the same workspace, current writer and stable fingerprint, with no false drift operation. - Cover a genuinely configured `project_primary` fresh remote Git retry followed by a real strategy change, preserving its replacement decision and stored fingerprint. - Document the execution rule and the separate live verification recipe. ## Verification - The final regression was run before and after the production change. Before: the next admission recorded a strategy replacement. After: unchanged reuse records no drift and retains the workspace and fingerprint. - All 312 targeted cases pass on `48b3e548afa42e740871feae2763d88fd5a00431` across workspace preparation, workspace/session policy and inherited session preparation. These include actual strategy/base-ref changes and the rule that explicit reuse must not hide real replacement-class drift. - All eight remote preparation cases also pass with the compatibility regression: a genuinely configured `project_primary` fresh Git retry creates a `git_worktree` row, and its later real strategy change retains the replacement decision and saved fingerprint. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full recursive typecheck and `pnpm build` pass locally on this head. - The [full current-head CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41524105868) passes: typecheck/build, all general server, chat, workspace and serialized test groups, plus Runner and E2E checks. The workflow and `run-vitest-stable.mjs` partition the complete suite. - The duplicate local `pnpm test:run` passed all 1,828 chat cases and was intentionally stopped during general-server after the same complete CI matrix passed. This is not recorded as a full local-suite pass. - Greptile and Paperclip Review both assessed this head at 5/5 with no new findings. Paperclip Review accepted the compatibility boundary after the counterexample; its check conclusion is neutral. The addressed review thread is resolved. - The regression uses real PostgreSQL and preparation policy. Runtime acquisition is controlled at its external boundary. The new fix has not been deployed; live provider resumption remains unverified for this patch. ## Risks - Existing incorrect fingerprints are not rewritten. The same stored shape can come from a genuinely configured `project_primary` remote Git retry; a Git workspace row alone cannot prove the old admission policy. Rewriting it would hide a real policy change. After preserving and inspecting retained work, an explicitly admitted fresh recovery after rollout establishes a correct new fingerprint. Ordinary reuse of the old workspace does not migrate it. Normal recovery gates remain required. - Real policy changes still use the current replacement guard. No schema, credential or authority change is included. ## Model Used OpenAI GPT-6 Codex with reasoning, repository editing, shell execution, PostgreSQL tests and browser inspection. The host does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:11:30 -06:00
DottaandPaperclip	482706e7dd	perf(tools): index pending profile review scopes (#18681) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - The Apps UI lists tool profiles and shows new tools that need review. > - Both services project new-tool review for each profile. > - The projection scans include rules and exact rules for each catalog row. > - This repeats millions of comparisons when many selected profiles exist. > - This pull request builds scope and selector indexes once per projection. > - The benefit is less server CPU with the same fresh review results. ## Linked Issues or Issue Description **What existing behavior does this improve?** New-tool review projection in profile listings, profile review, and app attention. **Current behavior** Each newer catalog row scans the profile's included scopes. Included catalog entries require a map lookup for each scope comparison. Rows in scope then scan all rules again to omit tools that were already reviewed. Profile listing and app attention repeat this work for every profile. **Proposed behavior** Index covered application and connection scopes once. Expand included catalog entries to their existing connection and non-null application. Index exact selectors across all rules once. Keep the ordered catalog filter and returned metadata. **Reason and benefit** A recent main-thread profile identified this scope-matching path as a large CPU consumer. The local reproduction below confirms multi-second CPU work in the actual owning services. This PR removes the repeated scans without caching authorization or changing database reads. Related [#18673](https://github.com/paperclipai/paperclip/pull/18673) indexes effective access and profile summaries. It leaves this pending-review path unchanged. This PR targets the separate pending-review mechanism and has no dependency on that PR. The selector helper is identical so the owners can share it when both changes land. ## What Changed - Build covered application and connection sets for included scopes. - Build exact selector sets for all rules, including exclusions, to omit already reviewed tools. - Preserve null application selectors, missing catalog references, review watermarks, status checks, output order, and names. - Add differential tests against the former scope/matching evaluator. - Add PostgreSQL checks through all three affected service methods, including current catalog and rule changes. - Add a reproducible PostgreSQL benchmark of complete profile-list and app-attention results. ## Verification Baseline: `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. Treatment: `fd59c0183ce64c44696f79604516f4d51ca5662b`. Node 24.22.2 on macOS arm64. The production projection file at this baseline is identical to the file at `410b0e27015495ab8951768b5b0b94727de33785`. The benchmark exercises the real `createToolProfileService(...).listProfiles` and `createToolAppAttention(...)` methods against disposable PostgreSQL. It uses 200 selected profiles, 1,900 active tools, 76,000 rules, and 12 agents. Profiles alternate between 65 and 695 rules. Half reference 695 removed catalog entries; half select current tools. This tests both the slow failed scope scans and the slow exact-rule scans. No provider calls or hosted traffic are generated. The complete ordered results from both services match baseline. After one equality/warmup invocation per version, five samples alternate implementation order. Forced GC runs before each measured pair of service calls. Other workers paused owned builds and benchmarks; unrelated host activity was not controlled. | One profile listing plus one attention listing | Baseline median (range) | Change median (range) | | --- | --- | --- | | Process CPU | 5,406.512 ms (5,221.533–5,648.262) | 1,770.116 ms (1,596.232–2,045.617) | | Elapsed time | 5,279.231 ms (4,932.419–5,537.105) | 1,532.414 ms (1,499.199–1,610.428) | | Queries | 14 | 14 | | Returned database rows | 156,215 | 156,215 | Median process CPU falls **67.3%** and median elapsed time falls **71.0%**. Profile summaries retain their old nested matching in this comparison, so the result isolates pending-review indexing. Query counts and row counts do not decrease. This is local source/tsx service proof. It does not establish hosted improvement, whole-run latency, or HTTP tail latency. Reproduce with `server/scripts/benchmark-profile-review-projection.ts`; baseline-module setup is documented in `doc/mcp-discovery-performance.md`. Passed: ```sh node node_modules/vitest/vitest.mjs run server/src/__tests__/tool-profile-pending-projection.test.ts server/src/__tests__/tool-profile-review-service.test.ts server/src/__tests__/tool-profile-entry-order.test.ts node scripts/check-module-boundaries.mjs ``` All five tests in three suites passed. The differential tests cover all selector kinds, include/exclude effects, null scope behavior, missing references, review watermarks, and exact special-character names. PostgreSQL checks preserve current pending counts after exclusions, catalog removal, and review; another company remains separate. `git diff --check` passes. The first new integration assertion used `summary` instead of the existing attention response's `totals` field. It failed before the mutation checks. The test assertion was corrected; no production code changed for that correction. The final run above passed. Server typecheck was attempted but did not pass local setup. The packaged command did not reach TypeScript because a global pnpm child tried to fetch and verify the project's pnpm version through an unavailable registry. A direct Node 24 TypeScript invocation then reported missing built `@paperclipai/paperclip-runner` and related package declarations, with cascading type errors in unchanged consumers. No error points to the changed projection. The prerequisite build was not expanded into a full workspace build. Full local unsharded tests and build were not run; final-head CI must supply broad acceptance. No merge or deployment was performed. ## Risks Scope expansion must preserve existing semantics. An individual catalog rule covers its connection and only a non-null application. An explicit null application selector still matches null applications. Removed references contribute no scope when absent from the supplied catalog. The oracle and PostgreSQL tests cover these boundaries. The indexes add linear temporary memory in rule count and live only for one projection. Current company-scoped database reads and review mutations remain in their owners. Exclusions remain explicitly reviewed; they do not create new review items. Revert this commit to roll back. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with source inspection, code execution, tests, and local PostgreSQL benchmarking. The context window size is not available in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:10:42 -06:00
DottaandPaperclip	2b374caba0	fix(recovery): preserve unfinished remote exports before Retry (#18678) ... ## Thinking Path > - Paperclip manages agent work and preserves each task's saved files. > - A completed native result can still need workspace export. > - Remote workspaces now use a different saved descriptor from legacy copyback. > - The execution hold checked only the legacy descriptor. > - Task Retry could admit another run before the original export finished. > - A later failure could then supersede the original recovery record. > - This change keeps unfinished remote exports fenced and permits explicit historical file recovery. > - Recovery keeps human review and newer execution authority intact. ## Linked Issues or Issue Description **What happened?** A native run completed successfully, but its remote workspace export failed. The recovery action used `native_workspace_sync_out_retry_exhausted`. Task Retry created a second task run while the original files still needed preservation. The newer failure then superseded the original action. A board file-recovery request could queue that cancelled record, but the worker excluded it from discovery. **Expected behavior** Keep Retry behind the unfinished export. Save the exact source and finalize the existing accepted result once. Preserve pending human review. An explicit request for a superseded source must save its files without applying an old result over newer work. **Steps to reproduce** 1. Save an accepted successful native result with a prepared `remoteWorkspace` descriptor. 2. Fail its final export and retain its exact sandbox lease. 3. Select Task Retry before preservation completes. 4. Observe another task run before the original workspace is saved. 5. Supersede the original action and request file recovery for its exact source. Observe queued work that the worker does not select. **Paperclip version** Observed at `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. The regression is present on current master before this patch. **Deployment mode** Authenticated hosted instance with native Paperclip Runner and remote sandbox workspaces. Related: #18403 introduced the remote workspace path. #18644 reduces export scratch disk use. #17916 covers a separate legacy warm-copyback policy case. This change repairs remote export admission and explicit historical preservation. ## What Changed - Include prepared remote descriptors in the existing native export hold. - Retain remote source identity in the blocker and continuation read projections. - Let the existing worker select a cancelled export-repair record only after an explicit board file-recovery request. Keep the current action and workspace binding unchanged. - Add real PostgreSQL, HTTP, filesystem and Git regressions. Update execution semantics and recovery maps. ## Verification - Red before the admission fix: HTTP Retry creates two task runs instead of one. - Red before the historical-source fix: the explicit recovery stays scheduled and saves no snapshot. - All 588 focused tests pass across the workspace recovery, recovery source, explicit continuation and native export retry suites. The three new integration cases verify duplicate request identity, original result finalization, restored bytes and Git HEAD, pending human review, current-writer exclusion, and preservation without older-result commitment over newer work. - `pnpm -r typecheck`, `pnpm build`, map validation and `git diff --check` pass. - Full hosted CI passes on `c598718379fa939ec539e8e91a15a94d90321b59`, including all server, workspace, serialized, Runner and browser lanes ([run](https://github.com/paperclipai/paperclip/actions/runs/41529739604)). Greptile reviewed that commit at 5/5 with no findings; there are no unresolved review threads. Paperclip Review also published a complete 5/5 assessment for that exact commit with zero findings. Its check conclusion is advisory/neutral, titled `5/5`. The first attempt lacked a structured assessment; one normal completion request recovered the saved analysis and published the validated assessment. - The local `pnpm test:run` passed all six chat partitions, then was stopped during the general server group after equivalent full CI completed. Its partial log is retained; it is not reported as a completed local full-suite pass. - The test provider boundary uses real temporary files and the production archive path. It simulates provider resume/stop calls. The deployed fix and original hosted task recovery remain unverified until merge and rollout. - The older local Node 24.11.0 runtime failed to load the installed tsx worker. Node 26.4.0 passed the isolated worker check. Required repository checks use an isolated Node 24.22.2 runtime. The earlier failure logs remain recorded. ## Risks - Recognizing another unfinished export can keep previously admitted work waiting. This is intentional until persistence is verified. - Explicit historical recovery can resume the exact stopped sandbox to read files. It retains source, allocation, ownership, writer and bounded-attempt checks. It starts no provider turn. - A newer run still prevents commitment of an older accepted result. This change does not infer that a cancelled run had no external effects. - No schema, credential, deployment or fleet configuration changes. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact model ID or context-window size. Reasoning, repository inspection, tool use and code execution assisted this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:03:09 -06:00
DottaandPaperclip	919ddbf46d	fix(workspaces): read committed files before live allocations (#18677) ... ## Thinking Path > - Paperclip keeps task files in managed workspaces. > - A final remote save commits a durable snapshot and clears writer authority. > - The old allocation can remain recorded while its transport becomes unavailable. > - Files still tried that live allocation before the accepted saved version. > - This PR reads the validated save when authority is clean and no writer owns it. > - Dirty work and active writers still require the live allocation. ## Linked Issues or Issue Description Related: #18403 introduced remote Files access. #18665 fixes a separate completion-settlement issue and is included in the base. **What happened?** A file preview returned `422 workspace_unavailable` after a remote task had saved its files. Its clean authority still had a live location. The same original bytes were readable from the committed save after that location cleared. The exact original live transport failure is unknown. **Expected behavior** Once the workspace is clean and has no writer, Files reads its exact current committed save. A retained location must not make accepted files depend on remote command execution. **Steps to reproduce** Create a real committed workspace archive. Keep its allocation active in the recorded authority. Clear dirty state and writer authority. Make live execution unavailable, then request the ordinary file preview route. Before this fix it returns 422. After this fix it returns the saved bytes without a provider command. ## What Changed - Select the saved-file loader before consulting a live location when authority is clean and has no writer. - Keep exact company, workspace, snapshot, archive, and authority-change checks. - Keep the live path for dirty work and active writers. Never substitute their older saved bytes. - Cover browse, resolve, preview, download, rejection controls, and saved-copy cleanup through real routes and storage. - Document the selection rule and update its feature and access-map recipes. ## Verification - Red: the new ordinary-route regression expected HTTP 200 but received 422 against the old service. - A separate compatibility control reproduced the optional-loader regression. Valid live reads still work when the loader is absent; no-location reads still fail, and a configured invalid save never falls back to live bytes. - Green: 98 ordinary file-route and authorization tests passed with real PostgreSQL and archive storage. - All 46 opt-in remote workspace lifecycle tests passed against one owned local disposable PostgreSQL database. Its database, server, and files were cleaned after the run. No live S3 or provider flags were enabled. - Full workspace typecheck passed after the optional-loader compatibility repair. Full build passed locally before that narrow repair. All normal CI checks pass on the current head, including build and server suites. The separate Paperclip Review job and the full local test suite remain in progress. - Feature/access-map validation and `git diff --check` passed. - A read-only staging check preserved and verified the original saved bytes. It did not rerun the agent or qualify the failed journey. The fixed active-location path has local regression evidence; deployed acceptance is pending. ## Risks A clean workspace now uses the existing bounded saved-archive loader while a location remains recorded. Each Files request can retrieve and extract that archive. Existing disk, size, concurrency, cancellation, and cleanup bounds remain. A dirty workspace or current writer still uses live files and fails closed when they are unavailable. This change neither releases compute nor changes access permissions. It does not identify the original provider RPC subreason. ## Model Used OpenAI GPT-6 through Codex with reasoning, code execution, and browser tools. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 16:00:04 -06:00
DottaandPaperclip	b4a66027b7	fix(server): deliver chat completions after nested workspace saves (#18665) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat returns completed worker tasks to the original conversation. > - Delivery must retain its wake receipt when execution gates refuse admission. > - A workspace save can contain a nested snapshot operation. > - The old checks lost the receipt and treated that saved snapshot as unfinished work. > - This pull request preserves the receipt and recognizes a successful owning save. > - The benefit is that completed work can reach the chat through the normal queue. ## Linked Issues or Issue Description Refs #18652. Related: #17206 covers failed finalize recovery. This repair handles a successful finalize with an enclosed snapshot and does not change failed-save recovery policy. **What happened?** A delegated worker finished and saved its result. The original chat received no update. The dependency gate compared operation start times. A nested snapshot started after its enclosing finalize and therefore left a permanent hold. The completion wake also lost its exact idempotency receipt when the queue merged a refused wake into a generic dependency wait. The outbox reported `Completion wake returned no run or durable receipt`. **Expected behavior** A successful finalize covers a completed operation enclosed by that save when company, workspace, issue, and run match. Completion wakes that disable coalescing keep their own durable receipt even when admission waits. Dependency, budget, Stop, and pause gates still apply. **Steps to reproduce** 1. Hand a task from a chat to a worker and make the chat wait on that task. 2. Finish a workspace finalize that starts before its nested snapshot and ends after the successful snapshot. 3. Mark the worker Done and deliver its completion outbox entry. 4. On the parent revision, the dependency stays blocked and the completion receipt is absent. **Paperclip version or commit** Reproduced against `53f480d1b3` with real disposable Postgres rows and the normal queue/outbox services. ## What Changed - Honor `allowRunCoalescing: false` when recording execution waits. - Accept an enclosed successful operation only when the successful finalize has the same company, workspace, issue, and run and covers its whole recorded interval. - Preserve the existing rule that a strictly later successful workspace finalize clears earlier work. - Add failure controls for foreign runs, unfinished snapshots, failed saves, snapshots that finish too late, and later operations. - Exercise the real outbox and queue together. Hold provider dispatch at the existing drain boundary and verify server-recorded Done facts reach the admitted turn. - Document receipt identity and workspace-save coverage. ## Verification - Before repair: the two causal regressions fail and five safety controls pass. The missing exact receipt and empty wakeable-dependent list match the observed failure. - After repair: those seven cases pass without assertion changes. - The full affected set passed 288 tests and failed one existing scheduling test. Its unchanged Node24 baseline also failed. A closed-phase probe showed the fake adapter finished successfully at 3.453 seconds, after the incidental 3-second poll. The test now joins the existing execution promise and retains its status and dependency assertions under the unchanged Vitest timeout. The synchronized full dependency suite passes all 13 cases. - `pnpm -r typecheck` and `pnpm build` pass on pinned Node24/pnpm9. `pnpm test:run` is running. The combined real outbox/queue case passes. Fresh Greptile is 5/5 on `d7cee24b7032c7fdf171bfd1246cd90d41db1cba`. Exact-head normal CI and required `ci / verify` and `ci / e2e` pass. - The original scheduling failures on Node26 and Node24, the unchanged-source baseline failure, and the closed-phase probe are retained in the validation record. No model, deadline, production assertion, or paid attempt was retried to change the failure grade. - Current-head CI attempt1 passed all product and workspace coverage but failed the unchanged serialized retry-worker case at `issue-scheduled-retry-routes.test.ts:606` (first batch not observed before the poll deadline;410 other shard tests passed). The exact unchanged case passes locally on Node24 with real Postgres. The one failed-jobs-only retry passed on the same head; the original grade is retained. - The full local command completed all1828 chat integration tests. Its general stage remains running and has retained failures in unchanged `workspace-runtime.test.ts` (30-second local port-conflict fixture) and `muse-control.test.ts` (current binding authentication fixture). These are not counted as passing local full-suite evidence. Their current-head hosted coverage passed. - The separate advisory Paperclip Review app is paused and has no code finding. Its in-progress state is preserved and is not counted as a pass; it is not a required repository status check. - The live Cloud completion journey remains failed and unqualified. It must pass at the deployed fixed revision before daily-plan admission. No QA oracle, model, budget, or run limit changes are included. ## Risks - The workspace coverage query must not clear a save from another owner or new work after finalization. The guard checks the full interval and exact owner identity, with negative controls. - Refused noncoalescible automatic wakes retain separate receipts. Replaceable automatic signals still merge. No new polling or provider retry path is added. - No migration or API change. Deployment and live acceptance are separate remaining gates. ## Model Used - OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 15:06:26 -06:00
DottaandPaperclip	cd12432947	fix(runner): recognize verified task document evidence (#18661) ... ## Thinking Path > - Paperclip manages AI agents and keeps their results attached to tasks. > - Native tools return links to saved task documents. > - Completion feedback asks agents to include those links. > - The file-delivery validator treated every such relative link as a workspace path. > - A valid document and CSV result could therefore fail completion when it cited the supplied document link. > - This change derives document links from the same verified publication projection for guidance and validation. > - Requested files still need their own verified delivery evidence. ## Linked Issues or Issue Description **What happened?** The native document tool and completion feedback produce a link such as `/EX/issues/EX-1#document-output`. When a task also requests a downloadable CSV, citing that link causes completion to reject it as a workspace-only file. This occurs even when the document is published and the file has a valid current-run delivery receipt. **Expected behavior** Accept a server-derived document link only when it names the current company/task and a currently attached revision with a matching native publication receipt. Keep the separate requested-file requirement and all foreign, stale, missing, and workspace-file guards. **Steps to reproduce** 1. Save a task document through the native `write_document` tool. 2. Publish a CSV through `register_deliverable`. 3. Ask completion feedback for the final-response guidance. 4. Cite its returned document link alongside the file receipt in the next completion report. 5. The old validator rejects the valid document link as a workspace-only file. **Paperclip version or commit** Current source checkout. The regression exercises the real native document producer and completion validator. **Deployment mode** Local source verification on macOS arm64 with Node 24.22.1, pinned pnpm 9.15.4, locked dependencies, temporary PostgreSQL, and a controlled remote file-reader boundary. No live provider or hosted deployment is included. ## What Changed - Derive the canonical document href in the existing company/task/current-revision publication query. - Use that same verified href in completion guidance and reference validation. - Keep file registration and hash/size receipt requirements separate from document evidence. - Cover evidence, artifact, and criterion references; foreign companies/tasks; bogus keys; local paths; and lost current-revision publication proof. - Exercise actual native document writes and CSV registration, including attachment download query and fragment links. - Update the execution contract and the document feature/access maps. ## Verification - Red before the production edit: native completion generated a real saved-document link, then rejected the same link alongside a valid published file. The failure was the workspace-only-file guard. - Green: 210 tests across native completion feedback, explicit output validation, and the existing native file-handoff suite pass. These use real PostgreSQL and native tool receipts; the remote file reader is controlled. - A saved document link alone still fails the explicit CSV requirement. A document whose current revision lacks its publication receipt is rejected. Foreign task/company links, missing keys, malformed links, and workspace paths remain rejected. - Full workspace typecheck, server typecheck, and production build pass. - `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check` pass. Map checks establish structure only. - Full `pnpm test:run` is running separately. There is no full-suite verdict yet. CI and review are pending. - Hosted acceptance and any recovery of existing tasks remain separate steps. This PR does not change final-response recovery or scheduling. ## Risks - An overly broad URL exception could admit foreign or unpublished output. The validator accepts only exact hrefs from the authenticated publication projection. - Document evidence could hide a missing requested file. A separate regression keeps that delivery gate effective. - The shared projection adds current task/company joins. Existing document feedback and publication tests cover those consumers. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code editing, shell execution, and automated tests. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 15:02:00 -06:00
DottaandPaperclip	aa6c9c3f25	perf(storage): overlap streamed S3 multipart uploads (#18659) ... ## Thinking Path > - Paperclip manages AI agents and preserves their work. > - Large workspace and skill objects use the shared S3 multipart uploader. > - The uploader waits for each part acknowledgement before it sends the next part. > - Storage request latency therefore accumulates once per part. > - This change overlaps up to four streamed part requests. > - One source reader preserves byte order and backpressure. > - Failure cancels and settles active requests before cleanup. ## Linked Issues or Issue Description Related: #18082 added the shared streamed multipart uploader. #18644 fixes temporary disk use during workspace capture. It changes a separate stage. **What happened?** The shared S3 multipart uploader sends large object parts in sequence. The code waits for each `UploadPartCommand` before it consumes the next part. A controlled HTTP test confirms that only one complete part reaches the endpoint while its acknowledgement is held. **Expected behavior** Upload independent parts concurrently within a fixed limit. Preserve exact bytes, ordered completion receipts, source backpressure, cancellation, and full snapshot verification. **Steps to reproduce** 1. Upload a large stream through the S3 multipart provider. 2. Hold each successful part acknowledgement at a disposable storage endpoint. 3. Observe that the old uploader sends only one complete body until its acknowledgement returns. 4. Repeat with this change. Four complete bodies can reach the endpoint before the first acknowledgement. **Paperclip version or commit** Current source checkout. The affected uploader was introduced in #18082. **Deployment mode** Local source verification on macOS arm64 with Node 24.22.1, pinned pnpm 9.15.4, and locked dependencies. The HTTP tests use synthetic data and a disposable loopback endpoint. ## What Changed - Send up to four part requests at once through bounded streams. - Keep one source reader and retain the existing adaptive part-size policy. - Store successful receipts by part number before completion. - Cancel a stalled asynchronous source read without waiting for the source to produce more bytes. - Preserve the first error. Cancel sibling requests and settle them before multipart abort. - Cover exact bytes, out-of-order acknowledgements, backpressure, the concurrency limit, source errors, cancellation, and the real S3 HTTP client. - Document the transfer bound. Snapshot encryption, publication, and full object verification retain their existing behavior. ## Verification - Failing before the change: the held-acknowledgement regression expected four complete part bodies and received one. The same regression now passes. - All 15 focused multipart and storage-diagnostic tests pass. The real AWS S3 SDK sends four exact 8 MiB bodies through a disposable HTTP endpoint before any acknowledgement. Completion retains source order. - Controlled transfer comparison: one synthetic 64 MiB stream, eight 8 MiB parts, real S3 SDK and loopback HTTP, each acknowledgement delayed 100 ms. Original: 904 ms, peak one request. Fixed: 247 ms, peak four. Both transfers verified every part hash and completion order. This is local boundary evidence, not a hosted latency claim. - A second safety regression reproduced cancellation waiting indefinitely on a stalled asynchronous source. Abort-aware reads now settle within the existing bounded cleanup budget. - All nine existing encrypted snapshot storage tests pass after correcting missing library aliases in the local embedded-Postgres dependency. The failed setup attempt had no test verdict. - Full workspace typecheck and production build pass. The rebased server typecheck also passes. - The first full `pnpm test:run` attempt stopped during chat test collection with `No test found ... in lines ...` before a suite verdict. The chat test and line-selection scripts are unchanged from master. A retry after local Postgres dependency setup passed its first chat shards, then was stopped to avoid redundant host load after current-head CI passed every type/build/general/serialized/runner group. The unrelated narrow-banner E2E job failed its first attempt and passed one bounded retry; all current-head GitHub CI jobs are now green. There is no full local verdict. The separate Paperclip native review concluded neutral after reporting that work was paused; it supplied no substantive code review. - Commands: `pnpm -r typecheck`, `pnpm build`, and `pnpm --filter @paperclipai/server exec vitest run --configLoader runner src/__tests__/storage-s3-multipart.test.ts src/__tests__/storage-s3-failure-diagnostics.test.ts src/services/remote-workspace-snapshot-storage.test.ts`. - Deployment and an end-to-end hosted acceptance run remain separate steps. ## Risks - Four requests per upload increase concurrent storage traffic. Multiple simultaneous uploads multiply that bound. - Shared source consumption could reorder bytes. One producer and exact-content tests protect that boundary. - Failed siblings could outlive multipart cleanup. Cancellation and settlement tests protect cleanup order. - Capture and full read-back verification still take time. This change only targets serial multipart latency. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code editing, shell execution, and automated tests. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 15:01:34 -06:00
Devin FoleyandPaperclip	49b3804830	fix(connections): load permissions and show AI account identity (#18653) ... ## Thinking Path > - Paperclip manages AI agents and their access to saved connections. > - The Permissions page loads account details and permitted agent choices. > - Company-wide AI accounts repeat retained installation checks for every candidate agent. > - This makes target checks grow with the square of the roster size. > - The browser waits without a deadline, and connection names do not identify the provider login. > - This pull request batches direct AI checks and bounds loading with recovery controls. > - Account details load separately and show a provider email or an explicit profile-permission limit. ## Linked Issues or Issue Description **What happened?** Opening a saved AI account can leave Permissions on “Loading AI account…” for minutes. The account itself can be connected. The existing capability batch excludes AI connections, so a company-wide install repeats checks for every installed agent for every candidate. Saved names and Paperclip ownership also do not establish which provider login a credential uses. **Expected behavior** Direct AI accounts reuse retained-target checks within one descriptive request. The page shows loaded account controls or a bounded failure with Retry. Permissions shows the provider email and organization when the selected credential permits profile access. An unavailable profile must not block maintenance or disconnect a working inference credential. Writes still authorize current permissions. **Steps to reproduce** 1. Save an AI account in a company with several agents. 2. Install the account company-wide. 3. Open its Permissions page. 4. Delay the account-list or grants response to exercise the loading failure. The maintained benchmark reproduces database work with synthetic local data. Refs #17945, which added batching for non-AI connections. Searches found no open PR for this AI-account loading fix. ## What Changed - Include direct AI accounts in the existing board capability batch. Referenced pool members, delegated actors, complex selectors, and mutation authorization keep their existing evaluator. - Add opt-in per-caller GET deadlines to the API client. A timed-out caller detaches the stuck coalesced request so Retry starts a fresh fetch. Other callers retain their own cancellation and results. - Apply a 15-second deadline to AI account lists and grants reads. The permissions view retries transient failures once and reports other failures immediately. - Share the parent Permissions grants query with AI account details. Keep loaded controls visible after a background error and offer Retry. - Add a grant-scoped identity GET and OpenAPI contract. The service checks current membership and credential audience before it reads the selected credential. It projects only the provider email and organization; provider IDs, tokens, and raw errors stay private. - Show account details independently of maintenance controls. Anthropic profile denial has an explicit message and preserves connected status. Transient profile failures have a separate Retry action. Observations are cached for five minutes under the company, connection, grant, and grant revision. - Extend real PostgreSQL and authenticated HTTP coverage, client/component regressions, map guidance, and the existing benchmark's AI mode. - Address the Apex file-size finding by moving gallery identity lookup into the existing app display helper module. `AppDetail.tsx` is 984 lines; lookup behavior is unchanged. ## Verification Local source checkout on macOS, Node 25.6.1, pnpm 9.15.4, and disposable PostgreSQL 18.1. Provider calls use fixtures. No hosted deployment or provider qualification is claimed. - Passed `pnpm -r typecheck` and `pnpm build`. - Passed all 87 tests in `connection-management.integration.test.ts` and all 33 tests in `connection-install-capabilities.test.ts`. - Passed 196 AI connection, identity-provider, and OpenAPI tests: `ai-connections.test.ts`, `ai-account-identity.test.ts`, and `openapi-routes.test.ts`. - Passed 107 focused UI/client tests: `client.test.ts`, `ManagedAiConnectionDetails.test.tsx`, `AiConnectionAccountDetails.test.tsx`, and `AppDetail.test.tsx`. - After the Apex extraction, passed UI typecheck/build and 106 component tests across `AppDetail.test.tsx`, `AppNotConnected.test.tsx`, `ManagedAiConnectionDetails.test.tsx`, and `AiConnectionAccountDetails.test.tsx`. - The authenticated HTTP tests compare complete grants responses across human roles, for tool and AI connections. They verify a real install mutation succeeds before revocation and fails afterward. - A company-wide personal AI fixture checks 20 targets instead of 400. Pool-member permission loss still denies the complete pool description. - UI states covered: shared cached grants, stuck account/grants fetches, one automatic retry, terminal failure, manual recovery, authorization failure, background failure with retained controls, and unmount cancellation. Client tests also cover stalled response bodies and independent coalesced callers. - Identity coverage includes actual bearer-key/JWT reads, private-user and cross-company denial before provider access, revoked credentials, provider 401/403/429/500, and grant-change isolation. A real local HTTP fixture stalls the response body, verifies the shared 15-second deadline, and keeps unrelated HTTP requests responsive. The test shortens the native deadline to 100 ms. Profile permission denial leaves the saved account connected. - Passed `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check`. - The full local `pnpm test:run` started with the loading-only change. All six chat shards passed. It recorded one failure in the unchanged `heartbeat-retry-scheduling.test.ts` quota-rejection case. That exact case passed a focused rerun on the final head. The initial failure cause remains unconfirmed and its local log is retained; this is not a full local-suite pass. The redundant local run was stopped with SIGINT after final-head CI completed successfully. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41515556725) passed on `ead8d276d4a2023614fff54b3e9a858f28c3178b`: typecheck, build, all 12 server shards, all 3 chat shards, workspace suites, all 9 serialized suites, all 8 end-to-end shards, Runner checks, and the aggregate verify gate. Security checks passed. Optional Storybook checks were skipped by workflow policy. GitHub reports the branch mergeable without conflicts. - Apex reviewed head `ead8d276d4a2023614fff54b3e9a858f28c3178b` at 5/5, confirmed the extraction resolves its earlier file-size finding, and has no unresolved review threads. The existing benchmark ran for the loading-only change now recorded in commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce`, with `--purpose=ai --baseline-ref=026fd9e3452`. The identity follow-up leaves that batch source unchanged. Two warmups and five alternating samples exercise each of eight fixture shapes. Complete positional capabilities, reasons, and operations match the baseline in every sample. | AI fixture | SQL calls before → after | Target reads before → after | | --- | ---: | ---: | | 1 agent, company install | 9 → 9 | 1 → 1 | | 32 agents, company install | 5,248 → 164 | 1,024 → 32 | | 100 agents, company install | 50,400 → 504 | 10,000 → 100 | | 100 agents, partial install | 13,175 → 503 | 2,575 → 100 | These counts measure the real service over local PostgreSQL. They do not predict hosted page latency. To reproduce the standalone benchmark, use a checkout of commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce` and run the command below. Its isolation check intentionally rejects the later identity follow-up's additional runtime changes. ```sh node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --purpose=ai --baseline-ref=026fd9e3452 ``` Run the final-head regression suites from the repository root: ```sh cd server pnpm exec vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/connection-install-capabilities.test.ts src/__tests__/ai-connections.test.ts src/services/ai-account-identity.test.ts src/__tests__/openapi-routes.test.ts cd ../ui pnpm exec vitest run src/api/client.test.ts src/components/ai-connections/ManagedAiConnectionDetails.test.tsx src/components/ai-connections/AiConnectionAccountDetails.test.tsx src/pages/apps/AppDetail.test.tsx ``` ## Risks Descriptive permissions reuse retained facts within one request. A concurrent change can leave a displayed choice stale; the next request and every write still authorize fresh. Pools with referenced members retain their existing cost. The account and grants clients now bound each caller's wait. A browser abort does not cancel an authorization read already admitted by the server. The independent profile request uses a fixed Anthropic origin, a bounded body, no redirects, and a 15-second deadline. Inference-only credentials can return no email; that is reported without guessing from a Paperclip owner or changing connection health. Profiles are fetched only for the caller's authorized selected grant. No ambient CLI identity, persistent identity cache, background polling, or billing-identity change is added. Other API reads keep their existing behavior. No migration is required. Staging has not been redeployed. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, code editing, execution, and test tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 12:29:03 -08:00
DottaandPaperclip	33fc7fcddc	fix(db): clean up owned PostgreSQL test servers (#18627) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Database, server, and CLI tests use real PostgreSQL fixtures. > - Each fixture must own its server until PostgreSQL has stopped. > - The old teardown returned after five seconds even when its server remained alive. > - A startup failure could then mark the host unsupported and skip database coverage. > - This pull request waits for owned shutdown and makes supported-host startup failures fail collection. > - A shared-server command reduces process creation while keeping each fixture's database isolated. ## Linked Issues or Issue Description Refs: #7756 for related development-process ownership. That development-server issue remains outside this test-fixture fix. Related PR: #16799 collects macOS startup diagnostics. This PR changes test shutdown and failure behavior. **What happened?** Repeated test runs can exhaust macOS PostgreSQL semaphore resources. The fixture abandoned slow shutdown and swallowed stop errors. The support probe converted ordinary startup failures into skipped integration coverage. **Expected behavior** Tests must stop their PostgreSQL servers, retain files if stop fails, and report resource exhaustion as a failure. Fixtures on a shared server must not close another database's clients. **Steps to reproduce** Start a fixture whose PostgreSQL stop exceeds five seconds, or make a supported-host startup fail with semaphore exhaustion. The former helper returned from teardown or reported unsupported coverage. The new regression tests exercise both paths without exhausting host resources. **Paperclip version or commit** Source checkout based on current master. **Deployment mode** Local source development and CI tests. ## What Changed - Add explicit ownership for test PostgreSQL children. Wait for fast shutdown, then request immediate PostgreSQL shutdown if needed. Keep cleanup idempotent and fail when exit cannot be confirmed. - Publish reusable templates only after clean shutdown. Do not publish a crashed or immediately stopped cluster. - Stop owned children on POSIX worker signals and process exit. Preserve test directories when shutdown fails. Remove conflicting dependency exit hooks. - Retry port collisions only. Fail collection for resource exhaustion, unavailable configured servers, and other startup failures on supported platforms. Preserve skips for explicitly unsupported platforms. - Move administrative-server support into the common fixture used by database, server, and CLI tests. Close clients at database scope and migrate cloned templates before use. - Add a directly invoked shared-server runner to own one test server and run suites against separate disposable databases. Preserve child exit status and support an existing disposable server. - Unify cancellation: stop and wait for the test process group before PostgreSQL teardown. Add nested-process coverage, real worker-exit coverage, shutdown regressions, shared-database isolation checks, and setup documentation. Replace ignored heartbeat completion polls with the existing execution drain before asserting persisted outcomes. - Finish owned cluster-directory cleanup after signal-driven cancellation. Preserve prepared templates across shared-fixture tests. Remove stopped directories asynchronously, using native removal on POSIX with a Node fallback. - Align database and CLI fixture hook budgets with the existing 30-second server budget. This lets the ten-second shutdown protocol finish and report errors. The idle-sleep suite uses the existing 90-second fixture budget for its unusually large final directory cleanup. ## Verification - The semaphore-exhaustion regression failed against the old helper because it resolved to `supported: false` instead of rejecting. - All 37 focused tests pass on the final runtime code (`e4de00ab40`). They include real worker exit, SIGTERM, nested-process SIGINT/SIGTERM cancellation with PostgreSQL available during child cleanup, failed template shutdown, unsupported platforms, stale SQL templates, and directory removal. A child exiting with status 7 makes the shared-server runner return status 7 after cleanup. - Full local `pnpm -r typecheck` and `pnpm build` pass on the final runtime code (`e4de00ab40`). Subsequent changes only adjust server-test lifecycle waits, the idle-sleep cleanup budget, and documentation; the server typecheck also passes with those final tests. - A fresh shared-server run on `90b65fd8ad` passed all 68 database files and 297 tests. Earlier preparation runs had two ten-second teardown timeouts and a stale-template failure; the final helper migrates cloned templates and permits the full shutdown budget. These earlier failures remain accounted for here. - The local full repository attempt stopped on a Discord receiver-lease test after 338 passing chat tests. That exact case passed in isolation. This full-suite attempt is not a pass. - On `90b65fd8ad`, [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/41496495253/job/125452949369) and its one unchanged retry reported two 15-second cached-input heartbeat timeouts and a 30-second idle-sleep teardown timeout. All ten cached-input cases then passed locally with their original deadlines. A local parallel run of the full files reported eight behavioral/hook failures and two suite teardown timeouts (281 tests passed). - Serial idle-sleep runs passed all 225 test assertions but overran the 30-second final hook. Tracing measured client closure in 4 ms, PostgreSQL exit after 5 seconds, and synchronous directory removal taking another 35 seconds. Asynchronous Node removal and native removal also exceeded that hook under the full workload. Cleanup now remains asynchronous and awaited, and only this heavy suite's final hook uses the existing 90-second fixture budget. No PostgreSQL shutdown deadline or test assertion was weakened. The serial run then passed the complete idle-sleep file, including all 225 tests and teardown. Its heartbeat file passed 63/64 cases, exposing an ignored three-second completion poll. After replacing that poll with an execution drain, another timer case exposed the same pattern (63/64 passed). All remaining ignored completion polls now await execution drain without changing test deadlines. The complete heartbeat file then passed all 64 tests. The intentional asserted lock-contention poll is unchanged. The final server typecheck passes, and a process audit found no PostgreSQL server left by the completed local runs. - On final head `2cf316d3f571e4516890cba743ecffeb9703cce3`, the previously failing server shard 6, database lane, full typecheck, and build passed. The [first CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41506358136) then lost self-hosted runners in the canary and serialized-server lanes, with additional jobs cancelled. The server-2 lane had one 30-second timeout in the unchanged workspace-recovery HTTP Retry test (2,282 other tests passed). That exact test passed locally with its original deadline (28.78 seconds including imports). One unchanged retry of only failed and cancelled jobs completed successfully on the exact final commit (workflow attempt 2). The workspace-recovery timeout case and every interrupted lane passed; successful first-attempt jobs were retained. The additional Paperclip Review check remains queued, waiting for its assigned agent; it has no unresolved GitHub review threads. Greptile completed review on this exact final head at 5/5, with zero unresolved review threads. All four original review findings and their threads are resolved; additional Windows shutdown, cancellation-directory, and template-reuse findings are fixed. - `pnpm check:module-boundaries` and `git diff --check` passed. Post-run process checks found no PostgreSQL servers left by the completed verification runs. ## Risks - Immediate PostgreSQL shutdown can require crash recovery. This fallback applies to disposable test clusters and runs only after fast shutdown times out. - No process can handle `SIGKILL` or machine power loss. Host resource cleanup remains a separate backstop. - Directory cleanup on cancellation applies to the shared-server runner's own cluster. Directly terminated test workers stop their PostgreSQL process but retain disposable files for inspection. The process-exit guard cannot await file removal. - The two-second `pg_ctl -W` deadline bounds signal dispatch. It is separate from the five-second and thirty-second server-exit waits. A control-command failure fails cleanup and retains its directory; it does not report successful cleanup or force-delete live files. - Shared fixtures require a disposable PostgreSQL server and a role that can create databases. The directly invoked shared command is opt-in; the existing `pnpm test` command remains available. - These changes do not stop unrelated development servers or delete host-wide IPC resources. Clean shutdown uses the owned `pg_ctl` binary on every supported platform. Its real control path was tested on macOS. A Windows host was not available for live verification; POSIX signal tests skip on Windows. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving variant or context-window size. The agent used reasoning, shell execution, code editing, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 14:24:48 -06:00
DottaandPaperclip	946735d03a	Persist task sorting in URLs and show newest routine tasks first ... Save task sort choices in URLs across task views. Open routine tasks by creation time, newest first, with stable sorting before pagination. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>	2026-11-02 13:32:40 -06:00
DottaandPaperclip	5a07c30c1a	fix(workspaces): reduce temporary disk use during remote capture (#18644) ... ## Thinking Path > - Paperclip manages AI agents and preserves their completed work. > - Native remote finalization must save workspace files before it releases the writer. > - Portable capture copied the working tree and Git history, repacked the history, and then wrote an archive. > - These temporary copies can exhaust a small sandbox after normal development work. > - A completed provider turn then fails finalization and file preservation repeats the same failure. > - This change streams checked working files and keeps only an independent Git copy in scratch. > - The save still rejects changed sources and incomplete output. ## Linked Issues or Issue Description Related: #18403 added portable remote workspace storage. #18172 handles disk pressure in Daytona directory transfer. This failure occurs earlier, while the portable archive helper creates its source archive. That earlier repair does not cover this path. **What happened?** A completed native run failed workspace finalization three times. Its file-preservation retries also failed. In a retained 10 GB Linux sandbox with about 1.5 GB free, the deployed helper failed during `git repack -a -d` with `Out of diskspace`. The working files used about 603 MB, and Git used about 444 MiB. **Expected behavior** Capture a coherent workspace when free space can hold one full archive plus an independent Git copy. Preserve working files, staged changes, unpushed commits, and existing recovery gates. **Steps to reproduce** 1. Use a disposable Linux sandbox with a Git repository and generated working files. 2. Leave enough free space for an archive and a Git copy, but not a second working tree and another repack. 3. Run the portable capture helper. The old helper fails while creating temporary data. 4. Run the fixed helper against the same source and free-space limit. Verify each archived file hash. The small regression uses real file serialization and a controlled filesystem capacity failure at archive creation. It failed with `ENOSPC` before the production edit and passes afterward. **Paperclip version or commit** The observed staging server served `42c64d4ec4b9b0898556fcdf6ba304c4285a7819`. The repair is based on `a92f616cd0`. **Deployment mode** Hosted staging. Linux x64, Node 25.9.0, Daytona sandbox. Local tests use a macOS source checkout. ## What Changed - Stream working files from checked file handles. Keep only normalized Git metadata in scratch. - Remove the second repack after a mirror clone that already uses `--dissociate` and `--no-hardlinks`. - Check each streamed file's hash, size, mode, and filesystem identity. Recheck the source inventory and Git state before returning a receipt. - Remove incomplete archives after failure or cancellation. Keep pre-existing archive files intact. - Add space-limit, concurrent-change, cancellation, and existing-file safety cases. Update execution semantics and workspace verification recipes. ## Verification - Red before the production change: the capacity regression failed with `ENOSPC`. Green afterward: 54 archive and transport tests pass. - The existing tests restore ordinary, linked, and borrowed Git repositories after their sources disappear. They retain staged, unstaged, and unpushed work. Credential and path guards also pass. - Live comparison on the same retained sandbox: the deployed helper failed after 19.4 seconds with Git disk exhaustion. The fixed helper completed in 35.3 seconds. It produced a 1,081,872,803-byte archive with 24,111 entries. An independent reader checked every archived file hash and the manifest digest. The source commit stayed unchanged. Diagnostic scratch was removed. - This live probe exercised the changed capture boundary. It did not deploy a server, upload a production snapshot, finalize the failed run, or replay its provider turn. Full task recovery still needs rollout and the normal export recovery path. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Current-head CI passes: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41499985180), commit `d42700f7f4b0bef14a5969a44c2da0b2513e0ed2`. Greptile rates the same head 5/5 with no actionable findings or unresolved threads. - The independent Ponytail advisory also rates this exact head 5/5 with no findings. Its structured check remains queued because the automatic request was bound to the earlier head. A [fresh current-head review request](https://github.com/paperclipai/paperclip/pull/18644#issuecomment-6585296011) was accepted and queued. Its predecessor has a successful workspace export but remains active in finalization, so the new review cannot start yet. This review-service publication gate must complete before claiming every check is green. - `pnpm -r typecheck` and `pnpm build` pass. The final server-only typecheck also passes after the test additions. - Supported-runtime verification on Node 24.11.0: all 54 archive/transport tests and all 46 existing workspace lifecycle tests pass. The lifecycle suite uses a disposable real Postgres database and local filesystem transport with encrypted snapshot storage. It checks save/restore after losing the original allocation, ownership, retention, and file continuity. This is separate from the live Daytona capture probe. - The first extra lifecycle attempt could not start its local Postgres binary because the installed Darwin package lacked library aliases. Local dependency setup was repaired before the passing run. No production or repository code changed for that setup. - The full local `pnpm test:run` on the host default Node 26.4.0 stops in chat-test collection with `No test found ... in lines ...`; it has no suite verdict. A clean retry repeats the failure. The unchanged `a92f616cd0` test wrapper reproduces it with the same installed dependencies, after correcting an initial baseline setup that lacked built package entries. This is a local test-runner limitation, not a full local pass. Current-head CI remains the full-suite gate. - One focused run timed out while deeply comparing 4 MB buffers under concurrent build load. Complete SHA-256 comparisons retain the content check without that assertion cost. The capacity case then passes in under one second, and the final 54-test run passes. The disk allowance and production code did not change. ## Risks - Streaming reads the live source instead of a copied working tree. File identity, content hashes, mode checks, the final inventory read, and Git signatures must reject concurrent changes. Regression tests exercise these checks. - Capture still needs space for the full archive and normalized Git metadata together. It cannot export from a completely full disk. - Archive framing and restore validation stay compatible. The change has no database migration or new execution authority. - The incident remains unrecovered until the fixed server performs normal workspace finalization. No deployment or merge is included in this PR. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell tools, browser diagnostics, and automated tests. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-02 13:22:23 -06:00