mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
58308a3e6343bb4ef71fa7f87899d730dab2e312
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
554a770c6f |
perf(apps): scope connection profile configuration reads (#18687)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Apps Permissions page reads saved access profiles for one connection. > - The page fetched every company profile, rule, and binding. > - It used only the app profile and eligible task-granted profiles for that connection. > - This pull request adds a saved configuration read owned by the profile service. > - The route keeps current connection and per-profile management checks. > - The benefit is fewer database rows, less Node work, and a smaller response for the same displayed configuration. ## Linked Issues or Issue Description **Subsystem affected** Server profile configuration reads and the Apps Permissions and Review data caller. **Problem or motivation** One connection page loaded and serialized unrelated company profiles. In the local fixture, this produced a 32.35 MB response with 76,001 rule rows. **Proposed solution** Select the displayed connection's app profile and exact eligible connection-intent profiles before loading their entries and bindings. Return an explicit configuration DTO with complete selected arrays. Keep summary and pending-review work on the existing full API. **Alternatives considered** Filtering the full response in the browser retains the database and Node cost. Truncating the full API would break other clients. Filtering entries to the connection would lose selectors and bindings needed by saves. **Roadmap alignment** This improves the existing governed Apps capability. It does not add a new access policy. Related merged work: #18673 and #18681. Related harness cleanup: #18683. Public PR searches found no duplicate connection configuration projection. ## What Changed - Add `ToolProfileConfiguration` and `GET /api/tool-connections/{connectionId}/profile-configuration`. - Select `app:<connectionId>` at any status and exact active, correctly bound `connection_intent` profiles. Run the existing per-profile visibility filter before configuration queries. - Preserve complete ordered entries and bindings. Keep all save behavior, empty-binding install fallback, and the existing full profile API. - Switch AppDetail and AgentConnectionAccess to the configuration DTO. Use a distinct cache key below the existing profiles prefix so mutation and reconnect invalidations still refresh it. - Add real PostgreSQL differential and authenticated HTTP tests, UI caller coverage, a reproducible benchmark, and access-map documentation. ## Verification Baseline: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`, after both #18673 and #18681. Final head: `b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`. Current implementation and reproduction details are in [the performance note](https://github.com/paperclipai/paperclip/blob/b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22/doc/performance/2026-11-02-connection-profile-configuration.md). - AppDetail: 79 tests passed. Existing access/action saves, install fallback, grant removal, errors, and refresh cases use the new caller. - Real PostgreSQL differential: passed. Full ordered selected configuration equals the old full-list selection. Cases cover extra bindings, sibling selectors, wrong source/key/status/binding, null and non-string agent metadata, fresh mutations, empty app bindings, visibility filtering, and company scope. - Production HTTP authentication: API key and run JWT cases passed. Hidden personal accounts omit spanning profiles. Private connection reads fail. Restored ownership restores visibility. Revoked management grants deny fresh calls. Final UID canonicalization and foreign-company 404 assertions passed on the current head. - Map structural check, token gate, and `git diff --check`: passed. - Required local `pnpm -r typecheck` and `pnpm build`: passed on the current head. `pnpm test:run` was started once. Its first chat shard passed (337 tests, 151.87 seconds); the second shard was intentionally interrupted after the complete final-head CI suite passed. This partial local attempt is not a full local test pass. The same stable-runner general and serialized partitions and Runner lanes passed in [final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41538805231), including aggregate `ci / verify`. No local assertion failures were observed before interruption. - Final-head focused acceptance: four files, 85 tests passed. Full token gates, maps, and module boundaries passed. - Actual browser acceptance passed on the current checkout with isolated real PostgreSQL and a local trusted board. An action saved as Off persisted after reload. Both app agent bindings stayed intact. Removing an agent-only task grant persisted after reload. Requests used only the narrow profile endpoint. No browser exceptions occurred. No responses were mocked. No provider calls or hosted changes were made. The helper stopped the owned runtime and database after the check. - Final exact-head checks: 54 success, one clean neutral Paperclip Review, three skipped, zero pending or failures. Legacy Snyk status: success. No inline review comments. - Greptile reviewed exact head `b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`: 5/5, no findings, completed successful check. Five alternating unprofiled pairs ran with real PostgreSQL on macOS/arm64, Node v24.22.2, and forced GC. The fixture has 210 profiles (200 heavy profiles plus ten eligibility cases), 76,001 rules, 1,900 active tools, 695 removed tools, and 12 agents. The selected configuration has two profiles and 1,391 rules. Each pair asserts complete ordered selected DTO equality. | Measurement | Full list | Configuration read | | --- | --- | --- | | Service wall ms, median [min, max] | 200.927 [193.319, 213.390] | 6.414 [5.508, 6.973] | | Node CPU ms, median [min, max] | 262.751 [246.727, 271.573] | 27.525 [22.576, 29.308] | | Select queries / returned rows | 7 / 78,139 | 3 / 1,397 | | Actual response JSON bytes | 32,353,534 | 592,233 | | Serialization wall ms, median [min, max] | 90.714 [88.715, 99.304] | 1.644 [1.637, 1.745] | | Serialization CPU ms, median [min, max] | 90.644 [88.569, 96.778] | 1.654 [1.637, 1.807] | Reproduce: `node --expose-gc --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-connection-profile-configuration.ts 5`. The benchmark measures the service and actual response JSON separately. It excludes HTTP and authorization work through an identity visibility callback. Node CPU does not include PostgreSQL CPU. No hosted or foreground-under-load improvement is claimed. Preserved failures: offline install lacked a locked tarball; frozen online install passed. Sandbox localhost and tsx IPC restrictions blocked initial DB/typecheck attempts; permitted retries use the same checks. The first HTTP suite needed the unbuilt plugin SDK; its owning build helper fixed setup. Shared typecheck caught a missing barrel export, now fixed. A broad typecheck inherited the host's unpinned nested pnpm executable; the owned stalled process was stopped and restarted with pinned pnpm 9. The complete current-head CI suite and aggregate verify passed. The separate Paperclip agent review published 5/5 with no findings at this head and completed with a clean neutral conclusion. An added foreign-company assertion expected 403; the existing guard correctly returned 404. Only the test expectation changed. The first real browser attempt saved and reloaded the action correctly, but grant deletion returned 422: the stress fixture had a company binding, which makes that profile the company default. The UI showed the existing deletion error. The second browser seed used the production agent-only intent shape and passed removal and reload. This does not change the stress fixture or product deletion gates. Initial external browser script attempts also needed an ESM extension and the correct dependency path. Logs, first-failure screenshots, traces, and result files are retained in the local evidence directory. ## Risks - SQL selection must stay aligned with connection-intent eligibility. Differential fixtures cover the exact current predicate. - A selected connection can still have many rules. This change preserves all selected configuration. - Per-profile authorization still reads current selected resources and can add work. Connection visibility does not bypass that check. - No migration, default-allow, effective-access, policy-review, provider, or deployment behavior changes. The old endpoint remains available. Rollback restores the old AppDetail caller and removes the additive endpoint. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with code execution and repository tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3f7811eef0 |
fix(runner): keep requests responsive during retired backup cleanup (#18619)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote native runs keep durable harness backups for continuation and
recovery.
> - Publishing a new backup retires earlier backup directories in the
server process.
> - The synchronous permission walk and recursive removal block
unrelated requests.
> - This pull request makes that cleanup asynchronous and keeps
publication waiting for completion.
> - The benefit is continued request progress while large retired
backups are removed.
## Linked Issues or Issue Description
**What happened?**
Publishing a backup with many small files blocks the Node event loop
during directory permission repair and removal. A local comparison with
20,000 retired files reproduced 1.37–1.95 second event-loop stalls. This
is a pause candidate. A short hosted profile has not established it as
the dominant source of sustained CPU use.
**Expected behavior**
Unrelated requests can progress during cleanup. Publication still waits
for complete removal, preserves rollback on a failed lease stamp, and
leaves live files and symlink targets unchanged.
**Steps to reproduce**
Run the benchmark in `doc/performance/native-backup-publication.md`. It
compiles the actual publication service from a pinned baseline and the
working tree, prepares immutable retired backups, and sends foreground
HTTP probes from a separate process.
**Paperclip version or commit**
Measured baseline: `774ddc72713d6aa7cb516691c71f1fc23bcfa5ff`. The
measured treatment source is committed in this branch. Current head:
`0a69cf2d43c2f9fcf032d96c1dceb56163bf21ba`. The runtime patch is
unchanged from the measured source; later commits only strengthen the
test and benchmark evidence paths.
**Deployment mode**
Local compiled service on macOS arm64 with Node v24.22.2. No hosted
rollout has occurred.
Related work: #18380 changed retention scheduling. #13676 included the
existing immutable-backup cleanup behavior. Neither change makes
publication cleanup asynchronous. The GitHub duplicate search also found
#11929, which concerns database dumps rather than native harness
backups. No matching open event-loop cleanup issue was found.
## What Changed
- Use asynchronous filesystem operations for retired-backup cleanup.
Visit directories serially and skip unnecessary metadata calls for file
entries.
- Await cleanup before publication proceeds, during rollback, after
successful rotation, and in executor finalization.
- Exercise immutable current and previous backups in the publication
test. Preserve the existing symlink-isolation and rollback tests. Add a
real separate-process HTTP probe that must complete while publication is
still active.
- Add a repeatable service benchmark and document its fixture, metrics,
and acceptance limits.
## Verification
Draft status: focused correctness tests, the local comparison, and
repository typecheck pass. The complete local test gate is incomplete.
The build passed. Complete CI, hosted acceptance, and final-head review
remain pending.
- Final focused Vitest run: 2 files, 4 tests passed in 649 ms with the
normal server configuration. The regression warms a separate Node
client, waits for its HTTP request during the stamp callback, queues the
response for the next event-loop turn, and then allows final
retired-tree removal to start. The async treatment serves that response
before publication completes. The original synchronous cleanup fails the
same final regression in a separate source fixture. That expected
control failure is retained.
- Syntax and `git diff --check` passed.
- `pnpm -r typecheck` passed on head `332c57b88c` in 152.18 seconds,
including the native Runner build. Validation uses Node v24.22.2 and the
repository-pinned pnpm v9.15.4 with isolated dependencies.
- `pnpm test:run` was intentionally stopped after 1,390.24 seconds (exit
143). All six chat shards passed, covering all 1,817 parameterized cases
across their exact partitions. The general server phase was advancing
but incomplete. The repository timing manifest estimates another roughly
two hours for the serial remainder, so validation moved to final-head
parallel CI. This is not a full local test pass. The command began on
`332c57b88c`; later changes only refine the regression and benchmark,
and the final regression has its own focused pass.
- Benchmark failure/metadata checks pass in a tiny isolated Git fixture:
an injected worker exit 17 is reported in 0.44 seconds without hanging,
and the result includes a staged source change when the unstaged diff is
empty.
- `pnpm build` passed in 41.80 seconds on `9f7e4e669b`. The final commit
only releases a pending test probe on failure; runtime and build sources
are unchanged. An injected client exit verifies that final test failure
path exits promptly in 0.97 seconds instead of hanging.
- Three alternating baseline/treatment pairs used fresh fixtures with
100 directories × 100 files in each of two retired backups. Each
publication removed 20,000 files. A separate process sent HTTP probes at
fixed 5 ms intervals, including overdue arrivals.
- Every run verified the replacement manifest, complete cleanup, and
unchanged symlink-target contents and mode. All 2,395 HTTP requests
completed without errors. The largest measured generator delay was 8.8
ms.
| Metric | Baseline median (range), ms | Treatment median (range), ms |
| --- | ---: | ---: |
| Foreground request p95 | 1,357.7 (1,287.4–2,014.0) | 1.8 (1.7–25.9) |
| Maximum event-loop delay | 1,439.7 (1,371.5–1,954.5) | 19.9
(17.4–40.7) |
| Publication completion | 1,429.1 (1,360.2–1,952.9) | 1,439.5
(897.9–1,649.6) |
| Process CPU during publication | 1,194.1 (1,041.8–1,225.5) | 2,423.1
(1,995.7–2,514.7) |
The treatment served 180–329 requests while publication was active. The
baseline served zero during that interval. Process CPU includes
foreground work that the baseline cannot serve until cleanup completes.
The result supports responsiveness, with a CPU tradeoff. It does not
establish lower total CPU or a hosted latency gain.
Reproduce with Node v24.22.2:
```sh
pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts
node server/scripts/benchmark-native-backup-publication.mjs --baseline-ref
|