mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-31 22:24:16 +01:00
2badf8a4a5a7fb07329d99d355cf0f9875de3b80
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c3a9a8e92a |
perf(server): scope run identity reads to operation needs (#17931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool and connection operations capture the current accepted requester. > - Capture locks the task and run, then reconciles pending steering. > - Most callers use identity or task fields from the returned run. > - The default query also transfers saved prompts, runner configuration, results, and logs. > - This pull request gives each production caller an explicit read scope. > - Narrow scopes keep their authority checks and return only selected columns. > - The Muse project path still reads its complete saved execution input. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operation-time run identity capture for native tools, gateway sessions, project and workspace access, Slack, voice, connection management, and connection token admission. **Current behavior** These callers use the default full run read. Large saved execution documents cross the database connection even when the caller only needs the current accepted requester or task binding. **Proposed behavior** Use typed identity, task, broker, and project-execution scopes. Keep task-before-run locks, company and agent predicates, active status validation, and pending-steering reconciliation. Return the complete saved native input only where project admission uses it. **Reason and benefit** A synthetic PostgreSQL 17.11 fixture matches a wide run with a 142 KB context and a 77 KB runner profile. Its locked run DataRow falls from 225,101 bytes to 190–605 bytes for authority reads. The project scope returns 66,004 bytes because it must retain a complete 64 KB execution input. These are local DataRow measurements, not a forecast of production billing or latency. **Breaking changes** None to public APIs, saved data, or authority policy. Internal narrow return types omit columns that were not selected. The default full compatibility read and the existing GitHub-credential and authentication scopes keep their current contracts. **Additional context** The public issue and PR duplicate searches found no matching open change. This is one independent query optimization. It does not require other performance PRs. ## What Changed - Add typed scopes for accepted identity metadata, task authorization, token attribution, and saved project execution input. - Assign explicit scopes at all production `captureRunIdentity` call sites. - Keep the full native execution input for Muse project admission while omitting unrelated profile, context, result, and log data. - Narrow the broker's initial read and the existing Slack and connection-management historical reads to the same required fields. - Extend steering and acknowledgement regressions to all new scopes. Add real PostgreSQL projection, malformed JSON, tenant and agent, session-generation, and saved-input coverage. - Add a service-generated SQL benchmark with exact raw DataRow accounting, retained-field equality, plans, and alternating transaction timings. - Document the scope contracts and reproducible benchmark. ## Verification - Baseline: `623f1dc289a14a699ebbe3f09b94179d38fc2ae2`. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts`: 54 tests passed on an owned disposable PostgreSQL 17.11 database. - The tests retain task-before-run and audit foreign-key lock behavior. Every scope rejects uncertain steering, accepts its durable acknowledgement, and preserves requester replay ordering. - A full-row mutation makes the narrow identity regression fail. The restored treatment passes. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-identity-projections.ts --baseline-ref=623f1dc289a14a699ebbe3f09b94179d38fc2ae2` captures the real locked SQL for both versions. - Every retained column and selected JSON field matches the baseline. Identity contexts match. The complete stored run remains unchanged. Capture keeps the same five queries. | Scope | Before DataRow bytes | After DataRow bytes | Reduction | | --- | ---: | ---: | ---: | | Identity | 225,101 | 190 | 99.92% | | Task authorization | 225,101 | 410 | 99.82% | | Token broker | 225,101 | 605 | 99.73% | | Project execution input | 225,101 | 66,004 | 70.68% | - DataRow bytes include row framing and raw text-format column values. They exclude response metadata, TLS, and TCP. - Two warmups and seven alternating samples measure the complete capture transaction. Several timing ranges overlap. The primary claim is byte reduction; production latency is not verified. - Initial fixture runs exposed an empty external schema and an incomplete synthetic conversation binding. The test setup now applies migrations and supplies the required user/state fields. The final 54 tests passed. - `pnpm -r typecheck` and `pnpm build` passed locally. - Four existing skill-setup tests passed. Four assembled connection-management/native-tool consumer cases passed with real JWT/API-key authentication. These include per-request identity freshness and task-scoped activity authority. - A CI-only stale mock assertion expected two capture arguments; it now verifies the explicit identity scope. A repository-wide capture mock audit found no other stale argument assertions. - Review follow-up: the projection suite uses the existing embedded-support guard while always enabling an explicitly supplied external database. The audit-lock regression merges URL query parameters and retains existing PostgreSQL options. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts server/src/__tests__/company-skills-routes.test.ts server/src/__tests__/project-skill-setup-identity.test.ts`: 123 tests passed. The external PostgreSQL URL included both `application_name` and `options`; lock tests verified `lock_timeout=1s` and retained `statement_timeout=5s`. - Full local tests are not run because the host's embedded PostgreSQL setup is unavailable. Greptile reviewed `06e1dd53a3fb24df74d5fbfb2ad397f291f1e30f` at 5/5 with zero unresolved threads. The full final-head CI matrix passes. All 58 checks are terminal: 55 success, one neutral, and two optional Storybook skips. - No migration, lockfile, workflow, new polling, or cache change. ## Risks A narrow read must retain each caller's actual fields. The caller audit includes Slack's wake-request fallback, conversation generation, legacy broker attribution, and the Muse parser's complete native input. JSON selection retains value types, including malformed legacy values. The saved-input scope preserves missing-key behavior. The default full read still returns the complete stored row for compatibility. Locks and live authority checks are unchanged. ## Model Used OpenAI GPT-6, accessed through Codex. The exact runtime model ID and context-window size were not exposed in this session. Used repository inspection, code execution, and PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
15176963b1 |
fix: stop idle skill-source polling (#17008)
## Thinking Path > - Paperclip manages agent work and the skills that agents use. > - Project repositories create durable import jobs, and installed sources can refresh automatically. > - The server queried these records every five seconds, even when no work existed. > - Idle tenant databases need to stop receiving those discovery queries. > - This change uses the existing transaction notifications and delivery coordinator. > - Saved refresh, retry, and lease deadlines determine when the worker runs next. > - Cloud does not yet wake these deadlines, so outstanding skill work retains the instance. ## Linked Issues or Issue Description **What existing behavior does this improve?** Skill-source background synchronization. This continues the idle-work migrations in #15625 and #17000. A search found no overlapping open polling-removal PR. **Current behavior** On master `b11010b0d4`, startup installs a five-second skill-source interval and also calls it from the external-object scheduler. Each sweep queries active companies, import capacity, pending imports, and automatic refreshes. **Proposed behavior** Run at startup, after a relevant transaction settles, or at a saved deadline. Stay SQL-quiet when no eligible work remains. Preserve import admission, authorization, atomic claims, retry backoff, and the existing five-minute upstream check. **Reason and benefit** Remove recurring database activity from inactive tenant instances without requiring new tables or a new scheduler. ## What Changed - Replace both recurring skill-sync calls with one delivery-coordinator registration. - Notify from attachment/detachment, source creation/settings/manual refresh/disconnection, project/company deletion, and company status changes, including budget resume. - Restore the earliest eligible `nextSyncAt` or lease-expiry deadline. Due backlogs and conflicts retry after five seconds; no idle recovery cadence remains. - Admit deferred imports again after a batch frees capacity. Preserve requests through drain, standby, restart, and commits during a sweep. - Add sleep-safety predicates for imports and eligible automatic sources until Cloud supports these deadlines. - Document the lifecycle and process-local writer contract. No schema or API changes. - Guard external test database URLs before migrations or cleanup: require a `_test` database name and reject database/options overrides. ## Verification - 147 targeted tests pass: 39 server-startup tests, 14 new real-PostgreSQL lifecycle tests, 20 shared coordinator/scheduler tests, 63 existing import/refresh/retry regression tests, and 11 test-database guard cases. - CI exposed a stale startup mock without the new `start` method. The fixture is updated and all 39 startup tests pass locally. - Tests exercise real enqueue/settings/company-reactivation producers, nested commit, rollback, saved retry/lease restart, manual refresh failure, capacity admission, concurrent commit, shutdown cancellation, and SQL quiet after startup or pause. - Sleep-safety tests reject sleep for future imports and automatic refreshes, and accept inactive retained state. - `pnpm -r typecheck` and `pnpm build` pass. - `pnpm test:run` was attempted and stopped after repeated embedded-PostgreSQL startup failures in unrelated suites. The two existing import/refresh suites initially skipped for the same host limitation; after adding explicit disposable-database options, all 63 regression tests passed against PostgreSQL 17. Full CI passed on `ddb3ee51369baa8c7153bae9fd88efb8b903f2bc`: 55 successful checks, one neutral review check, and two intentional skips. ## Risks - Notifications are process-local. Independent SQL writers and separate live replicas do not wake this worker; startup reconstructs persisted work. - Enabled automatic sources still contact GitHub every five minutes. Their upstream changes have no event ingress here. - Cloud scheduled wakes currently cover routines and issue monitors, not skill sync. This change retains instances with outstanding skill obligations. It does not claim measured host/database suspension or fleet savings. - No change to source authorization, installed content on failure, or existing claim semantics. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The host does not expose an exact model revision or context-window size to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
473c80d13f |
Automatically import and update project repository skills (#16066)
Automatically import skills from new GitHub project repositories and update enabled sources in the background. Preserve project-derived access, saved selections, immutable execution versions and explicit pins; add legacy adoption, refresh and connection repair in Sources. Include durable scheduling and bounded work credits, production Storybook journeys, real GitHub-to-routine acceptance receipts and final-head CI/security/Greptile qualification. Co-Authored-By: Paperclip <noreply@paperclip.ing> |