2 Commits

Include renames

Author	SHA1	Message	Date
Nicky LeachandPaperclip	313fb265db	feat(plugins): invalidate runtimes when connector access changes (#16605) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runtime preparation depends on access and credential state as well as tool definitions. > - Grant, membership, secret, and endpoint changes need committed invalidation. > - Their writers must acquire locks before journal fanout can lock agent rows. > - This pull request adds access hooks with the corresponding transaction guards. > - Plugins can refresh preparation immediately while task-time authorization remains authoritative. ## Linked Issues or Issue Description **What happened?** Grant changes, credential rotation and revocation, company access changes, and connector endpoint configuration do not request runtime synchronization. Journal fanout exposes lock inversions between credential persistence, access cleanup, and connection assignment writers. **Expected behavior** Committed access changes request runtime preparation without polling or credential values in the journal. Agent-specific secret bindings and new personal delegations affect their target agent. Company access changes remain conservative. Provider bookkeeping does not create preparation work. **Steps to reproduce** 1. Change an audience, delegation, membership permission, referenced credential, vault configuration, or endpoint instruction. 2. Expect a committed runtime event in the correct company and scope. 3. Run reconnect, credential refresh, SSH setup, or membership cleanup concurrently with related writes. 4. Expect both operations to preserve access checks and finish without a lock-order cycle. **Additional context** Final feature extraction from #16503. The earlier slices, #16604, and the separate delivery-test correction #16871 are merged. This PR is now the active change against master. The rebase preserves the merged assignment and definition recipient rules, Unicode selector semantics, and profile/reference transaction guards. ## What Changed - Add migration 0356 for grants, audiences, delegation, membership/permissions, secret bindings, referenced credentials and vaults, and chat/email endpoint configuration. - Keep events content-free. Ignore health, delivery, publication, Vercel verification, and GitHub webhook bookkeeping while retaining credential and repository-access changes. - Target agent secret bindings and all personal delegation changes to their old and new target agents. Coalesce delegation changes per transaction. - Lock existing grants before connections and credential persistence in OAuth and cleanup writers. - Reject superseded refresh leases before rotating stored credentials. Ignore lease acquisition and release when access metadata is unchanged. - Serialize audience replacement, validate membership, and preserve unchanged audiences. - Lock membership before permission-only saves and grant before Railway SSH setup. Individual delegation revocation locks its grant and target. Whole-grant and membership cleanup lock company recipients before deleting scoped delegations. - Lock grants and connection inputs before static-key reconnect writes. Lock company agents before cascading secret deletion; use a nonblocking secret lock and bounded retries to avoid a new cycle with credential writers. - Add PostgreSQL concurrency coverage for access writers and endpoint publication/configuration behavior. - Preserve the owner lock for plugin instance-secret removal; these secrets cannot use ordinary bindings. - Align Slack search refresh, callbacks, and disconnect with grant-before-connection locking. Recheck newly discovered grants without waiting before provider work. - Retry complete membership transactions when a credential deletion candidate is busy. Cover all four cleanup entry points and process company access in ID order. - Document final invalidation coverage and task-time authorization boundaries. ## Verification - Current head: `78aace55bef7f52bc5742b131e0b3961eb5c0f11`. All 1,061 tests across 23 affected suites pass. The named Slack authorization integration scenario also passes; it exercises authorization, refresh, disconnect, and rejection of a late callback. Server/database typechecks, migration safety and numbering, five-snapshot retention, feature maps, and diff checks pass. The complete PR diff was reviewed for secrets and personal data before push. Final acceptance: [CI run 39308303837, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/39308303837) is green, with 55 successful checks and two optional Storybook skips. The current-head Greptile reassessment returned **Apex 5/5** at 19:16 UTC on 2026-10-19. Its full summary reports no actionable issues. Security passed, all threads are resolved, and GitHub reports no merge conflicts. The reassessment was requested as a standard review after the fixes; Greptile retained the Apex tier. - On this head, [CI run 39308303837](https://github.com/paperclipai/paperclip/actions/runs/39308303837) failed the unchanged adapter-utils HTTP/2 fixture with `EADDRINUSE` before READY. Its closed probe left a port handoff race. That test and its gateway are byte-identical on master. No upstream fix was found. The deterministic fixture correction is isolated in #16888; no adapter test changes are included here. The failed jobs recovered on retry without a feature source change. The original failure remains documented here, and #16888 carries the fixture correction separately. That retry demonstrates recovery; the held-probe negative control establishes the port handoff race. - Rebased onto master `194fc11406`, including merged #16604 and #16871. Replayed only the access-hook commit. Resolved OAuth, test, and documentation conflicts while retaining both sets of tests. Migration 0356 preserves the merged assignment/definition recipient rules, Unicode selector semantics, and profile/reference guards. - Apex returned 3/5 on `0fb65bc27`. Negative controls reproduce its Slack delegation and member-cleanup deadlocks (`40P01`) and unwanted Vercel/GitHub bookkeeping events. Slack grant writers now take their locks before the connection. A separate late-grant control reproduces `40P01` without the post-lock recheck; the complete guard passes. All four membership cleanup commands retry from a fresh transaction when the credential is busy. Tests verify successful cleanup and no journal rows from failed attempts. Credential IDs and repository-access changes still notify runtimes. - Earlier review corrections remain covered: static-key reconnect and cascading secret removal each reproduce `40P01` without their guards. Grant lease acquisition/release is ignored. Secret removal releases agent locks when a credential writer is busy. Instance-secret removal retains the owner lock and passes its public plugin API contention regression. Delegation updates and revocations target old/new agents; unrelated-peer, coalescing, unchanged-update, and rollback cases pass. Three negative controls verify the required whole-grant and membership cleanup guards after narrowing delegation scope. - Preserve prior acceptance failures: the first rebased 20-suite run passed 1,025/1,026; its only failure was a draft uninstall fixture blocking a random peer. Blocking the assigned agent corrected that assumption, and all 44 access/assignment tests passed. Later, all 1,044 tests across 21 suites, [CI run 39302185527](https://github.com/paperclipai/paperclip/actions/runs/39302185527), standard Greptile 5/5, and security passed on `0fb65bc27`. That did not clear the later Apex findings described above. Draft run 38975428081 failed a server shard; its cause was not re-investigated. A later green run does not establish that earlier cause. - Local fixture corrections are recorded separately from product failures: an assignment filename filter initially matched no file; the actual suite was included in full runs. A delegation fixture omitted its required creator, and a guard run omitted a query import. The new global-access cleanup fixture now uses its own user to avoid earlier fixtures' memberships. All were corrected before publishing their final iterations. - Local checks target affected services and real PostgreSQL boundaries. A full local build and repository-wide test run were not performed. Hosted CI supplies broader verification. Provider responses in tests are controlled; live provider behavior was not qualified. ## Risks Migration 0356 adds triggers without changing table schema. Apply it through the normal migration flow. Most access changes retain synchronous company-wide invalidation. Company fanout takes O(eligible company agents) locks and journal rows, without a fanout ceiling or mutation quota. Agent secret bindings and individual delegation changes retain narrower recipients. Events contain no credentials. Physical removal of non-instance secrets takes ordered locks on all company agents before cascading into bindings. This costs O(company agents) locks even for an otherwise unreferenced secret. Instance secrets cannot use ordinary bindings and retain their owner lock. A busy secret lock releases the transaction guards and retries up to three times. Continued contention returns a retryable conflict. Provider deletion stays outside that transaction; a retry resumes removal of an already disabled secret. Membership cleanup uses nonblocking credential guards and retries the full membership transaction. Slack search adds two grant queries, scoped to the subject and connection. Its retries occur before provider work or credential persistence. Each retry loop is limited to three attempts. Invalidation requests preparation; task-time authorization and provider session activation remain authoritative. Provider calls use fixtures; no live provider qualification is claimed. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 12:49:11 -07:00
Nicky LeachandPaperclip	da9af67415	Persist agent-owned instance secrets through lifecycle results (#16191) ... ## Thinking Path > - Paperclip coordinates agents and controls when they can run. > - Lifecycle plugins prepare and remove resources for each agent. > - Those resources can require private connection credentials. > - The current SDK can resolve configured secrets but cannot save these credentials securely. > - Ordinary company secrets can be bound to agents and do not provide the required server-only boundary. > - This change saves agent-owned instance secrets through the existing local lifecycle result and native secret service. > - The tenant can retain runtime credentials through restarts without a separate encryption system or remote API. ## Linked Issues or Issue Description **Subsystem affected** Agent lifecycle, plugin contract, and native secret storage. **Problem or motivation** Lifecycle plugins need to save private runtime connection details for the owning agent. Worker memory loses these details on restart. Plugin state and company secrets do not provide a private runtime-only boundary. **Proposed solution** Add optional instance-secret changes to lifecycle results. Save them through the existing secret service and local encrypted provider before accepting the result. Require company, agent, plugin, phase, operation, revision, and current unexpired attempt ownership. Supply saved values through the existing local preparation/resume request. **Alternatives considered** A separate encrypted store duplicates the native secret system. Generic plugin company-secret writes proposed by #5007 do not supply agent ownership or a server-only secret scope. This change does not add that general write API. **Roadmap alignment** Extends the existing lifecycle contract from #15631. It does not add a resource provider or select an execution backend. ## What Changed - Add an instance scope to native secrets with mandatory agent/plugin ownership, a company-qualified agent foreign key, and key uniqueness per owner. Existing company and user scopes retain their constraints. - Reuse native creation, rotation, deletion, secret versions, access audit, and the local encrypted provider. Keep values out of company catalogs, normal bindings, agent access, and public agent records. - Accept local secret patches only after locking and checking the current lifecycle operation and required plugin selection. Storage failure or a stale reply cannot complete setup. Expired or replaced attempt owners cannot overwrite a successor's credentials. - Keep vault-import conflicts company-scoped. Protect instance-owned account-home paths during failed device-login cleanup with a private, fail-closed scan. - Skip decryption on stop and cleanup so a lost key cannot block removal. Cascade secret records when their agent or plugin is deleted. - Log fixed provider diagnostics without raw errors that can contain credentials. Document the local contract, runtime resolver, and access-map exclusions. Retain the five latest migration snapshots through the normal generator pruning. ## Verification - 239 focused tests passed across agent-instance-secrets, lifecycle, secrets-service, secrets-routes, plugin-secrets-handler, and agent-device-login-routes. Used disposable PostgreSQL 18 clusters through the existing test constructor seam because the bundled binary cannot start on this macOS host. The temporary harness is outside git. - Regression tests reproduced the vault-import conflict and missed cleanup claimant before the fixes; both pass after the fixes. - New coverage includes encrypted native versions, rotation, restart recovery, company/agent/plugin isolation, ordinary-read and binding denial, stale replies, storage failure, safe logging, cleanup without the master key, and same-name secret isolation, native account-home mutation guards, lease takeover/expiry, and live agent/plugin deletion cascades. - Direct server TypeScript check, DB/shared typechecks, migration numbering/safety checks, module boundaries, and git diff check passed. - Full workspace typecheck and build were attempted; both stop at the missing local cargo toolchain. The full test command was started and stopped after failures in unchanged native-runtime/workspace suites plus bundled-PostgreSQL skips; it did not complete. CI must verify the full toolchain paths. - Current head `c0f8bfd12477ad509252ecd40e168a6c84dbeb36`: all 55 CI checks passed, including the previously canceled e2e shard. Greptile Apex reviewed this exact head at 5/5 with no unresolved threads. GitHub reports the branch mergeable and clean. - No live provider probe, production data access, or deployment. Reviewed the diff for credentials, private references, and run artifacts. ## Risks Apply migration 0339_luxuriant_iron_monger.sql before deploying tenant code. It adds nullable ownership columns, a partial unique index, foreign keys, and a stricter scope constraint. Existing rows remain valid. Keep the schema on application rollback. Preserve the existing secrets master key through deployments and backup restoration. The lifecycle request/result now has a sensitive, server-only field. Plugins must never log it or return it to an agent or browser. Values stay in the tenant database through the existing encryption provider. Trusted runtime consumers must apply normal execution admission before resolving them. Older hosts omit instanceSecrets; plugins that require durable storage must reject those hosts. ## Model Used OpenAI GPT-6 via Codex, with reasoning, repository inspection, code editing, and local test execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-13 10:13:42 -07:00