Files
PaperClipAI/server/src/services
Nicky LeachandPaperclip 313fb265db feat(plugins): invalidate runtimes when connector access changes (#16605)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runtime preparation depends on access and credential state as well
as tool definitions.
> - Grant, membership, secret, and endpoint changes need committed
invalidation.
> - Their writers must acquire locks before journal fanout can lock
agent rows.
> - This pull request adds access hooks with the corresponding
transaction guards.
> - Plugins can refresh preparation immediately while task-time
authorization remains authoritative.

## Linked Issues or Issue Description

**What happened?**

Grant changes, credential rotation and revocation, company access
changes, and connector endpoint configuration do not request runtime
synchronization. Journal fanout exposes lock inversions between
credential persistence, access cleanup, and connection assignment
writers.

**Expected behavior**

Committed access changes request runtime preparation without polling or
credential values in the journal. Agent-specific secret bindings and new
personal delegations affect their target agent. Company access changes
remain conservative. Provider bookkeeping does not create preparation
work.

**Steps to reproduce**

1. Change an audience, delegation, membership permission, referenced
credential, vault configuration, or endpoint instruction.
2. Expect a committed runtime event in the correct company and scope.
3. Run reconnect, credential refresh, SSH setup, or membership cleanup
concurrently with related writes.
4. Expect both operations to preserve access checks and finish without a
lock-order cycle.

**Additional context**

Final feature extraction from #16503. The earlier slices, #16604, and
the separate delivery-test correction #16871 are merged. This PR is now
the active change against master. The rebase preserves the merged
assignment and definition recipient rules, Unicode selector semantics,
and profile/reference transaction guards.

## What Changed

- Add migration 0356 for grants, audiences, delegation,
membership/permissions, secret bindings, referenced credentials and
vaults, and chat/email endpoint configuration.
- Keep events content-free. Ignore health, delivery, publication, Vercel
verification, and GitHub webhook bookkeeping while retaining credential
and repository-access changes.
- Target agent secret bindings and all personal delegation changes to
their old and new target agents. Coalesce delegation changes per
transaction.
- Lock existing grants before connections and credential persistence in
OAuth and cleanup writers.
- Reject superseded refresh leases before rotating stored credentials.
Ignore lease acquisition and release when access metadata is unchanged.
- Serialize audience replacement, validate membership, and preserve
unchanged audiences.
- Lock membership before permission-only saves and grant before Railway
SSH setup. Individual delegation revocation locks its grant and target.
Whole-grant and membership cleanup lock company recipients before
deleting scoped delegations.
- Lock grants and connection inputs before static-key reconnect writes.
Lock company agents before cascading secret deletion; use a nonblocking
secret lock and bounded retries to avoid a new cycle with credential
writers.
- Add PostgreSQL concurrency coverage for access writers and endpoint
publication/configuration behavior.
- Preserve the owner lock for plugin instance-secret removal; these
secrets cannot use ordinary bindings.
- Align Slack search refresh, callbacks, and disconnect with
grant-before-connection locking. Recheck newly discovered grants without
waiting before provider work.
- Retry complete membership transactions when a credential deletion
candidate is busy. Cover all four cleanup entry points and process
company access in ID order.
- Document final invalidation coverage and task-time authorization
boundaries.

## Verification

- Current head: `78aace55bef7f52bc5742b131e0b3961eb5c0f11`. All 1,061
tests across 23 affected suites pass. The named Slack authorization
integration scenario also passes; it exercises authorization, refresh,
disconnect, and rejection of a late callback. Server/database
typechecks, migration safety and numbering, five-snapshot retention,
feature maps, and diff checks pass. The complete PR diff was reviewed
for secrets and personal data before push. Final acceptance: [CI run
39308303837, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/39308303837) is
green, with 55 successful checks and two optional Storybook skips. The
current-head Greptile reassessment returned **Apex 5/5** at 19:16 UTC on
2026-10-19. Its full summary reports no actionable issues. Security
passed, all threads are resolved, and GitHub reports no merge conflicts.
The reassessment was requested as a standard review after the fixes;
Greptile retained the Apex tier.
- On this head, [CI run
39308303837](https://github.com/paperclipai/paperclip/actions/runs/39308303837)
failed the unchanged adapter-utils HTTP/2 fixture with `EADDRINUSE`
before READY. Its closed probe left a port handoff race. That test and
its gateway are byte-identical on master. No upstream fix was found. The
deterministic fixture correction is isolated in #16888; no adapter test
changes are included here. The failed jobs recovered on retry without a
feature source change. The original failure remains documented here, and
#16888 carries the fixture correction separately. That retry
demonstrates recovery; the held-probe negative control establishes the
port handoff race.
- Rebased onto master `194fc11406`, including merged #16604 and #16871.
Replayed only the access-hook commit. Resolved OAuth, test, and
documentation conflicts while retaining both sets of tests. Migration
0356 preserves the merged assignment/definition recipient rules, Unicode
selector semantics, and profile/reference guards.
- Apex returned 3/5 on `0fb65bc27`. Negative controls reproduce its
Slack delegation and member-cleanup deadlocks (`40P01`) and unwanted
Vercel/GitHub bookkeeping events. Slack grant writers now take their
locks before the connection. A separate late-grant control reproduces
`40P01` without the post-lock recheck; the complete guard passes. All
four membership cleanup commands retry from a fresh transaction when the
credential is busy. Tests verify successful cleanup and no journal rows
from failed attempts. Credential IDs and repository-access changes still
notify runtimes.
- Earlier review corrections remain covered: static-key reconnect and
cascading secret removal each reproduce `40P01` without their guards.
Grant lease acquisition/release is ignored. Secret removal releases
agent locks when a credential writer is busy. Instance-secret removal
retains the owner lock and passes its public plugin API contention
regression. Delegation updates and revocations target old/new agents;
unrelated-peer, coalescing, unchanged-update, and rollback cases pass.
Three negative controls verify the required whole-grant and membership
cleanup guards after narrowing delegation scope.
- Preserve prior acceptance failures: the first rebased 20-suite run
passed 1,025/1,026; its only failure was a draft uninstall fixture
blocking a random peer. Blocking the assigned agent corrected that
assumption, and all 44 access/assignment tests passed. Later, all 1,044
tests across 21 suites, [CI run
39302185527](https://github.com/paperclipai/paperclip/actions/runs/39302185527),
standard Greptile 5/5, and security passed on `0fb65bc27`. That did not
clear the later Apex findings described above. Draft run 38975428081
failed a server shard; its cause was not re-investigated. A later green
run does not establish that earlier cause.
- Local fixture corrections are recorded separately from product
failures: an assignment filename filter initially matched no file; the
actual suite was included in full runs. A delegation fixture omitted its
required creator, and a guard run omitted a query import. The new
global-access cleanup fixture now uses its own user to avoid earlier
fixtures' memberships. All were corrected before publishing their final
iterations.
- Local checks target affected services and real PostgreSQL boundaries.
A full local build and repository-wide test run were not performed.
Hosted CI supplies broader verification. Provider responses in tests are
controlled; live provider behavior was not qualified.

## Risks

Migration 0356 adds triggers without changing table schema. Apply it
through the normal migration flow. Most access changes retain
synchronous company-wide invalidation. Company fanout takes O(eligible
company agents) locks and journal rows, without a fanout ceiling or
mutation quota. Agent secret bindings and individual delegation changes
retain narrower recipients. Events contain no credentials.

Physical removal of non-instance secrets takes ordered locks on all
company agents before cascading into bindings. This costs O(company
agents) locks even for an otherwise unreferenced secret. Instance
secrets cannot use ordinary bindings and retain their owner lock. A busy
secret lock releases the transaction guards and retries up to three
times. Continued contention returns a retryable conflict. Provider
deletion stays outside that transaction; a retry resumes removal of an
already disabled secret.

Membership cleanup uses nonblocking credential guards and retries the
full membership transaction. Slack search adds two grant queries, scoped
to the subject and connection. Its retries occur before provider work or
credential persistence. Each retry loop is limited to three attempts.
Invalidation requests preparation; task-time authorization and provider
session activation remain authoritative. Provider calls use fixtures; no
live provider qualification is claimed.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 12:49:11 -07:00
..
…
…
…