Nicky LeachandPaperclip
5f5602bba7
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins prepare agent runtimes outside the control plane. > - They need committed connection assignment changes without polling. > - A reader and its revision hooks must cover the same fields. > - This pull request adds a bounded standing-assignment snapshot and its matching hooks. > - Plugins can react to assignment changes while execution keeps its existing access checks. ## Linked Issues or Issue Description **What happened?** The agent runtime reader does not expose standing connection assignments. Changes to connection installs, active profiles, and connector feature switches do not request synchronization. Adding company-wide invalidation also exposes lock inversions in assignment writers and AI adoption. **Expected behavior** The owning lifecycle plugin can read a bounded assignment snapshot. Every committed change to its stored fields advances the runtime revision. Rollback removes both the change and its event. Assignment writers finish without a lock-order cycle. **Steps to reproduce** 1. Install a connection or bind an active profile to an agent. 2. Read its runtime inputs through the owning lifecycle plugin. 3. Rename or disable the connection, revoke the assignment, or switch connector features. 4. Expect a committed runtime event and an updated snapshot without periodic polling. **Additional context** Extracted from #16503. Catalog publication #16602 and the separate test maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Tool-definition hooks #16604 and access/credential hooks #16605 remain drafts without review requests. ## What Changed - Add SDK types and an owning-plugin runtime snapshot for installed connections, active profile IDs, and effective connector feature switches. - Reject more than 512 connections or 128 active profiles instead of returning partial data. - Add migration 0354 for the exact stored fields used by the snapshot, with transaction-coalesced invalidation. Agent-targeted bindings and installs notify only their old/new target agents; project/routine/issue/gateway bindings are outside the snapshot and produce no notification. - Use ordered agent locks that are compatible with foreign-key references and exclude runtime readers. Prelock complete old/new target sets before replacing assignments or restoring managed profiles. - Record one company baseline at delivery startup for process-managed feature overlays. - Lock assignment inputs before install saves, task completion, and email assignment. Setup finalization reads the locked connection state and prelocks company agents before activating, re-enabling, or reviving a profile. - Make profile binding and reassignment writes transactional and lock parent profiles first. Keep target validation and profile/detail reads on the reserved transaction connection, including with a one-connection pool. - Lock company agents in order before validated AI adoption. Keep adoption and shared configuration writes at `FOR NO KEY UPDATE`, avoiding an upgrade that can deadlock foreign-key inserts during gateway creation. Configuration fields cannot change agent identity; lifecycle transitions and deletion retain their existing locks. - Document supported coverage and the inputs reserved for later PRs. ## Verification - Thirteen affected suites passed: 757 tests covering runtime hooks/assignments/skills, concurrent skill publication, tool access/catalog, task connection intents, lifecycle delivery, email, AI accounts, migration histories, snapshot drift, and the agent lifecycle module. This includes the merged catalog PR's OAuth and overlapping-refresh regressions. - The assignment suite passes all 24 cases, including bounded snapshots, company/agent isolation, rollback, health-field no-ops, process-start feature overlays, and real lock contention. - Four one-connection PostgreSQL regressions reproduce the earlier helper-read hangs before the correction; bind, unbind, delete, and reassign complete with it. The earlier adoption/gateway probe exposed both lock upgrades: changing only validation still deadlocks in shared configuration. Current scoped coverage verifies adoption with agent-targeted binding writes; gateway-only bindings no longer invalidate this snapshot. - The original ordered-adoption regression reproduces PostgreSQL `40P01` without ordered company-agent locking. Scoped-recipient tests reproduce unrelated-peer journaling before the trigger correction and verify repeated bind/unbind operations leave peers untouched, ignored binding types produce no events, both reassignment recipients are reached, and coalescing remains intact. - Six replacement contention cases cover installs, existing setup, activation, re-enabling, profile revival, and profile reassignment. Negative controls reproduce `40P01` without the corresponding target/company prelocks; all cases pass with the guards. Activation decisions use the connection row reread under lock. - An initial contention fixture failed because its settings connection lacked an MCP URL. Supplying the fixture URL and controlled lookup corrected that test setup error. - Server typecheck, database migration numbering/safety, five-snapshot retention, feature-map structural validation, and `git diff --check` passed. Migration metadata preserves historical journal entries and adds no schema change. The complete diff was reviewed for secrets and personal data before push. - Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the superseded promotion run 38983440470 was cancelled. Superagent acknowledged the recipient fix and accepted the documented company-wide cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard Greptile summaries were conditional on a later setup correction and were not counted as independent acceptance; the setup correction is now included here. - At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run 38986958988 passed: all 55 checks succeeded, with two optional Storybook checks skipped. Superagent passed, all review threads are resolved, and standard Greptile independently returned 5/5 with no blocking findings (review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5 with no actionable findings (completed 2026-10-17 00:50 UTC). CI was rechecked after Apex and remains green; the branch is mergeable against master. Full local build and repository-wide tests were not run. ## Risks Migration 0354 adds database functions and triggers without changing table schema. Apply it through the normal migration flow. Agent-targeted bindings and installs only notify their affected agents. Explicit company-wide assignments, connection metadata, profile activation/removal, and connector feature switches retain synchronous, transaction-atomic company fanout. Each such transaction takes ordered row locks and inserts up to one connector runtime event per eligible company agent: O(number of eligible agents) work and journal storage, with no company-size cap or cross-transaction mutation quota. This remains an intentional scope tradeoff for this PR; deferred fanout is not implemented. Many of these conservative changes can request preparation even if an individual snapshot stays unchanged. Snapshot limits fail closed. This PR provides desired assignments, not credentials or task authorization. Later PRs must add mutation coverage before consumers can rely on immediate synchronization of tool definitions or credentials. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
8.6 KiB
Agent runtime synchronization
Plugins can opt into committed agent runtime changes with
resourceLifecycle: ["agent_runtime"] and events.subscribe.
Implement onResourceLifecycle and handle update and delete.
Existing agent and project subscriptions do not receive this new resource type.
Manual inbox reads retain both legacy streams regardless of automatic subscriptions.
They include runtime events only when the manifest opts into agent_runtime.
An event contains company identity, agent identity in resourceId, a numeric
sequence serialized as id, and its action. It contains no file contents,
configuration, or credentials. Treat id as an opaque decimal revision, not a
JavaScript floating-point value. Delivery is at least once and ordered per agent.
Return acknowledged only after external effects finish. Return retry after an
unfinished operation, or deferred when configuration is missing. Startup,
plugin startup, and configuration changes recover unacknowledged work.
The host records configuration changes at the database mutation boundary. This covers API saves, rollback, plugin writes, and background writers. No-op updates and heartbeat accounting do not create runtime work. Instruction edits, file deletions, restores, and run writeback use the file audit boundary. Pending hires produce no update until approval. Returning to the ready lifecycle phase requests synchronization after preparation or resume. Existing agents receive a one-time baseline when the migration runs. Agent deletion leaves a durable delete event.
PostgreSQL notifications wake the existing durable dispatcher only after commit,
including writes through independent clients. Notifications carry no identifiers
or contents. The journal remains authoritative. The listener scans on connection
and reconnection so a dropped notification does not lose committed work. There
is no recurring empty-inbox poll. Existing retry deadlines still retry failed work.
A process opens one listener connection only after a ready plugin opts into
agent_runtime. The database connection must support session-level LISTEN;
transaction-mode poolers are not supported for this opt-in feature. Startup waits
for listener registration. It releases the listener after the last runtime
subscriber is disabled or removed. Legacy subscriptions keep process-local work signals.
A plugin that also has agents.lifecycle.manage can call
ctx.agents.getRuntime(agentId, companyId) from a company-scoped invocation.
Only the enabled, ready plugin selected in that agent's required lifecycle set
can read these inputs. The result supplies the latest runtime revision, the
normalized agent configuration, instruction root and entry filename, and that
plugin's own instance secrets. Instruction paths use the same managed-folder
recovery as the instruction reader. A missing or cross-company agent returns null.
The instruction root identifies host files that the trusted worker can package
through its existing filesystem access. This API grants no arbitrary path reads.
Never expose instance secrets through UI data, logs, plugin state, or agent files.
Re-read current state before external effects. A queued event can be older than the returned revision. Fence stale revisions at the destination and use its conditional writes. Current configuration is desired state, not a frozen run snapshot or permission to change an active provider session. Execution admission and provider activation keep their existing checks. The private transport consumer is a separate integration.
Filesystem changes outside Paperclip's managed write APIs do not emit these hooks. Filesystem publication and the database journal are not one filesystem transaction. A failed managed write must be repaired through the same API before claiming synchronization. Upstream repository detection remains owned by the source integration.
Successful file saves record the current desired state. Identical retries and
already-applied deletions repair a missing journal event if a prior save published
bytes before its database transaction failed. Audit rows carry their runtime
revision, so repeated identical saves against that revision create no audit rows
or events. Later runtime changes invalidate that deduplication. The file API still
returns changed: false for identical content and keeps conditional-write checks.
Consumers can skip rewriting content whose hash already matches.
Skills
Migration 0348 invalidates referencing agents when installed skill content, companion files, versions, source bindings, or identities change. Rename checks both old and new keys. String selections, structured selections, and supported legacy aliases are recognized. Unrelated agents and other companies receive no content invalidation. Private-source bindings, membership, and privacy changes invalidate only agents assigned skills from the affected sources. Project workspace authorization-policy changes use the same boundary. Company policy changes conservatively invalidate the company. Source retry/lease bookkeeping and skill popularity counters do not emit changes. Multi-skill source publication defers its skill invalidation until transaction commit, after it acquires all skill rows. This avoids an agent-lock/skill-lock cycle with concurrent version saves. Individual file edits retain immediate journaling and their existing filesystem rollback behavior.
Assignment edits already use the configuration boundary. The runtime reader returns selected skill entries with host source paths and immutable version IDs. It uses the execution resolver, current agent access, the instance pin flag, and legacy operational-skill rules. It performs no upstream refresh. The resolver can rebuild an expired version cache from saved bytes. Slow materialization happens outside the agent row lock. The reader rechecks ownership and revision afterward; it retries concurrent changes three times and then returns a conflict.
Pinned assignments receive conservative content invalidations too. Their resolved version and bytes stay pinned when pin support is enabled. Consumers can compare versions or destination hashes and avoid rewriting unchanged trees. Task-only skill mentions and a running task's saved version selection remain task-owned.
Standing connection assignments
getRuntime also returns connectionAssignments: effective publicMcp and
chatConnectors feature switches, installed connections with id, name,
transport, enabled, and status, and active profile IDs. It includes both
company assignments and assignments for the requested agent. It excludes other
agents' assignments and has no credential values, grant details, or tool schemas.
The owning-plugin checks for the runtime reader apply to this field too.
Migration 0354 records committed changes to every stored field used by this snapshot. Agent-targeted installation and profile binding edits invalidate only their old and new target agents. Company-targeted assignments invalidate company agent runtimes. Project, routine, issue, and gateway bindings are outside this snapshot and do not invalidate it. Connection metadata, profile activation or removal, and connector feature switches still use conservative company-wide invalidation. Empty writes and changes to health bookkeeping do not. Multiple assignment statements in one transaction coalesce to one update per agent. Journal writes and their triggering changes commit or roll back together. Company-wide invalidation remains synchronous: it locks eligible company agents in order and journals one update per agent. Its work grows with company size; there is no fanout ceiling or cross-transaction mutation quota.
The reader orders and deduplicates assignments. It rejects more than 512 connections or 128 active profiles with status 422 instead of returning a partial snapshot. It rechecks the revision after reading assignments and skills, with the same three-attempt conflict behavior described above.
Each delivery-service startup records one company baseline when an enabled runtime subscriber becomes active. This covers feature overlays supplied at process startup. It does not add a recurring scan. Plugins must compare desired state and destination hashes before rewriting external files.
This assignment contract does not yet emit connector-specific invalidations for tool catalog contents, profile entries, policies, connection configuration, grants, credentials, or endpoint configuration. Those inputs need their own mutation coverage before a consumer can rely on their immediate synchronization. Existing agent and skill invalidations still apply independently. Standing assignments express desired state; they do not prove task-specific authorization.