mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 08:10:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins prepare agent runtimes outside the control plane. > - They need committed connection assignment changes without polling. > - A reader and its revision hooks must cover the same fields. > - This pull request adds a bounded standing-assignment snapshot and its matching hooks. > - Plugins can react to assignment changes while execution keeps its existing access checks. ## Linked Issues or Issue Description **What happened?** The agent runtime reader does not expose standing connection assignments. Changes to connection installs, active profiles, and connector feature switches do not request synchronization. Adding company-wide invalidation also exposes lock inversions in assignment writers and AI adoption. **Expected behavior** The owning lifecycle plugin can read a bounded assignment snapshot. Every committed change to its stored fields advances the runtime revision. Rollback removes both the change and its event. Assignment writers finish without a lock-order cycle. **Steps to reproduce** 1. Install a connection or bind an active profile to an agent. 2. Read its runtime inputs through the owning lifecycle plugin. 3. Rename or disable the connection, revoke the assignment, or switch connector features. 4. Expect a committed runtime event and an updated snapshot without periodic polling. **Additional context** Extracted from #16503. Catalog publication #16602 and the separate test maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Tool-definition hooks #16604 and access/credential hooks #16605 remain drafts without review requests. ## What Changed - Add SDK types and an owning-plugin runtime snapshot for installed connections, active profile IDs, and effective connector feature switches. - Reject more than 512 connections or 128 active profiles instead of returning partial data. - Add migration 0354 for the exact stored fields used by the snapshot, with transaction-coalesced invalidation. Agent-targeted bindings and installs notify only their old/new target agents; project/routine/issue/gateway bindings are outside the snapshot and produce no notification. - Use ordered agent locks that are compatible with foreign-key references and exclude runtime readers. Prelock complete old/new target sets before replacing assignments or restoring managed profiles. - Record one company baseline at delivery startup for process-managed feature overlays. - Lock assignment inputs before install saves, task completion, and email assignment. Setup finalization reads the locked connection state and prelocks company agents before activating, re-enabling, or reviving a profile. - Make profile binding and reassignment writes transactional and lock parent profiles first. Keep target validation and profile/detail reads on the reserved transaction connection, including with a one-connection pool. - Lock company agents in order before validated AI adoption. Keep adoption and shared configuration writes at `FOR NO KEY UPDATE`, avoiding an upgrade that can deadlock foreign-key inserts during gateway creation. Configuration fields cannot change agent identity; lifecycle transitions and deletion retain their existing locks. - Document supported coverage and the inputs reserved for later PRs. ## Verification - Thirteen affected suites passed: 757 tests covering runtime hooks/assignments/skills, concurrent skill publication, tool access/catalog, task connection intents, lifecycle delivery, email, AI accounts, migration histories, snapshot drift, and the agent lifecycle module. This includes the merged catalog PR's OAuth and overlapping-refresh regressions. - The assignment suite passes all 24 cases, including bounded snapshots, company/agent isolation, rollback, health-field no-ops, process-start feature overlays, and real lock contention. - Four one-connection PostgreSQL regressions reproduce the earlier helper-read hangs before the correction; bind, unbind, delete, and reassign complete with it. The earlier adoption/gateway probe exposed both lock upgrades: changing only validation still deadlocks in shared configuration. Current scoped coverage verifies adoption with agent-targeted binding writes; gateway-only bindings no longer invalidate this snapshot. - The original ordered-adoption regression reproduces PostgreSQL `40P01` without ordered company-agent locking. Scoped-recipient tests reproduce unrelated-peer journaling before the trigger correction and verify repeated bind/unbind operations leave peers untouched, ignored binding types produce no events, both reassignment recipients are reached, and coalescing remains intact. - Six replacement contention cases cover installs, existing setup, activation, re-enabling, profile revival, and profile reassignment. Negative controls reproduce `40P01` without the corresponding target/company prelocks; all cases pass with the guards. Activation decisions use the connection row reread under lock. - An initial contention fixture failed because its settings connection lacked an MCP URL. Supplying the fixture URL and controlled lookup corrected that test setup error. - Server typecheck, database migration numbering/safety, five-snapshot retention, feature-map structural validation, and `git diff --check` passed. Migration metadata preserves historical journal entries and adds no schema change. The complete diff was reviewed for secrets and personal data before push. - Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the superseded promotion run 38983440470 was cancelled. Superagent acknowledged the recipient fix and accepted the documented company-wide cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard Greptile summaries were conditional on a later setup correction and were not counted as independent acceptance; the setup correction is now included here. - At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run 38986958988 passed: all 55 checks succeeded, with two optional Storybook checks skipped. Superagent passed, all review threads are resolved, and standard Greptile independently returned 5/5 with no blocking findings (review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5 with no actionable findings (completed 2026-10-17 00:50 UTC). CI was rechecked after Apex and remains green; the branch is mergeable against master. Full local build and repository-wide tests were not run. ## Risks Migration 0354 adds database functions and triggers without changing table schema. Apply it through the normal migration flow. Agent-targeted bindings and installs only notify their affected agents. Explicit company-wide assignments, connection metadata, profile activation/removal, and connector feature switches retain synchronous, transaction-atomic company fanout. Each such transaction takes ordered row locks and inserts up to one connector runtime event per eligible company agent: O(number of eligible agents) work and journal storage, with no company-size cap or cross-transaction mutation quota. This remains an intentional scope tradeoff for this PR; deferred fanout is not implemented. Many of these conservative changes can request preparation even if an individual snapshot stays unchanged. Snapshot limits fail closed. This PR provides desired assignments, not credentials or task authorization. Later PRs must add mutation coverage before consumers can rely on immediate synchronization of tool definitions or credentials. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
185 lines
11 KiB
PL/PgSQL
185 lines
11 KiB
PL/PgSQL
CREATE OR REPLACE FUNCTION paperclip_record_agent_runtime(p_company uuid, p_agent uuid) RETURNS bigint
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE revision bigint;
|
|
BEGIN
|
|
PERFORM 1 FROM agents WHERE company_id = p_company AND id = p_agent
|
|
AND status NOT IN ('pending_approval', 'terminated') FOR NO KEY UPDATE;
|
|
IF FOUND THEN
|
|
INSERT INTO resource_lifecycle_events(company_id, resource_type, resource_id, action)
|
|
VALUES (p_company, 'agent_runtime', p_agent, 'update') RETURNING id INTO revision;
|
|
END IF;
|
|
RETURN revision;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_invalidate_company_agent_runtimes(p_company uuid) RETURNS void
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE candidate uuid;
|
|
BEGIN
|
|
FOR candidate IN SELECT id FROM agents WHERE company_id = p_company
|
|
AND status NOT IN ('pending_approval', 'terminated') ORDER BY id FOR NO KEY UPDATE LOOP
|
|
PERFORM paperclip_record_agent_runtime(p_company, candidate);
|
|
END LOOP;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_invalidate_skill_source_runtimes(p_company uuid, sources uuid[]) RETURNS void
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE candidate uuid;
|
|
BEGIN
|
|
FOR candidate IN SELECT a.id FROM agents a WHERE a.company_id = p_company
|
|
AND a.status NOT IN ('pending_approval', 'terminated') AND EXISTS (
|
|
SELECT 1 FROM company_skills s WHERE s.company_id = p_company AND s.access_source_id = ANY(sources)
|
|
AND paperclip_agent_selects_runtime_skill(a.adapter_config, a.adapter_type, s.key, s.slug))
|
|
ORDER BY a.id FOR NO KEY UPDATE LOOP
|
|
PERFORM paperclip_record_agent_runtime(p_company, candidate);
|
|
END LOOP;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_skill_runtime_changed() RETURNS trigger
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE previous jsonb; current jsonb; keys text[]; company uuid; candidate uuid;
|
|
BEGIN
|
|
IF TG_OP <> 'INSERT' THEN previous := to_jsonb(OLD); END IF;
|
|
IF TG_OP <> 'DELETE' THEN current := to_jsonb(NEW); END IF;
|
|
IF TG_OP = 'UPDATE' AND
|
|
jsonb_build_array(OLD.key, OLD.slug, OLD.markdown, OLD.source_type, OLD.source_locator, OLD.source_ref,
|
|
OLD.file_inventory, OLD.current_version_id, OLD.access_source_id, OLD.compatibility, OLD.trust_level, OLD.metadata)
|
|
IS NOT DISTINCT FROM
|
|
jsonb_build_array(NEW.key, NEW.slug, NEW.markdown, NEW.source_type, NEW.source_locator, NEW.source_ref,
|
|
NEW.file_inventory, NEW.current_version_id, NEW.access_source_id, NEW.compatibility, NEW.trust_level, NEW.metadata)
|
|
THEN RETURN NEW; END IF;
|
|
company := coalesce((current->>'company_id')::uuid, (previous->>'company_id')::uuid);
|
|
SELECT array_agg(DISTINCT lower(reference)) INTO keys FROM (
|
|
SELECT row->>'key' AS reference FROM unnest(ARRAY[previous, current]) row
|
|
UNION ALL SELECT regexp_replace(row->>'key', '.*/', '') FROM unnest(ARRAY[previous, current]) row
|
|
UNION ALL SELECT CASE WHEN row->>'key' LIKE 'paperclipai/paperclip/%' THEN row->>'slug'
|
|
ELSE (row->>'slug') || '--' || left(encode(sha256(convert_to(row->>'key', 'UTF8')), 'hex'), 10) END
|
|
FROM unnest(ARRAY[previous, current]) row
|
|
) refs WHERE reference IS NOT NULL;
|
|
FOR candidate IN SELECT a.id FROM agents a WHERE a.company_id = company
|
|
AND a.status NOT IN ('pending_approval', 'terminated') AND (
|
|
(a.adapter_type <> 'paperclip_runner' AND keys && ARRAY['paperclipai/paperclip/paperclip', 'paperclipai/paperclip/complain', 'paperclipai/paperclip/suggestion-box']) OR
|
|
EXISTS (SELECT 1 FROM jsonb_array_elements(CASE WHEN jsonb_typeof(a.adapter_config#>'{paperclipSkillSync,desiredSkills}') = 'array'
|
|
THEN a.adapter_config#>'{paperclipSkillSync,desiredSkills}' ELSE '[]'::jsonb END) choice
|
|
WHERE lower(trim(CASE WHEN jsonb_typeof(choice) = 'string' THEN choice#>>'{}' ELSE choice->>'key' END)) = ANY(keys)))
|
|
ORDER BY a.id FOR NO KEY UPDATE LOOP
|
|
PERFORM paperclip_record_agent_runtime(company, candidate);
|
|
END LOOP;
|
|
IF TG_OP = 'DELETE' THEN RETURN OLD; END IF;
|
|
RETURN NEW;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_invalidate_connector_agent(p_company uuid, p_agent uuid) RETURNS void
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE setting text := 'paperclip.runtime_agent_' || replace(p_agent::text, '-', '_'); xid text := pg_current_xact_id()::text;
|
|
BEGIN
|
|
IF current_setting('paperclip.runtime_' || replace(p_company::text, '-', '_'), true) IS NOT DISTINCT FROM xid
|
|
OR current_setting(setting, true) IS NOT DISTINCT FROM xid THEN RETURN; END IF;
|
|
PERFORM paperclip_record_agent_runtime(p_company, p_agent);
|
|
PERFORM set_config(setting, xid, true);
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_invalidate_connector_company(p_company uuid) RETURNS void
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE setting text := 'paperclip.runtime_' || replace(p_company::text, '-', '_'); xid text := pg_current_xact_id()::text; candidate uuid;
|
|
BEGIN
|
|
IF current_setting(setting, true) IS NOT DISTINCT FROM xid THEN RETURN; END IF;
|
|
FOR candidate IN SELECT id FROM agents WHERE company_id = p_company
|
|
AND status NOT IN ('pending_approval', 'terminated') ORDER BY id FOR NO KEY UPDATE LOOP
|
|
PERFORM paperclip_invalidate_connector_agent(p_company, candidate);
|
|
END LOOP;
|
|
PERFORM set_config(setting, xid, true);
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_connector_runtime_changed() RETURNS trigger
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE company uuid; query text; old_projection text; new_projection text; target record; candidate uuid;
|
|
BEGIN
|
|
IF TG_TABLE_NAME = 'tool_profiles' AND TG_OP = 'INSERT' THEN RETURN NULL; END IF;
|
|
IF TG_OP = 'UPDATE' THEN
|
|
SELECT 'jsonb_build_array(' || string_agg(format('to_jsonb(o)->%L', field), ',') || ')',
|
|
'jsonb_build_array(' || string_agg(format('to_jsonb(n)->%L', field), ',') || ')'
|
|
INTO old_projection, new_projection FROM unnest(TG_ARGV[1:TG_NARGS - 1]) field;
|
|
END IF;
|
|
IF TG_TABLE_NAME IN ('tool_profile_bindings', 'tool_connection_installs') THEN
|
|
IF TG_OP = 'INSERT' THEN query := 'SELECT company_id, target_type, target_id FROM new_rows';
|
|
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT company_id, target_type, target_id FROM old_rows';
|
|
ELSE query := format('WITH changed AS (
|
|
SELECT o.company_id AS old_company, o.target_type AS old_type, o.target_id AS old_target,
|
|
n.company_id AS new_company, n.target_type AS new_type, n.target_id AS new_target
|
|
FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I WHERE %s IS DISTINCT FROM %s)
|
|
SELECT old_company AS company_id, old_type AS target_type, old_target AS target_id FROM changed WHERE old_company IS NOT NULL UNION
|
|
SELECT new_company AS company_id, new_type AS target_type, new_target AS target_id FROM changed WHERE new_company IS NOT NULL',
|
|
TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
|
|
END IF;
|
|
FOR target IN EXECUTE 'SELECT DISTINCT company_id, target_type, target_id FROM (' || query || ') targets
|
|
WHERE target_type IN (''agent'', ''company'')
|
|
ORDER BY company_id, target_type DESC, target_id' LOOP
|
|
IF target.target_type = 'company' AND target.target_id = target.company_id::text THEN
|
|
PERFORM paperclip_invalidate_connector_company(target.company_id);
|
|
ELSIF target.target_type = 'agent' AND target.target_id ~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' THEN
|
|
SELECT id INTO candidate FROM agents WHERE company_id = target.company_id AND id = target.target_id::uuid;
|
|
IF FOUND THEN PERFORM paperclip_invalidate_connector_agent(target.company_id, candidate); END IF;
|
|
END IF;
|
|
END LOOP;
|
|
RETURN NULL;
|
|
END IF;
|
|
IF TG_OP = 'INSERT' THEN query := 'SELECT DISTINCT company_id FROM new_rows ORDER BY company_id';
|
|
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT DISTINCT company_id FROM old_rows ORDER BY company_id';
|
|
ELSE
|
|
query := format('WITH changed AS (
|
|
SELECT o.company_id AS old_company, n.company_id AS new_company FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I
|
|
WHERE %s IS DISTINCT FROM %s)
|
|
SELECT old_company AS company_id FROM changed WHERE old_company IS NOT NULL UNION
|
|
SELECT new_company AS company_id FROM changed WHERE new_company IS NOT NULL ORDER BY company_id', TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
|
|
END IF;
|
|
FOR company IN EXECUTE query LOOP
|
|
PERFORM paperclip_invalidate_connector_company(company);
|
|
END LOOP;
|
|
RETURN NULL;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
DO $$
|
|
DECLARE relation text; arguments text;
|
|
BEGIN
|
|
FOREACH relation IN ARRAY ARRAY['tool_connections', 'tool_connection_installs', 'tool_profiles', 'tool_profile_bindings'] LOOP
|
|
arguments := CASE relation
|
|
WHEN 'tool_connections' THEN '''id'', ''company_id'', ''name'', ''transport'', ''enabled'', ''status'''
|
|
WHEN 'tool_connection_installs' THEN '''id'', ''company_id'', ''connection_id'', ''target_type'', ''target_id'''
|
|
WHEN 'tool_profiles' THEN '''id'', ''company_id'', ''status'''
|
|
ELSE '''id'', ''company_id'', ''profile_id'', ''target_type'', ''target_id''' END;
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_insert ON %I', relation);
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_update ON %I', relation);
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_delete ON %I', relation);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_insert AFTER INSERT ON %I REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%s)', relation, arguments);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_update AFTER UPDATE ON %I REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%s)', relation, arguments);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_delete AFTER DELETE ON %I REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%s)', relation, arguments);
|
|
END LOOP;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_connector_flags_runtime_changed() RETURNS trigger
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE company uuid; previous jsonb := coalesce(to_jsonb(OLD)->'experimental', '{}'::jsonb);
|
|
current jsonb := coalesce(to_jsonb(NEW)->'experimental', '{}'::jsonb);
|
|
BEGIN
|
|
IF jsonb_build_array(coalesce(previous->'enablePublicMcp', 'false'::jsonb), coalesce(previous->'enableChatConnectors', 'false'::jsonb))
|
|
IS DISTINCT FROM jsonb_build_array(coalesce(current->'enablePublicMcp', 'false'::jsonb), coalesce(current->'enableChatConnectors', 'false'::jsonb)) THEN
|
|
FOR company IN SELECT DISTINCT company_id FROM agents ORDER BY company_id LOOP
|
|
PERFORM paperclip_invalidate_connector_company(company);
|
|
END LOOP;
|
|
END IF;
|
|
RETURN NULL;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
DROP TRIGGER IF EXISTS paperclip_connector_flags_runtime_changed ON instance_settings;
|
|
CREATE TRIGGER paperclip_connector_flags_runtime_changed AFTER INSERT OR UPDATE OR DELETE ON instance_settings
|
|
FOR EACH ROW EXECUTE FUNCTION paperclip_connector_flags_runtime_changed();
|