mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
## Thinking Path > - Paperclip manages work for AI agents. > - Server startup applies database migrations before accepting work. > - Some older embedded databases use SQL_ASCII encoding. > - Migration 0355 uses a Unicode escape literal that PostgreSQL rejects on those databases. > - A source rebuild can pass but the updated server cannot start. > - This change makes that migration portable and preserves its shipped receipt. > - Existing databases retain their newer connector triggers during upgrades. ## Linked Issues or Issue Description **What happened?** A source deployment failed at startup with `conversion between UTF8 and SQL_ASCII is not supported`. Migration 0355 could not compile its connector policy function on an older embedded database. The previous app was restored while this repair was prepared. **Expected behavior** Supported legacy databases complete startup migrations. Connector selector trimming matches JavaScript `trim()`. A database that already applied 0355 must not replay it over the later 0356 connector trigger definition. **Steps to reproduce** Create a PostgreSQL database from template0 with SQL_ASCII encoding, then apply the current migration chain. The new regression fails with the observed conversion error before this change. **Paperclip version or commit** Reproduced on `92b238137ba716cdf1746dd49969e2f5ab7af8fe`. The branch includes current master `f57ed25385`. **Database mode** Embedded PostgreSQL 18.1; legacy SQL_ASCII database. The UTF-8 path is also covered. Related background: #645 fixed encoding for new databases. Existing SQL_ASCII installations still need portable migrations. #16956 fixes a separate workspace lock timeout that motivated this deployment. ## What Changed - Build the whitespace pattern from UTF-8 bytes instead of a Unicode escape literal. - Match whole whitespace sequences so SQL_ASCII does not trim partial multibyte characters. - Recognize the original 0355 checksum as a valid receipt. This prevents replay of its older trigger body over 0356. - Add real PostgreSQL coverage for both encodings, selector behavior, and preservation of the later trigger definition and receipt count. - Document the legacy encoding and migration-history contract. ## Verification - Red: SQL_ASCII migration failed with the same conversion error; UTF-8 passed. - Second red: fixing only the SQL caused an existing receipt to replay and overwrite the later connector function. - Green: 29 focused tests passed across the new regression, migration replay tests, and database client tests. - Rebased regression: all 29 focused tests passed. - `pnpm -r typecheck`, `pnpm build`, and `pnpm --filter @paperclipai/db check:migrations` passed. - Linux ARM64: clean build and 37 focused tests passed, including both encoding and workspace-lock regressions. - Live acceptance: the same legacy SQL_ASCII database completed all seven remaining migrations. The server reports the tested commit and healthy status. The restart report has no lost runs. - All 55 current-head checks passed; two unrelated Storybook checks were skipped. Full test, typecheck, build, runner, release, and E2E CI gates passed. The complete suite ran in CI; local verification used focused tests. - Greptile reviewed this exact commit at 5/5 with no findings. - Preserved failures: the initial unmodified-master startup failed at migration 0355 and was rolled back. A later build-cache path error in the deployment commands was corrected before the successful clean build; it required no source change. ## Risks The migration content changes, so receipt compatibility matters. The original checksum remains recognized to avoid replaying historical DDL. The tests compare complete later function definitions and receipt counts. The whitespace pattern contains encoded bytes; tests cover every ECMAScript whitespace character and preserve non-whitespace Unicode input. Database data and schema shape are unchanged by this repair. ## Model Used OpenAI GPT-6 through Codex, with repository tools, code execution, and browser verification. The exact served model variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
109 lines
7.8 KiB
PL/PgSQL
109 lines
7.8 KiB
PL/PgSQL
CREATE OR REPLACE FUNCTION paperclip_connector_policy_targets(p_company uuid, selectors jsonb)
|
|
RETURNS TABLE(target_type text, target_id text)
|
|
LANGUAGE plpgsql IMMUTABLE SET search_path = public, pg_temp AS $$
|
|
DECLARE single_agent text := CASE WHEN jsonb_typeof(selectors->'agentId') = 'string' THEN nullif(selectors->>'agentId', '') END;
|
|
many_agents text[];
|
|
trim_pattern text := '[\x09-\x0d ]|' || convert_from(decode('c2a07ce19a807ce280807ce280817ce280827ce280837ce280847ce280857ce280867ce280877ce280887ce280897ce2808a7ce280a87ce280a97ce280af7ce2819f7ce380807cefbbbf', 'hex'), 'UTF8');
|
|
BEGIN
|
|
SELECT array_agg(value#>>'{}') INTO many_agents
|
|
FROM jsonb_array_elements(CASE jsonb_typeof(selectors->'agentIds')
|
|
WHEN 'array' THEN selectors->'agentIds'
|
|
WHEN 'string' THEN jsonb_build_array(regexp_replace(selectors->>'agentIds', '^(' || trim_pattern || ')+|(' || trim_pattern || ')+$', '', 'g')) ELSE '[]'::jsonb END) value
|
|
WHERE jsonb_typeof(value) = 'string' AND regexp_replace(value#>>'{}', '^(' || trim_pattern || ')+|(' || trim_pattern || ')+$', '', 'g') <> '';
|
|
IF single_agent IS NOT NULL THEN
|
|
IF coalesce(cardinality(many_agents), 0) = 0 OR single_agent = ANY(many_agents) THEN
|
|
RETURN QUERY SELECT 'agent'::text, single_agent;
|
|
END IF;
|
|
ELSIF coalesce(cardinality(many_agents), 0) > 0 THEN
|
|
RETURN QUERY SELECT 'agent'::text, unnest(many_agents);
|
|
ELSE
|
|
RETURN QUERY SELECT 'company'::text, p_company::text;
|
|
END IF;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
CREATE OR REPLACE FUNCTION paperclip_connector_runtime_changed() RETURNS trigger
|
|
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
|
DECLARE company uuid; query text; target record; candidate uuid;
|
|
old_projection text := '(to_jsonb(o) - $1)'; new_projection text := '(to_jsonb(n) - $1)';
|
|
ignored text[] := ARRAY['credential_refs', 'credential_secret_refs', 'credential_source', 'external_credential', 'credential_policy', 'updated_at', 'created_at', 'last_used_at', 'last_resolved_at', 'last_health_at',
|
|
'health_status', 'health_checked_at', 'health_message', 'last_catalog_refresh_at', 'last_seen_at', 'last_error', 'last_event_at', 'last_publication_at',
|
|
'last_sync_at', 'sync_checkpoint', 'activation_at', 'lease_token', 'lease_expires_at'];
|
|
BEGIN
|
|
IF TG_TABLE_NAME = 'tool_profiles' AND TG_OP = 'INSERT' THEN RETURN NULL; END IF;
|
|
IF TG_TABLE_NAME = 'tool_policies' THEN
|
|
old_projection := old_projection || ' #- ''{config,trustRule,hitCount}'' #- ''{config,trustRule,lastHitAt}''';
|
|
new_projection := new_projection || ' #- ''{config,trustRule,hitCount}'' #- ''{config,trustRule,lastHitAt}''';
|
|
END IF;
|
|
IF TG_TABLE_NAME IN ('tool_profile_bindings', 'tool_connection_installs', 'tool_profile_entries', 'tool_policies') THEN
|
|
IF TG_TABLE_NAME IN ('tool_profile_entries', 'tool_policies') THEN
|
|
IF TG_OP = 'INSERT' THEN query := 'SELECT to_jsonb(n) AS row_data FROM new_rows n';
|
|
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT to_jsonb(o) AS row_data FROM old_rows o';
|
|
ELSE query := format('WITH changed AS (
|
|
SELECT to_jsonb(o) AS previous, to_jsonb(n) AS current FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I
|
|
WHERE %s IS DISTINCT FROM %s)
|
|
SELECT previous AS row_data FROM changed WHERE previous IS NOT NULL UNION
|
|
SELECT current AS row_data FROM changed WHERE current IS NOT NULL', TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
|
|
END IF;
|
|
IF TG_TABLE_NAME = 'tool_profile_entries' THEN
|
|
query := 'SELECT b.company_id, b.target_type, b.target_id FROM (' || query || ') changed
|
|
JOIN tool_profile_bindings b ON b.company_id = (row_data->>''company_id'')::uuid
|
|
AND b.profile_id = (row_data->>''profile_id'')::uuid';
|
|
ELSE
|
|
query := 'SELECT (row_data->>''company_id'')::uuid AS company_id, t.target_type, t.target_id
|
|
FROM (' || query || ') changed CROSS JOIN LATERAL
|
|
paperclip_connector_policy_targets((row_data->>''company_id'')::uuid, row_data->''selectors'') t';
|
|
END IF;
|
|
ELSIF TG_OP = 'INSERT' THEN query := 'SELECT company_id, target_type, target_id FROM new_rows';
|
|
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT company_id, target_type, target_id FROM old_rows';
|
|
ELSE query := format('WITH changed AS (
|
|
SELECT o.company_id AS old_company, o.target_type AS old_type, o.target_id AS old_target,
|
|
n.company_id AS new_company, n.target_type AS new_type, n.target_id AS new_target
|
|
FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I WHERE %s IS DISTINCT FROM %s)
|
|
SELECT old_company AS company_id, old_type AS target_type, old_target AS target_id FROM changed WHERE old_company IS NOT NULL UNION
|
|
SELECT new_company AS company_id, new_type AS target_type, new_target AS target_id FROM changed WHERE new_company IS NOT NULL',
|
|
TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
|
|
END IF;
|
|
FOR target IN EXECUTE 'SELECT DISTINCT company_id, target_type, target_id FROM (' || query || ') targets
|
|
WHERE target_type IN (''agent'', ''company'')
|
|
ORDER BY company_id, target_type DESC, target_id' USING ignored LOOP
|
|
IF target.target_type = 'company' AND target.target_id = target.company_id::text THEN
|
|
PERFORM paperclip_invalidate_connector_company(target.company_id);
|
|
ELSIF target.target_type = 'agent' AND target.target_id ~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' THEN
|
|
SELECT id INTO candidate FROM agents WHERE company_id = target.company_id AND id = target.target_id::uuid;
|
|
IF FOUND THEN PERFORM paperclip_invalidate_connector_agent(target.company_id, candidate); END IF;
|
|
END IF;
|
|
END LOOP;
|
|
RETURN NULL;
|
|
END IF;
|
|
IF TG_OP = 'INSERT' THEN query := 'SELECT DISTINCT company_id FROM new_rows ORDER BY company_id';
|
|
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT DISTINCT company_id FROM old_rows ORDER BY company_id';
|
|
ELSE query := format('WITH changed AS (
|
|
SELECT o.company_id AS old_company, n.company_id AS new_company FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I
|
|
WHERE %s IS DISTINCT FROM %s)
|
|
SELECT old_company AS company_id FROM changed WHERE old_company IS NOT NULL UNION
|
|
SELECT new_company AS company_id FROM changed WHERE new_company IS NOT NULL ORDER BY company_id', TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
|
|
END IF;
|
|
FOR company IN EXECUTE query USING ignored LOOP
|
|
PERFORM paperclip_invalidate_connector_company(company);
|
|
END LOOP;
|
|
RETURN NULL;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|
|
DO $$
|
|
DECLARE relation text; key_column text;
|
|
BEGIN
|
|
FOREACH relation IN ARRAY ARRAY['tool_applications', 'tool_connections', 'tool_connection_installs', 'tool_catalog_entries', 'tool_profiles', 'tool_profile_entries', 'tool_profile_bindings', 'tool_policies', 'tool_mcp_gateways'] LOOP
|
|
key_column := CASE WHEN relation = 'email_endpoints' THEN 'endpoint_id' ELSE 'id' END;
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_insert ON %I', relation);
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_update ON %I', relation);
|
|
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_delete ON %I', relation);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_insert AFTER INSERT ON %I REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_update AFTER UPDATE ON %I REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
|
|
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_delete AFTER DELETE ON %I REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
|
|
END LOOP;
|
|
END;
|
|
$$;
|
|
--> statement-breakpoint
|