Files
PaperClipAI/packages/db/src/migrations/0355_agent_tool_definition_hooks.sql
T
DottaandPaperclip 96824a71d1 fix(db): migrate connector hooks on SQL_ASCII databases (#16958)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Server startup applies database migrations before accepting work.
> - Some older embedded databases use SQL_ASCII encoding.
> - Migration 0355 uses a Unicode escape literal that PostgreSQL rejects
on those databases.
> - A source rebuild can pass but the updated server cannot start.
> - This change makes that migration portable and preserves its shipped
receipt.
> - Existing databases retain their newer connector triggers during
upgrades.

## Linked Issues or Issue Description

**What happened?**

A source deployment failed at startup with `conversion between UTF8 and
SQL_ASCII is not supported`. Migration 0355 could not compile its
connector policy function on an older embedded database. The previous
app was restored while this repair was prepared.

**Expected behavior**

Supported legacy databases complete startup migrations. Connector
selector trimming matches JavaScript `trim()`. A database that already
applied 0355 must not replay it over the later 0356 connector trigger
definition.

**Steps to reproduce**

Create a PostgreSQL database from template0 with SQL_ASCII encoding,
then apply the current migration chain. The new regression fails with
the observed conversion error before this change.

**Paperclip version or commit**

Reproduced on `92b238137ba716cdf1746dd49969e2f5ab7af8fe`. The branch
includes current master `f57ed25385`.

**Database mode**

Embedded PostgreSQL 18.1; legacy SQL_ASCII database. The UTF-8 path is
also covered.

Related background: #645 fixed encoding for new databases. Existing
SQL_ASCII installations still need portable migrations. #16956 fixes a
separate workspace lock timeout that motivated this deployment.

## What Changed

- Build the whitespace pattern from UTF-8 bytes instead of a Unicode
escape literal.
- Match whole whitespace sequences so SQL_ASCII does not trim partial
multibyte characters.
- Recognize the original 0355 checksum as a valid receipt. This prevents
replay of its older trigger body over 0356.
- Add real PostgreSQL coverage for both encodings, selector behavior,
and preservation of the later trigger definition and receipt count.
- Document the legacy encoding and migration-history contract.

## Verification

- Red: SQL_ASCII migration failed with the same conversion error; UTF-8
passed.
- Second red: fixing only the SQL caused an existing receipt to replay
and overwrite the later connector function.
- Green: 29 focused tests passed across the new regression, migration
replay tests, and database client tests.
- Rebased regression: all 29 focused tests passed.
- `pnpm -r typecheck`, `pnpm build`, and `pnpm --filter @paperclipai/db
check:migrations` passed.
- Linux ARM64: clean build and 37 focused tests passed, including both
encoding and workspace-lock regressions.
- Live acceptance: the same legacy SQL_ASCII database completed all
seven remaining migrations. The server reports the tested commit and
healthy status. The restart report has no lost runs.
- All 55 current-head checks passed; two unrelated Storybook checks were
skipped. Full test, typecheck, build, runner, release, and E2E CI gates
passed. The complete suite ran in CI; local verification used focused
tests.
- Greptile reviewed this exact commit at 5/5 with no findings.
- Preserved failures: the initial unmodified-master startup failed at
migration 0355 and was rolled back. A later build-cache path error in
the deployment commands was corrected before the successful clean build;
it required no source change.

## Risks

The migration content changes, so receipt compatibility matters. The
original checksum remains recognized to avoid replaying historical DDL.
The tests compare complete later function definitions and receipt
counts. The whitespace pattern contains encoded bytes; tests cover every
ECMAScript whitespace character and preserve non-whitespace Unicode
input. Database data and schema shape are unchanged by this repair.

## Model Used

OpenAI GPT-6 through Codex, with repository tools, code execution, and
browser verification. The exact served model variant and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 07:56:32 -05:00

109 lines
7.8 KiB
PL/PgSQL

CREATE OR REPLACE FUNCTION paperclip_connector_policy_targets(p_company uuid, selectors jsonb)
RETURNS TABLE(target_type text, target_id text)
LANGUAGE plpgsql IMMUTABLE SET search_path = public, pg_temp AS $$
DECLARE single_agent text := CASE WHEN jsonb_typeof(selectors->'agentId') = 'string' THEN nullif(selectors->>'agentId', '') END;
many_agents text[];
trim_pattern text := '[\x09-\x0d ]|' || convert_from(decode('c2a07ce19a807ce280807ce280817ce280827ce280837ce280847ce280857ce280867ce280877ce280887ce280897ce2808a7ce280a87ce280a97ce280af7ce2819f7ce380807cefbbbf', 'hex'), 'UTF8');
BEGIN
SELECT array_agg(value#>>'{}') INTO many_agents
FROM jsonb_array_elements(CASE jsonb_typeof(selectors->'agentIds')
WHEN 'array' THEN selectors->'agentIds'
WHEN 'string' THEN jsonb_build_array(regexp_replace(selectors->>'agentIds', '^(' || trim_pattern || ')+|(' || trim_pattern || ')+$', '', 'g')) ELSE '[]'::jsonb END) value
WHERE jsonb_typeof(value) = 'string' AND regexp_replace(value#>>'{}', '^(' || trim_pattern || ')+|(' || trim_pattern || ')+$', '', 'g') <> '';
IF single_agent IS NOT NULL THEN
IF coalesce(cardinality(many_agents), 0) = 0 OR single_agent = ANY(many_agents) THEN
RETURN QUERY SELECT 'agent'::text, single_agent;
END IF;
ELSIF coalesce(cardinality(many_agents), 0) > 0 THEN
RETURN QUERY SELECT 'agent'::text, unnest(many_agents);
ELSE
RETURN QUERY SELECT 'company'::text, p_company::text;
END IF;
END;
$$;
--> statement-breakpoint
CREATE OR REPLACE FUNCTION paperclip_connector_runtime_changed() RETURNS trigger
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
DECLARE company uuid; query text; target record; candidate uuid;
old_projection text := '(to_jsonb(o) - $1)'; new_projection text := '(to_jsonb(n) - $1)';
ignored text[] := ARRAY['credential_refs', 'credential_secret_refs', 'credential_source', 'external_credential', 'credential_policy', 'updated_at', 'created_at', 'last_used_at', 'last_resolved_at', 'last_health_at',
'health_status', 'health_checked_at', 'health_message', 'last_catalog_refresh_at', 'last_seen_at', 'last_error', 'last_event_at', 'last_publication_at',
'last_sync_at', 'sync_checkpoint', 'activation_at', 'lease_token', 'lease_expires_at'];
BEGIN
IF TG_TABLE_NAME = 'tool_profiles' AND TG_OP = 'INSERT' THEN RETURN NULL; END IF;
IF TG_TABLE_NAME = 'tool_policies' THEN
old_projection := old_projection || ' #- ''{config,trustRule,hitCount}'' #- ''{config,trustRule,lastHitAt}''';
new_projection := new_projection || ' #- ''{config,trustRule,hitCount}'' #- ''{config,trustRule,lastHitAt}''';
END IF;
IF TG_TABLE_NAME IN ('tool_profile_bindings', 'tool_connection_installs', 'tool_profile_entries', 'tool_policies') THEN
IF TG_TABLE_NAME IN ('tool_profile_entries', 'tool_policies') THEN
IF TG_OP = 'INSERT' THEN query := 'SELECT to_jsonb(n) AS row_data FROM new_rows n';
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT to_jsonb(o) AS row_data FROM old_rows o';
ELSE query := format('WITH changed AS (
SELECT to_jsonb(o) AS previous, to_jsonb(n) AS current FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I
WHERE %s IS DISTINCT FROM %s)
SELECT previous AS row_data FROM changed WHERE previous IS NOT NULL UNION
SELECT current AS row_data FROM changed WHERE current IS NOT NULL', TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
END IF;
IF TG_TABLE_NAME = 'tool_profile_entries' THEN
query := 'SELECT b.company_id, b.target_type, b.target_id FROM (' || query || ') changed
JOIN tool_profile_bindings b ON b.company_id = (row_data->>''company_id'')::uuid
AND b.profile_id = (row_data->>''profile_id'')::uuid';
ELSE
query := 'SELECT (row_data->>''company_id'')::uuid AS company_id, t.target_type, t.target_id
FROM (' || query || ') changed CROSS JOIN LATERAL
paperclip_connector_policy_targets((row_data->>''company_id'')::uuid, row_data->''selectors'') t';
END IF;
ELSIF TG_OP = 'INSERT' THEN query := 'SELECT company_id, target_type, target_id FROM new_rows';
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT company_id, target_type, target_id FROM old_rows';
ELSE query := format('WITH changed AS (
SELECT o.company_id AS old_company, o.target_type AS old_type, o.target_id AS old_target,
n.company_id AS new_company, n.target_type AS new_type, n.target_id AS new_target
FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I WHERE %s IS DISTINCT FROM %s)
SELECT old_company AS company_id, old_type AS target_type, old_target AS target_id FROM changed WHERE old_company IS NOT NULL UNION
SELECT new_company AS company_id, new_type AS target_type, new_target AS target_id FROM changed WHERE new_company IS NOT NULL',
TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
END IF;
FOR target IN EXECUTE 'SELECT DISTINCT company_id, target_type, target_id FROM (' || query || ') targets
WHERE target_type IN (''agent'', ''company'')
ORDER BY company_id, target_type DESC, target_id' USING ignored LOOP
IF target.target_type = 'company' AND target.target_id = target.company_id::text THEN
PERFORM paperclip_invalidate_connector_company(target.company_id);
ELSIF target.target_type = 'agent' AND target.target_id ~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' THEN
SELECT id INTO candidate FROM agents WHERE company_id = target.company_id AND id = target.target_id::uuid;
IF FOUND THEN PERFORM paperclip_invalidate_connector_agent(target.company_id, candidate); END IF;
END IF;
END LOOP;
RETURN NULL;
END IF;
IF TG_OP = 'INSERT' THEN query := 'SELECT DISTINCT company_id FROM new_rows ORDER BY company_id';
ELSIF TG_OP = 'DELETE' THEN query := 'SELECT DISTINCT company_id FROM old_rows ORDER BY company_id';
ELSE query := format('WITH changed AS (
SELECT o.company_id AS old_company, n.company_id AS new_company FROM old_rows o FULL JOIN new_rows n ON o.%I = n.%I
WHERE %s IS DISTINCT FROM %s)
SELECT old_company AS company_id FROM changed WHERE old_company IS NOT NULL UNION
SELECT new_company AS company_id FROM changed WHERE new_company IS NOT NULL ORDER BY company_id', TG_ARGV[0], TG_ARGV[0], old_projection, new_projection);
END IF;
FOR company IN EXECUTE query USING ignored LOOP
PERFORM paperclip_invalidate_connector_company(company);
END LOOP;
RETURN NULL;
END;
$$;
--> statement-breakpoint
DO $$
DECLARE relation text; key_column text;
BEGIN
FOREACH relation IN ARRAY ARRAY['tool_applications', 'tool_connections', 'tool_connection_installs', 'tool_catalog_entries', 'tool_profiles', 'tool_profile_entries', 'tool_profile_bindings', 'tool_policies', 'tool_mcp_gateways'] LOOP
key_column := CASE WHEN relation = 'email_endpoints' THEN 'endpoint_id' ELSE 'id' END;
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_insert ON %I', relation);
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_update ON %I', relation);
EXECUTE format('DROP TRIGGER IF EXISTS paperclip_connector_runtime_delete ON %I', relation);
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_insert AFTER INSERT ON %I REFERENCING NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_update AFTER UPDATE ON %I REFERENCING OLD TABLE AS old_rows NEW TABLE AS new_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
EXECUTE format('CREATE TRIGGER paperclip_connector_runtime_delete AFTER DELETE ON %I REFERENCING OLD TABLE AS old_rows FOR EACH STATEMENT EXECUTE FUNCTION paperclip_connector_runtime_changed(%L)', relation, key_column);
END LOOP;
END;
$$;
--> statement-breakpoint