diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index 26eaaadc723..cabbcdad193 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -1540,6 +1540,28 @@ POST /api/tool-connections/:connectionId/test-calls DELETE /api/tool-connections/:connectionId ``` +Stored tool-call content endpoints: + +```text +GET /api/tool-connections/:connectionId/stored-content +POST /api/tool-connections/:connectionId/stored-content/deletion +GET /api/tool-connections/:connectionId/stored-content/deletion +DELETE /api/tool-connections/:connectionId?deleteStoredContent=true +``` + +Deletion is connector-agnostic. It clears the content of the connection's +tool-call summaries (including `tool_access_audit_events.details.argumentsSummary`), +approval previews, approval cards, delivered action results and app views, and +deletes the whole run transcript (every copy of the run log, run-event content, +output excerpts and the result text) of every settled run that called the +connection or received an approved action's result. Row ids, timestamps, status, +sizes and hashes stay. A step that cannot confirm its deletion fails, and the +request stays pending and retries; it completes only when every step succeeded. +A `tool_connection_content_deletions` row is the durable request; the +`tool-connection-content-deletion` delivery queue resumes it after a restart. +Runs that are still working are skipped and counted. The user-facing help is +`docs/guides/board-operator/deleting-connection-data.md`. + Responses may contain secret IDs, version selectors, header names, prefixes, scope names, expiry timestamps, and redacted provider metadata. They must not contain secret values. Logs and activity should identify the operation and diff --git a/docs/docs.json b/docs/docs.json index a1a008f94dd..55989359f45 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -57,6 +57,7 @@ "guides/board-operator/approvals", "guides/board-operator/costs-and-budgets", "guides/board-operator/activity-log", + "guides/board-operator/deleting-connection-data", "guides/board-operator/importing-and-exporting" ] }, diff --git a/docs/guides/board-operator/deleting-connection-data.md b/docs/guides/board-operator/deleting-connection-data.md new file mode 100644 index 00000000000..831a8db2ca1 --- /dev/null +++ b/docs/guides/board-operator/deleting-connection-data.md @@ -0,0 +1,58 @@ +--- +title: Deleting Data Read Through a Connection +summary: Permanently remove what Paperclip saved from a connected app's tool calls +--- + +When an agent uses a connected app, such as Google Workspace, Paperclip saves part of what the app returned so you can review the work later. You can delete that saved data at any time, for any connected app. + +## What Paperclip saves + +For each tool call an agent makes through a connection, Paperclip keeps: + +- A short preview of the request and the result (up to about 4,000 characters, with secrets removed) +- For actions that needed approval, the preview and arguments shown on the approval card, and the result passed back to the agent +- Interactive app views that a tool showed in a task +- The transcript of the run that made the call, which can contain the full result, and of the run that received an approved action's result + +This data stays until you delete it. Removing the connection alone does not delete it. + +## How to delete it + +You need permission to manage the connection. + +**From the connection's page:** + +1. Open **Connectors** and select the connection. +2. At the bottom of the **Permissions** tab, select **Delete data**. +3. Check the counts in the confirmation and select **Delete data**. + +**When you remove a connection:** + +1. Open **Connectors**, open the connection's menu, and select **Remove connection**. +2. Select **Also delete data read through this connection**. +3. Select **Remove connection**. + +Deletion starts right away and continues in the background. It finishes even if Paperclip restarts while it runs. If a step fails, for example because file storage is unavailable, Paperclip keeps the request open and tries again until every step succeeds. It reports the deletion as finished only then. The connection's page shows when the data was last deleted. + +## What is deleted + +- Request and result previews for every call made through the connection before you asked to delete, including the copies in the connection's audit history +- Approval previews, arguments and results for those calls, on the approval card and in what was passed back to the agent +- Interactive app views from those calls +- The whole transcript of every run that called the connection or received an approved action's result, including that run's other steps and its final summary + +## What is kept + +- The record that each call happened: when, which tool, which agent, the outcome, the size of the data, and a fingerprint of it. Paperclip needs these for its activity history and usage counts. They do not contain the data. +- **Anything your agents wrote into tasks: comments, documents and files.** This action does not remove them, even when an agent copied data from the app into them. Delete or edit those items yourself, or delete the task. +- The connection's audit activity, which records counts only. +- Transcripts of later runs that only read the context an interactive app view shared. Paperclip does not record which runs read it. Delete those tasks if you need that text gone. +- Raw provider traces. An administrator must turn them on, and they expire on their own within 24 hours. + +## Runs that are still working + +Paperclip does not change a run while it is still working. The confirmation tells you how many runs are skipped for this reason. Select **Delete data** again after they finish. + +## It cannot be undone + +Deleted data cannot be recovered. diff --git a/packages/db/src/migrations/0490_nappy_chat.sql b/packages/db/src/migrations/0490_nappy_chat.sql new file mode 100644 index 00000000000..3f470177466 --- /dev/null +++ b/packages/db/src/migrations/0490_nappy_chat.sql @@ -0,0 +1,21 @@ +CREATE TABLE "tool_connection_content_deletions" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "connection_id" uuid NOT NULL, + "status" text DEFAULT 'pending' NOT NULL, + "requested_by_actor_type" text NOT NULL, + "requested_by_actor_id" text, + "requested_at" timestamp with time zone DEFAULT now() NOT NULL, + "progress" jsonb DEFAULT '{}'::jsonb NOT NULL, + "attempts" integer DEFAULT 0 NOT NULL, + "last_error" text, + "completed_at" timestamp with time zone, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +ALTER TABLE "tool_connection_content_deletions" ADD CONSTRAINT "tool_connection_content_deletions_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "tool_connection_content_deletions" ADD CONSTRAINT "tool_connection_content_deletions_connection_id_tool_connections_id_fk" FOREIGN KEY ("connection_id") REFERENCES "public"."tool_connections"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "tool_connection_content_deletions_connection_idx" ON "tool_connection_content_deletions" USING btree ("company_id","connection_id","created_at");--> statement-breakpoint +CREATE UNIQUE INDEX "tool_connection_content_deletions_pending_uq" ON "tool_connection_content_deletions" USING btree ("company_id","connection_id") WHERE "tool_connection_content_deletions"."status" = 'pending';--> statement-breakpoint +CREATE INDEX "tool_connection_content_deletions_pending_idx" ON "tool_connection_content_deletions" USING btree ("requested_at") WHERE "tool_connection_content_deletions"."status" = 'pending'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0485_snapshot.json b/packages/db/src/migrations/meta/0490_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0485_snapshot.json rename to packages/db/src/migrations/meta/0490_snapshot.json index fcf5987ed8c..6a945283d61 100644 --- a/packages/db/src/migrations/meta/0485_snapshot.json +++ b/packages/db/src/migrations/meta/0490_snapshot.json @@ -1,6 +1,6 @@ { - "id": "78d10bc7-b440-4c00-a02e-90111c50071d", - "prevId": "9389124c-379a-4b64-9c20-c7b37538e7c8", + "id": "a8bc890a-cc97-4b54-b426-5687b134a8b0", + "prevId": "dc61c7f3-cf3d-44c5-9296-86973e5f0925", "version": "7", "dialect": "postgresql", "tables": { @@ -36134,6 +36134,78 @@ "method": "btree", "with": {} }, + "heartbeat_runs_company_finished_idx": { + "name": "heartbeat_runs_company_finished_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "finished_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"finished_at\" is not null", + "concurrently": true, + "method": "btree", + "with": {} + }, + "heartbeat_runs_settled_process_identity_idx": { + "name": "heartbeat_runs_settled_process_identity_idx", + "columns": [ + { + "expression": "process_host_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' and \"heartbeat_runs\".\"status\" in ('failed', 'cancelled', 'timed_out', 'interrupted')\n and \"heartbeat_runs\".\"finished_at\" is not null\n and (\"heartbeat_runs\".\"process_pid\" is not null or \"heartbeat_runs\".\"process_group_id\" is not null or \"heartbeat_runs\".\"process_started_at\" is not null)", + "concurrently": true, + "method": "btree", + "with": {} + }, + "heartbeat_runs_pending_api_stop_idx": { + "name": "heartbeat_runs_pending_api_stop_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"driver_kind\" = 'cursor_cloud_api'\n and (\"heartbeat_runs\".\"result_json\"->'executionCancellation'->>'state') = 'requested'", + "concurrently": true, + "method": "btree", + "with": {} + }, "heartbeat_runs_company_created_at_desc_idx": { "name": "heartbeat_runs_company_created_at_desc_idx", "columns": [ @@ -40827,6 +40899,131 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.issue_recovery_deferrals": { + "name": "issue_recovery_deferrals", + "schema": "", + "columns": { + "queue": { + "name": "queue", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "due_at": { + "name": "due_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": { + "issue_recovery_deferrals_issue_id_issues_id_fk": { + "name": "issue_recovery_deferrals_issue_id_issues_id_fk", + "tableFrom": "issue_recovery_deferrals", + "tableTo": "issues", + "columnsFrom": [ + "issue_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "issue_recovery_deferrals_company_id_companies_id_fk": { + "name": "issue_recovery_deferrals_company_id_companies_id_fk", + "tableFrom": "issue_recovery_deferrals", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "issue_recovery_deferrals_pk": { + "name": "issue_recovery_deferrals_pk", + "columns": [ + "queue", + "issue_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.issue_recovery_full_passes": { + "name": "issue_recovery_full_passes", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "queue": { + "name": "queue", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": { + "issue_recovery_full_passes_company_id_companies_id_fk": { + "name": "issue_recovery_full_passes_company_id_companies_id_fk", + "tableFrom": "issue_recovery_full_passes", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "issue_recovery_full_passes_pk": { + "name": "issue_recovery_full_passes_pk", + "columns": [ + "company_id", + "queue" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.issue_reference_mentions": { "name": "issue_reference_mentions", "schema": "", @@ -44884,6 +45081,123 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.link_previews": { + "name": "link_previews", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "viewer_id": { + "name": "viewer_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "channel": { + "name": "channel", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_kind": { + "name": "resource_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "canonical_url": { + "name": "canonical_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "checked_at": { + "name": "checked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "link_previews_scope_url_uq": { + "name": "link_previews_scope_url_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "viewer_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "canonical_url", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "link_previews_company_id_companies_id_fk": { + "name": "link_previews_company_id_companies_id_fk", + "tableFrom": "link_previews", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.managed_agent_profiles": { "name": "managed_agent_profiles", "schema": "", @@ -58387,6 +58701,197 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_connection_content_deletions": { + "name": "tool_connection_content_deletions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "requested_by_actor_type": { + "name": "requested_by_actor_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "requested_by_actor_id": { + "name": "requested_by_actor_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "requested_at": { + "name": "requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "progress": { + "name": "progress", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "attempts": { + "name": "attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_connection_content_deletions_connection_idx": { + "name": "tool_connection_content_deletions_connection_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_connection_content_deletions_pending_uq": { + "name": "tool_connection_content_deletions_pending_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"tool_connection_content_deletions\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_connection_content_deletions_pending_idx": { + "name": "tool_connection_content_deletions_pending_idx", + "columns": [ + { + "expression": "requested_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_connection_content_deletions\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_connection_content_deletions_company_id_companies_id_fk": { + "name": "tool_connection_content_deletions_company_id_companies_id_fk", + "tableFrom": "tool_connection_content_deletions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_connection_content_deletions_connection_id_tool_connections_id_fk": { + "name": "tool_connection_content_deletions_connection_id_tool_connections_id_fk", + "tableFrom": "tool_connection_content_deletions", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_connection_installs": { "name": "tool_connection_installs", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index ee03bddd7a5..b7bf09d719e 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3417,6 +3417,13 @@ "when": 1794841384470, "tag": "0489_real_sage", "breakpoints": true + }, + { + "idx": 490, + "version": "7", + "when": 1794923393849, + "tag": "0490_nappy_chat", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index f27b014644c..fd3261f77b3 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -291,3 +291,4 @@ export { budgetEvaluationWork } from "./budget_evaluation_work.js"; export { tailscaleConnections } from "./tailscale_connections.js"; export { nativeProviderArtifacts, nativeProviderArtifactReferences } from "./native_provider_artifacts.js"; export { linkPreviews } from "./link_previews.js"; +export { toolConnectionContentDeletions } from "./tool_connection_content_deletions.js"; diff --git a/packages/db/src/schema/tool_connection_content_deletions.ts b/packages/db/src/schema/tool_connection_content_deletions.ts new file mode 100644 index 00000000000..93c29a7f834 --- /dev/null +++ b/packages/db/src/schema/tool_connection_content_deletions.ts @@ -0,0 +1,32 @@ +import { sql } from "drizzle-orm"; +import { index, integer, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core"; +import { companies } from "./companies.js"; +import { toolConnections } from "./tool_access.js"; + +export const toolConnectionContentDeletions = pgTable( + "tool_connection_content_deletions", + { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), + connectionId: uuid("connection_id").notNull().references(() => toolConnections.id, { onDelete: "cascade" }), + status: text("status").$type<"pending" | "completed">().notNull().default("pending"), + requestedByActorType: text("requested_by_actor_type").notNull(), + requestedByActorId: text("requested_by_actor_id"), + requestedAt: timestamp("requested_at", { withTimezone: true }).notNull().defaultNow(), + progress: jsonb("progress").$type>().notNull().default({}), + attempts: integer("attempts").notNull().default(0), + lastError: text("last_error"), + completedAt: timestamp("completed_at", { withTimezone: true }), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [ + index("tool_connection_content_deletions_connection_idx").on(table.companyId, table.connectionId, table.createdAt), + uniqueIndex("tool_connection_content_deletions_pending_uq") + .on(table.companyId, table.connectionId) + .where(sql`${table.status} = 'pending'`), + index("tool_connection_content_deletions_pending_idx") + .on(table.requestedAt) + .where(sql`${table.status} = 'pending'`), + ], +); diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 825867d65bb..6a7bc609575 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1542,6 +1542,11 @@ export type { ToolConnectionInstallTargetType, ToolConnectionRemovalResult, ToolConnectionRemovalSummary, + ToolConnectionStoredContent, + ToolConnectionStoredContentCounts, + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionContentDeletionStatus, ConnectionAudienceMember, ConnectionGrant, ConnectionGrantCapabilities, diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 8abf1f4ed91..9c93d353993 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -562,6 +562,11 @@ export type { ToolConnectionInstallTargetType, ToolConnectionRemovalResult, ToolConnectionRemovalSummary, + ToolConnectionStoredContent, + ToolConnectionStoredContentCounts, + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionContentDeletionStatus, ConnectionTokenAttribution, ConnectionRecoverableErrorCode, ConnectionRecoverableErrorPayload, diff --git a/packages/shared/src/types/tool-access.ts b/packages/shared/src/types/tool-access.ts index 1d2c29fb7c0..32b0ad72fd9 100644 --- a/packages/shared/src/types/tool-access.ts +++ b/packages/shared/src/types/tool-access.ts @@ -113,6 +113,7 @@ export interface ToolRedactedValueSummary { sha256?: string | null; redactedFields?: string[]; artifactId?: string | null; + contentDeletedAt?: string; } export type VercelConnectPrincipalMode = "app" | "user"; @@ -433,6 +434,40 @@ export interface ToolConnectionRemovalResult { removal: ToolConnectionRemovalSummary; } +export interface ToolConnectionStoredContentCounts { + toolCalls: number; + approvals: number; + appViews: number; + runs: number; + runsInProgress: number; +} + +export interface ToolConnectionContentDeletionCounts { + toolCalls: number; + approvals: number; + auditEvents: number; + appViews: number; + runs: number; + runsInProgress: number; + itemsInProgress: number; +} + +export type ToolConnectionContentDeletionStatus = "pending" | "completed"; + +export interface ToolConnectionContentDeletion { + id: string; + status: ToolConnectionContentDeletionStatus; + requestedAt: string; + completedAt: string | null; + retrying: boolean; + counts: ToolConnectionContentDeletionCounts; +} + +export interface ToolConnectionStoredContent { + stored: ToolConnectionStoredContentCounts; + deletion: ToolConnectionContentDeletion | null; +} + export type ConnectionTokenScope = string | string[]; export type ConnectionTokenSubject = { type: "app" } | { type: "user"; userId: string }; diff --git a/server/src/__tests__/tool-connection-content-deletion-canary.test.ts b/server/src/__tests__/tool-connection-content-deletion-canary.test.ts new file mode 100644 index 00000000000..7aea9491925 --- /dev/null +++ b/server/src/__tests__/tool-connection-content-deletion-canary.test.ts @@ -0,0 +1,346 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { Readable } from "node:stream"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { sql } from "drizzle-orm"; +import { + agentWakeupRequests, + agents, + companies, + createDb, + heartbeatRunEvents, + heartbeatRuns, + issueComments, + issues, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolApplications, + toolCallEvents, + toolConnections, + toolInvocations, + toolMcpAppViews, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import { composeRunLogStore, createDurableRunLogStore, type RunLogHandle, type RunLogStore } from "../services/run-log-store.js"; +import { createObjectRunLogStore } from "../services/object-run-log-store.js"; +import { summarizeToolValue } from "../services/tool-content-guards.js"; +import type { StorageProvider } from "../storage/types.js"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +const OUT_OF_SCOPE_COLUMNS = new Map([ + ["issue_comments.body", "Comments an agent wrote into the task. The user removes them by editing or deleting the comment or the task."], +]); + +function memoryProvider() { + const objects = new Map(); + const etag = (bytes: Buffer) => String(bytes.length) + ":" + bytes.toString("base64").slice(0, 32); + const read = async (body: Buffer | AsyncIterable) => { + if (Buffer.isBuffer(body)) return Buffer.from(body); + const chunks: Buffer[] = []; + for await (const chunk of body) chunks.push(Buffer.from(chunk)); + return Buffer.concat(chunks); + }; + const provider: StorageProvider = { + id: "s3", + async putObject(input) { objects.set(input.objectKey, await read(input.body)); }, + async putObjectConditional(input) { + const old = objects.get(input.objectKey); + if (input.ifNoneMatch ? old : !old || etag(old) !== input.ifMatch) + throw Object.assign(new Error("conflict"), { name: "PreconditionFailed" }); + const bytes = await read(input.body); + objects.set(input.objectKey, bytes); + return { etag: etag(bytes) }; + }, + async getObject(input) { + const bytes = objects.get(input.objectKey); + if (!bytes) throw Object.assign(new Error("missing"), { name: "NoSuchKey" }); + const slice = input.range ? bytes.subarray(input.range.start, input.range.end + 1) : bytes; + return { stream: Readable.from([slice]), contentLength: slice.length, etag: etag(bytes) }; + }, + async headObject(input) { + const bytes = objects.get(input.objectKey); + return bytes ? { exists: true, contentLength: bytes.length, etag: etag(bytes) } : { exists: false }; + }, + async deleteObject(input) { objects.delete(input.objectKey); }, + async listObjects(input) { return { keys: [...objects.keys()].filter((key) => key.startsWith(input.prefix)) }; }, + }; + return { provider, objects }; +} + +async function filesContaining(dir: string, marker: string): Promise { + const hits: string[] = []; + const entries = await fs.readdir(dir, { withFileTypes: true }).catch(() => []); + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) hits.push(...await filesContaining(full, marker)); + else if ((await fs.readFile(full)).includes(marker)) hits.push(full); + } + return hits; +} + +describeEmbeddedPostgres("deleting a connection's stored content leaves no copy behind", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + let baseDir = ""; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-content-canary-"); + db = createDb(tempDb.connectionString); + baseDir = await fs.mkdtemp(path.join(os.tmpdir(), "content-canary-")); + }, 30_000); + + afterAll(async () => { + await tempDb?.cleanup(); + await fs.rm(baseDir, { recursive: true, force: true }); + }); + + function stores(dir: string) { + const mirror = memoryProvider(); + const objects = memoryProvider(); + const local = createDurableRunLogStore({ basePath: dir, s3: { provider: mirror.provider } }); + const external = createObjectRunLogStore({ provider: objects.provider, keyPrefix: "run-logs", assertWriterAuthority: async () => undefined }); + return { dir, mirror, objects, local, external, store: composeRunLogStore(local, external) }; + } + + async function writeLog(store: RunLogStore, ids: { companyId: string; agentId: string; runId: string }, text: string) { + const handle = await store.begin(ids); + await store.append(handle, { stream: "stdout", chunk: text, ts: new Date().toISOString(), seq: 1 }); + await store.finalize(handle); + return handle; + } + + async function seedCompany() { + const [company] = await db.insert(companies).values({ name: `Canary ${randomUUID()}`, issuePrefix: `CN${randomUUID().slice(0, 6).toUpperCase()}` }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company!.id, name: "Canary agent", role: "engineer", status: "active", adapterType: "process", adapterConfig: {}, runtimeConfig: {}, + }).returning(); + const [application] = await db.insert(toolApplications).values({ companyId: company!.id, name: `Docs ${randomUUID().slice(0, 6)}`, type: "mcp_http" }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Summarise the drive" }).returning(); + return { company: company!, agent: agent!, application: application!, issue: issue! }; + } + + async function seedConnectionFlow( + base: Awaited>, + logs: ReturnType, + marker: string, + ) { + const companyId = base.company.id; + const [connection] = await db.insert(toolConnections).values({ + companyId, applicationId: base.application.id, name: `Docs ${marker}`.slice(0, 40), uid: randomUUID(), transport: "mcp_remote", status: "active", + }).returning(); + const finishedAt = new Date(Date.now() - 60_000); + const created = new Date(Date.now() - 30_000); + + const callingRunId = randomUUID(); + const callingLog = await writeLog(logs.store, { companyId, agentId: base.agent.id, runId: callingRunId }, + JSON.stringify({ tool: "read_doc", output: `Document says ${marker}` })); + await db.insert(heartbeatRuns).values({ + id: callingRunId, companyId, agentId: base.agent.id, issueId: base.issue.id, status: "succeeded", finishedAt, runtimeMode: "native", + logStore: callingLog.store, logRef: callingLog.logRef, stdoutExcerpt: `read_doc -> ${marker}`, + resultJson: { + summary: `The drive says ${marker}`, + finalizationPhase: "committed", + nativeResult: { schema: "paperclip.run_result.v1", summary: `Answer quoting ${marker}`, reportedWorkDisposition: "completed", + continuation: { kind: "response_wake", summary: `Continue with ${marker}`, idempotencyKey: "k1" } }, + }, + }); + await db.insert(heartbeatRunEvents).values([ + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 1, eventType: "lifecycle", message: "run started", payload: { phase: "start" } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 2, eventType: "tool.execution.started", message: "read_doc", payload: { arguments: { query: marker } } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 3, eventType: "tool.execution.completed", message: marker, payload: { output: marker } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 4, eventType: "item.completed", payload: { prpEvent: { payload: { kind: "assistant_message", text: `I found ${marker}` } } } }, + ]); + + const args = summarizeToolValue({ query: marker }); + const result = summarizeToolValue({ text: `Result ${marker}` }); + const [readCall, writeCall] = await db.insert(toolInvocations).values([ + { companyId, connectionId: connection!.id, runId: callingRunId, agentId: base.agent.id, issueId: base.issue.id, toolName: "read_doc", status: "succeeded", + argumentsSummary: args, argumentsHash: args.sha256, resultSummary: result, resultHash: result.sha256, resultSizeBytes: result.sizeBytes, + progress: { message: `Reading ${marker}` } as never, createdAt: created }, + { companyId, connectionId: connection!.id, runId: callingRunId, agentId: base.agent.id, issueId: base.issue.id, toolName: "send_doc", status: "succeeded", + approvalState: "approved", argumentsSummary: args, resultSummary: result, createdAt: created }, + ]).returning(); + await db.insert(toolCallEvents).values([ + { companyId, connectionId: connection!.id, runId: callingRunId, invocationId: readCall!.id, eventType: "call_started", argumentsSummary: args, requestSummary: args, createdAt: created }, + { companyId, connectionId: connection!.id, runId: callingRunId, invocationId: readCall!.id, eventType: "call_completed", resultSummary: result, resultHash: result.sha256, createdAt: created }, + ]); + await db.insert(toolAccessAuditEvents).values({ + companyId, connectionId: connection!.id, action: "tool_access.policy_decision", outcome: "success", createdAt: created, + details: { decision: "allow", runId: callingRunId, toolName: "read_doc", argumentsSummary: args, redactionPlan: { redactedFieldCount: 0 } }, + }); + + const [card] = await db.insert(issueThreadInteractions).values({ + companyId, issueId: base.issue.id, kind: "request_confirmation", status: "resolved", + payload: { version: 1, prompt: "Approve send_doc?", detailsMarkdown: `Will send ${marker}`, + toolAction: { version: 1, toolName: "send_doc", previewMarkdown: `Preview ${marker}`, argumentsSummaryJson: args.summary } } as never, + result: { version: 1, outcome: "accepted", toolAction: { version: 1, status: "executed", resultSummary: result.summary } } as never, + }).returning(); + const [actionRequest] = await db.insert(toolActionRequests).values({ + companyId, invocationId: writeCall!.id, issueId: base.issue.id, interactionId: card!.id, status: "executed", + canonicalArgumentsHash: "args", canonicalArgumentsSummary: args, signedArguments: `signed:${marker}`, previewMarkdown: `Preview ${marker}`, + }).returning(); + + await db.insert(toolMcpAppViews).values({ + companyId, connectionId: connection!.id, issueId: base.issue.id, runId: callingRunId, invocationId: readCall!.id, + applicationName: "Docs", toolName: "read_doc", upstreamToolName: "read_doc", resourceUri: "ui://docs/view", + toolInput: { query: marker }, toolResult: { content: [{ type: "text", text: marker }] }, + modelContext: { content: [{ type: "text", text: marker }] } as never, createdAt: created, + }); + + await db.insert(issueComments).values({ companyId, issueId: base.issue.id, authorAgentId: base.agent.id, body: `Agent note: ${marker}` }); + + return { connection: connection!, actionRequest: actionRequest!, result, callingRunId }; + } + + async function seedDeliveryRun( + base: Awaited>, + logs: ReturnType, + delivered: Array<{ actionRequestId: string; marker: string }>, + ) { + const companyId = base.company.id; + const runId = randomUUID(); + const ids = { companyId, agentId: base.agent.id, runId }; + const text = delivered.map((entry) => `Approved result ${entry.marker}`).join("\n"); + await writeLog(logs.local, ids, text); + const migrated = await writeLog(logs.external, ids, text); + const actions = delivered.map((entry) => ({ actionRequestId: entry.actionRequestId, toolName: "send_doc", resultSummary: `Result ${entry.marker}` })); + const context = { + issueId: base.issue.id, toolAction: actions[0], toolActions: actions, toolActionRequestIds: delivered.map((entry) => entry.actionRequestId), + paperclipAgentMessage: { text: "There are recorded connection outcomes.", untrustedToolResults: actions }, + }; + await db.insert(heartbeatRuns).values({ + id: runId, companyId, agentId: base.agent.id, issueId: base.issue.id, status: "succeeded", finishedAt: new Date(Date.now() - 10_000), + logStore: migrated.store, logRef: migrated.logRef, contextSnapshot: context, resultJson: { summary: text }, + }); + await db.insert(agentWakeupRequests).values({ + companyId, agentId: base.agent.id, source: "automation", status: "completed", + payload: { issueId: base.issue.id, _paperclipWakeContext: context }, + }); + return runId; + } + + async function scan(markers: string[], logs: ReturnType) { + const columns = await db.execute(sql`select table_name, column_name from information_schema.columns + where table_schema = 'public' and data_type in ('text', 'character varying', 'jsonb', 'json')`); + const hits: string[] = []; + for (const row of columns as unknown as Array<{ table_name: string; column_name: string }>) { + const name = `${row.table_name}.${row.column_name}`; + if (OUT_OF_SCOPE_COLUMNS.has(name)) continue; + for (const marker of markers) { + const found = await db.execute(sql`select 1 from ${sql.identifier(row.table_name)} + where ${sql.identifier(row.column_name)}::text like ${`%${marker}%`} limit 1`); + if ((found as unknown as unknown[]).length > 0) hits.push(`${name} contains ${marker}`); + } + } + for (const marker of markers) { + for (const [label, objects] of [["object store", logs.objects.objects], ["legacy mirror", logs.mirror.objects]] as const) { + for (const [key, bytes] of objects) if (bytes.includes(marker)) hits.push(`${label} ${key} contains ${marker}`); + } + for (const file of await filesContaining(logs.dir, marker)) hits.push(`file ${file} contains ${marker}`); + } + return hits; + } + + async function outOfScopeStillHolds(marker: string) { + const rows = await db.execute(sql`select 1 from issue_comments where body like ${`%${marker}%`}`); + return (rows as unknown as unknown[]).length > 0; + } + + async function drain(service: ReturnType, deletionIds: string[]) { + for (let i = 0; i < 200; i++) { + const jobs = await Promise.all(deletionIds.map((id) => service.process(id, new AbortController().signal))); + if (jobs.every((job) => job?.status === "completed")) return; + } + throw new Error("deletion did not finish"); + } + + async function seenBefore(markers: string[], logs: ReturnType) { + const hits = await scan(markers, logs); + for (const marker of markers) expect(hits.some((hit) => hit.includes(marker))).toBe(true); + expect(hits.some((hit) => hit.startsWith("object store"))).toBe(true); + expect(hits.some((hit) => hit.startsWith("legacy mirror"))).toBe(true); + expect(hits.some((hit) => hit.startsWith("file "))).toBe(true); + } + + it("leaves the marker nowhere after a deletion runs to completion", async () => { + const logs = stores(path.join(baseDir, "complete")); + const base = await seedCompany(); + const marker = `CANARY${randomUUID().replace(/-/g, "")}`; + const flow = await seedConnectionFlow(base, logs, marker); + await seedDeliveryRun(base, logs, [{ actionRequestId: flow.actionRequest.id, marker }]); + await seenBefore([marker], logs); + + const service = createConnectionContentDeletion(db, { runLogStore: () => logs.store }); + const before = await service.storedContent(flow.connection); + expect(before.stored).toMatchObject({ toolCalls: 2, approvals: 1, appViews: 1, runs: 2, runsInProgress: 0 }); + const { deletion } = await service.request(flow.connection, { actorType: "user", actorId: "board-user" }); + await drain(service, [deletion.id]); + + expect(await scan([marker], logs)).toEqual([]); + expect(await outOfScopeStillHolds(marker)).toBe(true); + expect((await service.storedContent(flow.connection)).stored).toEqual({ toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 }); + }); + + it("leaves the marker nowhere after an interrupted deletion resumes", async () => { + const logs = stores(path.join(baseDir, "interrupted")); + const base = await seedCompany(); + const marker = `CANARY${randomUUID().replace(/-/g, "")}`; + const flow = await seedConnectionFlow(base, logs, marker); + await seedDeliveryRun(base, logs, [{ actionRequestId: flow.actionRequest.id, marker }]); + await seenBefore([marker], logs); + + let failures = 2; + const flaky: RunLogStore = { + ...logs.store, + remove: async (handle: RunLogHandle, signal?: AbortSignal) => { + if (failures > 0) { + failures -= 1; + throw new Error("object storage unavailable"); + } + return logs.store.remove!(handle, signal); + }, + }; + const service = createConnectionContentDeletion(db, { runLogStore: () => flaky }); + const { deletion } = await service.request(flow.connection, { actorType: "user", actorId: "board-user" }); + let interrupted = 0; + for (let i = 0; i < 200; i++) { + const job = await service.process(deletion.id, new AbortController().signal).catch(() => { + interrupted += 1; + return null; + }); + if (job?.status === "completed") break; + } + expect(interrupted).toBe(2); + expect(await scan([marker], logs)).toEqual([]); + expect(await outOfScopeStillHolds(marker)).toBe(true); + }); + + it("leaves no marker after two deletions for different connections run at the same time on one task", async () => { + const logs = stores(path.join(baseDir, "concurrent")); + const base = await seedCompany(); + const first = `CANARY${randomUUID().replace(/-/g, "")}`; + const second = `CANARY${randomUUID().replace(/-/g, "")}`; + const flowOne = await seedConnectionFlow(base, logs, first); + const flowTwo = await seedConnectionFlow(base, logs, second); + await seedDeliveryRun(base, logs, [ + { actionRequestId: flowOne.actionRequest.id, marker: first }, + { actionRequestId: flowTwo.actionRequest.id, marker: second }, + ]); + await seenBefore([first, second], logs); + + const service = createConnectionContentDeletion(db, { runLogStore: () => logs.store }); + const one = await service.request(flowOne.connection, { actorType: "user", actorId: "board-user" }); + const two = await service.request(flowTwo.connection, { actorType: "user", actorId: "board-user" }); + await drain(service, [one.deletion.id, two.deletion.id]); + + expect(await scan([first, second], logs)).toEqual([]); + expect(await outOfScopeStillHolds(first)).toBe(true); + expect(await outOfScopeStillHolds(second)).toBe(true); + }); +}); diff --git a/server/src/__tests__/tool-connection-content-deletion.test.ts b/server/src/__tests__/tool-connection-content-deletion.test.ts new file mode 100644 index 00000000000..c7123f12d61 --- /dev/null +++ b/server/src/__tests__/tool-connection-content-deletion.test.ts @@ -0,0 +1,420 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { and, eq, sql } from "drizzle-orm"; +import { + activityLog, + agentWakeupRequests, + agents, + companies, + createDb, + heartbeatRunEvents, + heartbeatRuns, + issues, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolApplications, + toolCallEvents, + toolConnectionContentDeletions, + toolConnections, + toolInvocations, + toolMcpAppViews, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import type { RunLogHandle, RunLogStore } from "../services/run-log-store.js"; +import { toolAccessRoutes } from "../routes/tool-access.js"; +import { errorHandler } from "../middleware/index.js"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +const SECRET = "Quarterly board notes from the shared drive"; + +function memoryRunLogs() { + const logs = new Map(); + let failuresLeft = 0; + const store = { + remove: async (handle: RunLogHandle) => { + if (failuresLeft > 0) { + failuresLeft -= 1; + throw new Error("object storage unavailable"); + } + logs.delete(handle.logRef); + }, + } as unknown as RunLogStore; + return { logs, store, failNext: (count: number) => { failuresLeft = count; } }; +} + +describeEmbeddedPostgres("deleting content stored from a connection's tool calls", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-content-deletion-"); + db = createDb(tempDb.connectionString); + }, 30_000); + + afterAll(async () => { + await tempDb?.cleanup(); + }); + + async function seedCompany(runLogs: ReturnType) { + const [company] = await db.insert(companies).values({ + name: `Deletion ${randomUUID()}`, + issuePrefix: `DL${randomUUID().slice(0, 6).toUpperCase()}`, + }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company!.id, + name: `Agent ${randomUUID().slice(0, 6)}`, + role: "engineer", + status: "active", + adapterType: "process", + adapterConfig: {}, + runtimeConfig: {}, + }).returning(); + const [application] = await db.insert(toolApplications).values({ + companyId: company!.id, + name: `Docs ${randomUUID().slice(0, 6)}`, + type: "mcp_http", + }).returning(); + const [connection, other] = await db.insert(toolConnections).values([ + { companyId: company!.id, applicationId: application!.id, name: "Docs", uid: randomUUID(), transport: "mcp_remote", status: "active" }, + { companyId: company!.id, applicationId: application!.id, name: "Other", uid: randomUUID(), transport: "mcp_remote", status: "active" }, + ]).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Read the notes" }).returning(); + const finishedAt = new Date(Date.now() - 60_000); + const [settledRun, liveRun, otherRun] = await db.insert(heartbeatRuns).values([ + { companyId: company!.id, agentId: agent!.id, status: "succeeded", finishedAt, logStore: "local_file", logRef: `${company!.id}/settled.ndjson`, logBytes: 120, logSha256: "abc", stdoutExcerpt: SECRET, resultJson: { summary: SECRET, executionRecovery: { kind: "none" }, nativeResult: { schema: "paperclip.run_result.v1", summary: SECRET, continuation: { kind: "response_wake", summary: SECRET } } } }, + { companyId: company!.id, agentId: agent!.id, status: "running", logStore: "local_file", logRef: `${company!.id}/live.ndjson`, stdoutExcerpt: SECRET }, + { companyId: company!.id, agentId: agent!.id, status: "succeeded", finishedAt, logStore: "local_file", logRef: `${company!.id}/other.ndjson`, stdoutExcerpt: SECRET }, + ]).returning(); + for (const run of [settledRun!, liveRun!, otherRun!]) runLogs.logs.set(run.logRef!, SECRET); + const summary = { summary: SECRET, sizeBytes: 42, sha256: "hash-42", redactedFields: [] }; + const created = new Date(Date.now() - 30_000); + const [settledCall, liveCall, otherCall] = await db.insert(toolInvocations).values([ + { companyId: company!.id, connectionId: connection!.id, runId: settledRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, resultHash: "hash-42", resultSizeBytes: 42, createdAt: created }, + { companyId: company!.id, connectionId: connection!.id, runId: liveRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, createdAt: created }, + { companyId: company!.id, connectionId: other!.id, runId: otherRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, createdAt: created }, + ]).returning(); + await db.insert(toolCallEvents).values([ + { companyId: company!.id, connectionId: connection!.id, runId: settledRun!.id, invocationId: settledCall!.id, eventType: "call_completed", requestSummary: summary, resultSummary: summary, resultHash: "hash-42", createdAt: created }, + { companyId: company!.id, connectionId: connection!.id, runId: liveRun!.id, invocationId: liveCall!.id, eventType: "call_completed", resultSummary: summary, createdAt: created }, + { companyId: company!.id, connectionId: other!.id, runId: otherRun!.id, invocationId: otherCall!.id, eventType: "call_completed", resultSummary: summary, createdAt: created }, + ]); + const [interaction] = await db.insert(issueThreadInteractions).values({ + companyId: company!.id, + issueId: issue!.id, + kind: "request_confirmation", + status: "resolved", + payload: { version: 1, prompt: "Approve?", detailsMarkdown: SECRET, toolAction: { version: 1, previewMarkdown: SECRET, argumentsSummaryJson: SECRET } } as never, + result: { version: 1, toolAction: { version: 1, status: "executed", resultSummary: SECRET } } as never, + }).returning(); + const [actionRequest] = await db.insert(toolActionRequests).values({ + companyId: company!.id, + issueId: issue!.id, + invocationId: settledCall!.id, + interactionId: interaction!.id, + status: "executed", + canonicalArgumentsHash: "args-hash", + canonicalArgumentsSummary: summary, + signedArguments: "signed", + previewMarkdown: SECRET, + }).returning(); + const delivered = { actionRequestId: actionRequest!.id, toolName: "read_doc", resultSummary: SECRET }; + const deliveryContext = { issueId: issue!.id, toolAction: delivered, toolActions: [delivered], toolActionRequestIds: [actionRequest!.id], paperclipAgentMessage: { untrustedToolResults: [delivered] } }; + const [deliveryRun] = await db.insert(heartbeatRuns).values({ + companyId: company!.id, agentId: agent!.id, issueId: issue!.id, status: "succeeded", finishedAt, contextSnapshot: deliveryContext, + logStore: "local_file", logRef: `${company!.id}/delivery.ndjson`, stdoutExcerpt: SECRET, + }).returning(); + runLogs.logs.set(deliveryRun!.logRef!, SECRET); + const [deliveryWake] = await db.insert(agentWakeupRequests).values({ + companyId: company!.id, agentId: agent!.id, source: "automation", status: "completed", + payload: { issueId: issue!.id, _paperclipWakeContext: deliveryContext }, + }).returning(); + const [accessAudit] = await db.insert(toolAccessAuditEvents).values({ + companyId: company!.id, connectionId: connection!.id, action: "tool_access.policy_decision", outcome: "success", + details: { decision: "allow", runId: settledRun!.id, toolName: "read_doc", argumentsSummary: summary }, + createdAt: created, + }).returning(); + await db.insert(toolMcpAppViews).values({ + companyId: company!.id, + connectionId: connection!.id, + runId: settledRun!.id, + invocationId: settledCall!.id, + applicationName: "Docs", + toolName: "read_doc", + upstreamToolName: "read_doc", + resourceUri: "ui://docs/view", + toolInput: { id: "doc-1" }, + toolResult: { text: SECRET }, + createdAt: created, + }); + await db.insert(heartbeatRunEvents).values([ + { companyId: company!.id, runId: settledRun!.id, agentId: agent!.id, seq: 1, eventType: "lifecycle", message: "run started", payload: { phase: "start" } }, + { companyId: company!.id, runId: settledRun!.id, agentId: agent!.id, seq: 2, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + { companyId: company!.id, runId: liveRun!.id, agentId: agent!.id, seq: 1, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + { companyId: company!.id, runId: otherRun!.id, agentId: agent!.id, seq: 1, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + ]); + return { deliveryRun: deliveryRun!, deliveryWake: deliveryWake!, accessAudit: accessAudit!, interaction: interaction!, company: company!, agent: agent!, connection: connection!, other: other!, settledRun: settledRun!, liveRun: liveRun!, otherRun: otherRun!, settledCall: settledCall!, liveCall: liveCall!, otherCall: otherCall! }; + } + + async function drain(service: ReturnType) { + for (let i = 0; i < 20 && await service.nextDeadline() !== null; i++) { + await service.processPending(new AbortController().signal); + } + } + + it("removes content from every store, keeps rows and hashes, and leaves other connections and companies alone", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const foreign = await seedCompany(runLogs); + + const before = await service.storedContent(seed.connection); + expect(before.stored).toEqual({ toolCalls: 2, approvals: 1, appViews: 1, runs: 3, runsInProgress: 1 }); + + const { created } = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + expect(created).toBe(true); + await drain(service); + + const [call] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, seed.settledCall.id)); + expect(call!.resultSummary).toMatchObject({ summary: "", sizeBytes: 42, sha256: "hash-42" }); + expect(call!.resultSummary?.contentDeletedAt).toBeTruthy(); + expect(call!.argumentsSummary?.summary).toBe(""); + expect(call!.resultHash).toBe("hash-42"); + expect(call!.status).toBe("succeeded"); + + const events = await db.select().from(toolCallEvents).where(eq(toolCallEvents.invocationId, seed.settledCall.id)); + expect(events).toHaveLength(1); + expect(events[0]!.resultSummary?.summary).toBe(""); + expect(events[0]!.requestSummary?.summary).toBe(""); + expect(events[0]!.resultHash).toBe("hash-42"); + + const [actionRequest] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.invocationId, seed.settledCall.id)); + expect(actionRequest!.previewMarkdown).toBeNull(); + expect(actionRequest!.signedArguments).toBeNull(); + expect(actionRequest!.canonicalArgumentsSummary.summary).toBe(""); + expect(actionRequest!.canonicalArgumentsHash).toBe("args-hash"); + const [interaction] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, actionRequest!.interactionId!)); + expect(JSON.stringify(interaction!.payload)).not.toContain(SECRET); + expect(JSON.stringify(interaction!.result)).not.toContain(SECRET); + + expect(await db.select().from(toolMcpAppViews).where(eq(toolMcpAppViews.connectionId, seed.connection.id))).toEqual([]); + + const [settledRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(settledRun).toMatchObject({ logStore: null, logRef: null, stdoutExcerpt: null, logBytes: 120, logSha256: "abc", resultJson: { executionRecovery: { kind: "none" }, nativeResult: { schema: "paperclip.run_result.v1", continuation: { kind: "response_wake" } } } }); + expect(JSON.stringify(settledRun!.resultJson)).not.toContain(SECRET); + const [audit] = await db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.id, seed.accessAudit.id)); + expect(audit!.details).toMatchObject({ decision: "allow", toolName: "read_doc", argumentsSummary: { summary: "", sha256: "hash-42", sizeBytes: 42 } }); + const [deliveryRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.deliveryRun.id)); + expect(JSON.stringify(deliveryRun!.contextSnapshot)).not.toContain(SECRET); + expect(deliveryRun!.contextSnapshot).toMatchObject({ toolActionRequestIds: [expect.any(String)] }); + expect(deliveryRun).toMatchObject({ logRef: null, stdoutExcerpt: null }); + expect(runLogs.logs.has(seed.deliveryRun.logRef!)).toBe(false); + const [deliveryWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, seed.deliveryWake.id)); + expect(JSON.stringify(deliveryWake!.payload)).not.toContain(SECRET); + expect(runLogs.logs.has(seed.settledRun.logRef!)).toBe(false); + const settledEvents = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seed.settledRun.id)); + expect(settledEvents).toHaveLength(2); + expect(JSON.stringify(settledEvents)).not.toContain(SECRET); + expect(settledEvents.find((event) => event.eventType === "lifecycle")!.message).toBe("run started"); + + const [liveRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.liveRun.id)); + expect(liveRun!.logRef).toBe(seed.liveRun.logRef); + expect(liveRun!.stdoutExcerpt).toBe(SECRET); + expect(runLogs.logs.get(seed.liveRun.logRef!)).toBe(SECRET); + const [liveCall] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, seed.liveCall.id)); + expect(liveCall!.resultSummary?.summary).toBe(SECRET); + const liveEvents = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seed.liveRun.id)); + expect(liveEvents[0]!.message).toBe(SECRET); + + for (const untouched of [seed.otherCall.id, foreign.settledCall.id]) { + const [row] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, untouched)); + expect(row!.resultSummary?.summary).toBe(SECRET); + } + for (const run of [seed.otherRun, foreign.settledRun]) { + const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(row!.logRef).toBe(run.logRef); + expect(runLogs.logs.get(run.logRef!)).toBe(SECRET); + } + expect(await db.select().from(toolMcpAppViews).where(eq(toolMcpAppViews.connectionId, foreign.connection.id))).toHaveLength(1); + + const after = await service.storedContent(seed.connection); + expect(after.stored).toEqual({ toolCalls: 1, approvals: 0, appViews: 0, runs: 1, runsInProgress: 1 }); + expect(after.deletion).toMatchObject({ + status: "completed", + counts: { toolCalls: 1, approvals: 1, auditEvents: 2, appViews: 1, runs: 2, runsInProgress: 1 }, + }); + const [activity] = await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deleted"), + )); + expect(activity!.entityId).toBe(seed.connection.id); + expect(JSON.stringify(activity!.details)).not.toContain(SECRET); + + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, seed.liveRun.id)); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + await drain(service); + expect(runLogs.logs.has(seed.liveRun.logRef!)).toBe(false); + expect((await service.storedContent(seed.connection)).stored).toEqual({ toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 }); + }); + + it("resumes an interrupted deletion to completion on the next pass", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + runLogs.failNext(1); + await expect(service.processPending(new AbortController().signal)).rejects.toThrow("object storage unavailable"); + + const [interrupted] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.connection.id)); + expect(interrupted).toMatchObject({ status: "pending", attempts: 1, lastError: "object storage unavailable" }); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "pending", retrying: true }); + expect(await service.nextDeadline()).not.toBeNull(); + + await drain(service); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(run!.logRef).toBeNull(); + expect(runLogs.logs.has(seed.settledRun.logRef!)).toBe(false); + const finished = await service.storedContent(seed.connection); + expect(finished.deletion).toMatchObject({ status: "completed", retrying: false, counts: { toolCalls: 1, runs: 2 } }); + }); + + it("returns the pending deletion instead of starting a second one", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const first = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + const second = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + expect(second).toEqual({ deletion: first.deletion, created: false }); + await drain(service); + }); + + it("clears the approval card on the retry after the card update failed once", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await db.execute(sql`create table if not exists content_deletion_card_failures (remaining int not null)`); + await db.execute(sql`delete from content_deletion_card_failures`); + await db.execute(sql`insert into content_deletion_card_failures values (1)`); + await db.execute(sql`create or replace function content_deletion_fail_card() returns trigger language plpgsql as $$ + begin + if exists (select 1 from content_deletion_card_failures where remaining > 0) then + raise exception 'card update failed'; + end if; + return new; + end $$`); + await db.execute(sql`create trigger content_deletion_fail_card before update on issue_thread_interactions + for each row execute function content_deletion_fail_card()`); + try { + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + await expect(service.processPending(new AbortController().signal)).rejects.toThrow(); + const [pending] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.connection.id)); + expect(pending!.status).toBe("pending"); + await db.execute(sql`delete from content_deletion_card_failures`); + await drain(service); + } finally { + await db.execute(sql`drop trigger if exists content_deletion_fail_card on issue_thread_interactions`); + } + const [card] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, seed.interaction.id)); + expect(JSON.stringify(card!.payload)).not.toContain(SECRET); + expect(JSON.stringify(card!.result)).not.toContain(SECRET); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "completed", counts: { approvals: 1, toolCalls: 1 } }); + }); + + it("keeps the job pending and the run log reference when the log backend is not recognised", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await db.update(heartbeatRuns).set({ logStore: "tape_archive" }).where(eq(heartbeatRuns.id, seed.settledRun.id)); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + for (let i = 0; i < 5; i++) await service.processPending(new AbortController().signal).catch(() => undefined); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(run).toMatchObject({ logStore: "tape_archive", logRef: seed.settledRun.logRef }); + const state = await service.storedContent(seed.connection); + expect(state.deletion).toMatchObject({ status: "pending", retrying: true }); + expect(await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deleted"), + ))).toEqual([]); + }); + + it("yields after every step when the time budget is spent and finishes over later passes", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const { deletion } = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + const phases: string[] = []; + for (let i = 0; i < 40; i++) { + const job = await service.process(deletion.id, new AbortController().signal, 0); + if (!job || job.status === "completed") break; + phases.push(String((job.progress as { phase: string }).phase)); + } + expect(phases.length).toBeGreaterThan(5); + expect(new Set(phases)).toEqual(new Set(["events", "auditEvents", "appViews", "deliveredResults", "runs", "done"])); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "completed", counts: { runs: 2 } }); + }); + + describe("routes", () => { + function app(actor: Express.Request["actor"]) { + const server = express(); + server.use(express.json()); + server.use((req, _res, next) => { req.actor = actor; next(); }); + server.use("/api", toolAccessRoutes(db)); + server.use(errorHandler); + return server; + } + + it("lets a board user see counts and request deletion, and refuses an agent and another company", async () => { + const runLogs = memoryRunLogs(); + const seed = await seedCompany(runLogs); + const board = app({ type: "board", userId: "board-user", userName: "Board", userEmail: null, isInstanceAdmin: true, source: "local_implicit" } as never); + + const counts = await request(board).get(`/api/tool-connections/${seed.connection.id}/stored-content`).expect(200); + expect(counts.body.stored).toEqual({ toolCalls: 2, approvals: 1, appViews: 1, runs: 3, runsInProgress: 1 }); + + const requested = await request(board).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`).expect(202); + expect(requested.body.deletion).toMatchObject({ status: "pending" }); + const [activity] = await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deletion_requested"), + )); + expect(activity!.details).toMatchObject({ toolCalls: 2, runs: 3, runsInProgress: 1 }); + const status = await request(board).get(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`).expect(200); + expect(status.body.deletion).toMatchObject({ status: "pending" }); + + const outsider = app({ + type: "board", + userId: "outsider", + userName: "Outsider", + userEmail: null, + isInstanceAdmin: false, + source: "session", + companyIds: [randomUUID()], + memberships: [], + } as never); + const refused = await request(outsider).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`); + expect([403, 404]).toContain(refused.status); + + const agent = app({ type: "agent", agentId: seed.agent.id, companyId: seed.company.id, runId: null, source: "agent_key" } as never); + const agentRefused = await request(agent).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`); + expect([401, 403]).toContain(agentRefused.status); + }); + + it("queues the deletion when a board user removes the connection with it", async () => { + const runLogs = memoryRunLogs(); + const seed = await seedCompany(runLogs); + const board = app({ type: "board", userId: "board-user", userName: "Board", userEmail: null, isInstanceAdmin: true, source: "local_implicit" } as never); + const removed = await request(board).delete(`/api/tool-connections/${seed.other.id}?deleteStoredContent=true`).expect(200); + expect(removed.body.contentDeletion).toMatchObject({ status: "pending" }); + const [pending] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.other.id)); + expect(pending!.status).toBe("pending"); + }); + }); +}); diff --git a/server/src/index.ts b/server/src/index.ts index 4efa6af8612..3488284e080 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -18,6 +18,7 @@ import { registerMergeConfirmationWork } from "./services/merged-pr-confirmation import { registerNativeHarnessBackupExpirationWork } from "./services/native-runtime/native-harness-backup-expiration-work.js"; import { registerGitHubConnectionEventWork, registerGitHubConnectionContinuityWork } from "./services/github-connection-work.js"; import { registerProviderTraceRetentionWork } from "./services/provider-trace-retention-work.js"; +import { registerToolConnectionContentDeletionWork } from "./services/tool-connection-content-deletion-work.js"; import { registerRunEvidenceMigrationWork } from "./services/run-evidence-migration-work.js"; import { registerSkillVersionRetentionWork } from "./services/skill-version-retention-work.js"; import { registerDatabaseBackupWork } from "./services/database-backup-work.js"; @@ -1446,6 +1447,10 @@ async function startServerWithDatabaseTeardown( () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()); trackHeartbeatSchedulerWork(providerTraceRetentionWork.ready); server.on("close", providerTraceRetentionWork.stop); + const toolConnectionContentDeletionWork = registerToolConnectionContentDeletionWork(db, deliveryWork, + () => !heartbeatSchedulerStopped && !isWarmStandby()); + trackHeartbeatSchedulerWork(toolConnectionContentDeletionWork.ready); + server.on("close", toolConnectionContentDeletionWork.stop); const executionSchedule = executionControlSchedule(db); const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager }); const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled @@ -2276,6 +2281,7 @@ async function startServerWithDatabaseTeardown( await runEvidenceMigrationWork?.stop(); }); await preDrain.run("provider_trace_retention_stop", () => providerTraceRetentionWork.stop()); + await preDrain.run("tool_connection_content_deletion_stop", () => toolConnectionContentDeletionWork.stop()); await preDrain.run("skill_version_retention_dispose", async () => { await skillVersionRetentionWorker?.dispose(); }); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 101c9bf9d38..1e10c07ea01 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -12574,6 +12574,35 @@ registerCurrentRoute({ path: "/api/tool-connections/{connectionId}", tags: ["tool-access"], summary: "Archive a tool connection", + query: z.object({ deleteStoredContent: z.enum(["true", "false"]).optional() }), +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "get", + path: "/api/tool-connections/{connectionId}/stored-content", + tags: ["tool-access"], + summary: "Count the content Paperclip stored from a connection's tool calls", +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/tool-connections/{connectionId}/stored-content/deletion", + tags: ["tool-access"], + summary: "Delete the content Paperclip stored from a connection's tool calls", + responses: { 202: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "get", + path: "/api/tool-connections/{connectionId}/stored-content/deletion", + tags: ["tool-access"], + summary: "Read the status of the latest stored-content deletion for a connection", }); const MACHINE_SIGN_IN_ERRORS = { diff --git a/server/src/routes/tool-access.ts b/server/src/routes/tool-access.ts index 146d1f61b4a..0faa1a9836d 100644 --- a/server/src/routes/tool-access.ts +++ b/server/src/routes/tool-access.ts @@ -71,6 +71,8 @@ import { accessService, logActivity, toolAccessPolicyService, toolAccessService, import { ToolGatewayHttpError, type ToolGatewayService } from "../services/tool-gateway.js"; import { authorizationService } from "../services/authorization.js"; import { mcpAppViewsService } from "../services/mcp-app-views.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import { getRunLogStore } from "../services/run-log-store.js"; import { listCompanyProviderToolIcons } from "../services/tool-access/catalog/icons.js"; import { mcpAppToolCallSchema } from "@paperclipai/shared"; import { RailwayError } from "../services/railway.js"; @@ -544,6 +546,7 @@ function connectorEnrollmentPrincipal(req: Request): string { const access = accessService(db); const managementGuard = connectionManagementGuard(db, "tool"); + const contentDeletion = createConnectionContentDeletion(db, { runLogStore: getRunLogStore }); async function assertBoardToolPermission(req: Request, companyId: string, permissionKey: PermissionKey) { if (req.actor.type === "agent") { @@ -2371,6 +2374,8 @@ function connectorEnrollmentPrincipal(req: Request): string { await assertToolConnectionConfigureAccess(req, existing); if (aiConnectionRouterPluginKey(existing)) throw badRequest("Remove this connection pool through its revision-checked pool settings."); const applicationBefore = await svc.getApplication(existing.applicationId); + const deleteStoredContent = req.query.deleteStoredContent === "true"; + if (deleteStoredContent) assertBoard(req); const { connection, removal } = await svc.archiveConnection( existing.id, existing.companyId, @@ -2402,7 +2407,54 @@ function connectorEnrollmentPrincipal(req: Request): string { details: { type: applicationAfter.type, name: applicationAfter.name, reason: "last_connection_removed" }, }); } - res.json({ ...connection, removal }); + const contentDeletion = deleteStoredContent + ? await requestStoredContentDeletion(req, connection) + : null; + res.json({ ...connection, removal, ...(contentDeletion ? { contentDeletion } : {}) }); + }); + + async function requestStoredContentDeletion(req: Request, connection: { id: string; companyId: string }) { + const actor = getActorInfo(req); + const { deletion, created } = await contentDeletion.request(connection, { + actorType: actor.actorType, + actorId: actor.actorId, + }); + if (created) { + const { stored } = await contentDeletion.storedContent(connection); + await logActivity(db, { + companyId: connection.companyId, + ...connectionRequestActor(req), + action: "tool_connection.content_deletion_requested", + entityType: "tool_connection", + entityId: connection.id, + details: { ...stored }, + }); + } + return deletion; + } + + router.get("/tool-connections/:connectionId/stored-content", managementGuard("connections:configure", "GET /tool-connections/:connectionId/stored-content", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.json(await contentDeletion.storedContent(existing)); + }); + + router.post("/tool-connections/:connectionId/stored-content/deletion", managementGuard("connections:configure", "POST /tool-connections/:connectionId/stored-content/deletion", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.status(202).json({ deletion: await requestStoredContentDeletion(req, existing) }); + }); + + router.get("/tool-connections/:connectionId/stored-content/deletion", managementGuard("connections:configure", "GET /tool-connections/:connectionId/stored-content/deletion", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.json({ deletion: await contentDeletion.latestDeletion(existing) }); }); router.post("/tool-connections/:connectionId/health-check", managementGuard("connections:configure", "POST /tool-connections/:connectionId/health-check", { personal: true }), async (req, res) => { diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index e419a441e56..66e5c3821d7 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -79,6 +79,7 @@ export const DELIVERY_QUEUES = { decisionArchiveDelivery: "decision-archive-delivery", toolConnectionHealth: "tool-connection-health", + toolConnectionContentDeletion: "tool-connection-content-deletion", toolReviewProjection: "tool-review-projection", toolReviewInterruption: "tool-review-interruption", toolReviewRecovery: "tool-review-recovery", diff --git a/server/src/services/object-run-log-store.test.ts b/server/src/services/object-run-log-store.test.ts index cc817d72f02..8187fbec7ef 100644 --- a/server/src/services/object-run-log-store.test.ts +++ b/server/src/services/object-run-log-store.test.ts @@ -66,6 +66,18 @@ const event = (seq: number) => ({ }); describe("S3 run log store", () => { + it("removes every object of a finalized log and stays removable", async () => { + const m = memory(), store = createObjectRunLogStore({ provider: m.provider, keyPrefix: "logs", + assertWriterAuthority: async () => undefined }); + const handle = await store.begin(input); + await store.append(handle, { ...event(1), chunk: "tool result content" }); + await store.finalize(handle); + expect([...m.objects.keys()].some((key) => key.includes(handle.logRef))).toBe(true); + await store.remove!(handle); + await store.remove!(handle); + expect([...m.objects.keys()].filter((key) => key.includes(handle.logRef) && !key.endsWith("/head.json"))).toEqual([]); + await expect(store.read(handle)).rejects.toThrow("Run log not found"); + }); it("awaits a successor flush started by another drain waiter before finalizing", async () => { const m = memory(), store = createObjectRunLogStore({ provider: m.provider, keyPrefix: "logs", flushMs: 1000, segmentBytes: 1024, assertWriterAuthority: async () => undefined }); diff --git a/server/src/services/object-run-log-store.ts b/server/src/services/object-run-log-store.ts index 8ffee47f9ad..00b16d092f4 100644 --- a/server/src/services/object-run-log-store.ts +++ b/server/src/services/object-run-log-store.ts @@ -399,6 +399,10 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL nextOffset: nextOffset ?? offset + bytes.length, }; }, + async remove(handle, signal) { + if (handle.store !== "s3_segments" && handle.store !== "s3_archive") throw new Error("Invalid object run log backend"); + await journal(handle.logRef, "background").remove(signal); + }, async flushInflightMirrors() { await Promise.all([...writers.values()].map(flush)); }, diff --git a/server/src/services/run-log-store.test.ts b/server/src/services/run-log-store.test.ts index aee7e202198..cf3b8546f25 100644 --- a/server/src/services/run-log-store.test.ts +++ b/server/src/services/run-log-store.test.ts @@ -4,7 +4,8 @@ import path from "node:path"; import os from "node:os"; import { createHash } from "node:crypto"; import { Readable } from "node:stream"; -import { createDurableRunLogStore } from "./run-log-store.js"; +import { composeRunLogStore, createDurableRunLogStore } from "./run-log-store.js"; +import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js"; import type { StorageProvider } from "../storage/types.js"; // In-memory StorageProvider stand-in: durable, survives the "pod roll" (local @@ -68,6 +69,46 @@ afterEach(async () => { const begin = { companyId: "co1", agentId: "ag1", runId: "run1" }; describe("createDurableRunLogStore", () => { + it("removes a migrated run's original local log and mirror along with its object journal", async () => { + const { provider, objects } = createMemoryProvider(); + const local = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); + const external = objectLogFixture(); + const store = composeRunLogStore(local, external); + const original = await local.begin(begin); + await local.append(original, { stream: "stdout", chunk: "tool result content", ts: "t1" }); + await local.finalize(original); + const migrated = await external.begin(begin); + await external.append(migrated, { stream: "stdout", chunk: "tool result content", ts: "t1", seq: 1 }); + await external.finalize(migrated); + expect(objects.has(original.logRef)).toBe(true); + await store.remove!(migrated); + await store.remove!(migrated); + expect(objects.has(original.logRef)).toBe(false); + await expect(fs.stat(path.join(baseDir, original.logRef))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(external.read(migrated)).rejects.toThrow("Run log not found"); + await expect(store.remove!({ store: "s3_segments", logRef: "../escape" })).rejects.toThrow("Invalid run log reference"); + }); + + it("refuses to report an object log removed when object storage is not configured", async () => { + const local = createDurableRunLogStore({ basePath: baseDir }); + const store = composeRunLogStore(local, null); + await expect(store.remove!({ store: "s3_segments", logRef: "co1/ag1/run1" })).rejects.toThrow("Object run log storage is not configured"); + }); + + it("removes a finalized log from local disk and its object mirror", async () => { + const { provider, objects } = createMemoryProvider(); + const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); + const handle = await store.begin(begin); + await store.append(handle, { stream: "stdout", chunk: "tool result content", ts: "t1" }); + await store.finalize(handle); + expect(objects.has(handle.logRef)).toBe(true); + await store.remove!(handle); + await store.remove!(handle); + expect(objects.has(handle.logRef)).toBe(false); + await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(store.read(handle)).rejects.toThrow("Run log not found"); + }); + it.each([false, true])("reads a complete long final record from a compact tail (S3 fallback: %s)", async (remote) => { const { provider, objects, calls } = createMemoryProvider(); const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); diff --git a/server/src/services/run-log-store.ts b/server/src/services/run-log-store.ts index 2138026afc7..495805081ef 100644 --- a/server/src/services/run-log-store.ts +++ b/server/src/services/run-log-store.ts @@ -85,6 +85,7 @@ export interface RunLogStore { recordVerifiedSourcePrefix?(handle: RunLogHandle, sourceCursor: number): Promise; finalize(handle: RunLogHandle, options?: { computeDigest?: boolean }): Promise; read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise; + remove?(handle: RunLogHandle, signal?: AbortSignal): Promise; // Optional so existing fakes/fixtures keep compiling: uploads every dirty // in-flight mirror immediately (graceful-shutdown path). No-op when the // in-flight mirror is not enabled. @@ -494,6 +495,14 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru }; }, + async remove(handle, signal) { + if (handle.store !== "local_file") throw new Error("Invalid local run log backend"); + signal?.throwIfAborted(); + await retireInflightMirror(handle.logRef); + await fs.rm(resolveWithin(basePath, handle.logRef), { force: true }); + if (s3) await s3.provider.deleteObject({ objectKey: s3Key(handle.logRef), signal }); + }, + async flushInflightMirrors() { if (!s3 || inflightMirrorMs <= 0) return; const flushEntry = async (logRef: string, entry: InflightMirrorEntry) => { @@ -558,20 +567,13 @@ export function resolveRunLogS3(): DurableRunLogStoreOptions["s3"] { }; } -let cachedStore: RunLogStore | null = null; +export function runLogBaseRef(handle: RunLogHandle): string { + const base = handle.store === "local_file" ? handle.logRef.replace(/\.ndjson$/, "") : handle.logRef.replace(/\.archive$/, ""); + if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(base)) throw new Error("Invalid run log reference"); + return base; +} -export function getRunLogStore() { - if (cachedStore) return cachedStore; - const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs"); - const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() }); - const s3 = resolveRunEvidenceStorage(); - const external = s3 ? createObjectRunLogStore({ - provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs", - assertWriterAuthority: async input => { - if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority"); - await input.assertWriterAuthority(); - }, - }) : null; +export function composeRunLogStore(local: RunLogStore, external: RunLogStore | null): RunLogStore { const writer = external ?? local; function writable(handle: RunLogHandle) { if (handle.store === "s3_archive") throw new Error("Historical run log archives are read-only"); @@ -579,7 +581,7 @@ export function getRunLogStore() { if (!external) throw new Error("Object run log storage is not configured"); return external; } - cachedStore = { + return { begin: input => { if (input.previousStore === "s3_archive") throw new Error("Historical run log archives are read-only"); if (input.previousStore === "local_file") return local.begin(input); @@ -603,8 +605,36 @@ export function getRunLogStore() { } return local.read(handle, opts); }, + remove: async (handle, signal) => { + if (!local.remove) throw new Error("Run log removal is unavailable"); + const base = runLogBaseRef(handle); + if (external?.remove) { + await external.remove({ store: "s3_segments", logRef: base }, signal); + await external.remove({ store: "s3_archive", logRef: `${base}.archive` }, signal); + } else if (handle.store !== "local_file") { + throw new Error("Object run log storage is not configured"); + } + await local.remove({ store: "local_file", logRef: `${base}.ndjson` }, signal); + }, flushInflightMirrors: async () => { await Promise.all([local.flushInflightMirrors?.(), external?.flushInflightMirrors?.()]); }, }; +} + +let cachedStore: RunLogStore | null = null; + +export function getRunLogStore() { + if (cachedStore) return cachedStore; + const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs"); + const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() }); + const s3 = resolveRunEvidenceStorage(); + const external = s3 ? createObjectRunLogStore({ + provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs", + assertWriterAuthority: async input => { + if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority"); + await input.assertWriterAuthority(); + }, + }) : null; + cachedStore = composeRunLogStore(local, external); return cachedStore; } diff --git a/server/src/services/tool-access/connections/content-deletion.ts b/server/src/services/tool-access/connections/content-deletion.ts new file mode 100644 index 00000000000..063d2dbd0de --- /dev/null +++ b/server/src/services/tool-access/connections/content-deletion.ts @@ -0,0 +1,761 @@ +import { and, asc, desc, eq, gt, inArray, lte, sql, type SQL } from "drizzle-orm"; +import { + agentWakeupRequests, + heartbeatRunEvents, + heartbeatRuns, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolCallEvents, + toolConnectionContentDeletions, + toolInvocations, + toolMcpAppViews, + type Db, +} from "@paperclipai/db"; +import type { + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionStoredContent, +} from "@paperclipai/shared"; +import { isRunLogStoreType, type RunLogHandle, type RunLogStore } from "../../run-log-store.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../delivery-work-notifications.js"; +import { logActivity } from "../../activity-log.js"; + +const BATCH_SIZE = 200; +const EVENT_BATCH_SIZE = 1000; +const STEP_BUDGET_MS = 10_000; + +const SETTLED_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]; +const UNSETTLED_INVOCATION_STATUSES = ["pending", "authorized", "awaiting_approval", "executing"]; +const UNSETTLED_ACTION_REQUEST_STATUSES = ["pending", "approved", "executing"]; +const UNSETTLED_WAKE_STATUSES = ["queued", "deferred_issue_execution", "claimed"]; +const RESULT_TEXT_KEYS = ["summary", "result", "message", "text", "output", "content", "body"]; +const STRUCTURAL_EVENT_TYPES = [ + "lifecycle", + "run.terminal", + "run.presentation.resolved", + "adapter.invoke", + "usage.reported", + "context.compacted", + "session.started", + "session.resumed", + "turn.started", + "turn.accepted", + "harness.ready", +]; + +type Phase = "invocations" | "events" | "auditEvents" | "appViews" | "deliveredResults" | "runs" | "done"; +const PHASES: Phase[] = ["invocations", "events", "auditEvents", "appViews", "deliveredResults", "runs", "done"]; +const RUN_BATCH_SIZE = 50; +const DELIVERY_BATCH_SIZE = 50; + +type RunInProgress = { id: string; eventAfterId: number; changed: boolean; stage?: "events" | "log" }; + +type Progress = { + phase: Phase; + cursor: { createdAt: string; id: string } | null; + runCursor: string | null; + currentRun: RunInProgress | null; + pendingRuns: string[]; + counts: ToolConnectionContentDeletionCounts; +}; + +type DeletionRow = typeof toolConnectionContentDeletions.$inferSelect; + +export type ContentDeletionActor = { actorType: "user" | "agent" | "system"; actorId: string | null }; + +class ProgressConflict extends Error {} + +export function emptyContentDeletionCounts(): ToolConnectionContentDeletionCounts { + return { toolCalls: 0, approvals: 0, auditEvents: 0, appViews: 0, runs: 0, runsInProgress: 0, itemsInProgress: 0 }; +} + +function readProgress(row: DeletionRow): Progress { + const raw = row.progress as Partial; + const counts = { ...emptyContentDeletionCounts(), ...(raw.counts ?? {}) }; + const phase = PHASES.includes(raw.phase as Phase) ? raw.phase as Phase : "invocations"; + return { phase, cursor: raw.cursor ?? null, runCursor: raw.runCursor ?? null, currentRun: raw.currentRun ?? null, pendingRuns: raw.pendingRuns ?? [], counts }; +} + +function nextPhase(phase: Phase): Phase { + return PHASES[Math.min(PHASES.indexOf(phase) + 1, PHASES.length - 1)]!; +} + +function listSql(values: string[]): SQL { + return sql.join(values.map((value) => sql`${value}`), sql`, `); +} + +export function settledRunSql(runId: SQL | typeof toolInvocations.runId): SQL { + return sql`exists (select 1 from ${heartbeatRuns} r where r.id = ${runId} + and r.status in (${listSql(SETTLED_RUN_STATUSES)}) + and r.finished_at is not null + and r.cost_accounting_pending = false + and not exists (select 1 from native_run_finalizations f where f.run_id = r.id + and f.phase not in ('committed', 'terminal_failure', 'applied')))`; +} + +export function runsWithStoredContentSql(scope: { companyId: string; connectionId: string; cutoff: Date }): SQL { + const calls = sql`${toolInvocations.companyId} = ${scope.companyId} and ${toolInvocations.connectionId} = ${scope.connectionId} + and ${toolInvocations.createdAt} <= ${scope.cutoff.toISOString()}::timestamptz`; + const resultText = sql.join(RESULT_TEXT_KEYS.map((key) => + sql.raw(`jsonb_path_exists(coalesce("heartbeat_runs"."result_json", '{}'::jsonb), 'lax $.**.${key} ? (@.type() == "string" && @ != "")')`)), sql` or `); + return sql`${heartbeatRuns.companyId} = ${scope.companyId} + and (${heartbeatRuns.id} in (select ${toolInvocations.runId} from ${toolInvocations} where ${calls} and ${toolInvocations.runId} is not null) + or ${heartbeatRuns.id} in (select ${toolCallEvents.runId} from ${toolCallEvents} where ${toolCallEvents.companyId} = ${scope.companyId} + and ${toolCallEvents.connectionId} = ${scope.connectionId} and ${toolCallEvents.createdAt} <= ${scope.cutoff.toISOString()}::timestamptz + and ${toolCallEvents.runId} is not null) + or (${heartbeatRuns.issueId} in (select ${toolActionRequests.issueId} from ${toolActionRequests} + join ${toolInvocations} on ${toolInvocations.id} = ${toolActionRequests.invocationId} where ${calls}) + and ${heartbeatRuns.contextSnapshot}->'toolActionRequestIds' ?| array(select ${toolActionRequests.id}::text from ${toolActionRequests} + join ${toolInvocations} on ${toolInvocations.id} = ${toolActionRequests.invocationId} where ${calls}))) + and (${heartbeatRuns.logRef} is not null or ${heartbeatRuns.logStore} is not null + or ${heartbeatRuns.stdoutExcerpt} is not null or ${heartbeatRuns.stderrExcerpt} is not null + or ${resultText} + or exists (select 1 from ${heartbeatRunEvents} where ${heartbeatRunEvents.companyId} = ${heartbeatRuns.companyId} + and ${heartbeatRunEvents.runId} = ${heartbeatRuns.id} and ${contentEventSql()}))`; +} + +function contentEventSql(): SQL { + return sql`${heartbeatRunEvents.eventType} not in (${listSql(STRUCTURAL_EVENT_TYPES)}) + and ${heartbeatRunEvents.eventType} not like 'native.%' + and not (${heartbeatRunEvents.eventType} = 'item.completed' and (${heartbeatRunEvents.payload} -> 'prpEvent' -> 'payload' ->> 'kind') = 'usage') + and (${heartbeatRunEvents.message} is not null or ${heartbeatRunEvents.payload} is not null)`; +} + +function clearedSummary(column: SQL, deletedAt: string): SQL { + return sql`case when ${column} is null then null + else ${column} || jsonb_build_object('summary', '', 'contentDeletedAt', ${deletedAt}::text) end`; +} + +export function withoutActionResults(value: unknown, actionRequestIds: ReadonlySet): { value: unknown; changed: boolean } { + let changed = false; + const walk = (node: unknown): unknown => { + if (Array.isArray(node)) return node.map(walk); + if (!node || typeof node !== "object") return node; + const record = node as Record; + const next: Record = {}; + for (const [key, entry] of Object.entries(record)) next[key] = walk(entry); + if (typeof record.actionRequestId === "string" && actionRequestIds.has(record.actionRequestId) + && typeof record.resultSummary === "string" && record.resultSummary !== "") { + next.resultSummary = ""; + changed = true; + } + return next; + }; + const result = walk(value); + return { value: changed ? result : value, changed }; +} + +export function withoutResultText(value: Record | null): { value: Record | null; changed: boolean } { + if (!value || typeof value !== "object") return { value, changed: false }; + let changed = false; + const walk = (node: unknown, topLevel: boolean): unknown => { + if (Array.isArray(node)) return node.map((entry) => walk(entry, false)); + if (!node || typeof node !== "object") return node; + const next: Record = {}; + for (const [key, entry] of Object.entries(node as Record)) { + if (RESULT_TEXT_KEYS.includes(key) && typeof entry === "string") { + if (entry !== "") changed = true; + if (!topLevel) next[key] = ""; + else if (entry === "") next[key] = entry; + continue; + } + next[key] = walk(entry, false); + } + return next; + }; + const next = walk(value, true) as Record; + return { value: changed ? next : value, changed }; +} + +function toDeletion(row: DeletionRow): ToolConnectionContentDeletion { + return { + id: row.id, + status: row.status, + requestedAt: row.requestedAt.toISOString(), + completedAt: row.completedAt?.toISOString() ?? null, + retrying: row.status === "pending" && row.lastError !== null, + counts: readProgress(row).counts, + }; +} + +export function createConnectionContentDeletion(db: Db, deps: { runLogStore: () => RunLogStore }) { + async function latestDeletion(connection: { id: string; companyId: string }) { + const [latest] = await db + .select() + .from(toolConnectionContentDeletions) + .where(and( + eq(toolConnectionContentDeletions.companyId, connection.companyId), + eq(toolConnectionContentDeletions.connectionId, connection.id), + )) + .orderBy(desc(toolConnectionContentDeletions.createdAt)) + .limit(1); + return latest ? toDeletion(latest) : null; + } + + async function storedContent(connection: { id: string; companyId: string }): Promise { + const scope = and(eq(toolInvocations.companyId, connection.companyId), eq(toolInvocations.connectionId, connection.id)); + const [calls] = await db + .select({ toolCalls: sql`count(*)::int` }) + .from(toolInvocations) + .where(and( + scope, + sql`coalesce(${toolInvocations.resultSummary}->>'contentDeletedAt', ${toolInvocations.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolInvocations.resultSummary} is not null or ${toolInvocations.argumentsSummary} is not null)`, + )); + const [approvals] = await db + .select({ count: sql`count(*)::int` }) + .from(toolActionRequests) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and( + scope, + eq(toolActionRequests.companyId, connection.companyId), + sql`(${toolActionRequests.canonicalArgumentsSummary}->>'contentDeletedAt' is null or ${toolActionRequests.previewMarkdown} is not null)`, + )); + const [views] = await db + .select({ count: sql`count(*)::int` }) + .from(toolMcpAppViews) + .where(and(eq(toolMcpAppViews.connectionId, connection.id), eq(toolMcpAppViews.companyId, connection.companyId))); + const [runs] = await db + .select({ + runs: sql`count(*)::int`, + runsInProgress: sql`count(*) filter (where not ${settledRunSql(sql`${heartbeatRuns.id}`)})::int`, + }) + .from(heartbeatRuns) + .where(runsWithStoredContentSql({ companyId: connection.companyId, connectionId: connection.id, cutoff: new Date() })); + return { + stored: { + toolCalls: calls?.toolCalls ?? 0, + approvals: approvals?.count ?? 0, + appViews: views?.count ?? 0, + runs: runs?.runs ?? 0, + runsInProgress: runs?.runsInProgress ?? 0, + }, + deletion: await latestDeletion(connection), + }; + } + + async function request(connection: { id: string; companyId: string }, actor: ContentDeletionActor) { + return db.transaction(async (tx) => { + const pendingFor = () => tx + .select() + .from(toolConnectionContentDeletions) + .where(and( + eq(toolConnectionContentDeletions.companyId, connection.companyId), + eq(toolConnectionContentDeletions.connectionId, connection.id), + eq(toolConnectionContentDeletions.status, "pending"), + )) + .limit(1); + const [existing] = await pendingFor(); + if (existing) return { deletion: toDeletion(existing), created: false }; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.toolConnectionContentDeletion); + const now = new Date(); + const [created] = await tx + .insert(toolConnectionContentDeletions) + .values({ + requestedAt: now, + createdAt: now, + updatedAt: now, + companyId: connection.companyId, + connectionId: connection.id, + requestedByActorType: actor.actorType, + requestedByActorId: actor.actorId, + progress: { phase: "invocations", cursor: null, runCursor: null, currentRun: null, pendingRuns: [], counts: emptyContentDeletionCounts() }, + }) + .onConflictDoNothing() + .returning(); + if (created) return { deletion: toDeletion(created), created: true }; + const [raced] = await pendingFor(); + return { deletion: toDeletion(raced!), created: false }; + }); + } + + async function settledRunIds(tx: Db, runIds: string[]) { + if (runIds.length === 0) return new Set(); + const rows = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where(and(inArray(heartbeatRuns.id, runIds), settledRunSql(sql`${heartbeatRuns.id}`))); + return new Set(rows.map((row) => row.id)); + } + + async function clearDeliveredResults(tx: Db, job: DeletionRow, progress: Progress, requests: Array<{ id: string; issueId: string | null }>) { + const issueIds = [...new Set(requests.flatMap((row) => row.issueId ? [row.issueId] : []))]; + if (issueIds.length === 0) return []; + const ids = new Set(requests.map((row) => row.id)); + const idList = sql`array[${listSql([...ids])}]::text[]`; + const runs = await tx + .select({ id: heartbeatRuns.id, contextSnapshot: heartbeatRuns.contextSnapshot }) + .from(heartbeatRuns) + .where(and( + eq(heartbeatRuns.companyId, job.companyId), + inArray(heartbeatRuns.issueId, issueIds), + sql`${heartbeatRuns.contextSnapshot}->'toolActionRequestIds' ?| ${idList}`, + )) + .for("update"); + const settled = await settledRunIds(tx, runs.map((row) => row.id)); + for (const run of runs) { + if (!settled.has(run.id)) { + progress.counts.itemsInProgress += 1; + continue; + } + const cleaned = withoutActionResults(run.contextSnapshot, ids); + if (cleaned.changed) { + await tx.update(heartbeatRuns).set({ contextSnapshot: cleaned.value as Record }).where(eq(heartbeatRuns.id, run.id)); + } + } + const wakes = await tx + .select({ id: agentWakeupRequests.id, status: agentWakeupRequests.status, payload: agentWakeupRequests.payload }) + .from(agentWakeupRequests) + .where(and( + eq(agentWakeupRequests.companyId, job.companyId), + inArray(sql`${agentWakeupRequests.payload}->>'issueId'`, issueIds), + sql`(${agentWakeupRequests.payload}->'toolActionRequestIds' ?| ${idList} + or ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'toolActionRequestIds' ?| ${idList})`, + )) + .for("update"); + for (const wake of wakes) { + const cleaned = withoutActionResults(wake.payload, ids); + if (!cleaned.changed) continue; + if (UNSETTLED_WAKE_STATUSES.includes(wake.status)) { + progress.counts.itemsInProgress += 1; + continue; + } + await tx.update(agentWakeupRequests).set({ payload: cleaned.value as Record }).where(eq(agentWakeupRequests.id, wake.id)); + } + return runs.filter((row) => settled.has(row.id)).map((row) => row.id); + } + + async function invocationsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolInvocations.id, createdAt: toolInvocations.createdAt, runId: toolInvocations.runId, status: toolInvocations.status }) + .from(toolInvocations) + .where(and( + eq(toolInvocations.companyId, job.companyId), + lte(toolInvocations.createdAt, job.requestedAt), + cursor ? sql`(${toolInvocations.createdAt}, ${toolInvocations.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolInvocations.connectionId, job.connectionId), + )) + .orderBy(asc(toolInvocations.createdAt), asc(toolInvocations.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const eligible = page.filter((row) => !UNSETTLED_INVOCATION_STATUSES.includes(row.status) && (!row.runId || settledRuns.has(row.runId))); + progress.counts.itemsInProgress += page.length - eligible.length; + const ids = eligible.map((row) => row.id); + if (ids.length > 0) { + const cleared = await tx + .update(toolInvocations) + .set({ + argumentsSummary: clearedSummary(sql`${toolInvocations.argumentsSummary}`, deletedAt) as never, + resultSummary: clearedSummary(sql`${toolInvocations.resultSummary}`, deletedAt) as never, + progress: null, + }) + .where(and( + eq(toolInvocations.companyId, job.companyId), + inArray(toolInvocations.id, ids), + sql`coalesce(${toolInvocations.resultSummary}->>'contentDeletedAt', ${toolInvocations.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolInvocations.resultSummary} is not null or ${toolInvocations.argumentsSummary} is not null or ${toolInvocations.progress} is not null)`, + )) + .returning({ id: toolInvocations.id }); + progress.counts.toolCalls += cleared.length; + const requests = await tx + .select({ id: toolActionRequests.id, interactionId: toolActionRequests.interactionId, issueId: toolActionRequests.issueId, status: toolActionRequests.status }) + .from(toolActionRequests) + .where(and(eq(toolActionRequests.companyId, job.companyId), inArray(toolActionRequests.invocationId, ids))); + const settledRequests = requests.filter((row) => !UNSETTLED_ACTION_REQUEST_STATUSES.includes(row.status)); + progress.counts.itemsInProgress += requests.length - settledRequests.length; + if (settledRequests.length > 0) { + const requestIds = settledRequests.map((row) => row.id); + const clearedRequests = await tx + .update(toolActionRequests) + .set({ + canonicalArgumentsSummary: clearedSummary(sql`${toolActionRequests.canonicalArgumentsSummary}`, deletedAt) as never, + previewMarkdown: null, + signedArguments: null, + }) + .where(and( + eq(toolActionRequests.companyId, job.companyId), + inArray(toolActionRequests.id, requestIds), + sql`(${toolActionRequests.canonicalArgumentsSummary}->>'contentDeletedAt' is null or ${toolActionRequests.previewMarkdown} is not null or ${toolActionRequests.signedArguments} is not null)`, + )) + .returning({ id: toolActionRequests.id }); + progress.counts.approvals += clearedRequests.length; + const interactionIds = settledRequests.flatMap((row) => row.interactionId ? [row.interactionId] : []); + if (interactionIds.length > 0) { + await tx + .update(issueThreadInteractions) + .set({ + payload: sql`case when jsonb_typeof(${issueThreadInteractions.payload}->'toolAction') = 'object' + then jsonb_set(jsonb_set(${issueThreadInteractions.payload} - 'detailsMarkdown', '{toolAction,previewMarkdown}', 'null'::jsonb), '{toolAction,argumentsSummaryJson}', 'null'::jsonb) + else ${issueThreadInteractions.payload} - 'detailsMarkdown' end` as never, + result: sql`case when jsonb_typeof(${issueThreadInteractions.result}->'toolAction') = 'object' + then jsonb_set(${issueThreadInteractions.result}, '{toolAction,resultSummary}', 'null'::jsonb) + else ${issueThreadInteractions.result} end` as never, + }) + .where(and( + eq(issueThreadInteractions.companyId, job.companyId), + inArray(issueThreadInteractions.id, interactionIds), + sql`(${issueThreadInteractions.payload} ? 'detailsMarkdown' + or ${issueThreadInteractions.payload}->'toolAction'->>'previewMarkdown' is not null + or ${issueThreadInteractions.payload}->'toolAction'->>'argumentsSummaryJson' is not null + or ${issueThreadInteractions.result}->'toolAction'->>'resultSummary' is not null)`, + )); + } + } + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function eventsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolCallEvents.id, createdAt: toolCallEvents.createdAt, runId: toolCallEvents.runId }) + .from(toolCallEvents) + .where(and( + eq(toolCallEvents.companyId, job.companyId), + lte(toolCallEvents.createdAt, job.requestedAt), + cursor ? sql`(${toolCallEvents.createdAt}, ${toolCallEvents.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolCallEvents.connectionId, job.connectionId), + )) + .orderBy(asc(toolCallEvents.createdAt), asc(toolCallEvents.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const ids = page.filter((row) => !row.runId || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const cleared = await tx + .update(toolCallEvents) + .set({ + argumentsSummary: clearedSummary(sql`${toolCallEvents.argumentsSummary}`, deletedAt) as never, + requestSummary: clearedSummary(sql`${toolCallEvents.requestSummary}`, deletedAt) as never, + resultSummary: clearedSummary(sql`${toolCallEvents.resultSummary}`, deletedAt) as never, + }) + .where(and( + eq(toolCallEvents.companyId, job.companyId), + inArray(toolCallEvents.id, ids), + sql`coalesce(${toolCallEvents.resultSummary}->>'contentDeletedAt', ${toolCallEvents.requestSummary}->>'contentDeletedAt', ${toolCallEvents.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolCallEvents.resultSummary} is not null or ${toolCallEvents.requestSummary} is not null or ${toolCallEvents.argumentsSummary} is not null)`, + )) + .returning({ id: toolCallEvents.id }); + progress.counts.auditEvents += cleared.length; + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function auditEventsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolAccessAuditEvents.id, createdAt: toolAccessAuditEvents.createdAt, runId: sql`${toolAccessAuditEvents.details}->>'runId'` }) + .from(toolAccessAuditEvents) + .where(and( + eq(toolAccessAuditEvents.companyId, job.companyId), + lte(toolAccessAuditEvents.createdAt, job.requestedAt), + cursor ? sql`(${toolAccessAuditEvents.createdAt}, ${toolAccessAuditEvents.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolAccessAuditEvents.connectionId, job.connectionId), + )) + .orderBy(asc(toolAccessAuditEvents.createdAt), asc(toolAccessAuditEvents.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + const runIds = [...new Set(page.flatMap((row) => row.runId && uuid.test(row.runId) ? [row.runId] : []))]; + const settledRuns = await settledRunIds(tx, runIds); + const ids = page.filter((row) => !row.runId || !uuid.test(row.runId) || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const cleared = await tx + .update(toolAccessAuditEvents) + .set({ + details: sql`jsonb_set(${toolAccessAuditEvents.details}, '{argumentsSummary}', + ${toolAccessAuditEvents.details}->'argumentsSummary' || jsonb_build_object('summary', '', 'contentDeletedAt', ${deletedAt}::text))` as never, + }) + .where(and( + eq(toolAccessAuditEvents.companyId, job.companyId), + inArray(toolAccessAuditEvents.id, ids), + sql`jsonb_typeof(${toolAccessAuditEvents.details}->'argumentsSummary') = 'object'`, + sql`${toolAccessAuditEvents.details}->'argumentsSummary'->>'contentDeletedAt' is null`, + )) + .returning({ id: toolAccessAuditEvents.id }); + progress.counts.auditEvents += cleared.length; + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function appViewsStep(tx: Db, job: DeletionRow, progress: Progress) { + const page = await tx + .select({ id: toolMcpAppViews.id, runId: toolMcpAppViews.runId }) + .from(toolMcpAppViews) + .where(and( + eq(toolMcpAppViews.connectionId, job.connectionId), + eq(toolMcpAppViews.companyId, job.companyId), + lte(toolMcpAppViews.createdAt, job.requestedAt), + progress.runCursor ? gt(toolMcpAppViews.id, progress.runCursor) : undefined, + )) + .orderBy(asc(toolMcpAppViews.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const ids = page.filter((row) => !row.runId || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const removed = await tx + .delete(toolMcpAppViews) + .where(and(eq(toolMcpAppViews.companyId, job.companyId), inArray(toolMcpAppViews.id, ids))) + .returning({ id: toolMcpAppViews.id }); + progress.counts.appViews += removed.length; + } + progress.runCursor = page[page.length - 1]!.id; + return page.length < BATCH_SIZE; + } + + async function runEventPage(tx: Db, companyId: string, run: RunInProgress) { + const batch = await tx + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where(and( + eq(heartbeatRunEvents.companyId, companyId), + eq(heartbeatRunEvents.runId, run.id), + gt(heartbeatRunEvents.id, run.eventAfterId), + )) + .orderBy(asc(heartbeatRunEvents.id)) + .limit(EVENT_BATCH_SIZE); + if (batch.length === 0) return true; + const ids = batch.map((row) => row.id); + const cleared = await tx + .update(heartbeatRunEvents) + .set({ message: null, payload: null as never }) + .where(and( + inArray(heartbeatRunEvents.id, ids), + contentEventSql(), + )) + .returning({ id: heartbeatRunEvents.id }); + if (cleared.length > 0) run.changed = true; + run.eventAfterId = ids[ids.length - 1]!; + return batch.length < EVENT_BATCH_SIZE; + } + + async function removeRunLog(companyId: string, runId: string, signal: AbortSignal) { + let removedAny = false; + for (let attempt = 0; attempt < 3; attempt++) { + const [run] = await db + .select({ logStore: heartbeatRuns.logStore, logRef: heartbeatRuns.logRef }) + .from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))); + if (!run?.logRef && !run?.logStore) return removedAny; + if (!run.logRef || !isRunLogStoreType(run.logStore)) throw new Error("run_log_store_unrecognised"); + const handle: RunLogHandle = { store: run.logStore, logRef: run.logRef }; + const store = deps.runLogStore(); + if (!store.remove) throw new Error("run_log_removal_unavailable"); + await store.remove(handle, signal); + const updated = await db + .update(heartbeatRuns) + .set({ logStore: null, logRef: null }) + .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.logStore, run.logStore), eq(heartbeatRuns.logRef, run.logRef))) + .returning({ id: heartbeatRuns.id }); + removedAny = true; + if (updated.length > 0) return true; + } + throw new Error("run_log_reference_changed"); + } + + async function finishRun(tx: Db, companyId: string, run: RunInProgress) { + const [row] = await tx + .select({ stdoutExcerpt: heartbeatRuns.stdoutExcerpt, stderrExcerpt: heartbeatRuns.stderrExcerpt, resultJson: heartbeatRuns.resultJson }) + .from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, companyId))) + .for("update"); + if (!row) return; + const result = withoutResultText(row.resultJson); + if (row.stdoutExcerpt === null && row.stderrExcerpt === null && !result.changed) return; + await tx + .update(heartbeatRuns) + .set({ stdoutExcerpt: null, stderrExcerpt: null, resultJson: result.value as Record | null }) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, companyId))); + run.changed = true; + } + + async function deliveredResultsStep(tx: Db, job: DeletionRow, progress: Progress) { + const requests = await tx + .select({ id: toolActionRequests.id, issueId: toolActionRequests.issueId }) + .from(toolActionRequests) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and( + eq(toolActionRequests.companyId, job.companyId), + eq(toolInvocations.companyId, job.companyId), + eq(toolInvocations.connectionId, job.connectionId), + lte(toolInvocations.createdAt, job.requestedAt), + progress.runCursor ? gt(toolActionRequests.id, progress.runCursor) : undefined, + )) + .orderBy(asc(toolActionRequests.id)) + .limit(DELIVERY_BATCH_SIZE); + if (requests.length === 0) return true; + progress.runCursor = requests[requests.length - 1]!.id; + await clearDeliveredResults(tx, job, progress, requests); + return requests.length < DELIVERY_BATCH_SIZE; + } + + async function runsStep(tx: Db, job: DeletionRow, progress: Progress) { + if (!progress.currentRun) { + let runId = progress.pendingRuns.shift() ?? null; + if (!runId) { + const rows = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where(and( + progress.runCursor ? gt(heartbeatRuns.id, progress.runCursor) : undefined, + runsWithStoredContentSql({ companyId: job.companyId, connectionId: job.connectionId, cutoff: job.requestedAt }), + )) + .orderBy(asc(heartbeatRuns.id)) + .limit(RUN_BATCH_SIZE); + if (rows.length === 0) return true; + progress.pendingRuns = rows.map((row) => row.id); + progress.runCursor = rows[rows.length - 1]!.id; + return false; + } + if (!(await settledRunIds(tx, [runId])).has(runId)) { + progress.counts.runsInProgress += 1; + return false; + } + progress.currentRun = { id: runId, eventAfterId: 0, changed: false, stage: "events" }; + return false; + } + const run = progress.currentRun; + if (run.stage === "log") { + if (run.changed) progress.counts.runs += 1; + progress.currentRun = null; + return false; + } + if (!(await runEventPage(tx, job.companyId, run))) return false; + await finishRun(tx, job.companyId, run); + run.stage = "log"; + return false; + } + + async function step(job: DeletionRow, signal: AbortSignal) { + const progress = readProgress(job); + const deletedAt = job.requestedAt.toISOString(); + if (progress.currentRun?.stage === "log" && await removeRunLog(job.companyId, progress.currentRun.id, signal)) { + progress.currentRun.changed = true; + } + return db.transaction(async (rawTx) => { + const tx = rawTx as unknown as Db; + const done = progress.phase === "invocations" ? await invocationsStep(tx, job, progress, deletedAt) + : progress.phase === "events" ? await eventsStep(tx, job, progress, deletedAt) + : progress.phase === "auditEvents" ? await auditEventsStep(tx, job, progress, deletedAt) + : progress.phase === "appViews" ? await appViewsStep(tx, job, progress) + : progress.phase === "deliveredResults" ? await deliveredResultsStep(tx, job, progress) + : await runsStep(tx, job, progress); + if (done) { + progress.phase = nextPhase(progress.phase); + progress.cursor = null; + progress.runCursor = null; + progress.currentRun = null; + progress.pendingRuns = []; + } + const [saved] = await tx + .update(toolConnectionContentDeletions) + .set({ progress: progress as unknown as Record, updatedAt: new Date() }) + .where(and( + eq(toolConnectionContentDeletions.id, job.id), + eq(toolConnectionContentDeletions.status, "pending"), + eq(toolConnectionContentDeletions.updatedAt, job.updatedAt), + )) + .returning(); + if (!saved) throw new ProgressConflict(); + return saved; + }); + } + + async function complete(job: DeletionRow) { + const progress = readProgress(job); + const now = new Date(); + return db.transaction(async (tx) => { + const [done] = await tx + .update(toolConnectionContentDeletions) + .set({ status: "completed", completedAt: now, updatedAt: now, lastError: null }) + .where(and( + eq(toolConnectionContentDeletions.id, job.id), + eq(toolConnectionContentDeletions.status, "pending"), + eq(toolConnectionContentDeletions.updatedAt, job.updatedAt), + sql`${toolConnectionContentDeletions.progress}->>'phase' = 'done'`, + )) + .returning(); + if (!done) return null; + await logActivity(tx as unknown as Db, { + companyId: job.companyId, + actorType: job.requestedByActorType === "agent" ? "agent" : job.requestedByActorType === "user" ? "user" : "system", + actorId: job.requestedByActorId ?? "tool-connection-content-deletion", + action: "tool_connection.content_deleted", + entityType: "tool_connection", + entityId: job.connectionId, + details: { ...progress.counts }, + }); + return done; + }); + } + + async function process(jobId: string, signal: AbortSignal, budgetMs = STEP_BUDGET_MS) { + const startedAt = Date.now(); + let [job] = await db.select().from(toolConnectionContentDeletions).where(eq(toolConnectionContentDeletions.id, jobId)); + if (!job || job.status !== "pending") return job ?? null; + try { + while (!signal.aborted) { + if (readProgress(job).phase === "done") return await complete(job); + job = await step(job, signal); + if (Date.now() - startedAt >= budgetMs) return job; + } + return job; + } catch (error) { + if (error instanceof ProgressConflict) return null; + await db + .update(toolConnectionContentDeletions) + .set({ + attempts: sql`${toolConnectionContentDeletions.attempts} + 1`, + lastError: error instanceof Error ? error.message.slice(0, 500) : "content deletion failed", + updatedAt: new Date(), + }) + .where(and(eq(toolConnectionContentDeletions.id, job.id), eq(toolConnectionContentDeletions.status, "pending"))); + throw error; + } + } + + async function processPending(signal: AbortSignal) { + const rows = await db + .select({ id: toolConnectionContentDeletions.id }) + .from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.status, "pending")) + .orderBy(asc(toolConnectionContentDeletions.updatedAt)) + .limit(10); + let failure: unknown = null; + for (const { id } of rows) { + if (signal.aborted) break; + try { + await process(id, signal); + } catch (error) { + failure ??= error; + } + } + if (failure) throw failure; + } + + async function nextDeadline() { + const [row] = await db + .select({ id: toolConnectionContentDeletions.id }) + .from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.status, "pending")) + .limit(1); + return row ? Date.now() : null; + } + + return { storedContent, latestDeletion, request, process, processPending, nextDeadline }; +} + +export type ConnectionContentDeletionService = ReturnType; diff --git a/server/src/services/tool-connection-content-deletion-work.ts b/server/src/services/tool-connection-content-deletion-work.ts new file mode 100644 index 00000000000..0be2a0d4f0f --- /dev/null +++ b/server/src/services/tool-connection-content-deletion-work.ts @@ -0,0 +1,40 @@ +import type { Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerIdleLocalWorkProbe } from "./idle-local-work.js"; +import { getRunLogStore } from "./run-log-store.js"; +import { createConnectionContentDeletion } from "./tool-access/connections/content-deletion.js"; + +export function registerToolConnectionContentDeletionWork( + db: Db, + coordinator: ReturnType, + canRun: () => boolean, +) { + const deletions = createConnectionContentDeletion(db, { runLogStore: getRunLogStore }); + let deadline: number | null | undefined; + const unregister = registerIdleLocalWorkProbe(() => { + if (deadline === undefined) return "unknown"; + return deadline === null ? "none" : "present"; + }); + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.toolConnectionContentDeletion, { + canRun, + retryMs: 30_000, + continuationMs: 250, + onSettled: () => { + deadline = undefined; + }, + run: (signal) => signal.aborted ? Promise.resolve() : deletions.processPending(signal), + next: async () => { + deadline = await deletions.nextDeadline(); + return deadline; + }, + }); + return { + ...worker, + stop: async () => { + unregister(); + await worker.stop(); + }, + }; +} diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index aa52f0ca8b1..09b7a1d2994 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -268,7 +268,7 @@ "packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d", "packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca", "packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699", - "packages/shared/src/index.ts": "825867d65bbb0783a2294e1838231e68669a8fc3", + "packages/shared/src/index.ts": "6a7bc6095752c6e6f7602fdccd889cdb8bafeb57", "packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca", "packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075", "packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9", @@ -408,7 +408,7 @@ "packages/shared/src/types/inbox-agent-policy.ts": "cb9d41d9681e80b834889268ad96015a39df92f6", "packages/shared/src/types/inbox-dismissal.ts": "58fb7013847396ac9971eb5799dd0bf0486e0371", "packages/shared/src/types/inbox-mine.ts": "1d302b531d34842189704abd2f75f8743d09490d", - "packages/shared/src/types/index.ts": "8abf1f4ed91e67f8e620a1a0b127799c793554ff", + "packages/shared/src/types/index.ts": "9c93d353993d6073389ae6dad752c7b8845377e1", "packages/shared/src/types/instance.ts": "ddabd7cc0d885b5fdf45b33a7937b7b704f8fb7d", "packages/shared/src/types/issue-tree-control.ts": "75489fffd491b8350b05caf57d7796419797d9df", "packages/shared/src/types/issue.ts": "7183a5a11238c297e43917dc66897cbe73703835", @@ -431,7 +431,7 @@ "packages/shared/src/types/tailscale.ts": "d02a917945d7779009a2dea4c4b535e9478e0680", "packages/shared/src/types/task-workspace.ts": "a23afa13d2cb58dc2eb72692ed28c16ed6fb84fb", "packages/shared/src/types/teams-catalog.ts": "632fe75f8e5a4cfc455843c2a1a2daff28a022be", - "packages/shared/src/types/tool-access.ts": "1d2c29fb7c0c22ae30b53ff18fe44545c18b4835", + "packages/shared/src/types/tool-access.ts": "32b0ad72fd977c9fb5dbf31321f55c27cad92d9f", "packages/shared/src/types/user-profile.ts": "1ac26e033a79fab21dc267fd983959814c8b680f", "packages/shared/src/types/voice-sessions.ts": "95e3b62cdd51f62290e68536728ddd7297bc0459", "packages/shared/src/types/work-product.ts": "26f9129f606a207ce7a218ccef40f96747283086", diff --git a/ui/src/api/tools.ts b/ui/src/api/tools.ts index 786afb8929b..bfa359b1fc4 100644 --- a/ui/src/api/tools.ts +++ b/ui/src/api/tools.ts @@ -9,6 +9,8 @@ import type { ToolConnectionInstall, ToolConnectionInstallSnapshot, ToolConnectionRemovalSummary, + ToolConnectionContentDeletion, + ToolConnectionStoredContent, ConnectToolAppResult, ConnectToolApp, FinishToolAppResult, @@ -429,10 +431,16 @@ export const toolsApi = { api.post(`/tool-connections/${connectionId}/railway/ssh`, input), // Removal is a credential-revoking teardown (PAP-17119), so the response // carries the cleanup receipt alongside the archived connection. - archiveConnection: (connectionId: string) => - api.delete( - `/tool-connections/${connectionId}`, + archiveConnection: (connectionId: string, options: { deleteStoredContent?: boolean } = {}) => + api.delete( + `/tool-connections/${connectionId}${options.deleteStoredContent ? "?deleteStoredContent=true" : ""}`, ), + getStoredContent: (connectionId: string) => + api.get(`/tool-connections/${connectionId}/stored-content`), + deleteStoredContent: (connectionId: string) => + api.post<{ deletion: ToolConnectionContentDeletion }>(`/tool-connections/${connectionId}/stored-content/deletion`, {}), + getStoredContentDeletion: (connectionId: string) => + api.get<{ deletion: ToolConnectionContentDeletion | null }>(`/tool-connections/${connectionId}/stored-content/deletion`), checkConnectionHealth: (connectionId: string) => api.post(`/tool-connections/${connectionId}/health-check`, {}), reconnectConnection: (connectionId: string, credentialValues: Record) => diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index 1bae37d0722..e12ab0f09e2 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -72,6 +72,10 @@ export const queryKeys = { ["tools", "connection", connectionId, "installs"] as const, connectionGrants: (connectionId: string) => ["tools", "connection", connectionId, "grants"] as const, + connectionStoredContent: (connectionId: string) => + ["tools", "connection", connectionId, "stored-content"] as const, + connectionStoredContentDeletion: (connectionId: string) => + ["tools", "connection", connectionId, "stored-content", "deletion"] as const, catalog: (connectionId: string) => ["tools", "connection", connectionId, "catalog"] as const, connectionActivity: (connectionId: string) => ["tools", "connection", connectionId, "activity"] as const, diff --git a/ui/src/pages/apps/AppDetail.tsx b/ui/src/pages/apps/AppDetail.tsx index 993d67873ba..4b047c38cee 100644 --- a/ui/src/pages/apps/AppDetail.tsx +++ b/ui/src/pages/apps/AppDetail.tsx @@ -67,6 +67,7 @@ import { actionPermissionMutation } from "./app-detail/action-permissions"; import { RailwayAccessPanel } from "./app-detail/RailwayAccessPanel"; import { ReviewPanel } from "./app-detail/ReviewPanel"; import { AccessRetryBanner } from "./app-detail/AccessRetryBanner"; +import { StoredContentDeletionRow } from "./app-detail/StoredContentDeletion"; import { ReconnectCard, DangerZone, @@ -738,6 +739,11 @@ function StandardAppDetail({ renderActions, renderAgentSettings, renderConnectio onManage={() => window.open(remoteMcpProviders[connection.config?.sourceTemplateKey as keyof typeof remoteMcpProviders].dashboardUrl, "_blank", "noopener,noreferrer")} onDisconnect={() => disconnectRemote.mutateAsync()} />} + {grantsQuery.data?.capabilities.canConfigure === true && connection.connectionPurpose === "tool" && ( +
+ +
+ )} } )} diff --git a/ui/src/pages/apps/Browse.tsx b/ui/src/pages/apps/Browse.tsx index bbe64341603..8e47e351cb6 100644 --- a/ui/src/pages/apps/Browse.tsx +++ b/ui/src/pages/apps/Browse.tsx @@ -84,6 +84,7 @@ import { AlertDialogHeader, AlertDialogTitle, } from "@/components/ui/alert-dialog"; +import { Checkbox } from "@/components/ui/checkbox"; import { Button } from "@/components/ui/button"; import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible"; import { @@ -414,6 +415,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne const [aggregatorToConnect, setAggregatorToConnect] = useState(null); const [aggregatorToManage, setAggregatorToManage] = useState<{ app: AggregatorAppCatalogEntry; connectionId?: string } | null>(null); const [initialAggregatorProvider, setInitialAggregatorProvider] = useState(); + const [deleteStoredContent, setDeleteStoredContent] = useState(false); const [connectionToRemove, setConnectionToRemove] = useState(null); @@ -539,7 +541,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne if (!target.poolRevision) throw new Error("Open the confirmation again before removing this pool."); await aiConnectionPoolsApi.remove(selectedCompanyId!, target.id, target.poolRevision); } else { - await toolsApi.archiveConnection(target.id); + await toolsApi.archiveConnection(target.id, { deleteStoredContent }); } }, onSuccess: (_connection, target) => { @@ -564,6 +566,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne : target.remainingConnectionCount > 0 ? `${target.providerName} still has ${target.remainingConnectionCount} active ${target.remainingConnectionCount === 1 ? "connection" : "connections"} available to agents.` : `${target.providerName} is no longer available to agents through this connection. Its saved credentials were deleted.`, + ...(deleteStoredContent && !target.pool && target.kind !== "chat" ? { title: "Connection removed. Deleting its saved data." } : {}), tone: "success", }); setConnectionToRemove(null); @@ -577,6 +580,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne }); async function requestConnectionRemoval(target: ConnectionRemovalTarget) { removeConnection.reset(); + setDeleteStoredContent(false); if (!target.pool) { setConnectionToRemove(target); return; } try { const pool = (await aiConnectionPoolsApi.list(selectedCompanyId!)).find(pool => pool.id === target.id); @@ -1125,6 +1129,22 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne : "The saved credentials are deleted and agents lose access immediately. Connecting it again later requires a new sign-in or key."} + {connectionToRemove && !connectionToRemove.pool && connectionToRemove.kind !== "chat" ? ( + + ) : null} {removeConnection.error ?

{describeError(removeConnection.error).body}

: null} diff --git a/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx b/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx new file mode 100644 index 00000000000..1986eb0c2a5 --- /dev/null +++ b/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx @@ -0,0 +1,84 @@ +// @vitest-environment jsdom + +import { act } from "react"; +import { createRoot } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { ToolConnectionStoredContent } from "@paperclipai/shared"; + +const api = vi.hoisted(() => ({ + getStoredContent: vi.fn(), + getStoredContentDeletion: vi.fn(), + deleteStoredContent: vi.fn(), +})); +vi.mock("@/api/tools", () => ({ toolsApi: api })); +vi.mock("@/context/ToastContext", () => ({ useToast: () => ({ pushToast: vi.fn() }) })); + +import { StoredContentDeletionDialog, storedContentStatus, storedContentSummary } from "./StoredContentDeletion"; + +// eslint-disable-next-line @typescript-eslint/no-explicit-any +(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true; + +const counts = { toolCalls: 0, approvals: 0, auditEvents: 0, appViews: 0, runs: 0, runsInProgress: 0, itemsInProgress: 0 }; +const stored: ToolConnectionStoredContent = { + stored: { toolCalls: 12, approvals: 2, appViews: 0, runs: 3, runsInProgress: 1 }, + deletion: null, +}; + +let container: HTMLDivElement | null = null; +afterEach(() => { + container?.remove(); + container = null; + vi.clearAllMocks(); +}); + +async function flush() { + for (let i = 0; i < 10; i++) await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); +} + +describe("stored content deletion", () => { + it("describes how much will be deleted in plain words", () => { + expect(storedContentSummary(stored, "Google Workspace")).toBe( + "Paperclip will delete what it saved from 12 tool calls, 2 approval requests and the full transcripts of 3 runs that used Google Workspace or received its results.", + ); + expect(storedContentSummary({ ...stored, stored: { toolCalls: 0, approvals: 0, appViews: 1, runs: 0, runsInProgress: 0 } }, "Docs")) + .toBe("Paperclip will delete what it saved from 1 app view that used Docs."); + expect(storedContentSummary({ ...stored, stored: { toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 } }, "Docs")) + .toBe("Paperclip found no saved data from Docs's tool calls. You can still delete to check again."); + expect(storedContentStatus({ ...stored, deletion: { id: "d", status: "pending", requestedAt: "", completedAt: null, retrying: true, counts } })) + .toBe("Deleting was interrupted. Paperclip keeps trying until it finishes."); + }); + + it("confirms with counts, warns about running work, and starts the deletion", async () => { + api.getStoredContent.mockResolvedValue(stored); + const pendingDeletion = { id: "d", status: "pending" as const, requestedAt: new Date().toISOString(), completedAt: null, retrying: false, counts }; + api.deleteStoredContent.mockResolvedValue({ deletion: pendingDeletion }); + api.getStoredContentDeletion.mockResolvedValue({ deletion: pendingDeletion }); + container = document.createElement("div"); + document.body.appendChild(container); + const root = createRoot(container); + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + await act(async () => { + root.render( + + undefined} /> + , + ); + }); + await flush(); + const text = document.body.textContent ?? ""; + expect(text).toContain("Delete data read through Google Workspace?"); + expect(text).toContain("12 tool calls"); + expect(text).toContain("1 run is still working and will be skipped."); + expect(text).toContain("This cannot be undone."); + expect(text).not.toContain("connection-1"); + + const confirm = Array.from(document.body.querySelectorAll("button")).find((button) => button.textContent === "Delete data"); + await act(async () => { confirm!.click(); }); + await flush(); + expect(api.deleteStoredContent).toHaveBeenCalledWith("connection-1"); + expect(document.body.textContent).toContain("Deleting now. You can close this window."); + expect(api.getStoredContent).toHaveBeenCalledTimes(1); + await act(async () => { root.unmount(); }); + }); +}); diff --git a/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx b/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx new file mode 100644 index 00000000000..b7b46f24210 --- /dev/null +++ b/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx @@ -0,0 +1,171 @@ +import { useEffect, useRef, useState } from "react"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { Loader2 } from "lucide-react"; +import type { ToolConnectionStoredContent } from "@paperclipai/shared"; +import { toolsApi } from "@/api/tools"; +import { describeError } from "@/api/errors"; +import { Button, buttonVariants } from "@/components/ui/button"; +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from "@/components/ui/alert-dialog"; +import { useToast } from "@/context/ToastContext"; +import { queryKeys } from "@/lib/queryKeys"; +import { relativeTime } from "@/lib/utils"; + +function plural(count: number, one: string, many: string) { + return `${count.toLocaleString()} ${count === 1 ? one : many}`; +} + +export function storedContentSummary(content: ToolConnectionStoredContent, appName: string): string { + const { toolCalls, approvals, appViews, runs } = content.stored; + const parts = [ + toolCalls > 0 ? plural(toolCalls, "tool call", "tool calls") : null, + approvals > 0 ? plural(approvals, "approval request", "approval requests") : null, + appViews > 0 ? plural(appViews, "app view", "app views") : null, + runs > 0 ? `the full transcripts of ${plural(runs, "run", "runs")}` : null, + ].filter((part): part is string => part !== null); + if (parts.length === 0) return `Paperclip found no saved data from ${appName}'s tool calls. You can still delete to check again.`; + const list = parts.length > 1 ? `${parts.slice(0, -1).join(", ")} and ${parts[parts.length - 1]}` : parts[0]; + return `Paperclip will delete what it saved from ${list} that used ${appName}${runs > 0 ? " or received its results" : ""}.`; +} + +export function storedContentStatus(content: ToolConnectionStoredContent): string | null { + const deletion = content.deletion; + if (deletion?.status === "pending") { + return deletion.retrying + ? "Deleting was interrupted. Paperclip keeps trying until it finishes." + : "Deleting now. You can close this window."; + } + if (deletion?.status === "completed" && deletion.completedAt) { + const skipped = deletion.counts.runsInProgress; + return skipped > 0 + ? `Last deleted ${relativeTime(deletion.completedAt)}. ${plural(skipped, "run was", "runs were")} still working and kept their data.` + : `Last deleted ${relativeTime(deletion.completedAt)}.`; + } + return null; +} + +export function StoredContentDeletionRow({ connectionId, appName }: { connectionId: string; appName: string }) { + const [open, setOpen] = useState(false); + return ( +
+
+

Delete data read through this app

+

+ Permanently removes the results and run transcripts Paperclip saved from this app's tool calls. +

+
+ + {open ? ( + setOpen(false)} /> + ) : null} +
+ ); +} + +export function StoredContentDeletionDialog({ + connectionId, + appName, + onClose, +}: { + connectionId: string; + appName: string; + onClose: () => void; +}) { + const queryClient = useQueryClient(); + const { pushToast } = useToast(); + const countsKey = queryKeys.tools.connectionStoredContent(connectionId); + const statusKey = queryKeys.tools.connectionStoredContentDeletion(connectionId); + const query = useQuery({ + queryKey: countsKey, + queryFn: () => toolsApi.getStoredContent(connectionId), + staleTime: Infinity, + refetchOnMount: "always", + refetchOnWindowFocus: false, + }); + const [tracking, setTracking] = useState(false); + const watching = tracking || query.data?.deletion?.status === "pending"; + const statusQuery = useQuery({ + queryKey: statusKey, + queryFn: () => toolsApi.getStoredContentDeletion(connectionId), + enabled: watching, + refetchInterval: (current) => current.state.data?.deletion?.status === "completed" ? false : 2_000, + }); + const latest = statusQuery.data?.deletion ?? query.data?.deletion ?? null; + const refreshedFor = useRef(null); + useEffect(() => { + if (!watching || latest?.status !== "completed" || refreshedFor.current === latest.id) return; + refreshedFor.current = latest.id; + setTracking(false); + void queryClient.invalidateQueries({ queryKey: queryKeys.tools.connectionStoredContent(connectionId) }); + }, [watching, latest?.status, latest?.id, queryClient, connectionId]); + const deletion = useMutation({ + mutationFn: () => toolsApi.deleteStoredContent(connectionId), + onSuccess: (result) => { + queryClient.setQueryData(statusKey, result); + setTracking(true); + }, + onError: (error) => pushToast({ title: "Couldn't delete the data", body: describeError(error).body, tone: "error" }), + }); + const content = query.data ? { ...query.data, deletion: latest } : undefined; + const pending = deletion.isPending || latest?.status === "pending"; + const inProgress = content?.stored.runsInProgress ?? 0; + const status = content ? storedContentStatus(content) : null; + + return ( + { if (!next) onClose(); }}> + + + {`Delete data read through ${appName}?`} + + {content ? storedContentSummary(content, appName) : "Counting saved data…"} + + + {query.error ? ( +

{describeError(query.error).body}

+ ) : null} + {content ? ( +
+

Whole transcripts are deleted, including other steps of those runs. Call times, sizes and outcomes stay in the activity history.

+

This does not remove anything your agents wrote into tasks, such as comments, documents and files. Delete or edit those yourself.

+ {inProgress > 0 ? ( +

{`${plural(inProgress, "run is", "runs are")} still working and will be skipped. Delete again after ${inProgress === 1 ? "it finishes" : "they finish"}.`}

+ ) : null} +

This cannot be undone.

+
+ ) : null} + {status ? ( +

+ {content?.deletion?.status === "pending" ? : null} + {status} +

+ ) : null} + + {pending ? "Close" : "Cancel"} + {content ? ( + { + event.preventDefault(); + deletion.mutate(); + }} + > + {pending ? : null} + Delete data + + ) : null} + +
+
+ ); +}