From d1b5cfe038e7b66fef345e6cbf5d60be40972d03 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 17 Nov 2026 08:55:35 -0600 Subject: [PATCH] feat(connections): let users delete data read through a connection (#22010) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call connected apps, such as Google Workspace, through Paperclip's tool gateway > - Paperclip saves part of each tool result, approval previews, app views and the full run transcripts that contain tool output > - No user action deleted this data. Removing the connection left it in place > - Google's restricted-scope review requires that users can delete the data an app obtained, and that the privacy policy says how > - This pull request adds "Delete data read through this connection" for any connection, on the connection's page and as an option when you remove it > - The benefit is that the picker text ("They stay there until you delete them") is now true, and the privacy policy can link to a help page ## Linked Issues or Issue Description **Subsystem affected** Connections (tool gateway audit tables, run logs, run events) and the Connectors UI. **Problem or motivation** Paperclip stores content from a connection's tool calls in several places and gives users no way to delete it: - `tool_invocations.arguments_summary/result_summary` and `tool_call_events.arguments_summary/request_summary/result_summary` keep the first ~4,000 characters of each redacted value (`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No retention limit. - `tool_action_requests` keeps `canonical_arguments_summary`, `preview_markdown` and `signed_arguments`. The linked approval card (`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`, `payload.toolAction.argumentsSummaryJson` and `result.toolAction.resultSummary`. - `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app views. - Run transcripts hold full tool output: the run log (`local_file` with an optional S3 mirror, or `s3_segments`/`s3_archive` object journals), `heartbeat_run_events.message/payload`, and `heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store had a delete path. - Connection removal (`removal.ts`) archives the connection and revokes credentials. It deletes no content. **Proposed solution** A durable, resumable, company-scoped deletion request per connection. See "Design" below. **Alternatives considered** - Delete synchronously in the request. Rejected: an interrupted request leaves partial state that needs a person to retry it. - Delete only the run events that mention the connection. Rejected: tool output also reaches the model's later messages, so the safe unit is the whole run transcript. - Delete the audit rows. Rejected: counts, outcomes and hashes feed the activity history and usage counts. ## Design **Rule.** The job never drops a reference to content it has not confirmed deleted. It is marked complete, and writes the `tool_connection.content_deleted` activity entry, only after every step succeeded. A step that cannot confirm its work throws. The request then stays pending, the UI says "Paperclip keeps trying until it finishes", and the worker retries it. **Store inventory.** Every place that stores tool arguments, results or previews. I found them from `summarizeToolValue`, `redactArguments`, `writeAudit`, and every column or JSON field named like summary, preview, arguments, result, toolInput or toolResult. | Store | Content | Cleared by this PR | Proof | | --- | --- | --- | --- | | `tool_invocations.arguments_summary`, `result_summary`, `progress` | Redacted request/result preview, provider progress | Yes: `summary` becomes `""` with `contentDeletedAt`; `progress` null; size and sha256 kept | content-deletion test 1 | | `tool_call_events.arguments_summary`, `request_summary`, `result_summary` | Same previews | Yes, same | test 1 | | `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) | Argument preview | Yes, same; decision, ids, size, hash kept | test 1 | | `tool_action_requests.canonical_arguments_summary`, `preview_markdown`, `signed_arguments` | Approval preview and arguments | Yes, for settled requests | test 1, card retry test | | Approval card in `issue_thread_interactions`: `payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`, `payload.toolAction.argumentsSummaryJson`, `result.toolAction.resultSummary` | Approval preview, arguments, executed result | Yes. Cards are found from every settled request of a batch, in the same transaction | test 1, card retry test | | `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload` (`toolAction(s)[].resultSummary`, `paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved action result delivered to the agent | Yes, for settled runs and settled wakes | test 1 | | `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App view input, result, shared context | Yes: rows deleted | test 1 | | Run log: object journal, archive journal, original `local_file` log and its legacy S3 mirror | Full transcript | Yes, every copy, for each settled run that called the connection or received an approved result | run-log-store tests, test 1 | | `heartbeat_run_events.message`, `payload` | Transcript events | Yes, except structural events (lifecycle, terminal, presentation, usage, compaction, session/turn start, `native.*`) | test 1 | | `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string `summary`, `result`, `message`, `text`, `output`, `content` or `body` at any depth of `result_json` (including `nativeResult`, `acceptedResult`, `semanticResult` and their `continuation`) | Output excerpts and final answer | Yes; schema and control fields kept | test 1, canary | | `approvals.payload` (formal board approval) | Tool name, risk and hashes only | Not needed: no content | — | | `tool_mcp_tasks` | Request hash and sealed provider task id; no result | Not needed: no content | — | | `tool_mcp_elicitations.questions` and their question cards | Questions the provider asked a person | No. These are provider prompts, not tool results, and the answered question stays in the task. Follow-up if wanted | — | | `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` | Skills and tool descriptions the provider publishes | No. These are provider definitions, not data read through tool calls | — | | Transcripts of later runs that read an app view's `model_context` | Shared app context | No. Nothing records which runs read it. The help page says so. Follow-up | — | | `issue_comments.body`, issue documents (including continuation summaries), files, chat messages | Task content written by agents or Paperclip | No, by design. The user removes it by editing or deleting it or the task. The confirmation says in bold that anything agents wrote into tasks is not removed, and so does the help page. The canary excludes `issue_comments.body` by name for this reason | canary asserts it is kept | | Raw provider traces | Full provider traffic | No. Admin opt-in, encrypted, expire within 24 hours | — | | Feedback exports already sent | Run logs the user chose to share | No. Already outside Paperclip | — | | Content of runs of a deleted task or company | Same stores | No. Follow-up | — | **Durable request (schema change).** I added one table, `tool_connection_content_deletions`, and migration `0490`. I followed `doc/DATABASE.md`: edited the schema, exported it from `packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the migration drift and compatibility tests. The classifier marks it `online_compatible` (a new table and its indexes only). No existing table can hold a durable per-connection request with a cutoff time, cursor, counts and retry state. Without it, an interrupted deletion could resume only when a person pressed the button again. **Resume after restart.** The worker uses the shared `registerDeadlineWork` lifecycle on a new `tool-connection-content-deletion` queue. The request insert signals the queue in the same transaction. At server start the worker runs once, `next()` finds any pending row, and processing continues from the saved phase and cursor. Nobody presses anything. `next()` returns a deadline only while a pending row exists, so an idle instance sends no recurring query. A failure records `attempts` and `last_error` and retries after 30 seconds. **One run selection.** `runsWithStoredContentSql` selects the runs whose transcripts are deleted: runs that made a tool call or tool-call event through the connection, and runs whose context lists one of the connection's approved action requests, limited to runs that still hold a log, an excerpt, result text or a content event. The dialog count, its in-progress warning and the job's `runs` phase all use this one function. **Phases.** `invocations`, `events`, `auditEvents`, `appViews`, `deliveredResults` (50 action requests per page; matching runs and wakes are locked with `FOR UPDATE` before their documents are rewritten), `runs` (the shared selection, 50 run ids per page, then one run at a time), `done`. **Canary test.** `tool-connection-content-deletion-canary.test.ts` puts a unique marker in a tool call's arguments and result, an approved action with a preview, a result delivered to a later run and its wake, a native run that quotes the marker in its answer and events, a run log migrated from a local file with an S3 mirror to an object journal, an MCP App view and the audit tables. It checks that the marker is first present in the database, the object store, the legacy mirror and the log files. After the deletion it scans every `text`, `varchar`, `json` and `jsonb` column of every table found through `information_schema` (no hand-written table list), every object in both stores and every file under the log directory. Any hit fails with the table and column or the key. It runs three times: a complete deletion, a deletion interrupted twice by storage failures and resumed, and two deletions for different connections on one task at the same time. The only exclusion is `issue_comments.body`, named with its reason. **Batching and time budget.** Tool rows are read in pages of 200 with keyset pagination on the existing `(company_id, created_at)` indexes, bounded by the request time. Each page's writes and the saved progress commit in one transaction. Counts are therefore exact, and a crash repeats at most one page, idempotently. Runs advance one at a time from the shared selection. Each 1,000-event page is its own step. Log removal is its own step and runs before its progress transaction, so object storage calls never run inside a database transaction. The 10-second budget is checked after every step. Nothing changes on the agent boot or tool-call path. **In-flight work.** A run counts as settled only when its status is terminal, `finished_at` is set, cost accounting is done, and no native finalization is open. Calls, approvals, events, app views, delivered results and transcripts tied to other runs or wakes are skipped and counted. The confirmation says "N runs are still working and will be skipped. Delete again after they finish." **Permissions.** Board users with configure access (connection manager or creator) can delete, like `DELETE /tool-connections/:id`. Agents, viewers and other companies are refused. The activity entries `tool_connection.content_deletion_requested` and `tool_connection.content_deleted` hold counts only. **UI reads.** The dialog loads counts once when it opens. While a deletion runs, it polls `GET /tool-connections/:id/stored-content/deletion`, a single indexed row read, every 2 seconds. Polling stops when the job completes or the dialog closes. The Delete button is not gated on counts. ## What Changed - `packages/db`: new `tool_connection_content_deletions` table and migration `0490` (online compatible: new table and its indexes only). - `packages/shared`: `ToolConnectionStoredContent`, `ToolConnectionContentDeletion` and counts types; optional `contentDeletedAt` on `ToolRedactedValueSummary`. - `server`: `RunLogStore.remove`, which deletes every copy of a run's log (`composeRunLogStore`); `content-deletion.ts` service; deadline worker wired into startup and pre-drain; `GET /tool-connections/:id/stored-content`, `POST` and `GET /tool-connections/:id/stored-content/deletion`, and `DELETE /tool-connections/:id?deleteStoredContent=true`; OpenAPI entries. - `ui`: "Delete data" row on the connection's Permissions tab with a confirmation that shows counts, skipped runs, and "This cannot be undone"; "Also delete data read through this connection" checkbox in the Connectors removal dialog. - Docs: user help `docs/guides/board-operator/deleting-connection-data.md` (for the privacy policy link) and an API note in `doc/connections/CONNECTOR-PLAYBOOK.md`. - `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the shared type change. `server/src/routes/tool-access.ts` (3,010 lines) and `ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20 lines. The routes reuse that file's connection access checks; the dialog reuses the existing removal confirmation. The new logic lives in its own files. ## Verification - `server/src/__tests__/tool-connection-content-deletion.test.ts` (embedded PostgreSQL): content is gone from every "Yes" store in the inventory; rows, hashes and sizes stay; another connection and another company are untouched; a running run is skipped and is deleted after it settles; a storage failure leaves the request pending with `retrying`, and the next pass completes it; an approval-card update that fails once is cleared on the retry; an unrecognised log backend keeps the pointer, keeps the job pending and writes no completion entry; a zero time budget yields after every step and still finishes; a repeat request returns the pending one; agents and outsiders are refused; removal with `deleteStoredContent=true` queues the deletion. - `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove` deletes local and mirrored logs and object journals, removes a migrated run's original local file and mirror with its journal, refuses an object log when object storage is not configured, and is idempotent. - `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts, skipped-run warning, and the request. - Red: these tests fail on `master` because `remove`, the service and the routes do not exist. Green on this branch. - Ran: the tests above, `openapi-routes`, `server-startup-feedback-export`, `tool-connection-removal`, `packages/db` migration drift and compatibility tests, all `ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI `tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so required CI supplies broad coverage. ## Risks - Deleting is permanent by design. The confirmation says so. - A `local_file` log written on another replica's local disk stays on that pod's disk until the pod is replaced; its S3 mirror is deleted. New logs on S3-configured instances use object journals, which are deleted. - On versioned buckets, earlier object versions remain until the bucket's lifecycle policy removes them (the existing journal delete path does not remove versions). - Not in this PR (follow-ups): the "No" rows of the inventory (except agent-written task content, which is out of scope by design), mainly the same deletion when a task or a company is deleted, and later runs that read an app view's shared context. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- doc/connections/CONNECTOR-PLAYBOOK.md | 22 + docs/docs.json | 1 + .../deleting-connection-data.md | 58 ++ .../db/src/migrations/0490_nappy_chat.sql | 21 + ...{0485_snapshot.json => 0490_snapshot.json} | 509 +++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/index.ts | 1 + .../tool_connection_content_deletions.ts | 32 + packages/shared/src/index.ts | 5 + packages/shared/src/types/index.ts | 5 + packages/shared/src/types/tool-access.ts | 35 + ...connection-content-deletion-canary.test.ts | 346 ++++++++ .../tool-connection-content-deletion.test.ts | 420 ++++++++++ server/src/index.ts | 6 + server/src/routes/openapi.ts | 29 + server/src/routes/tool-access.ts | 54 +- .../services/delivery-work-notifications.ts | 1 + .../src/services/object-run-log-store.test.ts | 12 + server/src/services/object-run-log-store.ts | 4 + server/src/services/run-log-store.test.ts | 43 +- server/src/services/run-log-store.ts | 58 +- .../connections/content-deletion.ts | 761 ++++++++++++++++++ .../tool-connection-content-deletion-work.ts | 40 + .../runner-e2e/source-sdk-public-inputs.json | 6 +- ui/src/api/tools.ts | 14 +- ui/src/lib/queryKeys.ts | 4 + ui/src/pages/apps/AppDetail.tsx | 6 + ui/src/pages/apps/Browse.tsx | 22 +- .../app-detail/StoredContentDeletion.test.tsx | 84 ++ .../apps/app-detail/StoredContentDeletion.tsx | 171 ++++ 30 files changed, 2752 insertions(+), 25 deletions(-) create mode 100644 docs/guides/board-operator/deleting-connection-data.md create mode 100644 packages/db/src/migrations/0490_nappy_chat.sql rename packages/db/src/migrations/meta/{0485_snapshot.json => 0490_snapshot.json} (99%) create mode 100644 packages/db/src/schema/tool_connection_content_deletions.ts create mode 100644 server/src/__tests__/tool-connection-content-deletion-canary.test.ts create mode 100644 server/src/__tests__/tool-connection-content-deletion.test.ts create mode 100644 server/src/services/tool-access/connections/content-deletion.ts create mode 100644 server/src/services/tool-connection-content-deletion-work.ts create mode 100644 ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx create mode 100644 ui/src/pages/apps/app-detail/StoredContentDeletion.tsx diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index 26eaaadc723..cabbcdad193 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -1540,6 +1540,28 @@ POST /api/tool-connections/:connectionId/test-calls DELETE /api/tool-connections/:connectionId ``` +Stored tool-call content endpoints: + +```text +GET /api/tool-connections/:connectionId/stored-content +POST /api/tool-connections/:connectionId/stored-content/deletion +GET /api/tool-connections/:connectionId/stored-content/deletion +DELETE /api/tool-connections/:connectionId?deleteStoredContent=true +``` + +Deletion is connector-agnostic. It clears the content of the connection's +tool-call summaries (including `tool_access_audit_events.details.argumentsSummary`), +approval previews, approval cards, delivered action results and app views, and +deletes the whole run transcript (every copy of the run log, run-event content, +output excerpts and the result text) of every settled run that called the +connection or received an approved action's result. Row ids, timestamps, status, +sizes and hashes stay. A step that cannot confirm its deletion fails, and the +request stays pending and retries; it completes only when every step succeeded. +A `tool_connection_content_deletions` row is the durable request; the +`tool-connection-content-deletion` delivery queue resumes it after a restart. +Runs that are still working are skipped and counted. The user-facing help is +`docs/guides/board-operator/deleting-connection-data.md`. + Responses may contain secret IDs, version selectors, header names, prefixes, scope names, expiry timestamps, and redacted provider metadata. They must not contain secret values. Logs and activity should identify the operation and diff --git a/docs/docs.json b/docs/docs.json index a1a008f94dd..55989359f45 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -57,6 +57,7 @@ "guides/board-operator/approvals", "guides/board-operator/costs-and-budgets", "guides/board-operator/activity-log", + "guides/board-operator/deleting-connection-data", "guides/board-operator/importing-and-exporting" ] }, diff --git a/docs/guides/board-operator/deleting-connection-data.md b/docs/guides/board-operator/deleting-connection-data.md new file mode 100644 index 00000000000..831a8db2ca1 --- /dev/null +++ b/docs/guides/board-operator/deleting-connection-data.md @@ -0,0 +1,58 @@ +--- +title: Deleting Data Read Through a Connection +summary: Permanently remove what Paperclip saved from a connected app's tool calls +--- + +When an agent uses a connected app, such as Google Workspace, Paperclip saves part of what the app returned so you can review the work later. You can delete that saved data at any time, for any connected app. + +## What Paperclip saves + +For each tool call an agent makes through a connection, Paperclip keeps: + +- A short preview of the request and the result (up to about 4,000 characters, with secrets removed) +- For actions that needed approval, the preview and arguments shown on the approval card, and the result passed back to the agent +- Interactive app views that a tool showed in a task +- The transcript of the run that made the call, which can contain the full result, and of the run that received an approved action's result + +This data stays until you delete it. Removing the connection alone does not delete it. + +## How to delete it + +You need permission to manage the connection. + +**From the connection's page:** + +1. Open **Connectors** and select the connection. +2. At the bottom of the **Permissions** tab, select **Delete data**. +3. Check the counts in the confirmation and select **Delete data**. + +**When you remove a connection:** + +1. Open **Connectors**, open the connection's menu, and select **Remove connection**. +2. Select **Also delete data read through this connection**. +3. Select **Remove connection**. + +Deletion starts right away and continues in the background. It finishes even if Paperclip restarts while it runs. If a step fails, for example because file storage is unavailable, Paperclip keeps the request open and tries again until every step succeeds. It reports the deletion as finished only then. The connection's page shows when the data was last deleted. + +## What is deleted + +- Request and result previews for every call made through the connection before you asked to delete, including the copies in the connection's audit history +- Approval previews, arguments and results for those calls, on the approval card and in what was passed back to the agent +- Interactive app views from those calls +- The whole transcript of every run that called the connection or received an approved action's result, including that run's other steps and its final summary + +## What is kept + +- The record that each call happened: when, which tool, which agent, the outcome, the size of the data, and a fingerprint of it. Paperclip needs these for its activity history and usage counts. They do not contain the data. +- **Anything your agents wrote into tasks: comments, documents and files.** This action does not remove them, even when an agent copied data from the app into them. Delete or edit those items yourself, or delete the task. +- The connection's audit activity, which records counts only. +- Transcripts of later runs that only read the context an interactive app view shared. Paperclip does not record which runs read it. Delete those tasks if you need that text gone. +- Raw provider traces. An administrator must turn them on, and they expire on their own within 24 hours. + +## Runs that are still working + +Paperclip does not change a run while it is still working. The confirmation tells you how many runs are skipped for this reason. Select **Delete data** again after they finish. + +## It cannot be undone + +Deleted data cannot be recovered. diff --git a/packages/db/src/migrations/0490_nappy_chat.sql b/packages/db/src/migrations/0490_nappy_chat.sql new file mode 100644 index 00000000000..3f470177466 --- /dev/null +++ b/packages/db/src/migrations/0490_nappy_chat.sql @@ -0,0 +1,21 @@ +CREATE TABLE "tool_connection_content_deletions" ( + "id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL, + "company_id" uuid NOT NULL, + "connection_id" uuid NOT NULL, + "status" text DEFAULT 'pending' NOT NULL, + "requested_by_actor_type" text NOT NULL, + "requested_by_actor_id" text, + "requested_at" timestamp with time zone DEFAULT now() NOT NULL, + "progress" jsonb DEFAULT '{}'::jsonb NOT NULL, + "attempts" integer DEFAULT 0 NOT NULL, + "last_error" text, + "completed_at" timestamp with time zone, + "created_at" timestamp with time zone DEFAULT now() NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL +); +--> statement-breakpoint +ALTER TABLE "tool_connection_content_deletions" ADD CONSTRAINT "tool_connection_content_deletions_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +ALTER TABLE "tool_connection_content_deletions" ADD CONSTRAINT "tool_connection_content_deletions_connection_id_tool_connections_id_fk" FOREIGN KEY ("connection_id") REFERENCES "public"."tool_connections"("id") ON DELETE cascade ON UPDATE no action;--> statement-breakpoint +CREATE INDEX "tool_connection_content_deletions_connection_idx" ON "tool_connection_content_deletions" USING btree ("company_id","connection_id","created_at");--> statement-breakpoint +CREATE UNIQUE INDEX "tool_connection_content_deletions_pending_uq" ON "tool_connection_content_deletions" USING btree ("company_id","connection_id") WHERE "tool_connection_content_deletions"."status" = 'pending';--> statement-breakpoint +CREATE INDEX "tool_connection_content_deletions_pending_idx" ON "tool_connection_content_deletions" USING btree ("requested_at") WHERE "tool_connection_content_deletions"."status" = 'pending'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0485_snapshot.json b/packages/db/src/migrations/meta/0490_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0485_snapshot.json rename to packages/db/src/migrations/meta/0490_snapshot.json index fcf5987ed8c..6a945283d61 100644 --- a/packages/db/src/migrations/meta/0485_snapshot.json +++ b/packages/db/src/migrations/meta/0490_snapshot.json @@ -1,6 +1,6 @@ { - "id": "78d10bc7-b440-4c00-a02e-90111c50071d", - "prevId": "9389124c-379a-4b64-9c20-c7b37538e7c8", + "id": "a8bc890a-cc97-4b54-b426-5687b134a8b0", + "prevId": "dc61c7f3-cf3d-44c5-9296-86973e5f0925", "version": "7", "dialect": "postgresql", "tables": { @@ -36134,6 +36134,78 @@ "method": "btree", "with": {} }, + "heartbeat_runs_company_finished_idx": { + "name": "heartbeat_runs_company_finished_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "finished_at", + "isExpression": false, + "asc": false, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"finished_at\" is not null", + "concurrently": true, + "method": "btree", + "with": {} + }, + "heartbeat_runs_settled_process_identity_idx": { + "name": "heartbeat_runs_settled_process_identity_idx", + "columns": [ + { + "expression": "process_host_hash", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' and \"heartbeat_runs\".\"status\" in ('failed', 'cancelled', 'timed_out', 'interrupted')\n and \"heartbeat_runs\".\"finished_at\" is not null\n and (\"heartbeat_runs\".\"process_pid\" is not null or \"heartbeat_runs\".\"process_group_id\" is not null or \"heartbeat_runs\".\"process_started_at\" is not null)", + "concurrently": true, + "method": "btree", + "with": {} + }, + "heartbeat_runs_pending_api_stop_idx": { + "name": "heartbeat_runs_pending_api_stop_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"driver_kind\" = 'cursor_cloud_api'\n and (\"heartbeat_runs\".\"result_json\"->'executionCancellation'->>'state') = 'requested'", + "concurrently": true, + "method": "btree", + "with": {} + }, "heartbeat_runs_company_created_at_desc_idx": { "name": "heartbeat_runs_company_created_at_desc_idx", "columns": [ @@ -40827,6 +40899,131 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.issue_recovery_deferrals": { + "name": "issue_recovery_deferrals", + "schema": "", + "columns": { + "queue": { + "name": "queue", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "due_at": { + "name": "due_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": { + "issue_recovery_deferrals_issue_id_issues_id_fk": { + "name": "issue_recovery_deferrals_issue_id_issues_id_fk", + "tableFrom": "issue_recovery_deferrals", + "tableTo": "issues", + "columnsFrom": [ + "issue_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "issue_recovery_deferrals_company_id_companies_id_fk": { + "name": "issue_recovery_deferrals_company_id_companies_id_fk", + "tableFrom": "issue_recovery_deferrals", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "issue_recovery_deferrals_pk": { + "name": "issue_recovery_deferrals_pk", + "columns": [ + "queue", + "issue_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, + "public.issue_recovery_full_passes": { + "name": "issue_recovery_full_passes", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "queue": { + "name": "queue", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "started_at": { + "name": "started_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": {}, + "foreignKeys": { + "issue_recovery_full_passes_company_id_companies_id_fk": { + "name": "issue_recovery_full_passes_company_id_companies_id_fk", + "tableFrom": "issue_recovery_full_passes", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "issue_recovery_full_passes_pk": { + "name": "issue_recovery_full_passes_pk", + "columns": [ + "company_id", + "queue" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.issue_reference_mentions": { "name": "issue_reference_mentions", "schema": "", @@ -44884,6 +45081,123 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.link_previews": { + "name": "link_previews", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "viewer_id": { + "name": "viewer_id", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "channel": { + "name": "channel", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "resource_kind": { + "name": "resource_kind", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "canonical_url": { + "name": "canonical_url", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "checked_at": { + "name": "checked_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "expires_at": { + "name": "expires_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + } + }, + "indexes": { + "link_previews_scope_url_uq": { + "name": "link_previews_scope_url_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "viewer_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "canonical_url", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "link_previews_company_id_companies_id_fk": { + "name": "link_previews_company_id_companies_id_fk", + "tableFrom": "link_previews", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.managed_agent_profiles": { "name": "managed_agent_profiles", "schema": "", @@ -58387,6 +58701,197 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.tool_connection_content_deletions": { + "name": "tool_connection_content_deletions", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "connection_id": { + "name": "connection_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "requested_by_actor_type": { + "name": "requested_by_actor_type", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "requested_by_actor_id": { + "name": "requested_by_actor_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "requested_at": { + "name": "requested_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "progress": { + "name": "progress", + "type": "jsonb", + "primaryKey": false, + "notNull": true, + "default": "'{}'::jsonb" + }, + "attempts": { + "name": "attempts", + "type": "integer", + "primaryKey": false, + "notNull": true, + "default": 0 + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "completed_at": { + "name": "completed_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "tool_connection_content_deletions_connection_idx": { + "name": "tool_connection_content_deletions_connection_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_connection_content_deletions_pending_uq": { + "name": "tool_connection_content_deletions_pending_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "connection_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"tool_connection_content_deletions\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_connection_content_deletions_pending_idx": { + "name": "tool_connection_content_deletions_pending_idx", + "columns": [ + { + "expression": "requested_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_connection_content_deletions\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "tool_connection_content_deletions_company_id_companies_id_fk": { + "name": "tool_connection_content_deletions_company_id_companies_id_fk", + "tableFrom": "tool_connection_content_deletions", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "tool_connection_content_deletions_connection_id_tool_connections_id_fk": { + "name": "tool_connection_content_deletions_connection_id_tool_connections_id_fk", + "tableFrom": "tool_connection_content_deletions", + "tableTo": "tool_connections", + "columnsFrom": [ + "connection_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.tool_connection_installs": { "name": "tool_connection_installs", "schema": "", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index ee03bddd7a5..b7bf09d719e 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3417,6 +3417,13 @@ "when": 1794841384470, "tag": "0489_real_sage", "breakpoints": true + }, + { + "idx": 490, + "version": "7", + "when": 1794923393849, + "tag": "0490_nappy_chat", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index f27b014644c..fd3261f77b3 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -291,3 +291,4 @@ export { budgetEvaluationWork } from "./budget_evaluation_work.js"; export { tailscaleConnections } from "./tailscale_connections.js"; export { nativeProviderArtifacts, nativeProviderArtifactReferences } from "./native_provider_artifacts.js"; export { linkPreviews } from "./link_previews.js"; +export { toolConnectionContentDeletions } from "./tool_connection_content_deletions.js"; diff --git a/packages/db/src/schema/tool_connection_content_deletions.ts b/packages/db/src/schema/tool_connection_content_deletions.ts new file mode 100644 index 00000000000..93c29a7f834 --- /dev/null +++ b/packages/db/src/schema/tool_connection_content_deletions.ts @@ -0,0 +1,32 @@ +import { sql } from "drizzle-orm"; +import { index, integer, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core"; +import { companies } from "./companies.js"; +import { toolConnections } from "./tool_access.js"; + +export const toolConnectionContentDeletions = pgTable( + "tool_connection_content_deletions", + { + id: uuid("id").primaryKey().defaultRandom(), + companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), + connectionId: uuid("connection_id").notNull().references(() => toolConnections.id, { onDelete: "cascade" }), + status: text("status").$type<"pending" | "completed">().notNull().default("pending"), + requestedByActorType: text("requested_by_actor_type").notNull(), + requestedByActorId: text("requested_by_actor_id"), + requestedAt: timestamp("requested_at", { withTimezone: true }).notNull().defaultNow(), + progress: jsonb("progress").$type>().notNull().default({}), + attempts: integer("attempts").notNull().default(0), + lastError: text("last_error"), + completedAt: timestamp("completed_at", { withTimezone: true }), + createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), + }, + (table) => [ + index("tool_connection_content_deletions_connection_idx").on(table.companyId, table.connectionId, table.createdAt), + uniqueIndex("tool_connection_content_deletions_pending_uq") + .on(table.companyId, table.connectionId) + .where(sql`${table.status} = 'pending'`), + index("tool_connection_content_deletions_pending_idx") + .on(table.requestedAt) + .where(sql`${table.status} = 'pending'`), + ], +); diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 825867d65bb..6a7bc609575 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1542,6 +1542,11 @@ export type { ToolConnectionInstallTargetType, ToolConnectionRemovalResult, ToolConnectionRemovalSummary, + ToolConnectionStoredContent, + ToolConnectionStoredContentCounts, + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionContentDeletionStatus, ConnectionAudienceMember, ConnectionGrant, ConnectionGrantCapabilities, diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 8abf1f4ed91..9c93d353993 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -562,6 +562,11 @@ export type { ToolConnectionInstallTargetType, ToolConnectionRemovalResult, ToolConnectionRemovalSummary, + ToolConnectionStoredContent, + ToolConnectionStoredContentCounts, + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionContentDeletionStatus, ConnectionTokenAttribution, ConnectionRecoverableErrorCode, ConnectionRecoverableErrorPayload, diff --git a/packages/shared/src/types/tool-access.ts b/packages/shared/src/types/tool-access.ts index 1d2c29fb7c0..32b0ad72fd9 100644 --- a/packages/shared/src/types/tool-access.ts +++ b/packages/shared/src/types/tool-access.ts @@ -113,6 +113,7 @@ export interface ToolRedactedValueSummary { sha256?: string | null; redactedFields?: string[]; artifactId?: string | null; + contentDeletedAt?: string; } export type VercelConnectPrincipalMode = "app" | "user"; @@ -433,6 +434,40 @@ export interface ToolConnectionRemovalResult { removal: ToolConnectionRemovalSummary; } +export interface ToolConnectionStoredContentCounts { + toolCalls: number; + approvals: number; + appViews: number; + runs: number; + runsInProgress: number; +} + +export interface ToolConnectionContentDeletionCounts { + toolCalls: number; + approvals: number; + auditEvents: number; + appViews: number; + runs: number; + runsInProgress: number; + itemsInProgress: number; +} + +export type ToolConnectionContentDeletionStatus = "pending" | "completed"; + +export interface ToolConnectionContentDeletion { + id: string; + status: ToolConnectionContentDeletionStatus; + requestedAt: string; + completedAt: string | null; + retrying: boolean; + counts: ToolConnectionContentDeletionCounts; +} + +export interface ToolConnectionStoredContent { + stored: ToolConnectionStoredContentCounts; + deletion: ToolConnectionContentDeletion | null; +} + export type ConnectionTokenScope = string | string[]; export type ConnectionTokenSubject = { type: "app" } | { type: "user"; userId: string }; diff --git a/server/src/__tests__/tool-connection-content-deletion-canary.test.ts b/server/src/__tests__/tool-connection-content-deletion-canary.test.ts new file mode 100644 index 00000000000..7aea9491925 --- /dev/null +++ b/server/src/__tests__/tool-connection-content-deletion-canary.test.ts @@ -0,0 +1,346 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { Readable } from "node:stream"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { sql } from "drizzle-orm"; +import { + agentWakeupRequests, + agents, + companies, + createDb, + heartbeatRunEvents, + heartbeatRuns, + issueComments, + issues, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolApplications, + toolCallEvents, + toolConnections, + toolInvocations, + toolMcpAppViews, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import { composeRunLogStore, createDurableRunLogStore, type RunLogHandle, type RunLogStore } from "../services/run-log-store.js"; +import { createObjectRunLogStore } from "../services/object-run-log-store.js"; +import { summarizeToolValue } from "../services/tool-content-guards.js"; +import type { StorageProvider } from "../storage/types.js"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +const OUT_OF_SCOPE_COLUMNS = new Map([ + ["issue_comments.body", "Comments an agent wrote into the task. The user removes them by editing or deleting the comment or the task."], +]); + +function memoryProvider() { + const objects = new Map(); + const etag = (bytes: Buffer) => String(bytes.length) + ":" + bytes.toString("base64").slice(0, 32); + const read = async (body: Buffer | AsyncIterable) => { + if (Buffer.isBuffer(body)) return Buffer.from(body); + const chunks: Buffer[] = []; + for await (const chunk of body) chunks.push(Buffer.from(chunk)); + return Buffer.concat(chunks); + }; + const provider: StorageProvider = { + id: "s3", + async putObject(input) { objects.set(input.objectKey, await read(input.body)); }, + async putObjectConditional(input) { + const old = objects.get(input.objectKey); + if (input.ifNoneMatch ? old : !old || etag(old) !== input.ifMatch) + throw Object.assign(new Error("conflict"), { name: "PreconditionFailed" }); + const bytes = await read(input.body); + objects.set(input.objectKey, bytes); + return { etag: etag(bytes) }; + }, + async getObject(input) { + const bytes = objects.get(input.objectKey); + if (!bytes) throw Object.assign(new Error("missing"), { name: "NoSuchKey" }); + const slice = input.range ? bytes.subarray(input.range.start, input.range.end + 1) : bytes; + return { stream: Readable.from([slice]), contentLength: slice.length, etag: etag(bytes) }; + }, + async headObject(input) { + const bytes = objects.get(input.objectKey); + return bytes ? { exists: true, contentLength: bytes.length, etag: etag(bytes) } : { exists: false }; + }, + async deleteObject(input) { objects.delete(input.objectKey); }, + async listObjects(input) { return { keys: [...objects.keys()].filter((key) => key.startsWith(input.prefix)) }; }, + }; + return { provider, objects }; +} + +async function filesContaining(dir: string, marker: string): Promise { + const hits: string[] = []; + const entries = await fs.readdir(dir, { withFileTypes: true }).catch(() => []); + for (const entry of entries) { + const full = path.join(dir, entry.name); + if (entry.isDirectory()) hits.push(...await filesContaining(full, marker)); + else if ((await fs.readFile(full)).includes(marker)) hits.push(full); + } + return hits; +} + +describeEmbeddedPostgres("deleting a connection's stored content leaves no copy behind", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + let baseDir = ""; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-content-canary-"); + db = createDb(tempDb.connectionString); + baseDir = await fs.mkdtemp(path.join(os.tmpdir(), "content-canary-")); + }, 30_000); + + afterAll(async () => { + await tempDb?.cleanup(); + await fs.rm(baseDir, { recursive: true, force: true }); + }); + + function stores(dir: string) { + const mirror = memoryProvider(); + const objects = memoryProvider(); + const local = createDurableRunLogStore({ basePath: dir, s3: { provider: mirror.provider } }); + const external = createObjectRunLogStore({ provider: objects.provider, keyPrefix: "run-logs", assertWriterAuthority: async () => undefined }); + return { dir, mirror, objects, local, external, store: composeRunLogStore(local, external) }; + } + + async function writeLog(store: RunLogStore, ids: { companyId: string; agentId: string; runId: string }, text: string) { + const handle = await store.begin(ids); + await store.append(handle, { stream: "stdout", chunk: text, ts: new Date().toISOString(), seq: 1 }); + await store.finalize(handle); + return handle; + } + + async function seedCompany() { + const [company] = await db.insert(companies).values({ name: `Canary ${randomUUID()}`, issuePrefix: `CN${randomUUID().slice(0, 6).toUpperCase()}` }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company!.id, name: "Canary agent", role: "engineer", status: "active", adapterType: "process", adapterConfig: {}, runtimeConfig: {}, + }).returning(); + const [application] = await db.insert(toolApplications).values({ companyId: company!.id, name: `Docs ${randomUUID().slice(0, 6)}`, type: "mcp_http" }).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Summarise the drive" }).returning(); + return { company: company!, agent: agent!, application: application!, issue: issue! }; + } + + async function seedConnectionFlow( + base: Awaited>, + logs: ReturnType, + marker: string, + ) { + const companyId = base.company.id; + const [connection] = await db.insert(toolConnections).values({ + companyId, applicationId: base.application.id, name: `Docs ${marker}`.slice(0, 40), uid: randomUUID(), transport: "mcp_remote", status: "active", + }).returning(); + const finishedAt = new Date(Date.now() - 60_000); + const created = new Date(Date.now() - 30_000); + + const callingRunId = randomUUID(); + const callingLog = await writeLog(logs.store, { companyId, agentId: base.agent.id, runId: callingRunId }, + JSON.stringify({ tool: "read_doc", output: `Document says ${marker}` })); + await db.insert(heartbeatRuns).values({ + id: callingRunId, companyId, agentId: base.agent.id, issueId: base.issue.id, status: "succeeded", finishedAt, runtimeMode: "native", + logStore: callingLog.store, logRef: callingLog.logRef, stdoutExcerpt: `read_doc -> ${marker}`, + resultJson: { + summary: `The drive says ${marker}`, + finalizationPhase: "committed", + nativeResult: { schema: "paperclip.run_result.v1", summary: `Answer quoting ${marker}`, reportedWorkDisposition: "completed", + continuation: { kind: "response_wake", summary: `Continue with ${marker}`, idempotencyKey: "k1" } }, + }, + }); + await db.insert(heartbeatRunEvents).values([ + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 1, eventType: "lifecycle", message: "run started", payload: { phase: "start" } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 2, eventType: "tool.execution.started", message: "read_doc", payload: { arguments: { query: marker } } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 3, eventType: "tool.execution.completed", message: marker, payload: { output: marker } }, + { companyId, runId: callingRunId, agentId: base.agent.id, seq: 4, eventType: "item.completed", payload: { prpEvent: { payload: { kind: "assistant_message", text: `I found ${marker}` } } } }, + ]); + + const args = summarizeToolValue({ query: marker }); + const result = summarizeToolValue({ text: `Result ${marker}` }); + const [readCall, writeCall] = await db.insert(toolInvocations).values([ + { companyId, connectionId: connection!.id, runId: callingRunId, agentId: base.agent.id, issueId: base.issue.id, toolName: "read_doc", status: "succeeded", + argumentsSummary: args, argumentsHash: args.sha256, resultSummary: result, resultHash: result.sha256, resultSizeBytes: result.sizeBytes, + progress: { message: `Reading ${marker}` } as never, createdAt: created }, + { companyId, connectionId: connection!.id, runId: callingRunId, agentId: base.agent.id, issueId: base.issue.id, toolName: "send_doc", status: "succeeded", + approvalState: "approved", argumentsSummary: args, resultSummary: result, createdAt: created }, + ]).returning(); + await db.insert(toolCallEvents).values([ + { companyId, connectionId: connection!.id, runId: callingRunId, invocationId: readCall!.id, eventType: "call_started", argumentsSummary: args, requestSummary: args, createdAt: created }, + { companyId, connectionId: connection!.id, runId: callingRunId, invocationId: readCall!.id, eventType: "call_completed", resultSummary: result, resultHash: result.sha256, createdAt: created }, + ]); + await db.insert(toolAccessAuditEvents).values({ + companyId, connectionId: connection!.id, action: "tool_access.policy_decision", outcome: "success", createdAt: created, + details: { decision: "allow", runId: callingRunId, toolName: "read_doc", argumentsSummary: args, redactionPlan: { redactedFieldCount: 0 } }, + }); + + const [card] = await db.insert(issueThreadInteractions).values({ + companyId, issueId: base.issue.id, kind: "request_confirmation", status: "resolved", + payload: { version: 1, prompt: "Approve send_doc?", detailsMarkdown: `Will send ${marker}`, + toolAction: { version: 1, toolName: "send_doc", previewMarkdown: `Preview ${marker}`, argumentsSummaryJson: args.summary } } as never, + result: { version: 1, outcome: "accepted", toolAction: { version: 1, status: "executed", resultSummary: result.summary } } as never, + }).returning(); + const [actionRequest] = await db.insert(toolActionRequests).values({ + companyId, invocationId: writeCall!.id, issueId: base.issue.id, interactionId: card!.id, status: "executed", + canonicalArgumentsHash: "args", canonicalArgumentsSummary: args, signedArguments: `signed:${marker}`, previewMarkdown: `Preview ${marker}`, + }).returning(); + + await db.insert(toolMcpAppViews).values({ + companyId, connectionId: connection!.id, issueId: base.issue.id, runId: callingRunId, invocationId: readCall!.id, + applicationName: "Docs", toolName: "read_doc", upstreamToolName: "read_doc", resourceUri: "ui://docs/view", + toolInput: { query: marker }, toolResult: { content: [{ type: "text", text: marker }] }, + modelContext: { content: [{ type: "text", text: marker }] } as never, createdAt: created, + }); + + await db.insert(issueComments).values({ companyId, issueId: base.issue.id, authorAgentId: base.agent.id, body: `Agent note: ${marker}` }); + + return { connection: connection!, actionRequest: actionRequest!, result, callingRunId }; + } + + async function seedDeliveryRun( + base: Awaited>, + logs: ReturnType, + delivered: Array<{ actionRequestId: string; marker: string }>, + ) { + const companyId = base.company.id; + const runId = randomUUID(); + const ids = { companyId, agentId: base.agent.id, runId }; + const text = delivered.map((entry) => `Approved result ${entry.marker}`).join("\n"); + await writeLog(logs.local, ids, text); + const migrated = await writeLog(logs.external, ids, text); + const actions = delivered.map((entry) => ({ actionRequestId: entry.actionRequestId, toolName: "send_doc", resultSummary: `Result ${entry.marker}` })); + const context = { + issueId: base.issue.id, toolAction: actions[0], toolActions: actions, toolActionRequestIds: delivered.map((entry) => entry.actionRequestId), + paperclipAgentMessage: { text: "There are recorded connection outcomes.", untrustedToolResults: actions }, + }; + await db.insert(heartbeatRuns).values({ + id: runId, companyId, agentId: base.agent.id, issueId: base.issue.id, status: "succeeded", finishedAt: new Date(Date.now() - 10_000), + logStore: migrated.store, logRef: migrated.logRef, contextSnapshot: context, resultJson: { summary: text }, + }); + await db.insert(agentWakeupRequests).values({ + companyId, agentId: base.agent.id, source: "automation", status: "completed", + payload: { issueId: base.issue.id, _paperclipWakeContext: context }, + }); + return runId; + } + + async function scan(markers: string[], logs: ReturnType) { + const columns = await db.execute(sql`select table_name, column_name from information_schema.columns + where table_schema = 'public' and data_type in ('text', 'character varying', 'jsonb', 'json')`); + const hits: string[] = []; + for (const row of columns as unknown as Array<{ table_name: string; column_name: string }>) { + const name = `${row.table_name}.${row.column_name}`; + if (OUT_OF_SCOPE_COLUMNS.has(name)) continue; + for (const marker of markers) { + const found = await db.execute(sql`select 1 from ${sql.identifier(row.table_name)} + where ${sql.identifier(row.column_name)}::text like ${`%${marker}%`} limit 1`); + if ((found as unknown as unknown[]).length > 0) hits.push(`${name} contains ${marker}`); + } + } + for (const marker of markers) { + for (const [label, objects] of [["object store", logs.objects.objects], ["legacy mirror", logs.mirror.objects]] as const) { + for (const [key, bytes] of objects) if (bytes.includes(marker)) hits.push(`${label} ${key} contains ${marker}`); + } + for (const file of await filesContaining(logs.dir, marker)) hits.push(`file ${file} contains ${marker}`); + } + return hits; + } + + async function outOfScopeStillHolds(marker: string) { + const rows = await db.execute(sql`select 1 from issue_comments where body like ${`%${marker}%`}`); + return (rows as unknown as unknown[]).length > 0; + } + + async function drain(service: ReturnType, deletionIds: string[]) { + for (let i = 0; i < 200; i++) { + const jobs = await Promise.all(deletionIds.map((id) => service.process(id, new AbortController().signal))); + if (jobs.every((job) => job?.status === "completed")) return; + } + throw new Error("deletion did not finish"); + } + + async function seenBefore(markers: string[], logs: ReturnType) { + const hits = await scan(markers, logs); + for (const marker of markers) expect(hits.some((hit) => hit.includes(marker))).toBe(true); + expect(hits.some((hit) => hit.startsWith("object store"))).toBe(true); + expect(hits.some((hit) => hit.startsWith("legacy mirror"))).toBe(true); + expect(hits.some((hit) => hit.startsWith("file "))).toBe(true); + } + + it("leaves the marker nowhere after a deletion runs to completion", async () => { + const logs = stores(path.join(baseDir, "complete")); + const base = await seedCompany(); + const marker = `CANARY${randomUUID().replace(/-/g, "")}`; + const flow = await seedConnectionFlow(base, logs, marker); + await seedDeliveryRun(base, logs, [{ actionRequestId: flow.actionRequest.id, marker }]); + await seenBefore([marker], logs); + + const service = createConnectionContentDeletion(db, { runLogStore: () => logs.store }); + const before = await service.storedContent(flow.connection); + expect(before.stored).toMatchObject({ toolCalls: 2, approvals: 1, appViews: 1, runs: 2, runsInProgress: 0 }); + const { deletion } = await service.request(flow.connection, { actorType: "user", actorId: "board-user" }); + await drain(service, [deletion.id]); + + expect(await scan([marker], logs)).toEqual([]); + expect(await outOfScopeStillHolds(marker)).toBe(true); + expect((await service.storedContent(flow.connection)).stored).toEqual({ toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 }); + }); + + it("leaves the marker nowhere after an interrupted deletion resumes", async () => { + const logs = stores(path.join(baseDir, "interrupted")); + const base = await seedCompany(); + const marker = `CANARY${randomUUID().replace(/-/g, "")}`; + const flow = await seedConnectionFlow(base, logs, marker); + await seedDeliveryRun(base, logs, [{ actionRequestId: flow.actionRequest.id, marker }]); + await seenBefore([marker], logs); + + let failures = 2; + const flaky: RunLogStore = { + ...logs.store, + remove: async (handle: RunLogHandle, signal?: AbortSignal) => { + if (failures > 0) { + failures -= 1; + throw new Error("object storage unavailable"); + } + return logs.store.remove!(handle, signal); + }, + }; + const service = createConnectionContentDeletion(db, { runLogStore: () => flaky }); + const { deletion } = await service.request(flow.connection, { actorType: "user", actorId: "board-user" }); + let interrupted = 0; + for (let i = 0; i < 200; i++) { + const job = await service.process(deletion.id, new AbortController().signal).catch(() => { + interrupted += 1; + return null; + }); + if (job?.status === "completed") break; + } + expect(interrupted).toBe(2); + expect(await scan([marker], logs)).toEqual([]); + expect(await outOfScopeStillHolds(marker)).toBe(true); + }); + + it("leaves no marker after two deletions for different connections run at the same time on one task", async () => { + const logs = stores(path.join(baseDir, "concurrent")); + const base = await seedCompany(); + const first = `CANARY${randomUUID().replace(/-/g, "")}`; + const second = `CANARY${randomUUID().replace(/-/g, "")}`; + const flowOne = await seedConnectionFlow(base, logs, first); + const flowTwo = await seedConnectionFlow(base, logs, second); + await seedDeliveryRun(base, logs, [ + { actionRequestId: flowOne.actionRequest.id, marker: first }, + { actionRequestId: flowTwo.actionRequest.id, marker: second }, + ]); + await seenBefore([first, second], logs); + + const service = createConnectionContentDeletion(db, { runLogStore: () => logs.store }); + const one = await service.request(flowOne.connection, { actorType: "user", actorId: "board-user" }); + const two = await service.request(flowTwo.connection, { actorType: "user", actorId: "board-user" }); + await drain(service, [one.deletion.id, two.deletion.id]); + + expect(await scan([first, second], logs)).toEqual([]); + expect(await outOfScopeStillHolds(first)).toBe(true); + expect(await outOfScopeStillHolds(second)).toBe(true); + }); +}); diff --git a/server/src/__tests__/tool-connection-content-deletion.test.ts b/server/src/__tests__/tool-connection-content-deletion.test.ts new file mode 100644 index 00000000000..c7123f12d61 --- /dev/null +++ b/server/src/__tests__/tool-connection-content-deletion.test.ts @@ -0,0 +1,420 @@ +import { randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { and, eq, sql } from "drizzle-orm"; +import { + activityLog, + agentWakeupRequests, + agents, + companies, + createDb, + heartbeatRunEvents, + heartbeatRuns, + issues, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolApplications, + toolCallEvents, + toolConnectionContentDeletions, + toolConnections, + toolInvocations, + toolMcpAppViews, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import type { RunLogHandle, RunLogStore } from "../services/run-log-store.js"; +import { toolAccessRoutes } from "../routes/tool-access.js"; +import { errorHandler } from "../middleware/index.js"; + +const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; + +const SECRET = "Quarterly board notes from the shared drive"; + +function memoryRunLogs() { + const logs = new Map(); + let failuresLeft = 0; + const store = { + remove: async (handle: RunLogHandle) => { + if (failuresLeft > 0) { + failuresLeft -= 1; + throw new Error("object storage unavailable"); + } + logs.delete(handle.logRef); + }, + } as unknown as RunLogStore; + return { logs, store, failNext: (count: number) => { failuresLeft = count; } }; +} + +describeEmbeddedPostgres("deleting content stored from a connection's tool calls", () => { + let db!: ReturnType; + let tempDb: Awaited> | null = null; + + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-content-deletion-"); + db = createDb(tempDb.connectionString); + }, 30_000); + + afterAll(async () => { + await tempDb?.cleanup(); + }); + + async function seedCompany(runLogs: ReturnType) { + const [company] = await db.insert(companies).values({ + name: `Deletion ${randomUUID()}`, + issuePrefix: `DL${randomUUID().slice(0, 6).toUpperCase()}`, + }).returning(); + const [agent] = await db.insert(agents).values({ + companyId: company!.id, + name: `Agent ${randomUUID().slice(0, 6)}`, + role: "engineer", + status: "active", + adapterType: "process", + adapterConfig: {}, + runtimeConfig: {}, + }).returning(); + const [application] = await db.insert(toolApplications).values({ + companyId: company!.id, + name: `Docs ${randomUUID().slice(0, 6)}`, + type: "mcp_http", + }).returning(); + const [connection, other] = await db.insert(toolConnections).values([ + { companyId: company!.id, applicationId: application!.id, name: "Docs", uid: randomUUID(), transport: "mcp_remote", status: "active" }, + { companyId: company!.id, applicationId: application!.id, name: "Other", uid: randomUUID(), transport: "mcp_remote", status: "active" }, + ]).returning(); + const [issue] = await db.insert(issues).values({ companyId: company!.id, title: "Read the notes" }).returning(); + const finishedAt = new Date(Date.now() - 60_000); + const [settledRun, liveRun, otherRun] = await db.insert(heartbeatRuns).values([ + { companyId: company!.id, agentId: agent!.id, status: "succeeded", finishedAt, logStore: "local_file", logRef: `${company!.id}/settled.ndjson`, logBytes: 120, logSha256: "abc", stdoutExcerpt: SECRET, resultJson: { summary: SECRET, executionRecovery: { kind: "none" }, nativeResult: { schema: "paperclip.run_result.v1", summary: SECRET, continuation: { kind: "response_wake", summary: SECRET } } } }, + { companyId: company!.id, agentId: agent!.id, status: "running", logStore: "local_file", logRef: `${company!.id}/live.ndjson`, stdoutExcerpt: SECRET }, + { companyId: company!.id, agentId: agent!.id, status: "succeeded", finishedAt, logStore: "local_file", logRef: `${company!.id}/other.ndjson`, stdoutExcerpt: SECRET }, + ]).returning(); + for (const run of [settledRun!, liveRun!, otherRun!]) runLogs.logs.set(run.logRef!, SECRET); + const summary = { summary: SECRET, sizeBytes: 42, sha256: "hash-42", redactedFields: [] }; + const created = new Date(Date.now() - 30_000); + const [settledCall, liveCall, otherCall] = await db.insert(toolInvocations).values([ + { companyId: company!.id, connectionId: connection!.id, runId: settledRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, resultHash: "hash-42", resultSizeBytes: 42, createdAt: created }, + { companyId: company!.id, connectionId: connection!.id, runId: liveRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, createdAt: created }, + { companyId: company!.id, connectionId: other!.id, runId: otherRun!.id, toolName: "read_doc", status: "succeeded", argumentsSummary: summary, resultSummary: summary, createdAt: created }, + ]).returning(); + await db.insert(toolCallEvents).values([ + { companyId: company!.id, connectionId: connection!.id, runId: settledRun!.id, invocationId: settledCall!.id, eventType: "call_completed", requestSummary: summary, resultSummary: summary, resultHash: "hash-42", createdAt: created }, + { companyId: company!.id, connectionId: connection!.id, runId: liveRun!.id, invocationId: liveCall!.id, eventType: "call_completed", resultSummary: summary, createdAt: created }, + { companyId: company!.id, connectionId: other!.id, runId: otherRun!.id, invocationId: otherCall!.id, eventType: "call_completed", resultSummary: summary, createdAt: created }, + ]); + const [interaction] = await db.insert(issueThreadInteractions).values({ + companyId: company!.id, + issueId: issue!.id, + kind: "request_confirmation", + status: "resolved", + payload: { version: 1, prompt: "Approve?", detailsMarkdown: SECRET, toolAction: { version: 1, previewMarkdown: SECRET, argumentsSummaryJson: SECRET } } as never, + result: { version: 1, toolAction: { version: 1, status: "executed", resultSummary: SECRET } } as never, + }).returning(); + const [actionRequest] = await db.insert(toolActionRequests).values({ + companyId: company!.id, + issueId: issue!.id, + invocationId: settledCall!.id, + interactionId: interaction!.id, + status: "executed", + canonicalArgumentsHash: "args-hash", + canonicalArgumentsSummary: summary, + signedArguments: "signed", + previewMarkdown: SECRET, + }).returning(); + const delivered = { actionRequestId: actionRequest!.id, toolName: "read_doc", resultSummary: SECRET }; + const deliveryContext = { issueId: issue!.id, toolAction: delivered, toolActions: [delivered], toolActionRequestIds: [actionRequest!.id], paperclipAgentMessage: { untrustedToolResults: [delivered] } }; + const [deliveryRun] = await db.insert(heartbeatRuns).values({ + companyId: company!.id, agentId: agent!.id, issueId: issue!.id, status: "succeeded", finishedAt, contextSnapshot: deliveryContext, + logStore: "local_file", logRef: `${company!.id}/delivery.ndjson`, stdoutExcerpt: SECRET, + }).returning(); + runLogs.logs.set(deliveryRun!.logRef!, SECRET); + const [deliveryWake] = await db.insert(agentWakeupRequests).values({ + companyId: company!.id, agentId: agent!.id, source: "automation", status: "completed", + payload: { issueId: issue!.id, _paperclipWakeContext: deliveryContext }, + }).returning(); + const [accessAudit] = await db.insert(toolAccessAuditEvents).values({ + companyId: company!.id, connectionId: connection!.id, action: "tool_access.policy_decision", outcome: "success", + details: { decision: "allow", runId: settledRun!.id, toolName: "read_doc", argumentsSummary: summary }, + createdAt: created, + }).returning(); + await db.insert(toolMcpAppViews).values({ + companyId: company!.id, + connectionId: connection!.id, + runId: settledRun!.id, + invocationId: settledCall!.id, + applicationName: "Docs", + toolName: "read_doc", + upstreamToolName: "read_doc", + resourceUri: "ui://docs/view", + toolInput: { id: "doc-1" }, + toolResult: { text: SECRET }, + createdAt: created, + }); + await db.insert(heartbeatRunEvents).values([ + { companyId: company!.id, runId: settledRun!.id, agentId: agent!.id, seq: 1, eventType: "lifecycle", message: "run started", payload: { phase: "start" } }, + { companyId: company!.id, runId: settledRun!.id, agentId: agent!.id, seq: 2, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + { companyId: company!.id, runId: liveRun!.id, agentId: agent!.id, seq: 1, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + { companyId: company!.id, runId: otherRun!.id, agentId: agent!.id, seq: 1, eventType: "tool.execution.completed", message: SECRET, payload: { output: SECRET } }, + ]); + return { deliveryRun: deliveryRun!, deliveryWake: deliveryWake!, accessAudit: accessAudit!, interaction: interaction!, company: company!, agent: agent!, connection: connection!, other: other!, settledRun: settledRun!, liveRun: liveRun!, otherRun: otherRun!, settledCall: settledCall!, liveCall: liveCall!, otherCall: otherCall! }; + } + + async function drain(service: ReturnType) { + for (let i = 0; i < 20 && await service.nextDeadline() !== null; i++) { + await service.processPending(new AbortController().signal); + } + } + + it("removes content from every store, keeps rows and hashes, and leaves other connections and companies alone", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const foreign = await seedCompany(runLogs); + + const before = await service.storedContent(seed.connection); + expect(before.stored).toEqual({ toolCalls: 2, approvals: 1, appViews: 1, runs: 3, runsInProgress: 1 }); + + const { created } = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + expect(created).toBe(true); + await drain(service); + + const [call] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, seed.settledCall.id)); + expect(call!.resultSummary).toMatchObject({ summary: "", sizeBytes: 42, sha256: "hash-42" }); + expect(call!.resultSummary?.contentDeletedAt).toBeTruthy(); + expect(call!.argumentsSummary?.summary).toBe(""); + expect(call!.resultHash).toBe("hash-42"); + expect(call!.status).toBe("succeeded"); + + const events = await db.select().from(toolCallEvents).where(eq(toolCallEvents.invocationId, seed.settledCall.id)); + expect(events).toHaveLength(1); + expect(events[0]!.resultSummary?.summary).toBe(""); + expect(events[0]!.requestSummary?.summary).toBe(""); + expect(events[0]!.resultHash).toBe("hash-42"); + + const [actionRequest] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.invocationId, seed.settledCall.id)); + expect(actionRequest!.previewMarkdown).toBeNull(); + expect(actionRequest!.signedArguments).toBeNull(); + expect(actionRequest!.canonicalArgumentsSummary.summary).toBe(""); + expect(actionRequest!.canonicalArgumentsHash).toBe("args-hash"); + const [interaction] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, actionRequest!.interactionId!)); + expect(JSON.stringify(interaction!.payload)).not.toContain(SECRET); + expect(JSON.stringify(interaction!.result)).not.toContain(SECRET); + + expect(await db.select().from(toolMcpAppViews).where(eq(toolMcpAppViews.connectionId, seed.connection.id))).toEqual([]); + + const [settledRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(settledRun).toMatchObject({ logStore: null, logRef: null, stdoutExcerpt: null, logBytes: 120, logSha256: "abc", resultJson: { executionRecovery: { kind: "none" }, nativeResult: { schema: "paperclip.run_result.v1", continuation: { kind: "response_wake" } } } }); + expect(JSON.stringify(settledRun!.resultJson)).not.toContain(SECRET); + const [audit] = await db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.id, seed.accessAudit.id)); + expect(audit!.details).toMatchObject({ decision: "allow", toolName: "read_doc", argumentsSummary: { summary: "", sha256: "hash-42", sizeBytes: 42 } }); + const [deliveryRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.deliveryRun.id)); + expect(JSON.stringify(deliveryRun!.contextSnapshot)).not.toContain(SECRET); + expect(deliveryRun!.contextSnapshot).toMatchObject({ toolActionRequestIds: [expect.any(String)] }); + expect(deliveryRun).toMatchObject({ logRef: null, stdoutExcerpt: null }); + expect(runLogs.logs.has(seed.deliveryRun.logRef!)).toBe(false); + const [deliveryWake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, seed.deliveryWake.id)); + expect(JSON.stringify(deliveryWake!.payload)).not.toContain(SECRET); + expect(runLogs.logs.has(seed.settledRun.logRef!)).toBe(false); + const settledEvents = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seed.settledRun.id)); + expect(settledEvents).toHaveLength(2); + expect(JSON.stringify(settledEvents)).not.toContain(SECRET); + expect(settledEvents.find((event) => event.eventType === "lifecycle")!.message).toBe("run started"); + + const [liveRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.liveRun.id)); + expect(liveRun!.logRef).toBe(seed.liveRun.logRef); + expect(liveRun!.stdoutExcerpt).toBe(SECRET); + expect(runLogs.logs.get(seed.liveRun.logRef!)).toBe(SECRET); + const [liveCall] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, seed.liveCall.id)); + expect(liveCall!.resultSummary?.summary).toBe(SECRET); + const liveEvents = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, seed.liveRun.id)); + expect(liveEvents[0]!.message).toBe(SECRET); + + for (const untouched of [seed.otherCall.id, foreign.settledCall.id]) { + const [row] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, untouched)); + expect(row!.resultSummary?.summary).toBe(SECRET); + } + for (const run of [seed.otherRun, foreign.settledRun]) { + const [row] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + expect(row!.logRef).toBe(run.logRef); + expect(runLogs.logs.get(run.logRef!)).toBe(SECRET); + } + expect(await db.select().from(toolMcpAppViews).where(eq(toolMcpAppViews.connectionId, foreign.connection.id))).toHaveLength(1); + + const after = await service.storedContent(seed.connection); + expect(after.stored).toEqual({ toolCalls: 1, approvals: 0, appViews: 0, runs: 1, runsInProgress: 1 }); + expect(after.deletion).toMatchObject({ + status: "completed", + counts: { toolCalls: 1, approvals: 1, auditEvents: 2, appViews: 1, runs: 2, runsInProgress: 1 }, + }); + const [activity] = await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deleted"), + )); + expect(activity!.entityId).toBe(seed.connection.id); + expect(JSON.stringify(activity!.details)).not.toContain(SECRET); + + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, seed.liveRun.id)); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + await drain(service); + expect(runLogs.logs.has(seed.liveRun.logRef!)).toBe(false); + expect((await service.storedContent(seed.connection)).stored).toEqual({ toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 }); + }); + + it("resumes an interrupted deletion to completion on the next pass", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + runLogs.failNext(1); + await expect(service.processPending(new AbortController().signal)).rejects.toThrow("object storage unavailable"); + + const [interrupted] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.connection.id)); + expect(interrupted).toMatchObject({ status: "pending", attempts: 1, lastError: "object storage unavailable" }); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "pending", retrying: true }); + expect(await service.nextDeadline()).not.toBeNull(); + + await drain(service); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(run!.logRef).toBeNull(); + expect(runLogs.logs.has(seed.settledRun.logRef!)).toBe(false); + const finished = await service.storedContent(seed.connection); + expect(finished.deletion).toMatchObject({ status: "completed", retrying: false, counts: { toolCalls: 1, runs: 2 } }); + }); + + it("returns the pending deletion instead of starting a second one", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const first = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + const second = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + expect(second).toEqual({ deletion: first.deletion, created: false }); + await drain(service); + }); + + it("clears the approval card on the retry after the card update failed once", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await db.execute(sql`create table if not exists content_deletion_card_failures (remaining int not null)`); + await db.execute(sql`delete from content_deletion_card_failures`); + await db.execute(sql`insert into content_deletion_card_failures values (1)`); + await db.execute(sql`create or replace function content_deletion_fail_card() returns trigger language plpgsql as $$ + begin + if exists (select 1 from content_deletion_card_failures where remaining > 0) then + raise exception 'card update failed'; + end if; + return new; + end $$`); + await db.execute(sql`create trigger content_deletion_fail_card before update on issue_thread_interactions + for each row execute function content_deletion_fail_card()`); + try { + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + await expect(service.processPending(new AbortController().signal)).rejects.toThrow(); + const [pending] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.connection.id)); + expect(pending!.status).toBe("pending"); + await db.execute(sql`delete from content_deletion_card_failures`); + await drain(service); + } finally { + await db.execute(sql`drop trigger if exists content_deletion_fail_card on issue_thread_interactions`); + } + const [card] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, seed.interaction.id)); + expect(JSON.stringify(card!.payload)).not.toContain(SECRET); + expect(JSON.stringify(card!.result)).not.toContain(SECRET); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "completed", counts: { approvals: 1, toolCalls: 1 } }); + }); + + it("keeps the job pending and the run log reference when the log backend is not recognised", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + await db.update(heartbeatRuns).set({ logStore: "tape_archive" }).where(eq(heartbeatRuns.id, seed.settledRun.id)); + await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + for (let i = 0; i < 5; i++) await service.processPending(new AbortController().signal).catch(() => undefined); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seed.settledRun.id)); + expect(run).toMatchObject({ logStore: "tape_archive", logRef: seed.settledRun.logRef }); + const state = await service.storedContent(seed.connection); + expect(state.deletion).toMatchObject({ status: "pending", retrying: true }); + expect(await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deleted"), + ))).toEqual([]); + }); + + it("yields after every step when the time budget is spent and finishes over later passes", async () => { + const runLogs = memoryRunLogs(); + const service = createConnectionContentDeletion(db, { runLogStore: () => runLogs.store }); + const seed = await seedCompany(runLogs); + const { deletion } = await service.request(seed.connection, { actorType: "user", actorId: "board-user" }); + const phases: string[] = []; + for (let i = 0; i < 40; i++) { + const job = await service.process(deletion.id, new AbortController().signal, 0); + if (!job || job.status === "completed") break; + phases.push(String((job.progress as { phase: string }).phase)); + } + expect(phases.length).toBeGreaterThan(5); + expect(new Set(phases)).toEqual(new Set(["events", "auditEvents", "appViews", "deliveredResults", "runs", "done"])); + expect((await service.storedContent(seed.connection)).deletion).toMatchObject({ status: "completed", counts: { runs: 2 } }); + }); + + describe("routes", () => { + function app(actor: Express.Request["actor"]) { + const server = express(); + server.use(express.json()); + server.use((req, _res, next) => { req.actor = actor; next(); }); + server.use("/api", toolAccessRoutes(db)); + server.use(errorHandler); + return server; + } + + it("lets a board user see counts and request deletion, and refuses an agent and another company", async () => { + const runLogs = memoryRunLogs(); + const seed = await seedCompany(runLogs); + const board = app({ type: "board", userId: "board-user", userName: "Board", userEmail: null, isInstanceAdmin: true, source: "local_implicit" } as never); + + const counts = await request(board).get(`/api/tool-connections/${seed.connection.id}/stored-content`).expect(200); + expect(counts.body.stored).toEqual({ toolCalls: 2, approvals: 1, appViews: 1, runs: 3, runsInProgress: 1 }); + + const requested = await request(board).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`).expect(202); + expect(requested.body.deletion).toMatchObject({ status: "pending" }); + const [activity] = await db.select().from(activityLog).where(and( + eq(activityLog.companyId, seed.company.id), + eq(activityLog.action, "tool_connection.content_deletion_requested"), + )); + expect(activity!.details).toMatchObject({ toolCalls: 2, runs: 3, runsInProgress: 1 }); + const status = await request(board).get(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`).expect(200); + expect(status.body.deletion).toMatchObject({ status: "pending" }); + + const outsider = app({ + type: "board", + userId: "outsider", + userName: "Outsider", + userEmail: null, + isInstanceAdmin: false, + source: "session", + companyIds: [randomUUID()], + memberships: [], + } as never); + const refused = await request(outsider).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`); + expect([403, 404]).toContain(refused.status); + + const agent = app({ type: "agent", agentId: seed.agent.id, companyId: seed.company.id, runId: null, source: "agent_key" } as never); + const agentRefused = await request(agent).post(`/api/tool-connections/${seed.connection.id}/stored-content/deletion`); + expect([401, 403]).toContain(agentRefused.status); + }); + + it("queues the deletion when a board user removes the connection with it", async () => { + const runLogs = memoryRunLogs(); + const seed = await seedCompany(runLogs); + const board = app({ type: "board", userId: "board-user", userName: "Board", userEmail: null, isInstanceAdmin: true, source: "local_implicit" } as never); + const removed = await request(board).delete(`/api/tool-connections/${seed.other.id}?deleteStoredContent=true`).expect(200); + expect(removed.body.contentDeletion).toMatchObject({ status: "pending" }); + const [pending] = await db.select().from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.connectionId, seed.other.id)); + expect(pending!.status).toBe("pending"); + }); + }); +}); diff --git a/server/src/index.ts b/server/src/index.ts index 4efa6af8612..3488284e080 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -18,6 +18,7 @@ import { registerMergeConfirmationWork } from "./services/merged-pr-confirmation import { registerNativeHarnessBackupExpirationWork } from "./services/native-runtime/native-harness-backup-expiration-work.js"; import { registerGitHubConnectionEventWork, registerGitHubConnectionContinuityWork } from "./services/github-connection-work.js"; import { registerProviderTraceRetentionWork } from "./services/provider-trace-retention-work.js"; +import { registerToolConnectionContentDeletionWork } from "./services/tool-connection-content-deletion-work.js"; import { registerRunEvidenceMigrationWork } from "./services/run-evidence-migration-work.js"; import { registerSkillVersionRetentionWork } from "./services/skill-version-retention-work.js"; import { registerDatabaseBackupWork } from "./services/database-backup-work.js"; @@ -1446,6 +1447,10 @@ async function startServerWithDatabaseTeardown( () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()); trackHeartbeatSchedulerWork(providerTraceRetentionWork.ready); server.on("close", providerTraceRetentionWork.stop); + const toolConnectionContentDeletionWork = registerToolConnectionContentDeletionWork(db, deliveryWork, + () => !heartbeatSchedulerStopped && !isWarmStandby()); + trackHeartbeatSchedulerWork(toolConnectionContentDeletionWork.ready); + server.on("close", toolConnectionContentDeletionWork.stop); const executionSchedule = executionControlSchedule(db); const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager }); const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled @@ -2276,6 +2281,7 @@ async function startServerWithDatabaseTeardown( await runEvidenceMigrationWork?.stop(); }); await preDrain.run("provider_trace_retention_stop", () => providerTraceRetentionWork.stop()); + await preDrain.run("tool_connection_content_deletion_stop", () => toolConnectionContentDeletionWork.stop()); await preDrain.run("skill_version_retention_dispose", async () => { await skillVersionRetentionWorker?.dispose(); }); diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 101c9bf9d38..1e10c07ea01 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -12574,6 +12574,35 @@ registerCurrentRoute({ path: "/api/tool-connections/{connectionId}", tags: ["tool-access"], summary: "Archive a tool connection", + query: z.object({ deleteStoredContent: z.enum(["true", "false"]).optional() }), +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "get", + path: "/api/tool-connections/{connectionId}/stored-content", + tags: ["tool-access"], + summary: "Count the content Paperclip stored from a connection's tool calls", +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "post", + path: "/api/tool-connections/{connectionId}/stored-content/deletion", + tags: ["tool-access"], + summary: "Delete the content Paperclip stored from a connection's tool calls", + responses: { 202: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound }, +}); + +registerCurrentRoute({ + "x-paperclip-transport": "rest", + "x-paperclip-connection-management": "connections:configure", + method: "get", + path: "/api/tool-connections/{connectionId}/stored-content/deletion", + tags: ["tool-access"], + summary: "Read the status of the latest stored-content deletion for a connection", }); const MACHINE_SIGN_IN_ERRORS = { diff --git a/server/src/routes/tool-access.ts b/server/src/routes/tool-access.ts index 146d1f61b4a..0faa1a9836d 100644 --- a/server/src/routes/tool-access.ts +++ b/server/src/routes/tool-access.ts @@ -71,6 +71,8 @@ import { accessService, logActivity, toolAccessPolicyService, toolAccessService, import { ToolGatewayHttpError, type ToolGatewayService } from "../services/tool-gateway.js"; import { authorizationService } from "../services/authorization.js"; import { mcpAppViewsService } from "../services/mcp-app-views.js"; +import { createConnectionContentDeletion } from "../services/tool-access/connections/content-deletion.js"; +import { getRunLogStore } from "../services/run-log-store.js"; import { listCompanyProviderToolIcons } from "../services/tool-access/catalog/icons.js"; import { mcpAppToolCallSchema } from "@paperclipai/shared"; import { RailwayError } from "../services/railway.js"; @@ -544,6 +546,7 @@ function connectorEnrollmentPrincipal(req: Request): string { const access = accessService(db); const managementGuard = connectionManagementGuard(db, "tool"); + const contentDeletion = createConnectionContentDeletion(db, { runLogStore: getRunLogStore }); async function assertBoardToolPermission(req: Request, companyId: string, permissionKey: PermissionKey) { if (req.actor.type === "agent") { @@ -2371,6 +2374,8 @@ function connectorEnrollmentPrincipal(req: Request): string { await assertToolConnectionConfigureAccess(req, existing); if (aiConnectionRouterPluginKey(existing)) throw badRequest("Remove this connection pool through its revision-checked pool settings."); const applicationBefore = await svc.getApplication(existing.applicationId); + const deleteStoredContent = req.query.deleteStoredContent === "true"; + if (deleteStoredContent) assertBoard(req); const { connection, removal } = await svc.archiveConnection( existing.id, existing.companyId, @@ -2402,7 +2407,54 @@ function connectorEnrollmentPrincipal(req: Request): string { details: { type: applicationAfter.type, name: applicationAfter.name, reason: "last_connection_removed" }, }); } - res.json({ ...connection, removal }); + const contentDeletion = deleteStoredContent + ? await requestStoredContentDeletion(req, connection) + : null; + res.json({ ...connection, removal, ...(contentDeletion ? { contentDeletion } : {}) }); + }); + + async function requestStoredContentDeletion(req: Request, connection: { id: string; companyId: string }) { + const actor = getActorInfo(req); + const { deletion, created } = await contentDeletion.request(connection, { + actorType: actor.actorType, + actorId: actor.actorId, + }); + if (created) { + const { stored } = await contentDeletion.storedContent(connection); + await logActivity(db, { + companyId: connection.companyId, + ...connectionRequestActor(req), + action: "tool_connection.content_deletion_requested", + entityType: "tool_connection", + entityId: connection.id, + details: { ...stored }, + }); + } + return deletion; + } + + router.get("/tool-connections/:connectionId/stored-content", managementGuard("connections:configure", "GET /tool-connections/:connectionId/stored-content", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.json(await contentDeletion.storedContent(existing)); + }); + + router.post("/tool-connections/:connectionId/stored-content/deletion", managementGuard("connections:configure", "POST /tool-connections/:connectionId/stored-content/deletion", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.status(202).json({ deletion: await requestStoredContentDeletion(req, existing) }); + }); + + router.get("/tool-connections/:connectionId/stored-content/deletion", managementGuard("connections:configure", "GET /tool-connections/:connectionId/stored-content/deletion", { personal: true }), async (req, res) => { + assertBoard(req); + const existing = await getAccessibleResource(req, res, svc.getConnection(req.params.connectionId as string), "Tool connection not found"); + if (!existing) return; + await assertToolConnectionConfigureAccess(req, existing); + res.json({ deletion: await contentDeletion.latestDeletion(existing) }); }); router.post("/tool-connections/:connectionId/health-check", managementGuard("connections:configure", "POST /tool-connections/:connectionId/health-check", { personal: true }), async (req, res) => { diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index e419a441e56..66e5c3821d7 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -79,6 +79,7 @@ export const DELIVERY_QUEUES = { decisionArchiveDelivery: "decision-archive-delivery", toolConnectionHealth: "tool-connection-health", + toolConnectionContentDeletion: "tool-connection-content-deletion", toolReviewProjection: "tool-review-projection", toolReviewInterruption: "tool-review-interruption", toolReviewRecovery: "tool-review-recovery", diff --git a/server/src/services/object-run-log-store.test.ts b/server/src/services/object-run-log-store.test.ts index cc817d72f02..8187fbec7ef 100644 --- a/server/src/services/object-run-log-store.test.ts +++ b/server/src/services/object-run-log-store.test.ts @@ -66,6 +66,18 @@ const event = (seq: number) => ({ }); describe("S3 run log store", () => { + it("removes every object of a finalized log and stays removable", async () => { + const m = memory(), store = createObjectRunLogStore({ provider: m.provider, keyPrefix: "logs", + assertWriterAuthority: async () => undefined }); + const handle = await store.begin(input); + await store.append(handle, { ...event(1), chunk: "tool result content" }); + await store.finalize(handle); + expect([...m.objects.keys()].some((key) => key.includes(handle.logRef))).toBe(true); + await store.remove!(handle); + await store.remove!(handle); + expect([...m.objects.keys()].filter((key) => key.includes(handle.logRef) && !key.endsWith("/head.json"))).toEqual([]); + await expect(store.read(handle)).rejects.toThrow("Run log not found"); + }); it("awaits a successor flush started by another drain waiter before finalizing", async () => { const m = memory(), store = createObjectRunLogStore({ provider: m.provider, keyPrefix: "logs", flushMs: 1000, segmentBytes: 1024, assertWriterAuthority: async () => undefined }); diff --git a/server/src/services/object-run-log-store.ts b/server/src/services/object-run-log-store.ts index 8ffee47f9ad..00b16d092f4 100644 --- a/server/src/services/object-run-log-store.ts +++ b/server/src/services/object-run-log-store.ts @@ -399,6 +399,10 @@ export function createObjectRunLogStore(options: ObjectRunLogStoreOptions): RunL nextOffset: nextOffset ?? offset + bytes.length, }; }, + async remove(handle, signal) { + if (handle.store !== "s3_segments" && handle.store !== "s3_archive") throw new Error("Invalid object run log backend"); + await journal(handle.logRef, "background").remove(signal); + }, async flushInflightMirrors() { await Promise.all([...writers.values()].map(flush)); }, diff --git a/server/src/services/run-log-store.test.ts b/server/src/services/run-log-store.test.ts index aee7e202198..cf3b8546f25 100644 --- a/server/src/services/run-log-store.test.ts +++ b/server/src/services/run-log-store.test.ts @@ -4,7 +4,8 @@ import path from "node:path"; import os from "node:os"; import { createHash } from "node:crypto"; import { Readable } from "node:stream"; -import { createDurableRunLogStore } from "./run-log-store.js"; +import { composeRunLogStore, createDurableRunLogStore } from "./run-log-store.js"; +import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js"; import type { StorageProvider } from "../storage/types.js"; // In-memory StorageProvider stand-in: durable, survives the "pod roll" (local @@ -68,6 +69,46 @@ afterEach(async () => { const begin = { companyId: "co1", agentId: "ag1", runId: "run1" }; describe("createDurableRunLogStore", () => { + it("removes a migrated run's original local log and mirror along with its object journal", async () => { + const { provider, objects } = createMemoryProvider(); + const local = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); + const external = objectLogFixture(); + const store = composeRunLogStore(local, external); + const original = await local.begin(begin); + await local.append(original, { stream: "stdout", chunk: "tool result content", ts: "t1" }); + await local.finalize(original); + const migrated = await external.begin(begin); + await external.append(migrated, { stream: "stdout", chunk: "tool result content", ts: "t1", seq: 1 }); + await external.finalize(migrated); + expect(objects.has(original.logRef)).toBe(true); + await store.remove!(migrated); + await store.remove!(migrated); + expect(objects.has(original.logRef)).toBe(false); + await expect(fs.stat(path.join(baseDir, original.logRef))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(external.read(migrated)).rejects.toThrow("Run log not found"); + await expect(store.remove!({ store: "s3_segments", logRef: "../escape" })).rejects.toThrow("Invalid run log reference"); + }); + + it("refuses to report an object log removed when object storage is not configured", async () => { + const local = createDurableRunLogStore({ basePath: baseDir }); + const store = composeRunLogStore(local, null); + await expect(store.remove!({ store: "s3_segments", logRef: "co1/ag1/run1" })).rejects.toThrow("Object run log storage is not configured"); + }); + + it("removes a finalized log from local disk and its object mirror", async () => { + const { provider, objects } = createMemoryProvider(); + const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); + const handle = await store.begin(begin); + await store.append(handle, { stream: "stdout", chunk: "tool result content", ts: "t1" }); + await store.finalize(handle); + expect(objects.has(handle.logRef)).toBe(true); + await store.remove!(handle); + await store.remove!(handle); + expect(objects.has(handle.logRef)).toBe(false); + await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(store.read(handle)).rejects.toThrow("Run log not found"); + }); + it.each([false, true])("reads a complete long final record from a compact tail (S3 fallback: %s)", async (remote) => { const { provider, objects, calls } = createMemoryProvider(); const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } }); diff --git a/server/src/services/run-log-store.ts b/server/src/services/run-log-store.ts index 2138026afc7..495805081ef 100644 --- a/server/src/services/run-log-store.ts +++ b/server/src/services/run-log-store.ts @@ -85,6 +85,7 @@ export interface RunLogStore { recordVerifiedSourcePrefix?(handle: RunLogHandle, sourceCursor: number): Promise; finalize(handle: RunLogHandle, options?: { computeDigest?: boolean }): Promise; read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise; + remove?(handle: RunLogHandle, signal?: AbortSignal): Promise; // Optional so existing fakes/fixtures keep compiling: uploads every dirty // in-flight mirror immediately (graceful-shutdown path). No-op when the // in-flight mirror is not enabled. @@ -494,6 +495,14 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru }; }, + async remove(handle, signal) { + if (handle.store !== "local_file") throw new Error("Invalid local run log backend"); + signal?.throwIfAborted(); + await retireInflightMirror(handle.logRef); + await fs.rm(resolveWithin(basePath, handle.logRef), { force: true }); + if (s3) await s3.provider.deleteObject({ objectKey: s3Key(handle.logRef), signal }); + }, + async flushInflightMirrors() { if (!s3 || inflightMirrorMs <= 0) return; const flushEntry = async (logRef: string, entry: InflightMirrorEntry) => { @@ -558,20 +567,13 @@ export function resolveRunLogS3(): DurableRunLogStoreOptions["s3"] { }; } -let cachedStore: RunLogStore | null = null; +export function runLogBaseRef(handle: RunLogHandle): string { + const base = handle.store === "local_file" ? handle.logRef.replace(/\.ndjson$/, "") : handle.logRef.replace(/\.archive$/, ""); + if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(base)) throw new Error("Invalid run log reference"); + return base; +} -export function getRunLogStore() { - if (cachedStore) return cachedStore; - const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs"); - const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() }); - const s3 = resolveRunEvidenceStorage(); - const external = s3 ? createObjectRunLogStore({ - provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs", - assertWriterAuthority: async input => { - if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority"); - await input.assertWriterAuthority(); - }, - }) : null; +export function composeRunLogStore(local: RunLogStore, external: RunLogStore | null): RunLogStore { const writer = external ?? local; function writable(handle: RunLogHandle) { if (handle.store === "s3_archive") throw new Error("Historical run log archives are read-only"); @@ -579,7 +581,7 @@ export function getRunLogStore() { if (!external) throw new Error("Object run log storage is not configured"); return external; } - cachedStore = { + return { begin: input => { if (input.previousStore === "s3_archive") throw new Error("Historical run log archives are read-only"); if (input.previousStore === "local_file") return local.begin(input); @@ -603,8 +605,36 @@ export function getRunLogStore() { } return local.read(handle, opts); }, + remove: async (handle, signal) => { + if (!local.remove) throw new Error("Run log removal is unavailable"); + const base = runLogBaseRef(handle); + if (external?.remove) { + await external.remove({ store: "s3_segments", logRef: base }, signal); + await external.remove({ store: "s3_archive", logRef: `${base}.archive` }, signal); + } else if (handle.store !== "local_file") { + throw new Error("Object run log storage is not configured"); + } + await local.remove({ store: "local_file", logRef: `${base}.ndjson` }, signal); + }, flushInflightMirrors: async () => { await Promise.all([local.flushInflightMirrors?.(), external?.flushInflightMirrors?.()]); }, }; +} + +let cachedStore: RunLogStore | null = null; + +export function getRunLogStore() { + if (cachedStore) return cachedStore; + const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs"); + const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() }); + const s3 = resolveRunEvidenceStorage(); + const external = s3 ? createObjectRunLogStore({ + provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs", + assertWriterAuthority: async input => { + if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority"); + await input.assertWriterAuthority(); + }, + }) : null; + cachedStore = composeRunLogStore(local, external); return cachedStore; } diff --git a/server/src/services/tool-access/connections/content-deletion.ts b/server/src/services/tool-access/connections/content-deletion.ts new file mode 100644 index 00000000000..063d2dbd0de --- /dev/null +++ b/server/src/services/tool-access/connections/content-deletion.ts @@ -0,0 +1,761 @@ +import { and, asc, desc, eq, gt, inArray, lte, sql, type SQL } from "drizzle-orm"; +import { + agentWakeupRequests, + heartbeatRunEvents, + heartbeatRuns, + issueThreadInteractions, + toolAccessAuditEvents, + toolActionRequests, + toolCallEvents, + toolConnectionContentDeletions, + toolInvocations, + toolMcpAppViews, + type Db, +} from "@paperclipai/db"; +import type { + ToolConnectionContentDeletion, + ToolConnectionContentDeletionCounts, + ToolConnectionStoredContent, +} from "@paperclipai/shared"; +import { isRunLogStoreType, type RunLogHandle, type RunLogStore } from "../../run-log-store.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../delivery-work-notifications.js"; +import { logActivity } from "../../activity-log.js"; + +const BATCH_SIZE = 200; +const EVENT_BATCH_SIZE = 1000; +const STEP_BUDGET_MS = 10_000; + +const SETTLED_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"]; +const UNSETTLED_INVOCATION_STATUSES = ["pending", "authorized", "awaiting_approval", "executing"]; +const UNSETTLED_ACTION_REQUEST_STATUSES = ["pending", "approved", "executing"]; +const UNSETTLED_WAKE_STATUSES = ["queued", "deferred_issue_execution", "claimed"]; +const RESULT_TEXT_KEYS = ["summary", "result", "message", "text", "output", "content", "body"]; +const STRUCTURAL_EVENT_TYPES = [ + "lifecycle", + "run.terminal", + "run.presentation.resolved", + "adapter.invoke", + "usage.reported", + "context.compacted", + "session.started", + "session.resumed", + "turn.started", + "turn.accepted", + "harness.ready", +]; + +type Phase = "invocations" | "events" | "auditEvents" | "appViews" | "deliveredResults" | "runs" | "done"; +const PHASES: Phase[] = ["invocations", "events", "auditEvents", "appViews", "deliveredResults", "runs", "done"]; +const RUN_BATCH_SIZE = 50; +const DELIVERY_BATCH_SIZE = 50; + +type RunInProgress = { id: string; eventAfterId: number; changed: boolean; stage?: "events" | "log" }; + +type Progress = { + phase: Phase; + cursor: { createdAt: string; id: string } | null; + runCursor: string | null; + currentRun: RunInProgress | null; + pendingRuns: string[]; + counts: ToolConnectionContentDeletionCounts; +}; + +type DeletionRow = typeof toolConnectionContentDeletions.$inferSelect; + +export type ContentDeletionActor = { actorType: "user" | "agent" | "system"; actorId: string | null }; + +class ProgressConflict extends Error {} + +export function emptyContentDeletionCounts(): ToolConnectionContentDeletionCounts { + return { toolCalls: 0, approvals: 0, auditEvents: 0, appViews: 0, runs: 0, runsInProgress: 0, itemsInProgress: 0 }; +} + +function readProgress(row: DeletionRow): Progress { + const raw = row.progress as Partial; + const counts = { ...emptyContentDeletionCounts(), ...(raw.counts ?? {}) }; + const phase = PHASES.includes(raw.phase as Phase) ? raw.phase as Phase : "invocations"; + return { phase, cursor: raw.cursor ?? null, runCursor: raw.runCursor ?? null, currentRun: raw.currentRun ?? null, pendingRuns: raw.pendingRuns ?? [], counts }; +} + +function nextPhase(phase: Phase): Phase { + return PHASES[Math.min(PHASES.indexOf(phase) + 1, PHASES.length - 1)]!; +} + +function listSql(values: string[]): SQL { + return sql.join(values.map((value) => sql`${value}`), sql`, `); +} + +export function settledRunSql(runId: SQL | typeof toolInvocations.runId): SQL { + return sql`exists (select 1 from ${heartbeatRuns} r where r.id = ${runId} + and r.status in (${listSql(SETTLED_RUN_STATUSES)}) + and r.finished_at is not null + and r.cost_accounting_pending = false + and not exists (select 1 from native_run_finalizations f where f.run_id = r.id + and f.phase not in ('committed', 'terminal_failure', 'applied')))`; +} + +export function runsWithStoredContentSql(scope: { companyId: string; connectionId: string; cutoff: Date }): SQL { + const calls = sql`${toolInvocations.companyId} = ${scope.companyId} and ${toolInvocations.connectionId} = ${scope.connectionId} + and ${toolInvocations.createdAt} <= ${scope.cutoff.toISOString()}::timestamptz`; + const resultText = sql.join(RESULT_TEXT_KEYS.map((key) => + sql.raw(`jsonb_path_exists(coalesce("heartbeat_runs"."result_json", '{}'::jsonb), 'lax $.**.${key} ? (@.type() == "string" && @ != "")')`)), sql` or `); + return sql`${heartbeatRuns.companyId} = ${scope.companyId} + and (${heartbeatRuns.id} in (select ${toolInvocations.runId} from ${toolInvocations} where ${calls} and ${toolInvocations.runId} is not null) + or ${heartbeatRuns.id} in (select ${toolCallEvents.runId} from ${toolCallEvents} where ${toolCallEvents.companyId} = ${scope.companyId} + and ${toolCallEvents.connectionId} = ${scope.connectionId} and ${toolCallEvents.createdAt} <= ${scope.cutoff.toISOString()}::timestamptz + and ${toolCallEvents.runId} is not null) + or (${heartbeatRuns.issueId} in (select ${toolActionRequests.issueId} from ${toolActionRequests} + join ${toolInvocations} on ${toolInvocations.id} = ${toolActionRequests.invocationId} where ${calls}) + and ${heartbeatRuns.contextSnapshot}->'toolActionRequestIds' ?| array(select ${toolActionRequests.id}::text from ${toolActionRequests} + join ${toolInvocations} on ${toolInvocations.id} = ${toolActionRequests.invocationId} where ${calls}))) + and (${heartbeatRuns.logRef} is not null or ${heartbeatRuns.logStore} is not null + or ${heartbeatRuns.stdoutExcerpt} is not null or ${heartbeatRuns.stderrExcerpt} is not null + or ${resultText} + or exists (select 1 from ${heartbeatRunEvents} where ${heartbeatRunEvents.companyId} = ${heartbeatRuns.companyId} + and ${heartbeatRunEvents.runId} = ${heartbeatRuns.id} and ${contentEventSql()}))`; +} + +function contentEventSql(): SQL { + return sql`${heartbeatRunEvents.eventType} not in (${listSql(STRUCTURAL_EVENT_TYPES)}) + and ${heartbeatRunEvents.eventType} not like 'native.%' + and not (${heartbeatRunEvents.eventType} = 'item.completed' and (${heartbeatRunEvents.payload} -> 'prpEvent' -> 'payload' ->> 'kind') = 'usage') + and (${heartbeatRunEvents.message} is not null or ${heartbeatRunEvents.payload} is not null)`; +} + +function clearedSummary(column: SQL, deletedAt: string): SQL { + return sql`case when ${column} is null then null + else ${column} || jsonb_build_object('summary', '', 'contentDeletedAt', ${deletedAt}::text) end`; +} + +export function withoutActionResults(value: unknown, actionRequestIds: ReadonlySet): { value: unknown; changed: boolean } { + let changed = false; + const walk = (node: unknown): unknown => { + if (Array.isArray(node)) return node.map(walk); + if (!node || typeof node !== "object") return node; + const record = node as Record; + const next: Record = {}; + for (const [key, entry] of Object.entries(record)) next[key] = walk(entry); + if (typeof record.actionRequestId === "string" && actionRequestIds.has(record.actionRequestId) + && typeof record.resultSummary === "string" && record.resultSummary !== "") { + next.resultSummary = ""; + changed = true; + } + return next; + }; + const result = walk(value); + return { value: changed ? result : value, changed }; +} + +export function withoutResultText(value: Record | null): { value: Record | null; changed: boolean } { + if (!value || typeof value !== "object") return { value, changed: false }; + let changed = false; + const walk = (node: unknown, topLevel: boolean): unknown => { + if (Array.isArray(node)) return node.map((entry) => walk(entry, false)); + if (!node || typeof node !== "object") return node; + const next: Record = {}; + for (const [key, entry] of Object.entries(node as Record)) { + if (RESULT_TEXT_KEYS.includes(key) && typeof entry === "string") { + if (entry !== "") changed = true; + if (!topLevel) next[key] = ""; + else if (entry === "") next[key] = entry; + continue; + } + next[key] = walk(entry, false); + } + return next; + }; + const next = walk(value, true) as Record; + return { value: changed ? next : value, changed }; +} + +function toDeletion(row: DeletionRow): ToolConnectionContentDeletion { + return { + id: row.id, + status: row.status, + requestedAt: row.requestedAt.toISOString(), + completedAt: row.completedAt?.toISOString() ?? null, + retrying: row.status === "pending" && row.lastError !== null, + counts: readProgress(row).counts, + }; +} + +export function createConnectionContentDeletion(db: Db, deps: { runLogStore: () => RunLogStore }) { + async function latestDeletion(connection: { id: string; companyId: string }) { + const [latest] = await db + .select() + .from(toolConnectionContentDeletions) + .where(and( + eq(toolConnectionContentDeletions.companyId, connection.companyId), + eq(toolConnectionContentDeletions.connectionId, connection.id), + )) + .orderBy(desc(toolConnectionContentDeletions.createdAt)) + .limit(1); + return latest ? toDeletion(latest) : null; + } + + async function storedContent(connection: { id: string; companyId: string }): Promise { + const scope = and(eq(toolInvocations.companyId, connection.companyId), eq(toolInvocations.connectionId, connection.id)); + const [calls] = await db + .select({ toolCalls: sql`count(*)::int` }) + .from(toolInvocations) + .where(and( + scope, + sql`coalesce(${toolInvocations.resultSummary}->>'contentDeletedAt', ${toolInvocations.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolInvocations.resultSummary} is not null or ${toolInvocations.argumentsSummary} is not null)`, + )); + const [approvals] = await db + .select({ count: sql`count(*)::int` }) + .from(toolActionRequests) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and( + scope, + eq(toolActionRequests.companyId, connection.companyId), + sql`(${toolActionRequests.canonicalArgumentsSummary}->>'contentDeletedAt' is null or ${toolActionRequests.previewMarkdown} is not null)`, + )); + const [views] = await db + .select({ count: sql`count(*)::int` }) + .from(toolMcpAppViews) + .where(and(eq(toolMcpAppViews.connectionId, connection.id), eq(toolMcpAppViews.companyId, connection.companyId))); + const [runs] = await db + .select({ + runs: sql`count(*)::int`, + runsInProgress: sql`count(*) filter (where not ${settledRunSql(sql`${heartbeatRuns.id}`)})::int`, + }) + .from(heartbeatRuns) + .where(runsWithStoredContentSql({ companyId: connection.companyId, connectionId: connection.id, cutoff: new Date() })); + return { + stored: { + toolCalls: calls?.toolCalls ?? 0, + approvals: approvals?.count ?? 0, + appViews: views?.count ?? 0, + runs: runs?.runs ?? 0, + runsInProgress: runs?.runsInProgress ?? 0, + }, + deletion: await latestDeletion(connection), + }; + } + + async function request(connection: { id: string; companyId: string }, actor: ContentDeletionActor) { + return db.transaction(async (tx) => { + const pendingFor = () => tx + .select() + .from(toolConnectionContentDeletions) + .where(and( + eq(toolConnectionContentDeletions.companyId, connection.companyId), + eq(toolConnectionContentDeletions.connectionId, connection.id), + eq(toolConnectionContentDeletions.status, "pending"), + )) + .limit(1); + const [existing] = await pendingFor(); + if (existing) return { deletion: toDeletion(existing), created: false }; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.toolConnectionContentDeletion); + const now = new Date(); + const [created] = await tx + .insert(toolConnectionContentDeletions) + .values({ + requestedAt: now, + createdAt: now, + updatedAt: now, + companyId: connection.companyId, + connectionId: connection.id, + requestedByActorType: actor.actorType, + requestedByActorId: actor.actorId, + progress: { phase: "invocations", cursor: null, runCursor: null, currentRun: null, pendingRuns: [], counts: emptyContentDeletionCounts() }, + }) + .onConflictDoNothing() + .returning(); + if (created) return { deletion: toDeletion(created), created: true }; + const [raced] = await pendingFor(); + return { deletion: toDeletion(raced!), created: false }; + }); + } + + async function settledRunIds(tx: Db, runIds: string[]) { + if (runIds.length === 0) return new Set(); + const rows = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where(and(inArray(heartbeatRuns.id, runIds), settledRunSql(sql`${heartbeatRuns.id}`))); + return new Set(rows.map((row) => row.id)); + } + + async function clearDeliveredResults(tx: Db, job: DeletionRow, progress: Progress, requests: Array<{ id: string; issueId: string | null }>) { + const issueIds = [...new Set(requests.flatMap((row) => row.issueId ? [row.issueId] : []))]; + if (issueIds.length === 0) return []; + const ids = new Set(requests.map((row) => row.id)); + const idList = sql`array[${listSql([...ids])}]::text[]`; + const runs = await tx + .select({ id: heartbeatRuns.id, contextSnapshot: heartbeatRuns.contextSnapshot }) + .from(heartbeatRuns) + .where(and( + eq(heartbeatRuns.companyId, job.companyId), + inArray(heartbeatRuns.issueId, issueIds), + sql`${heartbeatRuns.contextSnapshot}->'toolActionRequestIds' ?| ${idList}`, + )) + .for("update"); + const settled = await settledRunIds(tx, runs.map((row) => row.id)); + for (const run of runs) { + if (!settled.has(run.id)) { + progress.counts.itemsInProgress += 1; + continue; + } + const cleaned = withoutActionResults(run.contextSnapshot, ids); + if (cleaned.changed) { + await tx.update(heartbeatRuns).set({ contextSnapshot: cleaned.value as Record }).where(eq(heartbeatRuns.id, run.id)); + } + } + const wakes = await tx + .select({ id: agentWakeupRequests.id, status: agentWakeupRequests.status, payload: agentWakeupRequests.payload }) + .from(agentWakeupRequests) + .where(and( + eq(agentWakeupRequests.companyId, job.companyId), + inArray(sql`${agentWakeupRequests.payload}->>'issueId'`, issueIds), + sql`(${agentWakeupRequests.payload}->'toolActionRequestIds' ?| ${idList} + or ${agentWakeupRequests.payload}->'_paperclipWakeContext'->'toolActionRequestIds' ?| ${idList})`, + )) + .for("update"); + for (const wake of wakes) { + const cleaned = withoutActionResults(wake.payload, ids); + if (!cleaned.changed) continue; + if (UNSETTLED_WAKE_STATUSES.includes(wake.status)) { + progress.counts.itemsInProgress += 1; + continue; + } + await tx.update(agentWakeupRequests).set({ payload: cleaned.value as Record }).where(eq(agentWakeupRequests.id, wake.id)); + } + return runs.filter((row) => settled.has(row.id)).map((row) => row.id); + } + + async function invocationsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolInvocations.id, createdAt: toolInvocations.createdAt, runId: toolInvocations.runId, status: toolInvocations.status }) + .from(toolInvocations) + .where(and( + eq(toolInvocations.companyId, job.companyId), + lte(toolInvocations.createdAt, job.requestedAt), + cursor ? sql`(${toolInvocations.createdAt}, ${toolInvocations.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolInvocations.connectionId, job.connectionId), + )) + .orderBy(asc(toolInvocations.createdAt), asc(toolInvocations.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const eligible = page.filter((row) => !UNSETTLED_INVOCATION_STATUSES.includes(row.status) && (!row.runId || settledRuns.has(row.runId))); + progress.counts.itemsInProgress += page.length - eligible.length; + const ids = eligible.map((row) => row.id); + if (ids.length > 0) { + const cleared = await tx + .update(toolInvocations) + .set({ + argumentsSummary: clearedSummary(sql`${toolInvocations.argumentsSummary}`, deletedAt) as never, + resultSummary: clearedSummary(sql`${toolInvocations.resultSummary}`, deletedAt) as never, + progress: null, + }) + .where(and( + eq(toolInvocations.companyId, job.companyId), + inArray(toolInvocations.id, ids), + sql`coalesce(${toolInvocations.resultSummary}->>'contentDeletedAt', ${toolInvocations.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolInvocations.resultSummary} is not null or ${toolInvocations.argumentsSummary} is not null or ${toolInvocations.progress} is not null)`, + )) + .returning({ id: toolInvocations.id }); + progress.counts.toolCalls += cleared.length; + const requests = await tx + .select({ id: toolActionRequests.id, interactionId: toolActionRequests.interactionId, issueId: toolActionRequests.issueId, status: toolActionRequests.status }) + .from(toolActionRequests) + .where(and(eq(toolActionRequests.companyId, job.companyId), inArray(toolActionRequests.invocationId, ids))); + const settledRequests = requests.filter((row) => !UNSETTLED_ACTION_REQUEST_STATUSES.includes(row.status)); + progress.counts.itemsInProgress += requests.length - settledRequests.length; + if (settledRequests.length > 0) { + const requestIds = settledRequests.map((row) => row.id); + const clearedRequests = await tx + .update(toolActionRequests) + .set({ + canonicalArgumentsSummary: clearedSummary(sql`${toolActionRequests.canonicalArgumentsSummary}`, deletedAt) as never, + previewMarkdown: null, + signedArguments: null, + }) + .where(and( + eq(toolActionRequests.companyId, job.companyId), + inArray(toolActionRequests.id, requestIds), + sql`(${toolActionRequests.canonicalArgumentsSummary}->>'contentDeletedAt' is null or ${toolActionRequests.previewMarkdown} is not null or ${toolActionRequests.signedArguments} is not null)`, + )) + .returning({ id: toolActionRequests.id }); + progress.counts.approvals += clearedRequests.length; + const interactionIds = settledRequests.flatMap((row) => row.interactionId ? [row.interactionId] : []); + if (interactionIds.length > 0) { + await tx + .update(issueThreadInteractions) + .set({ + payload: sql`case when jsonb_typeof(${issueThreadInteractions.payload}->'toolAction') = 'object' + then jsonb_set(jsonb_set(${issueThreadInteractions.payload} - 'detailsMarkdown', '{toolAction,previewMarkdown}', 'null'::jsonb), '{toolAction,argumentsSummaryJson}', 'null'::jsonb) + else ${issueThreadInteractions.payload} - 'detailsMarkdown' end` as never, + result: sql`case when jsonb_typeof(${issueThreadInteractions.result}->'toolAction') = 'object' + then jsonb_set(${issueThreadInteractions.result}, '{toolAction,resultSummary}', 'null'::jsonb) + else ${issueThreadInteractions.result} end` as never, + }) + .where(and( + eq(issueThreadInteractions.companyId, job.companyId), + inArray(issueThreadInteractions.id, interactionIds), + sql`(${issueThreadInteractions.payload} ? 'detailsMarkdown' + or ${issueThreadInteractions.payload}->'toolAction'->>'previewMarkdown' is not null + or ${issueThreadInteractions.payload}->'toolAction'->>'argumentsSummaryJson' is not null + or ${issueThreadInteractions.result}->'toolAction'->>'resultSummary' is not null)`, + )); + } + } + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function eventsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolCallEvents.id, createdAt: toolCallEvents.createdAt, runId: toolCallEvents.runId }) + .from(toolCallEvents) + .where(and( + eq(toolCallEvents.companyId, job.companyId), + lte(toolCallEvents.createdAt, job.requestedAt), + cursor ? sql`(${toolCallEvents.createdAt}, ${toolCallEvents.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolCallEvents.connectionId, job.connectionId), + )) + .orderBy(asc(toolCallEvents.createdAt), asc(toolCallEvents.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const ids = page.filter((row) => !row.runId || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const cleared = await tx + .update(toolCallEvents) + .set({ + argumentsSummary: clearedSummary(sql`${toolCallEvents.argumentsSummary}`, deletedAt) as never, + requestSummary: clearedSummary(sql`${toolCallEvents.requestSummary}`, deletedAt) as never, + resultSummary: clearedSummary(sql`${toolCallEvents.resultSummary}`, deletedAt) as never, + }) + .where(and( + eq(toolCallEvents.companyId, job.companyId), + inArray(toolCallEvents.id, ids), + sql`coalesce(${toolCallEvents.resultSummary}->>'contentDeletedAt', ${toolCallEvents.requestSummary}->>'contentDeletedAt', ${toolCallEvents.argumentsSummary}->>'contentDeletedAt') is null`, + sql`(${toolCallEvents.resultSummary} is not null or ${toolCallEvents.requestSummary} is not null or ${toolCallEvents.argumentsSummary} is not null)`, + )) + .returning({ id: toolCallEvents.id }); + progress.counts.auditEvents += cleared.length; + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function auditEventsStep(tx: Db, job: DeletionRow, progress: Progress, deletedAt: string) { + const cursor = progress.cursor; + const page = await tx + .select({ id: toolAccessAuditEvents.id, createdAt: toolAccessAuditEvents.createdAt, runId: sql`${toolAccessAuditEvents.details}->>'runId'` }) + .from(toolAccessAuditEvents) + .where(and( + eq(toolAccessAuditEvents.companyId, job.companyId), + lte(toolAccessAuditEvents.createdAt, job.requestedAt), + cursor ? sql`(${toolAccessAuditEvents.createdAt}, ${toolAccessAuditEvents.id}) > (${cursor.createdAt}::timestamptz, ${cursor.id}::uuid)` : undefined, + eq(toolAccessAuditEvents.connectionId, job.connectionId), + )) + .orderBy(asc(toolAccessAuditEvents.createdAt), asc(toolAccessAuditEvents.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + const runIds = [...new Set(page.flatMap((row) => row.runId && uuid.test(row.runId) ? [row.runId] : []))]; + const settledRuns = await settledRunIds(tx, runIds); + const ids = page.filter((row) => !row.runId || !uuid.test(row.runId) || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const cleared = await tx + .update(toolAccessAuditEvents) + .set({ + details: sql`jsonb_set(${toolAccessAuditEvents.details}, '{argumentsSummary}', + ${toolAccessAuditEvents.details}->'argumentsSummary' || jsonb_build_object('summary', '', 'contentDeletedAt', ${deletedAt}::text))` as never, + }) + .where(and( + eq(toolAccessAuditEvents.companyId, job.companyId), + inArray(toolAccessAuditEvents.id, ids), + sql`jsonb_typeof(${toolAccessAuditEvents.details}->'argumentsSummary') = 'object'`, + sql`${toolAccessAuditEvents.details}->'argumentsSummary'->>'contentDeletedAt' is null`, + )) + .returning({ id: toolAccessAuditEvents.id }); + progress.counts.auditEvents += cleared.length; + } + const last = page[page.length - 1]!; + progress.cursor = { createdAt: last.createdAt.toISOString(), id: last.id }; + return page.length < BATCH_SIZE; + } + + async function appViewsStep(tx: Db, job: DeletionRow, progress: Progress) { + const page = await tx + .select({ id: toolMcpAppViews.id, runId: toolMcpAppViews.runId }) + .from(toolMcpAppViews) + .where(and( + eq(toolMcpAppViews.connectionId, job.connectionId), + eq(toolMcpAppViews.companyId, job.companyId), + lte(toolMcpAppViews.createdAt, job.requestedAt), + progress.runCursor ? gt(toolMcpAppViews.id, progress.runCursor) : undefined, + )) + .orderBy(asc(toolMcpAppViews.id)) + .limit(BATCH_SIZE); + if (page.length === 0) return true; + const settledRuns = await settledRunIds(tx, [...new Set(page.flatMap((row) => row.runId ? [row.runId] : []))]); + const ids = page.filter((row) => !row.runId || settledRuns.has(row.runId)).map((row) => row.id); + progress.counts.itemsInProgress += page.length - ids.length; + if (ids.length > 0) { + const removed = await tx + .delete(toolMcpAppViews) + .where(and(eq(toolMcpAppViews.companyId, job.companyId), inArray(toolMcpAppViews.id, ids))) + .returning({ id: toolMcpAppViews.id }); + progress.counts.appViews += removed.length; + } + progress.runCursor = page[page.length - 1]!.id; + return page.length < BATCH_SIZE; + } + + async function runEventPage(tx: Db, companyId: string, run: RunInProgress) { + const batch = await tx + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where(and( + eq(heartbeatRunEvents.companyId, companyId), + eq(heartbeatRunEvents.runId, run.id), + gt(heartbeatRunEvents.id, run.eventAfterId), + )) + .orderBy(asc(heartbeatRunEvents.id)) + .limit(EVENT_BATCH_SIZE); + if (batch.length === 0) return true; + const ids = batch.map((row) => row.id); + const cleared = await tx + .update(heartbeatRunEvents) + .set({ message: null, payload: null as never }) + .where(and( + inArray(heartbeatRunEvents.id, ids), + contentEventSql(), + )) + .returning({ id: heartbeatRunEvents.id }); + if (cleared.length > 0) run.changed = true; + run.eventAfterId = ids[ids.length - 1]!; + return batch.length < EVENT_BATCH_SIZE; + } + + async function removeRunLog(companyId: string, runId: string, signal: AbortSignal) { + let removedAny = false; + for (let attempt = 0; attempt < 3; attempt++) { + const [run] = await db + .select({ logStore: heartbeatRuns.logStore, logRef: heartbeatRuns.logRef }) + .from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, companyId))); + if (!run?.logRef && !run?.logStore) return removedAny; + if (!run.logRef || !isRunLogStoreType(run.logStore)) throw new Error("run_log_store_unrecognised"); + const handle: RunLogHandle = { store: run.logStore, logRef: run.logRef }; + const store = deps.runLogStore(); + if (!store.remove) throw new Error("run_log_removal_unavailable"); + await store.remove(handle, signal); + const updated = await db + .update(heartbeatRuns) + .set({ logStore: null, logRef: null }) + .where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.logStore, run.logStore), eq(heartbeatRuns.logRef, run.logRef))) + .returning({ id: heartbeatRuns.id }); + removedAny = true; + if (updated.length > 0) return true; + } + throw new Error("run_log_reference_changed"); + } + + async function finishRun(tx: Db, companyId: string, run: RunInProgress) { + const [row] = await tx + .select({ stdoutExcerpt: heartbeatRuns.stdoutExcerpt, stderrExcerpt: heartbeatRuns.stderrExcerpt, resultJson: heartbeatRuns.resultJson }) + .from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, companyId))) + .for("update"); + if (!row) return; + const result = withoutResultText(row.resultJson); + if (row.stdoutExcerpt === null && row.stderrExcerpt === null && !result.changed) return; + await tx + .update(heartbeatRuns) + .set({ stdoutExcerpt: null, stderrExcerpt: null, resultJson: result.value as Record | null }) + .where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, companyId))); + run.changed = true; + } + + async function deliveredResultsStep(tx: Db, job: DeletionRow, progress: Progress) { + const requests = await tx + .select({ id: toolActionRequests.id, issueId: toolActionRequests.issueId }) + .from(toolActionRequests) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and( + eq(toolActionRequests.companyId, job.companyId), + eq(toolInvocations.companyId, job.companyId), + eq(toolInvocations.connectionId, job.connectionId), + lte(toolInvocations.createdAt, job.requestedAt), + progress.runCursor ? gt(toolActionRequests.id, progress.runCursor) : undefined, + )) + .orderBy(asc(toolActionRequests.id)) + .limit(DELIVERY_BATCH_SIZE); + if (requests.length === 0) return true; + progress.runCursor = requests[requests.length - 1]!.id; + await clearDeliveredResults(tx, job, progress, requests); + return requests.length < DELIVERY_BATCH_SIZE; + } + + async function runsStep(tx: Db, job: DeletionRow, progress: Progress) { + if (!progress.currentRun) { + let runId = progress.pendingRuns.shift() ?? null; + if (!runId) { + const rows = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where(and( + progress.runCursor ? gt(heartbeatRuns.id, progress.runCursor) : undefined, + runsWithStoredContentSql({ companyId: job.companyId, connectionId: job.connectionId, cutoff: job.requestedAt }), + )) + .orderBy(asc(heartbeatRuns.id)) + .limit(RUN_BATCH_SIZE); + if (rows.length === 0) return true; + progress.pendingRuns = rows.map((row) => row.id); + progress.runCursor = rows[rows.length - 1]!.id; + return false; + } + if (!(await settledRunIds(tx, [runId])).has(runId)) { + progress.counts.runsInProgress += 1; + return false; + } + progress.currentRun = { id: runId, eventAfterId: 0, changed: false, stage: "events" }; + return false; + } + const run = progress.currentRun; + if (run.stage === "log") { + if (run.changed) progress.counts.runs += 1; + progress.currentRun = null; + return false; + } + if (!(await runEventPage(tx, job.companyId, run))) return false; + await finishRun(tx, job.companyId, run); + run.stage = "log"; + return false; + } + + async function step(job: DeletionRow, signal: AbortSignal) { + const progress = readProgress(job); + const deletedAt = job.requestedAt.toISOString(); + if (progress.currentRun?.stage === "log" && await removeRunLog(job.companyId, progress.currentRun.id, signal)) { + progress.currentRun.changed = true; + } + return db.transaction(async (rawTx) => { + const tx = rawTx as unknown as Db; + const done = progress.phase === "invocations" ? await invocationsStep(tx, job, progress, deletedAt) + : progress.phase === "events" ? await eventsStep(tx, job, progress, deletedAt) + : progress.phase === "auditEvents" ? await auditEventsStep(tx, job, progress, deletedAt) + : progress.phase === "appViews" ? await appViewsStep(tx, job, progress) + : progress.phase === "deliveredResults" ? await deliveredResultsStep(tx, job, progress) + : await runsStep(tx, job, progress); + if (done) { + progress.phase = nextPhase(progress.phase); + progress.cursor = null; + progress.runCursor = null; + progress.currentRun = null; + progress.pendingRuns = []; + } + const [saved] = await tx + .update(toolConnectionContentDeletions) + .set({ progress: progress as unknown as Record, updatedAt: new Date() }) + .where(and( + eq(toolConnectionContentDeletions.id, job.id), + eq(toolConnectionContentDeletions.status, "pending"), + eq(toolConnectionContentDeletions.updatedAt, job.updatedAt), + )) + .returning(); + if (!saved) throw new ProgressConflict(); + return saved; + }); + } + + async function complete(job: DeletionRow) { + const progress = readProgress(job); + const now = new Date(); + return db.transaction(async (tx) => { + const [done] = await tx + .update(toolConnectionContentDeletions) + .set({ status: "completed", completedAt: now, updatedAt: now, lastError: null }) + .where(and( + eq(toolConnectionContentDeletions.id, job.id), + eq(toolConnectionContentDeletions.status, "pending"), + eq(toolConnectionContentDeletions.updatedAt, job.updatedAt), + sql`${toolConnectionContentDeletions.progress}->>'phase' = 'done'`, + )) + .returning(); + if (!done) return null; + await logActivity(tx as unknown as Db, { + companyId: job.companyId, + actorType: job.requestedByActorType === "agent" ? "agent" : job.requestedByActorType === "user" ? "user" : "system", + actorId: job.requestedByActorId ?? "tool-connection-content-deletion", + action: "tool_connection.content_deleted", + entityType: "tool_connection", + entityId: job.connectionId, + details: { ...progress.counts }, + }); + return done; + }); + } + + async function process(jobId: string, signal: AbortSignal, budgetMs = STEP_BUDGET_MS) { + const startedAt = Date.now(); + let [job] = await db.select().from(toolConnectionContentDeletions).where(eq(toolConnectionContentDeletions.id, jobId)); + if (!job || job.status !== "pending") return job ?? null; + try { + while (!signal.aborted) { + if (readProgress(job).phase === "done") return await complete(job); + job = await step(job, signal); + if (Date.now() - startedAt >= budgetMs) return job; + } + return job; + } catch (error) { + if (error instanceof ProgressConflict) return null; + await db + .update(toolConnectionContentDeletions) + .set({ + attempts: sql`${toolConnectionContentDeletions.attempts} + 1`, + lastError: error instanceof Error ? error.message.slice(0, 500) : "content deletion failed", + updatedAt: new Date(), + }) + .where(and(eq(toolConnectionContentDeletions.id, job.id), eq(toolConnectionContentDeletions.status, "pending"))); + throw error; + } + } + + async function processPending(signal: AbortSignal) { + const rows = await db + .select({ id: toolConnectionContentDeletions.id }) + .from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.status, "pending")) + .orderBy(asc(toolConnectionContentDeletions.updatedAt)) + .limit(10); + let failure: unknown = null; + for (const { id } of rows) { + if (signal.aborted) break; + try { + await process(id, signal); + } catch (error) { + failure ??= error; + } + } + if (failure) throw failure; + } + + async function nextDeadline() { + const [row] = await db + .select({ id: toolConnectionContentDeletions.id }) + .from(toolConnectionContentDeletions) + .where(eq(toolConnectionContentDeletions.status, "pending")) + .limit(1); + return row ? Date.now() : null; + } + + return { storedContent, latestDeletion, request, process, processPending, nextDeadline }; +} + +export type ConnectionContentDeletionService = ReturnType; diff --git a/server/src/services/tool-connection-content-deletion-work.ts b/server/src/services/tool-connection-content-deletion-work.ts new file mode 100644 index 00000000000..0be2a0d4f0f --- /dev/null +++ b/server/src/services/tool-connection-content-deletion-work.ts @@ -0,0 +1,40 @@ +import type { Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerIdleLocalWorkProbe } from "./idle-local-work.js"; +import { getRunLogStore } from "./run-log-store.js"; +import { createConnectionContentDeletion } from "./tool-access/connections/content-deletion.js"; + +export function registerToolConnectionContentDeletionWork( + db: Db, + coordinator: ReturnType, + canRun: () => boolean, +) { + const deletions = createConnectionContentDeletion(db, { runLogStore: getRunLogStore }); + let deadline: number | null | undefined; + const unregister = registerIdleLocalWorkProbe(() => { + if (deadline === undefined) return "unknown"; + return deadline === null ? "none" : "present"; + }); + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.toolConnectionContentDeletion, { + canRun, + retryMs: 30_000, + continuationMs: 250, + onSettled: () => { + deadline = undefined; + }, + run: (signal) => signal.aborted ? Promise.resolve() : deletions.processPending(signal), + next: async () => { + deadline = await deletions.nextDeadline(); + return deadline; + }, + }); + return { + ...worker, + stop: async () => { + unregister(); + await worker.stop(); + }, + }; +} diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index aa52f0ca8b1..09b7a1d2994 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -268,7 +268,7 @@ "packages/shared/src/home-paths.ts": "fa4f1c440653b76b3dbb7a51026d557287deb45d", "packages/shared/src/humanize-connection.test.ts": "ba9e90a44c63452b0bf9f733adb90241921e8eca", "packages/shared/src/humanize-connection.ts": "fbd414476a0ef7106a7c80d8572208781b091699", - "packages/shared/src/index.ts": "825867d65bbb0783a2294e1838231e68669a8fc3", + "packages/shared/src/index.ts": "6a7bc6095752c6e6f7602fdccd889cdb8bafeb57", "packages/shared/src/issue-activity-identity.test.ts": "3dc294640f54c2be67943e3e0149cf4eff2c95ca", "packages/shared/src/issue-activity-identity.ts": "1b0f07ec3fe0506aaaf80a4fa9893f3baf8f0075", "packages/shared/src/issue-attribution.test.ts": "84563685e7f3c48029243cf1cc24a3a09ee3cde9", @@ -408,7 +408,7 @@ "packages/shared/src/types/inbox-agent-policy.ts": "cb9d41d9681e80b834889268ad96015a39df92f6", "packages/shared/src/types/inbox-dismissal.ts": "58fb7013847396ac9971eb5799dd0bf0486e0371", "packages/shared/src/types/inbox-mine.ts": "1d302b531d34842189704abd2f75f8743d09490d", - "packages/shared/src/types/index.ts": "8abf1f4ed91e67f8e620a1a0b127799c793554ff", + "packages/shared/src/types/index.ts": "9c93d353993d6073389ae6dad752c7b8845377e1", "packages/shared/src/types/instance.ts": "ddabd7cc0d885b5fdf45b33a7937b7b704f8fb7d", "packages/shared/src/types/issue-tree-control.ts": "75489fffd491b8350b05caf57d7796419797d9df", "packages/shared/src/types/issue.ts": "7183a5a11238c297e43917dc66897cbe73703835", @@ -431,7 +431,7 @@ "packages/shared/src/types/tailscale.ts": "d02a917945d7779009a2dea4c4b535e9478e0680", "packages/shared/src/types/task-workspace.ts": "a23afa13d2cb58dc2eb72692ed28c16ed6fb84fb", "packages/shared/src/types/teams-catalog.ts": "632fe75f8e5a4cfc455843c2a1a2daff28a022be", - "packages/shared/src/types/tool-access.ts": "1d2c29fb7c0c22ae30b53ff18fe44545c18b4835", + "packages/shared/src/types/tool-access.ts": "32b0ad72fd977c9fb5dbf31321f55c27cad92d9f", "packages/shared/src/types/user-profile.ts": "1ac26e033a79fab21dc267fd983959814c8b680f", "packages/shared/src/types/voice-sessions.ts": "95e3b62cdd51f62290e68536728ddd7297bc0459", "packages/shared/src/types/work-product.ts": "26f9129f606a207ce7a218ccef40f96747283086", diff --git a/ui/src/api/tools.ts b/ui/src/api/tools.ts index 786afb8929b..bfa359b1fc4 100644 --- a/ui/src/api/tools.ts +++ b/ui/src/api/tools.ts @@ -9,6 +9,8 @@ import type { ToolConnectionInstall, ToolConnectionInstallSnapshot, ToolConnectionRemovalSummary, + ToolConnectionContentDeletion, + ToolConnectionStoredContent, ConnectToolAppResult, ConnectToolApp, FinishToolAppResult, @@ -429,10 +431,16 @@ export const toolsApi = { api.post(`/tool-connections/${connectionId}/railway/ssh`, input), // Removal is a credential-revoking teardown (PAP-17119), so the response // carries the cleanup receipt alongside the archived connection. - archiveConnection: (connectionId: string) => - api.delete( - `/tool-connections/${connectionId}`, + archiveConnection: (connectionId: string, options: { deleteStoredContent?: boolean } = {}) => + api.delete( + `/tool-connections/${connectionId}${options.deleteStoredContent ? "?deleteStoredContent=true" : ""}`, ), + getStoredContent: (connectionId: string) => + api.get(`/tool-connections/${connectionId}/stored-content`), + deleteStoredContent: (connectionId: string) => + api.post<{ deletion: ToolConnectionContentDeletion }>(`/tool-connections/${connectionId}/stored-content/deletion`, {}), + getStoredContentDeletion: (connectionId: string) => + api.get<{ deletion: ToolConnectionContentDeletion | null }>(`/tool-connections/${connectionId}/stored-content/deletion`), checkConnectionHealth: (connectionId: string) => api.post(`/tool-connections/${connectionId}/health-check`, {}), reconnectConnection: (connectionId: string, credentialValues: Record) => diff --git a/ui/src/lib/queryKeys.ts b/ui/src/lib/queryKeys.ts index 1bae37d0722..e12ab0f09e2 100644 --- a/ui/src/lib/queryKeys.ts +++ b/ui/src/lib/queryKeys.ts @@ -72,6 +72,10 @@ export const queryKeys = { ["tools", "connection", connectionId, "installs"] as const, connectionGrants: (connectionId: string) => ["tools", "connection", connectionId, "grants"] as const, + connectionStoredContent: (connectionId: string) => + ["tools", "connection", connectionId, "stored-content"] as const, + connectionStoredContentDeletion: (connectionId: string) => + ["tools", "connection", connectionId, "stored-content", "deletion"] as const, catalog: (connectionId: string) => ["tools", "connection", connectionId, "catalog"] as const, connectionActivity: (connectionId: string) => ["tools", "connection", connectionId, "activity"] as const, diff --git a/ui/src/pages/apps/AppDetail.tsx b/ui/src/pages/apps/AppDetail.tsx index 993d67873ba..4b047c38cee 100644 --- a/ui/src/pages/apps/AppDetail.tsx +++ b/ui/src/pages/apps/AppDetail.tsx @@ -67,6 +67,7 @@ import { actionPermissionMutation } from "./app-detail/action-permissions"; import { RailwayAccessPanel } from "./app-detail/RailwayAccessPanel"; import { ReviewPanel } from "./app-detail/ReviewPanel"; import { AccessRetryBanner } from "./app-detail/AccessRetryBanner"; +import { StoredContentDeletionRow } from "./app-detail/StoredContentDeletion"; import { ReconnectCard, DangerZone, @@ -738,6 +739,11 @@ function StandardAppDetail({ renderActions, renderAgentSettings, renderConnectio onManage={() => window.open(remoteMcpProviders[connection.config?.sourceTemplateKey as keyof typeof remoteMcpProviders].dashboardUrl, "_blank", "noopener,noreferrer")} onDisconnect={() => disconnectRemote.mutateAsync()} />} + {grantsQuery.data?.capabilities.canConfigure === true && connection.connectionPurpose === "tool" && ( +
+ +
+ )} } )} diff --git a/ui/src/pages/apps/Browse.tsx b/ui/src/pages/apps/Browse.tsx index bbe64341603..8e47e351cb6 100644 --- a/ui/src/pages/apps/Browse.tsx +++ b/ui/src/pages/apps/Browse.tsx @@ -84,6 +84,7 @@ import { AlertDialogHeader, AlertDialogTitle, } from "@/components/ui/alert-dialog"; +import { Checkbox } from "@/components/ui/checkbox"; import { Button } from "@/components/ui/button"; import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible"; import { @@ -414,6 +415,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne const [aggregatorToConnect, setAggregatorToConnect] = useState(null); const [aggregatorToManage, setAggregatorToManage] = useState<{ app: AggregatorAppCatalogEntry; connectionId?: string } | null>(null); const [initialAggregatorProvider, setInitialAggregatorProvider] = useState(); + const [deleteStoredContent, setDeleteStoredContent] = useState(false); const [connectionToRemove, setConnectionToRemove] = useState(null); @@ -539,7 +541,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne if (!target.poolRevision) throw new Error("Open the confirmation again before removing this pool."); await aiConnectionPoolsApi.remove(selectedCompanyId!, target.id, target.poolRevision); } else { - await toolsApi.archiveConnection(target.id); + await toolsApi.archiveConnection(target.id, { deleteStoredContent }); } }, onSuccess: (_connection, target) => { @@ -564,6 +566,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne : target.remainingConnectionCount > 0 ? `${target.providerName} still has ${target.remainingConnectionCount} active ${target.remainingConnectionCount === 1 ? "connection" : "connections"} available to agents.` : `${target.providerName} is no longer available to agents through this connection. Its saved credentials were deleted.`, + ...(deleteStoredContent && !target.pool && target.kind !== "chat" ? { title: "Connection removed. Deleting its saved data." } : {}), tone: "success", }); setConnectionToRemove(null); @@ -577,6 +580,7 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne }); async function requestConnectionRemoval(target: ConnectionRemovalTarget) { removeConnection.reset(); + setDeleteStoredContent(false); if (!target.pool) { setConnectionToRemove(target); return; } try { const pool = (await aiConnectionPoolsApi.list(selectedCompanyId!)).find(pool => pool.id === target.id); @@ -1125,6 +1129,22 @@ export function Browse({ renderAccountDetails = (connection) => connection.conne : "The saved credentials are deleted and agents lose access immediately. Connecting it again later requires a new sign-in or key."} + {connectionToRemove && !connectionToRemove.pool && connectionToRemove.kind !== "chat" ? ( + + ) : null} {removeConnection.error ?

{describeError(removeConnection.error).body}

: null} diff --git a/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx b/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx new file mode 100644 index 00000000000..1986eb0c2a5 --- /dev/null +++ b/ui/src/pages/apps/app-detail/StoredContentDeletion.test.tsx @@ -0,0 +1,84 @@ +// @vitest-environment jsdom + +import { act } from "react"; +import { createRoot } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { ToolConnectionStoredContent } from "@paperclipai/shared"; + +const api = vi.hoisted(() => ({ + getStoredContent: vi.fn(), + getStoredContentDeletion: vi.fn(), + deleteStoredContent: vi.fn(), +})); +vi.mock("@/api/tools", () => ({ toolsApi: api })); +vi.mock("@/context/ToastContext", () => ({ useToast: () => ({ pushToast: vi.fn() }) })); + +import { StoredContentDeletionDialog, storedContentStatus, storedContentSummary } from "./StoredContentDeletion"; + +// eslint-disable-next-line @typescript-eslint/no-explicit-any +(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true; + +const counts = { toolCalls: 0, approvals: 0, auditEvents: 0, appViews: 0, runs: 0, runsInProgress: 0, itemsInProgress: 0 }; +const stored: ToolConnectionStoredContent = { + stored: { toolCalls: 12, approvals: 2, appViews: 0, runs: 3, runsInProgress: 1 }, + deletion: null, +}; + +let container: HTMLDivElement | null = null; +afterEach(() => { + container?.remove(); + container = null; + vi.clearAllMocks(); +}); + +async function flush() { + for (let i = 0; i < 10; i++) await act(async () => { await new Promise((resolve) => setTimeout(resolve, 0)); }); +} + +describe("stored content deletion", () => { + it("describes how much will be deleted in plain words", () => { + expect(storedContentSummary(stored, "Google Workspace")).toBe( + "Paperclip will delete what it saved from 12 tool calls, 2 approval requests and the full transcripts of 3 runs that used Google Workspace or received its results.", + ); + expect(storedContentSummary({ ...stored, stored: { toolCalls: 0, approvals: 0, appViews: 1, runs: 0, runsInProgress: 0 } }, "Docs")) + .toBe("Paperclip will delete what it saved from 1 app view that used Docs."); + expect(storedContentSummary({ ...stored, stored: { toolCalls: 0, approvals: 0, appViews: 0, runs: 0, runsInProgress: 0 } }, "Docs")) + .toBe("Paperclip found no saved data from Docs's tool calls. You can still delete to check again."); + expect(storedContentStatus({ ...stored, deletion: { id: "d", status: "pending", requestedAt: "", completedAt: null, retrying: true, counts } })) + .toBe("Deleting was interrupted. Paperclip keeps trying until it finishes."); + }); + + it("confirms with counts, warns about running work, and starts the deletion", async () => { + api.getStoredContent.mockResolvedValue(stored); + const pendingDeletion = { id: "d", status: "pending" as const, requestedAt: new Date().toISOString(), completedAt: null, retrying: false, counts }; + api.deleteStoredContent.mockResolvedValue({ deletion: pendingDeletion }); + api.getStoredContentDeletion.mockResolvedValue({ deletion: pendingDeletion }); + container = document.createElement("div"); + document.body.appendChild(container); + const root = createRoot(container); + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + await act(async () => { + root.render( + + undefined} /> + , + ); + }); + await flush(); + const text = document.body.textContent ?? ""; + expect(text).toContain("Delete data read through Google Workspace?"); + expect(text).toContain("12 tool calls"); + expect(text).toContain("1 run is still working and will be skipped."); + expect(text).toContain("This cannot be undone."); + expect(text).not.toContain("connection-1"); + + const confirm = Array.from(document.body.querySelectorAll("button")).find((button) => button.textContent === "Delete data"); + await act(async () => { confirm!.click(); }); + await flush(); + expect(api.deleteStoredContent).toHaveBeenCalledWith("connection-1"); + expect(document.body.textContent).toContain("Deleting now. You can close this window."); + expect(api.getStoredContent).toHaveBeenCalledTimes(1); + await act(async () => { root.unmount(); }); + }); +}); diff --git a/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx b/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx new file mode 100644 index 00000000000..b7b46f24210 --- /dev/null +++ b/ui/src/pages/apps/app-detail/StoredContentDeletion.tsx @@ -0,0 +1,171 @@ +import { useEffect, useRef, useState } from "react"; +import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { Loader2 } from "lucide-react"; +import type { ToolConnectionStoredContent } from "@paperclipai/shared"; +import { toolsApi } from "@/api/tools"; +import { describeError } from "@/api/errors"; +import { Button, buttonVariants } from "@/components/ui/button"; +import { + AlertDialog, + AlertDialogAction, + AlertDialogCancel, + AlertDialogContent, + AlertDialogDescription, + AlertDialogFooter, + AlertDialogHeader, + AlertDialogTitle, +} from "@/components/ui/alert-dialog"; +import { useToast } from "@/context/ToastContext"; +import { queryKeys } from "@/lib/queryKeys"; +import { relativeTime } from "@/lib/utils"; + +function plural(count: number, one: string, many: string) { + return `${count.toLocaleString()} ${count === 1 ? one : many}`; +} + +export function storedContentSummary(content: ToolConnectionStoredContent, appName: string): string { + const { toolCalls, approvals, appViews, runs } = content.stored; + const parts = [ + toolCalls > 0 ? plural(toolCalls, "tool call", "tool calls") : null, + approvals > 0 ? plural(approvals, "approval request", "approval requests") : null, + appViews > 0 ? plural(appViews, "app view", "app views") : null, + runs > 0 ? `the full transcripts of ${plural(runs, "run", "runs")}` : null, + ].filter((part): part is string => part !== null); + if (parts.length === 0) return `Paperclip found no saved data from ${appName}'s tool calls. You can still delete to check again.`; + const list = parts.length > 1 ? `${parts.slice(0, -1).join(", ")} and ${parts[parts.length - 1]}` : parts[0]; + return `Paperclip will delete what it saved from ${list} that used ${appName}${runs > 0 ? " or received its results" : ""}.`; +} + +export function storedContentStatus(content: ToolConnectionStoredContent): string | null { + const deletion = content.deletion; + if (deletion?.status === "pending") { + return deletion.retrying + ? "Deleting was interrupted. Paperclip keeps trying until it finishes." + : "Deleting now. You can close this window."; + } + if (deletion?.status === "completed" && deletion.completedAt) { + const skipped = deletion.counts.runsInProgress; + return skipped > 0 + ? `Last deleted ${relativeTime(deletion.completedAt)}. ${plural(skipped, "run was", "runs were")} still working and kept their data.` + : `Last deleted ${relativeTime(deletion.completedAt)}.`; + } + return null; +} + +export function StoredContentDeletionRow({ connectionId, appName }: { connectionId: string; appName: string }) { + const [open, setOpen] = useState(false); + return ( +
+
+

Delete data read through this app

+

+ Permanently removes the results and run transcripts Paperclip saved from this app's tool calls. +

+
+ + {open ? ( + setOpen(false)} /> + ) : null} +
+ ); +} + +export function StoredContentDeletionDialog({ + connectionId, + appName, + onClose, +}: { + connectionId: string; + appName: string; + onClose: () => void; +}) { + const queryClient = useQueryClient(); + const { pushToast } = useToast(); + const countsKey = queryKeys.tools.connectionStoredContent(connectionId); + const statusKey = queryKeys.tools.connectionStoredContentDeletion(connectionId); + const query = useQuery({ + queryKey: countsKey, + queryFn: () => toolsApi.getStoredContent(connectionId), + staleTime: Infinity, + refetchOnMount: "always", + refetchOnWindowFocus: false, + }); + const [tracking, setTracking] = useState(false); + const watching = tracking || query.data?.deletion?.status === "pending"; + const statusQuery = useQuery({ + queryKey: statusKey, + queryFn: () => toolsApi.getStoredContentDeletion(connectionId), + enabled: watching, + refetchInterval: (current) => current.state.data?.deletion?.status === "completed" ? false : 2_000, + }); + const latest = statusQuery.data?.deletion ?? query.data?.deletion ?? null; + const refreshedFor = useRef(null); + useEffect(() => { + if (!watching || latest?.status !== "completed" || refreshedFor.current === latest.id) return; + refreshedFor.current = latest.id; + setTracking(false); + void queryClient.invalidateQueries({ queryKey: queryKeys.tools.connectionStoredContent(connectionId) }); + }, [watching, latest?.status, latest?.id, queryClient, connectionId]); + const deletion = useMutation({ + mutationFn: () => toolsApi.deleteStoredContent(connectionId), + onSuccess: (result) => { + queryClient.setQueryData(statusKey, result); + setTracking(true); + }, + onError: (error) => pushToast({ title: "Couldn't delete the data", body: describeError(error).body, tone: "error" }), + }); + const content = query.data ? { ...query.data, deletion: latest } : undefined; + const pending = deletion.isPending || latest?.status === "pending"; + const inProgress = content?.stored.runsInProgress ?? 0; + const status = content ? storedContentStatus(content) : null; + + return ( + { if (!next) onClose(); }}> + + + {`Delete data read through ${appName}?`} + + {content ? storedContentSummary(content, appName) : "Counting saved data…"} + + + {query.error ? ( +

{describeError(query.error).body}

+ ) : null} + {content ? ( +
+

Whole transcripts are deleted, including other steps of those runs. Call times, sizes and outcomes stay in the activity history.

+

This does not remove anything your agents wrote into tasks, such as comments, documents and files. Delete or edit those yourself.

+ {inProgress > 0 ? ( +

{`${plural(inProgress, "run is", "runs are")} still working and will be skipped. Delete again after ${inProgress === 1 ? "it finishes" : "they finish"}.`}

+ ) : null} +

This cannot be undone.

+
+ ) : null} + {status ? ( +

+ {content?.deletion?.status === "pending" ? : null} + {status} +

+ ) : null} + + {pending ? "Close" : "Cancel"} + {content ? ( + { + event.preventDefault(); + deletion.mutate(); + }} + > + {pending ? : null} + Delete data + + ) : null} + +
+
+ ); +}