From e591e75b8c5682990babf0b6a47287b7c488f58e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:25:27 -0500 Subject: [PATCH 001/311] perf(db): index context_snapshot/payload issueId lookups used by recovery sweeps (#10969) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server runs a recovery sweep every 30 seconds. The sweep makes sure assigned issues do not stall. > - The sweep reads the latest heartbeat run for each candidate issue. It filters `heartbeat_runs` on `context_snapshot ->> 'issueId'`. No index covers this expression. > - Each lookup scans ~26k rows and detoasts each row's JSONB context (~645 ms per query, measured with EXPLAIN ANALYZE). With ~460 candidate issues, one sweep schedules ~500 seconds of database work every 30 seconds. > - The database saturates. Users see the whole server as slow. > - This pull request adds expression indexes so these lookups become index scans. > - The benefit is that the recovery sweep drops from ~500 seconds of database work per tick to milliseconds, and the server becomes responsive again. ## Linked Issues or Issue Description **What happened?** The server became slow for all users. `pg_stat_activity` sampling showed the same two recovery-sweep queries active continuously (15/15 samples). Each `getLatestIssueRun` call filtered `heartbeat_runs` on the unindexed expression `context_snapshot ->> 'issueId'`, scanned ~26k rows, and detoasted a 2.1 GB TOAST region. `heartbeat_runs` accumulated 10.8 billion sequential tuples read. `hasActiveExecutionPath` also scanned `agent_wakeup_requests` (1.8M rows) on the unindexed expression `payload ->> 'issueId'`. **Expected behavior** Per-issue run lookups in the recovery sweep complete in milliseconds. The sweep finishes well inside its 30-second interval. Background maintenance does not degrade interactive latency. **Steps to reproduce** 1. Run a board with several hundred issues in `todo` / `in_progress` / `in_review` and a large `heartbeat_runs` table with big `context_snapshot` payloads. 2. Let the heartbeat scheduler run its 30-second recovery sweep. 3. Observe `pg_stat_activity`: the per-issue `heartbeat_runs` lookups run continuously; `EXPLAIN ANALYZE` shows a filter on `context_snapshot ->> 'issueId'` removing tens of thousands of rows per call. **Paperclip version or commit** master (814cb336) ## What Changed - Add migration `0209_heartbeat_context_snapshot_indexes.sql` with three expression indexes: - `heartbeat_runs (company_id, (context_snapshot ->> 'issueId'), created_at DESC)` - `heartbeat_runs (company_id, (context_snapshot ->> 'taskId'), created_at DESC)` - `agent_wakeup_requests (company_id, (payload ->> 'issueId'))` β€” with a `large-create-index-not-concurrently` safety pragma and justification, following the migration 0206 precedent. - Mirror the three indexes in the drizzle schema files (`heartbeat_runs.ts`, `agent_wakeup_requests.ts`). - Add `heartbeat-context-snapshot-index-migration.test.ts`. The test boots a fresh embedded Postgres, applies the full migration chain, asserts the indexes exist, and asserts with `EXPLAIN` that the planner selects them for the exact hot query shapes. ## Verification - `pnpm --filter @paperclipai/db exec tsx src/check-migration-numbering.ts` passes. - `pnpm --filter @paperclipai/db exec tsx src/check-migration-safety.ts` passes. - `npx vitest run src/heartbeat-context-snapshot-index-migration.test.ts` passes (fresh embedded Postgres, full chain 0000β†’0209, planner uses all three indexes). - `npx vitest run src/check-migration-safety.test.ts` passes (25/25). - `tsc --noEmit` clean for `packages/db`. ## Risks - The index builds run inside the transactional migration (no `CONCURRENTLY`). The `heartbeat_runs` build must read its 2.1 GB TOAST once; expect the migration step to add roughly 1–3 minutes to one deploy while writes to the two tables wait. This is a one-time cost at startup, before the server accepts traffic. - Three new indexes add small write overhead to two hot-write tables. The read savings are several orders of magnitude larger. - No query or API behavior changes; the planner simply gains a better access path. ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic), extended thinking, with tool use (shell, Postgres EXPLAIN/ANALYZE against the live instance for measurement, vitest for verification). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge πŸ€– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 Co-authored-by: Paperclip --- ...t-context-snapshot-index-migration.test.ts | 65 +++++++++++++++++++ ...209_heartbeat_context_snapshot_indexes.sql | 4 ++ packages/db/src/migrations/meta/_journal.json | 7 ++ .../db/src/schema/agent_wakeup_requests.ts | 4 ++ packages/db/src/schema/heartbeat_runs.ts | 11 ++++ 5 files changed, 91 insertions(+) create mode 100644 packages/db/src/heartbeat-context-snapshot-index-migration.test.ts create mode 100644 packages/db/src/migrations/0209_heartbeat_context_snapshot_indexes.sql diff --git a/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts b/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts new file mode 100644 index 00000000000..e4147bc6dc0 --- /dev/null +++ b/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts @@ -0,0 +1,65 @@ +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it, afterEach } from "vitest"; +import postgres from "postgres"; +import { + getEmbeddedPostgresTestSupport, + startEmbeddedPostgresTestDatabase, +} from "./test-embedded-postgres.js"; + +const cleanups: Array<() => Promise> = []; +const support = await getEmbeddedPostgresTestSupport(); +const d = support.supported ? describe : describe.skip; + +afterEach(async () => { + while (cleanups.length > 0) await cleanups.pop()?.(); +}); + +d("heartbeat context_snapshot expression index migration", () => { + it("applies full migration chain and uses the new indexes", async () => { + const dbh = await startEmbeddedPostgresTestDatabase("pap16575-idx-"); + cleanups.push(() => dbh.cleanup()); + const sql = postgres(dbh.connectionString, { max: 1 }); + cleanups.push(async () => { await sql.end(); }); + + const idx = await sql`SELECT indexname FROM pg_indexes WHERE tablename IN ('heartbeat_runs','agent_wakeup_requests')`; + const names = idx.map((r) => r.indexname as string); + expect(names).toContain("heartbeat_runs_company_ctx_issue_created_idx"); + expect(names).toContain("heartbeat_runs_company_ctx_task_created_idx"); + expect(names).toContain("agent_wakeup_requests_company_payload_issue_idx"); + + await sql.unsafe("SET enable_seqscan = off"); + const plan = await sql.unsafe( + "EXPLAIN SELECT id FROM heartbeat_runs WHERE company_id = '00000000-0000-0000-0000-000000000001' AND context_snapshot ->> 'issueId' = 'x' ORDER BY created_at DESC, id DESC LIMIT 1", + ); + const planText = plan.map((r) => Object.values(r)[0]).join("\n"); + expect(planText).toContain("heartbeat_runs_company_ctx_issue_created_idx"); + + const taskPlan = await sql.unsafe( + "EXPLAIN SELECT id FROM heartbeat_runs WHERE company_id = '00000000-0000-0000-0000-000000000001' AND context_snapshot ->> 'taskId' = 'x' ORDER BY created_at DESC, id DESC LIMIT 1", + ); + const taskText = taskPlan.map((r) => Object.values(r)[0]).join("\n"); + expect(taskText).toContain("heartbeat_runs_company_ctx_task_created_idx"); + + const wakePlan = await sql.unsafe( + "EXPLAIN SELECT id FROM agent_wakeup_requests WHERE company_id = '00000000-0000-0000-0000-000000000001' AND status = 'deferred_issue_execution' AND payload ->> 'issueId' = 'x' LIMIT 1", + ); + const wakeText = wakePlan.map((r) => Object.values(r)[0]).join("\n"); + expect(wakeText).toContain("agent_wakeup_requests_company_payload_issue_idx"); + + // Idempotency: re-running the migration statements against an already + // migrated database must be a no-op, not an error. + const migrationSql = await readFile( + fileURLToPath(new URL("./migrations/0209_heartbeat_context_snapshot_indexes.sql", import.meta.url)), + "utf8", + ); + const statements = migrationSql + .split("--> statement-breakpoint") + .map((s) => s.trim()) + .filter((s) => s.length > 0); + expect(statements.length).toBeGreaterThan(0); + for (const statement of statements) { + await sql.unsafe(statement); + } + }, 240_000); +}); diff --git a/packages/db/src/migrations/0209_heartbeat_context_snapshot_indexes.sql b/packages/db/src/migrations/0209_heartbeat_context_snapshot_indexes.sql new file mode 100644 index 00000000000..9331d634508 --- /dev/null +++ b/packages/db/src/migrations/0209_heartbeat_context_snapshot_indexes.sql @@ -0,0 +1,4 @@ +CREATE INDEX IF NOT EXISTS "heartbeat_runs_company_ctx_issue_created_idx" ON "heartbeat_runs" USING btree ("company_id", ("context_snapshot" ->> 'issueId'), "created_at" DESC);--> statement-breakpoint +CREATE INDEX IF NOT EXISTS "heartbeat_runs_company_ctx_task_created_idx" ON "heartbeat_runs" USING btree ("company_id", ("context_snapshot" ->> 'taskId'), "created_at" DESC);--> statement-breakpoint +-- paperclip:migration-safety-ignore large-create-index-not-concurrently: Drizzle migrations run transactionally, so CONCURRENTLY is unavailable. This expression index removes per-issue full scans of agent_wakeup_requests in recovery sweeps (hasActiveExecutionPath), which currently dominate server load; the one-time build lock is the lesser cost. +CREATE INDEX IF NOT EXISTS "agent_wakeup_requests_company_payload_issue_idx" ON "agent_wakeup_requests" USING btree ("company_id", ("payload" ->> 'issueId')); diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 07ed0c584e2..06f3456dbaa 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -1450,6 +1450,13 @@ "when": 1785988680096, "tag": "0208_keen_sharon_carter", "breakpoints": true + }, + { + "idx": 209, + "version": "7", + "when": 1786020026023, + "tag": "0209_heartbeat_context_snapshot_indexes", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/agent_wakeup_requests.ts b/packages/db/src/schema/agent_wakeup_requests.ts index 7b0ec64fa92..194e1732a32 100644 --- a/packages/db/src/schema/agent_wakeup_requests.ts +++ b/packages/db/src/schema/agent_wakeup_requests.ts @@ -40,5 +40,9 @@ export const agentWakeupRequests = pgTable( reviewPathRecoveryIdempotencyUq: uniqueIndex("agent_wakeup_requests_review_path_recovery_idempotency_uq") .on(table.companyId, table.idempotencyKey) .where(sql`${table.idempotencyKey} LIKE 'issue_review_path_lost:%' AND ${table.status} <> 'skipped'`), + companyPayloadIssueIdx: index("agent_wakeup_requests_company_payload_issue_idx").on( + table.companyId, + sql`(${table.payload} ->> 'issueId')`, + ), }), ); diff --git a/packages/db/src/schema/heartbeat_runs.ts b/packages/db/src/schema/heartbeat_runs.ts index 0f3590bb0f7..49d3dd8f610 100644 --- a/packages/db/src/schema/heartbeat_runs.ts +++ b/packages/db/src/schema/heartbeat_runs.ts @@ -1,3 +1,4 @@ +import { sql } from "drizzle-orm"; import { type AnyPgColumn, pgTable, uuid, text, timestamp, jsonb, index, integer, bigint, boolean } from "drizzle-orm/pg-core"; import { companies } from "./companies.js"; import { agents } from "./agents.js"; @@ -88,5 +89,15 @@ export const heartbeatRuns = pgTable( table.companyId, table.createdAt.desc(), ), + companyCtxIssueCreatedIdx: index("heartbeat_runs_company_ctx_issue_created_idx").on( + table.companyId, + sql`(${table.contextSnapshot} ->> 'issueId')`, + table.createdAt.desc(), + ), + companyCtxTaskCreatedIdx: index("heartbeat_runs_company_ctx_task_created_idx").on( + table.companyId, + sql`(${table.contextSnapshot} ->> 'taskId')`, + table.createdAt.desc(), + ), }), ); From 656ecfa585b31938e2685ffab3db22e794474803 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:29:07 -0500 Subject: [PATCH 002/311] fix(server): keep Date fields intact through secret redaction; harden chat notice timestamps (#10984) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task chat thread renders issue comments, system notices, and run transcripts > - The server routes comment payloads through the run-secret redaction walker before it sends them > - The walker rebuilds each object with `Object.entries`, and this collapses `Date` instances to `{}` > - The chat renderer then calls `.toISOString()` on an invalid date and throws, and the thread falls back to the error banner > - This pull request keeps `Date` instances intact in redacted responses and makes the renderer safe against bad timestamps > - The benefit is that task threads with system notices render correctly again ## Linked Issues or Issue Description **What happened** Task threads that contain a system notice showed the banner "Chat renderer hit an internal state error." in place of the conversation. This occurred on many tasks. **Expected behavior** The thread renders all comments and system notices with correct timestamps. **Steps to reproduce** 1. Open a task that has at least one system notice comment (for example a "Workspace ready" notice). 2. `GET /api/issues/{id}/comments` returns `createdAt: {}` for every comment because the secret-redaction walker collapses `Date` objects. 3. The system-notice row calls `new Date({}).toISOString()`. This throws `RangeError: Invalid time value` and trips the thread error boundary. **Version / deployment** Regression from #9934 (`e43f187ca`). It applies to all deployments that include that commit. ## What Changed - `server/src/services/run-secret-redaction.ts`: `redactRegisteredSecretValues` now returns `Date` instances as-is. Dates hold no redactable text, and the `Object.entries` rebuild turned them into `{}`. - `ui/src/components/IssueChatThread.tsx`: the system-notice row formats its timestamp with a new `toValidIsoString` helper. A value that does not parse as a date now degrades to "no timestamp" instead of a render crash. - Regression tests at three layers: - Walker unit tests: `Date` values survive with and without registered secret values. - Route test: `GET /issues/:id/comments` serializes `createdAt` / `updatedAt` as ISO strings. - Render test: a system notice with a malformed `createdAt` renders without the error boundary. ## Verification - `npx vitest run --root server src/__tests__/run-secret-redaction.test.ts` β€” 5 passed. - `npx vitest run --root server src/__tests__/issue-comment-redaction.test.ts` β€” 4 passed (embedded Postgres route test). - `cd ui && npx vitest run src/components/IssueChatThread.test.tsx src/components/IssueChatThreadSystemNotice.test.tsx src/lib/issue-chat-messages.test.ts` β€” 121 passed. - Each new test was run against the unfixed code and failed there, which confirms it guards the regression. - A local sweep rendered 47 real issue threads through `IssueChatThread`: 7 tripped the boundary before the fix, 0 after. ## Risks - Low risk. The server change only preserves `Date` objects that the walker destroyed before. String redaction behavior does not change, and the registry-key stripping does not change. - The UI change only affects the timestamp of system-notice rows and omits it when the value is invalid. ## Model Used - Claude Fable 5 (`claude-fable-5`), Anthropic. Agentic coding session with tool use (file edits, shell, Vitest). No extended-context or special reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Claude Fable 5 Co-authored-by: Paperclip --- .../__tests__/issue-comment-redaction.test.ts | 21 ++++++++++++++ .../__tests__/run-secret-redaction.test.ts | 21 ++++++++++++++ server/src/services/run-secret-redaction.ts | 3 ++ ui/src/components/IssueChatThread.tsx | 14 +++++++++- .../IssueChatThreadSystemNotice.test.tsx | 28 +++++++++++++++++++ 5 files changed, 86 insertions(+), 1 deletion(-) diff --git a/server/src/__tests__/issue-comment-redaction.test.ts b/server/src/__tests__/issue-comment-redaction.test.ts index d53392056c3..423f1ab3454 100644 --- a/server/src/__tests__/issue-comment-redaction.test.ts +++ b/server/src/__tests__/issue-comment-redaction.test.ts @@ -187,6 +187,27 @@ describeEmbeddedPostgres("deleted issue comment redaction", () => { expect(JSON.stringify(wakePayload)).not.toContain("secret metadata"); }); + it("serializes comment timestamps as ISO strings through the redacted comments route (PAP-16607)", async () => { + const { companyId, issueId } = await seedIssue(); + const commentId = randomUUID(); + await db.insert(issueComments).values({ + id: commentId, + companyId, + issueId, + authorUserId: "board-user-1", + body: "ordinary comment", + }); + + const response = await request(createApp(companyId)).get(`/api/issues/${issueId}/comments`); + expect(response.status, JSON.stringify(response.body)).toBe(200); + expect(response.body).toHaveLength(1); + // Secret redaction must not collapse Date instances to `{}` β€” the chat + // renderer needs parseable timestamps. + expect(typeof response.body[0].createdAt).toBe("string"); + expect(Number.isNaN(new Date(response.body[0].createdAt).getTime())).toBe(false); + expect(typeof response.body[0].updatedAt).toBe("string"); + }); + it("excludes deleted comment bodies from company search", async () => { const { companyId, issueId } = await seedIssue(); await db.insert(issueComments).values({ diff --git a/server/src/__tests__/run-secret-redaction.test.ts b/server/src/__tests__/run-secret-redaction.test.ts index 5bff578f14c..fa2dd60a06e 100644 --- a/server/src/__tests__/run-secret-redaction.test.ts +++ b/server/src/__tests__/run-secret-redaction.test.ts @@ -54,4 +54,25 @@ describe("registered run secret redaction", () => { expect(redactRegisteredSecretValues("token-extended token", ["token-extended", "token"])) .toBe(`${REDACTED_EVENT_VALUE} ${REDACTED_EVENT_VALUE}`); }); + + it("preserves Date instances instead of collapsing them to empty objects (PAP-16607)", () => { + const createdAt = new Date("2026-08-06T12:00:00.000Z"); + const result = redactRegisteredSecretValues({ + comment: { body: `agent pasted ${secret}`, createdAt, updatedAt: createdAt }, + nested: [{ finishedAt: createdAt }], + }, [secret]); + + expect(result.comment.createdAt).toBeInstanceOf(Date); + expect(result.comment.createdAt.toISOString()).toBe("2026-08-06T12:00:00.000Z"); + expect(result.comment.updatedAt).toBeInstanceOf(Date); + expect(result.nested[0]?.finishedAt).toBeInstanceOf(Date); + expect(result.comment.body).toBe(`agent pasted ${REDACTED_EVENT_VALUE}`); + }); + + it("preserves Date instances when no secret values are registered", () => { + const createdAt = new Date("2026-08-06T12:00:00.000Z"); + const result = redactRegisteredSecretValues({ createdAt }, []); + expect(result.createdAt).toBeInstanceOf(Date); + expect(result.createdAt.toISOString()).toBe("2026-08-06T12:00:00.000Z"); + }); }); diff --git a/server/src/services/run-secret-redaction.ts b/server/src/services/run-secret-redaction.ts index 4604d54adaa..4ca7963e956 100644 --- a/server/src/services/run-secret-redaction.ts +++ b/server/src/services/run-secret-redaction.ts @@ -42,6 +42,9 @@ function redactText(input: string, values: string[]) { export function redactRegisteredSecretValues(input: T, values: string[]): T { if (typeof input === "string") return redactText(input, values) as T; if (Array.isArray(input)) return input.map((value) => redactRegisteredSecretValues(value, values)) as T; + // Dates carry no redactable text; rebuilding them via Object.entries would + // collapse them to `{}` and break every timestamp in redacted responses. + if (input instanceof Date) return input; const record = asRecord(input); if (!record) return input; return Object.fromEntries( diff --git a/ui/src/components/IssueChatThread.tsx b/ui/src/components/IssueChatThread.tsx index c540cab21ab..c6af475e502 100644 --- a/ui/src/components/IssueChatThread.tsx +++ b/ui/src/components/IssueChatThread.tsx @@ -773,6 +773,18 @@ function toIsoString(value: string | Date | null | undefined): string | null { return typeof value === "string" ? value : value.toISOString(); } +/** + * ISO timestamp for display, or undefined when the value does not parse as a + * real date. Comment timestamps can arrive malformed (e.g. a server + * serialization bug turning Dates into `{}`); formatting must degrade to "no + * timestamp" instead of throwing mid-render (PAP-16607). + */ +function toValidIsoString(value: Date | string | number | undefined): string | undefined { + if (value === undefined || value === null) return undefined; + const date = new Date(value); + return Number.isNaN(date.getTime()) ? undefined : date.toISOString(); +} + function loadDraft(draftKey: string): string { try { return localStorage.getItem(draftKey) ?? ""; @@ -2737,7 +2749,7 @@ function SystemNoticeCommentRow({ {bodyText} ), - timestamp: message.createdAt ? new Date(message.createdAt).toISOString() : undefined, + timestamp: toValidIsoString(message.createdAt), source, runAgentId, }); diff --git a/ui/src/components/IssueChatThreadSystemNotice.test.tsx b/ui/src/components/IssueChatThreadSystemNotice.test.tsx index 410d6b3068f..8419c54ebeb 100644 --- a/ui/src/components/IssueChatThreadSystemNotice.test.tsx +++ b/ui/src/components/IssueChatThreadSystemNotice.test.tsx @@ -150,6 +150,34 @@ describe("IssueChatThread system notice routing", () => { expect(container.querySelectorAll('[data-message-role="user"]').length).toBe(0); }); + it("renders a system notice with a malformed createdAt without tripping the error boundary (PAP-16607)", () => { + const comment: IssueChatComment = { + id: "comment-system-bad-date", + companyId: "company-1", + issueId: "issue-1", + authorType: "system", + authorAgentId: null, + authorUserId: null, + body: "Workspace ready.", + presentation: { + kind: "system_notice", + tone: "info", + title: "Workspace ready", + detailsDefaultOpen: false, + }, + metadata: { version: 1, sections: [] }, + // A server serialization bug (Dates collapsed to `{}` by secret + // redaction) shipped comments whose timestamps parse to Invalid Date. + createdAt: {} as unknown as Date, + updatedAt: {} as unknown as Date, + }; + + renderThread([comment]); + + expect(container.textContent).not.toContain("Chat renderer hit an internal state error."); + expect(container.textContent).toContain("Workspace ready"); + }); + it("expands metadata when detailsDefaultOpen is true", () => { const comment: IssueChatComment = { id: "comment-system-open", From 52b8741b8e2ee39f03c2e274fc95eb56184eabd9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 6 Aug 2026 11:56:40 -0500 Subject: [PATCH 003/311] perf(server): cut steady-state DB hot paths in dashboard, attention, and productivity sweeps (#10992) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server keeps fleet health with periodic sweeps and shows a dashboard with run activity > - The Paperclip instance became slow again after the first round of recovery-sweep indexes landed > - Live profiling found four steady-state hot paths that read much more data than they use > - This pull request bounds the dashboard recursion, adds the missing taskKey index, and narrows two wide reads > - The benefit is a large drop in constant database load and a responsive server ## Linked Issues or Issue Description **Describe the bug** The server becomes slow while agents work. Live query sampling shows four hot paths: 1. The dashboard run-activity recursive CTE reads every run a company ever had on each call. One call takes 2.85 seconds. The UI calls it after almost every fleet event through the dashboard and sidebar-badges routes. 2. The productivity-review sweep runs each 30 seconds. Its run-scope filter is `issueId OR taskId OR taskKey` on the run context JSONB. No index exists for `taskKey`. The planner must detoast every run snapshot for the agent. One query takes 444 ms and the sweep makes one for each of ~152 candidate issues. 3. The attention failed-run section selects the full `context_snapshot` for every run newer than the oldest exhausted run. That fetch moves 29 MB for each feed build. 4. The retention sweep pages the attention feed with a cursor. Each page makes a full feed rebuild. **Expected behavior** Periodic sweeps and dashboard queries read only the data they use, and use indexes. **Actual behavior** The database stays saturated. Users see a slow server. ## What Changed - `server/src/services/dashboard.ts`: bound both arms of the `recovered_runs` recursive CTE to the chart window. A retry is always newer than the run it retries, so the bound cannot change visible chart data. Live time went from 2,852 ms to 54 ms. - `packages/db/src/migrations/0210_heartbeat_context_taskkey_index.sql`: add the `taskKey` expression index that completes the issueId/taskId/taskKey trio. With all three, the planner uses a BitmapOr. Live time for the productivity run-scope query went from 444 ms to 1.9 ms. - `packages/db/src/schema/heartbeat_runs.ts`: mirror the new index in the Drizzle schema. - `server/src/services/productivity-review.ts`: select only the seven run fields the evidence code reads. Before, the query pulled full rows with `result_json` (up to 43 kB per row, 100 rows per issue). - `server/src/services/attention.ts`: project `issueId`/`taskId` text fields instead of the full `context_snapshot` in the failed-run newer-runs query (29 MB per feed build before). - `server/src/index.ts`: the retention sweep now builds the attention feed once per company with `all: true` instead of one full rebuild per cursor page. - `packages/db/src/heartbeat-context-snapshot-index-migration.test.ts`: cover the new index and re-run migration 0210 statements to prove idempotency. ## Verification - `pnpm --filter @paperclipai/db typecheck` (includes migration numbering and safety checks) β€” pass. - `npx tsc --noEmit` in `server/` β€” pass. - `npx vitest run packages/db/src/heartbeat-context-snapshot-index-migration.test.ts` β€” pass (embedded Postgres, full migration chain, planner assertions, idempotent re-run of 0209 and 0210). - `npx vitest run` on attention, dashboard, productivity-review, decision-retention, issue-blocker-attention, and issue-review-attention test files β€” 72/72 pass. - Live EXPLAIN ANALYZE before/after numbers are in the What Changed list. ## Risks - Migration 0210 builds one btree index without CONCURRENTLY inside the transactional migration runner. The table is not in the large-table bucket. The 0209 twin built in seconds on a 100k-row live table. - The CTE bound excludes retry ancestors that are older than the chart window. Those rows are not visible to the chart query, so chart output does not change. - The attention projection changes JSONB scalar handling in one edge case: a non-string `issueId`/`taskId` value now casts to text instead of reading as absent. These keys are always strings in practice. - The retention sweep now holds one full feed in memory per company. The cursor loop already accumulated all pages into one array, so peak memory is unchanged. ## Model Used Claude Fable 5 (`claude-fable-5`, Anthropic, Mythos-class tier, extended thinking + tool use) via Paperclip agent runtime. - [x] I searched existing PRs and issues and this change is not a duplicate. Co-authored-by: Claude Fable 5 --- ...t-context-snapshot-index-migration.test.ts | 38 +++++++++++++------ .../0210_heartbeat_context_taskkey_index.sql | 1 + packages/db/src/migrations/meta/_journal.json | 9 ++++- packages/db/src/schema/heartbeat_runs.ts | 5 +++ server/src/index.ts | 21 +++++----- server/src/services/attention.ts | 8 +++- server/src/services/dashboard.ts | 6 +++ server/src/services/productivity-review.ts | 18 ++++++++- 8 files changed, 78 insertions(+), 28 deletions(-) create mode 100644 packages/db/src/migrations/0210_heartbeat_context_taskkey_index.sql diff --git a/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts b/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts index e4147bc6dc0..36a4ea6a768 100644 --- a/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts +++ b/packages/db/src/heartbeat-context-snapshot-index-migration.test.ts @@ -26,6 +26,7 @@ d("heartbeat context_snapshot expression index migration", () => { const names = idx.map((r) => r.indexname as string); expect(names).toContain("heartbeat_runs_company_ctx_issue_created_idx"); expect(names).toContain("heartbeat_runs_company_ctx_task_created_idx"); + expect(names).toContain("heartbeat_runs_company_ctx_taskkey_created_idx"); expect(names).toContain("agent_wakeup_requests_company_payload_issue_idx"); await sql.unsafe("SET enable_seqscan = off"); @@ -41,6 +42,16 @@ d("heartbeat context_snapshot expression index migration", () => { const taskText = taskPlan.map((r) => Object.values(r)[0]).join("\n"); expect(taskText).toContain("heartbeat_runs_company_ctx_task_created_idx"); + // The productivity-review run scope ORs issueId/taskId/taskKey; all three + // expression indexes must exist so the planner can BitmapOr at real row + // counts instead of detoasting every run snapshot for the agent. An empty + // table plans a single index scan, so assert the taskKey index directly. + const taskKeyPlan = await sql.unsafe( + "EXPLAIN SELECT id FROM heartbeat_runs WHERE company_id = '00000000-0000-0000-0000-000000000001' AND context_snapshot ->> 'taskKey' = 'x' ORDER BY created_at DESC, id DESC LIMIT 1", + ); + const taskKeyText = taskKeyPlan.map((r) => Object.values(r)[0]).join("\n"); + expect(taskKeyText).toContain("heartbeat_runs_company_ctx_taskkey_created_idx"); + const wakePlan = await sql.unsafe( "EXPLAIN SELECT id FROM agent_wakeup_requests WHERE company_id = '00000000-0000-0000-0000-000000000001' AND status = 'deferred_issue_execution' AND payload ->> 'issueId' = 'x' LIMIT 1", ); @@ -49,17 +60,22 @@ d("heartbeat context_snapshot expression index migration", () => { // Idempotency: re-running the migration statements against an already // migrated database must be a no-op, not an error. - const migrationSql = await readFile( - fileURLToPath(new URL("./migrations/0209_heartbeat_context_snapshot_indexes.sql", import.meta.url)), - "utf8", - ); - const statements = migrationSql - .split("--> statement-breakpoint") - .map((s) => s.trim()) - .filter((s) => s.length > 0); - expect(statements.length).toBeGreaterThan(0); - for (const statement of statements) { - await sql.unsafe(statement); + for (const migration of [ + "./migrations/0209_heartbeat_context_snapshot_indexes.sql", + "./migrations/0210_heartbeat_context_taskkey_index.sql", + ]) { + const migrationSql = await readFile( + fileURLToPath(new URL(migration, import.meta.url)), + "utf8", + ); + const statements = migrationSql + .split("--> statement-breakpoint") + .map((s) => s.trim()) + .filter((s) => s.length > 0); + expect(statements.length).toBeGreaterThan(0); + for (const statement of statements) { + await sql.unsafe(statement); + } } }, 240_000); }); diff --git a/packages/db/src/migrations/0210_heartbeat_context_taskkey_index.sql b/packages/db/src/migrations/0210_heartbeat_context_taskkey_index.sql new file mode 100644 index 00000000000..0edc5eb44e8 --- /dev/null +++ b/packages/db/src/migrations/0210_heartbeat_context_taskkey_index.sql @@ -0,0 +1 @@ +CREATE INDEX IF NOT EXISTS "heartbeat_runs_company_ctx_taskkey_created_idx" ON "heartbeat_runs" USING btree ("company_id", ("context_snapshot" ->> 'taskKey'), "created_at" DESC); diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 06f3456dbaa..9961a106f56 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -1457,6 +1457,13 @@ "when": 1786020026023, "tag": "0209_heartbeat_context_snapshot_indexes", "breakpoints": true + }, + { + "idx": 210, + "version": "7", + "when": 1786032087897, + "tag": "0210_heartbeat_context_taskkey_index", + "breakpoints": true } ] -} \ No newline at end of file +} diff --git a/packages/db/src/schema/heartbeat_runs.ts b/packages/db/src/schema/heartbeat_runs.ts index 49d3dd8f610..6965e1669fa 100644 --- a/packages/db/src/schema/heartbeat_runs.ts +++ b/packages/db/src/schema/heartbeat_runs.ts @@ -99,5 +99,10 @@ export const heartbeatRuns = pgTable( sql`(${table.contextSnapshot} ->> 'taskId')`, table.createdAt.desc(), ), + companyCtxTaskKeyCreatedIdx: index("heartbeat_runs_company_ctx_taskkey_created_idx").on( + table.companyId, + sql`(${table.contextSnapshot} ->> 'taskKey')`, + table.createdAt.desc(), + ), }), ); diff --git a/server/src/index.ts b/server/src/index.ts index 86d05f1a143..7350e7d99ce 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1133,18 +1133,15 @@ export async function startServer(): Promise { const activeCompanies = await db.select({ id: companies.id }).from(companies).where(eq(companies.status, "active")); let archived = 0; for (const company of activeCompanies) { - const items = []; - let cursor: string | undefined; - do { - const page = await attentionService(db as any).list(company.id, { - includeDismissed: true, - limit: 100, - cursor, - }); - items.push(...page.items); - cursor = page.nextCursor ?? undefined; - } while (cursor); - archived += await retentionExecutor.autoArchive({ companyId: company.id, items }); + // Cursor pagination rebuilds the whole feed for every page; one + // unscoped all-items build keeps this sweep at a single feed build + // per company per tick. + const page = await attentionService(db as any).list(company.id, { + includeDismissed: true, + all: true, + allowUnscopedAll: true, + }); + archived += await retentionExecutor.autoArchive({ companyId: company.id, items: page.items }); } const notifications = await retentionExecutor.deliverNotifications(); return { archived, ...notifications }; diff --git a/server/src/services/attention.ts b/server/src/services/attention.ts index 6ccee4fbffc..0b4b72688e3 100644 --- a/server/src/services/attention.ts +++ b/server/src/services/attention.ts @@ -1704,7 +1704,10 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions .select({ agentId: heartbeatRuns.agentId, createdAt: heartbeatRuns.createdAt, - contextSnapshot: heartbeatRuns.contextSnapshot, + // Project just the ids readRunIssueId needs; pulling the whole + // context_snapshot detoasts megabytes per feed build. + runIssueId: sql`${heartbeatRuns.contextSnapshot} ->> 'issueId'`, + runTaskId: sql`${heartbeatRuns.contextSnapshot} ->> 'taskId'`, }) .from(heartbeatRuns) .where(and( @@ -1716,7 +1719,8 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions ]); const latestRunCreatedAtByKey = new Map(); for (const newerRun of newerRuns) { - const newerRunKey = `${newerRun.agentId}:${readRunIssueId(newerRun.contextSnapshot) ?? ""}`; + const newerRunIssueId = readRunIssueId({ issueId: newerRun.runIssueId, taskId: newerRun.runTaskId }); + const newerRunKey = `${newerRun.agentId}:${newerRunIssueId ?? ""}`; const latestCreatedAt = latestRunCreatedAtByKey.get(newerRunKey); if (!latestCreatedAt || newerRun.createdAt > latestCreatedAt) { latestRunCreatedAtByKey.set(newerRunKey, newerRun.createdAt); diff --git a/server/src/services/dashboard.ts b/server/src/services/dashboard.ts index 20d47dbfeff..70c1b52c84b 100644 --- a/server/src/services/dashboard.ts +++ b/server/src/services/dashboard.ts @@ -102,6 +102,10 @@ export function dashboardService(db: Db) { // restart-killed run whose retry succeeded is pulled out of the headline // failed count. error_code is carried through so a failure spike can be // attributed to an error class (e.g. process_lost, provider_quota). + // Both recursive arms are bounded to the chart window: a retry is always + // created after the run it retries, so ancestors of an out-of-window + // child are themselves out of window and invisible to the membership + // test below. Unbounded, the seed walks every run the company ever had. const runActivityRows = (await db.execute(sql` WITH RECURSIVE recovered_runs(id) AS ( SELECT parent.id @@ -109,11 +113,13 @@ export function dashboardService(db: Db) { JOIN ${heartbeatRuns} AS parent ON parent.id = child.retry_of_run_id WHERE child.company_id = ${companyId} AND child.status = 'succeeded' + AND child.created_at >= ${runActivityStart.toISOString()}::timestamptz UNION SELECT parent.id FROM recovered_runs rr JOIN ${heartbeatRuns} AS child ON child.id = rr.id JOIN ${heartbeatRuns} AS parent ON parent.id = child.retry_of_run_id + WHERE child.created_at >= ${runActivityStart.toISOString()}::timestamptz ) SELECT to_char(run.created_at AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS date, diff --git a/server/src/services/productivity-review.ts b/server/src/services/productivity-review.ts index e694f2e222f..199b122421c 100644 --- a/server/src/services/productivity-review.ts +++ b/server/src/services/productivity-review.ts @@ -44,6 +44,12 @@ export const PRODUCTIVITY_REVIEW_REFRESH_COMMENT_PREFIX = "Productivity review e type IssueRow = typeof issues.$inferSelect; type AgentRow = typeof agents.$inferSelect; type HeartbeatRunRow = typeof heartbeatRuns.$inferSelect; +// Evidence only reads these run fields; selecting the full row detoasts +// result_json/context_snapshot for up to MAX_RUNS_FOR_STREAK runs per issue. +type ProductivityRunSample = Pick< + HeartbeatRunRow, + "id" | "agentId" | "status" | "livenessState" | "createdAt" | "nextAction" | "usageJson" +>; type ProductivityReviewTrigger = "no_comment_streak" | "long_active_duration" | "high_churn"; type ProductivityReviewThresholds = { @@ -74,7 +80,7 @@ type ProductivityReviewEvidence = { commentCountLastHour: number; commentCountLastSixHours: number; elapsedMs: number | null; - latestRuns: HeartbeatRunRow[]; + latestRuns: ProductivityRunSample[]; latestComments: Array; costCents: number; usageSamples: Array<{ runId: string; usageJson: Record | null }>; @@ -447,7 +453,15 @@ export function productivityReviewService(db: Db, deps?: { enqueueWakeup?: Enque const sixHoursAgo = new Date(now.getTime() - 6 * 60 * 60 * 1000); const latestRuns = await db - .select() + .select({ + id: heartbeatRuns.id, + agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, + livenessState: heartbeatRuns.livenessState, + createdAt: heartbeatRuns.createdAt, + nextAction: heartbeatRuns.nextAction, + usageJson: heartbeatRuns.usageJson, + }) .from(heartbeatRuns) .where( and( From f258b34bbdb7ac318342e2954c0df4c66f3d1076 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 6 Aug 2026 12:13:14 -0500 Subject: [PATCH 004/311] fix(ui): unify cloud-managed sign-out (#10994) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip can run as a self-hosted app or as a Cloud-managed tenant. > - These modes need different sign-out sequences because Cloud owns three sessions. > - Several visible controls implemented sign-out separately and could choose different paths. > - This pull request adds one Cloud-aware sign-out action and moves every visible control to it. > - The benefit is one safe sign-out path in Cloud and unchanged local sign-out in self-hosted deployments. ## Linked Issues or Issue Description Refs #2073. That older PR adds a separate company-settings sign-out surface. This change centralizes the existing account, company, and instance-settings surfaces and preserves the self-hosted behavior described there. **What happened?** Visible sign-out controls used separate implementations. A Cloud-managed control could call the app-local sign-out endpoint and open the local auth page. That path did not enter the Cloud-owned logout sequence for the tenant, Cloud, and identity sessions. **Expected behavior** Every visible sign-out control must use one action. Cloud-managed instances must navigate the top-level window to the same-origin `/cloud/logout` route without a local sign-out call first. Authenticated self-hosted instances must keep the local sign-out API and cache invalidation behavior. **Steps to reproduce** 1. Open a Cloud-managed tenant. 2. Use the account menu, company menu, or instance-settings sign-out control. 3. Observe that independently implemented controls can enter different sign-out paths. **Paperclip version or commit** The problem reproduces at `656ecfa585b31938e2685ffab3db22e794474803`. **Deployment mode** Cloud-managed tenant built from source. The regression tests also cover authenticated self-hosted mode. ## What Changed - Added `useSignOut` as the shared Cloud-aware sign-out action. - Navigated Cloud-managed sessions to `/cloud/logout` exactly once without calling local auth first. - Preserved local API sign-out and cache invalidation for authenticated self-hosted sessions. - Migrated the account menu, company menu, and instance general settings to the shared action. - Added focused tests for mode selection, menu closure, pending state, failure state, and settings behavior. ## Verification - `pnpm exec vitest run ui/src/hooks/useSignOut.test.tsx ui/src/components/SidebarAccountMenu.test.tsx ui/src/components/SidebarCompanyMenu.test.tsx ui/src/pages/InstanceGeneralSettings.test.tsx` β€” 26 tests passed. - `pnpm --filter @paperclipai/ui typecheck` β€” passed. - `pnpm check:token-gates` β€” passed. - `git diff --check origin/master..HEAD` β€” passed. ## Risks - Low risk. The change centralizes existing behavior and adds no schema, API, telemetry, or style-token changes. - Cloud mode depends on the existing health decision. Tests pin both mode branches. - The change does not alter Fetch Metadata, CSRF, cookie, prefetch, or return-URL protections owned by the Cloud logout route. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5. The runtime does not expose a more specific model ID or context-window size. The agent used high-reasoning mode, shell tools, and API tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ui/src/components/SidebarAccountMenu.test.tsx | 31 ++- ui/src/components/SidebarAccountMenu.tsx | 22 +- ui/src/components/SidebarCompanyMenu.test.tsx | 28 +++ ui/src/components/SidebarCompanyMenu.tsx | 14 +- ui/src/hooks/useSignOut.test.tsx | 123 +++++++++++ ui/src/hooks/useSignOut.ts | 45 ++++ ui/src/pages/InstanceGeneralSettings.test.tsx | 196 ++++++++++++++++++ ui/src/pages/InstanceGeneralSettings.tsx | 46 ++-- 8 files changed, 453 insertions(+), 52 deletions(-) create mode 100644 ui/src/hooks/useSignOut.test.tsx create mode 100644 ui/src/hooks/useSignOut.ts create mode 100644 ui/src/pages/InstanceGeneralSettings.test.tsx diff --git a/ui/src/components/SidebarAccountMenu.test.tsx b/ui/src/components/SidebarAccountMenu.test.tsx index f596c625d1a..de2da8d8eae 100644 --- a/ui/src/components/SidebarAccountMenu.test.tsx +++ b/ui/src/components/SidebarAccountMenu.test.tsx @@ -177,6 +177,7 @@ describe("SidebarAccountMenu", () => { it("navigates cloud-managed sign-out through the harness without calling local auth", async () => { const root = createRoot(container); + const onOpenChange = vi.fn(); const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } }, }); @@ -194,7 +195,11 @@ describe("SidebarAccountMenu", () => { await act(async () => { root.render( - + , ); }); @@ -209,7 +214,31 @@ describe("SidebarAccountMenu", () => { await flushReact(); expect(mockAuthApi.signOut).not.toHaveBeenCalled(); + expect(mockNavigateTopLevel).toHaveBeenCalledOnce(); expect(mockNavigateTopLevel).toHaveBeenCalledWith("/cloud/logout"); + expect(onOpenChange).toHaveBeenCalledWith(false); + + await act(async () => { + root.unmount(); + }); + }); + + it("keeps sign-out hidden outside authenticated deployment mode", async () => { + const root = createRoot(container); + const queryClient = new QueryClient({ + defaultOptions: { queries: { retry: false } }, + }); + + await act(async () => { + root.render( + + + , + ); + }); + await flushReact(); + + expect(document.body.textContent).not.toContain("Sign out"); await act(async () => { root.unmount(); diff --git a/ui/src/components/SidebarAccountMenu.tsx b/ui/src/components/SidebarAccountMenu.tsx index 8713b12de1a..be58276d8a6 100644 --- a/ui/src/components/SidebarAccountMenu.tsx +++ b/ui/src/components/SidebarAccountMenu.tsx @@ -1,5 +1,5 @@ import { useState } from "react"; -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { useQuery } from "@tanstack/react-query"; import { BookOpen, LogOut, @@ -12,8 +12,7 @@ import type { DeploymentMode, ServerGitInfo } from "@paperclipai/shared"; import { Link } from "@/lib/router"; import { authApi } from "@/api/auth"; import { queryKeys } from "@/lib/queryKeys"; -import { navigateTopLevel } from "@/lib/browserNavigation"; -import { useCloudInstance } from "@/hooks/useCloudInstance"; +import { useSignOut } from "@/hooks/useSignOut"; import { useSidebar } from "../context/SidebarContext"; import { Popover, PopoverContent, PopoverTrigger } from "@/components/ui/popover"; import { Avatar, AvatarFallback, AvatarImage } from "@/components/ui/avatar"; @@ -26,7 +25,6 @@ const PROFILE_SETTINGS_PATH = "/company/settings/instance/profile"; const DOCS_URL = "https://docs.paperclip.ing/"; const FEEDBACK_URL = "https://paperclip.ing/feedback"; const SOURCE_REPOSITORY_URL = "https://github.com/paperclipai/paperclip"; -const MANAGED_SIGN_OUT_PATH = "/cloud/logout"; const SOURCE_VERSION_RE = /\+\d+\.git\.([0-9a-f]{7,40})(?:\.dirty)?$/i; interface SidebarAccountMenuProps { @@ -120,8 +118,6 @@ export function SidebarAccountMenu({ version, }: SidebarAccountMenuProps) { const [internalOpen, setInternalOpen] = useState(false); - const queryClient = useQueryClient(); - const isCloud = Boolean(useCloudInstance()); const { isMobile, setSidebarOpen, collapsed, peeking } = useSidebar(); const rail = collapsed && !peeking; const open = controlledOpen ?? internalOpen; @@ -132,14 +128,7 @@ export function SidebarAccountMenu({ retry: false, }); - const signOutMutation = useMutation({ - mutationFn: () => authApi.signOut(), - onSuccess: async () => { - setOpen(false); - await queryClient.invalidateQueries({ queryKey: queryKeys.auth.session }); - await queryClient.invalidateQueries({ queryKey: queryKeys.health }); - }, - }); + const signOutMutation = useSignOut({ onSignedOut: closeNavigationChrome }); const displayName = session?.user.name?.trim() || "Board"; const secondaryLabel = @@ -160,11 +149,6 @@ export function SidebarAccountMenu({ } function handleSignOut() { - if (isCloud) { - closeNavigationChrome(); - navigateTopLevel(MANAGED_SIGN_OUT_PATH); - return; - } signOutMutation.mutate(); } diff --git a/ui/src/components/SidebarCompanyMenu.test.tsx b/ui/src/components/SidebarCompanyMenu.test.tsx index c1c925d25e9..27b2a7a583c 100644 --- a/ui/src/components/SidebarCompanyMenu.test.tsx +++ b/ui/src/components/SidebarCompanyMenu.test.tsx @@ -296,6 +296,9 @@ describe("SidebarCompanyMenu", () => { await flushReact(); expect(mockAuthApi.signOut).toHaveBeenCalledTimes(1); + expect(mockNavigateTopLevel).not.toHaveBeenCalled(); + expect(queryClient.getQueryState(queryKeys.health)?.isInvalidated).toBe(true); + expect(document.body.textContent).not.toContain("Switch company"); act(() => { root.unmount(); @@ -456,6 +459,31 @@ describe("SidebarCompanyMenu", () => { }); describe("in Paperclip Cloud", () => { + it("closes the menu and enters the Cloud logout flow without local sign-out", async () => { + const { root } = renderMenu({ cloud: true }); + await flushReact(); + await flushReact(); + await openMenu("Open Acme Labs organization switcher"); + + const signOutItem = Array.from(document.body.querySelectorAll('[data-slot="dropdown-menu-item"]')) + .find((element) => element.textContent?.includes("Sign out")); + expect(signOutItem).toBeTruthy(); + + act(() => { + signOutItem?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + + expect(mockAuthApi.signOut).not.toHaveBeenCalled(); + expect(mockNavigateTopLevel).toHaveBeenCalledOnce(); + expect(mockNavigateTopLevel).toHaveBeenCalledWith("/cloud/logout"); + expect(document.body.textContent).not.toContain("Switch organization"); + + act(() => { + root.unmount(); + }); + }); + it("switches organizations instead of companies", async () => { const { root } = renderMenu({ cloud: true }); await flushReact(); diff --git a/ui/src/components/SidebarCompanyMenu.tsx b/ui/src/components/SidebarCompanyMenu.tsx index 928b6e0bac5..714c050fae3 100644 --- a/ui/src/components/SidebarCompanyMenu.tsx +++ b/ui/src/components/SidebarCompanyMenu.tsx @@ -1,5 +1,5 @@ import { useCallback, useMemo, useState } from "react"; -import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query"; +import { useQuery } from "@tanstack/react-query"; import { Check, ChevronsUpDown, @@ -37,6 +37,7 @@ import { useCompany } from "@/context/CompanyContext"; import { useDialogActions } from "@/context/DialogContext"; import { useCloudInstance } from "@/hooks/useCloudInstance"; import { useCompanyOrder } from "@/hooks/useCompanyOrder"; +import { useSignOut } from "@/hooks/useSignOut"; import { navigateTopLevel } from "@/lib/browserNavigation"; import { cloudStackCreateUrl, cloudStackEnterUrl } from "@/lib/cloudLinks"; import { queryKeys } from "@/lib/queryKeys"; @@ -198,7 +199,6 @@ function SortableCompanyItem({ export function SidebarCompanyMenu({ open: controlledOpen, onOpenChange }: SidebarCompanyMenuProps = {}) { const [internalOpen, setInternalOpen] = useState(false); const [isEditingOrder, setIsEditingOrder] = useState(false); - const queryClient = useQueryClient(); const { companies, selectedCompany, setSelectedCompanyId } = useCompany(); const { openOnboarding } = useDialogActions(); const { isMobile, setSidebarOpen, collapsed, peeking } = useSidebar(); @@ -257,15 +257,7 @@ export function SidebarCompanyMenu({ open: controlledOpen, onOpenChange }: Sideb ? currentStack?.displayName ?? cloud?.stackDisplayName ?? cloud?.stackSlug ?? null : selectedCompany?.name ?? null; - const signOutMutation = useMutation({ - mutationFn: () => authApi.signOut(), - onSuccess: async () => { - setOpen(false); - if (isMobile) setSidebarOpen(false); - await queryClient.invalidateQueries({ queryKey: queryKeys.auth.session }); - await queryClient.invalidateQueries({ queryKey: queryKeys.health }); - }, - }); + const signOutMutation = useSignOut({ onSignedOut: closeNavigationChrome }); function handleOpenChange(nextOpen: boolean) { if (!nextOpen) setIsEditingOrder(false); diff --git a/ui/src/hooks/useSignOut.test.tsx b/ui/src/hooks/useSignOut.test.tsx new file mode 100644 index 00000000000..0745ca1187e --- /dev/null +++ b/ui/src/hooks/useSignOut.test.tsx @@ -0,0 +1,123 @@ +// @vitest-environment jsdom + +import { flushSync } from "react-dom"; +import { createRoot } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { queryKeys } from "@/lib/queryKeys"; +import { useSignOut } from "./useSignOut"; + +const mockAuthApi = vi.hoisted(() => ({ signOut: vi.fn() })); +const mockNavigateTopLevel = vi.hoisted(() => vi.fn()); + +vi.mock("@/api/auth", () => ({ authApi: mockAuthApi })); +vi.mock("@/lib/browserNavigation", () => ({ navigateTopLevel: mockNavigateTopLevel })); + +// eslint-disable-next-line @typescript-eslint/no-explicit-any +(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true; + +let captured: ReturnType | null = null; + +function Harness({ onSignedOut }: { onSignedOut?: () => void }) { + captured = useSignOut({ onSignedOut }); + return null; +} + +describe("useSignOut", () => { + let container: HTMLDivElement; + let queryClient: QueryClient; + + beforeEach(() => { + captured = null; + container = document.createElement("div"); + document.body.appendChild(container); + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + }); + + afterEach(() => { + queryClient.clear(); + container.remove(); + vi.clearAllMocks(); + }); + + function renderHarness(onSignedOut?: () => void) { + const root = createRoot(container); + flushSync(() => { + root.render( + + + , + ); + }); + return root; + } + + it("closes caller chrome and navigates through Cloud without local sign-out", async () => { + const onSignedOut = vi.fn(); + queryClient.setQueryData(queryKeys.health, { + status: "ok", + cloud: { + managed: true, + managedBy: "paperclip-cloud", + stackSlug: "acme", + cloudBaseUrl: "https://cloud.example.test", + }, + }); + const root = renderHarness(onSignedOut); + + flushSync(() => captured?.mutate()); + + await vi.waitFor(() => expect(mockNavigateTopLevel).toHaveBeenCalledOnce()); + expect(mockNavigateTopLevel).toHaveBeenCalledWith("/cloud/logout"); + expect(onSignedOut).toHaveBeenCalledOnce(); + expect(mockAuthApi.signOut).not.toHaveBeenCalled(); + + flushSync(() => root.unmount()); + }); + + it("keeps self-hosted sign-out pending until the local request finishes, then invalidates caches", async () => { + let resolveSignOut: (() => void) | undefined; + mockAuthApi.signOut.mockImplementation(() => new Promise((resolve) => { + resolveSignOut = resolve; + })); + queryClient.setQueryData(queryKeys.health, { status: "ok", deploymentMode: "authenticated" }); + queryClient.setQueryData(queryKeys.auth.session, { session: { id: "session-1" } }); + const onSignedOut = vi.fn(); + const root = renderHarness(onSignedOut); + + flushSync(() => captured?.mutate()); + + await vi.waitFor(() => expect(captured?.isPending).toBe(true)); + expect(mockAuthApi.signOut).toHaveBeenCalledOnce(); + expect(mockNavigateTopLevel).not.toHaveBeenCalled(); + expect(onSignedOut).not.toHaveBeenCalled(); + + resolveSignOut?.(); + await vi.waitFor(() => expect(captured?.isPending).toBe(false)); + + expect(captured?.error).toBeNull(); + expect(onSignedOut).toHaveBeenCalledOnce(); + expect(queryClient.getQueryState(queryKeys.auth.session)?.isInvalidated).toBe(true); + expect(queryClient.getQueryState(queryKeys.health)?.isInvalidated).toBe(true); + + flushSync(() => root.unmount()); + }); + + it("exposes a stable error without closing chrome or invalidating caches", async () => { + mockAuthApi.signOut.mockRejectedValue(new Error("Sign-out request failed")); + queryClient.setQueryData(queryKeys.health, { status: "ok", deploymentMode: "authenticated" }); + queryClient.setQueryData(queryKeys.auth.session, { session: { id: "session-1" } }); + const onSignedOut = vi.fn(); + const root = renderHarness(onSignedOut); + + flushSync(() => captured?.mutate()); + + await vi.waitFor(() => expect(captured?.error?.message).toBe("Sign-out request failed")); + expect(captured?.isPending).toBe(false); + expect(onSignedOut).not.toHaveBeenCalled(); + expect(queryClient.getQueryState(queryKeys.auth.session)?.isInvalidated).toBe(false); + expect(queryClient.getQueryState(queryKeys.health)?.isInvalidated).toBe(false); + + flushSync(() => root.unmount()); + }); +}); diff --git a/ui/src/hooks/useSignOut.ts b/ui/src/hooks/useSignOut.ts new file mode 100644 index 00000000000..0d91d66a98f --- /dev/null +++ b/ui/src/hooks/useSignOut.ts @@ -0,0 +1,45 @@ +import { useMutation, useQueryClient } from "@tanstack/react-query"; +import { authApi } from "@/api/auth"; +import { navigateTopLevel } from "@/lib/browserNavigation"; +import { queryKeys } from "@/lib/queryKeys"; +import { useCloudInstance } from "./useCloudInstance"; + +const CLOUD_SIGN_OUT_PATH = "/cloud/logout"; + +interface UseSignOutOptions { + onSignedOut?: () => void; +} + +/** + * Owns the app-wide sign-out decision. + * + * Cloud-managed tenants must enter the harness-owned logout sequence without + * first clearing the tenant session. Authenticated self-hosted instances keep + * the local API flow and invalidate the auth-dependent caches afterward. + */ +export function useSignOut({ onSignedOut }: UseSignOutOptions = {}) { + const cloud = useCloudInstance(); + const queryClient = useQueryClient(); + + return useMutation({ + mutationFn: async () => { + if (cloud) { + onSignedOut?.(); + navigateTopLevel(CLOUD_SIGN_OUT_PATH); + return "cloud" as const; + } + + await authApi.signOut(); + return "self-hosted" as const; + }, + onSuccess: async (target) => { + if (target === "cloud") return; + + onSignedOut?.(); + await Promise.all([ + queryClient.invalidateQueries({ queryKey: queryKeys.auth.session }), + queryClient.invalidateQueries({ queryKey: queryKeys.health }), + ]); + }, + }); +} diff --git a/ui/src/pages/InstanceGeneralSettings.test.tsx b/ui/src/pages/InstanceGeneralSettings.test.tsx new file mode 100644 index 00000000000..9eff02c7a1b --- /dev/null +++ b/ui/src/pages/InstanceGeneralSettings.test.tsx @@ -0,0 +1,196 @@ +// @vitest-environment jsdom + +import { flushSync } from "react-dom"; +import { createRoot, type Root } from "react-dom/client"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { queryKeys } from "@/lib/queryKeys"; +import { InstanceGeneralSettings } from "./InstanceGeneralSettings"; + +const mockAuthApi = vi.hoisted(() => ({ signOut: vi.fn() })); +const mockHealthApi = vi.hoisted(() => ({ get: vi.fn() })); +const mockInstanceSettingsApi = vi.hoisted(() => ({ + getGeneral: vi.fn(), + updateGeneral: vi.fn(), +})); +const mockNavigateTopLevel = vi.hoisted(() => vi.fn()); + +vi.mock("@/api/auth", () => ({ authApi: mockAuthApi })); +vi.mock("@/api/health", () => ({ healthApi: mockHealthApi })); +vi.mock("@/api/instanceSettings", () => ({ instanceSettingsApi: mockInstanceSettingsApi })); +vi.mock("@/lib/browserNavigation", () => ({ navigateTopLevel: mockNavigateTopLevel })); +vi.mock("../context/BreadcrumbContext", () => ({ + useBreadcrumbs: () => ({ setBreadcrumbs: vi.fn() }), +})); + +// eslint-disable-next-line @typescript-eslint/no-explicit-any +(globalThis as any).IS_REACT_ACT_ENVIRONMENT = true; + +const SELF_HOSTED_HEALTH = { + status: "ok" as const, + deploymentMode: "authenticated" as const, + deploymentExposure: "private" as const, + authReady: true, + bootstrapStatus: "ready" as const, + bootstrapInviteActive: false, +}; + +const CLOUD_HEALTH = { + ...SELF_HOSTED_HEALTH, + cloud: { + managed: true as const, + managedBy: "paperclip-cloud" as const, + stackSlug: "acme", + cloudBaseUrl: "https://cloud.example.test", + }, +}; + +describe("InstanceGeneralSettings sign-out", () => { + let container: HTMLDivElement; + let root: Root | null; + let queryClient: QueryClient; + + beforeEach(() => { + container = document.createElement("div"); + document.body.appendChild(container); + root = null; + queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + mockInstanceSettingsApi.getGeneral.mockResolvedValue({ + censorUsernameInLogs: false, + keyboardShortcuts: false, + feedbackDataSharingPreference: "not_allowed", + backupRetention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }, + }); + mockInstanceSettingsApi.updateGeneral.mockResolvedValue(undefined); + mockAuthApi.signOut.mockResolvedValue({ success: true }); + }); + + afterEach(() => { + flushSync(() => root?.unmount()); + queryClient.clear(); + container.remove(); + vi.clearAllMocks(); + }); + + async function renderPage(health: typeof SELF_HOSTED_HEALTH | typeof CLOUD_HEALTH) { + mockHealthApi.get.mockResolvedValue(health); + queryClient.setQueryData(queryKeys.health, health); + root = createRoot(container); + flushSync(() => { + root?.render( + + + , + ); + }); + await vi.waitFor(() => expect(container.textContent).toContain("Deployment and auth")); + } + + function signOutButton() { + return Array.from(container.querySelectorAll("button")) + .find((button) => button.textContent?.trim() === "Sign out"); + } + + it("uses the Cloud-managed top-level logout without calling local auth", async () => { + await renderPage(CLOUD_HEALTH); + + flushSync(() => signOutButton()?.click()); + + await vi.waitFor(() => expect(mockNavigateTopLevel).toHaveBeenCalledOnce()); + expect(mockNavigateTopLevel).toHaveBeenCalledWith("/cloud/logout"); + expect(mockAuthApi.signOut).not.toHaveBeenCalled(); + }); + + it("keeps authenticated self-hosted sign-out local and invalidates auth caches", async () => { + const invalidateQueries = vi.spyOn(queryClient, "invalidateQueries"); + await renderPage(SELF_HOSTED_HEALTH); + + flushSync(() => signOutButton()?.click()); + + await vi.waitFor(() => expect(mockAuthApi.signOut).toHaveBeenCalledOnce()); + await vi.waitFor(() => expect(invalidateQueries).toHaveBeenCalledWith({ + queryKey: queryKeys.auth.session, + })); + expect(invalidateQueries).toHaveBeenCalledWith({ queryKey: queryKeys.health }); + expect(mockNavigateTopLevel).not.toHaveBeenCalled(); + }); + + it("shows a current sign-out failure instead of a stale settings error", async () => { + mockInstanceSettingsApi.updateGeneral.mockRejectedValue(new Error("Settings update failed")); + mockAuthApi.signOut.mockRejectedValue(new Error("Sign-out request failed")); + await renderPage(SELF_HOSTED_HEALTH); + + const keyboardToggle = container.querySelector( + '[aria-label="Toggle keyboard shortcuts"]', + ); + flushSync(() => keyboardToggle?.click()); + await vi.waitFor(() => expect(container.textContent).toContain("Settings update failed")); + + flushSync(() => signOutButton()?.click()); + + await vi.waitFor(() => expect(container.textContent).toContain("Sign-out request failed")); + expect(container.textContent).not.toContain("Settings update failed"); + }); + + it("clears a stale sign-out failure after a settings update succeeds", async () => { + mockAuthApi.signOut.mockRejectedValue(new Error("Sign-out request failed")); + await renderPage(SELF_HOSTED_HEALTH); + + flushSync(() => signOutButton()?.click()); + await vi.waitFor(() => expect(container.textContent).toContain("Sign-out request failed")); + + const keyboardToggle = container.querySelector( + '[aria-label="Toggle keyboard shortcuts"]', + ); + flushSync(() => keyboardToggle?.click()); + + await vi.waitFor(() => expect(mockInstanceSettingsApi.updateGeneral).toHaveBeenCalledOnce()); + await vi.waitFor(() => expect(container.textContent).not.toContain("Sign-out request failed")); + }); + + it("disables settings changes while sign-out is pending", async () => { + let resolveSignOut: ((result: { success: boolean }) => void) | undefined; + mockAuthApi.signOut.mockImplementation( + () => new Promise<{ success: boolean }>((resolve) => { + resolveSignOut = resolve; + }), + ); + await renderPage(SELF_HOSTED_HEALTH); + + const keyboardToggle = container.querySelector( + '[aria-label="Toggle keyboard shortcuts"]', + ); + flushSync(() => signOutButton()?.click()); + await vi.waitFor(() => expect(mockAuthApi.signOut).toHaveBeenCalledOnce()); + + expect(keyboardToggle?.disabled).toBe(true); + flushSync(() => keyboardToggle?.click()); + expect(mockInstanceSettingsApi.updateGeneral).not.toHaveBeenCalled(); + + resolveSignOut?.({ success: true }); + await vi.waitFor(() => expect(keyboardToggle?.disabled).toBe(false)); + }); + + it("disables sign-out while a settings update is pending", async () => { + let resolveSettings: (() => void) | undefined; + mockInstanceSettingsApi.updateGeneral.mockImplementation( + () => new Promise((resolve) => { + resolveSettings = resolve; + }), + ); + await renderPage(SELF_HOSTED_HEALTH); + + const keyboardToggle = container.querySelector( + '[aria-label="Toggle keyboard shortcuts"]', + ); + flushSync(() => keyboardToggle?.click()); + await vi.waitFor(() => expect(mockInstanceSettingsApi.updateGeneral).toHaveBeenCalledOnce()); + + expect(signOutButton()?.disabled).toBe(true); + flushSync(() => signOutButton()?.click()); + expect(mockAuthApi.signOut).not.toHaveBeenCalled(); + + resolveSettings?.(); + await vi.waitFor(() => expect(signOutButton()?.disabled).toBe(false)); + }); +}); diff --git a/ui/src/pages/InstanceGeneralSettings.tsx b/ui/src/pages/InstanceGeneralSettings.tsx index 3737e95c04e..1580f279ea8 100644 --- a/ui/src/pages/InstanceGeneralSettings.tsx +++ b/ui/src/pages/InstanceGeneralSettings.tsx @@ -8,7 +8,6 @@ import { DEFAULT_BACKUP_RETENTION, } from "@paperclipai/shared"; import { LogOut, SlidersHorizontal } from "lucide-react"; -import { authApi } from "@/api/auth"; import { healthApi } from "@/api/health"; import { instanceSettingsApi } from "@/api/instanceSettings"; import { ModeBadge } from "@/components/access/ModeBadge"; @@ -18,6 +17,7 @@ import { useBreadcrumbs } from "../context/BreadcrumbContext"; import { queryKeys } from "../lib/queryKeys"; import { ToggleSwitch } from "@/components/ui/toggle-switch"; import { cn } from "../lib/utils"; +import { useSignOut } from "@/hooks/useSignOut"; const FEEDBACK_TERMS_URL = import.meta.env.VITE_FEEDBACK_TERMS_URL?.trim() || "https://paperclip.ing/tos"; @@ -26,16 +26,7 @@ export function InstanceGeneralSettings() { const queryClient = useQueryClient(); const [actionError, setActionError] = useState(null); - const signOutMutation = useMutation({ - mutationFn: () => authApi.signOut(), - onSuccess: async () => { - await queryClient.invalidateQueries({ queryKey: queryKeys.auth.session }); - await queryClient.invalidateQueries({ queryKey: queryKeys.health }); - }, - onError: (error) => { - setActionError(error instanceof Error ? error.message : "Failed to sign out."); - }, - }); + const signOutMutation = useSignOut(); useEffect(() => { setBreadcrumbs([ @@ -57,8 +48,13 @@ export function InstanceGeneralSettings() { const updateGeneralMutation = useMutation({ mutationFn: instanceSettingsApi.updateGeneral, + onMutate: () => { + setActionError(null); + signOutMutation.reset(); + }, onSuccess: async () => { setActionError(null); + signOutMutation.reset(); await queryClient.invalidateQueries({ queryKey: queryKeys.instance.generalSettings }); }, onError: (error) => { @@ -84,6 +80,11 @@ export function InstanceGeneralSettings() { const keyboardShortcuts = generalQuery.data?.keyboardShortcuts === true; const feedbackDataSharingPreference = generalQuery.data?.feedbackDataSharingPreference ?? "prompt"; const backupRetention: BackupRetentionPolicy = generalQuery.data?.backupRetention ?? DEFAULT_BACKUP_RETENTION; + const visibleActionError = signOutMutation.error instanceof Error + ? signOutMutation.error.message + : signOutMutation.error + ? "Failed to sign out." + : actionError; return (
@@ -98,9 +99,9 @@ export function InstanceGeneralSettings() {

- {actionError && ( + {visibleActionError && (
- {actionError} + {visibleActionError}
)} @@ -150,7 +151,7 @@ export function InstanceGeneralSettings() { updateGeneralMutation.mutate({ censorUsernameInLogs: !censorUsernameInLogs })} - disabled={updateGeneralMutation.isPending} + disabled={updateGeneralMutation.isPending || signOutMutation.isPending} aria-label="Toggle username log censoring" /> @@ -168,7 +169,7 @@ export function InstanceGeneralSettings() { updateGeneralMutation.mutate({ keyboardShortcuts: !keyboardShortcuts })} - disabled={updateGeneralMutation.isPending} + disabled={updateGeneralMutation.isPending || signOutMutation.isPending} aria-label="Toggle keyboard shortcuts" /> @@ -194,7 +195,7 @@ export function InstanceGeneralSettings() { + {/* PAP-411: mobile priority section hidden behind SHOW_TASK_PRIORITY_UI. */} + {SHOW_TASK_PRIORITY_UI && (
Priority @@ -2211,6 +2216,7 @@ export function NewIssueDialog() { ))}
+ )}
+ {/* PAP-411: PriorityIcon showcase gated behind SHOW_TASK_PRIORITY_UI per board decision. */} + {SHOW_TASK_PRIORITY_UI && (
{["critical", "high", "medium", "low"].map((p) => ( @@ -651,6 +657,7 @@ export function DesignGuide() { Click the icon to change (current: {priority})
+ )}
@@ -1123,7 +1130,8 @@ export function DesignGuide() { leading={ <> - + {/* PAP-411: PriorityIcon hidden behind SHOW_TASK_PRIORITY_UI. */} + {SHOW_TASK_PRIORITY_UI && } } identifier="PAP-001" @@ -1136,7 +1144,7 @@ export function DesignGuide() { leading={ <> - + {SHOW_TASK_PRIORITY_UI && } } identifier="PAP-002" @@ -1149,7 +1157,7 @@ export function DesignGuide() { leading={ <> - + {SHOW_TASK_PRIORITY_UI && } } identifier="PAP-003" @@ -1161,7 +1169,7 @@ export function DesignGuide() { leading={ <> - + {SHOW_TASK_PRIORITY_UI && } } identifier="PAP-004" @@ -1249,7 +1257,10 @@ export function DesignGuide() { onClick={() => setFilters([ { key: "status", label: "Status", value: "Active" }, - { key: "priority", label: "Priority", value: "High" }, + // PAP-411: priority filter demo row suppressed while SHOW_TASK_PRIORITY_UI is off. + ...(SHOW_TASK_PRIORITY_UI + ? [{ key: "priority", label: "Priority", value: "High" } as FilterValue] + : []), ]) } > @@ -1429,10 +1440,13 @@ export function DesignGuide() { Status
-
- Priority - -
+ {/* PAP-411: priority metadata row hidden behind SHOW_TASK_PRIORITY_UI. */} + {SHOW_TASK_PRIORITY_UI && ( +
+ Priority + +
+ )}
Responsible
@@ -1500,14 +1514,15 @@ export function DesignGuide() { 2
+ {/* PAP-411: leading PriorityIcon hidden behind SHOW_TASK_PRIORITY_UI. */} } + leading={SHOW_TASK_PRIORITY_UI ? : undefined} identifier="PAP-101" title="Build agent heartbeat system" onClick={() => {}} /> } + leading={SHOW_TASK_PRIORITY_UI ? : undefined} identifier="PAP-102" title="Add cost tracking dashboard" onClick={() => {}} diff --git a/ui/src/pages/IssueDetail.test.tsx b/ui/src/pages/IssueDetail.test.tsx index fa0d1b5c4a8..8ef40448e7e 100644 --- a/ui/src/pages/IssueDetail.test.tsx +++ b/ui/src/pages/IssueDetail.test.tsx @@ -1108,7 +1108,7 @@ describe("IssueDetail", () => { expect(mockDecisionsApi.list).not.toHaveBeenCalled(); }); - it("updates status and priority from the task header controls", async () => { + it("updates status from the task header control and hides the priority control (PAP-411)", async () => { const issue = createIssue({ status: "todo", priority: "medium" }); mockIssuesApi.get.mockResolvedValue(issue); mockIssuesApi.update.mockImplementation(async (_issueId: string, data: Record) => ({ @@ -1133,7 +1133,9 @@ describe("IssueDetail", () => { 'button[aria-label="Change priority (current: medium)"]', ); expect(statusButton).not.toBeNull(); - expect(priorityButton).not.toBeNull(); + // PAP-411: priority UI is hidden behind SHOW_TASK_PRIORITY_UI (off), so the header + // priority control must not render. + expect(priorityButton).toBeNull(); await act(async () => { statusButton!.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true })); @@ -1141,13 +1143,10 @@ describe("IssueDetail", () => { await waitForAssertion(() => { expect(mockIssuesApi.update).toHaveBeenCalledWith(issue.identifier, { status: "done" }); }); - - await act(async () => { - priorityButton!.dispatchEvent(new MouseEvent("click", { bubbles: true, cancelable: true })); - }); - await waitForAssertion(() => { - expect(mockIssuesApi.update).toHaveBeenCalledWith(issue.identifier, { priority: "high" }); - }); + expect(mockIssuesApi.update).not.toHaveBeenCalledWith( + issue.identifier, + expect.objectContaining({ priority: expect.anything() }), + ); mockIssuesApi.update.mockReset(); }); diff --git a/ui/src/pages/IssueDetail.tsx b/ui/src/pages/IssueDetail.tsx index 483fb2860dc..1c632986e40 100644 --- a/ui/src/pages/IssueDetail.tsx +++ b/ui/src/pages/IssueDetail.tsx @@ -135,6 +135,7 @@ import { ArtifactFileChip } from "../components/ArtifactFileChip"; import { ScrollToBottom } from "../components/ScrollToBottom"; import { StatusIcon } from "../components/StatusIcon"; import { PriorityIcon } from "../components/PriorityIcon"; +import { SHOW_TASK_PRIORITY_UI } from "../lib/ui-flags"; import { ProductivityReviewBadge } from "../components/ProductivityReviewBadge"; import { Identity } from "../components/Identity"; import { PluginSlotMount, PluginSlotOutlet, usePluginSlots } from "@/plugins/slots"; @@ -706,7 +707,8 @@ function IssueDetailLoadingState({ {headerSeed ? ( <> - + {/* PAP-411: priority UI hidden behind SHOW_TASK_PRIORITY_UI. */} + {SHOW_TASK_PRIORITY_UI && } {identifier ? ( {identifier} ) : null} @@ -4195,10 +4197,13 @@ export function IssueDetail() { blockerAttention={issue.blockerAttention} onChange={(status) => updateIssue.mutate({ status })} /> - updateIssue.mutate({ priority })} - /> + {/* PAP-411: priority UI hidden behind SHOW_TASK_PRIORITY_UI. */} + {SHOW_TASK_PRIORITY_UI && ( + updateIssue.mutate({ priority })} + /> + )} {issue.identifier ?? issue.id.slice(0, 8)} {hasLiveRuns && ( From cfed36ea6b8b0f561435f729f0afcd9cd5585c4b Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 6 Aug 2026 17:38:00 -0700 Subject: [PATCH 007/311] feat(plugin-daytona): persistent session model with plain command dispatch (#10941) ## Thinking Path > - Paperclip is the control plane for autonomous AI companies > - One core subsystem runs agent work inside sandboxes > - The Daytona provider uses that path to run user commands > - The current one-shot model does not keep a shell alive across commands > - This pull request adds an opt-in persistent session model for Daytona > - The benefit is faster command dispatch with the same sandbox boundaries ## Linked Issues or Issue Description ### Subsystem affected Cross-cutting. This change touches `packages/adapters`, provider tests, span names, and sandbox command behavior. ### Problem or motivation The Daytona provider needs a persistent shell for repeated command dispatch. The old advisory wrapper path does not reach that goal. It also adds cost and removes the session speed gain. ### Proposed solution Add a `useSessions` driver flag. Keep it off by default. Open one Daytona session per lease when the flag is on. Send each user command into that session. Read stdout and stderr from the session logs endpoint. Run each command in a subshell so `exit` does not stop the shell. Remove the advisory `bwrap` wrapper path and its lease metadata. Add session setup and teardown spans. Keep a hard delete on teardown. ### Alternatives considered Keep the advisory `bwrap` wrapper. That path does not give a real persistent session. It also keeps extra command overhead. Keep a one-shot fallback for user commands. That would weaken the session model and hide a missing session case. ### Roadmap alignment This work fits the `Cloud / Sandbox agents` milestone in `ROADMAP.md`. It also supports the control plane goal of safe remote sandbox execution. ### Additional context The handoff verification reported `tsc --noEmit` clean and 119 Daytona unit tests passing. The handoff also reported a clean host span allowlist test and five expected commits on the branch. The security review gate remains required before merge. ## What Changed - Added an opt-in persistent session model for the Daytona sandbox provider. - Routed user commands through `executeSessionCommand` when sessions are enabled. - Removed the advisory `bwrap` command wrapper path and the lease metadata it used. - Added session lifecycle spans and span allowlist coverage. - Documented the leak bound in `DIRECTORY-CONSTRAINT-FINDINGS.md`. ## Verification - `tsc --noEmit` clean for the Daytona plugin, per handoff verification. - Daytona unit suite passes, with 119 tests, per handoff verification. - Host span allowlist test passes, per handoff verification. ## Risks - Persistent sessions can leak if teardown fails. - Session logs must keep stdout and stderr separate. - The flag stays off by default to limit rollout risk. ## Model Used OpenAI GPT-5, Codex, tool use enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../daytona/DIRECTORY-CONSTRAINT-FINDINGS.md | 68 ++ .../daytona/src/file-sync.ts | 2 +- .../daytona/src/plugin.test.ts | 918 +++++++++--------- .../sandbox-providers/daytona/src/plugin.ts | 538 +++++----- packages/plugins/sdk/src/protocol.ts | 14 + .../src/__tests__/environment-runtime.test.ts | 322 ++++++ .../plugin-host-services-span.test.ts | 15 + .../services/environment-run-orchestrator.ts | 7 + server/src/services/environment-runtime.ts | 111 ++- server/src/services/plugin-host-services.ts | 5 +- 10 files changed, 1295 insertions(+), 705 deletions(-) create mode 100644 packages/plugins/sandbox-providers/daytona/DIRECTORY-CONSTRAINT-FINDINGS.md diff --git a/packages/plugins/sandbox-providers/daytona/DIRECTORY-CONSTRAINT-FINDINGS.md b/packages/plugins/sandbox-providers/daytona/DIRECTORY-CONSTRAINT-FINDINGS.md new file mode 100644 index 00000000000..9ccabdb89f8 --- /dev/null +++ b/packages/plugins/sandbox-providers/daytona/DIRECTORY-CONSTRAINT-FINDINGS.md @@ -0,0 +1,68 @@ +# Directory-constraint (`bwrap`) findings for the Daytona session path + +## Summary + +The Daytona provider ran an advisory `bubblewrap` (`bwrap`) wrapper around each +user command. The wrapper gave an agent real-time feedback when the agent wrote +to a directory that the ephemeral sandbox does not keep. The wrapper was +advisory only. It added no security boundary; the ephemeral sandbox is the only +boundary. + +The provider now runs each user command plain, both on the persistent-session +path and on the one-shot fallback path. This note records why the provider drops +the directory-constraint goal for now, and what a future re-introduction needs. + +## Why the goal is dropped for now + +The session model runs one persistent shell per lease and feeds every command +into that shell. The plan aimed to enter the `bwrap` sandbox one time per session +and amortize its cost across every command in the session. A live validation +proved this aim is not reachable with the current Daytona session daemon. + +- An `exec`-replace of the session shell with `bwrap` stops the session from + running any later command. A side-channel file proved the inner shell never + ran the next command. A repeat without `--new-session` gave the same result, + so `--new-session` is not the cause. The `exec`-replace itself breaks the + session daemon's per-command delivery. +- A per-command `bwrap` wrap works inside a session (separated `stdout` and + `stderr`, correct exit codes, the session stays alive), but it costs about + 1.75 s per command. An un-wrapped session command takes about 0.29 s; a + `bwrap`-wrapped session command takes about 2.05 s. The wrap adds the full + cost back on every command, which removes the session model's speed win. + +The wrapper was advisory, not a security boundary. The provider already ran a +command plain when the `bwrap` capability probe failed, so the plain path is an +accepted, shipped behavior. Dropping the wrapper drops only the advisory +feedback; it does not change the command's privilege or isolation. The command +runs as the unprivileged sandbox user (`daytona`) in both cases. + +## Removal scope + +- The session dispatch runs the plain login-shell script, wrapped in a subshell + so a top-level `exit` cannot end the persistent session shell. +- The one-shot fallback (used when the session model is off) also runs the plain + login-shell script. +- The provider removed the per-command `bwrap` apply, the `bwrap` capability + probe, the `bwrap` command builder, and the `bwrapAvailable` / + `sandboxUsername` lease metadata that only the removed path used. +- The provider keeps the advisory writable-directory set, which a sync operation + records from each `access: "rw"` mapping. Nothing reads the set today. It stays + in place so a future isolation wrapper can consume it without a new sync + change. + +## What a future re-introduction needs + +- A persistent-namespace approach needs its own spike. Two candidates are: + - Enter one namespace per session and route each command into it with + `nsenter`. + - Feed each command to the namespace shell through a FIFO instead of an + `exec`-replace. +- Both candidates need a live test against the Daytona session daemon, because + the `exec`-replace model is proven unreachable and the per-command wrap is too + slow. +- The re-introduction must keep the session speed win. It must not pay the full + `bwrap` setup cost on every command. +- The writable-directory set is already tracked per lease, so a new wrapper can + bind those directories read-write without a sync change. +- The wrapper stays advisory. It must not become a security boundary; the + ephemeral sandbox stays the only boundary. diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index 3b91438138e..be9e51547da 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -43,7 +43,7 @@ const SPAN_STATUS_CODE_ERROR = 2; * their existing `*.wall_ms` attribute. A per-round-trip span omits it, so it * carries no `*.wall_ms` attribute and relies on the native span width. */ -async function withProviderSpan(input: { +export async function withProviderSpan(input: { name: string; wallMsAttr?: string; attributes?: Record; diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 4dd11030a83..7e3d04d9c9d 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -34,7 +34,6 @@ import plugin, { __resetDaytonaSandboxHandleCacheForTest, __getDaytonaWritableDirsForTest, __setDaytonaPluginContextForTest, - buildBwrapCommand, } from "./plugin.js"; import type { PluginContext } from "@paperclipai/plugin-sdk"; import manifest from "./manifest.js"; @@ -87,6 +86,15 @@ function createMockSandbox(overrides: { result: "bash", artifacts: { stdout: "bash" }, }), + // Session API (Daytona SDK 0.203.0). The exec hook opens one session per + // lease and dispatches every command into it. `executeSessionCommand` + // returns a `cmdId`; `getSessionCommand` reports the exit code; and + // `getSessionCommandLogs` returns separated `stdout` and `stderr`. + createSession: vi.fn().mockResolvedValue(undefined), + executeSessionCommand: vi.fn().mockResolvedValue({ cmdId: "cmd-1" }), + getSessionCommand: vi.fn().mockResolvedValue({ id: "cmd-1", command: "", exitCode: 0 }), + getSessionCommandLogs: vi.fn().mockResolvedValue({ stdout: "", stderr: "" }), + deleteSession: vi.fn().mockResolvedValue(undefined), }, }; } @@ -160,6 +168,7 @@ describe("Daytona sandbox provider plugin", () => { autoDeleteInterval: -1, reuseLease: true, archiveOnRelease: false, + useSessions: false, }, }); }); @@ -1075,6 +1084,457 @@ describe("Daytona sandbox provider plugin", () => { expect(warnSpy).toHaveBeenCalled(); }); + describe("session model lifecycle (per-lease session store)", () => { + // A recording plugin tracer that captures every provider span the session + // hooks open. It satisfies the structural plugin tracer contract. + const makeRecordingTracer = () => { + const spans: Array<{ + name: string; + attributes: Record; + status: { code: number; message?: string } | null; + ended: boolean; + }> = []; + const tracer = { + startSpan(name: string, options?: { attributes?: Record }) { + const span = { + name, + attributes: { ...(options?.attributes ?? {}) } as Record, + status: null as { code: number; message?: string } | null, + ended: false, + setAttribute(key: string, value: unknown) { + span.attributes[key] = value; + }, + setStatus(status: { code: number; message?: string }) { + span.status = status; + }, + end() { + span.ended = true; + }, + }; + spans.push(span); + return span; + }, + }; + return { tracer, spans }; + }; + + const sessionExecParams = (overrides: Record = {}) => ({ + driverKey: "daytona" as const, + companyId: "company-1", + environmentId: "env-1", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + lease: { providerLeaseId: "sandbox-123", metadata: {} }, + command: "printf", + args: ["hello"], + cwd: "/workspace", + timeoutMs: 1000, + ...overrides, + }); + + it("creates one session on the first execute and reuses it on the next", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + expect(sessionId).toMatch(/^paperclip-/); + }); + + it("opens one session when two first commands overlap", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + // Hold the first session create open, so the second first command reaches + // the session store before the first create resolves. Without a single + // flight guard, both commands would open a session for one lease and the + // first session id would leak. + let releaseCreate: () => void = () => {}; + const createGate = new Promise((resolve) => { + releaseCreate = resolve; + }); + sandbox.process.createSession.mockImplementationOnce(async () => { + await createGate; + }); + + const first = plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const second = plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + // Flush the pending microtasks, so both commands park on the held create. + await new Promise((resolve) => setTimeout(resolve, 0)); + releaseCreate(); + await Promise.all([first, second]); + + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + // Teardown deletes the same single session id, so no session leaks. + await plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + expect(sandbox.process.deleteSession).toHaveBeenCalledTimes(1); + expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); + }); + + it("opens no session when the session model is off", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.( + sessionExecParams({ config: { timeoutMs: 300000, reuseLease: false } }), + ); + + expect(sandbox.process.createSession).not.toHaveBeenCalled(); + expect(sandbox.process.executeCommand).toHaveBeenCalledTimes(1); + }); + + it("runs a bypassSession command one-shot and leaves the session closed", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + // The provision command runs before the run opens its trace root, so the + // host marks it `bypassSession`. The provider must not open the session for + // it, or the `session.setup` span loses its run parent. + await plugin.definition.onEnvironmentExecute?.( + sessionExecParams({ bypassSession: true }), + ); + + expect(sandbox.process.createSession).not.toHaveBeenCalled(); + expect(sandbox.process.executeCommand).toHaveBeenCalledTimes(1); + }); + + it("opens the session on the first in-run command after a bypassSession command", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + // A bypassSession command runs first (no session), then an in-run command + // opens the one session. The session-setup span then parents to the run + // trace, and every later in-run command reuses the same session. + await plugin.definition.onEnvironmentExecute?.( + sessionExecParams({ bypassSession: true }), + ); + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + expect(sessionId).toMatch(/^paperclip-/); + }); + + it("deletes the session and clears the store on release", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + + await plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + + expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); + }); + + it("deletes the session at destroy even when the sandbox delete throws", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.delete.mockRejectedValueOnce(new Error("delete failed")); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + + await expect( + plugin.definition.onEnvironmentDestroyLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }), + ).rejects.toThrow(/delete failed/); + + expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); + }); + + it("deletes the session on interactive-setup cancel", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + + await plugin.definition.onEnvironmentCancelInteractiveSetup?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + + expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); + }); + + it("logs loudly and does not throw when the session delete fails", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.process.deleteSession.mockRejectedValueOnce(new Error("session gone")); + mockGet.mockResolvedValue(sandbox); + const errorSpy = vi.spyOn(console, "error").mockImplementation(() => undefined); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + + await plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + + expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining(sessionId)); + // The rest of teardown still ran: the ephemeral sandbox was deleted. + expect(sandbox.delete).toHaveBeenCalledWith(300); + }); + + it("clears the session store after delete so no orphan id survives a second teardown", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + const releaseParams = { + driverKey: "daytona" as const, + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: true, useSessions: true }, + }; + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + await plugin.definition.onEnvironmentReleaseLease?.(releaseParams); + // The store is now clear. A second teardown finds no id and does not delete + // a session again, which proves no orphan id survived the first delete. + await plugin.definition.onEnvironmentReleaseLease?.(releaseParams); + + expect(sandbox.process.deleteSession).toHaveBeenCalledTimes(1); + }); + + it("clears the session store when resume restarts a stopped sandbox", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + // A stopped sandbox lost its session shell, so resume must clear the stale + // id and the next execute must open a fresh session. + const sandbox = createMockSandbox({ state: "stopped" }); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + + await plugin.definition.onEnvironmentResumeLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + leaseMetadata: { + remoteCwd: "/home/daytona/paperclip-workspace", + workspaceSentinel: { + path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json", + token: "token-1", + }, + }, + }); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + expect(sandbox.process.createSession).toHaveBeenCalledTimes(2); + }); + + it("keeps the session when resume runs on a still-running sandbox", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + // A running sandbox keeps its live session shell. Resume must not clear the + // stored id, or a later command opens a second session and teardown deletes + // only one, so the first session leaks until sandbox reaping. + const sandbox = createMockSandbox({ state: "started" }); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + const sessionId = sandbox.process.createSession.mock.calls[0]![0] as string; + + await plugin.definition.onEnvironmentResumeLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + leaseMetadata: { + remoteCwd: "/home/daytona/paperclip-workspace", + workspaceSentinel: { + path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json", + token: "token-1", + }, + }, + }); + + // The next command reuses the one live session, so no second session opens. + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + + // Teardown deletes the one session id, so no shell leaks. + await plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + expect(sandbox.process.deleteSession).toHaveBeenCalledTimes(1); + expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); + }); + + it("emits a session.setup span on create and a session.teardown span on delete", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + const { tracer, spans } = makeRecordingTracer(); + const restore = __setDaytonaPluginContextForTest({ tracer } as unknown as PluginContext); + try { + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + const setup = spans.find((span) => span.name === "session.setup"); + expect(setup).toBeDefined(); + expect(setup!.ended).toBe(true); + expect(setup!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); + + await plugin.definition.onEnvironmentReleaseLease?.({ + driverKey: "daytona", + companyId: "company-1", + environmentId: "env-1", + providerLeaseId: "sandbox-123", + config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, + }); + const teardown = spans.find((span) => span.name === "session.teardown"); + expect(teardown).toBeDefined(); + expect(teardown!.ended).toBe(true); + expect(teardown!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); + } finally { + restore(); + } + }); + + it("marks the session.setup span failed when the session create throws", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.process.createSession.mockRejectedValueOnce(new Error("create boom")); + mockGet.mockResolvedValue(sandbox); + const { tracer, spans } = makeRecordingTracer(); + const restore = __setDaytonaPluginContextForTest({ tracer } as unknown as PluginContext); + try { + await expect( + plugin.definition.onEnvironmentExecute?.(sessionExecParams()), + ).rejects.toThrow(/create boom/); + const setup = spans.find((span) => span.name === "session.setup"); + expect(setup).toBeDefined(); + expect(setup!.ended).toBe(true); + expect(setup!.status?.code).toBe(2); + } finally { + restore(); + } + }); + + it("dispatches commands into the session and returns separate stdout and stderr", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 7 }); + sandbox.process.getSessionCommandLogs.mockResolvedValue({ stdout: "out-here", stderr: "err-here" }); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + + // The command runs through the session, not the one-shot path. + expect(sandbox.process.executeSessionCommand).toHaveBeenCalledTimes(1); + expect(sandbox.process.executeCommand).not.toHaveBeenCalled(); + const [sid, req, timeoutArg] = sandbox.process.executeSessionCommand.mock.calls[0] as [ + string, + { command: string; runAsync?: boolean }, + number, + ]; + expect(sid).toMatch(/^paperclip-/); + expect(req.runAsync).toBe(true); + expect(timeoutArg).toBe(1); + // The built command carries the login-shell script and the user command. + expect(req.command).toMatch(/&& env .*'printf' 'hello'/); + // The session command runs plain: no bwrap wrapper and no su privilege drop. + expect(req.command).not.toContain("sudo -n bwrap"); + expect(req.command).not.toContain("su -s /bin/sh"); + // True separated streams come from the logs endpoint. + expect(result).toMatchObject({ exitCode: 7, timedOut: false, stdout: "out-here", stderr: "err-here" }); + expect(typeof (result!.metadata as Record)?.durationMs).toBe("number"); + }); + + it("wraps each user command in a subshell so a top-level exit cannot kill the session shell", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.( + sessionExecParams({ command: "exit", args: ["3"] }), + ); + + const [, req] = sandbox.process.executeSessionCommand.mock.calls[0] as [ + string, + { command: string }, + ]; + // The whole login-shell script (with the user `exit`) runs inside a + // subshell, so a top-level exit ends the subshell, not the session shell. + expect(req.command.trimStart()).toMatch(/^\(/); + expect(req.command.trimEnd()).toMatch(/\)$/); + expect(req.command).toMatch(/'exit' '3'/); + }); + + it("reuses the same session (one persistent shell) across commands", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + mockGet.mockResolvedValue(sandbox); + + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); + + expect(sandbox.process.createSession).toHaveBeenCalledTimes(1); + const firstSid = sandbox.process.executeSessionCommand.mock.calls[0]![0] as string; + const secondSid = sandbox.process.executeSessionCommand.mock.calls[1]![0] as string; + expect(firstSid).toBe(secondSid); + }); + + it("returns a session timeout when the command never reports an exit code", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + // The command stays running: the exit code never arrives. + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: undefined }); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.( + sessionExecParams({ timeoutMs: 1 }), + ); + + expect(result).toMatchObject({ exitCode: null, timedOut: true }); + expect(result!.stderr).toMatch(/timed out/); + }); + }); + it("executes commands one-shot and returns combined output via stdout", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); @@ -1233,7 +1693,7 @@ describe("Daytona sandbox provider plugin", () => { }); }); - it("wraps the command when the lease reports bwrap available and username known", async () => { + it("runs the one-shot command plain with no bwrap or su wrapper", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); mockGet.mockResolvedValue(sandbox); @@ -1243,226 +1703,17 @@ describe("Daytona sandbox provider plugin", () => { companyId: "company-1", environmentId: "env-1", config: { timeoutMs: 300000, reuseLease: false }, - lease: { - providerLeaseId: "sandbox-123", - metadata: { - remoteCwd: "/home/daytona/paperclip-workspace", - bwrapAvailable: true, - sandboxUsername: "daytona", - }, - }, + lease: { providerLeaseId: "sandbox-123", metadata: { remoteCwd: "/home/daytona/paperclip-workspace" } }, command: "printf", args: ["hello"], + cwd: "/workspace", timeoutMs: 1000, }); const [command] = sandbox.process.executeCommand.mock.calls[0] as [string]; - // The wrapper runs `sudo -n bwrap`, drops to the sandbox user via su, and - // binds the workspace directory read-write. - expect(command.startsWith("sudo -n bwrap")).toBe(true); - expect(command).toContain("su -s /bin/sh 'daytona'"); - expect(command).toContain( - "--bind-try '/home/daytona/paperclip-workspace' '/home/daytona/paperclip-workspace'", - ); - // The login-shell script still rides inside the wrapper through `su -c`. - expect(command).toContain("/etc/profile"); - }); - - it("binds collected rw sync directories", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const remoteCwd = "/home/daytona/paperclip-workspace"; - const collectedDir = `${remoteCwd}/data`; - const hostDir = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-bwrap-test-")); - const source = path.join(hostDir, "in-place.txt"); - await fs.writeFile(source, "bytes"); - const sandbox = createMockSandbox(); - mockGet.mockResolvedValue(sandbox); - - const scopeParams = { - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: false }, - }; - - // Record a read-write sync destination under the shared scope. The execute - // hook reads the same scope, so the wrapper must bind this directory too. - await plugin.definition.onEnvironmentSyncIn?.({ - ...scopeParams, - lease: { providerLeaseId: "sandbox-123", metadata: { remoteCwd } }, - operations: [ - { - operationId: "sync-op-rw", - files: [ - { - sourcePath: source, - targetPath: `${collectedDir}/in-place.txt`, - kind: "file" as const, - access: "rw" as const, - }, - ], - }, - ], - }); - sandbox.process.executeCommand.mockClear(); - - await plugin.definition.onEnvironmentExecute?.({ - ...scopeParams, - lease: { - providerLeaseId: "sandbox-123", - metadata: { remoteCwd, bwrapAvailable: true, sandboxUsername: "daytona" }, - }, - command: "printf", - args: ["hello"], - timeoutMs: 1000, - }); - - const [command] = sandbox.process.executeCommand.mock.calls[0] as [string]; - expect(command).toContain(`--bind-try '${collectedDir}' '${collectedDir}'`); - await fs.rm(hostDir, { recursive: true, force: true }); - }); - - it("keeps binding a collected rw directory after a later sync deletes it, so a stale path does not abort a later command", async () => { - // A sync records a read-write destination, then a later operation removes - // that path in the sandbox. The store still holds the path, so the wrapper - // still adds it to the command. `--bind-try` skips a missing source, so the - // command still runs. This test proves the wrapper uses `--bind-try` (not - // `--bind`) for the collected path, which stops a stale path from failing - // every later command for the scope. - process.env.DAYTONA_API_KEY = "host-key"; - const remoteCwd = "/home/daytona/paperclip-workspace"; - const collectedDir = `${remoteCwd}/scratch`; - const hostDir = await fs.mkdtemp(path.join(os.tmpdir(), "daytona-bwrap-stale-")); - const source = path.join(hostDir, "note.txt"); - await fs.writeFile(source, "bytes"); - const sandbox = createMockSandbox(); - mockGet.mockResolvedValue(sandbox); - - const scopeParams = { - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: false }, - }; - - await plugin.definition.onEnvironmentSyncIn?.({ - ...scopeParams, - lease: { providerLeaseId: "sandbox-123", metadata: { remoteCwd } }, - operations: [ - { - operationId: "sync-op-rw", - files: [ - { - sourcePath: source, - targetPath: `${collectedDir}/note.txt`, - kind: "file" as const, - access: "rw" as const, - }, - ], - }, - ], - }); - sandbox.process.executeCommand.mockClear(); - - await plugin.definition.onEnvironmentExecute?.({ - ...scopeParams, - lease: { - providerLeaseId: "sandbox-123", - metadata: { remoteCwd, bwrapAvailable: true, sandboxUsername: "daytona" }, - }, - command: "printf", - args: ["hello"], - timeoutMs: 1000, - }); - - const [command] = sandbox.process.executeCommand.mock.calls[0] as [string]; - // The stale collected path binds with `--bind-try`, so a missing source is - // skipped and the command still runs. - expect(command).toContain(`--bind-try '${collectedDir}' '${collectedDir}'`); - // No writable directory uses a plain `--bind`, so no stale source can abort. - expect(command).not.toMatch(/(^| )--bind '/); - await fs.rm(hostDir, { recursive: true, force: true }); - }); - - it("re-binds the stdin path when stdin is present and bwrap is available", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - mockGet.mockResolvedValue(sandbox); - - await plugin.definition.onEnvironmentExecute?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: false }, - lease: { - providerLeaseId: "sandbox-123", - metadata: { - remoteCwd: "/home/daytona/paperclip-workspace", - bwrapAvailable: true, - sandboxUsername: "daytona", - }, - }, - command: "cat", - args: [], - stdin: "input payload", - timeoutMs: 1000, - }); - - const stdinPath = sandbox.fs.uploadFile.mock.calls[0][1] as string; - expect(stdinPath).toMatch(/^\/tmp\/paperclip-stdin-/); - const [command] = sandbox.process.executeCommand.mock.calls[0] as [string]; - // The `--tmpfs /tmp` flag hides the uploaded stdin file, so the wrapper must - // re-bind it read-only after the tmpfs. - expect(command).toContain(`--ro-bind '${stdinPath}' '${stdinPath}'`); - }); - - it("runs the plain command when bwrap is unavailable or username is unknown", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - mockGet.mockResolvedValue(sandbox); - - // Case 1: the probe reported bwrap unavailable. - await plugin.definition.onEnvironmentExecute?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: false }, - lease: { - providerLeaseId: "sandbox-123", - metadata: { - remoteCwd: "/home/daytona/paperclip-workspace", - bwrapAvailable: false, - sandboxUsername: "daytona", - }, - }, - command: "printf", - args: ["hello"], - timeoutMs: 1000, - }); - expect((sandbox.process.executeCommand.mock.calls[0] as [string])[0]).not.toContain("bwrap"); - - // Case 2: bwrap is available but the username is unknown. A wrap without a - // username would run as root inside bwrap and give the agent's files root - // ownership, so the seam keeps the plain command. - sandbox.process.executeCommand.mockClear(); - await plugin.definition.onEnvironmentExecute?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { timeoutMs: 300000, reuseLease: false }, - lease: { - providerLeaseId: "sandbox-123", - metadata: { - remoteCwd: "/home/daytona/paperclip-workspace", - bwrapAvailable: true, - sandboxUsername: null, - }, - }, - command: "printf", - args: ["hello"], - timeoutMs: 1000, - }); - expect((sandbox.process.executeCommand.mock.calls[0] as [string])[0]).not.toContain("bwrap"); + expect(command).not.toContain("sudo -n bwrap"); + expect(command).not.toContain("su -s /bin/sh"); + expect(command).toMatch(/'printf' 'hello'$/); }); it("rejects invalid shell env keys before execution", async () => { @@ -3937,238 +4188,3 @@ describe("daytona manifest form defaults", () => { } }); }); - -describe("buildBwrapCommand advisory wrapper builder", () => { - it("uses su to drop to sandbox user without --unshare-user", () => { - const command = buildBwrapCommand( - "echo hi", - ["/home/daytona/paperclip-workspace"], - null, - "daytona", - ); - - expect(command).toBe( - "sudo -n bwrap " - + "--ro-bind / / --dev /dev --proc /proc --tmpfs /tmp " - + "--bind-try '/home/daytona/paperclip-workspace' '/home/daytona/paperclip-workspace' " - + "--new-session -- su -s /bin/sh 'daytona' -c 'echo hi'", - ); - expect(command).not.toContain("--unshare-user"); - expect(command).not.toContain("--uid"); - expect(command).not.toContain("--gid"); - }); - - it("re-binds the stdin path after tmpfs /tmp", () => { - const command = buildBwrapCommand( - "run-cmd", - ["/work"], - "/tmp/stdin.bin", - "daytona", - ); - - expect(command).toBe( - "sudo -n bwrap " - + "--ro-bind / / --dev /dev --proc /proc --tmpfs /tmp " - + "--bind-try '/work' '/work' " - + "--ro-bind '/tmp/stdin.bin' '/tmp/stdin.bin' " - + "--new-session -- su -s /bin/sh 'daytona' -c 'run-cmd'", - ); - // The stdin re-bind must come after the tmpfs, so the tmpfs does not hide it. - expect(command.indexOf("--ro-bind '/tmp/stdin.bin'")).toBeGreaterThan(command.indexOf("--tmpfs /tmp")); - }); - - it("quotes writable paths and inner script with single quotes", () => { - const command = buildBwrapCommand( - "echo 'hello'", - ["/data/o'brien"], - null, - "daytona", - ); - - // shellQuote rewrites each embedded single quote as the `'"'"'` token. - expect(command).toContain(`'"'"'`); - expect(command).toContain(`--bind-try '/data/o'"'"'brien' '/data/o'"'"'brien'`); - expect(command).toContain(`-c 'echo '"'"'hello'"'"''`); - }); - - it("falls back to sh -c when no username is given", () => { - const command = buildBwrapCommand("plain", ["/w"], null, null); - - expect(command).toBe( - "sudo -n bwrap " - + "--ro-bind / / --dev /dev --proc /proc --tmpfs /tmp " - + "--bind-try '/w' '/w' " - + "--new-session -- sh -c 'plain'", - ); - expect(command).not.toContain("--unshare-user"); - expect(command).not.toContain("su -s"); - }); - - it("binds each writable directory with --bind-try so a stale or deleted path does not abort bwrap", () => { - // The writable set holds advisory sandbox paths. A later sync can delete or - // replace one. `--bind` aborts when the source is absent; `--bind-try` skips - // a missing source and runs the command. Assert the builder emits - // `--bind-try` for every writable directory and never a plain `--bind` for - // them, so one stale path never poisons a later command. - const command = buildBwrapCommand( - "echo hi", - ["/home/daytona/paperclip-workspace", "/home/daytona/data"], - null, - "daytona", - ); - - expect(command).toContain( - "--bind-try '/home/daytona/paperclip-workspace' '/home/daytona/paperclip-workspace'", - ); - expect(command).toContain("--bind-try '/home/daytona/data' '/home/daytona/data'"); - // The only plain `--bind` family in the command is the read-only bind - // (`--ro-bind`). No writable directory uses a plain `--bind`. - expect(command).not.toMatch(/(^| )--bind '/); - }); -}); - -describe("advisory bwrap capability probe at lease time", () => { - // Route each probed command to a deterministic result so the hook exercises - // the real end-to-end path: shell detect, bwrap capability, and username read. - function bwrapExecMock(opts: { - bwrapExit?: number; - username?: string; - usernameExit?: number; - sentinelToken?: string; - } = {}) { - return async (command: string) => { - if (command.startsWith("cat ")) { - const token = opts.sentinelToken ?? "sentinel-token"; - return { exitCode: 0, result: JSON.stringify({ token }), artifacts: { stdout: JSON.stringify({ token }) } }; - } - if (command.includes("command -v bash")) { - return { exitCode: 0, result: "bash", artifacts: { stdout: "bash" } }; - } - if (command.startsWith("sudo -n bwrap")) { - return { exitCode: opts.bwrapExit ?? 0, result: "", artifacts: { stdout: "" } }; - } - if (command === "id -un") { - const username = opts.username ?? "daytona"; - return { exitCode: opts.usernameExit ?? 0, result: username, artifacts: { stdout: username } }; - } - return { exitCode: 0, result: "", artifacts: { stdout: "" } }; - }; - } - - const acquireParams = { - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - runId: "run-1", - agentId: "agent-1", - executionWorkspaceId: "workspace-1", - adapterType: "codex_local", - config: { image: "node:20", timeoutMs: 300000, reuseLease: true }, - }; - - it("records bwrap available and reads username when the probe exits zero", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - sandbox.process.executeCommand.mockImplementation(bwrapExecMock({ bwrapExit: 0, username: "daytona" })); - mockCreate.mockResolvedValue(sandbox); - - const lease = await plugin.definition.onEnvironmentAcquireLease?.(acquireParams); - - expect(lease).toMatchObject({ - metadata: { - bwrapAvailable: true, - sandboxUsername: "daytona", - }, - }); - }); - - it("bounds the probe timeout well under the hook deadline so the hook returns fallback metadata", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - sandbox.process.executeCommand.mockImplementation(bwrapExecMock({ bwrapExit: 0, username: "daytona" })); - mockCreate.mockResolvedValue(sandbox); - - // The hook deadline is 300 s; the probe must cap far below it. - await plugin.definition.onEnvironmentAcquireLease?.(acquireParams); - - const probeCalls = sandbox.process.executeCommand.mock.calls.filter( - ([command]: [string]) => command === "id -un" || command.startsWith("sudo -n bwrap"), - ); - expect(probeCalls.length).toBeGreaterThan(0); - for (const call of probeCalls) { - const timeoutArg = call[3] as number; - expect(timeoutArg).toBe(10); - } - }); - - it("records bwrap unavailable when the capability probe exits non-zero", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - sandbox.process.executeCommand.mockImplementation(bwrapExecMock({ bwrapExit: 1, username: "daytona" })); - mockCreate.mockResolvedValue(sandbox); - - const lease = await plugin.definition.onEnvironmentAcquireLease?.(acquireParams); - - expect(lease?.metadata).toMatchObject({ bwrapAvailable: false }); - }); - - it("records bwrap unavailable and does not throw when the probe throws", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - sandbox.process.executeCommand.mockRejectedValue(new Error("sandbox exec failed")); - mockCreate.mockResolvedValue(sandbox); - - const lease = await plugin.definition.onEnvironmentAcquireLease?.(acquireParams); - - expect(lease?.metadata).toMatchObject({ - bwrapAvailable: false, - sandboxUsername: null, - }); - }); - - it("runs the probe on the environment probe hook", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox(); - sandbox.process.executeCommand.mockImplementation(bwrapExecMock({ bwrapExit: 0, username: "daytona" })); - mockCreate.mockResolvedValue(sandbox); - - const result = await plugin.definition.onEnvironmentProbe?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - config: { snapshot: "base-snapshot", timeoutMs: 300000, reuseLease: false }, - }); - - expect(result).toMatchObject({ - ok: true, - metadata: { bwrapAvailable: true, sandboxUsername: "daytona" }, - }); - }); - - it("runs the probe on the resume-lease hook", async () => { - process.env.DAYTONA_API_KEY = "host-key"; - const sandbox = createMockSandbox({ id: "sandbox-reuse", state: "stopped" }); - sandbox.process.executeCommand.mockImplementation(bwrapExecMock({ bwrapExit: 0, username: "daytona" })); - mockGet.mockResolvedValue(sandbox); - - const lease = await plugin.definition.onEnvironmentResumeLease?.({ - driverKey: "daytona", - companyId: "company-1", - environmentId: "env-1", - providerLeaseId: "sandbox-reuse", - config: { timeoutMs: 300000, reuseLease: true }, - leaseMetadata: { - workspaceSentinel: { - path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json", - token: "sentinel-token", - result: "written", - }, - }, - }); - - expect(lease).toMatchObject({ - providerLeaseId: "sandbox-reuse", - metadata: { bwrapAvailable: true, sandboxUsername: "daytona" }, - }); - }); -}); diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 6efa06ed588..e00c0a2ca06 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -40,7 +40,7 @@ import type { PluginEnvironmentValidationResult, PluginSyncOperation, } from "@paperclipai/plugin-sdk"; -import { performSyncIn, performSyncOut } from "./file-sync.js"; +import { performSyncIn, performSyncOut, withProviderSpan } from "./file-sync.js"; // Injectable monotonic clock for provider-boundary timing (Open Q1). Defaults // to the real wall clock; `plugin.test.ts` overrides it via @@ -122,6 +122,7 @@ interface DaytonaDriverConfig { autoDeleteInterval: number | null; reuseLease: boolean; archiveOnRelease: boolean; + useSessions: boolean; } type WorkspaceSentinelResult = { @@ -175,14 +176,6 @@ const ARCHIVE_ON_RELEASE_AUTO_DELETE_MINUTES = 60; // RPC ceiling; callers always see an actionable error within this window. const GIT_NETWORK_TIMEOUT_MS = 120_000; -// Fail-fast cap for the advisory bwrap capability probe. The probe is -// best-effort, so it must return fallback metadata inside the lease hook. It -// must never consume the full hook deadline. A stalled probe command would -// otherwise expire the outer lease RPC before the probe records its unavailable -// result. This short cap keeps the probe well under the hook deadline, so the -// probe fails fast, records `bwrapAvailable: false`, and the hook still returns. -const BWRAP_PROBE_TIMEOUT_MS = 10_000; - // Noninteractive git credential defaults injected into every Daytona one-shot // command so that git operations never stall waiting for a terminal prompt. // Callers can override any of these via the env parameter. @@ -231,6 +224,11 @@ function parseDriverConfig(raw: Record): DaytonaDriverConfig { autoDeleteInterval: parseOptionalInteger(raw.autoDeleteInterval) ?? DEFAULT_AUTO_DELETE_INTERVAL_MINUTES, reuseLease: raw.reuseLease === true, archiveOnRelease: raw.archiveOnRelease === true, + // Session model opt-in. Default OFF. When off, the provider keeps the + // one-shot command path. When on, the exec hook opens one persistent + // Daytona session per lease and dispatches every command into it. The flag + // stays default off until a live leak soak passes. + useSessions: raw.useSessions === true, }; } @@ -330,14 +328,6 @@ function toTimeoutSeconds(timeoutMs: number): number { return Math.max(1, Math.ceil(timeoutMs / 1000)); } -// Bounded timeout for the advisory bwrap capability probe. The probe never uses -// the full hook deadline. It uses the smaller of the short probe cap and the -// hook timeout, so a stalled probe command returns fallback metadata inside the -// hook instead of expiring the outer lease RPC. -function toBwrapProbeTimeoutSeconds(config: DaytonaDriverConfig): number { - return toTimeoutSeconds(Math.min(BWRAP_PROBE_TIMEOUT_MS, config.timeoutMs)); -} - function resolveTimeoutMs(paramsTimeoutMs: number | undefined, config: DaytonaDriverConfig): number { return paramsTimeoutMs != null && Number.isFinite(paramsTimeoutMs) && paramsTimeoutMs > 0 ? Math.trunc(paramsTimeoutMs) @@ -415,75 +405,6 @@ function parseProbeInteger(value: string | undefined | null): number | null { return Number.isInteger(parsed) ? parsed : null; } -// Best-effort probe for the sandbox user's username. It runs `id -un` as the -// normal sandbox user (no `sudo`). The username is an image fact, not a code -// fact, so the probe is the only source of truth; the wrapper never assumes a -// hardcoded username. A non-zero exit code, an empty output, or a thrown error -// records no username. The probe never throws. -async function detectSandboxUsername( - sandbox: Sandbox, - timeoutSeconds: number, -): Promise { - try { - const result = await sandbox.process.executeCommand("id -un", undefined, undefined, timeoutSeconds); - if (result.exitCode !== 0) return null; - const username = result.result?.trim() ?? ""; - return username.length > 0 ? username : null; - } catch { - return null; - } -} - -// Best-effort probe for the advisory bwrap capability. The probe tests the real -// end-to-end capability using the su-based approach, not the old user-namespace -// approach. One command exercises the binary, the passwordless `sudo -n` rule, -// and the su user-switch together. The probe optionally binds the workspace -// directory (`--bind-try` suppresses ENOENT but not EACCES; on some Daytona -// images the home dir is `drwx------`, so including the workspace bind here -// catches that). A zero exit code means the capability is present. A non-zero -// exit code or a thrown error means the capability is absent. The probe never -// throws. It records the result, and the caller runs the command unwrapped when -// the capability is absent. -async function detectBwrapAvailable( - sandbox: Sandbox, - timeoutSeconds: number, - username: string, - remoteCwd?: string, -): Promise { - try { - const workspaceBind = remoteCwd - ? ` --bind-try ${shellQuote(remoteCwd)} ${shellQuote(remoteCwd)}` - : ""; - const result = await sandbox.process.executeCommand( - `sudo -n bwrap --ro-bind / /${workspaceBind} -- su -s /bin/sh ${shellQuote(username)} -c true`, - undefined, - undefined, - timeoutSeconds, - ); - return result.exitCode === 0; - } catch { - return false; - } -} - -// Run advisory bwrap probes sequentially (username first, then the bwrap probe -// which needs the username to construct the su command). The username is the -// ground-truth read; the wrapper relies on the probed name and never a -// hardcoded default. A missing username short-circuits and returns unavailable, -// so the caller runs the command unwrapped. Neither probe fails the lease. -async function detectBwrapCapability( - sandbox: Sandbox, - timeoutSeconds: number, - remoteCwd?: string, -): Promise<{ bwrapAvailable: boolean; sandboxUsername: string | null }> { - const username = await detectSandboxUsername(sandbox, timeoutSeconds); - if (username === null) { - return { bwrapAvailable: false, sandboxUsername: null }; - } - const bwrapAvailable = await detectBwrapAvailable(sandbox, timeoutSeconds, username, remoteCwd); - return { bwrapAvailable, sandboxUsername: username }; -} - function workspaceSentinelToken(input: { params: Pick; config: DaytonaDriverConfig; @@ -591,8 +512,6 @@ function leaseMetadata(input: { config: DaytonaDriverConfig; sandbox: Sandbox; shellCommand: "bash" | "sh"; - bwrapAvailable: boolean; - sandboxUsername: string | null; remoteCwd: string; resumedLease: boolean; workspaceSentinel?: WorkspaceSentinelResult; @@ -600,10 +519,6 @@ function leaseMetadata(input: { return { provider: "daytona", shellCommand: input.shellCommand, - // Advisory bwrap capability probed at lease time. `bwrapAvailable` false - // runs the command unwrapped; it never fails the lease. - bwrapAvailable: input.bwrapAvailable, - sandboxUsername: input.sandboxUsername, sandboxId: input.sandbox.id, sandboxName: input.sandbox.name, sandboxState: input.sandbox.state ?? null, @@ -631,64 +546,6 @@ function shellQuote(value: string): string { return `'${value.replace(/'/g, `'"'"'`)}'`; } -// Advisory bubblewrap (`bwrap`) wrapper. -// -// The wrapper gives an agent real-time feedback when the agent tries to change a -// file that the ephemeral sandbox will not keep. It adds NO security. The -// ephemeral sandbox stays the only security posture. The read-only root -// (`--ro-bind / /`) is a feedback signal, not a control: a write to a path -// outside the writable set fails at once, so the agent learns the change is not -// durable. -// -// `buildBwrapCommand` is pure. It builds one command string and runs no process. -// It needs no live sandbox. The flag order is load-bearing, because a later -// filesystem operation over the same path wins. So the writable `--bind` flags -// and the stdin re-bind must come after the read-only root and the fresh -// pseudo-filesystems. The function emits the flags in this fixed order: -// 1. `--ro-bind / /` (read-only root β€” the static system allowance base). -// 2. `--dev /dev --proc /proc --tmpfs /tmp` (fresh pseudo-filesystems). -// 3. one `--bind-try ` per writable directory, in the caller's order. -// 4. `--ro-bind ` when a stdin path is supplied. -// 5. `--new-session`. -// 6. `-- su -s /bin/sh '' -c ''` when a -// username is supplied, or `-- sh -c ''` otherwise. -// -// `sudo -n bwrap` runs as real root (for bind-mount capability). `su` then -// drops into the sandbox user, so inside uid= maps to outside uid= -// and workspace files owned by that uid are writable. The old `--unshare-user -// --uid`/`--gid` approach created a uid_map that made workspace files appear as -// overflow uid 65534 (nobody) from inside the namespace, causing EACCES. -// -// The writable binds use `--bind-try`, not `--bind`. The writable set is an -// advisory in-memory collection of sandbox paths. The host cannot check whether -// a sandbox path still exists. A path can be deleted or replaced after the store -// records it. `--bind` aborts bwrap when the source is absent, so one stale path -// would fail every later command for the scope. `--bind-try` skips a missing -// source and runs the command, which keeps the wrapper advisory and best-effort. -export function buildBwrapCommand( - innerScript: string, - writableDirs: string[], - stdinPath: string | null, - username: string | null, -): string { - const rootBinds = ["--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc", "--tmpfs", "/tmp"]; - const writableBinds = writableDirs.flatMap((dir) => ["--bind-try", shellQuote(dir), shellQuote(dir)]); - // Re-bind the stdin file after `--tmpfs /tmp`, so the tmpfs does not hide it. - const stdinReBind = stdinPath ? ["--ro-bind", shellQuote(stdinPath), shellQuote(stdinPath)] : []; - const tail = username - ? ["--new-session", "--", "su", "-s", "/bin/sh", shellQuote(username), "-c", shellQuote(innerScript)] - : ["--new-session", "--", "sh", "-c", shellQuote(innerScript)]; - return [ - "sudo", - "-n", - "bwrap", - ...rootBinds, - ...writableBinds, - ...stdinReBind, - ...tail, - ].join(" "); -} - function resolveConnectionExpiresInMinutes(value: number | null | undefined): number { if (typeof value !== "number" || !Number.isFinite(value)) return DEFAULT_SSH_ACCESS_MINUTES; return Math.min(24 * 60, Math.max(1, Math.trunc(value))); @@ -1296,19 +1153,16 @@ const sandboxHandleCache = (() => { })(); // Advisory writable-set store. It holds, per lease scope, the sandbox -// directories that a sync operation declared read-write (`access: "rw"`). An -// optional sandbox feedback wrapper reads this set later to bind those -// directories read-write, so an agent gets real-time feedback when a write to a -// non-persistent path fails. The store is advisory and best-effort in-memory -// state: it adds no security (the ephemeral sandbox stays the only boundary), -// and a cold store (for example after a worker restart) degrades to the -// workspace baseline, never to a crash. The store is keyed the same way as -// `sandboxHandleCache`, by `sandboxHandleCacheKey(scope)`. +// directories that a sync operation declared read-write (`access: "rw"`). The +// store is advisory and best-effort in-memory state: it adds no security (the +// ephemeral sandbox stays the only boundary). The store is keyed the same way +// as `sandboxHandleCache`, by `sandboxHandleCacheKey(scope)`. // -// The store keeps a path after a sync records it, so a path that a later -// operation deletes or replaces can stay in the set. The wrapper binds each -// path with `bwrap --bind-try`, which skips a missing source, so a stale path -// never fails a later command. See `buildBwrapCommand`. +// The command path no longer reads this set. The provider dropped the advisory +// `bwrap` wrapper that once bound these directories read-write for real-time +// feedback (see `DIRECTORY-CONSTRAINT-FINDINGS.md`). The store still records the +// read-write set, so a future isolation wrapper for the session can consume it +// without a new sync change. const sandboxHandleWritableDirs = (() => { const dirsByKey = new Map>(); @@ -1349,6 +1203,61 @@ const sandboxHandleWritableDirs = (() => { return { recordWritableTargets, get, reset }; })(); +// Per-lease Daytona session-id store. It holds, per lease scope, the id of the +// one persistent session the exec hook opened for that lease. The store is +// keyed the same way as `sandboxHandleCache`, by `sandboxHandleCacheKey(scope)`. +// The exec hook creates one session on a cache miss and records its id here. The +// teardown hooks delete the session and clear the id. A resume clears the id, +// because a restarted sandbox loses its session shell, so the next exec must +// open a fresh session. The store is process-memory only; it holds an id string, +// never a handle, a credential, or a command. +const sandboxHandleSessionStore = (() => { + const idByKey = new Map(); + // In-flight session creates, keyed the same way as `idByKey`. A create records + // its promise here for the time it runs, then removes it. The map lets two + // overlapping first commands for one lease share one create. See `runSingle`. + const pendingByKey = new Map>(); + + function get(scope: SandboxScope): string | undefined { + return idByKey.get(sandboxHandleCacheKey(scope)); + } + + function set(scope: SandboxScope, sessionId: string): void { + idByKey.set(sandboxHandleCacheKey(scope), sessionId); + } + + function clear(scope: SandboxScope): void { + idByKey.delete(sandboxHandleCacheKey(scope)); + } + + // Single-flight guard for the first-command session create. Two overlapping + // first commands for one lease must open at most one live session. The first + // caller runs `create` and records its in-flight promise; every concurrent + // caller awaits the same promise instead of a second `create`. The store keeps + // the promise only while `create` runs, then removes it, so a later command + // (for example, after a resume clears the id) can open a fresh session. A + // failed `create` removes the promise too, so the next command retries. + function runSingle(scope: SandboxScope, create: () => Promise): Promise { + const key = sandboxHandleCacheKey(scope); + const inFlight = pendingByKey.get(key); + if (inFlight) return inFlight; + const promise = create(); + pendingByKey.set(key, promise); + const settle = (): void => { + pendingByKey.delete(key); + }; + promise.then(settle, settle); + return promise; + } + + function reset(): void { + idByKey.clear(); + pendingByKey.clear(); + } + + return { get, set, clear, runSingle, reset }; +})(); + /** * Test seam: clear the process-scoped handle cache between tests so a handle * memoized under a reused composite key in one test never leaks into the next. @@ -1360,6 +1269,7 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void { sandboxHandleActivityGates.reset(); sandboxHandleLeaseAdmissionStates.reset(); sandboxHandleWritableDirs.reset(); + sandboxHandleSessionStore.reset(); } /** @@ -1403,47 +1313,72 @@ function evictSandboxHandle(scope: SandboxScope): void { sandboxHandleCache.clear(scope); } -// Advisory bwrap execution plan. When present, `executeOneShot` wraps the -// login-shell string with `buildBwrapCommand`. When null, it runs the plain -// string, which keeps today's behavior. `writableDirs` holds the workspace -// directory (baseline, always read-write) plus the collected read-write sync -// destinations. `username` is the sandbox user to su into inside bwrap. -type BwrapExecPlan = { - writableDirs: string[]; - username: string; -}; - -// Decide whether the advisory bwrap wrapper runs for one exec. The wrapper runs -// only when the lease reports bwrap available, a username is known, and the -// workspace directory is known. A wrap without a username would run as root -// inside bwrap and give the agent's files root ownership, so this returns null -// (run the plain command) in that case. The writable set is the workspace -// directory (baseline, always read-write) plus the per-scope read-write sync -// destinations. The baseline guarantees a safe result even when the collected -// store is cold. -function resolveBwrapExecPlan( - metadata: Record | null | undefined, - scope: SandboxScope, -): BwrapExecPlan | null { - if (metadata?.bwrapAvailable !== true) return null; - const username = typeof metadata.sandboxUsername === "string" ? metadata.sandboxUsername.trim() : ""; - if (username.length === 0) return null; - const remoteCwd = typeof metadata.remoteCwd === "string" ? metadata.remoteCwd.trim() : ""; - if (remoteCwd.length === 0) return null; - const writableDirs = new Set([remoteCwd]); - for (const dir of sandboxHandleWritableDirs.get(scope)) { - writableDirs.add(dir); - } - return { writableDirs: [...writableDirs], username }; +// Return the persistent session id for a lease, and open one session on a cache +// miss. The exec hook calls this once per command. The first call opens the +// session through `createSession` and records its id; every later call returns +// the stored id, so one lease runs every command in one persistent shell. The +// provider never falls back to a one-shot command to open a session. +// +// Leak bound: the Daytona SDK exposes NO per-session TTL. `createSession` takes +// only a session id, and there is no session update or expiry field. Two +// backstops bound the session against a leak. First, `teardownSession` runs a +// guaranteed `deleteSession` in every teardown hook's `try/finally`. Second, the +// sandbox-level `autoStopInterval` (15 minutes idle by default) stops the +// sandbox and, with it, every session; the `autoArchiveInterval` and +// `autoDeleteInterval` intervals then reap the sandbox. A session is a shell +// inside its sandbox and cannot outlive it. +async function getOrCreateSession(sandbox: Sandbox, scope: SandboxScope): Promise { + const existing = sandboxHandleSessionStore.get(scope); + if (existing) return existing; + // Single-flight the first-command create. Two overlapping first commands for + // one lease share one create promise, so the lease opens at most one live + // session. The guard checks and starts the create in one synchronous step, so + // no second command can slip in between the store read and the create start. + return sandboxHandleSessionStore.runSingle(scope, async () => { + const sessionId = `paperclip-${randomUUID()}`; + // Wrap the session create in a short `session.setup` provider span. The span + // carries no session id and no command text, only the provider family. The + // host maps the name to `sandbox.provider.session.setup`. + await withProviderSpan({ + name: "session.setup", + run: () => sandbox.process.createSession(sessionId), + }); + sandboxHandleSessionStore.set(scope, sessionId); + return sessionId; + }); } -// One-shot command execution via Daytona's `process.executeCommand`. The -// session-based API (`createSession` + `executeSessionCommand` with -// `runAsync: false`) hangs indefinitely when the supplied command ends with -// `exec `, which `buildLoginShellScript` always produces. Reproduced -// directly against the Daytona SDK: identical login-shell wrapper returns in -// ~600 ms via `executeCommand` but times out via `executeSessionCommand`. So we -// use the one-shot path, mirroring e2b's `sandbox.commands.run` model. +// Delete the persistent session for a lease and clear its stored id. Each +// teardown hook calls this inside its `try/finally`, so a failed delete never +// skips the rest of teardown. A failed delete logs the session id and the error +// loudly and does not throw past teardown; the sandbox stop or delete that +// follows removes the session shell anyway, and the sandbox-level +// `autoStopInterval` / `autoDeleteInterval` / `autoArchiveInterval` backstops +// bound any residual state (the session API exposes no per-session TTL). The +// store id is always cleared, so no orphan id survives. +async function teardownSession(sandbox: Sandbox, scope: SandboxScope): Promise { + const sessionId = sandboxHandleSessionStore.get(scope); + if (!sessionId) return; + try { + // Wrap the session delete in a short `session.teardown` provider span. The + // host maps the name to `sandbox.provider.session.teardown`. + await withProviderSpan({ + name: "session.teardown", + run: () => sandbox.process.deleteSession(sessionId), + }); + } catch (error) { + console.error( + `Failed to delete Daytona session ${sessionId} during teardown: ${formatErrorMessage(error)}`, + ); + } finally { + sandboxHandleSessionStore.clear(scope); + } +} + +// One-shot command execution via Daytona's `process.executeCommand`. This is the +// fallback path the exec hook uses when the session model is off. The command +// runs plain as the unprivileged sandbox user; the provider no longer wraps a +// user command with the advisory `bwrap` wrapper on any path. // // `executeCommand` returns combined stdout+stderr in `result`. We surface that // as `stdout` and leave `stderr` empty; callers that grep for error messages @@ -1452,7 +1387,6 @@ async function executeOneShot( sandbox: Sandbox, params: PluginEnvironmentExecuteParams, config: DaytonaDriverConfig, - bwrap: BwrapExecPlan | null, ): Promise { const gitNet = isGitNetworkCommand(params.command, params.args ?? []); const timeoutMs = resolveTimeoutMs(params.timeoutMs, config); @@ -1472,7 +1406,9 @@ async function executeOneShot( await sandbox.fs.uploadFile(Buffer.from(params.stdin ?? "", "utf8"), stdinPath, timeoutSeconds); } - const loginScript = buildLoginShellScript({ + // Run the plain login-shell script as the unprivileged sandbox user. The + // provider no longer wraps a user command with the advisory `bwrap` wrapper. + const command = buildLoginShellScript({ command: params.command, args: params.args ?? [], cwd: params.cwd, @@ -1480,16 +1416,6 @@ async function executeOneShot( stdinPath: stdinPath ?? undefined, }); - // Advisory bwrap wrapper (best-effort, automatic, no security boundary). When - // the lease reports bwrap available and a username is known, wrap the - // login-shell string so a write to a non-persistent path fails and the agent - // gets real-time feedback. The writable set binds the workspace and the - // read-write sync destinations; the stdin re-bind survives the `--tmpfs /tmp`. - // When the plan is null, run the plain string, which keeps today's behavior. - const command = bwrap - ? buildBwrapCommand(loginScript, bwrap.writableDirs, stdinPath, bwrap.username) - : loginScript; - // Pass cwd undefined: `buildLoginShellScript` already injects the `cd` after // it sources the login profiles, when params.cwd is set. The Daytona // executor's own cwd argument runs before that profile sourcing, which is @@ -1534,6 +1460,135 @@ async function executeOneShot( } } +// Poll interval for a session command's exit code. The live spike measured a +// session command resolving in about 260-300 ms, so a short interval keeps the +// poll responsive without a busy loop. +const SESSION_POLL_INTERVAL_MS = 50; + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +// Dispatch one user command into the persistent session and return its true +// stdout and stderr. +// +// The Daytona session is one persistent shell. A top-level `exit N` inside a +// session command ends that shell, so the next command then fails with "session +// process has exited". To stop a user `exit` from reaching the session shell, +// the dispatch wraps the whole login-shell script in a subshell `( ... )`. A +// user `exit` then ends only the subshell and reports its exit code, and the +// session shell stays alive. The provider passes the caller cwd and env inside +// the login-shell script on every command, so it never relies on implicit state +// that leaks between commands. +// +// The SDK exposes no built-in wait for a session command, so the dispatch runs +// the command with `runAsync: true` and polls `getSessionCommand` until the exit +// code is set. It then reads true `stdout` and `stderr` from +// `getSessionCommandLogs`, because the synchronous response fields are optional. +// The `runAsync: true` path also avoids the known `runAsync: false` login-shell +// hang. +async function executeInSession( + sandbox: Sandbox, + sessionId: string, + params: PluginEnvironmentExecuteParams, + config: DaytonaDriverConfig, +): Promise { + const gitNet = isGitNetworkCommand(params.command, params.args ?? []); + const timeoutMs = resolveTimeoutMs(params.timeoutMs, config); + const effectiveTimeoutMs = gitNet ? Math.min(timeoutMs, GIT_NETWORK_TIMEOUT_MS) : timeoutMs; + const timeoutSeconds = toTimeoutSeconds(effectiveTimeoutMs); + const stdinPath = params.stdin != null ? `/tmp/paperclip-stdin-${randomUUID()}` : null; + + // Marks the start of the session dispatch and poll. The timeout paths report + // the exec wall-time spent before the abort, so a slow command is still + // attributed to the provider boundary. + let execStart: number | null = null; + + try { + if (stdinPath) { + await sandbox.fs.uploadFile(Buffer.from(params.stdin ?? "", "utf8"), stdinPath, timeoutSeconds); + } + + const loginScript = buildLoginShellScript({ + command: params.command, + args: params.args ?? [], + cwd: params.cwd, + env: params.env, + stdinPath: stdinPath ?? undefined, + }); + // Subshell wrap: a top-level `exit` in the user command exits only the + // subshell, not the persistent session shell. + const command = `( ${loginScript} )`; + + execStart = timingNow(); + const dispatched = await sandbox.process.executeSessionCommand( + sessionId, + { command, runAsync: true }, + timeoutSeconds, + ); + const commandId = dispatched.cmdId; + + // Poll for the exit code; the SDK has no wait method. The poll deadline uses + // the wall clock, separate from the injected timing clock that measures the + // reported `durationMs`. + const deadlineMs = Date.now() + effectiveTimeoutMs; + let exitCode: number | null = null; + while (true) { + const status = await sandbox.process.getSessionCommand(sessionId, commandId); + if (typeof status.exitCode === "number") { + exitCode = status.exitCode; + break; + } + if (Date.now() >= deadlineMs) { + const durationMs = timingNow() - execStart; + const timeoutMessage = gitNet + ? `Git network operation timed out after ${Math.round(effectiveTimeoutMs / 1000)} s β€” the remote may be unreachable or noninteractive credentials are not configured.` + : `Command timed out after ${Math.round(effectiveTimeoutMs / 1000)} s.`; + return { + exitCode: null, + timedOut: true, + stdout: "", + stderr: `${timeoutMessage}\n`, + metadata: { durationMs }, + }; + } + await sleep(SESSION_POLL_INTERVAL_MS); + } + + // Read true, separated stdout and stderr from the logs endpoint. The + // synchronous dispatch response fields are optional, so the logs endpoint is + // the source of truth. + const logs = await sandbox.process.getSessionCommandLogs(sessionId, commandId); + const durationMs = timingNow() - execStart; + return { + exitCode, + timedOut: false, + stdout: logs.stdout ?? "", + stderr: logs.stderr ?? "", + metadata: { durationMs }, + }; + } catch (error) { + if (error instanceof DaytonaTimeoutError) { + const timeoutMessage = gitNet + ? `Git network operation timed out after ${Math.round(effectiveTimeoutMs / 1000)} s β€” the remote may be unreachable or noninteractive credentials are not configured.` + : error.message.trim(); + const durationMs = execStart != null ? timingNow() - execStart : undefined; + return { + exitCode: null, + timedOut: true, + stdout: "", + stderr: `${timeoutMessage}\n`, + ...(durationMs != null ? { metadata: { durationMs } } : {}), + }; + } + throw error; + } finally { + if (stdinPath) { + await sandbox.fs.deleteFile(stdinPath).catch(() => undefined); + } + } +} + const plugin = definePlugin({ async setup(ctx) { // Hoist the context to a module variable so the lifecycle hooks and the @@ -1613,15 +1668,12 @@ const plugin = definePlugin({ try { const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); const shellCommand = await detectSandboxShellCommand(sandbox, toTimeoutSeconds(config.timeoutMs)); - const bwrapCapability = await detectBwrapCapability(sandbox, toBwrapProbeTimeoutSeconds(config), remoteCwd); return { ok: true, summary: `Connected to Daytona sandbox ${sandbox.name}.`, metadata: { provider: "daytona", shellCommand, - bwrapAvailable: bwrapCapability.bwrapAvailable, - sandboxUsername: bwrapCapability.sandboxUsername, sandboxId: sandbox.id, sandboxName: sandbox.name, target: sandbox.target, @@ -1659,7 +1711,6 @@ const plugin = definePlugin({ try { const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); const shellCommand = await detectSandboxShellCommand(sandbox, toTimeoutSeconds(config.timeoutMs)); - const bwrapCapability = await detectBwrapCapability(sandbox, toBwrapProbeTimeoutSeconds(config), remoteCwd); const workspaceSentinel = await writeWorkspaceSentinel({ sandbox, remoteCwd, @@ -1693,8 +1744,6 @@ const plugin = definePlugin({ config, sandbox, shellCommand, - bwrapAvailable: bwrapCapability.bwrapAvailable, - sandboxUsername: bwrapCapability.sandboxUsername, remoteCwd, resumedLease: false, workspaceSentinel, @@ -1723,6 +1772,17 @@ const plugin = definePlugin({ return { providerLeaseId: null, metadata: { expired: true } }; } + // A stopped sandbox loses its session shell, so the stored session id is + // stale after a real restart. Clear the id only when the sandbox is not + // already running, and clear it before the restart. A stopped sandbox has + // no live session, so the clear drops a dead id and a later command opens + // a fresh session. A running sandbox keeps its live session, so the resume + // leaves the id in place; a concurrent command still finds it and teardown + // deletes one session. An unconditional clear would drop the id of a live + // session and leak its shell until sandbox reaping. + if (sandbox.state !== "started") { + sandboxHandleSessionStore.clear(scope); + } await ensureSandboxStarted(sandbox, toTimeoutSeconds(config.timeoutMs)); try { const remoteCwd = await resolveSandboxWorkingDirectory(sandbox); @@ -1741,7 +1801,6 @@ const plugin = definePlugin({ return { providerLeaseId: null, metadata: { expired: true, workspaceSentinel } }; } const shellCommand = await detectSandboxShellCommand(sandbox, toTimeoutSeconds(config.timeoutMs)); - const bwrapCapability = await detectBwrapCapability(sandbox, toBwrapProbeTimeoutSeconds(config), remoteCwd); sandboxHandleCache.markFresh(scope); sandboxHandleLeaseAdmissionStates.open(scope); return { @@ -1750,8 +1809,6 @@ const plugin = definePlugin({ config, sandbox, shellCommand, - bwrapAvailable: bwrapCapability.bwrapAvailable, - sandboxUsername: bwrapCapability.sandboxUsername, remoteCwd, resumedLease: true, workspaceSentinel, @@ -1788,6 +1845,7 @@ const plugin = definePlugin({ evictSandboxHandle(scope); await sandboxHandleActivityGates.waitForIdle(scope); + await teardownSession(sandbox, scope); if (config.reuseLease) { if (sandbox.state !== "stopped") { @@ -1851,6 +1909,7 @@ const plugin = definePlugin({ evictSandboxHandle(scope); await sandboxHandleActivityGates.waitForIdle(scope); + await teardownSession(sandbox, scope); await sandbox.delete(toTimeoutSeconds(config.timeoutMs)); } finally { sandboxHandleTeardownGates.end(scope, teardownGate); @@ -2088,6 +2147,7 @@ const plugin = definePlugin({ } evictSandboxHandle(scope); await sandboxHandleActivityGates.waitForIdle(scope); + await teardownSession(sandbox, scope); await sandbox.delete(toTimeoutSeconds(config.timeoutMs)); return { status: params.reason === "timed_out" ? "timed_out" : "cancelled", @@ -2179,24 +2239,34 @@ const plugin = definePlugin({ }); const getDurationMs = timingNow() - getStart; await ensureSandboxStarted(sandbox, toTimeoutSeconds(resolveTimeoutMs(params.timeoutMs, config))); - // Read the advisory bwrap flags from the lease metadata and read the - // collected writable directories from the same scope the sync-in hook uses. - const bwrapPlan = resolveBwrapExecPlan(params.lease.metadata, { + const scope: SandboxScope = { driverKey: params.driverKey, companyId: params.companyId, environmentId: params.environmentId, providerLeaseId, config, - }); - const result = await executeOneShot(sandbox, params, config, bwrapPlan); + }; + // Dispatch the command. When the session model is on, open the persistent + // session on a cache miss and run the command in it. The provider never + // falls back to a one-shot command to open a session; a cache miss creates + // one. When the session model is off, run the command on the one-shot path. + // + // A `bypassSession` command runs one-shot even when the session model is + // on, and it does NOT open the session. The host sets this flag on a + // pre-run command (the workspace provision command) that runs before the + // run opens its trace root. Opening the session there would emit a + // `session.setup` span with no run parent, and the span backend would drop + // it. With the bypass the session opens on the first in-run command, whose + // setup span parents to the run trace. + let result: PluginEnvironmentExecuteResult; + if (config.useSessions && !params.bypassSession) { + const sessionId = await getOrCreateSession(sandbox, scope); + result = await executeInSession(sandbox, sessionId, params, config); + } else { + result = await executeOneShot(sandbox, params, config); + } if (!result.timedOut) { - sandboxHandleCache.markFresh({ - driverKey: params.driverKey, - companyId: params.companyId, - environmentId: params.environmentId, - providerLeaseId, - config, - }); + sandboxHandleCache.markFresh(scope); } return { ...result, diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index 7e299764e1a..e14ba5fa26f 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -671,6 +671,20 @@ export interface PluginEnvironmentExecuteParams extends PluginEnvironmentDriverB env?: Record; stdin?: string; timeoutMs?: number; + /** + * Run this command outside the lease's persistent session. + * + * The host sets this flag on a command that runs before the run's agent work, + * for example the workspace provision command. A provider that opens a + * persistent session on the first command must NOT open the session for such a + * command; it runs the command one-shot and leaves the session closed. The + * session then opens on the first in-run command instead. A provider that does + * not use a persistent session ignores this flag. + * + * The default (absent or `false`) keeps the session path, so a normal in-run + * command opens and reuses the session as before. + */ + bypassSession?: boolean; } export interface PluginEnvironmentExecuteResult { diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 3c836883abd..5c13b76f522 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -32,6 +32,13 @@ import { environmentRuntimeService, findReusableSandboxLeaseId } from "../servic import { environmentService } from "../services/environments.ts"; import { secretService } from "../services/secrets.ts"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.ts"; +import { + getActiveStepContext, + runWithRuntimeParent, + type StartupSpanContext, +} from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; +import { traceparentFromContextToken } from "../instrumentation.ts"; +import { ROOT_CONTEXT, trace } from "@opentelemetry/api"; const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -699,6 +706,321 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentReleaseLease", expect.anything(), 31234); }); + // Build the fake plugin fixture for the run-parent release tests below. A + // plugin sandbox provider can open a persistent session on the first command + // and delete it on lease release; the delete emits a provider + // `session.teardown` span. The host mints that span's parent from the active + // step context at the release RPC. So the release must run under the run + // parent, or the span loses its traceparent and the backend drops it. + async function seedFakePluginSandbox() { + const pluginId = randomUUID(); + const { companyId, environment: baseEnvironment, runId } = await seedEnvironment(); + const fakePluginConfig = { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: false, + }; + const environment = { + ...baseEnvironment, + name: "Fake Plugin Sandbox", + driver: "sandbox", + config: fakePluginConfig, + }; + await environmentService(db).update(environment.id, { + driver: "sandbox", + name: environment.name, + config: fakePluginConfig, + }); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: "paperclip.fake-plugin-sandbox-provider", + packageName: "@paperclipai/plugin-fake-sandbox", + version: "1.0.0", + apiVersion: 1, + categories: ["automation"], + manifestJson: { + id: "paperclip.fake-plugin-sandbox-provider", + apiVersion: 1, + version: "1.0.0", + displayName: "Fake Plugin Sandbox Provider", + description: "Test fake plugin provider", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { worker: "dist/worker.js" }, + environmentDrivers: [ + { + driverKey: "fake-plugin", + kind: "sandbox_provider", + displayName: "Fake Plugin", + configSchema: { type: "object" }, + }, + ], + }, + status: "ready", + installOrder: 1, + updatedAt: new Date(), + } as any); + return { companyId, environment, runId, pluginId }; + } + + it("runs a plugin-backed lease release under the run-time exec parent, so the teardown span keeps a valid traceparent", async () => { + const { companyId, environment, runId, pluginId } = await seedFakePluginSandbox(); + + // The active step context observed at the moment the host issues the release + // RPC. The real worker manager mints the provider-span traceparent from + // exactly this value, so a non-null context with a valid parent proves the + // teardown span keeps a host-minted parent. + let releaseStepContext: ReturnType = null; + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "sandbox-1", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: false, + remoteCwd: "/workspace", + }, + }; + } + if (method === "environmentExecute") { + return { exitCode: 0, signal: null, timedOut: false, stdout: "ok\n", stderr: "" }; + } + if (method === "environmentReleaseLease") { + releaseStepContext = getActiveStepContext(); + return undefined; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + heartbeatRunId: runId, + persistedExecutionWorkspace: null, + }); + + // Build a real host run-parent context and run the first command under it, + // exactly as the run drives an exec. The driver records this context for the + // lease and replays it around the later release RPC. + const runParent: StartupSpanContext = trace.setSpanContext(ROOT_CONTEXT, { + traceId: "0af7651916cd43dd8448eb211c80319c", + spanId: "b7ad6b7169203331", + traceFlags: 1, + isRemote: false, + }); + await runWithRuntimeParent(runParent, () => + runtimeWithPlugin.execute({ + environment, + lease: acquired.lease, + command: "printf", + args: ["ok"], + cwd: "/workspace", + env: {}, + timeoutMs: 1000, + }), + ); + + await environmentService(db).update(environment.id, { driver: "local", config: {} }); + const released = await runtimeWithPlugin.releaseRunLeases(runId); + + expect(released).toHaveLength(1); + // The release RPC ran under the recorded run parent, so the host mints a + // valid W3C traceparent from it. A dropped span would show a null context. + expect(releaseStepContext).not.toBeNull(); + expect(releaseStepContext?.parentContext).toBe(runParent); + expect(traceparentFromContextToken(releaseStepContext?.parentContext)).toBe( + "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", + ); + }); + + it("releases a plugin-backed lease with no active step context when the run drove no exec", async () => { + const { companyId, environment, runId, pluginId } = await seedFakePluginSandbox(); + + let releaseStepContext: ReturnType = null; + let releaseCalled = false; + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "sandbox-1", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: false, + remoteCwd: "/workspace", + }, + }; + } + if (method === "environmentReleaseLease") { + releaseCalled = true; + releaseStepContext = getActiveStepContext(); + return undefined; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + heartbeatRunId: runId, + persistedExecutionWorkspace: null, + }); + await environmentService(db).update(environment.id, { driver: "local", config: {} }); + const released = await runtimeWithPlugin.releaseRunLeases(runId); + + expect(released).toHaveLength(1); + // No command ran, so no session opened and no teardown span is emitted. The + // release must not invent a parent from an unrelated ambient context; it + // runs unwrapped, exactly as before the fix. + expect(releaseCalled).toBe(true); + expect(releaseStepContext).toBeNull(); + }); + + it("forwards the bypassSession flag to the plugin execute RPC so a pre-run command skips the session", async () => { + const { companyId, environment, runId, pluginId } = await seedFakePluginSandbox(); + + // Capture the params of each environmentExecute RPC, so the test can assert + // the host forwards `bypassSession` to the provider unchanged. + const executeParams: Array> = []; + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string, params: Record) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "sandbox-1", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: false, + remoteCwd: "/workspace", + }, + }; + } + if (method === "environmentExecute") { + executeParams.push(params); + return { exitCode: 0, signal: null, timedOut: false, stdout: "ok\n", stderr: "" }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + heartbeatRunId: runId, + persistedExecutionWorkspace: null, + }); + + // A pre-run command (the provision command) sets bypassSession explicitly. + await runtimeWithPlugin.execute({ + environment, + lease: acquired.lease, + command: "bash", + args: ["-lc", "true"], + cwd: "/workspace", + env: {}, + timeoutMs: 1000, + bypassSession: true, + }); + // An in-run command runs under the run-parent context, so the host does not + // bypass the session and the provider opens/uses the session. + const runParent: StartupSpanContext = trace.setSpanContext(ROOT_CONTEXT, { + traceId: "0af7651916cd43dd8448eb211c80319c", + spanId: "b7ad6b7169203331", + traceFlags: 1, + isRemote: false, + }); + await runWithRuntimeParent(runParent, () => + runtimeWithPlugin.execute({ + environment, + lease: acquired.lease, + command: "printf", + args: ["ok"], + cwd: "/workspace", + env: {}, + timeoutMs: 1000, + }), + ); + + expect(executeParams).toHaveLength(2); + expect(executeParams[0]?.bypassSession).toBe(true); + // An in-run command carries a run parent, so it does not bypass the session. + expect(executeParams[1]?.bypassSession).toBe(false); + }); + + it("bypasses the session for a context-less command so the setup span keeps a run parent", async () => { + const { companyId, environment, runId, pluginId } = await seedFakePluginSandbox(); + + // Capture the params of each environmentExecute RPC, so the test can assert + // the host derives `bypassSession` from the active run-parent context. + const executeParams: Array> = []; + const workerManager = { + isRunning: vi.fn((id: string) => id === pluginId), + call: vi.fn(async (_pluginId: string, method: string, params: Record) => { + if (method === "environmentAcquireLease") { + return { + providerLeaseId: "sandbox-1", + metadata: { + provider: "fake-plugin", + image: "fake:test", + timeoutMs: 1234, + reuseLease: false, + remoteCwd: "/workspace", + }, + }; + } + if (method === "environmentExecute") { + executeParams.push(params); + return { exitCode: 0, signal: null, timedOut: false, stdout: "ok\n", stderr: "" }; + } + throw new Error(`Unexpected plugin method: ${method}`); + }), + } as unknown as PluginWorkerManager; + const runtimeWithPlugin = environmentRuntimeService(db, { pluginWorkerManager: workerManager }); + + const acquired = await runtimeWithPlugin.acquireRunLease({ + companyId, + environment, + issueId: null, + heartbeatRunId: runId, + persistedExecutionWorkspace: null, + }); + + // A command with no active run-parent context (a pre-run install/probe or + // the agent process launch) runs before the run trace is active. The host + // mints no plugin RPC traceparent for it, so a session opened here would drop + // its setup span. The host bypasses the session for such a command. + await runtimeWithPlugin.execute({ + environment, + lease: acquired.lease, + command: "sh", + args: ["-c", "command -v claude"], + cwd: "/workspace", + env: {}, + timeoutMs: 1000, + }); + + expect(executeParams).toHaveLength(1); + expect(executeParams[0]?.bypassSession).toBe(true); + }); + it("builds the workspace-realization record with referenced sources for a plugin-backed sandbox realize", async () => { // A provider plugin realize handler returns only its realized cwd and provider metadata; it does // not build the workspace-realization record. The server must build that record from the run diff --git a/server/src/__tests__/plugin-host-services-span.test.ts b/server/src/__tests__/plugin-host-services-span.test.ts index abc4ea058f8..110d06a0b63 100644 --- a/server/src/__tests__/plugin-host-services-span.test.ts +++ b/server/src/__tests__/plugin-host-services-span.test.ts @@ -143,6 +143,21 @@ describe("plugin provider span host handler", () => { ]); }); + it("admits the session setup and teardown span names", async () => { + const services = servicesFor(); + for (const name of ["session.setup", "session.teardown"]) { + await services.tracer.record( + { name }, + { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, + ); + } + const recorded = mockRecordSpan.mock.calls.map((c) => (c[0] as { name: string }).name); + expect(recorded).toEqual([ + "sandbox.provider.session.setup", + "sandbox.provider.session.teardown", + ]); + }); + it("forwards a valid start-time and end-time pair to the recorder", async () => { const services = servicesFor(); const startTimeMs = Date.now() - 4500; diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index 7ebad637394..8d60e77d8e6 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -432,6 +432,13 @@ export function environmentRunOrchestrator( SHELL: "/bin/bash", }, timeoutMs: 300_000, + // The provision command runs before the run opens its trace root, so it + // carries no run parent. A sandbox provider that opens a persistent + // session on the first command must not open the session here, or the + // session-setup span loses its parent and the span backend drops it. + // Bypass the session for this command; the session opens on the first + // in-run command instead, whose setup span parents to the run trace. + bypassSession: true, }); if (provisionResult.exitCode !== 0 || provisionResult.timedOut) { throw new Error(formatProvisionFailureDetail(provisionResult)); diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 7457718a50a..0e381340a9a 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -20,6 +20,11 @@ import type { PluginSyncOperation, } from "@paperclipai/plugin-sdk"; import { ensureSshWorkspaceReady } from "@paperclipai/adapter-utils/ssh"; +import { + getActiveStepContext, + runWithRuntimeParent, + type StartupSpanContext, +} from "@paperclipai/adapter-utils/acpx-engine/startup-timing"; import { environmentService } from "./environments.js"; import { collectEnvironmentSecretRefs, @@ -188,6 +193,15 @@ export interface EnvironmentDriverExecuteInput extends EnvironmentDriverLeaseInp env?: Record; stdin?: string; timeoutMs?: number; + /** + * Run this command outside the lease's persistent session. The run + * orchestrator sets this on the workspace provision command, which runs before + * the run opens its trace root. A sandbox provider that opens a persistent + * session on the first command must run this command one-shot and keep the + * session closed, so the session first opens on an in-run command whose setup + * span parents to the run trace. The default keeps the session path. + */ + bypassSession?: boolean; } export interface EnvironmentDriverSyncInput extends EnvironmentDriverLeaseInput { @@ -639,6 +653,31 @@ function createSandboxEnvironmentDriver( const pluginWorkerReadyPollMs = options.pluginWorkerReadyPollMs ?? DEFAULT_PLUGIN_SANDBOX_WORKER_READY_POLL_MS; const environmentsSvc = environmentService(db); + // The run-time exec parent context, held per lease id. A plugin sandbox + // provider can open a persistent session on the first command and delete it + // on lease release. The session open runs inside `execute`, under the run + // parent (the same context the `sandbox.exec` span reads). The session delete + // runs inside the lease-release RPC, which the run orchestrator calls after + // the run, outside that scope. Without a parent the host mints no + // `traceparent` and drops the provider `session.teardown` span. So `execute` + // records the exec parent here, and the release paths replay it around the + // release RPC. The host still mints and validates the `traceparent` itself; + // the value only widens which host calls carry a run parent. An entry is + // removed on release, so the map holds at most one context per live lease. + const runExecParentByLeaseId = new Map(); + + // Run a lease-release RPC under the lease's recorded exec parent context, and + // then drop the entry β€” the lease is gone. Under the parent the host mints a + // `traceparent`, so a provider `session.teardown` span reaches the span + // backend in the run trace. With no recorded context (no command ran, or a + // local target with no trace context) the call runs unwrapped, exactly as + // before, so the change never fails a release. + function runLeaseReleaseWithRunParent(leaseId: string, call: () => Promise): Promise { + const runParent = runExecParentByLeaseId.get(leaseId); + runExecParentByLeaseId.delete(leaseId); + return runParent !== undefined ? runWithRuntimeParent(runParent, call) : call(); + } + async function resolveSandboxProviderPlugin(input: { provider: string }) { const running = await resolvePluginSandboxProviderDriverByKey({ db, @@ -1274,6 +1313,31 @@ function createSandboxEnvironmentDriver( async execute(input) { // Plugin-backed sandbox providers: delegate command execution. if (input.lease.metadata?.sandboxProviderPlugin && pluginWorkerManager) { + // Read the active run-parent context once. The host mints the plugin + // RPC `traceparent` from this same context, so a provider `session.setup` + // span parents to the run trace only when this context is present. + const activeStep = getActiveStepContext(); + // Record the run-time exec parent context for this lease, so a later + // lease-release RPC that emits the provider `session.teardown` span can + // parent to the same run trace. This is the same context the + // `sandbox.exec` span reads. Keep only a defined context; a local or SSH + // target with no host trace context yields undefined and stores nothing. + const execParentContext = activeStep?.parentContext; + if (execParentContext !== undefined) { + runExecParentByLeaseId.set(input.lease.id, execParentContext); + } + // Bypass the persistent session for any command that runs with no active + // run-parent context. Such a command runs before the run trace is active: + // the workspace provision command, the CLI install command, the + // resolvability probe, and the agent process launch all run at the top of + // the adapter execute, outside a measured step and outside the run + // trace. A session opened on such a command emits a `session.setup` span + // with no host-minted parent, and the span backend drops it. So run the + // command one-shot and keep the session closed; the session then opens on + // the first in-run command that carries a run parent (an agent tool + // command runs under the run trace), whose setup span parents to the run + // trace. A command that sets `bypassSession` explicitly always bypasses. + const bypassSession = input.bypassSession === true || activeStep === null; const pluginId = readString(input.lease.metadata?.pluginId); const providerKey = readString(input.lease.metadata?.provider); if (pluginId && providerKey) { @@ -1300,6 +1364,13 @@ function createSandboxEnvironmentDriver( env: input.env, stdin: input.stdin, timeoutMs: input.timeoutMs, + // Forward the effective session-bypass flag so a provider that opens + // a persistent session skips it for a pre-run or context-less command + // (the workspace provision command, the CLI install command, the + // resolvability probe, the agent process launch). The session then + // opens on the first in-run command that carries a run parent, whose + // setup span parents to the run trace. + bypassSession, }, resolvePluginExecuteRpcTimeoutMs({ requestedTimeoutMs: input.timeoutMs, config: sanitizedConfig, @@ -1368,15 +1439,17 @@ function createSandboxEnvironmentDriver( lease: input.lease, provider: providerKey, }); - await pluginWorkerManager.call(pluginId, "environmentReleaseLease", { - driverKey: providerKey, - companyId: input.lease.companyId, - environmentId: input.environment.id, - issueId: input.lease.issueId, - config: stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig), - providerLeaseId: input.lease.providerLeaseId, - leaseMetadata: metadata, - }, resolvePluginSandboxRpcTimeoutMs(stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig))); + await runLeaseReleaseWithRunParent(input.lease.id, () => + pluginWorkerManager.call(pluginId, "environmentReleaseLease", { + driverKey: providerKey, + companyId: input.lease.companyId, + environmentId: input.environment.id, + issueId: input.lease.issueId, + config: stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig), + providerLeaseId: input.lease.providerLeaseId, + leaseMetadata: metadata, + }, resolvePluginSandboxRpcTimeoutMs(stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig))), + ); } catch { cleanupStatus = "failed"; } @@ -1414,15 +1487,17 @@ function createSandboxEnvironmentDriver( lease: input.lease, provider: providerKey, }); - await pluginWorkerManager.call(pluginId, "environmentDestroyLease", { - driverKey: providerKey, - companyId: input.lease.companyId, - environmentId: input.environment.id, - issueId: input.lease.issueId, - config: stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig), - providerLeaseId: input.lease.providerLeaseId, - leaseMetadata: metadata, - }, resolvePluginSandboxRpcTimeoutMs(stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig))); + await runLeaseReleaseWithRunParent(input.lease.id, () => + pluginWorkerManager.call(pluginId, "environmentDestroyLease", { + driverKey: providerKey, + companyId: input.lease.companyId, + environmentId: input.environment.id, + issueId: input.lease.issueId, + config: stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig), + providerLeaseId: input.lease.providerLeaseId, + leaseMetadata: metadata, + }, resolvePluginSandboxRpcTimeoutMs(stripSandboxProviderEnvelope(config as SandboxEnvironmentConfig))), + ); } } else { const metadataConfig = sandboxConfigFromLeaseMetadata(input.lease); diff --git a/server/src/services/plugin-host-services.ts b/server/src/services/plugin-host-services.ts index bf490c69a74..1eb3b8a01a7 100644 --- a/server/src/services/plugin-host-services.ts +++ b/server/src/services/plugin-host-services.ts @@ -510,7 +510,8 @@ const SPAN_ATTRS = SANDBOX_STARTUP_SPAN_ATTRS; /** The closed set of provider span names a plugin may emit. `pack` and * `transfer` are the host-local build and the byte upload. `mkdir`, `guard`, * `rename`, `extract`, and `provision` are the per-round-trip command spans in - * the inbound sync path. */ + * the inbound sync path. `session.setup` and `session.teardown` are the short + * spans that wrap a persistent-session create and delete. */ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet = new Set([ "pack", "transfer", @@ -519,6 +520,8 @@ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet = new Set([ "rename", "extract", "provision", + "session.setup", + "session.teardown", ]); /** Clamp the span name to a closed, namespaced set. A known name maps to From ea83c5c822f904b396320c84662200a0c2dff603 Mon Sep 17 00:00:00 2001 From: scotttong Date: Thu, 6 Aug 2026 18:26:44 -0700 Subject: [PATCH 008/311] =?UTF-8?q?feat(task-chat):=20bring=20back=20copy/?= =?UTF-8?q?=F0=9F=91=8D/=F0=9F=91=8E=20actions=20on=20the=20agent=20bubble?= =?UTF-8?q?=20footer=20(#11025)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task view shows a threaded conversation between the user and the agent, with agent replies rendered as bubbles that carry a "βœ“ Worked Β· N tools" summary line. > - The conference-room chat already offers per-message copy and thumbs-up / thumbs-down feedback, but the redesigned task thread dropped these controls from the agent bubble footer. > - Users lose a quick way to copy an agent reply or send feedback on it, and the redesign silently ignored the feedback-vote props it was already given. > - This pull request prepends a copy Β· thumbs-up Β· thumbs-down cluster to the bubble summary line and wires the existing feedback-vote props through. > - The benefit is a consistent feedback surface across both chat views, with no new API. ## Linked Issues or Issue Description No public GitHub issue exists for this change; the underlying issue is described inline below following the feature template (`.github/ISSUE_TEMPLATE/feature_request.yml`). #### Problem or motivation The redesigned task thread renders each agent reply with a "βœ“ Worked Β· N tools Β· " summary line, but it dropped the copy and thumbs-up / thumbs-down controls that the conference-room chat still shows. Users can no longer copy an agent reply or vote feedback from the task thread. The redesign component already received `feedbackVotes` and `onVote` props but ignored them. #### Proposed solution Prepend a copy Β· πŸ‘ Β· πŸ‘Ž cluster to the summary line, leading the always-visible timestamp, reusing the shared `IssueChatFeedbackButtons` so both chat views speak the same feedback language. Anchor the cluster to the turn's summary row (a sibling of the expandable tool-history fold) so it stays on the summary line whether the tool history is collapsed or expanded. #### Alternatives considered Placing the cluster inside the expandable fold β€” rejected because expanding the tool history then re-centered the actions to the middle of the tall fold. #### Roadmap alignment UI polish to the task thread; no core-roadmap overlap. ## What Changed - Add `TaskChatBubbleActions`: a copy Β· thumbs-up Β· thumbs-down cluster built on the shared `IssueChatFeedbackButtons`. - Render the cluster on the agent bubble's "βœ“ Worked Β· …" summary line, leading the timestamp; runless agent replies get the same cluster with the timestamp trailing. Human and system bubbles are unchanged. - Add a `leading` slot to `TaskChatTurn` so the actions sit on the summary row, a sibling of the tool-history fold, and stay anchored when the fold expands. - Wire the redesign to the `feedbackVotes` / `onVote` props it already received. - Add a demo binding in the `TaskChatLab` dev harness. ## Verification - `pnpm check:token-gates` β€” 3/3 gates CLEAN. - `pnpm typecheck` β€” clean across all packages. - `cd ui && pnpm vitest run src/components/task-chat/TaskChatBubble.test.tsx src/components/task-chat/TaskChatTurn.test.tsx` β€” 31/31 pass. - Manual: open a task thread, confirm the copy / πŸ‘ / πŸ‘Ž cluster shows on the agent bubble summary line before the timestamp, copy works, votes toggle, and the cluster stays on the summary line when the tool history is expanded. Visual change: snapshot baselines are intentionally not updated, per the `doc/design/DECISION-SHEET.md` entry "Per-change snapshot verification demoted to dormant (Jul 13 2026)". ## Risks Low risk. UI-only change scoped to the redesigned task-chat bubble footer. It reuses an existing shared feedback component and existing vote props; no API, schema, or server change. Human and system bubbles are untouched. ## Model Used Claude Opus 4.8 (Anthropic), model id `claude-opus-4-8`, extended thinking enabled, tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 --- ui/src/components/AgentBubbleActionRow.tsx | 49 +++++++---- ui/src/components/TaskChatThread.tsx | 45 ++++++++++ .../task-chat/TaskChatBubble.test.tsx | 59 +++++++++++++ .../components/task-chat/TaskChatBubble.tsx | 25 +++++- .../task-chat/TaskChatBubbleActions.tsx | 50 +++++++++++ .../task-chat/TaskChatThreadView.tsx | 29 ++++++- .../task-chat/TaskChatTurn.test.tsx | 22 +++++ ui/src/components/task-chat/TaskChatTurn.tsx | 85 ++++++++++++------- ui/src/pages/TaskChatLab.tsx | 28 +++++- 9 files changed, 334 insertions(+), 58 deletions(-) create mode 100644 ui/src/components/task-chat/TaskChatBubbleActions.tsx diff --git a/ui/src/components/AgentBubbleActionRow.tsx b/ui/src/components/AgentBubbleActionRow.tsx index 0f41b2f7af1..f761d942aee 100644 --- a/ui/src/components/AgentBubbleActionRow.tsx +++ b/ui/src/components/AgentBubbleActionRow.tsx @@ -40,6 +40,36 @@ export function agentBubbleDateLabel(date: Date | string | undefined): string { return formatShortDate(date); } +/** + * Copy-to-clipboard icon button shared by every agent-bubble footer β€” the + * conference-room {@link AgentBubbleActionRow} and the redesigned task thread's + * {@link TaskChatBubbleActions} (PAP-413). Single-sourced so both footers keep + * identical sizing, radius, and the "copied βœ“" feedback affordance rather than + * re-declaring the same button markup on each surface. + */ +export function BubbleCopyButton({ copyText }: { copyText: string }) { + const [copied, setCopied] = useState(false); + + return ( + + ); +} + /** * Shared agent-bubble action row β€” copy Β· πŸ‘ Β· πŸ‘Ž Β· timestamp Β· β‹― menu. * @@ -79,26 +109,9 @@ export function AgentBubbleActionRow({ menuItems?: ReactNode; className?: string; }) { - const [copied, setCopied] = useState(false); - return (
- + {feedback ? ( { @@ -388,6 +395,43 @@ export function TaskChatThread(props: TaskChatThreadProps) { ); }, [orderedEntries, runs, liveRun, transcriptByRun, linkedRunMetaById, lastCommentIdByRun, hasBrief]); + // Feedback votes keyed by the comment they target (targetType + // "issue_comment"), mirroring IssueChatThread β€” the redesign attaches the + // πŸ‘/πŸ‘Ž state to each agent bubble by its comment id (PAP-413). + const feedbackVoteByTargetId = useMemo(() => { + const map = new Map(); + for (const feedbackVote of feedbackVotes ?? []) { + if (feedbackVote.targetType !== "issue_comment") continue; + map.set(feedbackVote.targetId, feedbackVote.vote); + } + return map; + }, [feedbackVotes]); + + // copy Β· πŸ‘ Β· πŸ‘Ž cluster for an agent bubble's footer line (PAP-413). Human + // and system bubbles get nothing; copy is always available, and the feedback + // buttons render only when the host wired a vote handler. + const renderMessageActions = useCallback( + (item: TaskChatMessageItem) => { + if (item.author !== "agent" || item.optimistic) return null; + return ( + onVote(item.id, vote, options), + } + : null + } + /> + ); + }, + [onVote, feedbackVoteByTargetId, feedbackDataSharingPreference, feedbackTermsUrl], + ); + const renderInteraction = useCallback( (item: TaskChatInteractionItem) => ( : undefined} + renderMessageActions={renderMessageActions} scroll={!isMobile} /> )} diff --git a/ui/src/components/task-chat/TaskChatBubble.test.tsx b/ui/src/components/task-chat/TaskChatBubble.test.tsx index 9b3c0412007..875ee543cbc 100644 --- a/ui/src/components/task-chat/TaskChatBubble.test.tsx +++ b/ui/src/components/task-chat/TaskChatBubble.test.tsx @@ -209,3 +209,62 @@ describe("TaskChatBubble footer line (round 9)", () => { expect(plain).toHaveLength(0); }); }); + +describe("TaskChatBubble footer actions (PAP-413)", () => { + let container: HTMLDivElement; + let root: Root | null = null; + + beforeEach(() => { + container = document.createElement("div"); + document.body.appendChild(container); + root = createRoot(container); + }); + + afterEach(() => { + flushSync(() => root?.unmount()); + root = null; + container.remove(); + }); + + function render(item: TaskChatMessageItem, opts?: { attachedTurn?: ReactNode; actions?: ReactNode }) { + flushSync(() => + root!.render( + + + , + ), + ); + } + + const actions =
actions
; + + it("hands actions to the attached turn (not this wrapper) so they ride the summary row", () => { + render( + { id: "m1", kind: "message", author: "agent", authorName: "CEO", text: "Done.", timestamp: "2:34 PM" }, + { attachedTurn:
2:34 PM Β· Worked
, actions }, + ); + const slot = container.querySelector('[data-testid="task-chat-bubble-attached-turn"]'); + expect(slot).not.toBeNull(); + // The turn owns the footer line; the bubble no longer places actions beside + // it (they ride the turn's `leading` slot, anchored to the summary line). + expect(slot?.querySelector('[data-testid="fake-attached-turn"]')).not.toBeNull(); + expect(slot?.querySelector('[data-testid="fake-actions"]')).toBeNull(); + }); + + it("leads a runless agent reply with actions, timestamp trailing", () => { + render( + { id: "m1", kind: "message", author: "agent", authorName: "CEO", text: "Done.", timestamp: "2:34 PM" }, + { actions }, + ); + expect(container.querySelector('[data-testid="fake-actions"]')).not.toBeNull(); + const stamp = [...container.querySelectorAll("span")].find((el) => el.textContent === "2:34 PM"); + expect(stamp).not.toBeNull(); + }); + + it("omits the actions row entirely when none are supplied (human bubble)", () => { + render({ id: "m1", kind: "message", author: "human", text: "Hi", timestamp: "2:34 PM" }); + expect(container.querySelector('[data-testid="fake-actions"]')).toBeNull(); + const stamp = [...container.querySelectorAll("span")].find((el) => el.textContent === "2:34 PM"); + expect(stamp).not.toBeNull(); + }); +}); diff --git a/ui/src/components/task-chat/TaskChatBubble.tsx b/ui/src/components/task-chat/TaskChatBubble.tsx index af8f71e3bbe..528fbc985c9 100644 --- a/ui/src/components/task-chat/TaskChatBubble.tsx +++ b/ui/src/components/task-chat/TaskChatBubble.tsx @@ -25,6 +25,13 @@ interface TaskChatBubbleProps { * Supplied by TaskChatThreadView when `item.attachedTurn` is set. */ attachedTurn?: ReactNode; + /** + * copy Β· πŸ‘ Β· πŸ‘Ž controls for an agent bubble's footer line (PAP-413). + * Rendered here only for a runless reply (leading the bare timestamp); when + * an attached turn is present it owns these via its `leading` slot instead, + * so this bubble skips them. Human/system bubbles pass nothing. + */ + actions?: ReactNode; } function initialsForName(name: string) { @@ -41,7 +48,7 @@ function initialsForName(name: string) { * bubble with an avatar author header (the agent's assigned icon + name Β· mode * chip); system notices are centered and recede. */ -export function TaskChatBubble({ item, attachedTurn }: TaskChatBubbleProps) { +export function TaskChatBubble({ item, attachedTurn, actions }: TaskChatBubbleProps) { if (item.interstitial) { // Interstitial updates are ephemeral (PAP-361): while streaming the text // lives on the live parent row's line (TaskChatStatusItem.selfTalk), and @@ -146,10 +153,24 @@ export function TaskChatBubble({ item, attachedTurn }: TaskChatBubbleProps) { ) : attachedTurn ? ( // The settled turn takes over the footer line: timestamp + "βœ“ Worked" - // summary, always visible; expanding stretches beneath the bubble. + // summary, always visible; expanding stretches beneath the bubble. The + // copy/πŸ‘/πŸ‘Ž actions (PAP-413) ride the turn's summary row via its + // `leading` slot β€” not this wrapper β€” so they stay anchored to the + // summary line when the tool history expands beneath it.
{attachedTurn}
+ ) : actions ? ( + // Agent reply without run activity: the actions still lead the footer, + // with the always-visible timestamp trailing (PAP-413). +
+ {actions} + {item.timestamp ? ( + + {item.timestamp} + + ) : null} +
) : item.timestamp ? ( // Timestamps are always visible (round 9) β€” no longer hover-revealed. diff --git a/ui/src/components/task-chat/TaskChatBubbleActions.tsx b/ui/src/components/task-chat/TaskChatBubbleActions.tsx new file mode 100644 index 00000000000..33f6c9f179a --- /dev/null +++ b/ui/src/components/task-chat/TaskChatBubbleActions.tsx @@ -0,0 +1,50 @@ +import type { + FeedbackDataSharingPreference, + FeedbackVoteValue, +} from "@paperclipai/shared"; +import { + BubbleCopyButton, + IssueChatFeedbackButtons, +} from "@/components/AgentBubbleActionRow"; + +/** Feedback-vote wiring for an agent bubble, resolved per comment by the host. */ +export interface TaskChatBubbleFeedback { + activeVote: FeedbackVoteValue | null; + sharingPreference: FeedbackDataSharingPreference; + termsUrl: string | null; + onVote: ( + vote: FeedbackVoteValue, + options?: { allowSharing?: boolean; reason?: string }, + ) => Promise; +} + +/** + * Compact copy Β· πŸ‘ Β· πŸ‘Ž cluster prepended to an agent bubble's footer line + * (PAP-413), leading the "βœ“ Worked Β· …" turn summary (or the bare timestamp + * when the reply had no run activity). It reuses the shared + * {@link BubbleCopyButton} and {@link IssueChatFeedbackButtons} so the + * redesigned task thread speaks the same footer language as the conference + * room's {@link AgentBubbleActionRow} without re-declaring their markup; the + * timestamp stays owned by the summary/bubble, so it is not duplicated here. + */ +export function TaskChatBubbleActions({ + copyText, + feedback, +}: { + copyText: string; + feedback?: TaskChatBubbleFeedback | null; +}) { + return ( +
+ + {feedback ? ( + + ) : null} +
+ ); +} diff --git a/ui/src/components/task-chat/TaskChatThreadView.tsx b/ui/src/components/task-chat/TaskChatThreadView.tsx index 9916837b3c0..fa316c9699f 100644 --- a/ui/src/components/task-chat/TaskChatThreadView.tsx +++ b/ui/src/components/task-chat/TaskChatThreadView.tsx @@ -1,6 +1,10 @@ import type { ReactNode } from "react"; import { cn } from "@/lib/utils"; -import type { TaskChatInteractionItem, TaskChatItem } from "./task-chat-model"; +import type { + TaskChatInteractionItem, + TaskChatItem, + TaskChatMessageItem, +} from "./task-chat-model"; import { TaskChatTurn } from "./TaskChatTurn"; import { TaskChatBubble } from "./TaskChatBubble"; import { TaskChatMarker } from "./TaskChatMarker"; @@ -30,6 +34,12 @@ interface TaskChatThreadViewProps { * nothing without it. */ renderBrief?: () => ReactNode; + /** + * Renders the copy/πŸ‘/πŸ‘Ž action cluster prepended to an agent bubble's footer + * line (PAP-413). The live thread binds it to the feedback-vote API; harness + * fixtures omit it and the bubbles render actionless. + */ + renderMessageActions?: (item: TaskChatMessageItem) => ReactNode; className?: string; /** When false, render the list without the scroll container (e.g. previews). */ scroll?: boolean; @@ -40,23 +50,33 @@ function renderItem( onApprovalDecision?: (statusItemId: string, optionId: string) => void, renderInteraction?: (item: TaskChatInteractionItem) => ReactNode, renderBrief?: () => ReactNode, + renderMessageActions?: (item: TaskChatMessageItem) => ReactNode, ) { switch (item.kind) { - case "message": + case "message": { + // Compute the actions once: the bubble renders them for a runless reply + // (footer = actions + timestamp), while an attached turn hands them to + // TaskChatTurn's `leading` slot so they ride the summary line and stay + // put when the tool history expands (PAP-413). The two paths are mutually + // exclusive at runtime, so only one host ever mounts the node. + const actions = renderMessageActions?.(item); return ( renderItem(child, onApprovalDecision)} /> ) : undefined } /> ); + } case "marker": return ; case "thinking": @@ -107,6 +127,7 @@ export function TaskChatThreadView({ onApprovalDecision, renderInteraction, renderBrief, + renderMessageActions, className, scroll = true, }: TaskChatThreadViewProps) { @@ -118,7 +139,9 @@ export function TaskChatThreadView({
) : null} {items.map((item) => ( -
{renderItem(item, onApprovalDecision, renderInteraction, renderBrief)}
+
+ {renderItem(item, onApprovalDecision, renderInteraction, renderBrief, renderMessageActions)} +
))}
); diff --git a/ui/src/components/task-chat/TaskChatTurn.test.tsx b/ui/src/components/task-chat/TaskChatTurn.test.tsx index eec4c9e0833..1ec01cf23bc 100644 --- a/ui/src/components/task-chat/TaskChatTurn.test.tsx +++ b/ui/src/components/task-chat/TaskChatTurn.test.tsx @@ -226,6 +226,28 @@ describe("TaskChatTurn", () => { expect(fold()?.getAttribute("data-folded")).toBe("false"); }); + it("rides the `leading` slot on the header row, anchored above the fold (PAP-413)", () => { + flushSync(() => { + root.render( + actions
} + renderChild={(c) => {c.id}} + />, + ); + }); + const lead = container.querySelector('[data-testid="turn-lead"]'); + expect(lead).not.toBeNull(); + // It sits beside the summary button on the header row β€” NOT inside the + // expandable fold β€” so expanding the tool history can't drag it down. + expect(fold()?.contains(lead!)).toBe(false); + expect(summaryBtn()).not.toBeNull(); + // Expanding still works and the leading slot stays put outside the fold. + flushSync(() => summaryBtn()!.click()); + expect(fold()?.getAttribute("data-folded")).toBe("false"); + expect(fold()?.contains(lead!)).toBe(false); + }); + it("leads the settled summary with the bubble timestamp when attached (round 9)", () => { flushSync(() => { root.render( diff --git a/ui/src/components/task-chat/TaskChatTurn.tsx b/ui/src/components/task-chat/TaskChatTurn.tsx index 314396e5830..d66426a7431 100644 --- a/ui/src/components/task-chat/TaskChatTurn.tsx +++ b/ui/src/components/task-chat/TaskChatTurn.tsx @@ -13,6 +13,13 @@ interface TaskChatTurnProps { * always visible, in the slot the hover-only timestamp used to occupy. */ timestampPrefix?: string; + /** + * Content rendered on the header row, leading the summary line (PAP-413: the + * copy/πŸ‘/πŸ‘Ž action cluster). It sits beside the summary button β€” NOT inside + * the expandable fold β€” so it stays anchored to the summary line when the + * tool history expands beneath it, instead of drifting to the fold's center. + */ + leading?: ReactNode; } /** Metric segments after the label: "38s Β· 3 tools Β· +34 βˆ’3 Β· 12.3k tokens". */ @@ -50,7 +57,7 @@ export function turnSummaryText(summary: TaskChatTurnItem["summary"]): string { * `liveStatus` (harness fixtures) renders its children expanded with no header * and folds when it settles. */ -export function TaskChatTurn({ item, renderChild, timestampPrefix }: TaskChatTurnProps) { +export function TaskChatTurn({ item, renderChild, timestampPrefix, leading }: TaskChatTurnProps) { const parentRow = !item.settled && item.liveStatus != null; // Parent-row live turns and settled turns start as their one-line header; // only the headerless legacy live turn starts expanded. @@ -73,40 +80,52 @@ export function TaskChatTurn({ item, renderChild, timestampPrefix }: TaskChatTur const folded = (item.settled || parentRow) && !open; const SummaryIcon = item.summary.failed ? X : Check; + const header = item.settled ? ( + + ) : parentRow ? ( + // The pill renders the expand button itself, wrapped around only the + // gerund status line β€” the interstitial row above stays outside the + // hover/click target (PAP-376). Without activity there is no + // chevron/button. + setOpen((o) => !o) : undefined} + /> + ) : null; + return (
- {item.settled ? ( - - ) : parentRow ? ( - // The pill renders the expand button itself, wrapped around only the - // gerund status line β€” the interstitial row above stays outside the - // hover/click target (PAP-376). Without activity there is no - // chevron/button. - setOpen((o) => !o) : undefined} - /> - ) : null} + {leading ? ( + // Actions ride the header row (items-center matches them to the summary + // line), and the fold below is a separate sibling β€” so expanding the + // tool history never moves the actions off the summary line (PAP-413). +
+ {leading} + {header} +
+ ) : ( + header + )}
diff --git a/ui/src/pages/TaskChatLab.tsx b/ui/src/pages/TaskChatLab.tsx index ca79e6cb523..cc9590d0dae 100644 --- a/ui/src/pages/TaskChatLab.tsx +++ b/ui/src/pages/TaskChatLab.tsx @@ -8,8 +8,32 @@ import { import { buildScenario } from "@/components/task-chat/task-chat-fixtures"; import { TaskChatThreadView } from "@/components/task-chat/TaskChatThreadView"; import { TaskChatPlanView } from "@/components/task-chat/TaskChatPlanView"; +import { TaskChatBubbleActions } from "@/components/task-chat/TaskChatBubbleActions"; import { TweakPanel } from "@/components/task-chat/TweakPanel"; -import type { TaskChatItem } from "@/components/task-chat/task-chat-model"; +import type { + TaskChatItem, + TaskChatMessageItem, +} from "@/components/task-chat/task-chat-model"; + +/** + * Demo binding for the agent-bubble copy Β· πŸ‘ Β· πŸ‘Ž cluster (PAP-413). The live + * thread wires these to the feedback-vote API; here the votes no-op so the + * harness can show the footer without a control plane. + */ +function labMessageActions(item: TaskChatMessageItem) { + if (item.author !== "agent" || item.optimistic) return null; + return ( + {}, + }} + /> + ); +} /** Replay ticks of selfTalk gap between scripted interstitial updates (~0.7s at 1Γ—). */ const SELF_TALK_GAP_TICKS = 30; @@ -210,7 +234,7 @@ export function TaskChatLab() {
) : (
- +
)}
From 03cfad7ceb200499c7f383493f240ecffaf81203 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 6 Aug 2026 22:18:08 -0500 Subject: [PATCH 009/311] feat(apps): connect Notion through MCP OAuth (#11009) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps give agents governed access to external tools. > - The Apps gallery lists Notion, but the server required manually configured OAuth credentials. > - Notion's hosted MCP server supports OAuth discovery and dynamic client registration. > - Notion also requires HTTPS or a loopback HTTP redirect URI. > - This pull request adds a direct Notion MCP OAuth path with PKCE and reusable dynamic clients. > - It also adds the current Apps UI states for connect and reauthorization. > - The benefit is a secure Notion connection with no manual client credential setup. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Apps gallery, Apps connect route, OAuth token lifecycle, and managed MCP gateway. **Subsystem affected** `server/`, `packages/shared/`, `scripts/`, and `ui/`. **Current behavior** The Notion gallery cards are disabled. The server uses the classic Notion OAuth endpoints and requires operator-supplied client credentials. It does not register an OAuth client from provider metadata. Concurrent refreshes can also replay a rotating refresh token. **Proposed behavior** Enable the Notion Apps flow. Discover OAuth metadata from `https://mcp.notion.com/mcp`. Register and reuse a public RFC 7591 client with PKCE. Require HTTPS or loopback HTTP callbacks. Serialize refreshes, store each rotated refresh token before the new access token can be used, and show a reconnect state for `invalid_grant`. **Reason and benefit** Operators can connect the built-in Notion MCP app without creating or copying OAuth credentials. Paperclip keeps dynamic clients and rotating tokens in the company secret store. **Breaking changes** None. Explicit environment client credentials still take priority. Existing Slack and Linear OAuth endpoint hints remain unchanged. Other OAuth apps remain disabled unless they are allowlisted. **Additional context** PR #10910 is a related, broader Connections v3 wizard replacement. This PR is the focused current Apps flow. The MCP Tool Gateway and Connected Apps items in `ROADMAP.md` cover this planned capability. ## What Changed - Classify all 20 reviewed Notion MCP tools with provider-scoped read and write defaults. - Require approval for selected Notion mutations, including move, duplicate, and convert actions that generic verb matching missed. - Preserve company-scoped connection and catalog resolution for Notion profiles and policies. - Add RFC 7591 dynamic client registration with `token_endpoint_auth_method=none` and mandatory PKCE. - Store the dynamic client ID on the connection and store any returned client secret in the company secret store. - Reuse the registered client for later connects and keep explicit environment credentials as the first choice. - Discover protected-resource and authorization-server metadata from the Notion MCP endpoint. - Add `redirectConstraints: "https-or-loopback-http"` to the generated Notion app definition and shared contract. - Reject non-loopback plain HTTP callbacks before network access with a TLS setup error. - Serialize client registration and token refresh operations within the server process. - Store a rotated refresh token before publishing the refreshed access token. - Treat `invalid_grant` as terminal and move the connection to a clear reauthorization state. - Add focused coverage for registration reuse, callback constraints, refresh rotation, and terminal grants. - Enable the Notion Apps route and add connect, redirect, success, error, and reconnect UI states. - Keep non-allowlisted OAuth apps blocked and cover the UI policy with regression tests. ## Verification - The focused Notion policy integration test passed with embedded PostgreSQL. - The focused 20-tool classification test passed. - The server typecheck passed on the governance head. - `pnpm -r typecheck` passed on the rebased head. - `pnpm --filter @paperclipai/server typecheck` passed after the security follow-up. - `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts -t \u0027DCR|refresh tokens|invalid_grant|abandoned lease\u0027` passed 10 focused security tests. - `pnpm build` passed on the rebased head. - `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts -t 'OAuth|oauth'` passed 14 tests. - `pnpm exec vitest run packages/shared/src/app-definitions.test.ts` passed 5 tests. - The complete server group passed 3,686 tests with 4 skipped. - The complete UI group passed 3,656 tests. - The full local runner found one environment-only CLI failure because this agent runtime injects static AWS credentials into a test that expects `AWS_PROFILE` only. `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest run cli/src/__tests__/secrets.test.ts` passed all 8 tests. - The prior UI verification passed 55 focused tests, `pnpm check:token-gates`, the Storybook build, and review of six 1440 x 1000 screenshots. - OAuth request sequence: protected-resource metadata `GET https://mcp.notion.com/.well-known/oauth-protected-resource/mcp`; authorization metadata `GET https://mcp.notion.com/.well-known/oauth-authorization-server`; dynamic registration `POST https://mcp.notion.com/register`; authorization `GET https://mcp.notion.com/authorize`; token exchange and refresh `POST https://mcp.notion.com/token`; MCP traffic `POST https://mcp.notion.com/mcp`. - The live metadata and registration probe confirmed that Notion accepts HTTPS and loopback HTTP redirects. It rejects a plain HTTP private hostname. - A later QA task owns the full browser consent and managed gateway tool-list dry run against a configured HTTPS deployment. ## Risks - Notion can add tools. Unrecognized names use the generic classifier, and new or changed risky tools stay quarantined after connection activation. - A deployment that uses a private non-loopback hostname must configure HTTPS before it can connect Notion. - Dynamic registration creates a provider-side client. Paperclip reuses it because registration does not provide a standard delete operation. - Refresh coordination uses a database CAS lease across service instances. An unclean crash leaves an uncertain lease and requires reconnect instead of risking refresh-token replay. - The current Apps surface overlaps with PR #10910. Merge order can require a small conflict resolution if that PR lands first. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex on a GPT-5 runtime. The exact deployment ID and context window are not exposed. The runtime used reasoning, repository tools, code execution, and network tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Claude Fable 5 --- doc/connections/CONNECTOR-PLAYBOOK.md | 364 ++++++++ packages/shared/src/app-definitions.test.ts | 10 +- .../shared/src/app-definitions/notion.json | 11 +- packages/shared/src/index.ts | 1 + packages/shared/src/types/app-definition.ts | 3 +- .../shared/src/validators/app-definition.ts | 2 +- scripts/ingest-app-definitions.mjs | 4 +- scripts/screenshot-notion-connect-flow.mjs | 96 +++ .../src/__tests__/tool-access-service.test.ts | 569 ++++++++++++- server/src/services/tool-access.ts | 805 ++++++++++++++++-- ui/src/App.tsx | 3 +- ui/src/pages/apps/AppDetail.test.tsx | 64 ++ ui/src/pages/apps/AppDetail.tsx | 3 +- ui/src/pages/apps/AppsConnect.test.tsx | 319 ++++++- ui/src/pages/apps/AppsConnect.tsx | 415 +++++++-- ui/src/pages/apps/Browse.test.tsx | 15 +- ui/src/pages/apps/Browse.tsx | 20 +- ui/src/pages/apps/Connections.tsx | 4 +- ui/src/pages/apps/app-connect-policy.test.ts | 28 + ui/src/pages/apps/app-connect-policy.ts | 14 + .../pages/apps/app-detail/AdvancedPanel.tsx | 36 +- ui/src/pages/apps/app-detail/SetupPanel.tsx | 11 +- ui/src/pages/apps/store-cards.tsx | 4 + .../stories/notion-connect-flow.stories.tsx | 177 ++++ 24 files changed, 2820 insertions(+), 158 deletions(-) create mode 100644 scripts/screenshot-notion-connect-flow.mjs create mode 100644 ui/src/pages/apps/app-connect-policy.test.ts create mode 100644 ui/src/pages/apps/app-connect-policy.ts create mode 100644 ui/storybook/stories/notion-connect-flow.stories.tsx diff --git a/doc/connections/CONNECTOR-PLAYBOOK.md b/doc/connections/CONNECTOR-PLAYBOOK.md index 2c954e7090f..d1f94027983 100644 --- a/doc/connections/CONNECTOR-PLAYBOOK.md +++ b/doc/connections/CONNECTOR-PLAYBOOK.md @@ -183,6 +183,93 @@ Recommended defaults for a new catalog entry: If a gallery card cannot pass this path against a real vendor, de-list it or mark it unavailable until the missing auth, transport, or governance dependency is fixed. +## MCP-Direct Connections (Hosted MCP + OAuth) + +Many vendors now expose an official hosted MCP server whose authorization +server is discovered from the MCP endpoint itself, instead of documenting fixed +OAuth URLs. For these connectors the manifest's `oauth` block is a hint at +most; the broker resolves endpoints at connect time: + +1. `GET ` unauthenticated returns `401` with a `WWW-Authenticate` + header naming the protected-resource metadata URL (RFC 9728). +2. `GET /.well-known/oauth-protected-resource[/]` names the + authorization server(s). +3. `GET /.well-known/oauth-authorization-server` (RFC 8414) yields + `authorization_endpoint`, `token_endpoint`, and β€” when the vendor supports + dynamic registration β€” `registration_endpoint`. + +The broker implements this in `discoverOAuthEndpoints` +(`server/src/services/tool-access.ts`); connections with transport +`mcp_remote` and `auth: "oauth"` prefer discovered endpoints over manifest +hints. Keep manifest hints current anyway so reviewers can read the expected +endpoints without running discovery. + +### Dynamic client registration (RFC 7591) + +Vendors whose authorization server advertises a `registration_endpoint` and +supports public clients (`token_endpoint_auth_method: "none"` plus PKCE S256) +need **no pre-provisioned OAuth app at all**. At first connect the broker +registers a client on the fly and stores it on the connection: + +- Registration request: `client_name` `Paperclip ()`, + `redirect_uris` = the instance's own callback, `grant_types` + `["authorization_code", "refresh_token"]`, `response_types` `["code"]`, + `token_endpoint_auth_method` `"none"`. +- The issued `client_id` is persisted in the connection's OAuth config and any + issued `client_secret` becomes a `company_secrets` ref. The registered + client is **reused** for every later authorize/refresh on that connection β€” + re-registering orphans prior grants on providers that bind grants to the + client. +- Env-registered clients always win: when + `PAPERCLIP_TOOL_OAUTH__CLIENT_ID/_SECRET` are configured, the + broker uses them (`customer` ownership) and skips registration. List both + `customer` and `dcr` in the method's `ownershipModes` when the vendor + supports both. + +**DCR needs neither Paperclip ID nor Paperclip Connect.** DCR is always +instance-local (ratified in the PAP-14828 connector-service spec, section 10 +item 8.4: "DCR is always instance-local; the service has no DCR involvement"). +Each instance registers its own public client with the vendor and uses its own +`/api/tools/oauth/callback` redirect. **Cloud-hosted and self-hosted instances +use the SAME path** β€” the only per-instance difference is the hostname inside +the redirect URI. `id.paperclip.ing` authenticates operators only and never +holds resource tokens; `connect.paperclip.ing` is a fallback only for +providers that genuinely require a pre-registered public redirect, which a DCR +provider by definition does not. + +### Redirect-URI constraints + +Vendors restrict what `redirect_uris` a dynamic client may register. Record +the probed constraint in the `AppDefinition` `redirectConstraints` field and +enforce it before starting OAuth. The first supported value is +`https-or-loopback-http` (Notion's rule): HTTPS on any host β€” public or +private β€” or plain HTTP only on loopback (`localhost`, `*.localhost`, `::1`, +`127.0.0.0/8`). A plain-HTTP non-loopback origin fails fast with +`oauth_redirect_origin_unsupported` ("This provider requires an HTTPS or +loopback origin. Configure TLS before connecting.") and a pointer to the TLS +deployment docs, instead of a confusing vendor-side `invalid_redirect_uri`. +Probe the constraint with real registration attempts before writing the +manifest β€” the redirect-URI rule and browser-reachability are independent +axes; a private HTTPS host can be fine even when plain HTTP is not. + +### Documentation standards for every connection doc + +Every connection doc β€” playbook appendix, proposal, or user-facing doc β€” +must include all three of the following (they are part of the template below): + +1. **Service involvement statement.** Say explicitly whether Paperclip ID or + Paperclip Connect participates in the flow. For RFC 7591 DCR providers the + answer is always: neither β€” DCR is instance-local and cloud vs self-hosted + use the same path. +2. **Sequence diagram + exact endpoints.** A sequence diagram of how the + connection works, and the exact paths/endpoints used for auth: authorize, + token, registration (if DCR), and the Paperclip callback. Keep mermaid + sources next to the doc; do not put semicolons inside mermaid message text + (they parse as statement separators). +3. **Administrator setup instructions.** Step-by-step: what (if anything) an + admin must register β€” callback URLs? client credentials? nothing, for DCR? β€” + where to register it, and how to verify the connection works end to end. + ## Template Copy this section into a connector proposal or implementation issue. @@ -208,6 +295,25 @@ Copy this section into a connector proposal or implementation issue. - Secret storage: company_secrets refs only - Revocation behavior: +## Connection Flow (mandatory) + +- Sequence diagram: +- Auth endpoints (exact paths): + - Authorize: + - Token: + - Registration (if DCR): + - Discovery (.well-known), if any: + - Paperclip callback: `/api/tools/oauth/callback` (or n/a) +- Redirect constraints (probed): none / https-or-loopback-http / requires-public-redirect +- Paperclip ID / Paperclip Connect involvement: <"none β€” DCR is instance-local; cloud and self-hosted use the same path" for RFC 7591 providers; otherwise name the role> + +## Administrator Setup (mandatory) + +- What the admin must register (callback URLs? client credentials? nothing for DCR?): +- Where to register it: +- Instance prerequisites (TLS, base URL, feature flags): +- How to verify the connection works: + ## Resource Filters - Required filters: @@ -363,3 +469,261 @@ Linear's real-vendor evidence belongs in [PAP-12373](/PAP/issues/PAP-12373). The Connector proposals now target the versioned `AppDefinition` contract in `packages/shared/src/types/app-definition.ts`. Seed data is one JSON file per provider under `packages/shared/src/app-definitions/`; regenerate Wave 1 with `pnpm connections:ingest-app-definitions`. The generator parses all 99 captured templates, validates required placeholders, OAuth ownership modes, and API-key placement, and produces deterministic output for review. FIRST-30 remains authoritative for `riskTier` and `requiredResourceFilters`; managed ownership modes stay data-visible but runtime-hidden until availability is injected. +## Appendix: Notion Dry Run (MCP-Direct With DCR) + +This dry run applies the template to Notion, the first MCP-direct connector to +ship with RFC 7591 dynamic client registration (PAP-16637; server +implementation PAP-16649, PR #11009). Unlike the Linear appendix, every +endpoint and constraint below comes from a live request log, not vendor docs +alone. + +### Vendor + +- App key: `notion` +- App name: Notion +- First-30 classification: MCP-direct. Notion ships an official hosted MCP + server; its ~20 `notion-*` tools map directly to Paperclip grants. +- Reason for classification: no shim or wrapper needed β€” the hosted server + speaks Streamable HTTP, which `server/src/services/mcp-http.ts` already + handles. The FIRST-30 matrix's "thin wrapper for block/database policy" is + explicitly deferred; v1 enforcement is gateway policy plus filters-as-config. +- Security tier: S3 β€” workspace content read/write, but no payments, tenant + admin, or production infrastructure. +- Plugin needed: No. Gallery card, OAuth connect, filters, catalog, profiles, + policies, and audit cover the UX. + +### Transport And Auth + +- Transport: `mcp_remote` +- Endpoint: `https://mcp.notion.com/mcp` (Streamable HTTP; `/sse` fallback exists) +- Auth mode: OAuth, endpoints resolved by discovery (RFC 9728 β†’ RFC 8414), + public client via RFC 7591 DCR with PKCE S256 mandatory. +- Ownership modes: `dcr` (default, zero setup) and `customer` + (env-registered classic integration via + `PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID/_SECRET`, which always wins when set). +- Token behavior: access tokens last ~8 h (`expires_in` authoritative). + Refresh tokens **rotate on every refresh** β€” the old token is invalidated + (at most 2 valid per grant) and replaying a stale one can revoke the whole + grant, so the broker persists the rotated token before publishing the new + access token and serializes refresh per connection. Absolute expiry 180 + days, inactivity expiry 30 days. `invalid_grant` on refresh is terminal: + clear tokens, require re-auth, never retry. +- Secret storage: access/refresh tokens and any DCR `client_secret` are + `company_secrets` refs; the DCR `client_id` persists on the connection and + is reused β€” re-registering would orphan prior grants. +- Revocation behavior: disabling or revoking the connection removes + `notion-*` tools from agent sessions and denies brokered execution on the + next gateway check. + +### Connection Flow (mandatory) + +Paperclip ID / Paperclip Connect involvement: **none β€” DCR is instance-local** +(PAP-14828 spec section 10 item 8.4); **cloud-hosted and self-hosted use the +same path**. The only per-instance difference is the hostname in the redirect +URI. + +Auth endpoints (exact paths, from the live discovery chain): + +| Role | Endpoint | +| --- | --- | +| MCP server | `https://mcp.notion.com/mcp` | +| Protected-resource metadata (RFC 9728) | `https://mcp.notion.com/.well-known/oauth-protected-resource/mcp` | +| AS metadata (RFC 8414) | `https://mcp.notion.com/.well-known/oauth-authorization-server` | +| Authorize | `https://mcp.notion.com/authorize` | +| Token (exchange + refresh) | `https://mcp.notion.com/token` | +| Registration (RFC 7591 DCR) | `https://mcp.notion.com/register` | +| Paperclip connect (wizard) | `POST /api/companies/:companyId/tools/apps/connect` | +| Paperclip OAuth start | `POST /api/tools/oauth/:connectionId/start` | +| Paperclip callback | `GET /api/tools/oauth/callback` | + +Redirect constraints (probed): `https-or-loopback-http`. + +```mermaid +sequenceDiagram + autonumber + actor U as User's browser + participant UI as Paperclip UI
/PAP/apps/connect?source=notion + participant S as Paperclip instance server
(cloud or self-hosted β€” same path) + participant M as mcp.notion.com
(MCP server + OAuth AS) + participant N as Notion web
(app.notion.com, notion.com) + + U->>UI: Click "Connect" (deep link ?source=notion) + UI->>S: POST /companies/:id/tools/apps/connect { appKey: "notion" } + S->>M: GET /.well-known/oauth-protected-resource (RFC 9728) + M-->>S: authorization_servers β†’ mcp.notion.com + S->>M: GET /.well-known/oauth-authorization-server (RFC 8414) + M-->>S: authorize / token / registration endpoints + alt First connect on this instance (no stored client, no env client) + S->>M: POST registration_endpoint (RFC 7591 DCR, public client, PKCE-only) + M-->>S: client_id (persisted, REUSED for every later connect) + else Client already known + S->>S: Reuse stored DCR client_id (or env-registered client if configured) + end + S-->>UI: auth.startUrl (authorize URL + PKCE S256 challenge + state) + UI->>U: Redirect browser to startUrl + U->>M: GET /authorize?client_id + code_challenge + state + M->>N: 302 to app.notion.com/install-integration + N->>N: notion.com/login (only if signed out) + N-->>U: Consent page: pick workspace, approve integration + U->>S: 302 to GET /api/tools/oauth/callback?code&state (instance's OWN callback) + S->>M: POST token_endpoint (code + code_verifier) + M-->>S: access_token (~8 h) + rotating refresh_token + S->>S: Store tokens as company_secrets refs (server-side only) + S-->>U: Redirect to wizard actions/review step (?oauth=connected) + Note over S,M: Later: agent runs reach notion-* tools via the managed MCP gateway.
Server refreshes ahead of use β€” each refresh ROTATES the refresh token. +``` + +### Dry-Run Request Log (PAP-16649, 2026-08-06/07) + +The verified request sequence for a first connect: + +1. `GET https://mcp.notion.com/mcp` β†’ `401` with `WWW-Authenticate` naming + `https://mcp.notion.com/.well-known/oauth-protected-resource/mcp`. +2. `GET https://mcp.notion.com/.well-known/oauth-protected-resource/mcp` β†’ + `200`; authorization server `https://mcp.notion.com`, scope `default`. +3. `GET https://mcp.notion.com/.well-known/oauth-authorization-server` β†’ + `200`; `/authorize`, `/token`, `/register`; `token_endpoint_auth_method` + `none` supported; PKCE `S256` supported. +4. `POST https://mcp.notion.com/register` (RFC 7591). +5. Browser `GET https://mcp.notion.com/authorize` β†’ Notion consent + (`app.notion.com/install-integration`, `notion.com/login` if signed out). +6. `POST https://mcp.notion.com/token` for code exchange and every refresh. +7. `POST https://mcp.notion.com/mcp` for MCP traffic. + +Redirect-URI probes against `/register`: + +| Probed `redirect_uris` value | Result | +| --- | --- | +| `http://paperclip-dev:3100/api/tools/oauth/callback` | 400 `invalid_redirect_uri` β€” "Redirect URI must use HTTPS unless it is a loopback HTTP URI" | +| `https://paperclip-dev:3100/api/tools/oauth/callback` | Accepted β€” private host is fine over HTTPS | +| `http://localhost:3100/api/tools/oauth/callback` | Accepted | +| `http://127.0.0.1:3100/api/tools/oauth/callback` | Accepted | + +Hence `redirectConstraints: "https-or-loopback-http"` in `notion.json`, and +the broker's fail-fast `oauth_redirect_origin_unsupported` error for +plain-HTTP non-loopback origins. + +### Administrator Setup (mandatory) + +- What the admin must register: **nothing**. Notion's authorization server + supports RFC 7591 DCR, so the instance registers its own public client on + first connect. No Notion integration, no client credentials, no callback + registration, no Paperclip ID or Paperclip Connect involvement. +- Optional escape hatch: to use a pre-registered classic Notion integration + instead, set `PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID` and + `PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_SECRET`; the env client always takes + precedence (`customer` ownership). +- Instance prerequisites: the instance base URL must be HTTPS on any host or + loopback HTTP (Notion's redirect-URI rule). A plain-HTTP non-loopback origin + gets "This provider requires an HTTPS or loopback origin. Configure TLS + before connecting." β€” add TLS first (e.g. a tailscale cert, as + paperclip-dev did). The `enableApps` experimental setting must be on for + `/apps/*` routes. The connecting user must be allowed to install + integrations in their Notion workspace. +- How to verify: visit `/PAP/apps/connect?source=notion`, complete the Notion + consent flow, and land on the wizard's actions step listing `notion-*` + tools. Then confirm an agent run sees Notion tools through the runtime MCP + gateway and that a write call (e.g. `notion-create-pages`) opens an + ask-first action request. + +### Resource Filters + +- Required filters: workspace, page, database (per FIRST-30). +- Optional filters: object type, database/data-source scope. +- Write-enabling filters: workspace plus page/database scope for + create/update. +- Enforced by: gateway policy plus filters-as-config in v1; the FIRST-30 + "thin wrapper for block/database policy" is explicitly deferred. Notion-side + scoping also applies β€” the consent step lets the user share only selected + pages/databases with the integration. + +### Manifest Sketch + +The shipped `packages/shared/src/app-definitions/notion.json` (regenerate via +`pnpm connections:ingest-app-definitions`): + +```json +{ + "schemaVersion": 1, + "slug": "notion", + "name": "Notion", + "description": "Read and update pages in your Notion workspace.", + "urlPatterns": ["https://mcp.notion.com/*"], + "methods": [ + { + "key": "mcp-oauth", + "transport": "mcp_remote", + "auth": "oauth", + "ownershipModes": ["customer", "dcr"], + "defaults": { "serverUrl": "https://mcp.notion.com/mcp" }, + "riskTier": "S3", + "requiredResourceFilters": ["workspace", "page", "database"] + } + ], + "redirectConstraints": "https-or-loopback-http" +} +``` + +### Actions + +Notion's hosted server exposes ~20 `notion-*` tools. Representative risk +classes below; the full catalog review with per-tool defaults is PAP-16652 +(P4), and changed-action quarantine applies as usual. + +| Tool | Risk | Default status | Filters | Approval default | Audit fields | Negative case | +| --- | --- | --- | --- | --- | --- | --- | +| `notion-search` | read | active after catalog review; plan-gated by Notion (needs Notion AI) β€” may list but fail at call time | workspace | allow when profile includes Notion reads | query summary, result count | Ungranted agent cannot invoke. | +| `notion-fetch` | read | active after catalog review | workspace, page, database | allow when profile includes Notion reads | page/database id | Fetch outside shared pages fails Notion-side and is audited. | +| `notion-create-pages` | write | active only after review; changed versions quarantined | workspace, page, database | ask-first by default | parent id, title hash, created page id | Missing workspace/page filter denies. | +| `notion-update-page` | write | active only after review; changed versions quarantined | workspace, page | ask-first by default | page id, redaction summary | Revoked connection blocks retry. | +| `notion-query-data-sources` | read | active after catalog review | workspace, database | allow when profile includes Notion reads | data-source id, result count | Granted agent cannot query a disallowed database. | + +No destructive Notion action ships in the first pass; any future +delete/archive/bulk action starts quarantined pending SecurityEngineer review. + +### Wizard Path + +1. Operator opens `/PAP/apps/connect?source=notion` (or the Notion gallery + card β†’ Connect). The deep link POSTs connect immediately and redirects the + browser to `auth.startUrl`. +2. Operator completes Notion consent (workspace picker β†’ approve). +3. Notion redirects to the instance's own `GET /api/tools/oauth/callback`; + Paperclip exchanges the code, stores token material in `company_secrets`, + and returns the operator to the wizard (`?oauth=connected`). +4. Operator confirms resource filters and default ask-first writes. +5. Paperclip runs health check and catalog refresh; `notion-*` tools appear + on the actions step. +6. Write actions stay ask-first until the operator approves calls or creates + narrow trust rules. + +Error state: on a plain-HTTP non-loopback instance, step 1 fails fast with +the TLS guidance error above β€” the operator never reaches Notion. + +### Governance Defaults + +- Default profile: Notion read actions for the selected scope; writes opt-in. +- Policy defaults: ask-first for `notion-create-pages`, `notion-update-page`, + and comment writes; block any unreviewed destructive action. +- Quarantine: new or schema-changed write actions receive + `quarantineReason: "pending_review"` and are hidden from agent tool lists. +- Rate limits: per-connection search/fetch budget to protect vendor quota. +- Audit: log connect, DCR registration, config/filter changes, grant changes, + action requests, allowed/denied calls, token refresh failures, revoke, and + catalog quarantine events. + +### Validation Hook + +End-to-end evidence belongs to PAP-16654 (P6) and the PAP-12373 matrix: + +- Zero-setup OAuth connect succeeds on + `https://paperclip-dev.tail29c1aa.ts.net/PAP/apps/connect?source=notion` + with no pre-provisioned OAuth env vars (proves DCR). +- Catalog discovery lists `notion-*` tools; new/changed risky actions are + quarantined. +- An agent run sees Notion tools through the managed runtime MCP gateway. +- `notion-create-pages` opens ask-first review and executes only after + approval. +- Revocation removes Notion tools and blocks execution. +- Audit rows prove actor, run/issue context, connection, tool, decision, + reason code, and outcome. + diff --git a/packages/shared/src/app-definitions.test.ts b/packages/shared/src/app-definitions.test.ts index 8e93700c441..489f5e0595c 100644 --- a/packages/shared/src/app-definitions.test.ts +++ b/packages/shared/src/app-definitions.test.ts @@ -4,9 +4,11 @@ import { appDefinitionsSchema } from "./validators/app-definition.js"; describe("AppDefinition catalog",()=>{ it("validates all Wave 1 definitions",()=>expect(()=>appDefinitionsSchema.parse(APP_DEFINITIONS)).not.toThrow()); it("contains twelve reviewed providers",()=>expect(APP_DEFINITIONS.map((app)=>app.slug)).toEqual(["zapier","github","slack","notion","linear","google-sheets","context7","oauth-generic","api-key-generic","sentry","vercel","anthropic"])); - it.each([ - ["notion",["read_content","update_content"]], - ["linear",["read","write"]], - ])("preserves required OAuth scopes for %s",(slug,scopes)=>expect(APP_DEFINITIONS.find((app)=>app.slug===slug)?.methods[0]?.defaults?.scopesHint).toEqual(scopes)); + it("uses discovery-first Notion MCP OAuth metadata",()=>{ + const notion=APP_DEFINITIONS.find((app)=>app.slug==="notion"); + expect(notion?.redirectConstraints).toBe("https-or-loopback-http"); + expect(notion?.methods[0]?.defaults).toEqual({serverUrl:"https://mcp.notion.com/mcp"}); + }); + it("preserves required Linear OAuth scopes",()=>expect(APP_DEFINITIONS.find((app)=>app.slug==="linear")?.methods[0]?.defaults?.scopesHint).toEqual(["read","write"])); it("enforces method and field invariants",()=>{for(const app of APP_DEFINITIONS)for(const method of app.methods){if(method.auth==="api_key")expect(method.keyPlacement).toBeTruthy();if(method.auth==="oauth")expect(method.ownershipModes.length).toBeGreaterThan(0);for(const field of method.credentialFields??[])if(field.required&&field.type!=="checkbox")expect(field.placeholder).toBeTruthy()}}); }); diff --git a/packages/shared/src/app-definitions/notion.json b/packages/shared/src/app-definitions/notion.json index 8e95443e8d6..fe5ab22b2f0 100644 --- a/packages/shared/src/app-definitions/notion.json +++ b/packages/shared/src/app-definitions/notion.json @@ -24,13 +24,7 @@ ], "whenToUse": "Use the provider-hosted connection for the quickest setup.", "defaults": { - "serverUrl": "https://mcp.notion.com/mcp", - "authorizationEndpoint": "https://api.notion.com/v1/oauth/authorize", - "tokenEndpoint": "https://api.notion.com/v1/oauth/token", - "scopesHint": [ - "read_content", - "update_content" - ] + "serverUrl": "https://mcp.notion.com/mcp" }, "guidanceMd": "Connect Notion for workspace content. Share only the pages and databases agents should use.", "riskTier": "S3", @@ -40,5 +34,6 @@ "database" ] } - ] + ], + "redirectConstraints": "https-or-loopback-http" } diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index cbb2b9e49bf..512f54aeeb8 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -1393,6 +1393,7 @@ export type { AppDefinition, ConnectionMethodDef, FieldDef, + OAuthRedirectConstraints, QuotaWindow, ProviderQuotaResult, } from "./types/index.js"; diff --git a/packages/shared/src/types/app-definition.ts b/packages/shared/src/types/app-definition.ts index abbc3abfc4c..69a7c8606b7 100644 --- a/packages/shared/src/types/app-definition.ts +++ b/packages/shared/src/types/app-definition.ts @@ -1,5 +1,6 @@ import type { ToolConnectionOwnership, ToolConnectionTransport } from "./tool-access.js"; export type AppCategory = "ai"|"analytics"|"commerce"|"communication"|"content"|"data"|"developer"|"productivity"|"other"; +export type OAuthRedirectConstraints = "https-or-loopback-http"; export interface FieldDef { key:string; label:string; type:"text"|"password"|"textarea"|"datetime"|"select"|"checkbox"; required?:boolean; placeholder?:string; helperMd?:string; secret?:boolean; prefix?:string; validation?:{pattern?:string;maxLength?:number}; options?:Array<{value:string;label:string}> } export interface ConnectionMethodDef { key:string; transport:ToolConnectionTransport; auth:"oauth"|"api_key"|"none"; ownershipModes:ToolConnectionOwnership[]; whenToUse:string; defaults?:{serverUrl?:string;discoveryUrl?:string|null;serviceHost?:string;templateKey?:string;authorizationEndpoint?:string;tokenEndpoint?:string;metadataUrl?:string;scopesHint?:string[]}; tenantFields?:FieldDef[]; extensionFields?:FieldDef[]; credentialFields?:FieldDef[]; keyPlacement?:{location:"header"|"query"|"body_json"|"env";name:string;prefix?:string|null}; guidanceMd:string; consoleLinks?:{register?:string;keys?:string;settings?:string;docs?:string}; warnings?:string[]; variants?:Array<{key:string;label:string;whenToUse:string;tenantFields?:FieldDef[]}>; riskTier:"S1"|"S2"|"S3"|"S4"; requiredResourceFilters?:string[] } -export interface AppDefinition { schemaVersion:1; slug:string; name:string; description:string; categories:AppCategory[]; featured?:boolean; branding:{logoUrl:string;darkLogoUrl?:string;backgroundColor?:string;accentColor?:string}; urlPatterns:string[]; docsUrl?:string; methods:ConnectionMethodDef[]; suggestable?:boolean; availability?:{available:boolean;reason?:string;robotEmail?:string}; ownershipAvailability?:Partial> } +export interface AppDefinition { schemaVersion:1; slug:string; name:string; description:string; categories:AppCategory[]; featured?:boolean; branding:{logoUrl:string;darkLogoUrl?:string;backgroundColor?:string;accentColor?:string}; urlPatterns:string[]; docsUrl?:string; redirectConstraints?:OAuthRedirectConstraints; methods:ConnectionMethodDef[]; suggestable?:boolean; availability?:{available:boolean;reason?:string;robotEmail?:string}; ownershipAvailability?:Partial> } diff --git a/packages/shared/src/validators/app-definition.ts b/packages/shared/src/validators/app-definition.ts index 4aa559b2ae5..11ca80f0f0e 100644 --- a/packages/shared/src/validators/app-definition.ts +++ b/packages/shared/src/validators/app-definition.ts @@ -2,5 +2,5 @@ import { z } from "zod"; import { toolConnectionOwnershipSchema, toolConnectionTransportSchema } from "./tool-access.js"; const field=z.object({key:z.string().min(1),label:z.string().min(1),type:z.enum(["text","password","textarea","datetime","select","checkbox"]),required:z.boolean().optional(),placeholder:z.string().optional(),helperMd:z.string().optional(),secret:z.boolean().optional(),prefix:z.string().optional()}).superRefine((v,c)=>{if(v.required&&v.type!=="checkbox"&&!v.placeholder)c.addIssue({code:"custom",message:"Required fields need placeholders",path:["placeholder"]})}); export const connectionMethodDefSchema=z.object({key:z.string().min(1),transport:toolConnectionTransportSchema,auth:z.enum(["oauth","api_key","none"]),ownershipModes:z.array(toolConnectionOwnershipSchema).min(1),whenToUse:z.string().min(1),defaults:z.object({serverUrl:z.string().url().optional(),discoveryUrl:z.string().url().nullable().optional(),serviceHost:z.string().optional(),templateKey:z.string().optional(),authorizationEndpoint:z.string().url().optional(),tokenEndpoint:z.string().url().optional(),metadataUrl:z.string().url().optional(),scopesHint:z.array(z.string()).optional()}).optional(),tenantFields:z.array(field).optional(),extensionFields:z.array(field).optional(),credentialFields:z.array(field).optional(),keyPlacement:z.object({location:z.enum(["header","query","body_json","env"]),name:z.string().min(1),prefix:z.string().nullable().optional()}).optional(),guidanceMd:z.string().min(1),consoleLinks:z.object({register:z.string().url().optional(),keys:z.string().url().optional(),settings:z.string().url().optional(),docs:z.string().url().optional()}).optional(),warnings:z.array(z.string()).optional(),variants:z.array(z.object({key:z.string(),label:z.string(),whenToUse:z.string(),tenantFields:z.array(field).optional()})).optional(),riskTier:z.enum(["S1","S2","S3","S4"]),requiredResourceFilters:z.array(z.string()).optional()}).superRefine((v,c)=>{if(v.auth==="api_key"&&!v.keyPlacement)c.addIssue({code:"custom",message:"API-key methods require keyPlacement",path:["keyPlacement"]})}); -export const appDefinitionSchema=z.object({schemaVersion:z.literal(1),slug:z.string().regex(/^[a-z0-9]+(?:-[a-z0-9]+)*$/),name:z.string().min(1),description:z.string().min(1),categories:z.array(z.enum(["ai","analytics","commerce","communication","content","data","developer","productivity","other"])).min(1),featured:z.boolean().optional(),branding:z.object({logoUrl:z.string().url(),darkLogoUrl:z.string().url().optional(),backgroundColor:z.string().optional(),accentColor:z.string().optional()}),urlPatterns:z.array(z.string()),docsUrl:z.string().url().optional(),methods:z.array(connectionMethodDefSchema).min(1),suggestable:z.boolean().optional(),availability:z.object({available:z.boolean(),reason:z.string().optional(),robotEmail:z.string().optional()}).optional(),ownershipAvailability:z.object({platform_shared:z.boolean().optional(),platform_provisioned:z.boolean().optional(),customer:z.boolean().optional(),dcr:z.boolean().optional()}).optional()}); +export const appDefinitionSchema=z.object({schemaVersion:z.literal(1),slug:z.string().regex(/^[a-z0-9]+(?:-[a-z0-9]+)*$/),name:z.string().min(1),description:z.string().min(1),categories:z.array(z.enum(["ai","analytics","commerce","communication","content","data","developer","productivity","other"])).min(1),featured:z.boolean().optional(),branding:z.object({logoUrl:z.string().url(),darkLogoUrl:z.string().url().optional(),backgroundColor:z.string().optional(),accentColor:z.string().optional()}),urlPatterns:z.array(z.string()),docsUrl:z.string().url().optional(),redirectConstraints:z.enum(["https-or-loopback-http"]).optional(),methods:z.array(connectionMethodDefSchema).min(1),suggestable:z.boolean().optional(),availability:z.object({available:z.boolean(),reason:z.string().optional(),robotEmail:z.string().optional()}).optional(),ownershipAvailability:z.object({platform_shared:z.boolean().optional(),platform_provisioned:z.boolean().optional(),customer:z.boolean().optional(),dcr:z.boolean().optional()}).optional()}); export const appDefinitionsSchema=z.array(appDefinitionSchema).superRefine((v,c)=>{const s=new Set();v.forEach((a,i)=>{if(s.has(a.slug))c.addIssue({code:"custom",message:"Duplicate slug",path:[i,"slug"]});s.add(a.slug)})}); diff --git a/scripts/ingest-app-definitions.mjs b/scripts/ingest-app-definitions.mjs index 498a37594fe..05fc773ff06 100644 --- a/scripts/ingest-app-definitions.mjs +++ b/scripts/ingest-app-definitions.mjs @@ -7,7 +7,7 @@ const apps=[ ["zapier","Zapier","Reach thousands of apps through your Zapier account.","productivity","zapier.com",["https://mcp.zapier.com/*"],method("mcp-key","mcp_remote","api_key",{serverUrl:"https://mcp.zapier.com/api/mcp"},"S3","Create a Zapier MCP connection, then paste its token here.",{credentialFields:[field("authorization","Zapier MCP token","Paste your Zapier token")],keyPlacement:{location:"header",name:"Authorization",prefix:"Bearer "}})], ["github","GitHub","Read code and pull requests, and coordinate repository work.","developer","github.com",["https://api.githubcopilot.com/mcp/*"],method("mcp-key","mcp_remote","api_key",{serverUrl:"https://api.githubcopilot.com/mcp/"},"S3","Create a fine-grained token limited to the repositories agents should use.",{credentialFields:[field("authorization","GitHub token","github_pat_...")],keyPlacement:{location:"header",name:"Authorization",prefix:"Bearer "},requiredResourceFilters:["organization","repository"]})], ["slack","Slack","Search channels and coordinate team communication.","communication","slack.com",["https://mcp.slack.com/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.slack.com/mcp",authorizationEndpoint:"https://slack.com/oauth/v2/authorize",tokenEndpoint:"https://slack.com/api/oauth.v2.access",scopesHint:["channels:read","chat:write","search:read"]},"S3","Connect a Slack workspace and limit access to the channels agents need.",{requiredResourceFilters:["workspace","channel"]})], -["notion","Notion","Read and update pages in your Notion workspace.","content","notion.so",["https://mcp.notion.com/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.notion.com/mcp",authorizationEndpoint:"https://api.notion.com/v1/oauth/authorize",tokenEndpoint:"https://api.notion.com/v1/oauth/token",scopesHint:["read_content","update_content"]},"S3","Connect Notion for workspace content. Share only the pages and databases agents should use.",{requiredResourceFilters:["workspace","page","database"]})], +["notion","Notion","Read and update pages in your Notion workspace.","content","notion.so",["https://mcp.notion.com/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.notion.com/mcp"},"S3","Connect Notion for workspace content. Share only the pages and databases agents should use.",{requiredResourceFilters:["workspace","page","database"]}),{redirectConstraints:"https-or-loopback-http"}], ["linear","Linear","Create, update, and read Linear issues.","productivity","linear.app",["https://mcp.linear.app/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.linear.app/mcp",authorizationEndpoint:"https://linear.app/oauth/authorize",tokenEndpoint:"https://api.linear.app/oauth/token",scopesHint:["read","write"]},"S2","Register a Linear OAuth app and add Paperclip's redirect URI before connecting.",{requiredResourceFilters:["workspace","team","project"]})], ["google-sheets","Google Sheets","Read and update selected spreadsheets.","data","sheets.google.com",["https://docs.google.com/spreadsheets/*","https://sheets.google.com/*"],method("local","local_stdio","none",{templateKey:"paperclip.google-sheets"},"S3","Share each spreadsheet with the Paperclip robot email, then paste the sheet links.",{requiredResourceFilters:["spreadsheet"]})], ["context7","Context7","Look up current documentation for software libraries.","developer","context7.com",["https://mcp.context7.com/*"],method("mcp","mcp_remote","none",{serverUrl:"https://mcp.context7.com/mcp"},"S1","Connect Context7 to give agents current library documentation.")], @@ -16,7 +16,7 @@ const apps=[ ["sentry","Sentry","Investigate errors, releases, and production issues.","developer","sentry.io",["https://mcp.sentry.dev/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.sentry.dev/mcp",discoveryUrl:"https://sentry.io/.well-known/oauth-authorization-server"},"S2","Connect the Sentry organization and projects agents need for incident work.",{requiredResourceFilters:["organization","project","environment"]})], ["vercel","Vercel","Inspect projects, deployments, and runtime logs.","developer","vercel.com",["https://mcp.vercel.com/*"],method("mcp-oauth","mcp_remote","oauth",{serverUrl:"https://mcp.vercel.com/mcp"},"S3","Connect the Vercel team and projects agents should operate.",{requiredResourceFilters:["team","project","environment"]})], ["anthropic","Anthropic","Use Anthropic APIs with a restricted key.","ai","anthropic.com",["https://api.anthropic.com/*"],method("api-key","rest_api","api_key",{serviceHost:"api.anthropic.com"},"S3","Create a key in the Anthropic Console and rotate it if it has been exposed.",{credentialFields:[field("apiKey","API key","sk-ant-api03-...")],keyPlacement:{location:"header",name:"x-api-key"}})], -].map(([slug,name,description,category,domain,urlPatterns,m])=>({schemaVersion:1,slug,name,description,categories:[category],featured:["zapier","github","slack","notion","linear"].includes(slug),branding:{logoUrl:favicon(domain)},urlPatterns,methods:[m]})); +].map(([slug,name,description,category,domain,urlPatterns,m,extra={}])=>({schemaVersion:1,slug,name,description,categories:[category],featured:["zapier","github","slack","notion","linear"].includes(slug),branding:{logoUrl:favicon(domain)},urlPatterns,methods:[m],...extra})); const parseTableRow=(line)=>line.slice(1,-1).split("|").map((cell)=>cell.trim()); const parseCapture=(fileName)=>{ const markdown=fs.readFileSync(path.join(corpus,fileName),"utf8"); diff --git a/scripts/screenshot-notion-connect-flow.mjs b/scripts/screenshot-notion-connect-flow.mjs new file mode 100644 index 00000000000..bfa11f819c1 --- /dev/null +++ b/scripts/screenshot-notion-connect-flow.mjs @@ -0,0 +1,96 @@ +#!/usr/bin/env node +// Capture the PAP-16650 Notion connection-flow Storybook stories. +// Usage: node scripts/screenshot-notion-connect-flow.mjs + +import http from "node:http"; +import path from "node:path"; +import fs from "node:fs/promises"; +import { chromium } from "@playwright/test"; + +async function main() { + const [, , staticDir, outDir] = process.argv; + if (!staticDir || !outDir) { + console.error( + "usage: node scripts/screenshot-notion-connect-flow.mjs ", + ); + process.exit(1); + } + + await fs.mkdir(outDir, { recursive: true }); + const absStaticDir = path.resolve(staticDir); + const server = http.createServer(async (req, res) => { + try { + let urlPath = decodeURIComponent((req.url || "/").split("?")[0]); + if (urlPath.endsWith("/")) urlPath += "iframe.html"; + const filePath = path.resolve(absStaticDir, `.${urlPath}`); + if (!filePath.startsWith(absStaticDir + path.sep) && filePath !== absStaticDir) { + res.writeHead(403); + res.end("Forbidden"); + return; + } + + const buf = await fs.readFile(filePath); + const ext = path.extname(filePath).toLowerCase(); + const mime = + { + ".html": "text/html; charset=utf-8", + ".js": "application/javascript", + ".css": "text/css", + ".json": "application/json", + ".svg": "image/svg+xml", + ".png": "image/png", + ".woff": "font/woff", + ".woff2": "font/woff2", + ".map": "application/json", + }[ext] || "application/octet-stream"; + res.writeHead(200, { "content-type": mime }); + res.end(buf); + } catch (error) { + res.writeHead(404); + res.end(String(error)); + } + }); + + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Failed to start screenshot server"); + const baseUrl = `http://127.0.0.1:${address.port}/iframe.html`; + + const stories = [ + ["browse-entry", "01-browse-entry.png"], + ["connect-entry", "02-connect-entry.png"], + ["in-flight", "03-in-flight.png"], + ["connected", "04-connected.png"], + ["connect-error", "05-connect-error.png"], + ["reconnect-required", "06-reconnect-required.png"], + ]; + + const browser = await chromium.launch(); + try { + for (const [story, file] of stories) { + const context = await browser.newContext({ + viewport: { width: 1440, height: 1000 }, + deviceScaleFactor: 2, + colorScheme: "light", + }); + const page = await context.newPage(); + await page.goto( + `${baseUrl}?id=apps-notion-mcp-connect-flow-pap-16650--${story}&viewMode=story`, + { waitUntil: "networkidle" }, + ); + await page.waitForTimeout(500); + const out = path.join(outDir, file); + await page.screenshot({ path: out, fullPage: true }); + console.log("wrote", out); + await context.close(); + } + } finally { + await browser.close(); + server.close(); + } +} + +main().catch((error) => { + console.error(error); + process.exit(1); +}); diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index d3e8f799a4c..7a938d83563 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -3240,11 +3240,214 @@ describeEmbeddedPostgres("tool access service", () => { await expect(db.select().from(toolOauthStates)).resolves.toHaveLength(0); }); - it("refreshes expired OAuth access tokens before remote app calls", async () => { + it("discovers Notion MCP OAuth metadata, registers one public client, and reuses it", async () => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_SECRET", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_CLIENT_ID", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_CLIENT_SECRET", ""); + const company = await createCompany(db); + const service = toolAccessService(db); + const connected = await service.connectGalleryApp(company.id, { + galleryKey: "notion", + name: "Notion DCR", + }); + const redirectUri = "https://paperclip-dev.tail29c1aa.ts.net/api/tools/oauth/callback"; + const registrationBodies: Array> = []; + const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation(async (url, init) => { + const href = String(url); + if (href === "https://mcp.notion.com/.well-known/oauth-protected-resource/mcp") { + return mcpHttpResponse({ + authorization_servers: ["https://mcp.notion.com"], + scopes_supported: ["default"], + }); + } + if (href === "https://mcp.notion.com/.well-known/oauth-authorization-server") { + return mcpHttpResponse({ + issuer: "https://mcp.notion.com", + authorization_endpoint: "https://mcp.notion.com/authorize", + token_endpoint: "https://mcp.notion.com/token", + registration_endpoint: "https://mcp.notion.com/register", + code_challenge_methods_supported: ["S256"], + token_endpoint_auth_methods_supported: ["none"], + }); + } + if (href === "https://mcp.notion.com/register") { + expect(init?.method).toBe("POST"); + registrationBodies.push(JSON.parse(String(init?.body)) as Record); + return mcpHttpResponse({ + client_id: "notion-dcr-client", + client_secret: "notion-dcr-secret", + redirect_uris: [redirectUri], + grant_types: ["authorization_code", "refresh_token"], + response_types: ["code"], + token_endpoint_auth_method: "none", + }); + } + throw new Error(`unexpected fetch ${href}`); + }); + + const [first, concurrent] = await Promise.all([ + service.startOAuth(company.id, connected.connectionId, { + redirectUri, + actor: { actorType: "user", actorId: "board" }, + }), + service.startOAuth(company.id, connected.connectionId, { + redirectUri, + actor: { actorType: "user", actorId: "board" }, + }), + ]); + + expect(new URL(first.authorizationUrl).origin).toBe("https://mcp.notion.com"); + expect(new URL(concurrent.authorizationUrl).searchParams.get("client_id")).toBe("notion-dcr-client"); + expect(registrationBodies).toEqual([{ + client_name: "Paperclip (paperclip-dev.tail29c1aa.ts.net)", + redirect_uris: [redirectUri], + grant_types: ["authorization_code", "refresh_token"], + response_types: ["code"], + token_endpoint_auth_method: "none", + }]); + + fetchMock.mockClear(); + const reused = await service.startOAuth(company.id, connected.connectionId, { + redirectUri, + actor: { actorType: "user", actorId: "board" }, + }); + expect(new URL(reused.authorizationUrl).searchParams.get("client_id")).toBe("notion-dcr-client"); + expect(fetchMock).not.toHaveBeenCalled(); + + const [connection] = await db.select().from(toolConnections).where(eq(toolConnections.id, connected.connectionId)); + expect(connection).toMatchObject({ ownership: "dcr" }); + expect(connection.config).toMatchObject({ + oauth: { + provider: "notion", + clientId: "notion-dcr-client", + clientRegistrationSource: "dcr", + clientTokenEndpointAuthMethod: "none", + clientRedirectUri: redirectUri, + registrationUrl: "https://mcp.notion.com/register", + }, + }); + expect(connection.credentialSecretRefs).toEqual([ + expect.objectContaining({ configPath: "oauth.client_secret", required: false }), + ]); + expect(JSON.stringify(connection.config)).not.toContain("notion-dcr-secret"); + }); + + it.each([ + [ + "a confidential token endpoint auth method", + { client_id: "notion-dcr-client", token_endpoint_auth_method: "client_secret_basic" }, + "token_endpoint_auth_method", + ], + [ + "a different redirect URI", + { client_id: "notion-dcr-client", redirect_uris: ["https://attacker.example/callback"] }, + "redirect_uris", + ], + [ + "a reduced grant set", + { client_id: "notion-dcr-client", grant_types: ["authorization_code"] }, + "grant_types", + ], + [ + "missing response types", + { client_id: "notion-dcr-client", response_types: undefined }, + "response_types", + ], + [ + "an oversized client id", + { client_id: "x".repeat(4_097) }, + "client_id", + ], + ])("rejects DCR responses that return %s", async (_label, registrationResponse, field) => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_SECRET", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_CLIENT_ID", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_CLIENT_SECRET", ""); + const company = await createCompany(db); + const service = toolAccessService(db); + const connected = await service.connectGalleryApp(company.id, { + galleryKey: "notion", + name: `Notion invalid DCR ${field}`, + }); + const redirectUri = "https://paperclip-dev.tail29c1aa.ts.net/api/tools/oauth/callback"; + vi.spyOn(globalThis, "fetch").mockImplementation(async (url) => { + const href = String(url); + if (href === "https://mcp.notion.com/.well-known/oauth-protected-resource/mcp") { + return mcpHttpResponse({ authorization_servers: ["https://mcp.notion.com"] }); + } + if (href === "https://mcp.notion.com/.well-known/oauth-authorization-server") { + return mcpHttpResponse({ + authorization_endpoint: "https://mcp.notion.com/authorize", + token_endpoint: "https://mcp.notion.com/token", + registration_endpoint: "https://mcp.notion.com/register", + code_challenge_methods_supported: ["S256"], + token_endpoint_auth_methods_supported: ["none"], + }); + } + if (href === "https://mcp.notion.com/register") { + return mcpHttpResponse({ + client_id: "notion-dcr-client", + redirect_uris: [redirectUri], + grant_types: ["authorization_code", "refresh_token"], + response_types: ["code"], + token_endpoint_auth_method: "none", + ...registrationResponse, + }); + } + throw new Error(`unexpected fetch ${href}`); + }); + + await expect(service.startOAuth(company.id, connected.connectionId, { + redirectUri, + actor: { actorType: "user", actorId: "board" }, + })).rejects.toMatchObject({ + status: 502, + details: { + code: "oauth_dcr_response_invalid", + field, + }, + }); + + const [connection] = await db + .select() + .from(toolConnections) + .where(eq(toolConnections.id, connected.connectionId)); + expect(connection.ownership).not.toBe("dcr"); + expect((connection.config.oauth as Record | undefined)?.clientId).toBeUndefined(); + }); + + it("fails fast when Notion DCR is attempted from a non-loopback HTTP origin", async () => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_NOTION_CLIENT_ID", ""); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_CLIENT_ID", ""); + const company = await createCompany(db); + const service = toolAccessService(db); + const connected = await service.connectGalleryApp(company.id, { + galleryKey: "notion", + name: "Notion invalid origin", + }); + const fetchMock = vi.spyOn(globalThis, "fetch"); + + await expect(service.startOAuth(company.id, connected.connectionId, { + redirectUri: "http://paperclip-dev:3100/api/tools/oauth/callback", + actor: { actorType: "user", actorId: "board" }, + })).rejects.toMatchObject({ + status: 422, + message: "This provider requires an HTTPS or loopback origin. Configure TLS before connecting.", + details: expect.objectContaining({ + code: "oauth_redirect_origin_unsupported", + docsPath: "docs/deploy", + }), + }); + expect(fetchMock).not.toHaveBeenCalled(); + }); + + it("leases rotating OAuth refresh tokens across service instances before concurrent remote app calls", async () => { vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); const company = await createCompany(db); const service = toolAccessService(db); + const concurrentService = toolAccessService(db); const connect = await service.connectGalleryApp(company.id, { galleryKey: "slack", name: "Slack refresh" }); const start = await service.startOAuth(company.id, connect.connectionId, { @@ -3252,6 +3455,7 @@ describeEmbeddedPostgres("tool access service", () => { actor: { actorType: "user", actorId: "board" }, }); const state = new URL(start.authorizationUrl).searchParams.get("state")!; + let refreshCallCount = 0; vi.spyOn(globalThis, "fetch").mockImplementation(async (url, init) => { const href = String(url); if (href === "https://slack.com/api/oauth.v2.access") { @@ -3270,6 +3474,8 @@ describeEmbeddedPostgres("tool access service", () => { } expect(body.get("grant_type")).toBe("refresh_token"); expect(body.get("refresh_token")).toBe("refresh-token"); + refreshCallCount += 1; + await new Promise((resolve) => setTimeout(resolve, 10)); return { ok: true, json: async () => ({ @@ -3311,9 +3517,14 @@ describeEmbeddedPostgres("tool access service", () => { }) .where(eq(toolConnections.id, connect.connectionId)); - const health = await service.checkHealth(connect.connectionId); + const [health, concurrentHealth] = await Promise.all([ + service.checkHealth(connect.connectionId), + concurrentService.checkHealth(connect.connectionId), + ]); expect(health.connection.healthStatus).toBe("ok"); + expect(concurrentHealth.connection.healthStatus).toBe("ok"); + expect(refreshCallCount).toBe(1); const fetchCalls = vi.mocked(globalThis.fetch).mock.calls; const mcpCalls = fetchCalls.filter(([url]) => String(url) === "https://mcp.slack.com/mcp"); expect(mcpCalls.at(-1)?.[1]?.headers).toEqual(expect.objectContaining({ Authorization: "Bearer new-access-token" })); @@ -3336,6 +3547,231 @@ describeEmbeddedPostgres("tool access service", () => { ])); }); + it("treats invalid_grant as terminal without replaying a rotated refresh token", async () => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); + const company = await createCompany(db); + const service = toolAccessService(db); + const connect = await service.connectGalleryApp(company.id, { galleryKey: "slack", name: "Slack invalid grant" }); + const start = await service.startOAuth(company.id, connect.connectionId, { + redirectUri: "http://paperclip.test/api/tools/oauth/callback", + actor: { actorType: "user", actorId: "board" }, + }); + const state = new URL(start.authorizationUrl).searchParams.get("state")!; + const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation(async (url) => { + const href = String(url); + if (href === "https://slack.com/api/oauth.v2.access") { + return mcpHttpResponse({ + ok: true, + access_token: "expired-access-token", + refresh_token: "single-use-refresh-token", + expires_in: 3600, + token_type: "Bearer", + }); + } + if (href === "https://mcp.slack.com/mcp") { + return mcpHttpResponse({ jsonrpc: "2.0", id: "paperclip-catalog-refresh", result: { tools: [] } }); + } + throw new Error(`unexpected fetch ${href}`); + }); + await service.completeOAuthCallback({ + state, + code: "oauth-code", + redirectUri: "http://paperclip.test/api/tools/oauth/callback", + actor: { actorType: "user", actorId: "board" }, + }); + const [connected] = await db.select().from(toolConnections).where(eq(toolConnections.id, connect.connectionId)); + await db.update(toolConnections).set({ + config: { + ...connected.config, + oauth: { + ...(connected.config.oauth as Record), + expiresAt: "2000-01-01T00:00:00.000Z", + }, + }, + }).where(eq(toolConnections.id, connect.connectionId)); + + let refreshCallCount = 0; + fetchMock.mockImplementation(async (url, init) => { + const href = String(url); + if (href === "https://slack.com/api/oauth.v2.access") { + const body = init?.body as URLSearchParams; + expect(body.get("grant_type")).toBe("refresh_token"); + expect(body.get("refresh_token")).toBe("single-use-refresh-token"); + refreshCallCount += 1; + await new Promise((resolve) => setTimeout(resolve, 10)); + return { + ok: false, + status: 400, + headers: { get: () => null }, + json: async () => ({ error: "invalid_grant", error_description: "Refresh token was already used" }), + } as unknown as Response; + } + throw new Error(`unexpected fetch ${href}`); + }); + + const results = await Promise.allSettled([ + service.checkHealth(connect.connectionId), + service.checkHealth(connect.connectionId), + ]); + expect(results).toEqual([ + expect.objectContaining({ status: "rejected" }), + expect.objectContaining({ status: "rejected" }), + ]); + for (const result of results) { + expect(result.status === "rejected" ? result.reason : null).toMatchObject({ + details: expect.objectContaining({ code: "oauth_reauthorization_required" }), + }); + } + expect(refreshCallCount).toBe(1); + const [reauthorizationRequired] = await db + .select() + .from(toolConnections) + .where(eq(toolConnections.id, connect.connectionId)); + expect(reauthorizationRequired).toMatchObject({ + status: "draft", + enabled: false, + healthStatus: "error", + }); + expect(reauthorizationRequired.credentialSecretRefs.map((ref) => ref.configPath)).not.toContain("oauth.access_token"); + expect(reauthorizationRequired.credentialSecretRefs.map((ref) => ref.configPath)).not.toContain("oauth.refresh_token"); + expect(reauthorizationRequired.credentialRefs.map((ref) => ref.name)).not.toContain("oauth.access_token"); + }); + + it("does not disable a connection when invalid_grant used a superseded refresh-token version", async () => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); + const company = await createCompany(db); + const service = toolAccessService(db); + const connect = await service.connectGalleryApp(company.id, { + galleryKey: "slack", + name: "Slack stale invalid grant", + }); + const start = await service.startOAuth(company.id, connect.connectionId, { + redirectUri: "http://paperclip.test/api/tools/oauth/callback", + actor: { actorType: "user", actorId: "board" }, + }); + const state = new URL(start.authorizationUrl).searchParams.get("state")!; + const fetchMock = vi.spyOn(globalThis, "fetch").mockImplementation(async (url) => { + const href = String(url); + if (href === "https://slack.com/api/oauth.v2.access") { + return mcpHttpResponse({ + ok: true, + access_token: "expired-access-token", + refresh_token: "submitted-refresh-token", + expires_in: 3600, + token_type: "Bearer", + }); + } + if (href === "https://mcp.slack.com/mcp") { + return mcpHttpResponse({ jsonrpc: "2.0", id: "paperclip-catalog-refresh", result: { tools: [] } }); + } + throw new Error(`unexpected fetch ${href}`); + }); + await service.completeOAuthCallback({ + state, + code: "oauth-code", + redirectUri: "http://paperclip.test/api/tools/oauth/callback", + actor: { actorType: "user", actorId: "board" }, + }); + const [connected] = await db + .select() + .from(toolConnections) + .where(eq(toolConnections.id, connect.connectionId)); + const refreshRef = connected.credentialSecretRefs.find((ref) => ref.configPath === "oauth.refresh_token")!; + await db.update(toolConnections).set({ + config: { + ...connected.config, + oauth: { + ...(connected.config.oauth as Record), + expiresAt: "2000-01-01T00:00:00.000Z", + }, + }, + }).where(eq(toolConnections.id, connect.connectionId)); + + fetchMock.mockImplementation(async (url, init) => { + const href = String(url); + if (href === "https://slack.com/api/oauth.v2.access") { + const body = init?.body as URLSearchParams; + expect(body.get("refresh_token")).toBe("submitted-refresh-token"); + await secretService(db).rotate(refreshRef.secretId, { value: "newer-refresh-token" }); + return { + ok: false, + status: 400, + headers: { get: () => null }, + json: async () => ({ error: "invalid_grant" }), + } as unknown as Response; + } + throw new Error(`unexpected fetch ${href}`); + }); + + await expect(service.checkHealth(connect.connectionId)).rejects.toMatchObject({ + status: 502, + details: expect.objectContaining({ code: "oauth_refresh_superseded" }), + }); + const [preserved] = await db + .select() + .from(toolConnections) + .where(eq(toolConnections.id, connect.connectionId)); + expect(preserved).toMatchObject({ status: "active", enabled: false }); + expect(preserved.credentialSecretRefs.map((ref) => ref.configPath)).toEqual(expect.arrayContaining([ + "oauth.access_token", + "oauth.refresh_token", + ])); + expect((preserved.config.oauth as Record).refreshLease).toBeUndefined(); + }); + + it("fails closed instead of replaying a refresh token after an abandoned lease", async () => { + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_ID", "slack-client-id"); + vi.stubEnv("PAPERCLIP_TOOL_OAUTH_SLACK_CLIENT_SECRET", "slack-client-secret"); + const company = await createCompany(db); + const service = toolAccessService(db); + const fixture = await createOAuthConnection(db, company.id); + const refreshSecret = await secretService(db).create(company.id, { + provider: "local_encrypted", + name: `OAuth refresh ${randomUUID()}`, + key: `oauth.refresh.${randomUUID()}`, + value: "possibly-consumed-refresh-token", + }); + await db.insert(companySecretBindings).values({ + companyId: company.id, + secretId: refreshSecret.id, + targetType: "tool_connection", + targetId: fixture.connection.id, + configPath: "oauth.refresh_token", + }); + await db.update(toolConnections).set({ + config: { + ...fixture.connection.config, + oauth: { + ...(fixture.connection.config.oauth as Record), + expiresAt: "2000-01-01T00:00:00.000Z", + refreshLease: { + id: "abandoned-refresh", + expiresAt: "2000-01-01T00:00:00.000Z", + }, + }, + }, + credentialSecretRefs: [ + ...fixture.connection.credentialSecretRefs, + { + secretId: refreshSecret.id, + versionSelector: "latest", + configPath: "oauth.refresh_token", + required: false, + label: "OAuth refresh token", + }, + ], + }).where(eq(toolConnections.id, fixture.connection.id)); + const fetchMock = vi.spyOn(globalThis, "fetch"); + + await expect(service.checkHealth(fixture.connection.id)).rejects.toMatchObject({ + status: 502, + details: expect.objectContaining({ code: "oauth_refresh_outcome_unknown" }), + }); + expect(fetchMock).not.toHaveBeenCalled(); + }); + it("uses OAuth client credentials for shared machine-to-machine MCP connections", async () => { vi.stubEnv("PAPERCLIP_TOOL_OAUTH_M2M_CLIENT_ID", "m2m-client-id"); vi.stubEnv("PAPERCLIP_TOOL_OAUTH_M2M_CLIENT_SECRET", "m2m-client-secret"); @@ -4650,6 +5086,99 @@ describeEmbeddedPostgres("tool access service", () => { }); }); + it("resolves Notion reads as allowed, mutations as ask-first, and denies cross-company use", async () => { + const company = await createCompany(db); + const otherCompany = await createCompany(db); + const agent = await createAgent(db, company.id); + const service = toolAccessService(db); + const [application] = await db.insert(toolApplications).values({ + companyId: company.id, + applicationKey: `app-gallery:notion:${randomUUID()}`, + name: "Notion workspace", + type: "mcp_http", + status: "draft", + metadata: { sourceTemplateKey: "notion", galleryKey: "notion" }, + }).returning(); + const [connection] = await db.insert(toolConnections).values({ + companyId: company.id, + applicationId: application.id, + name: "Notion workspace", + uid: `notion/${randomUUID()}`, + transport: "mcp_remote", + authKind: "oauth", + status: "draft", + enabled: false, + config: { + url: "https://mcp.notion.com/mcp", + sourceTemplateKey: "notion", + quarantineNewEntries: true, + }, + transportConfig: { url: "https://mcp.notion.com/mcp" }, + healthStatus: "ok", + }).returning(); + mockToolsList([ + { name: "notion-fetch", annotations: { readOnlyHint: true } }, + // These two mutations do not contain a generic create/update verb. + { name: "notion-move-pages" }, + { name: "notion-duplicate-page", annotations: { readOnlyHint: true } }, + ]); + + const refresh = await service.refreshCatalog(connection.id); + const fetchEntry = refresh.catalog.find((entry) => entry.toolName === "notion-fetch")!; + const moveEntry = refresh.catalog.find((entry) => entry.toolName === "notion-move-pages")!; + const duplicateEntry = refresh.catalog.find((entry) => entry.toolName === "notion-duplicate-page")!; + expect(refresh.catalog).toEqual(expect.arrayContaining([ + expect.objectContaining({ id: fetchEntry.id, riskLevel: "read", isReadOnly: true }), + expect.objectContaining({ id: moveEntry.id, riskLevel: "write", isWrite: true }), + expect.objectContaining({ id: duplicateEntry.id, riskLevel: "write", isWrite: true }), + ])); + + await expect(service.finishGalleryAppConnection(otherCompany.id, connection.id, { + enabledCatalogEntryIds: [fetchEntry.id], + askFirstCatalogEntryIds: [], + access: "all_agents", + })).rejects.toMatchObject({ status: 404 }); + + await service.finishGalleryAppConnection(company.id, connection.id, { + enabledCatalogEntryIds: [fetchEntry.id, moveEntry.id, duplicateEntry.id], + askFirstCatalogEntryIds: [moveEntry.id, duplicateEntry.id], + access: "all_agents", + }); + + const policyService = toolAccessPolicyService(db); + const decide = (catalogEntryId: string, toolName: string) => policyService.decide({ + companyId: company.id, + actor: { actorType: "agent", actorId: agent.id }, + request: { connectionId: connection.id, catalogEntryId, toolName, arguments: {} }, + }); + await expect(decide(fetchEntry.id, fetchEntry.toolName)).resolves.toMatchObject({ + decision: "allow", + reasonCode: "allow_profile", + }); + await expect(decide(moveEntry.id, moveEntry.toolName)).resolves.toMatchObject({ + decision: "require_approval", + reasonCode: "requires_approval_policy", + }); + await expect(decide(duplicateEntry.id, duplicateEntry.toolName)).resolves.toMatchObject({ + decision: "require_approval", + reasonCode: "requires_approval_policy", + }); + + await expect(policyService.decide({ + companyId: otherCompany.id, + actor: { actorType: "user", actorId: "other-board" }, + request: { + connectionId: connection.id, + catalogEntryId: fetchEntry.id, + toolName: fetchEntry.toolName, + arguments: {}, + }, + })).resolves.toMatchObject({ + decision: "deny", + reasonCode: "deny_missing_tool", + }); + }); + it("rolls back gallery app finish when a later write fails after clearing profile state", async () => { const company = await createCompany(db); const service = toolAccessService(db); @@ -6348,4 +6877,40 @@ describe("classifyRisk", () => { expect(risk("list_items", { writeHint: true })).toBe("write"); expect(risk("list_items", { readOnlyHint: false })).toBe("write"); }); + + it("classifies the reviewed Notion MCP catalog with provider-scoped defaults", () => { + const notionRisk = (name: string, annotations?: Record) => + classifyRisk({ name, annotations }, "notion"); + const readTools = [ + "notion-search", + "notion-fetch", + "notion-query-data-sources", + "notion-query-database-view", + "notion-query-meeting-notes", + "notion-get-comments", + "notion-get-teams", + "notion-get-users", + "notion-get-async-task", + ]; + const writeTools = [ + "notion-create-pages", + "notion-update-page", + "notion-convert-page-to-skill", + "notion-move-pages", + "notion-duplicate-page", + "notion-create-database", + "notion-create-folder", + "notion-update-data-source", + "notion-create-view", + "notion-update-view", + "notion-create-comment", + ]; + + for (const toolName of readTools) expect(notionRisk(toolName)).toBe("read"); + for (const toolName of writeTools) { + expect(notionRisk(toolName, { readOnlyHint: true })).toBe("write"); + } + expect(notionRisk("notion-delete-page")).toBe("destructive"); + expect(classifyRisk({ name: "move_pages" })).toBe("read"); + }); }); diff --git a/server/src/services/tool-access.ts b/server/src/services/tool-access.ts index 0abcf380190..a399f9fba86 100644 --- a/server/src/services/tool-access.ts +++ b/server/src/services/tool-access.ts @@ -128,16 +128,42 @@ type ActorInfo = { const ACTIVE_BROKER_RUN_STATUSES = new Set(["running"]); const REMOTE_HTTP_REDIRECT_STATUSES = new Set([301, 302, 303, 307, 308]); const MAX_REMOTE_HTTP_REDIRECTS = 5; +const MAX_OAUTH_DCR_CLIENT_ID_LENGTH = 4_096; +const MAX_OAUTH_DCR_CLIENT_SECRET_LENGTH = 16_384; +const OAUTH_REFRESH_LEASE_MS = 120_000; +const OAUTH_REFRESH_LEASE_WAIT_MS = 30_000; +const OAUTH_REFRESH_LEASE_POLL_MS = 25; type OAuthProviderEndpoints = { provider: string; scopes: string[]; authorizationUrl: string; tokenUrl: string; + registrationUrl?: string | null; + codeChallengeMethodsSupported?: string[]; + tokenEndpointAuthMethodsSupported?: string[]; grantType?: "authorization_code" | "client_credentials"; metadataUrl?: string | null; }; +const oauthRegistrationFlights = new Map>(); + +async function oauthSingleFlight( + flights: Map>, + key: string, + operation: () => Promise, +): Promise { + const existing = flights.get(key) as Promise | undefined; + if (existing) return existing; + const pending = operation(); + flights.set(key, pending); + try { + return await pending; + } finally { + if (flights.get(key) === pending) flights.delete(key); + } +} + type ToolAccessServiceOptions = { deploymentMode?: DeploymentMode; deploymentExposure?: DeploymentExposure; @@ -1210,23 +1236,63 @@ function verbMatches(toolName: string, verbs: string): boolean { return new RegExp(`\\b(${verbs})\\b|(^|[:._-])(${verbs})([:._-]|$)`).test(normalized); } -export function classifyRisk(tool: McpToolDescriptor): ToolRiskLevel { +const NOTION_READ_TOOLS = new Set([ + "notion-fetch", + "notion-get-async-task", + "notion-get-comments", + "notion-get-teams", + "notion-get-users", + "notion-query-data-sources", + "notion-query-database-view", + "notion-query-meeting-notes", + "notion-search", +]); + +const NOTION_WRITE_TOOLS = new Set([ + "notion-convert-page-to-skill", + "notion-create-comment", + "notion-create-database", + "notion-create-folder", + "notion-create-pages", + "notion-create-view", + "notion-duplicate-page", + "notion-move-pages", + "notion-update-data-source", + "notion-update-page", + "notion-update-view", +]); + +function normalizedProviderToolName(toolName: string): string { + return toolName + .replace(/([a-z0-9])([A-Z])/g, "$1-$2") + .toLowerCase() + .replace(/[:._-]+/g, "-"); +} + +export function classifyRisk(tool: McpToolDescriptor, sourceTemplateKey?: string | null): ToolRiskLevel { const annotations = tool.annotations ?? {}; if (annotations.destructiveHint === true || annotations.destructive === true) return "destructive"; + const normalizedToolName = normalizedProviderToolName(tool.name); + // Notion's hosted MCP catalog contains mutations whose names do not use one + // of the generic create/update/delete verbs (move, duplicate, and convert). + // Keep all reviewed tools explicit so provider changes are visible in code, + // while an annotation may still escalate a known read to a write. + if (sourceTemplateKey === "notion" && NOTION_WRITE_TOOLS.has(normalizedToolName)) return "write"; if (annotations.readOnlyHint === false || annotations.writeHint === true) return "write"; + if (sourceTemplateKey === "notion" && NOTION_READ_TOOLS.has(normalizedToolName)) return "read"; if (verbMatches(tool.name, "delete|remove|destroy|unpublish")) return "destructive"; if (verbMatches(tool.name, "create|update|write|set|send|publish|post|mutate|mark|archive")) return "write"; return "read"; } -function descriptorHash(tool: McpToolDescriptor): string { +function descriptorHash(tool: McpToolDescriptor, riskLevel: ToolRiskLevel): string { return stableHash({ name: tool.name, title: tool.title ?? null, description: tool.description ?? null, inputSchema: tool.inputSchema ?? {}, annotations: tool.annotations ?? {}, - riskLevel: classifyRisk(tool), + riskLevel, }); } @@ -1247,6 +1313,24 @@ function sanitizeHttpFailure(error: unknown): { status: ToolConnectionHealthStat code: "oauth_refresh_missing", }; } + if (code === "oauth_reauthorization_required") { + return { + status: "error", + message: "OAuth authorization expired. Reconnect this app to continue.", + code: "oauth_reauthorization_required", + }; + } + if ( + code === "oauth_refresh_in_progress" + || code === "oauth_refresh_superseded" + || code === "oauth_refresh_outcome_unknown" + ) { + return { + status: "error", + message: error.message, + code, + }; + } if (code === "binding_missing" || code === "secret_deleted" || code === "secret_inactive" || code === "version_missing") { return { status: "missing_secret", @@ -1288,6 +1372,9 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} const policySvc = toolAccessPolicyService(db); const now = options.now ?? (() => new Date()); const runtimeSupervisor = createToolRuntimeSupervisor(db, options); + // This map only removes duplicate work inside one service instance. The + // database refresh lease below is the cross-process serialization boundary. + const oauthRefreshFlights = new Map>(); function allowPrivateRemoteEndpoints() { return options.deploymentMode !== "authenticated" || options.deploymentExposure !== "public"; @@ -1725,7 +1812,11 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} ?? readConfigString(broker, "credentialConfigPath") ?? readConfigString(broker, "secretConfigPath"); const configuredName = readConfigString(broker, "parentCredentialName") ?? readConfigString(broker, "credentialName"); - const secretCandidates = connection.credentialSecretRefs.filter((ref) => ref.configPath !== "oauth.access_token" && ref.configPath !== "oauth.refresh_token"); + const secretCandidates = connection.credentialSecretRefs.filter((ref) => + ref.configPath !== "oauth.access_token" + && ref.configPath !== "oauth.refresh_token" + && ref.configPath !== "oauth.client_secret" + ); const secretRef = configuredPath ? connection.credentialSecretRefs.find((ref) => ref.configPath === configuredPath) : secretCandidates.find((ref) => ref.configPath === "credentials.deploy_token") @@ -2798,8 +2889,11 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} provider: endpoints.provider, authorizationUrl: endpoints.authorizationUrl, tokenUrl: endpoints.tokenUrl, + registrationUrl: endpoints.registrationUrl ?? null, metadataUrl: endpoints.metadataUrl ?? null, scopes: endpoints.scopes, + codeChallengeMethodsSupported: endpoints.codeChallengeMethodsSupported ?? [], + tokenEndpointAuthMethodsSupported: endpoints.tokenEndpointAuthMethodsSupported ?? [], grantType: endpoints.grantType ?? "authorization_code", discoveredAt: new Date().toISOString(), }, @@ -2946,9 +3040,12 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} let quarantinedCount = 0; const quarantineOnRefresh = shouldQuarantineNewEntries(connection) && connection.status === "active"; const safeDefault = asRecord(connection.config).safeDefault === true; + const sourceTemplateKey = typeof asRecord(connection.config).sourceTemplateKey === "string" + ? String(asRecord(connection.config).sourceTemplateKey) + : null; for (const descriptor of descriptors) { - const riskLevel = classifyRisk(descriptor); - const hash = descriptorHash(descriptor); + const riskLevel = classifyRisk(descriptor, sourceTemplateKey); + const hash = descriptorHash(descriptor, riskLevel); const schemaHash = stableHash(descriptor.inputSchema ?? {}); const existing = existingByName.get(descriptor.name); const changed = existing && (existing.versionHash !== hash || existing.schemaHash !== schemaHash); @@ -3642,7 +3739,7 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} }; } - function oauthClientForConnection( + function configuredOAuthClientForConnection( connection: typeof toolConnections.$inferSelect, provider: string, ) { @@ -3657,6 +3754,43 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} return oauthClientConfig(provider); } + async function oauthClientForConnection( + connection: typeof toolConnections.$inferSelect, + provider: string, + actor?: ActorInfo, + ) { + const configured = configuredOAuthClientForConnection(connection, provider); + if (configured.clientId) return configured; + const oauth = oauthConfig(connection); + const clientId = typeof oauth.clientId === "string" && oauth.clientId.trim() + ? oauth.clientId.trim() + : null; + if (!clientId) return configured; + const clientSecretRef = oauth.clientRegistrationSource === "dcr" + ? undefined + : connection.credentialSecretRefs.find((ref) => ref.configPath === "oauth.client_secret"); + const clientSecret = clientSecretRef + ? await secrets.resolveSecretValue( + connection.companyId, + clientSecretRef.secretId, + clientSecretRef.versionSelector ?? "latest", + { + consumerType: "tool_connection", + consumerId: connection.id, + configPath: "oauth.client_secret", + actorType: actor?.actorType ?? "system", + actorId: actor?.actorId ?? null, + }, + ) + : null; + return { + clientIdEnv: null, + clientSecretEnv: null, + clientId, + clientSecret, + }; + } + function base64UrlSha256(input: string) { return createHash("sha256").update(input).digest("base64url"); } @@ -3736,6 +3870,10 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} function oauthProviderForConnection(connection: typeof toolConnections.$inferSelect, metadataUrl?: string | null): string { const oauth = oauthConfig(connection); if (typeof oauth.provider === "string" && oauth.provider.trim()) return oauth.provider.trim(); + const sourceTemplateKey = typeof connection.config.sourceTemplateKey === "string" + ? connection.config.sourceTemplateKey.trim() + : ""; + if (sourceTemplateKey) return sourceTemplateKey; const url = metadataUrl ?? remoteEndpoint(connection.config); try { return new URL(url).hostname.replace(/[^a-z0-9]+/gi, "_").replace(/^_+|_+$/g, "").toLowerCase() || "generic"; @@ -3767,7 +3905,7 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} function oauthSecretRef( connection: typeof toolConnections.$inferSelect, - configPath: "oauth.access_token" | "oauth.refresh_token", + configPath: "oauth.access_token" | "oauth.refresh_token" | "oauth.client_secret", ) { return connection.credentialSecretRefs.find((ref) => ref.configPath === configPath) ?? null; } @@ -3820,21 +3958,34 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} if (!metadata) return null; let authorizationUrl = typeof metadata.authorization_endpoint === "string" ? metadata.authorization_endpoint : null; let tokenUrl = typeof metadata.token_endpoint === "string" ? metadata.token_endpoint : null; - if (!authorizationUrl || !tokenUrl) { - for (const authMetadataUrl of await authServerMetadataUrls(metadata)) { - const authMetadata = await fetchJsonRecord(authMetadataUrl); - if (!authMetadata) continue; - authorizationUrl = authorizationUrl ?? (typeof authMetadata.authorization_endpoint === "string" ? authMetadata.authorization_endpoint : null); - tokenUrl = tokenUrl ?? (typeof authMetadata.token_endpoint === "string" ? authMetadata.token_endpoint : null); - if (authorizationUrl && tokenUrl) break; + let registrationUrl = typeof metadata.registration_endpoint === "string" ? metadata.registration_endpoint : null; + let scopes = normalizeOauthScopes(metadata.scopes_supported); + let codeChallengeMethodsSupported = normalizeOauthScopes(metadata.code_challenge_methods_supported); + let tokenEndpointAuthMethodsSupported = normalizeOauthScopes(metadata.token_endpoint_auth_methods_supported); + for (const authMetadataUrl of await authServerMetadataUrls(metadata)) { + const authMetadata = await fetchJsonRecord(authMetadataUrl); + if (!authMetadata) continue; + authorizationUrl = authorizationUrl ?? (typeof authMetadata.authorization_endpoint === "string" ? authMetadata.authorization_endpoint : null); + tokenUrl = tokenUrl ?? (typeof authMetadata.token_endpoint === "string" ? authMetadata.token_endpoint : null); + registrationUrl = registrationUrl ?? (typeof authMetadata.registration_endpoint === "string" ? authMetadata.registration_endpoint : null); + if (scopes.length === 0) scopes = normalizeOauthScopes(authMetadata.scopes_supported); + if (codeChallengeMethodsSupported.length === 0) { + codeChallengeMethodsSupported = normalizeOauthScopes(authMetadata.code_challenge_methods_supported); } + if (tokenEndpointAuthMethodsSupported.length === 0) { + tokenEndpointAuthMethodsSupported = normalizeOauthScopes(authMetadata.token_endpoint_auth_methods_supported); + } + if (authorizationUrl && tokenUrl) break; } if (!authorizationUrl || !tokenUrl) return null; return { provider: oauthProviderForConnection(connection, metadataUrl), - scopes: normalizeOauthScopes(metadata.scopes_supported), + scopes, authorizationUrl, tokenUrl, + registrationUrl, + codeChallengeMethodsSupported, + tokenEndpointAuthMethodsSupported, metadataUrl, }; } @@ -3863,6 +4014,9 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} scopes, authorizationUrl: configuredAuthorizationUrl, tokenUrl: configuredTokenUrl, + registrationUrl: typeof oauth.registrationUrl === "string" ? oauth.registrationUrl : null, + codeChallengeMethodsSupported: normalizeOauthScopes(oauth.codeChallengeMethodsSupported), + tokenEndpointAuthMethodsSupported: normalizeOauthScopes(oauth.tokenEndpointAuthMethodsSupported), grantType, metadataUrl: typeof oauth.metadataUrl === "string" ? oauth.metadataUrl : hints?.metadataUrl ?? null, }; @@ -3874,6 +4028,10 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} ].filter((value): value is string => Boolean(value)); if (metadataCandidates.length === 0) { const endpoint = new URL(await assertRemoteEndpointAllowed(connection.config)); + const protectedResourcePath = endpoint.pathname === "/" + ? "/.well-known/oauth-protected-resource" + : `/.well-known/oauth-protected-resource${endpoint.pathname}`; + metadataCandidates.push(new URL(protectedResourcePath, endpoint.origin).toString()); metadataCandidates.push(new URL("/.well-known/oauth-protected-resource", endpoint.origin).toString()); metadataCandidates.push(new URL("/.well-known/oauth-authorization-server", endpoint.origin).toString()); metadataCandidates.push(new URL("/.well-known/openid-configuration", endpoint.origin).toString()); @@ -3912,10 +4070,18 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} const smokeLabEndpoints = smokeLabOAuthEndpoints(connection, redirectUri); const sourceTemplateKey = typeof connection.config.sourceTemplateKey === "string" ? connection.config.sourceTemplateKey : null; const galleryEntry = sourceTemplateKey ? getConnectableAppDefinition(sourceTemplateKey) : null; + const galleryMethod = galleryEntry ? connectionMethodFor(galleryEntry) : null; + const hasCompleteGalleryEndpointHints = Boolean( + galleryMethod?.defaults?.authorizationEndpoint && galleryMethod.defaults.tokenEndpoint, + ); + const discovered = connection.transport === "mcp_remote" && !hasCompleteGalleryEndpointHints + ? await discoverOAuthEndpoints(connection, challenge) + : null; const endpoints = smokeLabEndpoints + ?? discovered ?? (galleryEntry && connectionMethodFor(galleryEntry).auth === "oauth" - ? await oauthProviderEndpoints(galleryEntry) - : await discoverOAuthEndpoints(connection, challenge)); + ? await oauthProviderEndpoints(galleryEntry) + : await discoverOAuthEndpoints(connection, challenge)); if (!endpoints) throw unprocessable("This app connection does not advertise OAuth sign in"); assertNotSmokeLabOAuthEndpoints(connection, endpoints); return endpoints; @@ -3934,7 +4100,7 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} async function createOrRotateOAuthSecret(input: { companyId: string; connection: typeof toolConnections.$inferSelect; - configPath: "oauth.access_token" | "oauth.refresh_token"; + configPath: "oauth.access_token" | "oauth.refresh_token" | "oauth.client_secret"; label: string; value: string; actor?: ActorInfo; @@ -3963,6 +4129,273 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} }; } + function assertOAuthRedirectConstraints(app: AppDefinition | null, redirectUri: string) { + if (app?.redirectConstraints !== "https-or-loopback-http") return; + let redirect: URL; + try { + redirect = new URL(redirectUri); + } catch { + throw unprocessable("OAuth callback URL is invalid", { code: "oauth_redirect_uri_invalid" }); + } + const hostname = redirect.hostname.replace(/^\[|\]$/g, "").toLowerCase(); + const isLoopback = hostname === "localhost" + || hostname.endsWith(".localhost") + || hostname === "::1" + || /^127(?:\.\d{1,3}){3}$/.test(hostname); + if (redirect.protocol === "https:" || (redirect.protocol === "http:" && isLoopback)) return; + throw unprocessable( + "This provider requires an HTTPS or loopback origin. Configure TLS before connecting.", + { + code: "oauth_redirect_origin_unsupported", + redirectConstraints: app.redirectConstraints, + docsPath: "docs/deploy", + }, + ); + } + + function invalidOAuthDcrResponse(field: string, reason: string): HttpError { + return new HttpError(502, "OAuth provider returned incompatible dynamic client metadata", { + code: "oauth_dcr_response_invalid", + field, + reason, + }); + } + + function parseOAuthDcrString( + record: Record, + field: "client_id" | "client_secret", + input: { required: boolean; maxLength: number }, + ): string | null { + const value = record[field]; + if (value === undefined || value === null) { + if (input.required) throw invalidOAuthDcrResponse(field, "missing"); + return null; + } + if (typeof value !== "string" || value.length === 0 || value.length > input.maxLength) { + throw invalidOAuthDcrResponse(field, "invalid_string"); + } + if (field === "client_id" && value.trim() !== value) { + throw invalidOAuthDcrResponse(field, "invalid_string"); + } + return value; + } + + function assertOAuthDcrArray( + record: Record, + field: "redirect_uris" | "grant_types" | "response_types", + expected: string[], + ) { + if (record[field] === undefined) throw invalidOAuthDcrResponse(field, "missing"); + const value = record[field]; + if ( + !Array.isArray(value) + || value.length !== expected.length + || value.some((entry) => typeof entry !== "string" || entry.length === 0 || entry.length > 2_048) + ) { + throw invalidOAuthDcrResponse(field, "invalid_array"); + } + const actual = [...value].sort(); + const required = [...expected].sort(); + if (actual.some((entry, index) => entry !== required[index])) { + throw invalidOAuthDcrResponse(field, "registered_value_mismatch"); + } + } + + function parseOAuthDcrTimestamp(record: Record, field: string): number | null { + const value = record[field]; + if (value === undefined || value === null) return null; + if (typeof value !== "number" || !Number.isSafeInteger(value) || value < 0) { + throw invalidOAuthDcrResponse(field, "invalid_timestamp"); + } + return value; + } + + async function registerOAuthClient(input: { + connection: typeof toolConnections.$inferSelect; + endpoints: OAuthProviderEndpoints; + redirectUri: string; + actor?: ActorInfo; + }) { + if (!input.endpoints.registrationUrl) { + throw unprocessable("OAuth provider does not advertise dynamic client registration", { + code: "oauth_dcr_not_supported", + }); + } + if ( + input.endpoints.codeChallengeMethodsSupported?.length + && !input.endpoints.codeChallengeMethodsSupported.includes("S256") + ) { + throw unprocessable("OAuth provider does not support the required PKCE S256 method", { + code: "oauth_pkce_s256_required", + }); + } + if ( + input.endpoints.tokenEndpointAuthMethodsSupported?.length + && !input.endpoints.tokenEndpointAuthMethodsSupported.includes("none") + ) { + throw unprocessable("OAuth provider does not support public dynamic clients", { + code: "oauth_dcr_public_client_unsupported", + }); + } + + const host = new URL(input.redirectUri).host; + const requestedMetadata = { + client_name: `Paperclip (${host})`, + redirect_uris: [input.redirectUri], + grant_types: ["authorization_code", "refresh_token"], + response_types: ["code"], + token_endpoint_auth_method: "none", + }; + const response = await fetchRemoteHttpUrl(input.endpoints.registrationUrl, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(requestedMetadata), + }); + const record = asRecord(await response.json().catch(() => ({})) as unknown); + if (!response.ok) { + const providerError = typeof record.error === "string" ? record.error : null; + const message = typeof record.error_description === "string" + ? record.error_description + : "OAuth dynamic client registration failed"; + throw new HttpError(502, message, { + code: "oauth_dynamic_client_registration_failed", + providerError, + status: response.status, + }); + } + const clientId = parseOAuthDcrString(record, "client_id", { + required: true, + maxLength: MAX_OAUTH_DCR_CLIENT_ID_LENGTH, + })!; + const clientSecret = parseOAuthDcrString(record, "client_secret", { + required: false, + maxLength: MAX_OAUTH_DCR_CLIENT_SECRET_LENGTH, + }); + assertOAuthDcrArray(record, "redirect_uris", requestedMetadata.redirect_uris); + assertOAuthDcrArray(record, "grant_types", requestedMetadata.grant_types); + assertOAuthDcrArray(record, "response_types", requestedMetadata.response_types); + if ( + record.token_endpoint_auth_method !== requestedMetadata.token_endpoint_auth_method + ) { + throw invalidOAuthDcrResponse("token_endpoint_auth_method", "registered_value_mismatch"); + } + const clientIdIssuedAt = parseOAuthDcrTimestamp(record, "client_id_issued_at"); + const clientSecretExpiresAt = parseOAuthDcrTimestamp(record, "client_secret_expires_at"); + if (clientSecretExpiresAt !== null && clientSecret === null) { + throw invalidOAuthDcrResponse("client_secret_expires_at", "client_secret_missing"); + } + const existingClientSecretRef = oauthSecretRef(input.connection, "oauth.client_secret"); + const nextCredentialSecretRefs = input.connection.credentialSecretRefs.filter( + (ref) => ref.configPath !== "oauth.client_secret", + ); + if (clientSecret) { + const clientSecretRef = await createOrRotateOAuthSecret({ + companyId: input.connection.companyId, + connection: input.connection, + configPath: "oauth.client_secret", + label: "OAuth client secret", + value: clientSecret, + actor: input.actor, + }); + nextCredentialSecretRefs.push(clientSecretRef); + } else if (existingClientSecretRef && oauthConfig(input.connection).clientId === clientId) { + nextCredentialSecretRefs.push(existingClientSecretRef); + } + + const oauth = oauthConfig(input.connection); + const nextConfig = { + ...input.connection.config, + oauth: { + ...oauth, + provider: input.endpoints.provider, + authorizationUrl: input.endpoints.authorizationUrl, + tokenUrl: input.endpoints.tokenUrl, + registrationUrl: input.endpoints.registrationUrl, + metadataUrl: input.endpoints.metadataUrl ?? null, + scopes: input.endpoints.scopes, + codeChallengeMethodsSupported: input.endpoints.codeChallengeMethodsSupported ?? [], + tokenEndpointAuthMethodsSupported: input.endpoints.tokenEndpointAuthMethodsSupported ?? [], + clientId, + clientRegistrationSource: "dcr", + clientTokenEndpointAuthMethod: "none", + clientRedirectUri: input.redirectUri, + clientIdIssuedAt, + clientSecretExpiresAt, + }, + }; + const [updated] = await db + .update(toolConnections) + .set({ + ownership: "dcr", + config: nextConfig, + transportConfig: nextConfig, + credentialSecretRefs: nextCredentialSecretRefs, + updatedAt: now(), + }) + .where(and( + eq(toolConnections.id, input.connection.id), + eq(toolConnections.companyId, input.connection.companyId), + )) + .returning(); + if (!updated) throw notFound("Tool connection not found"); + await syncCredentialBindings(updated); + return updated; + } + + async function ensureOAuthClient(input: { + connection: typeof toolConnections.$inferSelect; + endpoints: OAuthProviderEndpoints; + redirectUri: string; + galleryEntry: AppDefinition | null; + actor?: ActorInfo; + }) { + const configured = configuredOAuthClientForConnection(input.connection, input.endpoints.provider); + if (configured.clientId) return { connection: input.connection, client: configured }; + const oauth = oauthConfig(input.connection); + if ( + typeof oauth.clientId === "string" + && oauth.clientId.trim() + && oauth.clientRedirectUri === input.redirectUri + ) { + return { + connection: input.connection, + client: await oauthClientForConnection(input.connection, input.endpoints.provider, input.actor), + }; + } + const method = input.galleryEntry ? connectionMethodFor(input.galleryEntry) : null; + if (!method?.ownershipModes.includes("dcr")) { + throw unprocessable(`OAuth client id is not configured for ${input.endpoints.provider}`); + } + + const key = `${input.connection.id}:${input.redirectUri}`; + return oauthSingleFlight(oauthRegistrationFlights, key, async () => { + const latest = await getConnectionRow(input.connection.id, input.connection.companyId); + const latestConfigured = configuredOAuthClientForConnection(latest, input.endpoints.provider); + if (latestConfigured.clientId) return { connection: latest, client: latestConfigured }; + const latestOauth = oauthConfig(latest); + if ( + typeof latestOauth.clientId === "string" + && latestOauth.clientId.trim() + && latestOauth.clientRedirectUri === input.redirectUri + ) { + return { + connection: latest, + client: await oauthClientForConnection(latest, input.endpoints.provider, input.actor), + }; + } + const registered = await registerOAuthClient({ + connection: latest, + endpoints: input.endpoints, + redirectUri: input.redirectUri, + actor: input.actor, + }); + return { + connection: registered, + client: await oauthClientForConnection(registered, input.endpoints.provider, input.actor), + }; + }); + } + async function exchangeOAuthToken(input: { tokenUrl: string; clientId: string; @@ -3998,12 +4431,24 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} const payload = await response.json().catch(() => ({})) as unknown; const record = asRecord(payload); if (!response.ok || record.ok === false) { + const providerError = typeof record.error === "string" ? record.error : null; const message = typeof record.error_description === "string" ? record.error_description - : typeof record.error === "string" - ? record.error + : providerError + ? providerError : "OAuth token exchange failed"; - throw new HttpError(502, message, { code: "oauth_token_exchange_failed", status: response.status }); + if (input.grantType === "refresh_token" && providerError === "invalid_grant") { + throw new HttpError(422, "OAuth authorization has expired. Reconnect this app to continue.", { + code: "oauth_reauthorization_required", + providerError, + status: response.status, + }); + } + throw new HttpError(502, message, { + code: "oauth_token_exchange_failed", + providerError, + status: response.status, + }); } const accessToken = typeof record.access_token === "string" ? record.access_token : null; if (!accessToken) throw new HttpError(502, "OAuth provider did not return an access token", { code: "oauth_access_token_missing" }); @@ -4018,8 +4463,158 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} }; } - async function maybeRefreshOAuthCredentials( + function withoutOAuthRefreshLease(oauth: Record) { + const { refreshLease: _refreshLease, ...rest } = oauth; + return rest; + } + + function oauthRefreshLeaseId(connection: typeof toolConnections.$inferSelect): string | null { + const lease = asRecord(oauthConfig(connection).refreshLease); + return typeof lease.id === "string" && lease.id ? lease.id : null; + } + + async function clearOAuthRefreshLease( connection: typeof toolConnections.$inferSelect, + leaseId: string, + ) { + const latest = await getConnectionRow(connection.id, connection.companyId); + if (oauthRefreshLeaseId(latest) !== leaseId) return latest; + const nextConfig = { + ...latest.config, + oauth: withoutOAuthRefreshLease(oauthConfig(latest)), + }; + const [updated] = await db + .update(toolConnections) + .set({ config: nextConfig, transportConfig: nextConfig, updatedAt: now() }) + .where(and( + eq(toolConnections.id, latest.id), + eq(toolConnections.companyId, latest.companyId), + sql`${toolConnections.config} -> 'oauth' -> 'refreshLease' ->> 'id' = ${leaseId}`, + )) + .returning(); + return updated ?? getConnectionRow(connection.id, connection.companyId); + } + + async function acquireOAuthRefreshLease( + connection: typeof toolConnections.$inferSelect, + ): Promise<{ connection: typeof toolConnections.$inferSelect; leaseId: string | null }> { + const waitDeadline = Date.now() + OAUTH_REFRESH_LEASE_WAIT_MS; + while (true) { + const latest = await getConnectionRow(connection.id, connection.companyId); + const latestExpiresAtMs = oauthExpiresAtMs(latest); + if (latestExpiresAtMs && latestExpiresAtMs > Date.now() + 60_000) { + return { connection: latest, leaseId: null }; + } + + const oauth = oauthConfig(latest); + const currentLease = asRecord(oauth.refreshLease); + const currentLeaseExpiresAt = typeof currentLease.expiresAt === "string" + ? Date.parse(currentLease.expiresAt) + : Number.NaN; + const currentLeaseId = typeof currentLease.id === "string" && currentLease.id + ? currentLease.id + : null; + const leaseIsActive = currentLeaseId !== null + && Number.isFinite(currentLeaseExpiresAt) + && currentLeaseExpiresAt > Date.now(); + if (!currentLeaseId) { + const leaseId = randomUUID(); + const claimedAt = now(); + const nextConfig = { + ...latest.config, + oauth: { + ...withoutOAuthRefreshLease(oauth), + refreshLease: { + id: leaseId, + expiresAt: new Date(Date.now() + OAUTH_REFRESH_LEASE_MS).toISOString(), + }, + }, + }; + const [claimed] = await db + .update(toolConnections) + .set({ config: nextConfig, transportConfig: nextConfig, updatedAt: claimedAt }) + .where(and( + eq(toolConnections.id, latest.id), + eq(toolConnections.companyId, latest.companyId), + sql`${toolConnections.config} = ${JSON.stringify(latest.config)}::jsonb`, + sql`${toolConnections.config} #>> '{oauth,refreshLease,id}' is null`, + )) + .returning(); + if (claimed) return { connection: claimed, leaseId }; + } + + if (currentLeaseId && !leaseIsActive) { + throw new HttpError(422, "The previous OAuth refresh did not finish. Reconnect this app before retrying.", { + code: "oauth_refresh_outcome_unknown", + setupUrl: connectionSetupUrl(latest), + reconnectUrl: connectionReconnectUrl(latest), + }); + } + + if (Date.now() >= waitDeadline) { + throw conflict("OAuth credential refresh is already in progress", { + code: "oauth_refresh_in_progress", + retryable: true, + }); + } + await new Promise((resolve) => setTimeout(resolve, OAUTH_REFRESH_LEASE_POLL_MS)); + } + } + + async function markOAuthReauthorizationRequired( + connection: typeof toolConnections.$inferSelect, + guard: { + leaseId: string; + refreshSecretId: string; + refreshTokenVersion: number; + }, + ) { + const oauth = oauthConfig(connection); + const nextCredentialSecretRefs = connection.credentialSecretRefs.filter( + (ref) => ref.configPath !== "oauth.access_token" && ref.configPath !== "oauth.refresh_token", + ); + const nextCredentialRefs = connection.credentialRefs.filter((ref) => ref.name !== "oauth.access_token"); + const nextConfig = { + ...connection.config, + oauth: { + ...withoutOAuthRefreshLease(oauth), + expiresAt: null, + reauthorizationRequiredAt: now().toISOString(), + }, + }; + const [updated] = await db + .update(toolConnections) + .set({ + status: "draft", + enabled: false, + healthStatus: "error", + healthMessage: "OAuth authorization expired. Reconnect this app to continue.", + lastError: "oauth_reauthorization_required", + config: nextConfig, + transportConfig: nextConfig, + credentialSecretRefs: nextCredentialSecretRefs, + credentialRefs: nextCredentialRefs, + updatedAt: now(), + }) + .where(and( + eq(toolConnections.id, connection.id), + eq(toolConnections.companyId, connection.companyId), + sql`${toolConnections.config} -> 'oauth' -> 'refreshLease' ->> 'id' = ${guard.leaseId}`, + sql`exists ( + select 1 from ${companySecrets} + where ${companySecrets.id} = ${guard.refreshSecretId} + and ${companySecrets.companyId} = ${connection.companyId} + and ${companySecrets.latestVersion} = ${guard.refreshTokenVersion} + )`, + )) + .returning(); + if (updated) await syncCredentialBindings(updated); + return updated ?? null; + } + + async function refreshOAuthCredentials( + connection: typeof toolConnections.$inferSelect, + leaseId: string, actor?: ActorInfo, accessContext?: { actorSource?: "local_implicit" | "session" | "board_key" | "agent_key" | "agent_jwt" | "cloud_tenant"; @@ -4042,10 +4637,21 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} reconnectUrl: connectionReconnectUrl(connection), }); } - const client = oauthClientForConnection(connection, oauth.provider); + const client = await oauthClientForConnection(connection, oauth.provider, actor); if (!client.clientId) throw unprocessable(`OAuth client id is not configured for ${oauth.provider}`); - const refreshToken = refreshRef - ? await secrets.resolveSecretValue(connection.companyId, refreshRef.secretId, refreshRef.versionSelector ?? "latest", { + const [refreshSecret] = refreshRef + ? await db + .select({ latestVersion: companySecrets.latestVersion }) + .from(companySecrets) + .where(and( + eq(companySecrets.id, refreshRef.secretId), + eq(companySecrets.companyId, connection.companyId), + )) + .limit(1) + : [undefined]; + const refreshTokenVersion = refreshSecret?.latestVersion ?? null; + const refreshToken = refreshRef && refreshTokenVersion !== null + ? await secrets.resolveSecretValue(connection.companyId, refreshRef.secretId, refreshTokenVersion, { consumerType: "tool_connection", consumerId: connection.id, configPath: "oauth.refresh_token", @@ -4056,14 +4662,56 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} heartbeatRunId: accessContext?.heartbeatRunId, }) : null; - const token = await exchangeOAuthToken({ - tokenUrl: oauth.tokenUrl, - clientId: client.clientId, - clientSecret: client.clientSecret, - grantType, - scopes: normalizeOauthScopes(oauth.scopes).length > 0 ? normalizeOauthScopes(oauth.scopes) : normalizeOauthScopes(oauth.scope), - refreshToken, - }); + let token: Awaited>; + try { + token = await exchangeOAuthToken({ + tokenUrl: oauth.tokenUrl, + clientId: client.clientId, + clientSecret: client.clientSecret, + grantType, + scopes: normalizeOauthScopes(oauth.scopes).length > 0 ? normalizeOauthScopes(oauth.scopes) : normalizeOauthScopes(oauth.scope), + refreshToken, + }); + } catch (error) { + if (error instanceof HttpError && asRecord(error.details).code === "oauth_reauthorization_required") { + const marked = refreshRef && refreshTokenVersion !== null + ? await markOAuthReauthorizationRequired(connection, { + leaseId, + refreshSecretId: refreshRef.secretId, + refreshTokenVersion, + }) + : null; + if (!marked) { + const latest = await getConnectionRow(connection.id, connection.companyId); + const latestExpiresAtMs = oauthExpiresAtMs(latest); + if (latestExpiresAtMs && latestExpiresAtMs > Date.now() + 60_000) return latest; + throw conflict("OAuth credentials changed while refresh was in progress. Retry the request.", { + code: "oauth_refresh_superseded", + retryable: true, + }); + } + throw new HttpError(error.status, error.message, { + ...asRecord(error.details), + setupUrl: connectionSetupUrl(connection), + reconnectUrl: connectionReconnectUrl(connection), + }); + } + throw error; + } + // Rotating providers invalidate the submitted refresh token immediately. + // Persist its replacement before the new access token can be returned to a + // caller, so a crash cannot leave the grant with only the consumed token. + let nextRefreshRef: Awaited> | null = null; + if (token.refreshToken) { + nextRefreshRef = await createOrRotateOAuthSecret({ + companyId: connection.companyId, + connection, + configPath: "oauth.refresh_token", + label: "OAuth refresh token", + value: token.refreshToken, + actor, + }); + } const accessRef = await createOrRotateOAuthSecret({ companyId: connection.companyId, connection, @@ -4073,26 +4721,18 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} actor, }); const nextCredentialSecretRefs = [ - ...connection.credentialSecretRefs.filter((ref) => ref.configPath !== "oauth.access_token"), + ...connection.credentialSecretRefs.filter((ref) => + ref.configPath !== "oauth.access_token" + && (!nextRefreshRef || ref.configPath !== "oauth.refresh_token") + ), accessRef, + ...(nextRefreshRef ? [nextRefreshRef] : []), ]; - if (token.refreshToken) { - const nextRefreshRef = await createOrRotateOAuthSecret({ - companyId: connection.companyId, - connection, - configPath: "oauth.refresh_token", - label: "OAuth refresh token", - value: token.refreshToken, - actor, - }); - const filtered = nextCredentialSecretRefs.filter((ref) => ref.configPath !== "oauth.refresh_token"); - nextCredentialSecretRefs.splice(0, nextCredentialSecretRefs.length, ...filtered, nextRefreshRef); - } const expiresAt = token.expiresIn ? new Date(Date.now() + token.expiresIn * 1000).toISOString() : null; const nextConfig = { ...connection.config, oauth: { - ...oauth, + ...withoutOAuthRefreshLease(oauth), grantType: grantType === "client_credentials" ? grantType : oauth.grantType ?? "authorization_code", expiresAt, scope: token.scope ?? oauth.scope ?? null, @@ -4127,12 +4767,54 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} ], updatedAt: new Date(), }) - .where(eq(toolConnections.id, connection.id)) + .where(and( + eq(toolConnections.id, connection.id), + eq(toolConnections.companyId, connection.companyId), + sql`${toolConnections.config} -> 'oauth' -> 'refreshLease' ->> 'id' = ${leaseId}`, + )) .returning(); - await syncCredentialBindings(updated); + if (!updated) { + throw conflict("OAuth credentials changed while refresh was in progress. Retry the request.", { + code: "oauth_refresh_superseded", + retryable: true, + }); + } + const previousBindingKeys = new Set(connection.credentialSecretRefs.map( + (ref) => `${ref.secretId}:${ref.configPath}`, + )); + const nextBindingKeys = new Set(nextCredentialSecretRefs.map( + (ref) => `${ref.secretId}:${ref.configPath}`, + )); + const bindingsChanged = previousBindingKeys.size !== nextBindingKeys.size + || [...previousBindingKeys].some((key) => !nextBindingKeys.has(key)); + if (bindingsChanged) await syncCredentialBindings(updated); return updated; } + async function maybeRefreshOAuthCredentials( + connection: typeof toolConnections.$inferSelect, + actor?: ActorInfo, + accessContext?: { + actorSource?: "local_implicit" | "session" | "board_key" | "agent_key" | "agent_jwt" | "cloud_tenant"; + issueId?: string | null; + heartbeatRunId?: string | null; + }, + ): Promise { + const oauth = oauthConfig(connection); + if (typeof oauth.tokenUrl !== "string" || typeof oauth.provider !== "string") return connection; + const expiresAtMs = oauthExpiresAtMs(connection); + if (expiresAtMs && expiresAtMs > Date.now() + 60_000) return connection; + return oauthSingleFlight(oauthRefreshFlights, connection.id, async () => { + const lease = await acquireOAuthRefreshLease(connection); + if (!lease.leaseId) return lease.connection; + try { + return await refreshOAuthCredentials(lease.connection, lease.leaseId, actor, accessContext); + } finally { + await clearOAuthRefreshLease(lease.connection, lease.leaseId).catch(() => undefined); + } + }); + } + function policyNameForApp(connection: typeof toolConnections.$inferSelect, entry: typeof toolCatalogEntries.$inferSelect) { const base = `Ask first ${connection.id.slice(0, 8)} ${entry.toolName}`; return base.length <= 160 ? base : base.slice(0, 160); @@ -4740,13 +5422,24 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} connectionId: string, input: { redirectUri: string; actor: ActorInfo; subjectUserId?: string; scopes?: string[]; returnTo?: string; issueId?: string }, ): Promise { - const connection = await getConnectionRow(connectionId, companyId); + let connection = await getConnectionRow(connectionId, companyId); if (connection.status === "archived") throw conflict("Archived app connections cannot start sign in"); + const sourceTemplateKey = typeof connection.config.sourceTemplateKey === "string" ? connection.config.sourceTemplateKey : null; + const galleryEntry = sourceTemplateKey ? getConnectableAppDefinition(sourceTemplateKey) : null; + assertOAuthRedirectConstraints(galleryEntry, input.redirectUri); const endpoints = await oauthEndpointsForConnection(connection, null, input.redirectUri); if (endpoints.grantType === "client_credentials") { throw unprocessable("This app uses shared machine credentials and does not need browser sign in"); } - const client = oauthClientForConnection(connection, endpoints.provider); + const resolvedClient = await ensureOAuthClient({ + connection, + endpoints, + redirectUri: input.redirectUri, + galleryEntry, + actor: input.actor, + }); + connection = resolvedClient.connection; + const client = resolvedClient.client; if (!client.clientId) throw unprocessable(`OAuth client id is not configured for ${endpoints.provider}`); await db.delete(toolOauthStates).where(lt(toolOauthStates.expiresAt, new Date())); @@ -4837,8 +5530,11 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} provider: endpoints.provider, authorizationUrl: endpoints.authorizationUrl, tokenUrl: endpoints.tokenUrl, + registrationUrl: endpoints.registrationUrl ?? null, metadataUrl: endpoints.metadataUrl ?? null, scopes: endpoints.scopes, + codeChallengeMethodsSupported: endpoints.codeChallengeMethodsSupported ?? [], + tokenEndpointAuthMethodsSupported: endpoints.tokenEndpointAuthMethodsSupported ?? [], grantType: "authorization_code", clientIdEnv: client.clientIdEnv, clientSecretEnv: client.clientSecret ? client.clientSecretEnv : null, @@ -4896,8 +5592,9 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} let connection = await getConnectionRow(stateRow.connectionId, stateRow.companyId); const sourceTemplateKey = typeof connection.config.sourceTemplateKey === "string" ? connection.config.sourceTemplateKey : null; const galleryEntry = sourceTemplateKey ? getConnectableAppDefinition(sourceTemplateKey) : null; + assertOAuthRedirectConstraints(galleryEntry, input.redirectUri); const endpoints = await oauthEndpointsForConnection(connection, null, input.redirectUri); - const client = oauthClientForConnection(connection, endpoints.provider); + const client = await oauthClientForConnection(connection, endpoints.provider, input.actor); if (!client.clientId) throw unprocessable(`OAuth client id is not configured for ${endpoints.provider}`); const token = await exchangeOAuthToken({ @@ -5000,7 +5697,7 @@ export function toolAccessService(db: Db, options: ToolAccessServiceOptions = {} const nextConfig = { ...connection.config, oauth: { - ...oauthConfig(connection), + ...withoutOAuthRefreshLease(oauthConfig(connection)), provider: endpoints.provider, authorizationUrl: endpoints.authorizationUrl, tokenUrl: endpoints.tokenUrl, diff --git a/ui/src/App.tsx b/ui/src/App.tsx index f772a71251b..0e309f4fa20 100644 --- a/ui/src/App.tsx +++ b/ui/src/App.tsx @@ -60,6 +60,7 @@ import { ProfileDetailRoute } from "./pages/tools/profiles/ProfileDetailRoute"; import { Connections } from "./pages/apps/Connections"; import { Browse } from "./pages/apps/Browse"; import { AppsConnect } from "./pages/apps/AppsConnect"; +import { canEnterAppsConnect } from "./pages/apps/app-connect-policy"; import { AppsReview } from "./pages/apps/AppsReview"; import { AppDetail } from "./pages/apps/AppDetail"; import { AppNotConnected } from "./pages/apps/AppNotConnected"; @@ -308,7 +309,7 @@ function boardRoutes() { function AppsConnectEntryRoute() { const location = useLocation(); const searchParams = new URLSearchParams(location.search); - return searchParams.get("byo") === "1" ? : ; + return canEnterAppsConnect(searchParams) ? : ; } function InboxRootRedirect() { diff --git a/ui/src/pages/apps/AppDetail.test.tsx b/ui/src/pages/apps/AppDetail.test.tsx index 2aa7bd21991..5125be35c8e 100644 --- a/ui/src/pages/apps/AppDetail.test.tsx +++ b/ui/src/pages/apps/AppDetail.test.tsx @@ -23,6 +23,7 @@ const startOAuthMock = vi.hoisted(() => vi.fn()); const mockNavigate = vi.hoisted(() => vi.fn()); const mockParams = vi.hoisted(() => ({ connectionId: "conn-1", tab: "setup" as string | undefined })); const navigateComponentMock = vi.hoisted(() => vi.fn()); +const navigateTopLevelMock = vi.hoisted(() => vi.fn()); vi.mock("@/api/tools", () => ({ toolsApi: { @@ -57,6 +58,10 @@ vi.mock("@/api/agents", () => ({ }, })); +vi.mock("@/lib/browserNavigation", () => ({ + navigateTopLevel: (target: string) => navigateTopLevelMock(target), +})); + vi.mock("@/lib/router", () => ({ useParams: () => mockParams, useNavigate: () => mockNavigate, @@ -371,6 +376,40 @@ describe("AppDetail", () => { ).toBe(true); }); + it("matches connected Notion guidance to the reconnect action", async () => { + getConnectionMock.mockResolvedValue(connection({ + name: "Notion", + config: { + sourceTemplateKey: "notion", + oauth: { + provider: "notion", + connectedAt: "2026-08-06T20:00:00.000Z", + }, + }, + })); + listGalleryMock.mockResolvedValue({ + apps: [{ + key: "notion", + name: "Notion", + logoUrl: "https://example.com/notion.png", + tagline: "Search and update your Notion workspace.", + description: "Give agents governed access to Notion.", + authKind: "oauth", + transportTemplate: { transport: "mcp_remote", url: "https://mcp.notion.com/mcp" }, + credentialFields: [], + recommendedDefaults: {}, + urlPatterns: [], + }], + }); + + await renderAppDetail(); + + expect(container.textContent).toContain( + "Your workspace authorization is active. Reconnect any time to replace it.", + ); + expect(container.textContent).not.toContain("Sign in again any time"); + }); + it("lets Google Sheets connections add spreadsheet links from setup", async () => { mockParams.tab = "setup"; getConnectionMock.mockResolvedValue(connection({ @@ -681,4 +720,29 @@ describe("AppDetail", () => { expect(container.textContent).toContain("Token expired."); expect(container.textContent).toContain("Who can use it"); }); + + it("shows terminal OAuth failures as reconnect-required sign-in", async () => { + mockParams.tab = "permissions"; + getConnectionMock.mockResolvedValue(connection({ + authKind: "oauth", + healthStatus: "failed", + healthMessage: "Authorization expired (invalid_grant).", + })); + + await renderAppDetail(); + + expect(container.textContent).toContain("Reconnect required"); + expect(container.textContent).toContain("Authorization expired (invalid_grant)."); + expect(container.querySelector('input[placeholder="Paste your new key"]')).toBeNull(); + + await act(async () => { + Array.from(container.querySelectorAll("button")) + .find((button) => button.textContent?.trim() === "Reconnect") + ?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + + expect(startOAuthMock).toHaveBeenCalledWith("conn-1"); + expect(navigateTopLevelMock).toHaveBeenCalledWith("http://example.test/oauth"); + }); }); diff --git a/ui/src/pages/apps/AppDetail.tsx b/ui/src/pages/apps/AppDetail.tsx index 010311c37af..9d1222df805 100644 --- a/ui/src/pages/apps/AppDetail.tsx +++ b/ui/src/pages/apps/AppDetail.tsx @@ -22,6 +22,7 @@ import { accessApi } from "@/api/access"; import { authApi } from "@/api/auth"; import { buildCompanyUserLabelMap } from "@/lib/company-members"; import { installPayload, installStateFrom, type InstallState } from "@/lib/tool-installs"; +import { navigateTopLevel } from "@/lib/browserNavigation"; import { Button } from "@/components/ui/button"; import { Input } from "@/components/ui/input"; import { Skeleton } from "@/components/ui/skeleton"; @@ -239,7 +240,7 @@ export function AppDetail() { const startOAuth = useMutation({ mutationFn: () => toolsApi.startOAuth(connectionId), onSuccess: ({ authorizationUrl }) => { - window.location.assign(authorizationUrl); + navigateTopLevel(authorizationUrl); }, onError: (error) => pushToast({ diff --git a/ui/src/pages/apps/AppsConnect.test.tsx b/ui/src/pages/apps/AppsConnect.test.tsx index ac2b0fde6c2..966e8893b8e 100644 --- a/ui/src/pages/apps/AppsConnect.test.tsx +++ b/ui/src/pages/apps/AppsConnect.test.tsx @@ -5,24 +5,33 @@ import { createRoot } from "react-dom/client"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { CONNECTABLE_APP_DEFINITIONS } from "@paperclipai/shared"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { queryKeys } from "@/lib/queryKeys"; import { AppsConnect } from "./AppsConnect"; const listGalleryMock = vi.hoisted(() => vi.fn()); +const listApplicationsMock = vi.hoisted(() => vi.fn()); +const listConnectionsMock = vi.hoisted(() => vi.fn()); const connectAppMock = vi.hoisted(() => vi.fn()); +const startOAuthMock = vi.hoisted(() => vi.fn()); const finishAppMock = vi.hoisted(() => vi.fn()); const putConnectionInstallsMock = vi.hoisted(() => vi.fn()); const listAgentsMock = vi.hoisted(() => vi.fn()); const mockNavigate = vi.hoisted(() => vi.fn()); +const navigateTopLevelMock = vi.hoisted(() => vi.fn()); const mockSearch = vi.hoisted(() => ({ value: "" })); const mockParams = vi.hoisted(() => ({ appKey: undefined as string | undefined })); const ZAPIER = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "zapier")!; +const NOTION = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "notion")!; const GOOGLE_SHEETS = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "google-sheets")!; vi.mock("@/api/tools", () => ({ toolsApi: { listGallery: (companyId: string) => listGalleryMock(companyId), + listApplications: (companyId: string) => listApplicationsMock(companyId), + listConnections: (companyId: string) => listConnectionsMock(companyId), connectApp: (companyId: string, input: unknown) => connectAppMock(companyId, input), + startOAuth: (connectionId: string) => startOAuthMock(connectionId), finishApp: (companyId: string, connectionId: string, input: unknown) => finishAppMock(companyId, connectionId, input), putConnectionInstalls: (connectionId: string, installs: unknown) => @@ -34,6 +43,10 @@ vi.mock("@/api/agents", () => ({ agentsApi: { list: (companyId: string) => listAgentsMock(companyId) }, })); +vi.mock("@/lib/browserNavigation", () => ({ + navigateTopLevel: (target: string) => navigateTopLevelMock(target), +})); + vi.mock("@/lib/router", () => ({ useNavigate: () => mockNavigate, useParams: () => mockParams, @@ -129,6 +142,14 @@ describe("AppsConnect β€” Connect with a link (M4 frame)", () => { ZAPIER, ], }); + listApplicationsMock.mockResolvedValue({ applications: [] }); + listConnectionsMock.mockResolvedValue({ connections: [] }); + startOAuthMock.mockResolvedValue({ + connectionId: "conn-notion", + provider: "notion", + authorizationUrl: "https://mcp.notion.com/authorize?state=resumed", + expiresAt: "2099-01-01T00:00:00.000Z", + }); finishAppMock.mockResolvedValue({}); putConnectionInstallsMock.mockResolvedValue({ connectionId: "conn-1", installs: [] }); connectAppMock.mockResolvedValue({ @@ -150,12 +171,12 @@ describe("AppsConnect β€” Connect with a link (M4 frame)", () => { vi.clearAllMocks(); }); - async function render() { + async function render(queryClient?: QueryClient) { const root = createRoot(container); - const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + const client = queryClient ?? new QueryClient({ defaultOptions: { queries: { retry: false } } }); await act(async () => { root.render( - + , ); @@ -190,6 +211,298 @@ describe("AppsConnect β€” Connect with a link (M4 frame)", () => { expect(container.textContent).not.toContain("Pick the app you want your agents to use."); }); + it("auto-starts the allowlisted Notion source deep link and opens provider sign-in", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + connectAppMock.mockResolvedValueOnce({ + connectionId: "conn-notion", + application: { id: "app-notion", name: "Notion" }, + connection: { id: "conn-notion" }, + actions: { readOnly: [], canMakeChanges: [] }, + catalog: [], + suggestedDefaults: {}, + auth: { kind: "oauth", startUrl: "https://mcp.notion.com/authorize?state=opaque" }, + }); + + await render(); + + expect(connectAppMock).toHaveBeenCalledTimes(1); + expect(connectAppMock).toHaveBeenCalledWith("company-1", { + galleryKey: "notion", + name: "Notion", + credentialValues: {}, + configValues: undefined, + applicationId: undefined, + }); + expect(navigateTopLevelMock).toHaveBeenCalledWith( + "https://mcp.notion.com/authorize?state=opaque", + ); + }); + + it("shows an in-flight state while Paperclip prepares Notion sign-in", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + connectAppMock.mockReturnValueOnce(new Promise(() => {})); + + await render(); + + expect(container.textContent).toContain("Connect Notion"); + expect(container.textContent).toContain("Preparing secure sign-in"); + expect(container.textContent).toContain("Preparing…"); + expect(connectAppMock).toHaveBeenCalledTimes(1); + }); + + it("resumes an existing Notion OAuth connection instead of creating another draft", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + listApplicationsMock.mockResolvedValueOnce({ + applications: [{ + id: "app-notion", + status: "draft", + metadata: { sourceTemplateKey: "notion" }, + }], + }); + listConnectionsMock.mockResolvedValueOnce({ + connections: [{ + id: "conn-existing", + applicationId: "app-notion", + authKind: "oauth", + status: "draft", + config: { sourceTemplateKey: "notion" }, + transportConfig: {}, + }], + }); + startOAuthMock.mockResolvedValueOnce({ + connectionId: "conn-existing", + provider: "notion", + authorizationUrl: "https://mcp.notion.com/authorize?state=existing", + expiresAt: "2099-01-01T00:00:00.000Z", + }); + + await render(); + + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).toHaveBeenCalledWith("conn-existing"); + expect(navigateTopLevelMock).toHaveBeenCalledWith( + "https://mcp.notion.com/authorize?state=existing", + ); + }); + + it("waits for fresh connection data before creating a Notion OAuth draft", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + const queryClient = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + queryClient.setQueryData(queryKeys.tools.applications("company-1"), { applications: [] }); + queryClient.setQueryData(queryKeys.tools.connections("company-1"), { connections: [] }); + + let resolveApplications!: (value: unknown) => void; + let resolveConnections!: (value: unknown) => void; + listApplicationsMock.mockReturnValueOnce(new Promise((resolve) => { + resolveApplications = resolve; + })); + listConnectionsMock.mockReturnValueOnce(new Promise((resolve) => { + resolveConnections = resolve; + })); + + await render(queryClient); + + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).not.toHaveBeenCalled(); + + resolveApplications({ + applications: [{ + id: "app-notion", + status: "draft", + metadata: { sourceTemplateKey: "notion" }, + }], + }); + resolveConnections({ + connections: [{ + id: "conn-refreshed", + applicationId: "app-notion", + authKind: "oauth", + status: "draft", + config: { sourceTemplateKey: "notion" }, + transportConfig: {}, + }], + }); + await flushReact(); + await flushReact(); + + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).toHaveBeenCalledWith("conn-refreshed"); + }); + + it("resumes Notion OAuth after a failed connection lookup is retried", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + listApplicationsMock + .mockRejectedValueOnce(new Error("Lookup unavailable")) + .mockResolvedValueOnce({ + applications: [{ + id: "app-notion", + status: "draft", + metadata: { sourceTemplateKey: "notion" }, + }], + }); + listConnectionsMock + .mockResolvedValueOnce({ connections: [] }) + .mockResolvedValueOnce({ + connections: [{ + id: "conn-after-retry", + applicationId: "app-notion", + authKind: "oauth", + status: "draft", + config: { sourceTemplateKey: "notion" }, + transportConfig: {}, + }], + }); + + await render(); + + expect(container.textContent).toContain("couldn’t check for an existing connection"); + await act(async () => { + buttonByText("Try again")?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + await flushReact(); + + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).toHaveBeenCalledWith("conn-after-retry"); + expect(navigateTopLevelMock).toHaveBeenCalledWith( + "https://mcp.notion.com/authorize?state=resumed", + ); + }); + + it("restores the Notion lookup error when retrying still fails", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + listApplicationsMock.mockRejectedValue(new Error("Lookup unavailable")); + + await render(); + + await act(async () => { + buttonByText("Try again")?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + await flushReact(); + + expect(container.textContent).toContain("couldn’t check for an existing connection"); + expect(buttonByText("Try again")).toBeTruthy(); + expect(connectAppMock).not.toHaveBeenCalled(); + expect(startOAuthMock).not.toHaveBeenCalled(); + }); + + it("retries OAuth on the prepared connection without creating another draft", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + connectAppMock.mockResolvedValueOnce({ + connectionId: "conn-prepared", + application: { id: "app-notion", name: "Notion" }, + connection: { id: "conn-prepared" }, + actions: { readOnly: [], canMakeChanges: [] }, + catalog: [], + suggestedDefaults: {}, + auth: { kind: "oauth", startUrl: null }, + }); + startOAuthMock + .mockRejectedValueOnce(new Error("Provider unavailable")) + .mockResolvedValueOnce({ + connectionId: "conn-prepared", + provider: "notion", + authorizationUrl: "https://mcp.notion.com/authorize?state=retry", + expiresAt: "2099-01-01T00:00:00.000Z", + }); + + await render(); + + expect(container.textContent).toContain("Provider unavailable"); + await act(async () => { + buttonByText("Try again")?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + + expect(connectAppMock).toHaveBeenCalledTimes(1); + expect(startOAuthMock).toHaveBeenCalledTimes(2); + expect(startOAuthMock).toHaveBeenLastCalledWith("conn-prepared"); + expect(navigateTopLevelMock).toHaveBeenCalledWith( + "https://mcp.notion.com/authorize?state=retry", + ); + }); + + it("recovers a response-lost Notion draft before retrying creation", async () => { + mockSearch.value = "source=notion"; + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + listApplicationsMock + .mockResolvedValueOnce({ applications: [] }) + .mockResolvedValueOnce({ + applications: [{ + id: "app-response-lost", + status: "draft", + metadata: { sourceTemplateKey: "notion" }, + }], + }); + listConnectionsMock + .mockResolvedValueOnce({ connections: [] }) + .mockResolvedValueOnce({ + connections: [{ + id: "conn-response-lost", + applicationId: "app-response-lost", + authKind: "oauth", + status: "draft", + config: { sourceTemplateKey: "notion" }, + transportConfig: {}, + }], + }); + connectAppMock.mockRejectedValueOnce(new Error("Response lost")); + startOAuthMock.mockResolvedValueOnce({ + connectionId: "conn-response-lost", + provider: "notion", + authorizationUrl: "https://mcp.notion.com/authorize?state=recovered", + expiresAt: "2099-01-01T00:00:00.000Z", + }); + + await render(); + + expect(container.textContent).toContain("Response lost"); + await act(async () => { + buttonByText("Try again")?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + await flushReact(); + await flushReact(); + + expect(connectAppMock).toHaveBeenCalledTimes(1); + expect(startOAuthMock).toHaveBeenCalledWith("conn-response-lost"); + expect(navigateTopLevelMock).toHaveBeenCalledWith( + "https://mcp.notion.com/authorize?state=recovered", + ); + }); + + it("keeps non-allowlisted OAuth apps blocked", async () => { + const slack = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "slack")!; + mockParams.appKey = "slack"; + listGalleryMock.mockResolvedValueOnce({ apps: [slack] }); + + await render(); + + expect(connectAppMock).not.toHaveBeenCalled(); + expect(mockNavigate).toHaveBeenCalledWith("/apps/connect", { replace: true }); + }); + + it("routes the enabled Notion gallery tile through the generic source deep link", async () => { + listGalleryMock.mockResolvedValueOnce({ apps: [NOTION] }); + await render(); + + const notionTile = buttonContaining("Notion"); + expect(notionTile?.disabled).toBe(false); + expect(notionTile?.textContent).toContain("Connect"); + + await act(async () => { + notionTile?.dispatchEvent(new MouseEvent("click", { bubbles: true })); + }); + expect(mockNavigate).toHaveBeenCalledWith("/apps/connect?source=notion"); + }); + it("choosing No and clicking Check link connects with no credentials", async () => { await render(); await gotoLinkFrame(container, "https://www.example.com/actions"); diff --git a/ui/src/pages/apps/AppsConnect.tsx b/ui/src/pages/apps/AppsConnect.tsx index a1fbd41ea3b..2de37c4a241 100644 --- a/ui/src/pages/apps/AppsConnect.tsx +++ b/ui/src/pages/apps/AppsConnect.tsx @@ -17,6 +17,8 @@ import type { Agent, AppDefinition, ConnectToolAppResult, + ToolApplication, + ToolConnection, ToolAppConnectionActionSummary, } from "@paperclipai/shared"; import { credentialConfigPath, getAppDefinitionForUrl, getAvailableConnectionMethod } from "@paperclipai/shared"; @@ -40,11 +42,14 @@ import { ToggleSwitch } from "@/components/ui/toggle-switch"; import { Skeleton } from "@/components/ui/skeleton"; import { cn } from "@/lib/utils"; import { copyTextToClipboard } from "@/lib/clipboard"; +import { navigateTopLevel } from "@/lib/browserNavigation"; import { AppLogo } from "./AppLogo"; +import { appSourceConnectHref, isMcpDirectOAuthConnectSlug } from "./app-connect-policy"; import { parseGoogleSheetIds } from "./google-sheets"; import { autoExtendNotice, INSTALL_ALL_WARNING, installInfoNotice, installPayload } from "@/lib/tool-installs"; type Step = "gallery" | "key" | "actions" | "who" | "install" | "success"; +export type OAuthConnectPhase = "entry" | "starting" | "redirecting" | "error"; const ROUTE_STAGE_BY_STEP: Partial> = { key: "setup", @@ -88,6 +93,36 @@ function isGoogleSheetsEntry(entry: AppDefinition | null): boolean { return entry?.slug === "google-sheets"; } +function appSourceSlug(application: ToolApplication): string | null { + const metadata = application.metadata; + if (!metadata) return null; + const source = metadata.sourceTemplateKey ?? metadata.galleryKey; + return typeof source === "string" ? source : null; +} + +function connectionSourceSlug(connection: ToolConnection): string | null { + const source = connection.config?.sourceTemplateKey ?? connection.transportConfig.sourceTemplateKey; + return typeof source === "string" ? source : null; +} + +function reusableOAuthConnection( + sourceSlug: string | null, + applications: ToolApplication[], + connections: ToolConnection[], +): ToolConnection | null { + if (!sourceSlug) return null; + const matchingApplicationIds = new Set( + applications + .filter((application) => application.status !== "archived" && appSourceSlug(application) === sourceSlug) + .map((application) => application.id), + ); + return connections.find((connection) => + connection.status !== "archived" && + connection.authKind === "oauth" && + (matchingApplicationIds.has(connection.applicationId) || connectionSourceSlug(connection) === sourceSlug) + ) ?? null; +} + export function AppsConnect() { const navigate = useNavigate(); const routeParams = useParams<{ appKey?: string }>(); @@ -96,7 +131,10 @@ export function AppsConnect() { const { pushToast } = useToast(); const [searchParams] = useSearchParams(); const appKey = routeParams.appKey ?? searchParams.get("appKey") ?? undefined; - const zapierSource = searchParams.get("source") === "zapier"; + const sourceSlug = searchParams.get("source")?.trim() || null; + const directOAuthSource = isMcpDirectOAuthConnectSlug(sourceSlug) ? sourceSlug : null; + const requestedAppKey = appKey ?? directOAuthSource ?? undefined; + const zapierSource = sourceSlug === "zapier"; // Prefill arrives from the app page for reconnects; read once so later // wizard navigation doesn't fight the URL. @@ -109,7 +147,7 @@ export function AppsConnect() { }; }); - const [step, setStep] = useState(appKey || prefill.link || zapierSource ? "key" : "gallery"); + const [step, setStep] = useState(requestedAppKey || prefill.link || zapierSource ? "key" : "gallery"); const [entry, setEntry] = useState(null); const [galleryName, setGalleryName] = useState(""); const [linkUrl, setLinkUrl] = useState(prefill.link); @@ -125,6 +163,10 @@ export function AppsConnect() { const [agentIds, setAgentIds] = useState>(new Set()); const [installMode, setInstallMode] = useState("none"); const [installAgentIds, setInstallAgentIds] = useState>(new Set()); + const [oauthPhase, setOAuthPhase] = useState("entry"); + const [oauthError, setOAuthError] = useState(null); + const directOAuthStartedRef = useRef(false); + const directOAuthRetryingRef = useRef(false); const openGallery = () => { setEntry(null); @@ -157,12 +199,142 @@ export function AppsConnect() { queryFn: () => toolsApi.listGallery(selectedCompanyId!), enabled: !!selectedCompanyId, }); + const applicationsQuery = useQuery({ + queryKey: queryKeys.tools.applications(selectedCompanyId ?? "__none__"), + queryFn: () => toolsApi.listApplications(selectedCompanyId!), + enabled: !!selectedCompanyId && !!directOAuthSource, + refetchOnMount: "always", + }); + const connectionsQuery = useQuery({ + queryKey: queryKeys.tools.connections(selectedCompanyId ?? "__none__"), + queryFn: () => toolsApi.listConnections(selectedCompanyId!), + enabled: !!selectedCompanyId && !!directOAuthSource, + refetchOnMount: "always", + }); + const existingOAuthConnection = useMemo( + () => reusableOAuthConnection( + directOAuthSource, + applicationsQuery.data?.applications ?? [], + connectionsQuery.data?.connections ?? [], + ), + [applicationsQuery.data, connectionsQuery.data, directOAuthSource], + ); + + const directOAuthEntry = entry && + getAvailableConnectionMethod(entry)?.auth === "oauth" && + isMcpDirectOAuthConnectSlug(entry.slug) + ? entry + : null; + + const setAppStep = (nextStep: Step) => { + setStep(nextStep); + if (entry) navigate(appConnectHref(entry.slug, nextStep)); + }; + + const oauthStartMutation = useMutation({ + mutationFn: (connectionId: string) => toolsApi.startOAuth(connectionId), + onSuccess: ({ authorizationUrl }) => { + setOAuthPhase("redirecting"); + navigateTopLevel(authorizationUrl); + }, + onError: (error) => { + const details = error instanceof ApiError && error.body && typeof error.body === "object" + ? (error.body as { details?: { code?: unknown } }).details + : null; + setOAuthPhase("error"); + setOAuthError( + details?.code === "invalid_grant" + ? "Your authorization expired or was revoked. Reconnect to continue." + : error instanceof Error + ? error.message + : "Paperclip couldn’t start secure sign-in. Try again.", + ); + }, + }); + const startOAuth = oauthStartMutation.mutate; + + const connectMutation = useMutation({ + mutationFn: (entryOverride?: AppDefinition) => { + const connectEntry = entryOverride ?? entry; + if (connectEntry) { + const sheetIds = isGoogleSheetsEntry(connectEntry) ? parseGoogleSheetIds(googleSheetsLinks).ids : []; + const trimmedGalleryName = galleryName.trim(); + return toolsApi.connectApp(selectedCompanyId!, { + galleryKey: connectEntry.slug, + name: trimmedGalleryName || connectEntry.name, + credentialValues: credentials, + configValues: isGoogleSheetsEntry(connectEntry) ? { allowedSpreadsheetIds: sheetIds } : undefined, + applicationId: prefill.applicationId, + }); + } + const trimmedKey = linkNeedsKey ? linkKey.trim() : ""; + const trimmedName = linkName.trim(); + return toolsApi.connectApp(selectedCompanyId!, { + link: linkUrl, + name: trimmedName || undefined, + credentialValues: trimmedKey ? { [LINK_CREDENTIAL_CONFIG_PATH]: trimmedKey } : undefined, + applicationId: prefill.applicationId, + }); + }, + onSuccess: (result) => { + if (result.auth?.kind === "oauth") { + setConnectResult(result); + const startUrl = result.auth.startUrl?.trim(); + if (!startUrl) { + setOAuthPhase("starting"); + startOAuth(result.connectionId); + return; + } + setOAuthPhase("redirecting"); + navigateTopLevel(startUrl); + return; + } + setConnectResult(result); + const defaults: Record = {}; + for (const a of result.actions.readOnly) defaults[a.catalogEntryId] = true; + for (const a of result.actions.canMakeChanges) defaults[a.catalogEntryId] = false; + setEnabled(defaults); + setInstallMode("none"); + setInstallAgentIds(new Set()); + setAppStep("actions"); + }, + onError: (error) => { + const details = error instanceof ApiError && error.body && typeof error.body === "object" + ? (error.body as { details?: { code?: unknown } }).details + : null; + if (isMcpDirectOAuthConnectSlug(requestedAppKey)) { + setOAuthPhase("error"); + setOAuthError( + details?.code === "invalid_grant" + ? "Your authorization expired or was revoked. Reconnect to continue." + : error instanceof Error + ? error.message + : "Paperclip couldn’t start secure sign-in. Try again.", + ); + return; + } + const oauthRequired = details?.code === "oauth_challenge"; + pushToast({ + title: oauthRequired ? "Sign-in required" : "Couldn’t connect", + body: oauthRequired + ? "This app needs you to sign in - coming soon." + : error instanceof Error + ? error.message + : "Please check your key and try again.", + tone: "error", + }); + }, + }); + const connectApp = connectMutation.mutate; useEffect(() => { - if (!appKey || galleryQuery.isLoading || !galleryQuery.data) return; + if (!requestedAppKey || galleryQuery.isLoading || !galleryQuery.data) return; - const requestedEntry = galleryQuery.data.apps.find((candidate) => candidate.slug === appKey); - if (!requestedEntry || getAvailableConnectionMethod(requestedEntry)?.auth === "oauth" || requestedEntry.availability?.available === false) { + const requestedEntry = galleryQuery.data.apps.find((candidate) => candidate.slug === requestedAppKey); + const method = requestedEntry ? getAvailableConnectionMethod(requestedEntry) : null; + const directOAuth = method?.auth === "oauth" && isMcpDirectOAuthConnectSlug(requestedEntry?.slug); + const unsupportedOAuth = method?.auth === "oauth" && !directOAuth; + if (!requestedEntry || unsupportedOAuth || requestedEntry.availability?.available === false) { setEntry(null); setStep("gallery"); navigate("/apps/connect", { replace: true }); @@ -184,61 +356,42 @@ export function AppsConnect() { setInstallMode("none"); setInstallAgentIds(new Set()); setStep("key"); - }, [appKey, entry?.slug, galleryQuery.data, galleryQuery.isLoading, navigate]); - const setAppStep = (nextStep: Step) => { - setStep(nextStep); - if (entry) navigate(appConnectHref(entry.slug, nextStep)); - }; + if (directOAuth && ( + !applicationsQuery.isFetchedAfterMount || + !connectionsQuery.isFetchedAfterMount + )) return; + if (directOAuth && directOAuthRetryingRef.current) return; + if (directOAuth && (applicationsQuery.isError || connectionsQuery.isError)) { + setOAuthPhase("error"); + setOAuthError("Paperclip couldn’t check for an existing connection. Try again."); + return; + } - const connectMutation = useMutation({ - mutationFn: () => { - if (entry) { - const sheetIds = isGoogleSheetsEntry(entry) ? parseGoogleSheetIds(googleSheetsLinks).ids : []; - const trimmedGalleryName = galleryName.trim(); - return toolsApi.connectApp(selectedCompanyId!, { - galleryKey: entry.slug, - name: trimmedGalleryName || undefined, - credentialValues: credentials, - configValues: isGoogleSheetsEntry(entry) ? { allowedSpreadsheetIds: sheetIds } : undefined, - applicationId: prefill.applicationId, - }); + if (directOAuth && !directOAuthStartedRef.current) { + directOAuthStartedRef.current = true; + setOAuthError(null); + setOAuthPhase("starting"); + if (existingOAuthConnection) { + startOAuth(existingOAuthConnection.id); + } else { + connectApp(requestedEntry); } - const trimmedKey = linkNeedsKey ? linkKey.trim() : ""; - const trimmedName = linkName.trim(); - return toolsApi.connectApp(selectedCompanyId!, { - link: linkUrl, - name: trimmedName || undefined, - credentialValues: trimmedKey ? { [LINK_CREDENTIAL_CONFIG_PATH]: trimmedKey } : undefined, - applicationId: prefill.applicationId, - }); - }, - onSuccess: (result) => { - setConnectResult(result); - const defaults: Record = {}; - for (const a of result.actions.readOnly) defaults[a.catalogEntryId] = true; - for (const a of result.actions.canMakeChanges) defaults[a.catalogEntryId] = false; - setEnabled(defaults); - setInstallMode("none"); - setInstallAgentIds(new Set()); - setAppStep("actions"); - }, - onError: (error) => { - const details = error instanceof ApiError && error.body && typeof error.body === "object" - ? (error.body as { details?: { code?: unknown } }).details - : null; - const oauthRequired = details?.code === "oauth_challenge"; - pushToast({ - title: oauthRequired ? "Sign-in required" : "Couldn’t connect", - body: oauthRequired - ? "This app needs you to sign in - coming soon." - : error instanceof Error - ? error.message - : "Please check your key and try again.", - tone: "error", - }); - }, - }); + } + }, [ + applicationsQuery.isError, + applicationsQuery.isFetchedAfterMount, + connectApp, + connectionsQuery.isError, + connectionsQuery.isFetchedAfterMount, + entry?.slug, + existingOAuthConnection, + galleryQuery.data, + galleryQuery.isLoading, + navigate, + requestedAppKey, + startOAuth, + ]); const finishMutation = useMutation({ mutationFn: async () => { @@ -280,6 +433,55 @@ export function AppsConnect() { return
Select a company to connect apps.
; } + if (directOAuthEntry && step === "key") { + return ( + { + setOAuthError(null); + setOAuthPhase("starting"); + const connectionId = connectResult?.connectionId ?? existingOAuthConnection?.id; + if (connectionId) { + startOAuth(connectionId); + return; + } + + // The create request may have reached the server even when its + // response did not reach the browser. Re-read both resources before + // creating again so Retry resumes that durable draft instead of + // duplicating it. + directOAuthRetryingRef.current = true; + try { + const [applicationsResult, connectionsResult] = await Promise.all([ + applicationsQuery.refetch(), + connectionsQuery.refetch(), + ]); + if (applicationsResult.isError || connectionsResult.isError) { + setOAuthPhase("error"); + setOAuthError("Paperclip couldn’t check for an existing connection. Try again."); + return; + } + const refreshedConnection = reusableOAuthConnection( + directOAuthSource, + applicationsResult.data?.applications ?? [], + connectionsResult.data?.connections ?? [], + ); + if (refreshedConnection) { + startOAuth(refreshedConnection.id); + } else { + connectMutation.mutate(directOAuthEntry); + } + } finally { + directOAuthRetryingRef.current = false; + } + }} + onCancel={() => navigate("/apps/browse")} + /> + ); + } + const appName = connectResult?.application.name ?? entry?.name ?? @@ -326,6 +528,13 @@ export function AppsConnect() { byo={searchParams.get("byo") === "1"} source={searchParams.get("source")} onPick={(picked) => { + if ( + getAvailableConnectionMethod(picked)?.auth === "oauth" && + isMcpDirectOAuthConnectSlug(picked.slug) + ) { + navigate(appSourceConnectHref(picked.slug)); + return; + } setEntry(picked); setGalleryName(picked.name); setLinkUrl(""); @@ -388,7 +597,7 @@ export function AppsConnect() { return; } } - connectMutation.mutate(); + connectMutation.mutate(undefined); }} /> )} @@ -407,7 +616,7 @@ export function AppsConnect() { onKeyChange={setLinkKey} submitting={connectMutation.isPending} onBack={() => setStep("gallery")} - onConnect={() => connectMutation.mutate()} + onConnect={() => connectMutation.mutate(undefined)} /> )} @@ -417,7 +626,7 @@ export function AppsConnect() { onLinkChange={setLinkUrl} submitting={connectMutation.isPending} onBack={() => navigate("/apps/browse")} - onConnect={() => connectMutation.mutate()} + onConnect={() => connectMutation.mutate(undefined)} /> )} @@ -533,6 +742,85 @@ function StepHeader({ ); } +export function OAuthConnectStateScreen({ + entry, + phase, + error, + onRetry, + onCancel, +}: { + entry: AppDefinition; + phase: OAuthConnectPhase; + error?: string | null; + onRetry: () => void; + onCancel: () => void; +}) { + const status = phase === "entry" + ? { + title: `Connect ${entry.name} to Paperclip`, + body: `Paperclip will open ${entry.name} so you can choose a workspace and approve access.`, + } + : phase === "starting" + ? { + title: "Preparing secure sign-in", + body: `Paperclip is creating a secure ${entry.name} connection.`, + } + : phase === "redirecting" + ? { + title: `Opening ${entry.name}`, + body: `Continue in ${entry.name} to choose a workspace and approve access.`, + } + : { + title: `${entry.name} couldn’t connect`, + body: error ?? "Paperclip couldn’t start secure sign-in. Try again.", + }; + + return ( +
+ +
+
+ + {phase === "error" ? ( + + ) : phase === "entry" ? ( + + ) : ( + + )} + +
+

{status.title}

+

{status.body}

+
+
+ +
+ {phase === "error" ? ( + + ) : ( + + )} + +
+

+ + Your authorization stays in Paperclip’s encrypted secret store. +

+
+
+ ); +} + function ZapierConnectStep({ link, onLinkChange, @@ -675,12 +963,13 @@ function GalleryStep({ {filtered.map((app) => { const copy = appCopyFor(app.slug, app.description); const oauth = getAvailableConnectionMethod(app)?.auth === "oauth"; + const oauthBlocked = oauth && !isMcpDirectOAuthConnectSlug(app.slug); const unavailable = app.availability?.available === false; return ( + ) : ( + + )}
); diff --git a/ui/src/pages/apps/app-detail/SetupPanel.tsx b/ui/src/pages/apps/app-detail/SetupPanel.tsx index 6d031242964..105e57204ca 100644 --- a/ui/src/pages/apps/app-detail/SetupPanel.tsx +++ b/ui/src/pages/apps/app-detail/SetupPanel.tsx @@ -46,7 +46,7 @@ export function SetupPanel({ {hasOAuthSignIn && ( ).connectedAt)} - isSmokeLabFixture={isSmokeLabFixture} + providerName={appDefinitionSlug(galleryEntry) === "notion" ? "Notion" : isSmokeLabFixture ? "Smoke OAuth" : "OAuth"} disabled={oauthStartDisabled} onStart={onStartOAuth} /> @@ -58,26 +58,25 @@ export function SetupPanel({ function OAuthConnectionSection({ connected, - isSmokeLabFixture, + providerName, disabled, onStart, }: { connected: boolean; - isSmokeLabFixture: boolean; + providerName: string; disabled: boolean; onStart: () => void; }) { - const providerName = isSmokeLabFixture ? "Smoke OAuth" : "OAuth"; return (

- {connected ? `Connected with ${providerName}` : `Connect with ${providerName}`} + {connected ? `${providerName} connected` : `Connect with ${providerName}`}

{connected - ? "Sign in again to replace this connection's OAuth session." + ? "Your workspace authorization is active. Reconnect any time to replace it." : "Open the provider's consent page to finish connecting this app."}

diff --git a/ui/src/pages/apps/store-cards.tsx b/ui/src/pages/apps/store-cards.tsx index 0cc505c12cc..2bda5a0c204 100644 --- a/ui/src/pages/apps/store-cards.tsx +++ b/ui/src/pages/apps/store-cards.tsx @@ -1,6 +1,7 @@ import { ServerCog, Wrench } from "lucide-react"; import { Link } from "@/lib/router"; import { advancedTabHref } from "@/pages/tools/tool-tabs"; +import { appSourceConnectHref } from "./app-connect-policy"; /** Popular gallery keys surfaced first in the Browse store (PAP-13254, door 1). */ export const POPULAR_KEYS = ["zapier", "github", "slack", "notion", "linear"]; @@ -11,6 +12,9 @@ export const BYO_CONNECT_HREF = "/apps/connect?byo=1"; /** Zapier connects with the complete MCP URL issued by Zapier. */ export const ZAPIER_CONNECT_HREF = "/apps/connect?byo=1&source=zapier"; +/** MCP-direct OAuth apps enter through the generic source deep link. */ +export const NOTION_CONNECT_HREF = appSourceConnectHref("notion"); + /** * First-class "Connect your own tool" card (PAP-12371, Finding C; PAP-13254). * Lives in Browse as a persistent row and launches the guided URL flow. diff --git a/ui/storybook/stories/notion-connect-flow.stories.tsx b/ui/storybook/stories/notion-connect-flow.stories.tsx new file mode 100644 index 00000000000..b5f4e584557 --- /dev/null +++ b/ui/storybook/stories/notion-connect-flow.stories.tsx @@ -0,0 +1,177 @@ +import { useMemo } from "react"; +import type { Meta, StoryObj } from "@storybook/react-vite"; +import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; +import { + CONNECTABLE_APP_DEFINITIONS, + type AppDefinition, + type ToolConnection, +} from "@paperclipai/shared"; +import { queryKeys } from "@/lib/queryKeys"; +import { Browse } from "@/pages/apps/Browse"; +import { AppLogo } from "@/pages/apps/AppLogo"; +import { + OAuthConnectStateScreen, + type OAuthConnectPhase, +} from "@/pages/apps/AppsConnect"; +import { SetupPanel } from "@/pages/apps/app-detail/SetupPanel"; +import { ReconnectCard } from "@/pages/apps/app-detail/AdvancedPanel"; + +const COMPANY_ID = "company-storybook"; +const NOTION = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "notion") as AppDefinition; +const ZAPIER = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "zapier") as AppDefinition; +const GITHUB = CONNECTABLE_APP_DEFINITIONS.find((app) => app.slug === "github") as AppDefinition; + +function seededClient() { + const client = new QueryClient({ + defaultOptions: { + queries: { staleTime: Infinity, gcTime: Infinity, retry: false, refetchOnMount: false }, + }, + }); + client.setQueryData(queryKeys.apps.gallery(COMPANY_ID), { apps: [NOTION, ZAPIER, GITHUB] }); + return client; +} + +function BrowseHost() { + const client = useMemo(() => seededClient(), []); + return ( + +
+ +
+ + ); +} + +function OAuthStateHost({ phase, error }: { phase: OAuthConnectPhase; error?: string }) { + return ( +
+ undefined} + onCancel={() => undefined} + /> +
+ ); +} + +function notionConnection(overrides: Partial = {}): ToolConnection { + return { + id: "connection-notion", + companyId: COMPANY_ID, + applicationId: "application-notion", + name: "Notion", + uid: "notion-storybook", + connectionKind: "managed", + ownership: "dcr", + transport: "mcp_remote", + authKind: "oauth", + status: "active", + transportConfig: { url: "https://mcp.notion.com/mcp" }, + config: { + url: "https://mcp.notion.com/mcp", + sourceTemplateKey: "notion", + oauth: { provider: "notion", connectedAt: "2026-08-06T19:00:00.000Z" }, + }, + credentialSecretRefs: [], + credentialRefs: [], + healthStatus: "healthy", + healthMessage: null, + healthCheckedAt: new Date("2026-08-06T19:00:00.000Z"), + lastError: null, + enabled: true, + createdByAgentId: null, + createdByUserId: "board-user", + createdAt: new Date("2026-08-06T18:55:00.000Z"), + updatedAt: new Date("2026-08-06T19:00:00.000Z"), + ...overrides, + }; +} + +function ConnectedHost() { + return ( +
+
+ +
+

Notion

+

Connected app setup

+
+
+ undefined} + appToggleDisabled={false} + onUpdateConfig={() => undefined} + configUpdateDisabled={false} + onStartOAuth={() => undefined} + oauthStartDisabled={false} + /> +
+ ); +} + +function ReconnectRequiredHost() { + return ( +
+
+

Notion

+

Connection needs attention

+
+ undefined} + /> +
+ ); +} + +const meta: Meta = { + title: "Apps/Notion MCP connect flow (PAP-16650)", + parameters: { layout: "fullscreen" }, +}; +export default meta; + +type Story = StoryObj; + +export const BrowseEntry: Story = { + name: "1 β€” Browse entry", + render: () => , +}; + +export const ConnectEntry: Story = { + name: "2 β€” Connect entry", + render: () => , +}; + +export const InFlight: Story = { + name: "3 β€” In flight", + render: () => , +}; + +export const Connected: Story = { + name: "4 β€” Connected", + render: () => , +}; + +export const ConnectError: Story = { + name: "5 β€” Connect error", + render: () => ( + + ), +}; + +export const ReconnectRequired: Story = { + name: "6 β€” Reconnect required", + render: () => , +}; From 9ace548fd21536f83ca5dd8181bf978bc2a723ea Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 6 Aug 2026 22:31:15 -0700 Subject: [PATCH 010/311] feat(observability): rename sandbox provider spans and add run-time wrapper spans (#10999) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip uses adapter and sandbox code to start agents and run sandbox work > - The current sandbox spans use mixed names and do not group related run-time work > - Mixed names make traces harder to read and compare across providers > - This pull request renames provider spans, adds run-time wrapper spans, and keeps the host allowlist closed > - The benefit is clearer traces with the same sandbox behavior and trust boundary ## Linked Issues or Issue Description **What existing behavior does this improve?** This improves OpenTelemetry span names and grouping for sandbox startup, execution, callback relay, and agent session work. **Subsystem affected** Cross-cutting (multiple of the above): adapter utilities, sandbox providers, shared telemetry documentation, and server instrumentation. **Current behavior** Sandbox provider spans use mixed names. Related run-time operations expose inner `sandbox.exec` spans without a named wrapper span. The host mapper uses a closed allowlist for provider span names. **Proposed behavior** Use descriptive provider-scoped span names. Add wrapper spans for agent session input, agent session output polling, and callback relay. Keep the host mapper allowlist closed and map unknown names to `other`. **Reason and benefit** Clear names make traces easier to read and reduce ambiguity during sandbox operation analysis. Wrapper spans show the full operation while preserving the inner execution spans. **Breaking changes** None. This change updates telemetry span names and grouping only. It does not change sandbox behavior, endpoint behavior, or the host trust boundary. **Additional context** Related prior work: [#10758](https://github.com/paperclipai/paperclip/pull/10758). ## What Changed - Rename Daytona provider sync and session spans with descriptive provider-scoped names. - Add three run-time wrapper spans for agent session input, output polling, and callback relay. - Add a shared span runner that preserves no-op behavior without a real tracer. - Keep the host mapper allowlist closed and map unknown names to `other`. - Update telemetry documentation and span-name tests. ## Verification - Focused adapter-utils span tests pass for startup timing, callback relay, and sandbox execution. - Focused Daytona plugin span tests pass for renamed leaf spans and session open or close spans. - Focused server tests pass for host mapping and instrumentation. - The stacked diff contains one commit on top of `feat/daytona-persistent-session-model`. ## Risks - Span names change for existing telemetry consumers. - The wrapper spans add trace structure but do not change sandbox execution. - The host mapper keeps the existing closed allowlist and `other` bucket. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-5 (Codex agent); exact deployment revision and context window are not exposed in this run; tool use and code execution enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- .../adapter-utils/src/acpx-engine/execute.ts | 18 ++- .../src/acpx-engine/startup-timing.test.ts | 61 ++++++++- .../src/acpx-engine/startup-timing.ts | 60 +++++++++ .../src/execution-target-sandbox.test.ts | 125 ++++++++++++++++++ .../adapter-utils/src/execution-target.ts | 82 ++++++++---- .../src/sandbox-callback-bridge.test.ts | 48 +++++++ .../src/sandbox-callback-bridge.ts | 34 +++-- .../daytona/src/file-sync.ts | 32 +++-- .../daytona/src/plugin.test.ts | 56 +++++--- .../sandbox-providers/daytona/src/plugin.ts | 20 +-- packages/shared/src/telemetry/README.md | 44 ++++-- .../src/__tests__/environment-runtime.test.ts | 2 +- server/src/__tests__/instrumentation.test.ts | 6 +- .../plugin-host-services-span.test.ts | 40 +++--- server/src/services/plugin-host-services.ts | 34 ++--- 15 files changed, 539 insertions(+), 123 deletions(-) diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 868df3897f6..a43c2543e1e 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -84,12 +84,14 @@ import { DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS, } from "./constants.js"; import { + createRuntimeSpanRunner, emitSkippedStartupStep, measureStartupStep, NOOP_STARTUP_SPAN, NOOP_STARTUP_TRACE_CONTEXT, runWithRuntimeParent, setSandboxRootSpanAttributes, + type RuntimeSpanRunner, type SandboxRootSpanContext, type StartupSpan, type StartupSpanContext, @@ -1367,6 +1369,12 @@ async function buildRuntime(input: { // run closure passes the run-scoped getter here; when it is absent, each // bridge site keeps its earlier unparented run-time behavior. getRuntimeParentContext?: () => StartupSpanContext | undefined; + // Wrap each unit of bridge run-time work in its own named span. + // `buildRuntime` threads it into the two remote bridge factories, so the + // socket handler, the poll loop, and the callback worker each open a wrapper + // span per unit of work. The run closure passes the run-scoped runner here; + // when it is absent, each bridge site opens no wrapper span. + runtimeSpan?: RuntimeSpanRunner; }): Promise { const { runId, agent, config, context, authToken } = input.ctx; // Injectable monotonic clock for per-step startup timing. Hoisted above the @@ -1963,6 +1971,7 @@ async function buildRuntime(input: { hostApiToken: env.PAPERCLIP_API_KEY, onLog: input.ctx.onLog, getRuntimeParentContext: input.getRuntimeParentContext, + runtimeSpan: input.runtimeSpan, }), concurrentBridgeStepMetrics, ); @@ -1994,6 +2003,7 @@ async function buildRuntime(input: { timeoutSec, onLog: input.ctx.onLog, getRuntimeParentContext: input.getRuntimeParentContext, + runtimeSpan: input.runtimeSpan, }), concurrentBridgeStepMetrics, ); @@ -3131,6 +3141,12 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // reads it through `getRuntimeParentContext` to parent to the live span. let currentRunParentContext: StartupSpanContext | undefined = runRootSpan.parentContext; const getRuntimeParentContext = (): StartupSpanContext | undefined => currentRunParentContext; + // Wrap each unit of bridge run-time work (one outbound ACP message, one poll + // tick, one callback request) in its own named span, parented to the live run + // span. The runner reads the run parent per call through + // `getRuntimeParentContext`, so a wrapper span always parents to the current + // run span. On a no-op trace context the runner opens no real span. + const runRuntimeSpan = createRuntimeSpanRunner(tracing, getRuntimeParentContext); // `runFailed` marks the run root span status at end time. It stays `true` // until the run reaches a clean completed turn, so every failure and every // early exit closes the span with error status. @@ -3190,7 +3206,7 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { // parents to its step span. On a local or SSH target // `spanParent.parentContext` is a no-op token, so the wrap is inert. prepared = await runWithRuntimeParent(spanParent.parentContext, () => - buildRuntime({ ctx, engine, deps, spanParent, getRuntimeParentContext }), + buildRuntime({ ctx, engine, deps, spanParent, getRuntimeParentContext, runtimeSpan: runRuntimeSpan }), ); } catch (err) { rootSpan.end(true); diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts index 1678282143d..14c7e0bfe22 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts @@ -1,12 +1,14 @@ import { describe, expect, it, vi } from "vitest"; import type { AdapterRuntimeEvent } from "../types.js"; -import type { StartupSpan, StartupTracer } from "./startup-timing.js"; +import type { StartupSpan, StartupTraceContext, StartupTracer } from "./startup-timing.js"; import { clampSpanLabel, + createRuntimeSpanRunner, emitSkippedStartupStep, getActiveStepContext, measureStartupStep, NOOP_STARTUP_SPAN, + NOOP_STARTUP_TRACE_CONTEXT, normalizeProviderFamily, runWithoutActiveStep, runWithRuntimeParent, @@ -653,3 +655,60 @@ describe("clampSpanLabel", () => { expect(clampSpanLabel("stdout", "secret output")).toBeUndefined(); }); }); + +describe("createRuntimeSpanRunner", () => { + it("opens a named wrapper span and parents the wrapped work to it", async () => { + const { tracer, spans } = makeMockTracer(); + const runParent = { marker: "run-parent" }; + const traceContext: StartupTraceContext = { + tracer, + contextWithSpan: (span) => ({ span }), + }; + const run = createRuntimeSpanRunner(traceContext, () => runParent); + + let childParent: unknown = "unset"; + const result = await run("sandbox.agentSession.sendInput", async () => { + childParent = getActiveStepContext()?.parentContext; + return "ok"; + }); + + expect(result).toBe("ok"); + expect(spans).toHaveLength(1); + expect(spans[0]!.name).toBe("sandbox.agentSession.sendInput"); + expect(spans[0]!.endCount).toBe(1); + // The wrapped work parents to the wrapper span, not straight to the run + // parent, so the inner exec spans group under the wrapper span. + expect((childParent as { span?: unknown }).span).toBe(spans[0]); + }); + + it("marks the wrapper span failed and ends it when the work throws", async () => { + const { tracer, spans } = makeMockTracer(); + const traceContext: StartupTraceContext = { + tracer, + contextWithSpan: (span) => ({ span }), + }; + const run = createRuntimeSpanRunner(traceContext, () => ({ marker: "run-parent" })); + + await expect( + run("sandbox.callbackBridge.relayRequest", async () => { + throw new Error("boom"); + }), + ).rejects.toThrow(/boom/); + expect(spans).toHaveLength(1); + expect(spans[0]!.status?.code).toBe(2); + expect(spans[0]!.endCount).toBe(1); + }); + + it("runs the work unwrapped under a no-op trace context", async () => { + const run = createRuntimeSpanRunner(NOOP_STARTUP_TRACE_CONTEXT, () => ({ marker: "run-parent" })); + // The no-op `contextWithSpan` yields no child token, so the store empties for + // the work, exactly like the earlier unparented run-time behavior. + let childStep: unknown = "unset"; + const result = await run("sandbox.agentSession.pollOutput", async () => { + childStep = getActiveStepContext(); + return 7; + }); + expect(result).toBe(7); + expect(childStep).toBeNull(); + }); +}); diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index f49f3ccfd62..ce91ca6a5bd 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -402,6 +402,66 @@ export function runWithRuntimeParent( return activeStepContextStorage.run(activeStep, work); } +/** + * Run one run-time operation inside its own wrapper span. The runner opens a + * wrapper span parented to the current run span, publishes the wrapper span as + * the runtime parent while `work` runs, and ends the span when `work` settles. + * A child `sandbox.exec` span inside `work` parents to the wrapper span, so the + * trace groups the operation's execs under one named span. A throwing `work` + * sets the wrapper span error status before the span ends. + * + * The runner reads the run parent per call, so it always parents to the live + * span (`agent.turn` during the turn, `task.run` otherwise). The default runner + * opens no real span; it only runs `work` under the current run parent, so the + * span path stays a no-op until the server injects a real tracer. + */ +export type RuntimeSpanRunner = (name: string, work: () => Promise) => Promise; + +/** + * Build a {@link RuntimeSpanRunner} from a trace context and the run-parent + * getter. The runner opens the wrapper span through `traceContext.tracer`, and + * it derives the wrapper span's child parent token through + * `traceContext.contextWithSpan`. A no-op trace context yields a runner that + * opens no real span and runs `work` under the current run parent, so the span + * path stays inert until the server injects a real tracer. Every tracer call + * sits inside an error swallow, so a throwing tracer never changes control flow. + */ +export function createRuntimeSpanRunner( + traceContext: StartupTraceContext, + getRuntimeParentContext: () => StartupSpanContext | undefined, +): RuntimeSpanRunner { + return async (name: string, work: () => Promise): Promise => { + const parentContext = getRuntimeParentContext(); + let span: StartupSpan; + try { + span = traceContext.tracer.startSpan(name, undefined, parentContext); + } catch { + // A throwing tracer must not change control flow; run `work` unwrapped. + return runWithRuntimeParent(parentContext, work); + } + let childContext: StartupSpanContext; + try { + childContext = traceContext.contextWithSpan(span); + } catch { + childContext = parentContext; + } + let failed = false; + try { + return await runWithRuntimeParent(childContext, work); + } catch (err) { + failed = true; + throw err; + } finally { + try { + if (failed) span.setStatus({ code: SPAN_STATUS_CODE_ERROR }); + span.end(); + } catch { + // Observability must not change control flow. + } + } + }; +} + /** * Set a numeric span attribute only when the value is a finite number. A reader * that returns `undefined` (the counter is unavailable) yields no attribute, diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts index 321e88cb154..d4425a6ddfc 100644 --- a/packages/adapter-utils/src/execution-target-sandbox.test.ts +++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts @@ -481,6 +481,131 @@ describe("sandbox adapter execution targets", () => { } }); + it("wraps a stdin write in a sandbox.agentSession.sendInput span", async () => { + // With a span runner injected, the socket handler wraps one outbound ACP + // message to the agent in a `sandbox.agentSession.sendInput` span. This test + // connects a socket, sends one stdin line, and proves the handler opens that + // wrapper span around the write. + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-sendinput-span-")); + cleanupDirs.push(rootDir); + const childPath = path.join(rootDir, "noop-acp-child.mjs"); + await writeFile(childPath, "process.stdin.on('data', () => {});\n", "utf8"); + + const spanNames: string[] = []; + let resolveSendInput: () => void = () => {}; + const sendInputObserved = new Promise((resolve) => { + resolveSendInput = resolve; + }); + + const target: AdapterSandboxExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "local-test", + remoteCwd: rootDir, + timeoutMs: 30_000, + runner: createLocalSandboxRunner(), + }; + + const bridge = await startAdapterExecutionTargetProcessSessionBridge({ + runId: "run-process-session-sendinput-span", + target, + runtimeRootDir: path.posix.join(rootDir, ".paperclip-runtime", "acpx"), + adapterKey: "acpx", + command: process.execPath, + args: [childPath], + cwd: rootDir, + env: {}, + timeoutSec: 5, + onLog: async () => {}, + // Record each wrapper span name, then run the wrapped work. + runtimeSpan: async (name, work) => { + spanNames.push(name); + if (name === "sandbox.agentSession.sendInput") resolveSendInput(); + return work(); + }, + }); + expect(bridge).not.toBeNull(); + + let peer: net.Socket | null = null; + try { + const proxySource = await readFile(bridge!.agentCommand, "utf8"); + const port = Number(/port: (\d+)/.exec(proxySource)?.[1] ?? Number.NaN); + const tokenLiteral = /const token = (".*?");/.exec(proxySource)?.[1]; + const token = JSON.parse(tokenLiteral as string) as string; + + const peerSocket = net.createConnection({ host: "127.0.0.1", port }); + peer = peerSocket; + peerSocket.on("error", () => undefined); + await new Promise((resolve, reject) => { + peerSocket.once("connect", () => resolve()); + peerSocket.once("error", reject); + }); + + // The first token-bearing message authenticates and writes the stdin file. + peerSocket.write( + `${JSON.stringify({ token, type: "stdin", data: Buffer.from("hi").toString("base64") })}\n`, + ); + + await sendInputObserved; + expect(spanNames).toContain("sandbox.agentSession.sendInput"); + } finally { + peer?.destroy(); + await bridge?.stop(); + } + }); + + it("wraps each poll tick in a sandbox.agentSession.pollOutput span", async () => { + // With a span runner injected, the poll timer wraps each 100 ms poll tick in + // a `sandbox.agentSession.pollOutput` span. This test lets the first poll tick + // fire and proves the timer opens that wrapper span. + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-poll-span-")); + cleanupDirs.push(rootDir); + const childPath = path.join(rootDir, "noop-acp-child.mjs"); + await writeFile(childPath, "process.stdin.on('data', () => {});\n", "utf8"); + + const spanNames: string[] = []; + let resolvePoll: () => void = () => {}; + const pollObserved = new Promise((resolve) => { + resolvePoll = resolve; + }); + + const target: AdapterSandboxExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "local-test", + remoteCwd: rootDir, + timeoutMs: 30_000, + runner: createLocalSandboxRunner(), + }; + + const bridge = await startAdapterExecutionTargetProcessSessionBridge({ + runId: "run-process-session-poll-span", + target, + runtimeRootDir: path.posix.join(rootDir, ".paperclip-runtime", "acpx"), + adapterKey: "acpx", + command: process.execPath, + args: [childPath], + cwd: rootDir, + env: {}, + timeoutSec: 5, + onLog: async () => {}, + // Record each wrapper span name, then run the wrapped work. + runtimeSpan: async (name, work) => { + spanNames.push(name); + if (name === "sandbox.agentSession.pollOutput") resolvePoll(); + return work(); + }, + }); + expect(bridge).not.toBeNull(); + + try { + await pollObserved; + expect(spanNames).toContain("sandbox.agentSession.pollOutput"); + } finally { + await bridge?.stop(); + } + }); + it("bridges bidirectional sandbox process sessions through a local ACPX-spawnable proxy", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-")); cleanupDirs.push(rootDir); diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index 0eb776e8cd2..5e7aa5719f2 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -48,6 +48,7 @@ import { sanitizeRemoteExecutionEnv } from "./remote-execution-env.js"; import { preferredShellForSandbox, shellCommandArgs } from "./sandbox-shell.js"; import { runWithRuntimeParent, + type RuntimeSpanRunner, type StartupSpanContext, } from "./acpx-engine/startup-timing.js"; import type { RuntimeProgressSink, RuntimeStatusSink } from "./runtime-progress.js"; @@ -1354,6 +1355,14 @@ async function waitForLocalServerListen(server: net.Server): Promise { return address.port; } +/** Span name that wraps the socket handler's one `writeTextFile` exec β€” one + * outbound ACP message to the agent. */ +const AGENT_SESSION_SEND_INPUT_SPAN = "sandbox.agentSession.sendInput"; + +/** Span name that wraps one 100 ms poll tick β€” `list`, then `read`+`remove` per + * file found (`1 + 2n` execs). */ +const AGENT_SESSION_POLL_OUTPUT_SPAN = "sandbox.agentSession.pollOutput"; + export async function startAdapterExecutionTargetProcessSessionBridge(input: { runId: string; target: AdapterExecutionTarget | null | undefined; @@ -1376,6 +1385,12 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: { // turn, `task.run` otherwise). When it is absent, the work runs with an empty // store, exactly like the earlier `runWithoutActiveStep` behavior. getRuntimeParentContext?: () => StartupSpanContext | undefined; + // Wrap each unit of run-time work in its own named span. The socket handler + // uses it for `sandbox.agentSession.sendInput` and the poll timer for + // `sandbox.agentSession.pollOutput`, so each unit's inner `sandbox.exec` spans + // group under one wrapper span. When it is absent, the work runs under the run + // parent with no wrapper span, exactly like the earlier behavior. + runtimeSpan?: RuntimeSpanRunner; }): Promise { if (!input.target || input.target.kind !== "remote" || input.target.transport !== "sandbox") { return null; @@ -1384,6 +1399,14 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: { const target = input.target; const onLog = input.onLog ?? (async () => {}); const runner = requireSandboxRunner(target); + // Run one unit of run-time work under its named wrapper span when a span + // runner is injected. Without a runner, run the work under the current run + // parent, so the inner `sandbox.exec` spans parent to the live run span, + // exactly like the earlier behavior. + const runRuntimeWork = (name: string, work: () => Promise): Promise => + input.runtimeSpan + ? input.runtimeSpan(name, work) + : runWithRuntimeParent(input.getRuntimeParentContext?.(), work); const shellCommand = preferredSandboxShell(target); const timeoutMs = typeof input.timeoutSec === "number" && Number.isFinite(input.timeoutSec) && input.timeoutSec > 0 @@ -1548,26 +1571,28 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: { socket = nextSocket; flushPendingRemoteEvents(); } - // Read the current-run parent now, at send time. The live parent - // switches to `agent.turn` during the turn and back to `task.run` - // after it. With no getter the store stays empty, exactly like the - // earlier unparented behavior. - runWithRuntimeParent(input.getRuntimeParentContext?.(), () => { - void (async () => { - if (message.type === "stdin" && typeof message.data === "string") { - stdinSeq += 1; - const name = `${String(stdinSeq).padStart(12, "0")}.json`; - await client.writeTextFile(path.posix.join(stdinDir, name), jsonLine({ type: "stdin", data: message.data })); - } else if (message.type === "stdinEnd") { - stdinSeq += 1; - const name = `${String(stdinSeq).padStart(12, "0")}.json`; - await client.writeTextFile(path.posix.join(stdinDir, name), jsonLine({ type: "stdinEnd" })); - } - })().catch((error) => { + // Wrap one outbound ACP message to the agent in a + // `sandbox.agentSession.sendInput` span, so its one `writeTextFile` exec + // groups under one named span. The span runner reads the current-run + // parent at send time: the live parent switches to `agent.turn` during + // the turn and back to `task.run` after it. A message that is neither + // `stdin` nor `stdinEnd` writes nothing, so it opens no span. + const stdinPayload = + message.type === "stdin" && typeof message.data === "string" + ? { type: "stdin", data: message.data } + : message.type === "stdinEnd" + ? { type: "stdinEnd" } + : null; + if (stdinPayload) { + stdinSeq += 1; + const name = `${String(stdinSeq).padStart(12, "0")}.json`; + void runRuntimeWork(AGENT_SESSION_SEND_INPUT_SPAN, () => + client.writeTextFile(path.posix.join(stdinDir, name), jsonLine(stdinPayload)), + ).catch((error) => { nextSocket.write(jsonLine({ type: "error", message: error instanceof Error ? error.message : String(error) })); nextSocket.destroy(); }); - }); + } } }); }); @@ -1600,15 +1625,16 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: { } }; - // Schedule the long-lived poll timer under the current-run parent context. - // The poll loop reads remote event files with run-time execs, not startup - // work, so a poll `sandbox.exec` span parents to the live run span, not to the - // ended bridge step. Read the getter per tick, because the re-arm timer that - // the poll body schedules reads it again: the live parent switches to - // `agent.turn` during the turn and back to `task.run` after it. With no getter - // the store stays empty, exactly like the earlier unparented behavior. + // Schedule the long-lived poll timer. Wrap each 100 ms poll tick in a + // `sandbox.agentSession.pollOutput` span, so the tick's `list` plus per-file + // `read`/`remove` execs group under one named span. The poll loop reads remote + // event files with run-time execs, not startup work, so the wrapper span and + // its child execs parent to the live run span, not to the ended bridge step. + // The span runner reads the run parent per tick, because the re-arm timer that + // the poll body schedules opens a new tick span: the live parent switches to + // `agent.turn` during the turn and back to `task.run` after it. const schedulePoll = () => { - pollTimer = setTimeout(() => runWithRuntimeParent(input.getRuntimeParentContext?.(), () => void poll()), 100); + pollTimer = setTimeout(() => void runRuntimeWork(AGENT_SESSION_POLL_OUTPUT_SPAN, poll), 100); pollTimer.unref?.(); }; @@ -1764,6 +1790,11 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { // `sandbox.exec` span parents to the live run span. When it is absent, the // request work runs with an empty store, exactly like the earlier behavior. getRuntimeParentContext?: () => StartupSpanContext | undefined; + // Wrap each callback request in a `sandbox.callbackBridge.relayRequest` span. + // The factory threads it into the worker, which uses it per request so each + // request's execs group under one wrapper span. When it is absent, the request + // work runs under the run parent with no wrapper span. + runtimeSpan?: RuntimeSpanRunner; }): Promise { if (!adapterExecutionTargetUsesPaperclipBridge(input.target)) { return null; @@ -1835,6 +1866,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { queueDir, maxBodyBytes, getRuntimeParentContext: input.getRuntimeParentContext, + runtimeSpan: input.runtimeSpan, handleRequest: async (request) => { const method = request.method.trim().toUpperCase() || "GET"; if (bridgeDebugEnabled) { diff --git a/packages/adapter-utils/src/sandbox-callback-bridge.test.ts b/packages/adapter-utils/src/sandbox-callback-bridge.test.ts index 85a361f431d..e69fef32c8c 100644 --- a/packages/adapter-utils/src/sandbox-callback-bridge.test.ts +++ b/packages/adapter-utils/src/sandbox-callback-bridge.test.ts @@ -611,6 +611,54 @@ describe("sandbox callback bridge", () => { expect((requestStep as { criticalPath?: boolean }).criticalPath).toBe(false); }); + it("wraps each request in a sandbox.callbackBridge.relayRequest span", async () => { + // With a span runner injected, the worker wraps each request in one + // `sandbox.callbackBridge.relayRequest` span, so the request's read, write, + // and remove execs group under one named span. This test drives the worker + // with a recording runner and proves it opens the wrapper span around the + // request work. + const wrapped: string[] = []; + let served = false; + let resolveServed: () => void = () => {}; + const requestServed = new Promise((resolve) => { + resolveServed = resolve; + }); + + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-relay-span-")); + cleanupDirs.push(rootDir); + const queueDir = path.posix.join(rootDir, "queue"); + + const worker = await startSandboxCallbackBridgeWorker({ + client: { + makeDir: async () => {}, + makeDirs: async () => {}, + listJsonFiles: async () => (served ? [] : ["000000000001.json"]), + readTextFile: async () => + JSON.stringify({ id: "req-1", method: "GET", path: "/", query: "", headers: {}, body: "" }), + writeTextFile: async () => {}, + rename: async () => {}, + remove: async () => {}, + }, + queueDir, + authorizeRequest: async () => null, + handleRequest: async () => { + served = true; + resolveServed(); + return { status: 200, body: "ok" }; + }, + // Record each wrapper span name, then run the wrapped work. + runtimeSpan: async (name, work) => { + wrapped.push(name); + return work(); + }, + }); + + await requestServed; + await worker.stop(); + + expect(wrapped).toContain("sandbox.callbackBridge.relayRequest"); + }); + it("test_paperclip_loop_exec_stays_unparented_without_getter", async () => { // With no `getRuntimeParentContext`, a request runs with an empty active // step store, exactly like the earlier `runWithoutActiveStep` behavior. So a diff --git a/packages/adapter-utils/src/sandbox-callback-bridge.ts b/packages/adapter-utils/src/sandbox-callback-bridge.ts index 48f64211d72..bd5a83ab5f7 100644 --- a/packages/adapter-utils/src/sandbox-callback-bridge.ts +++ b/packages/adapter-utils/src/sandbox-callback-bridge.ts @@ -6,6 +6,7 @@ import path from "node:path"; import { runWithoutActiveStep, runWithRuntimeParent, + type RuntimeSpanRunner, type StartupSpanContext, } from "./acpx-engine/startup-timing.js"; import type { CommandManagedRuntimeRunner } from "./command-managed-runtime.js"; @@ -23,6 +24,10 @@ const SANDBOX_CALLBACK_BRIDGE_ENTRYPOINT = "paperclip-bridge-server.mjs"; const SANDBOX_EXEC_CHANNEL_ENV = "PAPERCLIP_SANDBOX_EXEC_CHANNEL"; const SANDBOX_EXEC_CHANNEL_BRIDGE = "bridge"; +/** Span name that wraps one Paperclip-API callback request β€” read the request, + * write the response, and remove the request file. */ +const CALLBACK_BRIDGE_RELAY_REQUEST_SPAN = "sandbox.callbackBridge.relayRequest"; + export const DEFAULT_SANDBOX_CALLBACK_BRIDGE_MAX_BODY_BYTES = DEFAULT_BRIDGE_MAX_BODY_BYTES; export interface SandboxCallbackBridgeRouteRule { @@ -630,6 +635,12 @@ export async function startSandboxCallbackBridgeWorker(input: { // otherwise). When it is absent, the request work runs with an empty store, // exactly like the earlier `runWithoutActiveStep` behavior. getRuntimeParentContext?: () => StartupSpanContext | undefined; + // Wrap each Paperclip-API callback request in a + // `sandbox.callbackBridge.relayRequest` span, so the request's read, write, and + // remove execs group under one named span. When it is absent, the request work + // runs under the run parent with no wrapper span, exactly like the earlier + // behavior. + runtimeSpan?: RuntimeSpanRunner; }): Promise { const pollIntervalMs = normalizeTimeoutMs(input.pollIntervalMs, DEFAULT_BRIDGE_POLL_INTERVAL_MS); const maxBodyBytes = normalizeTimeoutMs(input.maxBodyBytes, DEFAULT_BRIDGE_MAX_BODY_BYTES); @@ -777,15 +788,20 @@ export async function startSandboxCallbackBridgeWorker(input: { if (stopping && Date.now() >= stopDeadline) break; inFlight += 1; try { - // A request is run-time work, not startup work. Read the run parent - // context now and run the request under it, so the request - // `sandbox.exec` span parents to the live run span. Read the getter - // per request: the live parent switches to `agent.turn` during the - // turn and back to `task.run` after it. With no getter the store - // stays empty, exactly like the earlier unparented behavior. - await runWithRuntimeParent(input.getRuntimeParentContext?.(), () => - processRequestFile(fileName), - ); + // A request is run-time work, not startup work. Wrap it in a + // `sandbox.callbackBridge.relayRequest` span, so its read, write, and + // remove execs group under one named span that parents to the live + // run span. The span runner reads the run parent per request: the + // live parent switches to `agent.turn` during the turn and back to + // `task.run` after it. Without a runner, the request runs under the + // run parent with no wrapper span, exactly like the earlier behavior. + await (input.runtimeSpan + ? input.runtimeSpan(CALLBACK_BRIDGE_RELAY_REQUEST_SPAN, () => + processRequestFile(fileName), + ) + : runWithRuntimeParent(input.getRuntimeParentContext?.(), () => + processRequestFile(fileName), + )); } finally { inFlight -= 1; } diff --git a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts index be9e51547da..2e3a192caaf 100644 --- a/packages/plugins/sandbox-providers/daytona/src/file-sync.ts +++ b/packages/plugins/sandbox-providers/daytona/src/file-sync.ts @@ -460,8 +460,9 @@ async function syncInFileMappings(input: { // Ensure every target directory exists before the bulk upload writes its temp. const mkdirCommand = [...parentDirs].map((dir) => `mkdir -p ${shellQuote(dir)}`).join(" && "); + // `ensureDirectory` span: `mkdir -p` β€” ensure a directory exists before a write. await withProviderSpan({ - name: "mkdir", + name: "ensureDirectory", run: () => assertSandboxCommandOk(sandbox, mkdirCommand, timeoutSeconds, "syncIn mkdir"), }); guardRoundTrips += 1; @@ -470,8 +471,10 @@ async function syncInFileMappings(input: { // can replace a target parent with a symlink to `/etc` so the string check // passes but the upload + `mv -f` resolve through it. Canonicalize every parent // dir (now materialized) and fail closed if any escapes, BEFORE any bytes land. + // `checkSymlinkEscape` span: re-check a path resolves inside the workspace root + // before use. await withProviderSpan({ - name: "guard", + name: "checkSymlinkEscape", run: () => assertSandboxPathsConfined({ sandbox, @@ -488,6 +491,7 @@ async function syncInFileMappings(input: { // retry never accumulates stale `.paperclip-upload-*` scratch. try { // One batched bulk upload (single /files/bulk-upload) for all file mappings. + // `transfer` span: the real byte upload β€” `sandbox.fs.uploadFiles`. await withProviderSpan({ name: "transfer", wallMsAttr: SPAN_ATTR.transferWallMs, @@ -534,8 +538,10 @@ async function syncInFileMappings(input: { `exec 8>&-;`, ); } + // `promote` span: atomically move the staged temp onto its target via a + // pinned dir handle. await withProviderSpan({ - name: "rename", + name: "promote", run: () => assertSandboxCommandOk( sandbox, @@ -564,6 +570,7 @@ async function syncInDirectoryMapping(input: { const archivePath = path.join(tmp, "sync-in.tar"); // The pack step is host-local: it builds the tarball and makes no sandbox // round trip. The `pack` span records its wall time. + // `pack` span: build a tarball on the host β€” no sandbox round trip. await withProviderSpan({ name: "pack", wallMsAttr: SPAN_ATTR.packWallMs, @@ -585,8 +592,9 @@ async function syncInDirectoryMapping(input: { // components, then confirm it (and any existing parent) canonicalizes inside // the remote dir β€” `tar -C` would otherwise follow a sandbox-planted symlink // and extract our archive outside the workspace root. + // `ensureDirectory` span: `mkdir -p` β€” ensure a directory exists before a write. await withProviderSpan({ - name: "mkdir", + name: "ensureDirectory", run: () => assertSandboxCommandOk( sandbox, @@ -596,8 +604,10 @@ async function syncInDirectoryMapping(input: { ), }); guardRoundTrips += 1; + // `checkSymlinkEscape` span: re-check a path resolves inside the workspace + // root before use. await withProviderSpan({ - name: "guard", + name: "checkSymlinkEscape", run: () => assertSandboxPathsConfined({ sandbox, @@ -608,6 +618,7 @@ async function syncInDirectoryMapping(input: { }), }); guardRoundTrips += 1; + // `transfer` span: the real byte upload β€” `sandbox.fs.uploadFiles`. await withProviderSpan({ name: "transfer", wallMsAttr: SPAN_ATTR.transferWallMs, @@ -641,8 +652,10 @@ async function syncInDirectoryMapping(input: { `exec 9>&-;`, `rm -f ${shellQuote(remoteTar)};`, ].join("\n"); + // `extractTarball` span: one round trip β€” re-check the path, `tar -xf`, and + // remove the scratch tarball. await withProviderSpan({ - name: "extract", + name: "extractTarball", run: () => assertSandboxCommandOk( sandbox, @@ -687,8 +700,10 @@ async function runPostUploadCommands(input: { let cwd = remoteDir; if (command.cwd != null) { assertConfinedSandboxPath(remoteDir, command.cwd, "post-upload command cwd"); + // `checkSymlinkEscape` span: re-check a path resolves inside the workspace + // root before use. await withProviderSpan({ - name: "guard", + name: "checkSymlinkEscape", run: () => assertSandboxPathsConfined({ sandbox, @@ -704,8 +719,9 @@ async function runPostUploadCommands(input: { // C4: first non-zero exit or timeout throws and aborts the remaining commands. const commandTimeoutSeconds = command.timeoutMs != null ? toTimeoutSeconds(command.timeoutMs) : timeoutSeconds; + // `postUploadCommand` span: run one caller-supplied post-upload command. const result = await withProviderSpan({ - name: "provision", + name: "postUploadCommand", run: () => sandbox.process.executeCommand(command.command, cwd, undefined, commandTimeoutSeconds), }); diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index 7e3d04d9c9d..e37d2b348f1 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -1202,7 +1202,7 @@ describe("Daytona sandbox provider plugin", () => { // The provision command runs before the run opens its trace root, so the // host marks it `bypassSession`. The provider must not open the session for - // it, or the `session.setup` span loses its run parent. + // it, or the `session.open` span loses its run parent. await plugin.definition.onEnvironmentExecute?.( sessionExecParams({ bypassSession: true }), ); @@ -1406,7 +1406,7 @@ describe("Daytona sandbox provider plugin", () => { expect(sandbox.process.deleteSession).toHaveBeenCalledWith(sessionId); }); - it("emits a session.setup span on create and a session.teardown span on delete", async () => { + it("emits a session.open span on create and a session.close span on delete", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); mockGet.mockResolvedValue(sandbox); @@ -1414,7 +1414,7 @@ describe("Daytona sandbox provider plugin", () => { const restore = __setDaytonaPluginContextForTest({ tracer } as unknown as PluginContext); try { await plugin.definition.onEnvironmentExecute?.(sessionExecParams()); - const setup = spans.find((span) => span.name === "session.setup"); + const setup = spans.find((span) => span.name === "session.open"); expect(setup).toBeDefined(); expect(setup!.ended).toBe(true); expect(setup!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); @@ -1426,7 +1426,7 @@ describe("Daytona sandbox provider plugin", () => { providerLeaseId: "sandbox-123", config: { timeoutMs: 300000, reuseLease: false, useSessions: true }, }); - const teardown = spans.find((span) => span.name === "session.teardown"); + const teardown = spans.find((span) => span.name === "session.close"); expect(teardown).toBeDefined(); expect(teardown!.ended).toBe(true); expect(teardown!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); @@ -1435,7 +1435,7 @@ describe("Daytona sandbox provider plugin", () => { } }); - it("marks the session.setup span failed when the session create throws", async () => { + it("marks the session.open span failed when the session create throws", async () => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); sandbox.process.createSession.mockRejectedValueOnce(new Error("create boom")); @@ -1446,7 +1446,7 @@ describe("Daytona sandbox provider plugin", () => { await expect( plugin.definition.onEnvironmentExecute?.(sessionExecParams()), ).rejects.toThrow(/create boom/); - const setup = spans.find((span) => span.name === "session.setup"); + const setup = spans.find((span) => span.name === "session.open"); expect(setup).toBeDefined(); expect(setup!.ended).toBe(true); expect(setup!.status?.code).toBe(2); @@ -3028,7 +3028,7 @@ describe("daytona native file-sync hooks", () => { expect(transfer!.attributes["paperclip.sandbox.startup.transfer.guard.count"]).toBe(2); }); - it("opens mkdir, guard, transfer, rename spans in call order for a file-mapping sync", async () => { + it("opens ensureDirectory, checkSymlinkEscape, transfer, promote spans in call order for a file-mapping sync", async () => { const hostDir = await makeHostDir(); const source = path.join(hostDir, "config.txt"); await fs.writeFile(source, "plain"); @@ -3055,21 +3055,26 @@ describe("daytona native file-sync hooks", () => { restore(); } - expect(spans.map((span) => span.name)).toEqual(["mkdir", "guard", "transfer", "rename"]); + expect(spans.map((span) => span.name)).toEqual([ + "ensureDirectory", + "checkSymlinkEscape", + "transfer", + "promote", + ]); for (const span of spans) { expect(span.ended).toBe(true); expect(span.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); // A per-round-trip span carries no `*.wall_ms` attribute; the native span // width carries its time. Only `pack` and `transfer` keep a wall_ms value. if (span.name !== "transfer") { - expect(span.attributes["paperclip.sandbox.startup.mkdir.wall_ms"]).toBeUndefined(); - expect(span.attributes["paperclip.sandbox.startup.rename.wall_ms"]).toBeUndefined(); - expect(span.attributes["paperclip.sandbox.startup.guard.wall_ms"]).toBeUndefined(); + expect(span.attributes["paperclip.sandbox.startup.ensureDirectory.wall_ms"]).toBeUndefined(); + expect(span.attributes["paperclip.sandbox.startup.promote.wall_ms"]).toBeUndefined(); + expect(span.attributes["paperclip.sandbox.startup.checkSymlinkEscape.wall_ms"]).toBeUndefined(); } } }); - it("opens pack, mkdir, guard, transfer, extract spans in call order for a directory-mapping sync", async () => { + it("opens pack, ensureDirectory, checkSymlinkEscape, transfer, extractTarball spans in call order for a directory-mapping sync", async () => { const hostDir = await makeHostDir(); const sourceDir = path.join(hostDir, "assets"); await fs.mkdir(sourceDir, { recursive: true }); @@ -3099,7 +3104,13 @@ describe("daytona native file-sync hooks", () => { restore(); } - expect(spans.map((span) => span.name)).toEqual(["pack", "mkdir", "guard", "transfer", "extract"]); + expect(spans.map((span) => span.name)).toEqual([ + "pack", + "ensureDirectory", + "checkSymlinkEscape", + "transfer", + "extractTarball", + ]); for (const span of spans) { expect(span.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); } @@ -3141,7 +3152,7 @@ describe("daytona native file-sync hooks", () => { expect(pack!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); }); - it("opens a guard span and a provision span in call order for a post-upload command with a working directory", async () => { + it("opens a checkSymlinkEscape span and a postUploadCommand span in call order for a post-upload command with a working directory", async () => { const hostDir = await makeHostDir(); const source = path.join(hostDir, "config.txt"); await fs.writeFile(source, "plain"); @@ -3169,17 +3180,18 @@ describe("daytona native file-sync hooks", () => { restore(); } - // The full order: the file mapping opens mkdir, guard, transfer, rename; the - // post-upload command then opens its own cwd guard and the provision span. + // The full order: the file mapping opens ensureDirectory, checkSymlinkEscape, + // transfer, promote; the post-upload command then opens its own cwd + // checkSymlinkEscape and the postUploadCommand span. expect(spans.map((span) => span.name)).toEqual([ - "mkdir", - "guard", + "ensureDirectory", + "checkSymlinkEscape", "transfer", - "rename", - "guard", - "provision", + "promote", + "checkSymlinkEscape", + "postUploadCommand", ]); - const provision = spans.find((span) => span.name === "provision"); + const provision = spans.find((span) => span.name === "postUploadCommand"); expect(provision!.ended).toBe(true); expect(provision!.attributes["paperclip.sandbox.startup.provider"]).toBe("daytona"); }); diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index e00c0a2ca06..18306045b3c 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -1336,11 +1336,13 @@ async function getOrCreateSession(sandbox: Sandbox, scope: SandboxScope): Promis // no second command can slip in between the store read and the create start. return sandboxHandleSessionStore.runSingle(scope, async () => { const sessionId = `paperclip-${randomUUID()}`; - // Wrap the session create in a short `session.setup` provider span. The span + // Wrap the session create in a short `session.open` provider span. The span // carries no session id and no command text, only the provider family. The - // host maps the name to `sandbox.provider.session.setup`. + // host maps the name to `sandbox.daytona.session.open`. + // `session.open` span: create the one persistent Daytona session for a lease, + // on the first in-run command β€” `sandbox.process.createSession`. await withProviderSpan({ - name: "session.setup", + name: "session.open", run: () => sandbox.process.createSession(sessionId), }); sandboxHandleSessionStore.set(scope, sessionId); @@ -1360,10 +1362,12 @@ async function teardownSession(sandbox: Sandbox, scope: SandboxScope): Promise sandbox.process.deleteSession(sessionId), }); } catch (error) { @@ -2255,9 +2259,9 @@ const plugin = definePlugin({ // on, and it does NOT open the session. The host sets this flag on a // pre-run command (the workspace provision command) that runs before the // run opens its trace root. Opening the session there would emit a - // `session.setup` span with no run parent, and the span backend would drop + // `session.open` span with no run parent, and the span backend would drop // it. With the bypass the session opens on the first in-run command, whose - // setup span parents to the run trace. + // open span parents to the run trace. let result: PluginEnvironmentExecuteResult; if (config.useSessions && !params.bypassSession) { const sessionId = await getOrCreateSession(sandbox, scope); diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index b86c7ea09db..79ae3e25818 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -93,10 +93,17 @@ absent, never a misleading `0`. | `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` | | `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` | | `acp.handshake` | ACP session handshake step. | `sandbox.startup` | -| `sandbox.exec` | One host-to-sandbox execution. | the active step span | +| `sandbox.agentSession.sendInput` | One outbound ACP message to the agent β€” the socket handler's one `writeTextFile` exec. | the active run span | +| `sandbox.agentSession.pollOutput` | One 100 ms poll tick β€” `list`, then `read`+`remove` per file found (`1 + 2n` execs). | the active run span | +| `sandbox.callbackBridge.relayRequest` | One Paperclip-API callback request β€” read the request, write the response, remove it. | the active run span | +| `sandbox.exec` | One host-to-sandbox execution. | the active step or wrapper span | A step span name is the step name. The `sandbox.exec` span parents to the step -span that runs the execution, so each execution nests under its step. With no +span that runs the execution, so each execution nests under its step. A run-time +`sandbox.exec` span parents instead to the run-time wrapper span that runs it +(`sandbox.agentSession.sendInput`, `sandbox.agentSession.pollOutput`, or +`sandbox.callbackBridge.relayRequest`). Each run-time wrapper span parents to the +live run span (`agent.turn` during the turn, `task.run` otherwise). With no active trace context the exec span opens unparented. The root span sets the error status when the bring-up fails. Each step span sets @@ -172,7 +179,7 @@ the code first. Keep the attribute low-cardinality and free of user content. ### Provider spans A sandbox provider plugin also opens spans for its own sync steps. These spans -use the `sandbox.provider.` name prefix. They share the +use the `sandbox.daytona.` name prefix. They share the `paperclip.sandbox.startup.` attribute prefix and obey the same opt-in and no-user-content rules as the startup spans above. @@ -183,15 +190,26 @@ before it records the span. | Span | Scope | Parent | | --- | --- | --- | -| `sandbox.provider.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span | -| `sandbox.provider.transfer` | The transfer step that uploads the files to the sandbox. | the active startup step span | -| `sandbox.provider.other` | Any span name outside the known set. | the active startup step span | +| `sandbox.daytona.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span | +| `sandbox.daytona.transfer` | The transfer step that uploads the files to the sandbox. | the active startup step span | +| `sandbox.daytona.ensureDirectory` | The `mkdir -p` step that ensures a directory exists before a write. | the active startup step span | +| `sandbox.daytona.checkSymlinkEscape` | The re-check step that a path resolves inside the workspace root before use. | the active startup step span | +| `sandbox.daytona.promote` | The atomic move of a staged temp onto its target via a pinned dir handle. | the active startup step span | +| `sandbox.daytona.extractTarball` | The one round trip that re-checks the path, runs `tar -xf`, and removes the scratch tarball. | the active startup step span | +| `sandbox.daytona.postUploadCommand` | One caller-supplied post-upload command. | the active startup step span | +| `sandbox.daytona.session.open` | The create of the one persistent session for a lease, on the first in-run command. | the active run span | +| `sandbox.daytona.session.close` | The delete of that persistent session on lease release. | the active run span | +| `sandbox.daytona.other` | Any span name outside the known set. | the active startup step span | -The host clamps the span name to the closed set `pack` and `transfer`. The host -maps a known name to `sandbox.provider.`. The host maps any other value to -`sandbox.provider.other`, so a span name never carries free-form data. +The host clamps the span name to the closed set of leaf names above (`pack`, +`transfer`, `ensureDirectory`, `checkSymlinkEscape`, `promote`, `extractTarball`, +`postUploadCommand`, `session.open`, and `session.close`). The host maps a known +name to `sandbox.daytona.`. The host maps any other value to +`sandbox.daytona.other`, so a span name never carries free-form data. Only the +daytona provider emits these spans today, so the segment is the literal +`daytona`. -The `sandbox.provider.*` spans use this closed attribute allowlist. The host +The `sandbox.daytona.*` spans use this closed attribute allowlist. The host drops every other key, so a command, an argument, a path, an id, a standard output, or a standard error never rides a provider span. The host records only the attributes that the producer sends for one span. @@ -200,9 +218,9 @@ the attributes that the producer sends for one span. | --- | --- | --- | --- | | `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | | `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). | -| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.provider.pack` span. | -| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.provider.transfer` span. | -| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.provider.transfer` span. | +| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. | +| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. | +| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.daytona.transfer` span. | The `span.record` host handler enforces the allowlist. It re-maps `provider` through the provider-family normalizer. It keeps `outcome` only when the value diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 5c13b76f522..7397b997323 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -709,7 +709,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { // Build the fake plugin fixture for the run-parent release tests below. A // plugin sandbox provider can open a persistent session on the first command // and delete it on lease release; the delete emits a provider - // `session.teardown` span. The host mints that span's parent from the active + // `session.close` span. The host mints that span's parent from the active // step context at the release RPC. So the release must run under the run // parent, or the span loses its traceparent and the backend drops it. async function seedFakePluginSandbox() { diff --git a/server/src/__tests__/instrumentation.test.ts b/server/src/__tests__/instrumentation.test.ts index b0b94e07a3b..e2a3937ceb8 100644 --- a/server/src/__tests__/instrumentation.test.ts +++ b/server/src/__tests__/instrumentation.test.ts @@ -178,7 +178,7 @@ describe.skipIf(!otelSdk)("recordProviderPluginSpan native duration", () => { const startTimeMs = Date.now() - 4500; const endTimeMs = startTimeMs + 4500; recordProviderPluginSpan({ - name: "sandbox.provider.mkdir", + name: "sandbox.daytona.ensureDirectory", parent: { traceId: "0af7651916cd43dd8448eb211c80319c", spanId: "b7ad6b7169203331", @@ -191,7 +191,7 @@ describe.skipIf(!otelSdk)("recordProviderPluginSpan native duration", () => { const finished = exporter.getFinishedSpans(); expect(finished).toHaveLength(1); const span = finished[0]!; - expect(span.name).toBe("sandbox.provider.mkdir"); + expect(span.name).toBe("sandbox.daytona.ensureDirectory"); expect(Math.round(hrTimeToMs(span.startTime as [number, number]))).toBe(startTimeMs); expect(Math.round(hrTimeToMs(span.endTime as [number, number]))).toBe(endTimeMs); // The native width equals the true wall-clock difference, not near zero. @@ -211,7 +211,7 @@ describe.skipIf(!otelSdk)("recordProviderPluginSpan native duration", () => { try { const { recordProviderPluginSpan } = await import("../instrumentation.js"); recordProviderPluginSpan({ - name: "sandbox.provider.pack", + name: "sandbox.daytona.pack", parent: { traceId: "0af7651916cd43dd8448eb211c80319c", spanId: "b7ad6b7169203331", diff --git a/server/src/__tests__/plugin-host-services-span.test.ts b/server/src/__tests__/plugin-host-services-span.test.ts index 110d06a0b63..bb2a72bb568 100644 --- a/server/src/__tests__/plugin-host-services-span.test.ts +++ b/server/src/__tests__/plugin-host-services-span.test.ts @@ -63,7 +63,7 @@ describe("plugin provider span host handler", () => { parent: { traceId: string; spanId: string; traceFlags: number }; attributes: Record; }; - expect(call.name).toBe("sandbox.provider.pack"); + expect(call.name).toBe("sandbox.daytona.pack"); expect(call.parent).toEqual({ traceId: "0af7651916cd43dd8448eb211c80319c", spanId: "b7ad6b7169203331", @@ -114,20 +114,26 @@ describe("plugin provider span host handler", () => { expect(call.status).not.toHaveProperty("message"); }); - it("clamps an unknown span name to sandbox.provider.other", async () => { + it("clamps an unknown span name to sandbox.daytona.other", async () => { const services = servicesFor(); await services.tracer.record( { name: "rm -rf / --no-preserve-root" }, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, ); expect((mockRecordSpan.mock.calls[0]![0] as { name: string }).name).toBe( - "sandbox.provider.other", + "sandbox.daytona.other", ); }); - it("admits each per-round-trip span name to sandbox.provider.", async () => { + it("admits each per-round-trip span name to sandbox.daytona.", async () => { const services = servicesFor(); - for (const name of ["mkdir", "guard", "rename", "extract", "provision"]) { + for (const name of [ + "ensureDirectory", + "checkSymlinkEscape", + "promote", + "extractTarball", + "postUploadCommand", + ]) { await services.tracer.record( { name }, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, @@ -135,17 +141,17 @@ describe("plugin provider span host handler", () => { } const recorded = mockRecordSpan.mock.calls.map((c) => (c[0] as { name: string }).name); expect(recorded).toEqual([ - "sandbox.provider.mkdir", - "sandbox.provider.guard", - "sandbox.provider.rename", - "sandbox.provider.extract", - "sandbox.provider.provision", + "sandbox.daytona.ensureDirectory", + "sandbox.daytona.checkSymlinkEscape", + "sandbox.daytona.promote", + "sandbox.daytona.extractTarball", + "sandbox.daytona.postUploadCommand", ]); }); - it("admits the session setup and teardown span names", async () => { + it("admits the session open and close span names", async () => { const services = servicesFor(); - for (const name of ["session.setup", "session.teardown"]) { + for (const name of ["session.open", "session.close"]) { await services.tracer.record( { name }, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, @@ -153,8 +159,8 @@ describe("plugin provider span host handler", () => { } const recorded = mockRecordSpan.mock.calls.map((c) => (c[0] as { name: string }).name); expect(recorded).toEqual([ - "sandbox.provider.session.setup", - "sandbox.provider.session.teardown", + "sandbox.daytona.session.open", + "sandbox.daytona.session.close", ]); }); @@ -163,7 +169,7 @@ describe("plugin provider span host handler", () => { const startTimeMs = Date.now() - 4500; const endTimeMs = startTimeMs + 4500; await services.tracer.record( - { name: "mkdir", startTimeMs, endTimeMs }, + { name: "ensureDirectory", startTimeMs, endTimeMs }, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, ); const call = mockRecordSpan.mock.calls[0]![0] as { @@ -181,7 +187,7 @@ describe("plugin provider span host handler", () => { const startTimeMs = Date.now() + 5000; const endTimeMs = startTimeMs + 1000; await services.tracer.record( - { name: "mkdir", startTimeMs, endTimeMs }, + { name: "ensureDirectory", startTimeMs, endTimeMs }, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, ); const call = mockRecordSpan.mock.calls[0]![0] as { @@ -207,7 +213,7 @@ describe("plugin provider span host handler", () => { for (const pair of invalidPairs) { mockRecordSpan.mockReset(); await services.tracer.record( - { name: "mkdir", startTimeMs: pair.startTimeMs, endTimeMs: pair.endTimeMs } as never, + { name: "ensureDirectory", startTimeMs: pair.startTimeMs, endTimeMs: pair.endTimeMs } as never, { traceparent: VALID_TRACEPARENT } as WorkerHostCallContext, ); // The span still records (the synchronous path), but without a timestamp. diff --git a/server/src/services/plugin-host-services.ts b/server/src/services/plugin-host-services.ts index 1eb3b8a01a7..1428f42fd4e 100644 --- a/server/src/services/plugin-host-services.ts +++ b/server/src/services/plugin-host-services.ts @@ -507,29 +507,33 @@ const SESSION_EVENT_SUBSCRIPTION_TIMEOUT_MS = 30 * 60 * 1_000; // 30 minutes const SPAN_ATTRS = SANDBOX_STARTUP_SPAN_ATTRS; -/** The closed set of provider span names a plugin may emit. `pack` and - * `transfer` are the host-local build and the byte upload. `mkdir`, `guard`, - * `rename`, `extract`, and `provision` are the per-round-trip command spans in - * the inbound sync path. `session.setup` and `session.teardown` are the short - * spans that wrap a persistent-session create and delete. */ +/** The closed set of provider span leaf names a plugin may emit. `pack` and + * `transfer` are the host-local build and the byte upload. `ensureDirectory`, + * `checkSymlinkEscape`, `promote`, `extractTarball`, and `postUploadCommand` + * are the per-round-trip command spans in the inbound sync path. `session.open` + * and `session.close` are the short spans that wrap a persistent-session create + * and delete. */ const KNOWN_PROVIDER_SPAN_NAMES: ReadonlySet = new Set([ "pack", "transfer", - "mkdir", - "guard", - "rename", - "extract", - "provision", - "session.setup", - "session.teardown", + "ensureDirectory", + "checkSymlinkEscape", + "promote", + "extractTarball", + "postUploadCommand", + "session.open", + "session.close", ]); /** Clamp the span name to a closed, namespaced set. A known name maps to - * `sandbox.provider.`; any other value maps to `sandbox.provider.other`, - * so a span name never carries free-form data. */ + * `sandbox.daytona.`; any other value maps to `sandbox.daytona.other`, so + * a span name never carries free-form data. Only the daytona provider emits + * these spans today, so the segment is the literal `daytona`. When a second + * provider emits provider spans, derive the segment from the normalized + * `provider` family attribute on the span instead of this literal. */ function clampProviderSpanName(raw: unknown): string { const name = typeof raw === "string" && KNOWN_PROVIDER_SPAN_NAMES.has(raw) ? raw : "other"; - return `sandbox.provider.${name}`; + return `sandbox.daytona.${name}`; } /** The closed allowlist of attribute keys a provider span may carry. The host From 35132af1610c7a0435830e2d7cd834536514ffd4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 00:41:19 -0500 Subject: [PATCH 011/311] fix(config): preserve extensions and guard invalid repairs (#11005) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The CLI and server share a JSON configuration contract for local installations and worktrees. > - Existing config writes removed extension keys because Zod stripped unknown object properties. > - Invalid config files could also be replaced with defaults before an operator preserved the original bytes. > - Configuration updates must preserve operator edits and must not rewrite files when the effective value is unchanged. > - This pull request adds extension-preserving merges, guarded invalid-config repair, atomic writes, and focused regression tests. > - The benefit is safe setup and configuration reruns without data loss or unnecessary mtime changes. ## Linked Issues or Issue Description **What happened?** Known-field updates through the CLI or server removed unknown top-level and nested config keys. Non-interactive configure and onboard paths could replace a present but invalid config with defaults. **Expected behavior** Writers preserve extension keys, skip semantic no-op writes, and require explicit interactive confirmation before an invalid config is replaced. Repair preserves an exact collision-safe backup first. **Steps to reproduce** 1. Add an unknown top-level key and an unknown nested provider key to `config.json`. 2. Update a known field through the CLI or worktree config writer. 3. Observe that the extension keys are removed on the base branch. 4. Write invalid JSON and run configure or onboard without an interactive terminal. 5. Observe that the original file can be replaced without a durable invalid-file backup on the base branch. **Paperclip version or commit** `master` at the pull request base commit. ## What Changed - Accept unknown properties at each extensible config object boundary while keeping every known field validated. - Merge known-field updates into the parsed source config and preserve only unknown extension data. - Warn about near-match key names without deleting or changing them. - Skip writes when the effective config is unchanged, which keeps file mtimes stable. - Write config changes through a temporary file, file sync, rename, and directory sync. - Distinguish a missing config from an invalid config in configure and onboard. - Back up invalid bytes as `config.json.invalid-N` and verify the source still matches that backup before repair. - Require interactive repair confirmation and reject non-interactive replacement with an actionable message. - Document the config preservation and repair behavior. ## Verification - `pnpm exec vitest run packages/shared/src/config-schema.test.ts cli/src/__tests__/config-store.test.ts cli/src/__tests__/configure-repair.test.ts cli/src/__tests__/configure.test.ts cli/src/__tests__/onboard.test.ts server/src/__tests__/config-file.test.ts server/src/__tests__/worktree-config.test.ts` - `pnpm -r typecheck` - `AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= VITEST_MAX_WORKERS=1 pnpm test:run` - `pnpm build` - Confirm all pull request checks are green on the latest commit. - Confirm Greptile reports 5/5 with no unresolved comments. ## Risks - Passthrough keeps misspelled keys. Near-match warnings make this visible without destructive cleanup. - Merge behavior must distinguish unknown extension keys from optional known keys. Schema-aware regression tests cover preservation and known-key deletion. - Repair must not overwrite bytes that changed after backup. The writer compares the current source with the selected backup before atomic replacement. - The change does not alter database schema, company scoping, or activity logging. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 model family. The exact deployment model ID and context window are not exposed. Agentic reasoning, tool use, and code execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- cli/src/__tests__/config-store.test.ts | 139 +++++++++++++++ cli/src/__tests__/configure-repair.test.ts | 83 +++++++++ cli/src/__tests__/configure.test.ts | 25 +++ cli/src/__tests__/onboard.test.ts | 16 ++ cli/src/commands/configure.ts | 56 +++++- cli/src/commands/onboard.ts | 53 +++++- cli/src/config/schema.ts | 5 + cli/src/config/store.ts | 127 ++++++++++++- doc/DEVELOPING.md | 8 + packages/shared/src/config-schema.test.ts | 112 +++++++++++- packages/shared/src/config-schema.ts | 176 +++++++++++++++++-- packages/shared/src/index.ts | 5 + server/src/__tests__/config-file.test.ts | 24 ++- server/src/__tests__/worktree-config.test.ts | 78 ++++++++ server/src/config-file.ts | 14 +- server/src/worktree-config.ts | 62 ++++++- 16 files changed, 940 insertions(+), 43 deletions(-) create mode 100644 cli/src/__tests__/config-store.test.ts create mode 100644 cli/src/__tests__/configure-repair.test.ts diff --git a/cli/src/__tests__/config-store.test.ts b/cli/src/__tests__/config-store.test.ts new file mode 100644 index 00000000000..21abb4bf60e --- /dev/null +++ b/cli/src/__tests__/config-store.test.ts @@ -0,0 +1,139 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + backupInvalidConfig, + readConfig, + writeConfig, +} from "../config/store.js"; +import { paperclipConfigSchema, type PaperclipConfig } from "../config/schema.js"; + +const roots: string[] = []; + +afterEach(() => { + vi.restoreAllMocks(); + for (const root of roots.splice(0)) { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +function createConfigPath(): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-config-store-")); + roots.push(root); + return path.join(root, "config.json"); +} + +function defaultConfig(): PaperclipConfig { + return paperclipConfigSchema.parse({ + $meta: { + version: 1, + updatedAt: "2026-08-06T00:00:00.000Z", + source: "configure", + }, + database: { mode: "embedded-postgres" }, + logging: { mode: "file" }, + server: {}, + }); +} + +describe("config store", () => { + it("preserves top-level and nested extension keys during a known-field update", () => { + const configPath = createConfigPath(); + fs.writeFileSync(configPath, JSON.stringify({ + ...defaultConfig(), + topLevelExtension: { enabled: true }, + server: { + ...defaultConfig().server, + serverExtension: "keep", + }, + storage: { + ...defaultConfig().storage, + localDisk: { + ...defaultConfig().storage.localDisk, + driverExtension: "keep", + }, + }, + }, null, 2)); + + const source = readConfig(configPath)!; + const { topLevelExtension: _topLevelExtension, ...knownConfig } = source; + const { serverExtension: _serverExtension, ...knownServer } = source.server; + const { driverExtension: _driverExtension, ...knownLocalDisk } = source.storage.localDisk; + const update: PaperclipConfig = { + ...knownConfig, + server: { + ...knownServer, + port: 3200, + }, + storage: { + ...source.storage, + localDisk: knownLocalDisk, + }, + }; + + expect(writeConfig(update, configPath)).toBe(true); + expect(JSON.parse(fs.readFileSync(configPath, "utf8"))).toMatchObject({ + topLevelExtension: { enabled: true }, + server: { + port: 3200, + serverExtension: "keep", + }, + storage: { + localDisk: { + driverExtension: "keep", + }, + }, + }); + }); + + it("skips semantic no-op writes and keeps the config mtime stable", () => { + const configPath = createConfigPath(); + const source = defaultConfig(); + fs.writeFileSync(configPath, `${JSON.stringify(source, null, 2)}\n`); + const stableTime = new Date("2020-01-01T00:00:00.000Z"); + fs.utimesSync(configPath, stableTime, stableTime); + + const update = { + ...source, + $meta: { + ...source.$meta, + source: "doctor" as const, + updatedAt: "2026-08-06T01:00:00.000Z", + }, + }; + + expect(writeConfig(update, configPath)).toBe(false); + expect(fs.statSync(configPath).mtimeMs).toBe(stableTime.getTime()); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + + it("backs up invalid bytes collision-safely and only replaces them through an atomic repair", () => { + const configPath = createConfigPath(); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + fs.writeFileSync(`${configPath}.invalid-1`, "existing backup"); + + const open = vi.spyOn(fs, "openSync"); + const sync = vi.spyOn(fs, "fsyncSync"); + const backupPath = backupInvalidConfig(configPath); + expect(backupPath).toBe(`${configPath}.invalid-2`); + expect(fs.readFileSync(backupPath)).toEqual(invalidBytes); + expect(open).toHaveBeenCalledWith(backupPath, "r"); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(sync).toHaveBeenCalled(); + expect(() => writeConfig(defaultConfig(), configPath)).toThrow(/Refusing to overwrite invalid config/); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + + open.mockClear(); + sync.mockClear(); + const rename = vi.spyOn(fs, "renameSync"); + expect(writeConfig(defaultConfig(), configPath, { invalidBackupPath: backupPath })).toBe(true); + expect(rename).toHaveBeenCalledWith(expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), configPath); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(open.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(rename.mock.invocationCallOrder.at(-1)!); + expect(sync.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(open.mock.invocationCallOrder.at(-1)!); + expect(readConfig(configPath)).not.toBeNull(); + expect(fs.readdirSync(path.dirname(configPath)).some((entry) => entry.includes(".tmp-"))).toBe(false); + }); +}); diff --git a/cli/src/__tests__/configure-repair.test.ts b/cli/src/__tests__/configure-repair.test.ts new file mode 100644 index 00000000000..8ffe3834695 --- /dev/null +++ b/cli/src/__tests__/configure-repair.test.ts @@ -0,0 +1,83 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import * as prompts from "@clack/prompts"; +import { configure } from "../commands/configure.js"; +import { readConfig } from "../config/store.js"; + +vi.mock("@clack/prompts", () => ({ + intro: vi.fn(), + outro: vi.fn(), + cancel: vi.fn(), + confirm: vi.fn(), + select: vi.fn(), + isCancel: vi.fn(() => false), + log: { + error: vi.fn(), + message: vi.fn(), + step: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + }, +})); + +vi.mock("../prompts/server.js", () => ({ + promptServer: vi.fn(async ({ currentServer, currentAuth }) => ({ + server: currentServer, + auth: currentAuth, + })), +})); + +const ORIGINAL_EXIT_CODE = process.exitCode; +let originalStdinIsTTY: boolean | undefined; +let originalStdoutIsTTY: boolean | undefined; + +beforeEach(() => { + originalStdinIsTTY = process.stdin.isTTY; + originalStdoutIsTTY = process.stdout.isTTY; + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: true }); + Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: true }); + vi.mocked(prompts.confirm).mockResolvedValue(true); + vi.spyOn(console, "log").mockImplementation(() => undefined); +}); + +afterEach(() => { + Object.defineProperty(process.stdin, "isTTY", { + configurable: true, + value: originalStdinIsTTY, + }); + Object.defineProperty(process.stdout, "isTTY", { + configurable: true, + value: originalStdoutIsTTY, + }); + process.exitCode = ORIGINAL_EXIT_CODE; + vi.restoreAllMocks(); +}); + +describe("configure invalid-config repair", () => { + it("repairs only after confirmation and commits the staged config atomically", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-repair-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + const rename = vi.spyOn(fs, "renameSync"); + + try { + await configure({ config: configPath, section: "server" }); + + expect(prompts.confirm).toHaveBeenCalledWith({ + message: `Repair from defaults? The invalid original is backed up at ${configPath}.invalid-1.`, + initialValue: false, + }); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(readConfig(configPath)).not.toBeNull(); + expect(rename).toHaveBeenCalledWith( + expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), + configPath, + ); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/cli/src/__tests__/configure.test.ts b/cli/src/__tests__/configure.test.ts index 74a37fc8fcc..cbd7016879d 100644 --- a/cli/src/__tests__/configure.test.ts +++ b/cli/src/__tests__/configure.test.ts @@ -96,4 +96,29 @@ describe("configure command", () => { fs.rmSync(root, { recursive: true, force: true }); } }); + + it("backs up invalid config bytes and refuses non-interactive replacement", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-invalid-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + const stdinDescriptor = Object.getOwnPropertyDescriptor(process.stdin, "isTTY"); + fs.writeFileSync(configPath, invalidBytes); + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: false }); + + try { + await configure({ config: configPath, section: "server" }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + } finally { + if (stdinDescriptor) { + Object.defineProperty(process.stdin, "isTTY", stdinDescriptor); + } else { + delete (process.stdin as { isTTY?: boolean }).isTTY; + } + fs.rmSync(root, { recursive: true, force: true }); + } + }); }); diff --git a/cli/src/__tests__/onboard.test.ts b/cli/src/__tests__/onboard.test.ts index 33890e703de..59a578b1f86 100644 --- a/cli/src/__tests__/onboard.test.ts +++ b/cli/src/__tests__/onboard.test.ts @@ -8,6 +8,7 @@ import type { PaperclipConfig } from "../config/schema.js"; const ORIGINAL_ENV = { ...process.env }; const ORIGINAL_CWD = process.cwd(); const ORIGINAL_PATH = process.env.PATH; +const ORIGINAL_EXIT_CODE = process.exitCode; function createExistingConfigFixture() { const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-onboard-")); @@ -107,6 +108,7 @@ describe("onboard", () => { afterEach(() => { process.env = { ...ORIGINAL_ENV }; process.chdir(ORIGINAL_CWD); + process.exitCode = ORIGINAL_EXIT_CODE; }); it("preserves an existing config when rerun without flags", async () => { @@ -129,6 +131,20 @@ describe("onboard", () => { expect(fs.existsSync(path.join(path.dirname(fixture.configPath), ".env"))).toBe(true); }); + it("backs up invalid config bytes and refuses --yes replacement", async () => { + const configPath = createFreshConfigPath(); + const invalidBytes = Buffer.from('{"database": invalid}\n', "utf8"); + fs.mkdirSync(path.dirname(configPath), { recursive: true }); + fs.writeFileSync(configPath, invalidBytes); + + await onboard({ config: configPath, yes: true, invokedByRun: true }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + it("keeps --yes onboarding on local trusted loopback defaults", async () => { const configPath = createFreshConfigPath(); process.env.HOST = "0.0.0.0"; diff --git a/cli/src/commands/configure.ts b/cli/src/commands/configure.ts index 7c9b391bf0f..126fcc414bf 100644 --- a/cli/src/commands/configure.ts +++ b/cli/src/commands/configure.ts @@ -1,7 +1,16 @@ import * as p from "@clack/prompts"; import pc from "picocolors"; -import { readConfig, writeConfig, configExists, resolveConfigPath } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + readConfig, + writeConfig, + configExists, + resolveConfigPath, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; import { promptLlm } from "../prompts/llm.js"; @@ -88,15 +97,39 @@ export async function configure(opts: { } let config: PaperclipConfig; + let invalidBackupPath: string | undefined; try { config = readConfig(opts.config) ?? defaultConfig(); + for (const warning of findPaperclipConfigKeyWarnings(config)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config is invalid. Loading defaults so you can repair it now.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + if (!process.stdin.isTTY || !process.stdout.isTTY) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + config = defaultConfig(); + invalidBackupPath = backupPath; } let section: Section | undefined = opts.section as Section | undefined; @@ -179,8 +212,15 @@ export async function configure(opts: { config.$meta.updatedAt = new Date().toISOString(); config.$meta.source = "configure"; - writeConfig(config, opts.config); - p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + const written = writeConfig(config, opts.config, { + invalidBackupPath, + }); + invalidBackupPath = undefined; + if (written) { + p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + } else { + p.log.message(pc.dim(`${SECTION_LABELS[section]} configuration unchanged.`)); + } // If section was provided via CLI flag, don't loop if (opts.section) { diff --git a/cli/src/commands/onboard.ts b/cli/src/commands/onboard.ts index 3024360dda5..31850714d73 100644 --- a/cli/src/commands/onboard.ts +++ b/cli/src/commands/onboard.ts @@ -17,8 +17,17 @@ import { type SecretProvider, type StorageProvider, } from "@paperclipai/shared"; -import { configExists, readConfig, resolveConfigPath, writeConfig } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + configExists, + readConfig, + resolveConfigPath, + writeConfig, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureAgentJwtSecret, resolveAgentJwtEnvFile } from "../config/env.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; @@ -356,17 +365,45 @@ export async function onboard(opts: OnboardOptions): Promise { ); let existingConfig: PaperclipConfig | null = null; + let invalidBackupPath: string | undefined; if (configExists(opts.config)) { p.log.message(pc.dim(`${configPath} exists`)); try { existingConfig = readConfig(opts.config); + for (const warning of findPaperclipConfigKeyWarnings(existingConfig)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config appears invalid and will be updated.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + const canConfirmRepair = + opts.yes !== true && + opts.invokedByRun !== true && + process.stdin.isTTY === true && + process.stdout.isTTY === true; + if (!canConfirmRepair) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + invalidBackupPath = backupPath; } } @@ -646,7 +683,9 @@ export async function onboard(opts: OnboardOptions): Promise { p.log.message(pc.dim(`Using existing local secrets key file at ${keyResult.path}`)); } - writeConfig(config, opts.config); + writeConfig(config, opts.config, { + invalidBackupPath, + }); if (tc) trackInstallCompleted(tc, { adapterType: server.deploymentMode, diff --git a/cli/src/config/schema.ts b/cli/src/config/schema.ts index 65ddeab7334..799d8ba0d75 100644 --- a/cli/src/config/schema.ts +++ b/cli/src/config/schema.ts @@ -8,11 +8,15 @@ export { serverConfigSchema, authConfigSchema, telemetryConfigSchema, + updatesConfigSchema, storageConfigSchema, storageLocalDiskConfigSchema, storageS3ConfigSchema, secretsConfigSchema, secretsLocalEncryptedConfigSchema, + mergePaperclipConfig, + findPaperclipConfigKeyWarnings, + type ConfigKeyWarning, type PaperclipConfig, type LlmConfig, type DatabaseBackupConfig, @@ -27,4 +31,5 @@ export { type SecretsConfig, type SecretsLocalEncryptedConfig, type ConfigMeta, + type UpdatesConfig, } from "../../../packages/shared/src/config-schema.js"; diff --git a/cli/src/config/store.ts b/cli/src/config/store.ts index 8dddc777064..b1ab0229d17 100644 --- a/cli/src/config/store.ts +++ b/cli/src/config/store.ts @@ -1,6 +1,11 @@ import fs from "node:fs"; import path from "node:path"; -import { paperclipConfigSchema, type PaperclipConfig } from "./schema.js"; +import { isDeepStrictEqual } from "node:util"; +import { + mergePaperclipConfig, + paperclipConfigSchema, + type PaperclipConfig, +} from "./schema.js"; import { resolveDefaultConfigPath, resolvePaperclipInstanceId, @@ -95,24 +100,132 @@ export function readConfig(configPath?: string): PaperclipConfig | null { return parsed.data; } +function effectiveConfig(config: PaperclipConfig): Record { + const meta = { ...config.$meta } as Record; + delete meta.updatedAt; + delete meta.source; + return { + ...config, + $meta: meta, + }; +} + +function syncDirectory(directoryPath: string): void { + let directoryDescriptor: number | null = null; + try { + directoryDescriptor = fs.openSync(directoryPath, "r"); + fs.fsyncSync(directoryDescriptor); + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (process.platform !== "win32" || !["EACCES", "EINVAL", "EISDIR", "ENOTSUP", "EPERM"].includes(String(code))) { + throw error; + } + } finally { + if (directoryDescriptor !== null) fs.closeSync(directoryDescriptor); + } +} + +function durableCopyFile(sourcePath: string, destinationPath: string, flags = 0): void { + fs.copyFileSync(sourcePath, destinationPath, flags); + fs.chmodSync(destinationPath, 0o600); + + const backupDescriptor = fs.openSync(destinationPath, "r"); + try { + fs.fsyncSync(backupDescriptor); + } finally { + fs.closeSync(backupDescriptor); + } + syncDirectory(path.dirname(destinationPath)); +} + +function atomicWriteFile(filePath: string, contents: string): void { + let attempt = 0; + + while (true) { + const temporaryPath = `${filePath}.tmp-${process.pid}-${attempt}`; + attempt += 1; + let fileDescriptor: number | null = null; + try { + fileDescriptor = fs.openSync(temporaryPath, "wx", 0o600); + fs.writeFileSync(fileDescriptor, contents, "utf8"); + fs.fsyncSync(fileDescriptor); + fs.closeSync(fileDescriptor); + fileDescriptor = null; + fs.renameSync(temporaryPath, filePath); + syncDirectory(path.dirname(filePath)); + return; + } catch (error) { + if (fileDescriptor !== null) fs.closeSync(fileDescriptor); + fs.rmSync(temporaryPath, { force: true }); + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + +export function backupInvalidConfig(configPath?: string): string { + const filePath = resolveConfigPath(configPath); + if (!fs.existsSync(filePath)) { + throw new Error(`Cannot back up missing config at ${filePath}`); + } + + for (let suffix = 1; ; suffix += 1) { + const backupPath = `${filePath}.invalid-${suffix}`; + try { + durableCopyFile(filePath, backupPath, fs.constants.COPYFILE_EXCL); + return backupPath; + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + export function writeConfig( config: PaperclipConfig, configPath?: string, -): void { + options: { invalidBackupPath?: string } = {}, +): boolean { const filePath = resolveConfigPath(configPath); const dir = path.dirname(filePath); fs.mkdirSync(dir, { recursive: true }); + let nextConfig = paperclipConfigSchema.parse(config); + if (fs.existsSync(filePath)) { + try { + const source = paperclipConfigSchema.parse(migrateLegacyConfig(parseJson(filePath))); + nextConfig = paperclipConfigSchema.parse(mergePaperclipConfig(source, nextConfig)); + if (isDeepStrictEqual(effectiveConfig(source), effectiveConfig(nextConfig))) { + return false; + } + } catch (error) { + const invalidBackupPath = options.invalidBackupPath; + if (!invalidBackupPath) { + throw new Error( + `Refusing to overwrite invalid config at ${filePath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + if ( + !fs.existsSync(invalidBackupPath) || + !fs.readFileSync(filePath).equals(fs.readFileSync(invalidBackupPath)) + ) { + throw new Error( + `Refusing to overwrite ${filePath} because it changed after the invalid backup was created`, + ); + } + } + } + // Backup existing config before overwriting if (fs.existsSync(filePath)) { const backupPath = filePath + ".backup"; - fs.copyFileSync(filePath, backupPath); - fs.chmodSync(backupPath, 0o600); + durableCopyFile(filePath, backupPath); } - fs.writeFileSync(filePath, JSON.stringify(config, null, 2) + "\n", { - mode: 0o600, - }); + atomicWriteFile(filePath, JSON.stringify(nextConfig, null, 2) + "\n"); + return true; } export function configExists(configPath?: string): boolean { diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 436c7bc85c2..79af9ef36a0 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -336,6 +336,14 @@ Every local install keeps runtime state directly under the selected instance roo `PAPERCLIP_HOME` and `PAPERCLIP_INSTANCE_ID` override the home root and instance id respectively. `paperclipai onboard` echoes the resolved values in its banner (`Local home: | instance: | config: `) so you can confirm where state will land before continuing. +Config updates preserve unrecognized top-level and nested keys so provider or +plugin extensions survive `configure` and worktree port repair. Likely +misspellings of known keys produce a warning but are not removed. If an +existing `config.json` is malformed, `onboard` and `configure` first create a +byte-for-byte sibling backup named `config.json.invalid-1` (then `-2`, and so +on). Repair from defaults requires an interactive confirmation; non-interactive +runs stop without replacing the original. + ## Database in Dev (Auto-Handled) For local development, leave `DATABASE_URL` unset. diff --git a/packages/shared/src/config-schema.test.ts b/packages/shared/src/config-schema.test.ts index 9e2a75ccf78..5810f2c152e 100644 --- a/packages/shared/src/config-schema.test.ts +++ b/packages/shared/src/config-schema.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it } from "vitest"; -import { paperclipConfigSchema } from "./config-schema.js"; +import { + findPaperclipConfigKeyWarnings, + mergePaperclipConfig, + paperclipConfigSchema, +} from "./config-schema.js"; describe("paperclip config schema", () => { it("defaults omitted runtime paths to legacy instance-root locations", () => { @@ -24,4 +28,110 @@ describe("paperclip config schema", () => { expect(parsed.storage.localDisk.baseDir).toBe("~/.paperclip/instances/default/data/storage"); expect(parsed.secrets.localEncrypted.keyFilePath).toBe("~/.paperclip/instances/default/secrets/master.key"); }); + + it("retains extension keys at the top level and every nested config boundary", () => { + const parsed = paperclipConfigSchema.parse({ + $meta: { + version: 1, + updatedAt: "2026-05-10T00:00:00.000Z", + source: "configure", + extensionMeta: "keep", + }, + database: { + mode: "embedded-postgres", + backup: { + backupExtension: "keep", + }, + }, + logging: { + mode: "file", + }, + server: { + serverExtension: "keep", + }, + storage: { + localDisk: { + driverExtension: "keep", + }, + s3: { + s3Extension: "keep", + }, + }, + secrets: { + localEncrypted: { + providerExtension: "keep", + }, + }, + topLevelExtension: { + enabled: true, + }, + }); + + expect(parsed).toMatchObject({ + $meta: { extensionMeta: "keep" }, + database: { backup: { backupExtension: "keep" } }, + server: { serverExtension: "keep" }, + storage: { + localDisk: { driverExtension: "keep" }, + s3: { s3Extension: "keep" }, + }, + secrets: { localEncrypted: { providerExtension: "keep" } }, + topLevelExtension: { enabled: true }, + }); + }); + + it("merges retained extensions without resurrecting removed known fields", () => { + const source = paperclipConfigSchema.parse({ + $meta: { + version: 1, + updatedAt: "2026-05-10T00:00:00.000Z", + source: "configure", + }, + llm: { + provider: "openai", + apiKey: "remove-me", + providerExtension: "keep", + }, + database: { mode: "embedded-postgres" }, + logging: { mode: "file" }, + server: { port: 3100, serverExtension: "keep" }, + topLevelExtension: "keep", + }); + const update = paperclipConfigSchema.parse({ + ...source, + llm: { + provider: "openai", + }, + server: { + ...source.server, + port: 3200, + }, + }); + + const merged = mergePaperclipConfig(source, update); + + expect(merged.server.port).toBe(3200); + expect(merged.server.serverExtension).toBe("keep"); + expect(merged.topLevelExtension).toBe("keep"); + expect(merged.llm?.providerExtension).toBe("keep"); + expect(merged.llm).not.toHaveProperty("apiKey"); + }); + + it("warns about likely misspellings while leaving arbitrary extensions alone", () => { + expect(findPaperclipConfigKeyWarnings({ + servr: {}, + server: { + ports: 3100, + pluginOptions: { + arbitrary: true, + }, + }, + extensionNamespace: { + port: 9999, + }, + })).toEqual([ + { path: "servr", suggestion: "server" }, + { path: "server.ports", suggestion: "server.port" }, + ]); + }); }); diff --git a/packages/shared/src/config-schema.ts b/packages/shared/src/config-schema.ts index 229abd1a02d..190645062ef 100644 --- a/packages/shared/src/config-schema.ts +++ b/packages/shared/src/config-schema.ts @@ -13,19 +13,19 @@ export const configMetaSchema = z.object({ version: z.literal(1), updatedAt: z.string(), source: z.enum(["onboard", "configure", "doctor"]), -}); +}).passthrough(); export const llmConfigSchema = z.object({ provider: z.enum(["claude", "openai"]), apiKey: z.string().optional(), -}); +}).passthrough(); export const databaseBackupConfigSchema = z.object({ enabled: z.boolean().default(true), intervalMinutes: z.number().int().min(1).max(7 * 24 * 60).default(60), retentionDays: z.number().int().min(1).max(3650).default(7), dir: z.string().default("~/.paperclip/instances/default/data/backups"), -}); +}).passthrough(); export const databaseConfigSchema = z.object({ mode: z.enum(["embedded-postgres", "postgres"]).default("embedded-postgres"), @@ -38,12 +38,12 @@ export const databaseConfigSchema = z.object({ retentionDays: 7, dir: "~/.paperclip/instances/default/data/backups", }), -}); +}).passthrough(); export const loggingConfigSchema = z.object({ mode: z.enum(["file", "cloud"]), logDir: z.string().default("~/.paperclip/instances/default/logs"), -}); +}).passthrough(); export const serverConfigSchema = z.object({ deploymentMode: z.enum(DEPLOYMENT_MODES).default("local_trusted"), @@ -54,17 +54,17 @@ export const serverConfigSchema = z.object({ port: z.number().int().min(1).max(65535).default(3100), allowedHostnames: z.array(z.string().min(1)).default([]), serveUi: z.boolean().default(true), -}); +}).passthrough(); export const authConfigSchema = z.object({ baseUrlMode: z.enum(AUTH_BASE_URL_MODES).default("auto"), publicBaseUrl: z.string().url().optional(), disableSignUp: z.boolean().default(false), -}); +}).passthrough(); export const storageLocalDiskConfigSchema = z.object({ baseDir: z.string().default("~/.paperclip/instances/default/data/storage"), -}); +}).passthrough(); export const storageS3ConfigSchema = z.object({ bucket: z.string().min(1).default("paperclip"), @@ -72,7 +72,7 @@ export const storageS3ConfigSchema = z.object({ endpoint: z.string().optional(), prefix: z.string().default(""), forcePathStyle: z.boolean().default(false), -}); +}).passthrough(); export const storageConfigSchema = z.object({ provider: z.enum(STORAGE_PROVIDERS).default("local_disk"), @@ -85,11 +85,11 @@ export const storageConfigSchema = z.object({ prefix: "", forcePathStyle: false, }), -}); +}).passthrough(); export const secretsLocalEncryptedConfigSchema = z.object({ keyFilePath: z.string().default("~/.paperclip/instances/default/secrets/master.key"), -}); +}).passthrough(); export const secretsConfigSchema = z.object({ provider: z.enum(SECRET_PROVIDERS).default("local_encrypted"), @@ -97,15 +97,15 @@ export const secretsConfigSchema = z.object({ localEncrypted: secretsLocalEncryptedConfigSchema.default({ keyFilePath: "~/.paperclip/instances/default/secrets/master.key", }), -}); +}).passthrough(); export const telemetryConfigSchema = z.object({ enabled: z.boolean().default(true), -}).default({}); +}).passthrough().default({}); export const updatesConfigSchema = z.object({ checkEnabled: z.boolean().default(true), -}).default({}); +}).passthrough().default({}); export const paperclipConfigSchema = z .object({ @@ -140,6 +140,7 @@ export const paperclipConfigSchema = z }, }), }) + .passthrough() .superRefine((value, ctx) => { if (value.server.deploymentMode === "local_trusted" && value.server.exposure !== "private") { ctx.addIssue({ @@ -203,3 +204,150 @@ export type TelemetryConfig = z.infer; export type UpdatesConfig = z.infer; export type ConfigMeta = z.infer; export type DatabaseBackupConfig = z.infer; + +function isRecord(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +function unwrapConfigSchema(schema: z.ZodTypeAny): z.ZodTypeAny { + let current = schema; + while (true) { + if (current instanceof z.ZodEffects) { + current = current.innerType(); + continue; + } + if (current instanceof z.ZodOptional) { + current = current.unwrap(); + continue; + } + if (current instanceof z.ZodDefault) { + current = current.removeDefault(); + continue; + } + return current; + } +} + +function mergeUnknownConfigKeys( + source: Record, + update: Record, + schema: z.ZodTypeAny, +): Record { + const objectSchema = unwrapConfigSchema(schema); + if (!(objectSchema instanceof z.ZodObject)) return { ...update }; + + const shape = objectSchema.shape as Record; + const merged = { ...update }; + + for (const [key, sourceValue] of Object.entries(source)) { + const childSchema = shape[key]; + if (childSchema === undefined) { + merged[key] = sourceValue; + continue; + } + + const updateValue = update[key]; + if (isRecord(sourceValue) && isRecord(updateValue)) { + merged[key] = mergeUnknownConfigKeys(sourceValue, updateValue, childSchema); + } + } + + return merged; +} + +/** + * Applies a config update while retaining extension keys from the parsed source. + * Known optional keys that are absent from the update stay absent, so callers can + * intentionally clear values such as llm.apiKey or auth.publicBaseUrl. + */ +export function mergePaperclipConfig( + source: PaperclipConfig, + update: PaperclipConfig, +): PaperclipConfig { + return mergeUnknownConfigKeys( + source as Record, + update as Record, + paperclipConfigSchema, + ) as PaperclipConfig; +} + +export type ConfigKeyWarning = { + path: string; + suggestion: string; +}; + +function editDistance(left: string, right: string): number { + const previous = Array.from({ length: right.length + 1 }, (_, index) => index); + + for (let leftIndex = 1; leftIndex <= left.length; leftIndex += 1) { + const current = [leftIndex]; + for (let rightIndex = 1; rightIndex <= right.length; rightIndex += 1) { + const substitutionCost = left[leftIndex - 1] === right[rightIndex - 1] ? 0 : 1; + current[rightIndex] = Math.min( + current[rightIndex - 1] + 1, + previous[rightIndex] + 1, + previous[rightIndex - 1] + substitutionCost, + ); + } + previous.splice(0, previous.length, ...current); + } + + return previous[right.length]; +} + +function nearMatch(key: string, candidates: string[]): string | null { + const normalizedKey = key.toLowerCase(); + let best: { candidate: string; distance: number } | null = null; + + for (const candidate of candidates) { + const normalizedCandidate = candidate.toLowerCase(); + if (normalizedKey === normalizedCandidate && key !== candidate) return candidate; + + const distance = editDistance(normalizedKey, normalizedCandidate); + const threshold = Math.max(normalizedKey.length, normalizedCandidate.length) >= 8 ? 2 : 1; + if (distance > threshold || (best && distance >= best.distance)) continue; + best = { candidate, distance }; + } + + return best?.candidate ?? null; +} + +function collectConfigKeyWarnings( + value: Record, + schema: z.ZodTypeAny, + prefix: string, + warnings: ConfigKeyWarning[], +): void { + const objectSchema = unwrapConfigSchema(schema); + if (!(objectSchema instanceof z.ZodObject)) return; + + const shape = objectSchema.shape as Record; + const knownKeys = Object.keys(shape); + + for (const [key, childValue] of Object.entries(value)) { + const childSchema = shape[key]; + const childPath = prefix ? `${prefix}.${key}` : key; + if (childSchema === undefined) { + const suggestion = nearMatch(key, knownKeys); + if (suggestion) { + warnings.push({ + path: childPath, + suggestion: prefix ? `${prefix}.${suggestion}` : suggestion, + }); + } + continue; + } + + if (isRecord(childValue)) { + collectConfigKeyWarnings(childValue, childSchema, childPath, warnings); + } + } +} + +/** Returns likely misspellings among retained extension keys without modifying them. */ +export function findPaperclipConfigKeyWarnings(config: unknown): ConfigKeyWarning[] { + if (!isRecord(config)) return []; + const warnings: ConfigKeyWarning[] = []; + collectConfigKeyWarnings(config, paperclipConfigSchema, "", warnings); + return warnings; +} diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 512f54aeeb8..2b3d6b918bb 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -2262,7 +2262,12 @@ export { storageS3ConfigSchema, secretsLocalEncryptedConfigSchema, telemetryConfigSchema, + updatesConfigSchema, + mergePaperclipConfig, + findPaperclipConfigKeyWarnings, + type ConfigKeyWarning, type TelemetryConfig, + type UpdatesConfig, type PaperclipConfig, type LlmConfig, type DatabaseBackupConfig, diff --git a/server/src/__tests__/config-file.test.ts b/server/src/__tests__/config-file.test.ts index 9d5b2701ca7..9cc9451138a 100644 --- a/server/src/__tests__/config-file.test.ts +++ b/server/src/__tests__/config-file.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { readConfigFile } from "../config-file.js"; const ORIGINAL_PAPERCLIP_CONFIG = process.env.PAPERCLIP_CONFIG; @@ -42,6 +42,7 @@ describe("readConfigFile", () => { }); afterEach(() => { + vi.restoreAllMocks(); if (ORIGINAL_PAPERCLIP_CONFIG === undefined) { delete process.env.PAPERCLIP_CONFIG; } else { @@ -89,4 +90,25 @@ describe("readConfigFile", () => { }, }); }); + + it("warns about likely misspellings without stripping them", () => { + const config = { + ...(minimalConfig() as Record), + server: { + ports: 3200, + }, + }; + const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); + writeConfig(configPath, config); + + expect(readConfigFile()).toMatchObject({ + server: { + port: 3100, + ports: 3200, + }, + }); + expect(warn).toHaveBeenCalledWith( + "Unknown config key server.ports; did you mean server.port? It will be preserved.", + ); + }); }); diff --git a/server/src/__tests__/worktree-config.test.ts b/server/src/__tests__/worktree-config.test.ts index 1ea6859f99e..36a859d12b2 100644 --- a/server/src/__tests__/worktree-config.test.ts +++ b/server/src/__tests__/worktree-config.test.ts @@ -1,3 +1,4 @@ +import fsSync from "node:fs"; import fs from "node:fs/promises"; import os from "node:os"; import path from "node:path"; @@ -981,6 +982,83 @@ describe("worktree config repair", () => { expect(writtenConfig.auth.publicBaseUrl).toBe("https://paperclip.example"); }); + it("preserves top-level and nested config extensions while persisting runtime ports", async () => { + const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-worktree-config-extensions-")); + const worktreeRoot = path.join(tempRoot, "config-extensions"); + const paperclipDir = path.join(worktreeRoot, ".paperclip"); + const configPath = path.join(paperclipDir, "config.json"); + const isolatedHome = path.join(tempRoot, ".paperclip-worktrees"); + const instanceRoot = path.join(isolatedHome, "instances", "config-extensions"); + const base = buildIsolatedConfig(instanceRoot, 3101, 54331); + const config = { + ...base, + topLevelExtension: { enabled: true }, + database: { + ...base.database, + backup: { + ...base.database.backup, + backupExtension: "keep", + }, + }, + server: { + ...base.server, + serverExtension: "keep", + }, + storage: { + ...base.storage, + localDisk: { + ...base.storage.localDisk, + driverExtension: "keep", + }, + }, + }; + + await fs.mkdir(paperclipDir, { recursive: true }); + await fs.writeFile(configPath, `${JSON.stringify(config, null, 2)}\n`, "utf8"); + await fs.writeFile( + path.join(paperclipDir, ".env"), + ["# Paperclip environment variables", "PAPERCLIP_IN_WORKTREE=true", ""].join("\n"), + "utf8", + ); + + process.chdir(worktreeRoot); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + process.env.PAPERCLIP_WORKTREE_NAME = "config-extensions"; + process.env.PAPERCLIP_HOME = isolatedHome; + process.env.PAPERCLIP_INSTANCE_ID = "config-extensions"; + process.env.PAPERCLIP_CONFIG = configPath; + delete process.env.PORT; + delete process.env.DATABASE_URL; + + const open = vi.spyOn(fsSync, "openSync"); + const sync = vi.spyOn(fsSync, "fsyncSync"); + maybePersistWorktreeRuntimePorts({ serverPort: 3103, databasePort: 54335 }); + + expect(open).toHaveBeenCalledWith(paperclipDir, "r"); + expect(sync).toHaveBeenCalled(); + + const writtenConfig = JSON.parse(await fs.readFile(configPath, "utf8")); + expect(writtenConfig).toMatchObject({ + topLevelExtension: { enabled: true }, + database: { + embeddedPostgresPort: 54335, + backup: { backupExtension: "keep" }, + }, + server: { + port: 3103, + serverExtension: "keep", + }, + storage: { + localDisk: { driverExtension: "keep" }, + }, + }); + + const stableTime = new Date("2020-01-01T00:00:00.000Z"); + await fs.utimes(configPath, stableTime, stableTime); + maybePersistWorktreeRuntimePorts({ serverPort: 3103, databasePort: 54335 }); + expect((await fs.stat(configPath)).mtimeMs).toBe(stableTime.getTime()); + }); + it("can update the in-memory config when auth URL already includes a port", () => { const { config, changed } = applyRuntimePortSelectionToConfig( buildLegacyConfig("/tmp/shared", "http://my-host.ts.net:3100"), diff --git a/server/src/config-file.ts b/server/src/config-file.ts index 9c150f4a782..c66685599f0 100644 --- a/server/src/config-file.ts +++ b/server/src/config-file.ts @@ -1,5 +1,9 @@ import fs from "node:fs"; -import { paperclipConfigSchema, type PaperclipConfig } from "@paperclipai/shared"; +import { + findPaperclipConfigKeyWarnings, + paperclipConfigSchema, + type PaperclipConfig, +} from "@paperclipai/shared"; import { ZodError } from "zod"; import { resolvePaperclipConfigPath } from "./paths.js"; @@ -26,7 +30,13 @@ export function readConfigFile(): PaperclipConfig | null { } try { - return paperclipConfigSchema.parse(raw); + const config = paperclipConfigSchema.parse(raw); + for (const warning of findPaperclipConfigKeyWarnings(config)) { + console.warn( + `Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`, + ); + } + return config; } catch (error) { if (error instanceof ZodError) { throw new Error(`Invalid Paperclip config at ${configPath}: ${formatConfigValidationError(error)}`); diff --git a/server/src/worktree-config.ts b/server/src/worktree-config.ts index de8d824c18c..de999134bdd 100644 --- a/server/src/worktree-config.ts +++ b/server/src/worktree-config.ts @@ -1,7 +1,12 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import type { PaperclipConfig } from "@paperclipai/shared"; +import { isDeepStrictEqual } from "node:util"; +import { + mergePaperclipConfig, + paperclipConfigSchema, + type PaperclipConfig, +} from "@paperclipai/shared"; import { resolvePaperclipConfigPath, resolvePaperclipEnvPath } from "./paths.js"; function nonEmpty(value: string | null | undefined): string | null { @@ -305,9 +310,60 @@ function resolveWorktreeRuntimeContext( }; } -function writeConfigFile(configPath: string, config: PaperclipConfig): void { +function atomicWriteFile(filePath: string, contents: string): void { + let attempt = 0; + + while (true) { + const temporaryPath = `${filePath}.tmp-${process.pid}-${attempt}`; + attempt += 1; + let fileDescriptor: number | null = null; + try { + fileDescriptor = fs.openSync(temporaryPath, "wx", 0o600); + fs.writeFileSync(fileDescriptor, contents, "utf8"); + fs.fsyncSync(fileDescriptor); + fs.closeSync(fileDescriptor); + fileDescriptor = null; + fs.renameSync(temporaryPath, filePath); + let directoryDescriptor: number | null = null; + try { + directoryDescriptor = fs.openSync(path.dirname(filePath), "r"); + fs.fsyncSync(directoryDescriptor); + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if ( + process.platform !== "win32" || + !["EACCES", "EINVAL", "EISDIR", "ENOTSUP", "EPERM"].includes(String(code)) + ) { + throw error; + } + } finally { + if (directoryDescriptor !== null) fs.closeSync(directoryDescriptor); + } + return; + } catch (error) { + if (fileDescriptor !== null) fs.closeSync(fileDescriptor); + fs.rmSync(temporaryPath, { force: true }); + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + +function writeConfigFile(configPath: string, config: PaperclipConfig): boolean { fs.mkdirSync(path.dirname(configPath), { recursive: true }); - fs.writeFileSync(configPath, JSON.stringify(config, null, 2) + "\n", { mode: 0o600 }); + const update = paperclipConfigSchema.parse(config); + const source = fs.existsSync(configPath) + ? paperclipConfigSchema.parse(JSON.parse(fs.readFileSync(configPath, "utf8"))) + : null; + const nextConfig = source + ? paperclipConfigSchema.parse(mergePaperclipConfig(source, update)) + : update; + + if (source && isDeepStrictEqual(source, nextConfig)) return false; + + atomicWriteFile(configPath, JSON.stringify(nextConfig, null, 2) + "\n"); + return true; } function resolveRepoManagedWorktreesRoot(worktreeRoot: string): string | null { From f6c6452b25bdd61f0f8d98a19f4b2b907ea4f7ac Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 00:41:42 -0500 Subject: [PATCH 012/311] fix(server): preserve managed environment drift on boot (#10979) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip creates a managed sandbox environment for each company during boot > - Operators can change environment fields after Paperclip creates the environment > - The boot reconciler replaced those changes without checking for drift > - This pull request adds stock hashes and transactional drift reconciliation > - The benefit is that Paperclip can update untouched stock fields without losing operator work ## Linked Issues or Issue Description **What happened?** The managed sandbox boot reconciler rewrote the stock description, configuration, metadata, and status on every start. It did not detect operator changes first. A restart could therefore remove an operator's changes. **Expected behavior** Paperclip must preserve operator changes by default. It must update an untouched stock environment when Paperclip ships new stock values. It must perform each row update and stock-hash update atomically. **Steps to reproduce** 1. Start Paperclip and let it create the managed sandbox environment. 2. Change one Paperclip-owned stock field on that environment. 3. Restart Paperclip. 4. Observe that the previous reconciler replaced the change. **Paperclip version or commit** This bug reproduces on `master` before this change. **Deployment mode** Local development and self-hosted server boot are affected. ## What Changed - Add a shared deterministic stock-hash and drift classifier for built-in resources. - Track the managed sandbox stock hash with the company-scoped built-in resource binding. - Reconcile the environment and its stock metadata in one transaction with a row lock. - Preserve operator-modified and unmanaged rows and report their skipped update state. - Use archive ownership tokens so provider recovery reactivates only Paperclip-archived rows and preserves later operator archive decisions. - Keep operator-owned environment variables and unrelated metadata out of the stock fingerprint. - Add activity records for managed environment creation, updates, skipped drift, tracking initialization, and archive changes. - Add regression tests for current stock, available stock updates, operator drift, unmanaged rows, archive and reactivation, user-owned fields, and concurrent reconciliation. ## Verification - `pnpm -r typecheck` - `pnpm build` - `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm test:run -- --mode serialized` (128 suites passed) - Repository general server, UI, CLI, shared, skills catalog, database, adapter, plugin SDK, and plugin creator projects passed. Two embedded-Postgres tests exceeded the host's five-second default under the aggregate run and passed in the complete database project with `--testTimeout=20000`. One timing-sensitive sandbox stream test passed on its focused retry. - Focused managed-environment unit and integration coverage passed: 49 tests across the drift classifier, boot report, and environment service suites. ## Risks - The main risk is an incorrect ownership boundary in the stock fingerprint. The fingerprint includes only Paperclip-owned stock fields. Tests confirm that environment variables and unrelated metadata survive reconciliation. - Concurrent reconciliation could otherwise overwrite a late operator edit. The implementation locks the environment row and updates the row and hash binding in one transaction. A concurrency test covers this path. - There is no schema migration. Existing managed rows initialize tracking without replacing their current values. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-5 Codex. The exact serving snapshot and context-window size were not exposed. The model used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../src/__tests__/environment-service.test.ts | 318 +++++++- server/src/services/built-in-agents.ts | 42 +- server/src/services/environments.ts | 683 ++++++++++++++---- .../src/services/managed-environments.test.ts | 74 +- server/src/services/managed-environments.ts | 95 ++- .../services/managed-resource-drift.test.ts | 43 ++ server/src/services/managed-resource-drift.ts | 36 + 7 files changed, 1088 insertions(+), 203 deletions(-) create mode 100644 server/src/services/managed-resource-drift.test.ts create mode 100644 server/src/services/managed-resource-drift.ts diff --git a/server/src/__tests__/environment-service.test.ts b/server/src/__tests__/environment-service.test.ts index 4ce7c8f8254..9e81c9cf700 100644 --- a/server/src/__tests__/environment-service.test.ts +++ b/server/src/__tests__/environment-service.test.ts @@ -2,7 +2,9 @@ import { randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { and, eq } from "drizzle-orm"; import { + activityLog, agents, + builtInManagedResources, companies, companySecretBindings, companySecrets, @@ -44,6 +46,7 @@ describeEmbeddedPostgres("environmentService leases", () => { }); afterEach(async () => { + await db.delete(activityLog); await db.delete(companySecretBindings); await db.delete(environmentCustomImageSetupSessions); await db.delete(environmentLeases); @@ -115,6 +118,18 @@ describeEmbeddedPostgres("environmentService leases", () => { return { companyId, agentId, environmentId, runId }; } + async function seedCompany(name = "Acme") { + const companyId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name, + status: "active", + createdAt: new Date(), + updatedAt: new Date(), + }); + return companyId; + } + it("acquires and releases a lease for a run", async () => { const { companyId, environmentId, runId } = await seedEnvironment(); @@ -704,12 +719,16 @@ describeEmbeddedPostgres("environmentService leases", () => { }); it("ensures and refreshes a managed sandbox environment for an arbitrary provider", async () => { - const created = await svc.ensureManagedSandboxEnvironment({ + const companyId = await seedCompany(); + const createdResult = await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", description: "Managed Daytona sandbox environment.", provider: "daytona", config: { target: "us" }, }); + const created = createdResult.environment; + expect(createdResult).toMatchObject({ action: "added", stockStatus: "missing" }); expect(created.driver).toBe("sandbox"); expect(created.name).toBe("Daytona"); @@ -718,18 +737,45 @@ describeEmbeddedPostgres("environmentService leases", () => { expect(created.metadata?.managedByPaperclip).toBe(true); expect(created.metadata?.managedSandboxProvider).toBe("daytona"); - // Idempotent: a second call refreshes config and name in place, and a - // description omitted from the spec is cleared, not pinned forever. - const refreshed = await svc.ensureManagedSandboxEnvironment({ + // A stock update advances config and name in place, and a description + // omitted from the spec is cleared rather than pinned forever. + const refreshedResult = await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona (EU)", provider: "daytona", config: { target: "eu" }, }); + const refreshed = refreshedResult.environment; + expect(refreshedResult).toMatchObject({ + action: "updated", + stockStatus: "stock_update_available", + }); expect(refreshed.id).toBe(created.id); expect(refreshed.name).toBe("Daytona (EU)"); expect(refreshed.config.target).toBe("eu"); expect(refreshed.description).toBeNull(); + const [binding] = await db + .select() + .from(builtInManagedResources) + .where(and( + eq(builtInManagedResources.companyId, companyId), + eq(builtInManagedResources.resourceKind, "environment"), + )); + expect(binding).toMatchObject({ + resourceId: created.id, + stockHash: refreshedResult.stockHash, + }); + const activity = await db + .select({ action: activityLog.action }) + .from(activityLog) + .where(eq(activityLog.companyId, companyId)) + .orderBy(activityLog.createdAt); + expect(activity.map((entry) => entry.action)).toEqual([ + "environment.managed_stock_added", + "environment.managed_stock_updated", + ]); + const rows = await db .select() .from(environments) @@ -737,15 +783,136 @@ describeEmbeddedPostgres("environmentService leases", () => { expect(rows).toHaveLength(1); }); + it("treats stock_current as an environment-row no-op and preserves user-owned fields", async () => { + const companyId = await seedCompany(); + const created = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + description: "Managed stock", + provider: "daytona", + config: { target: "us" }, + stockVersion: "v1", + }); + const operatorUpdatedAt = new Date("2026-08-06T12:00:00.000Z"); + await db + .update(environments) + .set({ + envVars: { OPERATOR_FLAG: "kept" }, + metadata: { + ...created.environment.metadata, + operatorNote: "keep me", + }, + updatedAt: operatorUpdatedAt, + }) + .where(eq(environments.id, created.environment.id)); + const [bindingBefore] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + const activityBefore = await db + .select({ id: activityLog.id }) + .from(activityLog) + .where(eq(activityLog.companyId, companyId)); + + const reconciled = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + description: "Managed stock", + provider: "daytona", + config: { target: "us" }, + stockVersion: "v1", + }); + const [bindingAfter] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + + expect(reconciled).toMatchObject({ + action: "unchanged", + stockStatus: "stock_current", + updateAvailable: false, + }); + expect(reconciled.environment.updatedAt).toEqual(operatorUpdatedAt); + expect(reconciled.environment.envVars).toEqual({ OPERATOR_FLAG: "kept" }); + expect(reconciled.environment.metadata?.operatorNote).toBe("keep me"); + expect(bindingAfter?.updatedAt).toEqual(bindingBefore?.updatedAt); + const activityAfter = await db + .select({ id: activityLog.id }) + .from(activityLog) + .where(eq(activityLog.companyId, companyId)); + expect(activityAfter).toHaveLength(activityBefore.length); + }); + + it("classifies operator drift, preserves the row, and exposes the pending stock update", async () => { + const companyId = await seedCompany(); + const created = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + description: "Managed stock", + provider: "daytona", + config: { target: "us" }, + stockVersion: "v1", + }); + const [bindingBefore] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + const operatorUpdatedAt = new Date("2026-08-06T12:01:00.000Z"); + await db + .update(environments) + .set({ + description: "Operator description", + config: { provider: "daytona", target: "operator" }, + updatedAt: operatorUpdatedAt, + }) + .where(eq(environments.id, created.environment.id)); + + const reconciled = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona v2", + description: "Managed stock v2", + provider: "daytona", + config: { target: "eu" }, + stockVersion: "v2", + }); + const [bindingAfter] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + + expect(reconciled).toMatchObject({ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + }); + expect(reconciled.environment).toMatchObject({ + name: "Daytona", + description: "Operator description", + config: { provider: "daytona", target: "operator" }, + updatedAt: operatorUpdatedAt, + }); + expect(bindingAfter?.stockHash).toBe(bindingBefore?.stockHash); + expect(bindingAfter?.stockVersion).toBe("v1"); + expect(reconciled.stockHash).not.toBe(bindingAfter?.stockHash); + const activity = await db + .select({ action: activityLog.action }) + .from(activityLog) + .where(eq(activityLog.companyId, companyId)) + .orderBy(activityLog.createdAt); + expect(activity.at(-1)?.action).toBe("environment.managed_stock_skipped"); + }); + it("adopts the managed slot on a provider switch and drops the stale kubernetes marker", async () => { - const kubernetes = await svc.ensureKubernetesEnvironment({ inCluster: true, backend: "job" }); + const companyId = await seedCompany(); + const kubernetes = await svc.ensureKubernetesEnvironment(companyId, { inCluster: true, backend: "job" }); expect(kubernetes.metadata?.managedKubernetesSandbox).toBe(true); - const daytona = await svc.ensureManagedSandboxEnvironment({ + const daytona = (await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", provider: "daytona", config: { target: "us" }, - }); + })).environment; expect(daytona.id).toBe(kubernetes.id); expect(daytona.name).toBe("Daytona"); @@ -756,20 +923,22 @@ describeEmbeddedPostgres("environmentService leases", () => { expect(await svc.findKubernetesEnvironment()).toBeNull(); // And back: the kubernetes wrapper re-adopts the same row. - const restored = await svc.ensureKubernetesEnvironment({ inCluster: true, backend: "job" }); + const restored = await svc.ensureKubernetesEnvironment(companyId, { inCluster: true, backend: "job" }); expect(restored.id).toBe(kubernetes.id); expect(restored.metadata?.managedKubernetesSandbox).toBe(true); }); it("archives the managed sandbox row only for its own provider and reactivates on ensure", async () => { + const companyId = await seedCompany(); // Nothing provisioned yet: archiving is a no-op. expect(await svc.archiveManagedSandboxEnvironment({ provider: "daytona" })).toBeNull(); - const created = await svc.ensureManagedSandboxEnvironment({ + const created = (await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", provider: "daytona", config: { target: "us" }, - }); + })).environment; expect(created.status).toBe("active"); // Another provider's unavailability leaves this provider's row alone. @@ -778,21 +947,121 @@ describeEmbeddedPostgres("environmentService leases", () => { const archived = await svc.archiveManagedSandboxEnvironment({ provider: "daytona" }); expect(archived?.id).toBe(created.id); expect(archived?.status).toBe("archived"); + const archiveActivity = await db + .select({ action: activityLog.action }) + .from(activityLog) + .where(and( + eq(activityLog.companyId, companyId), + eq(activityLog.entityId, created.id), + )) + .orderBy(activityLog.createdAt); + expect(archiveActivity.at(-1)?.action).toBe( + "environment.managed_provider_unavailable_archived", + ); // Already archived: a repeat call is a no-op. expect(await svc.archiveManagedSandboxEnvironment({ provider: "daytona" })).toBeNull(); // The next healthy boot's ensure re-activates the same row. - const restored = await svc.ensureManagedSandboxEnvironment({ + const restored = (await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", provider: "daytona", config: { target: "us" }, - }); + })).environment; expect(restored.id).toBe(created.id); expect(restored.status).toBe("active"); }); - it("adopts an existing unmanaged sandbox row holding the desired name", async () => { + it("reactivates a reconciler-archived row without replacing operator drift", async () => { + const companyId = await seedCompany(); + const created = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + description: "Managed stock", + provider: "daytona", + config: { target: "us" }, + stockVersion: "v1", + }); + await db + .update(environments) + .set({ + description: "Operator description", + config: { provider: "daytona", target: "operator" }, + }) + .where(eq(environments.id, created.environment.id)); + + expect((await svc.archiveManagedSandboxEnvironment({ provider: "daytona" }))?.status) + .toBe("archived"); + const [archivedBinding] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + expect(archivedBinding?.defaultsJson).toMatchObject({ + description: "Managed stock", + status: "archived", + }); + + const restored = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona v2", + description: "Managed stock v2", + provider: "daytona", + config: { target: "eu" }, + stockVersion: "v2", + }); + expect(restored).toMatchObject({ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + environment: { + status: "active", + name: "Daytona", + description: "Operator description", + config: { provider: "daytona", target: "operator" }, + }, + }); + const [reactivatedBinding] = await db + .select() + .from(builtInManagedResources) + .where(eq(builtInManagedResources.companyId, companyId)); + expect(reactivatedBinding?.defaultsJson).toMatchObject({ + description: "Managed stock", + status: "active", + }); + expect(reactivatedBinding?.stockVersion).toBe("v1"); + expect(restored.stockHash).not.toBe(reactivatedBinding?.stockHash); + }); + + it("preserves an operator archive decision made after provider archival", async () => { + const companyId = await seedCompany(); + const created = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + provider: "daytona", + config: { target: "us" }, + }); + expect((await svc.archiveManagedSandboxEnvironment({ provider: "daytona" }))?.status) + .toBe("archived"); + expect((await svc.update(created.environment.id, { status: "archived" }))?.status) + .toBe("archived"); + + const reconciled = await svc.ensureManagedSandboxEnvironment({ + companyId, + name: "Daytona", + provider: "daytona", + config: { target: "us" }, + }); + expect(reconciled).toMatchObject({ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + environment: { status: "archived" }, + }); + }); + + it("preserves an existing unmanaged sandbox row holding the desired name", async () => { + const companyId = await seedCompany(); const handMade = await svc.create({ name: "Daytona", driver: "sandbox", @@ -801,15 +1070,20 @@ describeEmbeddedPostgres("environmentService leases", () => { }); expect(handMade.metadata?.managedByPaperclip).toBeUndefined(); - const adopted = await svc.ensureManagedSandboxEnvironment({ + const reconciliation = await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", provider: "daytona", config: { target: "eu" }, }); - expect(adopted.id).toBe(handMade.id); - expect(adopted.config.target).toBe("eu"); - expect(adopted.metadata?.managedByPaperclip).toBe(true); - expect(adopted.metadata?.managedSandboxProvider).toBe("daytona"); + expect(reconciliation).toMatchObject({ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + }); + expect(reconciliation.environment.id).toBe(handMade.id); + expect(reconciliation.environment.config.target).toBe("us"); + expect(reconciliation.environment.metadata?.managedByPaperclip).toBeUndefined(); const rows = await db .select() @@ -819,6 +1093,7 @@ describeEmbeddedPostgres("environmentService leases", () => { }); it("keeps the current name when the desired name belongs to another row", async () => { + const companyId = await seedCompany(); await svc.create({ name: "Daytona", driver: "ssh", @@ -830,15 +1105,16 @@ describeEmbeddedPostgres("environmentService leases", () => { remoteWorkspacePath: "/srv/paperclip", }, }); - const kubernetes = await svc.ensureKubernetesEnvironment({ inCluster: true }); + const kubernetes = await svc.ensureKubernetesEnvironment(companyId, { inCluster: true }); // The managed slot is adopted, but the rename would collide with the ssh // row on environments_name_idx; the ensure keeps the existing name. - const adopted = await svc.ensureManagedSandboxEnvironment({ + const adopted = (await svc.ensureManagedSandboxEnvironment({ + companyId, name: "Daytona", provider: "daytona", config: { target: "us" }, - }); + })).environment; expect(adopted.id).toBe(kubernetes.id); expect(adopted.name).toBe(kubernetes.name); expect(adopted.config.provider).toBe("daytona"); diff --git a/server/src/services/built-in-agents.ts b/server/src/services/built-in-agents.ts index 253686f7ba9..b8a25460c72 100644 --- a/server/src/services/built-in-agents.ts +++ b/server/src/services/built-in-agents.ts @@ -1,4 +1,3 @@ -import { createHash } from "node:crypto"; import { readFileSync } from "node:fs"; import { createRequire } from "node:module"; import path from "node:path"; @@ -22,6 +21,12 @@ import { companySkillService } from "./company-skills.js"; import { routineService } from "./routines.js"; import { accessService } from "./access.js"; import { listAdapterModels } from "../adapters/registry.js"; +import { + resourceStatus, + stableJson, + stockHash, + type ManagedResourceStockStatus, +} from "./managed-resource-drift.js"; export type BuiltInAgentStatus = "not_provisioned" | "pending_approval" | "needs_setup" | "ready" | "paused"; @@ -72,11 +77,7 @@ export interface BuiltInAgentProvisionResult { } export type BuiltInManagedResourceKind = "instructions" | "skill" | "routine"; -export type BuiltInManagedResourceStockStatus = - | "missing" - | "stock_current" - | "stock_update_available" - | "operator_modified"; +export type BuiltInManagedResourceStockStatus = ManagedResourceStockStatus; export interface BuiltInManagedResourceState { resourceKind: BuiltInManagedResourceKind; @@ -495,21 +496,6 @@ function uniqueNonEmptyStrings(values: string[]) { return result; } -function stableJson(value: unknown): string { - if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`; - if (value && typeof value === "object") { - return `{${Object.entries(value as Record) - .sort(([left], [right]) => left.localeCompare(right)) - .map(([key, entry]) => `${JSON.stringify(key)}:${stableJson(entry)}`) - .join(",")}}`; - } - return JSON.stringify(value); -} - -function stockHash(value: unknown) { - return `sha256:${createHash("sha256").update(stableJson(value)).digest("hex")}`; -} - function changedFileList(currentFiles: Record, stockFiles: Record) { const paths = new Set([...Object.keys(currentFiles), ...Object.keys(stockFiles)]); return [...paths] @@ -517,20 +503,6 @@ function changedFileList(currentFiles: Record, stockFiles .sort((left, right) => left.localeCompare(right)); } -function resourceStatus(input: { - resourceId: string | null; - currentHash: string | null; - bindingStockHash: string | null; - latestStockHash: string; -}): BuiltInManagedResourceStockStatus { - if (!input.resourceId || !input.currentHash) return "missing"; - if (input.currentHash === input.latestStockHash) return "stock_current"; - if (input.bindingStockHash && input.currentHash === input.bindingStockHash) { - return "stock_update_available"; - } - return "operator_modified"; -} - function stockState(input: { resourceKind: BuiltInManagedResourceKind; resourceKey: string; diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index 3cfb7b38f2f..657d366a024 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -1,7 +1,10 @@ +import { randomUUID } from "node:crypto"; import { and, desc, eq, inArray, ne, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, + builtInManagedResources, + companies, companySecretBindings, environmentCustomImageSetupSessions, environmentLeases, @@ -28,7 +31,13 @@ import { type UpdateEnvironment, } from "@paperclipai/shared"; import { conflict } from "../errors.js"; +import { logActivity } from "./activity-log.js"; import { isCloudManagedInstance } from "./cloud-instance.js"; +import { + resourceStatus, + stockHash, + type ManagedResourceStockStatus, +} from "./managed-resource-drift.js"; type EnvironmentRow = typeof environments.$inferSelect; type EnvironmentLeaseRow = typeof environmentLeases.$inferSelect; @@ -81,6 +90,8 @@ export interface KubernetesEnvironmentConfigInput { * lease time. */ export interface ManagedSandboxEnvironmentInput { + /** Company whose managed-resource binding should be reconciled. Omit at instance boot to bind every company. */ + companyId?: string; name: string; description?: string; /** Sandbox provider key (the plugin's driverKey, e.g. "kubernetes", "daytona"). */ @@ -92,6 +103,24 @@ export interface ManagedSandboxEnvironmentInput { * `findKubernetesEnvironment` keys on). */ extraMetadata?: Record; + /** Version label recorded with the stock binding; hashes remain the drift authority. */ + stockVersion?: string; +} + +export type ManagedSandboxEnvironmentReconcileAction = + | "added" + | "updated" + | "unchanged" + | "skipped"; + +export interface ManagedSandboxEnvironmentReconcileResult { + environment: Environment; + action: ManagedSandboxEnvironmentReconcileAction; + /** Classification observed before this reconciliation wrote anything. */ + stockStatus: ManagedResourceStockStatus; + /** True only when operator drift prevented the available stock update. */ + updateAvailable: boolean; + stockHash: string; } function cloneRecord(value: unknown, fallback: Record | null = null): Record | null { @@ -204,21 +233,81 @@ function countFromRows(rows: Array<{ count: number | string | null | undefined } type DbTransaction = Parameters[0]>[0]; type EnvironmentWriteDb = Pick; -export function environmentService(db: Db) { - /** The single Paperclip-managed sandbox row (`environments_managed_sandbox_idx`), if present. */ - const findManagedSandboxRow = () => - db - .select() - .from(environments) - .where(eq(environments.driver, "sandbox")) - .then( - (rows) => - rows.find( - (row) => - (row.metadata as Record | null)?.managedByPaperclip === true, - ) ?? null, - ); +const MANAGED_ENVIRONMENT_BUNDLE_KEY = "managed-sandbox-environment"; +const MANAGED_ENVIRONMENT_RESOURCE_KIND = "environment"; +const MANAGED_ENVIRONMENT_RESOURCE_KEY = "managed-sandbox"; +const MANAGED_ENVIRONMENT_STOCK_VERSION = "managed-environment-v1"; +const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY = "_paperclipManagedArchiveToken"; +const MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY = "_paperclipManagedArchiveToken"; +function managedEnvironmentBaselineDefaults( + defaultsJson: Record, +): Record { + const baseline = { ...defaultsJson }; + delete baseline[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]; + return baseline; +} + +function withoutManagedEnvironmentArchiveToken( + metadata: Record | null, +): Record | null { + if (!metadata) return null; + const next = { ...metadata }; + delete next[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]; + return next; +} + +function managedMetadataKeys( + desiredMetadata: Record, + bindings: Array<{ defaultsJson: Record }>, +): string[] { + const keys = new Set([ + "managedByPaperclip", + "managedSandboxProvider", + KUBERNETES_MANAGED_MARKER, + ...Object.keys(desiredMetadata), + ]); + for (const binding of bindings) { + const metadata = cloneRecord(binding.defaultsJson.metadata); + for (const key of Object.keys(metadata ?? {})) keys.add(key); + } + return [...keys].sort((left, right) => left.localeCompare(right)); +} + +function managedEnvironmentStock(input: { + name: string; + description: string | null; + config: Record; + metadata: Record | null; + status: string; +}, metadataKeys: readonly string[]): Record { + const metadata = input.metadata ?? {}; + return { + name: input.name, + description: input.description, + config: input.config, + metadata: Object.fromEntries( + metadataKeys.map((key) => [key, Object.prototype.hasOwnProperty.call(metadata, key) ? metadata[key] : null]), + ), + status: input.status, + }; +} + +function mergeManagedEnvironmentMetadata( + current: Record | null, + desired: Record, + keys: readonly string[], +): Record { + const merged = { ...(current ?? {}) }; + for (const key of keys) { + const value = Object.prototype.hasOwnProperty.call(desired, key) ? desired[key] : null; + if (value === null || value === undefined) delete merged[key]; + else merged[key] = value; + } + return merged; +} + +export function environmentService(db: Db) { /** * Idempotently ensure THE Paperclip-managed sandbox environment for this * instance, configured for an arbitrary sandbox provider plugin. Mirrors @@ -227,18 +316,15 @@ export function environmentService(db: Db) { * row per instance, so this function owns that single slot regardless of * provider: * - * - An existing managed row is adopted and refreshed (name, description, - * config, provider) on every call, so operator/control-plane changes flow - * via redeploy without recreating the row β€” including a provider switch, - * which also drops a stale provider-specific metadata marker. - * - An existing UNmanaged sandbox row holding the desired name is adopted - * and stamped as managed, so a row created by hand before the instance - * became config-managed converges instead of colliding on - * `environments_name_idx` on every boot. + * - A stock-controlled managed row advances to a new stock hash in the same + * transaction as its managed fields, including provider switches. + * - A stock-current row is returned without an environment write. + * - An operator-modified or previously unmanaged row is preserved and + * reported as skipped; its user-owned fields are never folded into stock. */ const ensureManagedSandboxEnvironment = async ( input: ManagedSandboxEnvironmentInput, - ): Promise => { + ): Promise => { const desiredConfig: Record = { ...(input.config ?? {}), provider: input.provider, @@ -248,99 +334,363 @@ export function environmentService(db: Db) { managedSandboxProvider: input.provider, ...(input.extraMetadata ?? {}), }; + if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) { + desiredMetadata[KUBERNETES_MANAGED_MARKER] = null; + } - const adopt = async (row: EnvironmentRow): Promise => { - const metadata: Record = { ...(row.metadata ?? {}), ...desiredMetadata }; - // A provider switch must not leave the previous provider's marker - // behind (`findKubernetesEnvironment` keys on it). - if (desiredMetadata[KUBERNETES_MANAGED_MARKER] !== true) { - delete metadata[KUBERNETES_MANAGED_MARKER]; - } - const now = new Date(); - const runUpdate = (values: { name?: string }) => - db + let activityCompanyIds: string[] = []; + let trackingInitialized = false; + let providerReactivated = false; + const reconciliation = await db.transaction( + async (tx): Promise => { + const companyIds = input.companyId + ? [input.companyId] + : await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id)); + activityCompanyIds = companyIds; + const bindingConditions = and( + eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY), + eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND), + eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY), + ...(companyIds.length > 0 ? [inArray(builtInManagedResources.companyId, companyIds)] : []), + ); + const bindings = companyIds.length > 0 + ? await tx.select().from(builtInManagedResources).where(bindingConditions) + : []; + trackingInitialized = bindings.length < companyIds.length; + const keys = managedMetadataKeys(desiredMetadata, bindings); + + const sandboxRows = await tx + .select() + .from(environments) + .where(eq(environments.driver, "sandbox")) + .for("update"); + let row = sandboxRows.find( + (candidate) => (candidate.metadata as Record | null)?.managedByPaperclip === true, + ) ?? sandboxRows.find((candidate) => candidate.name === input.name) ?? null; + + const writeBindings = async ( + environmentId: string, + stockVersion: string, + installedStockHash: string, + defaultsJson: Record, + replace: boolean, + ) => { + const targetCompanyIds = replace + ? companyIds + : companyIds.filter( + (companyId) => !bindings.some((binding) => binding.companyId === companyId), + ); + if (targetCompanyIds.length === 0) return; + const values = targetCompanyIds.map((companyId) => ({ + companyId, + bundleKey: MANAGED_ENVIRONMENT_BUNDLE_KEY, + resourceKind: MANAGED_ENVIRONMENT_RESOURCE_KIND, + resourceKey: MANAGED_ENVIRONMENT_RESOURCE_KEY, + resourceId: environmentId, + stockVersion, + stockHash: installedStockHash, + defaultsJson, + })); + const insert = tx.insert(builtInManagedResources).values(values); + if (!replace) { + await insert.onConflictDoNothing({ + target: [ + builtInManagedResources.companyId, + builtInManagedResources.bundleKey, + builtInManagedResources.resourceKind, + builtInManagedResources.resourceKey, + ], + }); + return; + } + await insert.onConflictDoUpdate({ + target: [ + builtInManagedResources.companyId, + builtInManagedResources.bundleKey, + builtInManagedResources.resourceKind, + builtInManagedResources.resourceKey, + ], + set: { + resourceId: environmentId, + stockVersion, + stockHash: installedStockHash, + defaultsJson, + updatedAt: new Date(), + }, + }); + }; + + if (!row) { + const nameOwner = await tx + .select() + .from(environments) + .where(eq(environments.name, input.name)) + .then((rows) => rows[0] ?? null); + if (nameOwner && nameOwner.driver !== "sandbox") { + throw new Error( + `Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`, + ); + } + const now = new Date(); + const inserted = await tx + .insert(environments) + .values({ + name: input.name, + description: input.description ?? null, + driver: "sandbox", + status: "active", + config: desiredConfig, + envVars: {}, + metadata: mergeManagedEnvironmentMetadata(null, desiredMetadata, keys), + createdAt: now, + updatedAt: now, + }) + // Either the managed-slot partial index or the global name index + // can select the concurrent winner. Treat both as convergence and + // reselect that winner under lock below. + .onConflictDoNothing() + .returning() + .then((rows) => rows[0] ?? null); + if (inserted) { + const stock = managedEnvironmentStock({ + name: inserted.name, + description: inserted.description ?? null, + config: inserted.config, + metadata: inserted.metadata, + status: inserted.status, + }, keys); + const latestStockHash = stockHash(stock); + await writeBindings( + inserted.id, + input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION, + latestStockHash, + stock, + true, + ); + return { + environment: toEnvironment(inserted), + action: "added", + stockStatus: "missing", + updateAvailable: false, + stockHash: latestStockHash, + }; + } + row = await tx + .select() + .from(environments) + .where(eq(environments.driver, "sandbox")) + .for("update") + .then( + (rows) => rows.find( + (candidate) => (candidate.metadata as Record | null)?.managedByPaperclip === true, + ) ?? null, + ); + if (!row) throw new Error("Failed to ensure managed sandbox environment"); + } + + const nameOwner = await tx + .select({ id: environments.id }) + .from(environments) + .where(eq(environments.name, input.name)) + .then((rows) => rows[0] ?? null); + const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name; + const desiredStock = managedEnvironmentStock({ + name: desiredName, + description: input.description ?? null, + config: desiredConfig, + metadata: desiredMetadata, + status: "active", + }, keys); + const latestStockHash = stockHash(desiredStock); + const currentStock = managedEnvironmentStock({ + name: row.name, + description: row.description ?? null, + config: row.config, + metadata: row.metadata, + status: row.status, + }, keys); + const currentHash = stockHash(currentStock); + const matchingBindings = bindings.filter((binding) => binding.resourceId === row!.id); + const rowArchiveToken = (row.metadata as Record | null) + ?.[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]; + let stockStatus = resourceStatus({ + resourceId: row.id, + currentHash, + bindingStockHash: matchingBindings[0]?.stockHash ?? null, + latestStockHash, + }); + if (stockStatus === "operator_modified") { + const stockControlledBinding = matchingBindings.find( + (binding) => resourceStatus({ + resourceId: row!.id, + currentHash, + bindingStockHash: binding.stockHash, + latestStockHash, + }) === "stock_update_available", + ); + if (stockControlledBinding) stockStatus = "stock_update_available"; + } + const operatorReaffirmedArchive = row.status === "archived" && matchingBindings.some( + (binding) => { + const bindingToken = binding.defaultsJson[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]; + return binding.defaultsJson.status === "archived" && + typeof bindingToken === "string" && + bindingToken !== rowArchiveToken; + }, + ); + if (operatorReaffirmedArchive) stockStatus = "operator_modified"; + + if (stockStatus === "operator_modified") { + const baseline = matchingBindings[0]; + let baselineDefaults = baseline + ? managedEnvironmentBaselineDefaults(baseline.defaultsJson) + : desiredStock; + let baselineHash = baseline?.stockHash ?? latestStockHash; + + // Provider unavailability is an operational state transition, not + // an operator edit. If the binding records that Paperclip archived + // this row, restore only its availability status. Keep every other + // operator-modified field intact and leave the stock update pending. + // A manually archived row still has an active binding baseline, so + // it remains operator_modified and is not reactivated here. + const archivedByReconciler = row.status === "archived" && + typeof rowArchiveToken === "string" && + matchingBindings.some((binding) => { + const bindingDefaults = binding.defaultsJson; + return bindingDefaults.status === "archived" && + bindingDefaults[MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY] === rowArchiveToken; + }); + if (archivedByReconciler) { + const reactivated = await tx + .update(environments) + .set({ + status: "active", + metadata: withoutManagedEnvironmentArchiveToken(row.metadata), + updatedAt: new Date(), + }) + .where(and(eq(environments.id, row.id), eq(environments.status, "archived"))) + .returning() + .then((rows) => rows[0] ?? null); + if (!reactivated) { + throw new Error("Managed sandbox environment changed during reactivation"); + } + row = reactivated; + providerReactivated = true; + + for (const binding of matchingBindings) { + const reactivatedDefaults = { + ...managedEnvironmentBaselineDefaults(binding.defaultsJson), + status: "active", + }; + const reactivatedHash = stockHash(reactivatedDefaults); + await tx + .update(builtInManagedResources) + .set({ + stockHash: reactivatedHash, + defaultsJson: reactivatedDefaults, + updatedAt: new Date(), + }) + .where(and( + eq(builtInManagedResources.id, binding.id), + eq(builtInManagedResources.resourceId, row.id), + )); + if (binding.id === baseline?.id) { + baselineDefaults = reactivatedDefaults; + baselineHash = reactivatedHash; + } + } + } + await writeBindings( + row.id, + baseline?.stockVersion ?? input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION, + baselineHash, + baselineDefaults, + false, + ); + return { + environment: toEnvironment(row), + action: "skipped", + stockStatus, + updateAvailable: true, + stockHash: latestStockHash, + }; + } + + if (stockStatus === "stock_current") { + await writeBindings( + row.id, + input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION, + latestStockHash, + desiredStock, + false, + ); + return { + environment: toEnvironment(row), + action: "unchanged", + stockStatus, + updateAvailable: false, + stockHash: latestStockHash, + }; + } + + const updated = await tx .update(environments) .set({ - ...values, - // The row mirrors the managed spec: omitting `description` clears - // a previously configured one rather than pinning it forever. + name: desiredName, description: input.description ?? null, config: desiredConfig, - metadata, + metadata: withoutManagedEnvironmentArchiveToken( + mergeManagedEnvironmentMetadata(row.metadata, desiredMetadata, keys), + ), status: "active", - updatedAt: now, + updatedAt: new Date(), }) .where(eq(environments.id, row.id)) .returning() - .then((rows) => rows[0] ?? row); - const updated = await runUpdate({ name: input.name }).catch((error: unknown) => { - // Another row already holds the desired name; keep the current name - // rather than failing a boot-time ensure over a display label. - if (hasConstraintName(error, "environments_name_idx")) { - return runUpdate({}); - } - throw error; - }); - return toEnvironment(updated); - }; - - const existing = await findManagedSandboxRow(); - if (existing) return adopt(existing); - - // The partial unique index `environments_managed_sandbox_idx` enforces - // "at most one Paperclip-managed sandbox row per instance" at the DB - // level. Use ON CONFLICT DO NOTHING keyed on that index so concurrent - // callers can race the INSERT; losers re-read the surviving row. - const now = new Date(); - const inserted = await db - .insert(environments) - .values({ - name: input.name, - description: input.description ?? null, - driver: "sandbox", - status: "active", - config: desiredConfig, - envVars: {}, - metadata: desiredMetadata, - createdAt: now, - updatedAt: now, - }) - .onConflictDoNothing({ - target: [environments.driver], - where: - sql`${environments.driver} = 'sandbox' AND (${environments.metadata} ->> 'managedByPaperclip')::boolean = true`, - }) - .returning() - .then((rows) => rows[0] ?? null) - .catch((error) => { - if ( - hasConstraintName(error, "environments_name_idx") - || hasConstraintName(error, "environments_managed_sandbox_idx") - ) { - return null; - } - throw error; - }); - if (inserted) return toEnvironment(inserted); - - // Either a concurrent caller won the managed slot, or an unmanaged row - // holds the desired name. Adopt whichever exists. - const winner = await findManagedSandboxRow(); - if (winner) return adopt(winner); - const sameName = await db - .select() - .from(environments) - .where(eq(environments.name, input.name)) - .then((rows) => rows[0] ?? null); - if (sameName) { - if (sameName.driver !== "sandbox") { - throw new Error( - `Failed to ensure managed sandbox environment: environment "${input.name}" already exists with driver "${sameName.driver}"`, + .then((rows) => rows[0] ?? null); + if (!updated) throw new Error("Managed sandbox environment changed during reconciliation"); + await writeBindings( + updated.id, + input.stockVersion ?? MANAGED_ENVIRONMENT_STOCK_VERSION, + latestStockHash, + desiredStock, + true, ); - } - return adopt(sameName); + return { + environment: toEnvironment(updated), + action: "updated", + stockStatus, + updateAvailable: false, + stockHash: latestStockHash, + }; + }, + ); + if (reconciliation.action !== "unchanged" || trackingInitialized) { + const action = reconciliation.action === "added" + ? "environment.managed_stock_added" + : reconciliation.action === "updated" + ? "environment.managed_stock_updated" + : reconciliation.action === "skipped" + ? "environment.managed_stock_skipped" + : "environment.managed_stock_tracking_initialized"; + await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, { + companyId, + actorType: "system", + actorId: "managed-environment-reconciler", + action, + entityType: "environment", + entityId: reconciliation.environment.id, + details: { + provider: input.provider, + reconciliationAction: reconciliation.action, + stockStatus: reconciliation.stockStatus, + updateAvailable: reconciliation.updateAvailable, + stockHash: reconciliation.stockHash, + providerReactivated, + }, + }))); } - throw new Error("Failed to ensure managed sandbox environment"); + return reconciliation; }; /** @@ -359,21 +709,95 @@ export function environmentService(db: Db) { * managed row for this provider. */ const archiveManagedSandboxEnvironment = async ( - input: { provider: string }, + input: { provider: string; companyId?: string }, ): Promise => { - const existing = await findManagedSandboxRow(); - if (!existing || existing.status !== "active") return null; - const rowProvider = (existing.metadata as Record | null) - ?.managedSandboxProvider; - if (rowProvider !== input.provider) return null; - const archived = await db - .update(environments) - .set({ status: "archived", updatedAt: new Date() }) - // Guarded on status so a concurrent re-activation is not clobbered. - .where(and(eq(environments.id, existing.id), eq(environments.status, "active"))) - .returning() - .then((rows) => rows[0] ?? null); - return archived ? toEnvironment(archived) : null; + let activityCompanyIds: string[] = []; + const archived = await db.transaction(async (tx) => { + const existing = await tx + .select() + .from(environments) + .where(eq(environments.driver, "sandbox")) + .for("update") + .then( + (rows) => rows.find( + (row) => (row.metadata as Record | null)?.managedByPaperclip === true, + ) ?? null, + ); + if (!existing || existing.status !== "active") return null; + const rowProvider = (existing.metadata as Record | null) + ?.managedSandboxProvider; + if (rowProvider !== input.provider) return null; + + const companyIds = input.companyId + ? [input.companyId] + : await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id)); + activityCompanyIds = companyIds; + const bindings = companyIds.length > 0 + ? await tx + .select() + .from(builtInManagedResources) + .where(and( + inArray(builtInManagedResources.companyId, companyIds), + eq(builtInManagedResources.bundleKey, MANAGED_ENVIRONMENT_BUNDLE_KEY), + eq(builtInManagedResources.resourceKind, MANAGED_ENVIRONMENT_RESOURCE_KIND), + eq(builtInManagedResources.resourceKey, MANAGED_ENVIRONMENT_RESOURCE_KEY), + eq(builtInManagedResources.resourceId, existing.id), + )) + : []; + const archiveToken = randomUUID(); + const archivedMetadata = { + ...(existing.metadata ?? {}), + [MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_METADATA_KEY]: archiveToken, + }; + const archived = await tx + .update(environments) + .set({ + status: "archived", + metadata: archivedMetadata, + updatedAt: new Date(), + }) + .where(and(eq(environments.id, existing.id), eq(environments.status, "active"))) + .returning() + .then((rows) => rows[0] ?? null); + if (!archived) return null; + + // Archival is a Paperclip-owned availability transition. Record only + // that status change in each installed baseline. Deriving the new hash + // from defaultsJson keeps operator-modified row fields out of stock. + for (const binding of bindings) { + const archivedStock = { + ...managedEnvironmentBaselineDefaults(binding.defaultsJson), + status: "archived", + }; + await tx + .update(builtInManagedResources) + .set({ + stockHash: stockHash(archivedStock), + defaultsJson: { + ...archivedStock, + [MANAGED_ENVIRONMENT_ARCHIVE_TOKEN_DEFAULTS_KEY]: archiveToken, + }, + updatedAt: new Date(), + }) + .where(and( + eq(builtInManagedResources.id, binding.id), + eq(builtInManagedResources.resourceId, existing.id), + )); + } + return toEnvironment(archived); + }); + if (archived) { + await Promise.all(activityCompanyIds.map((companyId) => logActivity(db, { + companyId, + actorType: "system", + actorId: "managed-environment-reconciler", + action: "environment.managed_provider_unavailable_archived", + entityType: "environment", + entityId: archived.id, + details: { provider: input.provider }, + }))); + } + return archived; }; return { @@ -488,8 +912,8 @@ export function environmentService(db: Db) { * an instance, configured from instance/operator-supplied config. A thin * wrapper over `ensureManagedSandboxEnvironment` that pins the provider to * "kubernetes" and stamps the legacy marker `findKubernetesEnvironment` - * keys on. On subsequent calls the config is refreshed (so operators can - * update egress/runtimeClass via gitops without recreating the row). + * keys on. On subsequent calls stock-controlled config advances in place; + * operator modifications remain untouched for explicit review. */ ensureKubernetesEnvironment: async ( companyIdOrConfig: string | KubernetesEnvironmentConfigInput, @@ -497,12 +921,13 @@ export function environmentService(db: Db) { ): Promise => { const config = resolveKubernetesConfig(companyIdOrConfig, maybeConfig); return ensureManagedSandboxEnvironment({ + companyId: typeof companyIdOrConfig === "string" ? companyIdOrConfig : undefined, name: DEFAULT_KUBERNETES_ENVIRONMENT_NAME, description: DEFAULT_KUBERNETES_ENVIRONMENT_DESCRIPTION, provider: KUBERNETES_PROVIDER_KEY, config, extraMetadata: { [KUBERNETES_MANAGED_MARKER]: true }, - }); + }).then((result) => result.environment); }, /** @@ -570,6 +995,7 @@ export function environmentService(db: Db) { patch: UpdateEnvironment, options?: { db?: EnvironmentWriteDb }, ): Promise => { + const writeDb = options?.db ?? db; const values: Partial = { updatedAt: new Date(), }; @@ -581,9 +1007,18 @@ export function environmentService(db: Db) { if ("envVars" in patch && patch.envVars !== undefined) { values.envVars = (patch.envVars ?? {}) as Record; } - if (patch.metadata !== undefined) values.metadata = patch.metadata ?? null; + if (patch.metadata !== undefined) { + values.metadata = withoutManagedEnvironmentArchiveToken(patch.metadata ?? null); + } else if (patch.status !== undefined) { + const existingMetadata = await writeDb + .select({ metadata: environments.metadata }) + .from(environments) + .where(eq(environments.id, id)) + .then((rows) => rows[0]?.metadata ?? null); + values.metadata = withoutManagedEnvironmentArchiveToken(existingMetadata); + } - const row = await (options?.db ?? db) + const row = await writeDb .update(environments) .set(values) .where(eq(environments.id, id)) diff --git a/server/src/services/managed-environments.test.ts b/server/src/services/managed-environments.test.ts index 4857c26a8b6..11b4d4bb8b9 100644 --- a/server/src/services/managed-environments.test.ts +++ b/server/src/services/managed-environments.test.ts @@ -40,6 +40,19 @@ function environmentRow(overrides: Record = {}) { }; } +function reconciliationResult( + overrides: Record = {}, +) { + return { + environment: environmentRow(), + action: "added" as const, + stockStatus: "missing" as const, + updateAvailable: false, + stockHash: "sha256:stock", + ...overrides, + }; +} + function readyDriverResolver(status = "ready") { return vi.fn(async () => ({ plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status }, @@ -83,7 +96,7 @@ type EnvironmentsSeam = NonNullable = {}): EnvironmentsSeam { return { ensureManagedSandboxEnvironment: - overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(environmentRow()), + overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()), archiveManagedSandboxEnvironment: overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null), }; @@ -109,7 +122,7 @@ describe("applyManagedEnvironments", () => { it("ensures each declared environment through the provider-agnostic service call", async () => { const ensureManagedSandboxEnvironment = vi .fn() - .mockResolvedValue(environmentRow()); + .mockResolvedValue(reconciliationResult()); const config = parsedConfig({ environments: [ { @@ -130,7 +143,8 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver, }); - expect(result).toEqual({ ensured: 1, failed: 0 }); + expect(result).toMatchObject({ ensured: 1, failed: 0, added: 1, skipped: 0 }); + expect(result).toMatchObject({ removed: 0, backedUp: 0 }); expect(resolveSandboxProviderDriver).toHaveBeenCalledWith({ db: noDb, driverKey: "daytona" }); expect(workerManager.isRunning).toHaveBeenCalledWith("plugin-1"); expect(ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1); @@ -139,6 +153,7 @@ describe("applyManagedEnvironments", () => { description: "Managed Daytona sandbox.", provider: "daytona", config: { target: "us" }, + stockVersion: "2026.720.0", }); // The frozen parsed config must not leak into the service (the row's // config is mutated downstream when the provider key is forced in). @@ -149,7 +164,7 @@ describe("applyManagedEnvironments", () => { it("waits for the bundled-plugin startup pass before ensuring anything", async () => { const ensureManagedSandboxEnvironment = vi .fn() - .mockResolvedValue(environmentRow()); + .mockResolvedValue(reconciliationResult()); const config = parsedConfig({ environments: [{ name: "Daytona", provider: "daytona" }], }); @@ -172,10 +187,42 @@ describe("applyManagedEnvironments", () => { expect(ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); releasePlugins(); - expect(await pending).toEqual({ ensured: 1, failed: 0 }); + expect(await pending).toMatchObject({ ensured: 1, failed: 0, added: 1 }); expect(ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1); }); + it("reports operator-modified rows as skipped with a stock update still available", async () => { + const environments = environmentsSeam({ + ensureManagedSandboxEnvironment: vi.fn().mockResolvedValue(reconciliationResult({ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + })), + }); + const result = await applyManagedEnvironments(noDb, parsedConfig({ + environments: [{ name: "Daytona", provider: "daytona" }], + }), { + env: {}, + workerManager: runningWorkerManager(), + environments, + resolveSandboxProviderDriver: readyDriverResolver(), + }); + + expect(result).toMatchObject({ + ensured: 0, + failed: 0, + added: 0, + updated: 0, + unchanged: 0, + skipped: 1, + outcomes: [{ + action: "skipped", + stockStatus: "operator_modified", + updateAvailable: true, + }], + }); + }); + it("skips an entry whose provider plugin is missing and archives its stale row", async () => { const environments = environmentsSeam({ archiveManagedSandboxEnvironment: vi.fn().mockResolvedValue(environmentRow({ status: "archived" })), @@ -191,7 +238,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: vi.fn(async () => null), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({ provider: "daytona", @@ -211,7 +258,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: readyDriverResolver("disabled"), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({ provider: "daytona", @@ -232,7 +279,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: readyDriverResolver(), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(workerManager.isRunning).toHaveBeenCalledWith("plugin-1"); expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({ @@ -263,7 +310,7 @@ describe("applyManagedEnvironments", () => { }); // The boot pass itself stays degraded: archived, counted failed. - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({ provider: "daytona", }); @@ -280,6 +327,7 @@ describe("applyManagedEnvironments", () => { description: undefined, provider: "daytona", config: { target: "us" }, + stockVersion: "2026.720.0", }); expect(handle.off).toHaveBeenCalledTimes(1); }); @@ -308,7 +356,7 @@ describe("applyManagedEnvironments", () => { }); await tick(); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1); }); @@ -371,7 +419,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: readyDriverResolver(), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled(); }); @@ -390,7 +438,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: vi.fn(async () => null), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledTimes(1); }); @@ -409,7 +457,7 @@ describe("applyManagedEnvironments", () => { resolveSandboxProviderDriver: readyDriverResolver(), }); - expect(result).toEqual({ ensured: 0, failed: 1 }); + expect(result).toMatchObject({ ensured: 0, failed: 1 }); expect(environments.archiveManagedSandboxEnvironment).not.toHaveBeenCalled(); }); }); diff --git a/server/src/services/managed-environments.ts b/server/src/services/managed-environments.ts index fbdf2c69851..8efb75dea72 100644 --- a/server/src/services/managed-environments.ts +++ b/server/src/services/managed-environments.ts @@ -48,7 +48,9 @@ import type { Db } from "@paperclipai/db"; import { logger } from "../middleware/logger.js"; import { environmentService } from "./environments.js"; +import type { ManagedSandboxEnvironmentReconcileAction } from "./environments.js"; import type { ManagedInstanceConfig } from "./managed-config.js"; +import type { ManagedResourceStockStatus } from "./managed-resource-drift.js"; import { parseExecutionPolicyBootstrapEnv } from "./execution-policy-bootstrap.js"; import { resolvePluginSandboxProviderDriverByKey } from "./plugin-environment-driver.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; @@ -93,6 +95,29 @@ export interface ApplyManagedEnvironmentsOptions { }) => Promise<{ plugin: { id: string; pluginKey: string; status: string } } | null>; } +export interface ManagedEnvironmentReconciliationOutcome { + environmentId: string; + name: string; + provider: string; + action: ManagedSandboxEnvironmentReconcileAction; + stockStatus: ManagedResourceStockStatus; + updateAvailable: boolean; +} + +export interface ApplyManagedEnvironmentsResult { + ensured: number; + failed: number; + added: number; + updated: number; + unchanged: number; + skipped: number; + /** Managed reconciliation never removes rows in preserve-only mode. */ + removed: number; + /** Preserve-only reconciliation never creates replacement backups. */ + backedUp: number; + outcomes: ManagedEnvironmentReconciliationOutcome[]; +} + /** * Ensure every environment declared in the managed-config document. Returns * null when there is nothing to do (self-hosted, or no `environments` @@ -103,7 +128,7 @@ export async function applyManagedEnvironments( db: Db, managedConfig: ManagedInstanceConfig | null, opts: ApplyManagedEnvironmentsOptions = {}, -): Promise<{ ensured: number; failed: number } | null> { +): Promise { if (!managedConfig || managedConfig.environments.length === 0) return null; // The forced-execution-mode bootstrap (`PAPERCLIP_EXECUTION_MODE=kubernetes`) @@ -155,10 +180,18 @@ export async function applyManagedEnvironments( description: spec.description, provider: spec.provider, config: { ...spec.config }, + stockVersion: managedConfig.catalogVersion, }) - .then((environment) => { + .then((result) => { logger.info( - { environmentId: environment.id, name: spec.name, provider: spec.provider }, + { + environmentId: result.environment.id, + name: spec.name, + provider: spec.provider, + action: result.action, + stockStatus: result.stockStatus, + updateAvailable: result.updateAvailable, + }, "managed sandbox environment reactivated after provider worker recovery", ); }) @@ -175,6 +208,11 @@ export async function applyManagedEnvironments( let ensured = 0; let failed = 0; + let added = 0; + let updated = 0; + let unchanged = 0; + let skipped = 0; + const outcomes: ManagedEnvironmentReconciliationOutcome[] = []; for (const spec of managedConfig.environments) { try { const resolved = await resolveDriver({ db, driverKey: spec.provider }); @@ -216,17 +254,44 @@ export async function applyManagedEnvironments( } continue; } - const environment = await environments.ensureManagedSandboxEnvironment({ + const reconciliation = await environments.ensureManagedSandboxEnvironment({ name: spec.name, description: spec.description, provider: spec.provider, config: { ...spec.config }, + stockVersion: managedConfig.catalogVersion, }); - ensured += 1; - logger.info( - { environmentId: environment.id, name: environment.name, provider: spec.provider }, - "managed sandbox environment ensured", - ); + if (reconciliation.action === "skipped") skipped += 1; + else { + ensured += 1; + if (reconciliation.action === "added") added += 1; + else if (reconciliation.action === "updated") updated += 1; + else unchanged += 1; + } + outcomes.push({ + environmentId: reconciliation.environment.id, + name: reconciliation.environment.name, + provider: spec.provider, + action: reconciliation.action, + stockStatus: reconciliation.stockStatus, + updateAvailable: reconciliation.updateAvailable, + }); + const logContext = { + environmentId: reconciliation.environment.id, + name: reconciliation.environment.name, + provider: spec.provider, + action: reconciliation.action, + stockStatus: reconciliation.stockStatus, + updateAvailable: reconciliation.updateAvailable, + }; + if (reconciliation.action === "skipped") { + logger.warn( + logContext, + "managed sandbox environment has operator modifications; preserving row and skipping stock update", + ); + } else { + logger.info(logContext, "managed sandbox environment reconciled"); + } } catch (err) { failed += 1; logger.error( @@ -235,5 +300,15 @@ export async function applyManagedEnvironments( ); } } - return { ensured, failed }; + return { + ensured, + failed, + added, + updated, + unchanged, + skipped, + removed: 0, + backedUp: 0, + outcomes, + }; } diff --git a/server/src/services/managed-resource-drift.test.ts b/server/src/services/managed-resource-drift.test.ts new file mode 100644 index 00000000000..5c1408748d3 --- /dev/null +++ b/server/src/services/managed-resource-drift.test.ts @@ -0,0 +1,43 @@ +import { describe, expect, it } from "vitest"; +import { resourceStatus, stockHash } from "./managed-resource-drift.js"; + +describe("managed resource drift", () => { + it("uses deterministic object-key ordering for stock hashes", () => { + expect(stockHash({ a: 1, b: { c: 2 } })).toBe( + stockHash({ b: { c: 2 }, a: 1 }), + ); + }); + + it.each([ + { + expected: "missing", + resourceId: null, + currentHash: null, + bindingStockHash: null, + latestStockHash: "latest", + }, + { + expected: "stock_current", + resourceId: "resource-1", + currentHash: "latest", + bindingStockHash: "previous", + latestStockHash: "latest", + }, + { + expected: "stock_update_available", + resourceId: "resource-1", + currentHash: "previous", + bindingStockHash: "previous", + latestStockHash: "latest", + }, + { + expected: "operator_modified", + resourceId: "resource-1", + currentHash: "operator", + bindingStockHash: "previous", + latestStockHash: "latest", + }, + ] as const)("classifies $expected", ({ expected, ...input }) => { + expect(resourceStatus(input)).toBe(expected); + }); +}); diff --git a/server/src/services/managed-resource-drift.ts b/server/src/services/managed-resource-drift.ts new file mode 100644 index 00000000000..addce116bc5 --- /dev/null +++ b/server/src/services/managed-resource-drift.ts @@ -0,0 +1,36 @@ +import { createHash } from "node:crypto"; + +export type ManagedResourceStockStatus = + | "missing" + | "stock_current" + | "stock_update_available" + | "operator_modified"; + +export function stableJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`; + if (value && typeof value === "object") { + return `{${Object.entries(value as Record) + .sort(([left], [right]) => left.localeCompare(right)) + .map(([key, entry]) => `${JSON.stringify(key)}:${stableJson(entry)}`) + .join(",")}}`; + } + return JSON.stringify(value); +} + +export function stockHash(value: unknown) { + return `sha256:${createHash("sha256").update(stableJson(value)).digest("hex")}`; +} + +export function resourceStatus(input: { + resourceId: string | null; + currentHash: string | null; + bindingStockHash: string | null; + latestStockHash: string; +}): ManagedResourceStockStatus { + if (!input.resourceId || !input.currentHash) return "missing"; + if (input.currentHash === input.latestStockHash) return "stock_current"; + if (input.bindingStockHash && input.currentHash === input.bindingStockHash) { + return "stock_update_available"; + } + return "operator_modified"; +} From 5da382fd59222819cff110f85ca738a74fb335d1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 00:42:08 -0500 Subject: [PATCH 013/311] feat(skills): require explicit merge modes (#10978) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents can select company skills and synchronize them to adapter runtimes > - The skill sync API replaced the complete selection without an explicit destructive choice > - Company package import also replaced conflicting skills by default > - These defaults could remove operator edits during setup and import reruns > - This pull request adds explicit assignment merge modes and safe package conflict handling > - The benefit is that reruns preserve operator work unless the caller explicitly requests replacement ## Linked Issues or Issue Description **What existing behavior does this improve?** This change improves agent skill synchronization and company package import. **Subsystem affected** This is a cross-cutting change across the shared contracts, server, CLI, and UI. **Current behavior** Agent skill synchronization replaces the full desired skill set from a modeless request. Package import replaces a conflicting skill when the caller does not select a conflict mode. **Proposed behavior** Agent skill synchronization requires `add`, `remove`, or `replace`. Package import skips conflicts by default. Each imported skill reports whether it was created, renamed, replaced, or skipped. **Reason and benefit** Setup and import reruns must preserve operator edits by default. Explicit destructive modes make data loss less likely and make each outcome inspectable. **Breaking changes** Callers of the agent skill sync API must now send `mode`. Callers that need the former behavior must send `replace`. Package import now uses `skip` when `onConflict` is absent. ## What Changed - Added required `add`, `remove`, and `replace` modes to the shared agent skill sync contract. - Added actionable `422` validation for missing or invalid modes. - Updated first-party UI and CLI callers with explicit modes. - Changed package skill conflict handling to use `skip` by default. - Kept plugin-owned and built-in stock skill imports on explicit `replace`. - Added created, renamed, replaced, and skipped results to company imports. - Added regression coverage for merge modes and package conflict outcomes. ## Verification - `pnpm check:token-gates` - `pnpm -r typecheck` - `pnpm build` - `pnpm test:run:serialized` (128 suites passed) - `pnpm --filter @paperclipai/skills-catalog test` (20 tests passed) - Focused agent skill route, company skill service, portability, CLI, and UI tests passed. - GitHub CI passed build, typecheck, canary, all general and serialized test shards, all browser shards, policy, security, and final verification on commit `2cfbb3e4c5`. - Greptile reviewed the latest commit at 5/5 with zero unresolved threads. ## Risks - This change intentionally rejects modeless agent skill sync requests. - The safe package default can leave an existing skill unchanged where the old default overwrote it. - All first-party callers now select a mode. Regression tests cover each outcome. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected β€” check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI `gpt-5.6-sol` through Codex. The runtime used agentic reasoning, tool use, code execution, and repository editing. The runtime did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- cli/src/__tests__/agent-lifecycle.test.ts | 10 +- cli/src/__tests__/company.test.ts | 13 +++ cli/src/__tests__/skills.test.ts | 4 +- cli/src/commands/client/agent.ts | 10 +- cli/src/commands/client/company.ts | 21 ++++ cli/src/commands/client/skills.ts | 42 ++++--- doc/CLI.md | 17 +-- docs/cli/control-plane-commands.md | 2 +- packages/shared/src/index.ts | 2 + packages/shared/src/types/adapter-skills.ts | 3 + .../shared/src/types/company-portability.ts | 9 ++ packages/shared/src/types/index.ts | 1 + .../shared/src/validators/adapter-skills.ts | 7 ++ packages/shared/src/validators/index.ts | 1 + .../reflection-coach/SKILL.md | 2 +- .../src/__tests__/agent-skills-routes.test.ts | 103 +++++++++++++++++- .../src/__tests__/company-portability.test.ts | 29 ++++- .../__tests__/company-skills-service.test.ts | 66 +++++++++++ server/src/routes/agents.ts | 86 ++++++++++++--- server/src/routes/openapi.ts | 2 +- server/src/services/company-portability.ts | 13 ++- server/src/services/company-skills.ts | 44 +++++--- skills/paperclip/SKILL.md | 2 +- skills/paperclip/references/company-skills.md | 7 ++ ui/src/api/agents.ts | 9 +- .../skill-studio/AgentsUsingSkillDialog.tsx | 2 +- ui/src/pages/CompanyImport.test.tsx | 1 + ui/src/pages/CompanyImport.tsx | 21 ++++ ui/src/pages/CompanySkills.tsx | 2 +- ui/src/pages/agent-skills/AgentSkillsTab.tsx | 2 +- ui/storybook/stories/team-catalog.stories.tsx | 1 + 31 files changed, 459 insertions(+), 75 deletions(-) diff --git a/cli/src/__tests__/agent-lifecycle.test.ts b/cli/src/__tests__/agent-lifecycle.test.ts index 4e2917ce83e..41817823db9 100644 --- a/cli/src/__tests__/agent-lifecycle.test.ts +++ b/cli/src/__tests__/agent-lifecycle.test.ts @@ -83,7 +83,15 @@ describe("agent lifecycle commands", () => { await run(["agent", "runtime-state:reset-session", AGENT_ID, "--task-key", "task-1"]); await run(["agent", "task-sessions", AGENT_ID]); await run(["agent", "skills", AGENT_ID]); - await run(["agent", "skills:sync", AGENT_ID, "--desired-skills", "paperclip,github"]); + await run([ + "agent", + "skills:sync", + AGENT_ID, + "--desired-skills", + "paperclip,github", + "--mode", + "replace", + ]); await run(["agent", "instructions-path:update", AGENT_ID, "--payload-json", JSON.stringify({ path: "/tmp/AGENTS.md" })]); await run(["agent", "instructions-bundle", AGENT_ID]); await run(["agent", "instructions-bundle:update", AGENT_ID, "--payload-json", JSON.stringify({ mode: "managed" })]); diff --git a/cli/src/__tests__/company.test.ts b/cli/src/__tests__/company.test.ts index 1532047f6f8..a548c8821a4 100644 --- a/cli/src/__tests__/company.test.ts +++ b/cli/src/__tests__/company.test.ts @@ -504,6 +504,17 @@ describe("renderCompanyImportResult", () => { { slug: "cto", id: "agent-2", action: "updated", name: "CTO", reason: "replace strategy" }, { slug: "ops", id: null, action: "skipped", name: "Ops", reason: "skip strategy" }, ], + skills: [ + { + originalKey: "company/source/review", + originalSlug: "review", + key: "company/target/review-2", + slug: "review-2", + id: "skill-1", + action: "renamed", + reason: "rename strategy", + }, + ], projects: [ { slug: "app", id: "project-1", action: "created", name: "App", reason: null }, { slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" }, @@ -525,8 +536,10 @@ describe("renderCompanyImportResult", () => { expect(rendered).toContain("Company"); expect(rendered).toContain("https://paperclip.example/PAP/dashboard"); expect(rendered).toContain("3 agents total (1 created, 1 updated, 1 skipped)"); + expect(rendered).toContain("1 skill total (1 renamed)"); expect(rendered).toContain("3 projects total (1 created, 1 updated, 1 skipped)"); expect(rendered).toContain("Agent results"); + expect(rendered).toContain("Skill results"); expect(rendered).toContain("Project results"); expect(rendered).toContain("Using claude-local adapter"); expect(rendered).toContain("Review API keys"); diff --git a/cli/src/__tests__/skills.test.ts b/cli/src/__tests__/skills.test.ts index 038b9812f1b..80d0dd3c791 100644 --- a/cli/src/__tests__/skills.test.ts +++ b/cli/src/__tests__/skills.test.ts @@ -479,6 +479,8 @@ describe("skills CLI commands", () => { "review-prs", "--skill", "paperclip/qa", + "--mode", + "add", "--company-id", "company-1", "--api-base", @@ -498,7 +500,7 @@ describe("skills CLI commands", () => { "http://paperclip.test/api/agents/agent-1/skills/sync", expect.objectContaining({ method: "POST", - body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"] }), + body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"], mode: "add" }), }), ); expect(JSON.parse(String(logSpy.mock.calls[0]?.[0]))).toEqual(snapshot); diff --git a/cli/src/commands/client/agent.ts b/cli/src/commands/client/agent.ts index 8144352c491..6d6677e4244 100644 --- a/cli/src/commands/client/agent.ts +++ b/cli/src/commands/client/agent.ts @@ -71,6 +71,7 @@ interface AgentResetSessionOptions extends BaseClientOptions { interface AgentSkillsSyncOptions extends BaseClientOptions { desiredSkills: string; + mode: string; } interface AgentInstructionsFileOptions extends BaseClientOptions { @@ -589,10 +590,17 @@ export function registerAgentCommands(program: Command): void { .description("Sync desired skills onto an agent") .argument("", "Agent ID") .requiredOption("--desired-skills ", "Desired skill names") + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentId: string, opts: AgentSkillsSyncOptions) => { try { const ctx = resolveCommandContext(opts); - const payload = agentSkillSyncSchema.parse({ desiredSkills: parseCsv(opts.desiredSkills) }); + const payload = agentSkillSyncSchema.parse({ + desiredSkills: parseCsv(opts.desiredSkills), + mode: opts.mode, + }); const result = await ctx.api.post(apiPath`/api/agents/${agentId}/skills/sync`, payload); printOutput(result, { json: ctx.json }); } catch (err) { diff --git a/cli/src/commands/client/company.ts b/cli/src/commands/client/company.ts index bd0d162a502..72864f6482c 100644 --- a/cli/src/commands/client/company.ts +++ b/cli/src/commands/client/company.ts @@ -548,6 +548,16 @@ function summarizeImportAgentResults(agents: CompanyPortabilityImportResult["age return `${agents.length} ${pluralize(agents.length, "agent")} total (${parts.join(", ")})`; } +function summarizeImportSkillResults(skills: CompanyPortabilityImportResult["skills"]): string { + if (skills.length === 0) return "0 skills changed"; + const actions = ["created", "renamed", "replaced", "skipped"] as const; + const parts = actions.flatMap((action) => { + const count = skills.filter((skill) => skill.action === action).length; + return count > 0 ? [`${count} ${action}`] : []; + }); + return `${skills.length} ${pluralize(skills.length, "skill")} total (${parts.join(", ")})`; +} + function summarizeImportProjectResults(projects: CompanyPortabilityImportResult["projects"]): string { if (projects.length === 0) return "0 projects changed"; const created = projects.filter((project) => project.action === "created").length; @@ -681,10 +691,12 @@ export function renderCompanyImportResult( result: CompanyPortabilityImportResult, meta: { targetLabel: string; companyUrl?: string; infoMessages?: string[] }, ): string { + const skills = result.skills ?? []; const lines: string[] = [ `${pc.bold("Target")} ${meta.targetLabel}`, `${pc.bold("Company")} ${result.company.name} (${actionChip(result.company.action)})`, `${pc.bold("Agents")} ${summarizeImportAgentResults(result.agents)}`, + `${pc.bold("Skills")} ${summarizeImportSkillResults(skills)}`, `${pc.bold("Projects")} ${summarizeImportProjectResults(result.projects)}`, ]; @@ -701,6 +713,15 @@ export function renderCompanyImportResult( reason: agent.reason, })), ); + appendPreviewExamples( + lines, + "Skill results", + skills.map((skill) => ({ + action: skill.action, + label: `${skill.originalSlug} -> ${skill.slug}`, + reason: skill.reason, + })), + ); appendPreviewExamples( lines, "Project results", diff --git a/cli/src/commands/client/skills.ts b/cli/src/commands/client/skills.ts index c97ebd18df1..f24316c9c49 100644 --- a/cli/src/commands/client/skills.ts +++ b/cli/src/commands/client/skills.ts @@ -1,17 +1,19 @@ import { Command } from "commander"; -import type { - Agent, - AgentSkillSnapshot, - CatalogSkill, - CompanySkill, - CompanySkillAuditResult, - CompanySkillDetail, - CompanySkillFileDetail, - CompanySkillImportResult, - CompanySkillInstallCatalogResult, - CompanySkillListItem, - CompanySkillProjectScanResult, - CompanySkillUpdateStatus, +import { + agentSkillAssignmentModeSchema, + type AgentSkillAssignmentMode, + type Agent, + type AgentSkillSnapshot, + type CatalogSkill, + type CompanySkill, + type CompanySkillAuditResult, + type CompanySkillDetail, + type CompanySkillFileDetail, + type CompanySkillImportResult, + type CompanySkillInstallCatalogResult, + type CompanySkillListItem, + type CompanySkillProjectScanResult, + type CompanySkillUpdateStatus, } from "@paperclipai/shared"; import { readFile } from "node:fs/promises"; import { stdin as input, stdout as output } from "node:process"; @@ -69,6 +71,7 @@ interface ConfirmedSkillOptions extends SkillsOptions { interface AgentSkillSyncOptions extends SkillsOptions { skill?: string[]; + mode: AgentSkillAssignmentMode; } type CompanySkillReferenceTarget = Pick; @@ -502,9 +505,13 @@ function registerAgentSkillCommands(skills: Command): void { addCommonClientOptions( agent .command("sync") - .description("Replace an agent's desired company skills and sync runtime state") + .description("Merge an agent's desired company skills and sync runtime state") .argument("", "Agent ID or shortname/url-key") .option("--skill ", "Desired company skill ID, key, or slug; may be repeated", collectOptionValue, [] as string[]) + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentRef: string, opts: AgentSkillSyncOptions) => { try { const desiredSkills = opts.skill ?? []; @@ -513,16 +520,17 @@ function registerAgentSkillCommands(skills: Command): void { } const ctx = resolveCommandContext(opts, { requireCompany: true }); const agentRow = await resolveAgent(ctx, agentRef); + const mode = agentSkillAssignmentModeSchema.parse(opts.mode); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills }, + { desiredSkills, mode }, ); if (ctx.json) { printOutput(snapshot, { json: true }); return; } console.log( - `Desired company skills replaced for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, + `Desired company skills updated with ${mode} mode for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, ); printAgentSkillSnapshot(snapshot, agentRow); } catch (err) { @@ -548,7 +556,7 @@ function registerAgentSkillCommands(skills: Command): void { ); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills: [] }, + { desiredSkills: [], mode: "replace" }, ); if (ctx.json) { printOutput(snapshot, { json: true }); diff --git a/doc/CLI.md b/doc/CLI.md index 5b36f799589..5c75aeb4e85 100644 --- a/doc/CLI.md +++ b/doc/CLI.md @@ -365,7 +365,7 @@ pnpm paperclipai agent runtime-state pnpm paperclipai agent runtime-state:reset-session [--task-key ] pnpm paperclipai agent task-sessions pnpm paperclipai agent skills -pnpm paperclipai agent skills:sync --desired-skills paperclip,github +pnpm paperclipai agent skills:sync --desired-skills paperclip,github --mode add pnpm paperclipai agent instructions-path:update --payload-json '{"path":"/path/to/AGENTS.md"}' pnpm paperclipai agent instructions-bundle pnpm paperclipai agent instructions-bundle:update --payload-json '{"mode":"managed"}' @@ -465,9 +465,10 @@ By default the command creates a `todo` issue assigned to the target agent and w 1. **Company install** β€” adds or updates a row in `company_skills` for the whole company. This is what `skills install`, `skills import`, `skills create`, and `skills scan-projects` do. -2. **Agent attach** β€” replaces an agent's *desired* company skill set - (`skills agent sync`/`clear`). This is a desired-state operation on the - agent's adapter config; it does not change the company library. +2. **Agent attach** β€” merges an agent's *desired* company skill set with an + explicit `add`, `remove`, or `replace` mode (`skills agent sync`/`clear`). + This is a desired-state operation on the agent's adapter config; it does not + change the company library. 3. **Adapter runtime sync** β€” the adapter reconciles the desired skill set with files on disk and reports an `AgentSkillSnapshot` (`skills agent list`). `skills agent sync` triggers this automatically after updating desired state. @@ -564,12 +565,14 @@ maintenance loop for catalog-installed skills: ```sh pnpm paperclipai skills agent list --company-id -pnpm paperclipai skills agent sync --skill [--skill ...] --company-id +pnpm paperclipai skills agent sync --skill [--skill ...] --mode --company-id pnpm paperclipai skills agent clear --yes --company-id ``` -`skills agent sync` replaces the agent's non-required desired skill set (it is -not additive) and returns the resulting adapter `AgentSkillSnapshot`. +`skills agent sync` requires a merge mode and returns the resulting adapter +`AgentSkillSnapshot`. `add` preserves all unnamed assignments, `remove` deletes +only named assignments, and `replace` destructively overwrites the complete +non-required desired skill set. `skills agent clear` sends an empty desired list. Required Paperclip skills are still enforced by the server in both cases. diff --git a/docs/cli/control-plane-commands.md b/docs/cli/control-plane-commands.md index 406b3190bd7..70adf99ad24 100644 --- a/docs/cli/control-plane-commands.md +++ b/docs/cli/control-plane-commands.md @@ -91,7 +91,7 @@ pnpm paperclipai skills import ./skills/my-skill --company-id pnpm paperclipai skills import owner/repo/path/to/skill --company-id # Attach desired company skills to an agent after install/import -pnpm paperclipai skills agent sync --skill github-pr-workflow --company-id +pnpm paperclipai skills agent sync --skill github-pr-workflow --mode add --company-id ``` ## Approval Commands diff --git a/packages/shared/src/index.ts b/packages/shared/src/index.ts index 2b3d6b918bb..7821153a275 100644 --- a/packages/shared/src/index.ts +++ b/packages/shared/src/index.ts @@ -764,6 +764,7 @@ export type { CatalogTeamInstallResult, InstalledCatalogTeam, AgentSkillSyncMode, + AgentSkillAssignmentMode, AgentSkillState, AgentSkillOrigin, AgentDesiredSkillEntry, @@ -1596,6 +1597,7 @@ export { type ProbeEnvironmentConfig, agentSkillStateSchema, agentSkillSyncModeSchema, + agentSkillAssignmentModeSchema, agentDesiredSkillEntrySchema, agentDesiredSkillSelectionSchema, agentSkillEntrySchema, diff --git a/packages/shared/src/types/adapter-skills.ts b/packages/shared/src/types/adapter-skills.ts index 998dda9a54d..ac77f7e013e 100644 --- a/packages/shared/src/types/adapter-skills.ts +++ b/packages/shared/src/types/adapter-skills.ts @@ -1,5 +1,7 @@ export type AgentSkillSyncMode = "unsupported" | "persistent" | "ephemeral"; +export type AgentSkillAssignmentMode = "add" | "remove" | "replace"; + export type AgentSkillState = | "available" | "configured" @@ -46,5 +48,6 @@ export interface AgentSkillSnapshot { } export interface AgentSkillSyncRequest { + mode: AgentSkillAssignmentMode; desiredSkills: Array; } diff --git a/packages/shared/src/types/company-portability.ts b/packages/shared/src/types/company-portability.ts index b43afdda2e3..3cc0eb4a6f0 100644 --- a/packages/shared/src/types/company-portability.ts +++ b/packages/shared/src/types/company-portability.ts @@ -402,6 +402,15 @@ export interface CompanyPortabilityImportResult { name: string; reason: string | null; }[]; + skills: { + originalKey: string; + originalSlug: string; + key: string; + slug: string; + id: string; + action: "created" | "renamed" | "replaced" | "skipped"; + reason: string | null; + }[]; projects: { slug: string; id: string | null; diff --git a/packages/shared/src/types/index.ts b/packages/shared/src/types/index.ts index 6027c15d39e..e23ae3daeb7 100644 --- a/packages/shared/src/types/index.ts +++ b/packages/shared/src/types/index.ts @@ -250,6 +250,7 @@ export type { } from "./teams-catalog.js"; export type { AgentSkillSyncMode, + AgentSkillAssignmentMode, AgentSkillState, AgentSkillOrigin, AgentDesiredSkillEntry, diff --git a/packages/shared/src/validators/adapter-skills.ts b/packages/shared/src/validators/adapter-skills.ts index 96c0c4998d5..d0befc91103 100644 --- a/packages/shared/src/validators/adapter-skills.ts +++ b/packages/shared/src/validators/adapter-skills.ts @@ -21,6 +21,12 @@ export const agentSkillSyncModeSchema = z.enum([ "ephemeral", ]); +export const agentSkillAssignmentModeSchema = z.enum([ + "add", + "remove", + "replace", +]); + export const agentDesiredSkillEntrySchema = z.object({ key: z.string().min(1), versionId: z.string().uuid().nullable(), @@ -59,6 +65,7 @@ export const agentSkillSnapshotSchema = z.object({ }); export const agentSkillSyncSchema = z.object({ + mode: agentSkillAssignmentModeSchema, desiredSkills: z.array(agentDesiredSkillSelectionSchema), }); diff --git a/packages/shared/src/validators/index.ts b/packages/shared/src/validators/index.ts index d93c6649dd6..b0923b60848 100644 --- a/packages/shared/src/validators/index.ts +++ b/packages/shared/src/validators/index.ts @@ -282,6 +282,7 @@ export { export { agentSkillStateSchema, agentSkillSyncModeSchema, + agentSkillAssignmentModeSchema, agentDesiredSkillEntrySchema, agentDesiredSkillSelectionSchema, agentSkillEntrySchema, diff --git a/packages/skills-catalog/catalog/bundled/paperclip-operations/reflection-coach/SKILL.md b/packages/skills-catalog/catalog/bundled/paperclip-operations/reflection-coach/SKILL.md index b5167838920..205a12494e4 100644 --- a/packages/skills-catalog/catalog/bundled/paperclip-operations/reflection-coach/SKILL.md +++ b/packages/skills-catalog/catalog/bundled/paperclip-operations/reflection-coach/SKILL.md @@ -176,7 +176,7 @@ Server-enforced mutation target keys: When the interaction resolves **accepted**, apply the change in a *separate* run: - **AGENTS.md** β€” update the target's managed instruction file exactly as the accepted diff specified. -- **Skill** β€” install/update the skill in the company library, then `POST /api/agents//skills/sync` when the target should receive it. +- **Skill** β€” install/update the skill in the company library, then `POST /api/agents//skills/sync` with `{"mode":"add","desiredSkills":[""]}` when the target should receive it. Use `remove` only for the named assignments. Use `replace` only after explicit confirmation to overwrite the complete desired skill set. - **Tool description** β€” update the target agent's description/profile field that the accepted diff named. The server rejects Reflection Coach mutations unless the accepted `request_confirmation` was created by Reflection Coach in a previous run, has a displayed diff, and is bound to the resource by one of the target keys above. If the interaction was rejected or is still pending, apply nothing. If you were asked to apply without a reviewed diff and an accepted interaction, refuse and name the gate β€” no-same-run-apply is load-bearing. diff --git a/server/src/__tests__/agent-skills-routes.test.ts b/server/src/__tests__/agent-skills-routes.test.ts index db6df76cc17..6dd77801ed1 100644 --- a/server/src/__tests__/agent-skills-routes.test.ts +++ b/server/src/__tests__/agent-skills-routes.test.ts @@ -488,7 +488,7 @@ describe.sequential("agent skill routes", () => { await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclip"] }), + .send({ desiredSkills: ["paperclip"], mode: "replace" }), ); expect(syncRes.status, JSON.stringify(syncRes.body)).toBe(200); const syncCall = mockSecretService.resolveAdapterConfigForRuntime.mock.calls.at(-1); @@ -588,7 +588,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclip"] })); + .send({ desiredSkills: ["paperclip"], mode: "replace" })); expect(res.status, JSON.stringify(res.body)).toBe(200); expect(mockAgentService.update).toHaveBeenCalledWith( @@ -615,12 +615,102 @@ describe.sequential("agent skill routes", () => { ); }); + it("requires an explicit actionable merge mode for skill sync", async () => { + const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) + .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") + .send({ desiredSkills: ["paperclip"] })); + + expect(res.status, JSON.stringify(res.body)).toBe(422); + expect(res.body.error).toContain('"add", "remove", or "replace"'); + expect(res.body.error).toContain('"replace" only to overwrite'); + expect(mockAgentService.update).not.toHaveBeenCalled(); + }); + + it("adds only named desired skills while preserving existing assignments", async () => { + mockAgentService.getById.mockResolvedValue({ + ...makeAgent("claude_local"), + adapterConfig: { + paperclipSkillSync: { desiredSkills: ["company-1/keep"] }, + }, + }); + + const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) + .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") + .send({ desiredSkills: ["paperclip"], mode: "add" })); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(mockAgentService.update).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ + adapterConfig: expect.objectContaining({ + paperclipSkillSync: { + desiredSkills: ["company-1/keep", "paperclipai/paperclip/paperclip"], + }, + }), + }), + expect.any(Object), + ); + }); + + it("removes only named desired skills while preserving other assignments", async () => { + mockAgentService.getById.mockResolvedValue({ + ...makeAgent("claude_local"), + adapterConfig: { + paperclipSkillSync: { + desiredSkills: ["company-1/keep", "paperclipai/paperclip/paperclip"], + }, + }, + }); + + const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) + .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") + .send({ desiredSkills: ["paperclip"], mode: "remove" })); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(mockAgentService.update).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ + adapterConfig: expect.objectContaining({ + paperclipSkillSync: { desiredSkills: ["company-1/keep"] }, + }), + }), + expect.any(Object), + ); + }); + + it("replaces the complete desired skill set only when explicitly requested", async () => { + mockAgentService.getById.mockResolvedValue({ + ...makeAgent("claude_local"), + adapterConfig: { + paperclipSkillSync: { desiredSkills: ["company-1/keep"] }, + }, + }); + + const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) + .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") + .send({ desiredSkills: ["paperclip"], mode: "replace" })); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(mockAgentService.update).toHaveBeenCalledWith( + expect.any(String), + expect.objectContaining({ + adapterConfig: expect.objectContaining({ + paperclipSkillSync: { + desiredSkills: ["paperclipai/paperclip/paperclip"], + }, + }), + }), + expect.any(Object), + ); + }); + it("rejects version pins while beta skills are disabled", async () => { mockAgentService.getById.mockResolvedValue(makeAgent("claude_local")); const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") .send({ + mode: "replace", desiredSkills: [{ key: "paperclipai/paperclip/paperclip", versionId: "22222222-2222-4222-8222-222222222222", @@ -640,6 +730,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") .send({ + mode: "replace", desiredSkills: [{ key: "paperclipai/paperclip/paperclip", versionId }], })); @@ -687,7 +778,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclip", "stale/removed/skill"] })); + .send({ desiredSkills: ["paperclip", "stale/removed/skill"], mode: "replace" })); expect(res.status, JSON.stringify(res.body)).toBe(200); // Stale key preserved in the persisted config alongside the resolved skill. @@ -741,7 +832,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclipai/paperclip/paperclip"] })); + .send({ desiredSkills: ["paperclipai/paperclip/paperclip"], mode: "replace" })); expect(res.status, JSON.stringify(res.body)).toBe(200); expect(mockAdapter.syncSkills).toHaveBeenCalled(); @@ -796,7 +887,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclipai/paperclip/paperclip"] })); + .send({ desiredSkills: ["paperclipai/paperclip/paperclip"], mode: "replace" })); expect(res.status, JSON.stringify(res.body)).toBe(200); expect(mockAdapter.syncSkills).toHaveBeenCalledWith( @@ -816,7 +907,7 @@ describe.sequential("agent skill routes", () => { const res = await requestApp(await createApp(), (baseUrl) => request(baseUrl) .post("/api/agents/11111111-1111-4111-8111-111111111111/skills/sync?companyId=company-1") - .send({ desiredSkills: ["paperclip"] })); + .send({ desiredSkills: ["paperclip"], mode: "replace" })); expect(res.status, JSON.stringify(res.body)).toBe(200); expect(mockAgentService.update).toHaveBeenCalledWith( diff --git a/server/src/__tests__/company-portability.test.ts b/server/src/__tests__/company-portability.test.ts index 63c3a149438..9631ddd3b7b 100644 --- a/server/src/__tests__/company-portability.test.ts +++ b/server/src/__tests__/company-portability.test.ts @@ -3005,6 +3005,18 @@ describe("company portability", () => { id: "agent-created", name: "ClaudeCoder", }); + companySkillSvc.importPackageFiles.mockResolvedValueOnce([{ + skill: { + id: "skill-imported", + key: paperclipKey, + slug: "paperclip", + }, + action: "renamed", + originalKey: "paperclip", + originalSlug: "paperclip", + requestedRefs: ["paperclip"], + reason: "Existing skill matched; renamed to paperclip-2.", + }]); const exported = await portability.exportBundle("company-1", { include: { @@ -3017,7 +3029,7 @@ describe("company portability", () => { agentSvc.list.mockResolvedValue([]); - await portability.importBundle({ + const result = await portability.importBundle({ source: { type: "inline", rootPath: exported.rootPath, @@ -3039,8 +3051,17 @@ describe("company portability", () => { const textOnlyFiles = Object.fromEntries(Object.entries(exported.files).filter(([, v]) => typeof v === "string")); expect(companySkillSvc.importPackageFiles).toHaveBeenCalledWith("company-imported", textOnlyFiles, { - onConflict: "replace", + onConflict: "rename", }); + expect(result.skills).toEqual([{ + originalKey: "paperclip", + originalSlug: "paperclip", + key: paperclipKey, + slug: "paperclip", + id: "skill-imported", + action: "renamed", + reason: "Existing skill matched; renamed to paperclip-2.", + }]); expect(agentSvc.create).toHaveBeenCalledWith("company-imported", expect.objectContaining({ adapterConfig: expect.objectContaining({ paperclipSkillSync: { @@ -3301,7 +3322,7 @@ describe("company portability", () => { "agents/cmo/AGENTS.md": expect.any(String), }), { - onConflict: "replace", + onConflict: "rename", }, ); expect(companySkillSvc.importPackageFiles).toHaveBeenCalledWith( @@ -3310,7 +3331,7 @@ describe("company portability", () => { "agents/claudecoder/AGENTS.md": expect.any(String), }), { - onConflict: "replace", + onConflict: "rename", }, ); expect(agentSvc.create).toHaveBeenCalledTimes(1); diff --git a/server/src/__tests__/company-skills-service.test.ts b/server/src/__tests__/company-skills-service.test.ts index c2c3541bfcc..1de80114abb 100644 --- a/server/src/__tests__/company-skills-service.test.ts +++ b/server/src/__tests__/company-skills-service.test.ts @@ -1605,6 +1605,72 @@ describeEmbeddedPostgres("companySkillService.list", () => { ]); }); + it("defaults package conflicts to skip and reports skip, rename, and explicit replace outcomes", async () => { + const companyId = randomUUID(); + await db.insert(companies).values({ + id: companyId, + name: "Paperclip", + issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`, + requireBoardApprovalForNewAgents: false, + }); + const original = await svc.createLocalSkill(companyId, { + name: "Conflict Skill", + slug: "conflict-skill", + markdown: "---\nname: Conflict Skill\n---\n\n# Original\n", + }); + const packageFiles = { + "skills/conflict-skill/SKILL.md": [ + "---", + "name: Imported Conflict Skill", + "slug: conflict-skill", + "description: Incoming package version", + "---", + "", + "# Imported", + "", + ].join("\n"), + }; + + const skipped = await svc.importPackageFiles(companyId, packageFiles); + expect(skipped).toEqual([ + expect.objectContaining({ + action: "skipped", + originalSlug: "conflict-skill", + skill: expect.objectContaining({ id: original.id, name: "Conflict Skill" }), + }), + ]); + await expect(svc.getById(companyId, original.id)).resolves.toMatchObject({ + name: "Conflict Skill", + markdown: expect.stringContaining("# Original"), + }); + + const renamed = await svc.importPackageFiles(companyId, packageFiles, { onConflict: "rename" }); + expect(renamed).toEqual([ + expect.objectContaining({ + action: "renamed", + originalSlug: "conflict-skill", + skill: expect.objectContaining({ + name: "Imported Conflict Skill", + slug: "conflict-skill-2", + }), + }), + ]); + expect((await svc.list(companyId)).filter((skill) => skill.slug.startsWith("conflict-skill"))).toHaveLength(2); + + const replaced = await svc.importPackageFiles(companyId, packageFiles, { onConflict: "replace" }); + expect(replaced).toEqual([ + expect.objectContaining({ + action: "replaced", + originalSlug: "conflict-skill", + skill: expect.objectContaining({ id: original.id, name: "Imported Conflict Skill" }), + }), + ]); + await expect(svc.getById(companyId, original.id)).resolves.toMatchObject({ + name: "Imported Conflict Skill", + markdown: expect.stringContaining("# Imported"), + }); + }); + it("rejects executable external package skills before persistence", async () => { const companyId = randomUUID(); await db.insert(companies).values({ diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index f8d991895f0..97754e352d8 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -1,4 +1,4 @@ -import { Router, type Request, type Response } from "express"; +import { Router, type NextFunction, type Request, type Response } from "express"; import { generateKeyPairSync, randomUUID } from "node:crypto"; import path from "node:path"; import type { Db } from "@paperclipai/db"; @@ -18,6 +18,7 @@ import { resetAgentSessionSchema, testAdapterEnvironmentSchema, type AgentDesiredSkillEntry, + type AgentSkillAssignmentMode, type AgentSkillSnapshot, type InstanceSchedulerHeartbeatAgent, upsertAgentInstructionsFileSchema, @@ -117,6 +118,35 @@ import { touchesAgentProfileChangeConsentFields, } from "../services/change-consent-gate.js"; +const AGENT_SKILL_ASSIGNMENT_MODES = ["add", "remove", "replace"] as const; + +function requireAgentSkillAssignmentMode(req: Request, _res: Response, next: NextFunction) { + if (!AGENT_SKILL_ASSIGNMENT_MODES.includes(req.body?.mode)) { + throw unprocessable( + 'Skill sync requires mode: "add", "remove", or "replace". ' + + 'Use "replace" only to overwrite the complete desired skill set.', + ); + } + next(); +} + +function mergeDesiredSkillEntries( + current: AgentDesiredSkillEntry[], + requested: AgentDesiredSkillEntry[], + mode: AgentSkillAssignmentMode, +) { + if (mode === "replace") return requested; + + const requestedKeys = new Set(requested.map((entry) => entry.key)); + if (mode === "remove") { + return current.filter((entry) => !requestedKeys.has(entry.key)); + } + + const merged = new Map(current.map((entry) => [entry.key, entry])); + for (const entry of requested) merged.set(entry.key, entry); + return Array.from(merged.values()); +} + const RUN_LOG_DEFAULT_LIMIT_BYTES = 256_000; const RUN_LOG_MAX_LIMIT_BYTES = 1024 * 1024; @@ -1625,6 +1655,7 @@ export function agentRoutes( adapterType: string, adapterConfig: Record, requestedDesiredSkills: AgentDesiredSkillEntry[] | undefined, + mode: AgentSkillAssignmentMode, options: { tolerateUnknownDesiredSkills?: boolean } = {}, ) { if (!requestedDesiredSkills) { @@ -1647,22 +1678,44 @@ export function agentRoutes( await companySkills.resolveRequestedSkillEntries(companyId, requestedDesiredSkills, { tolerateUnknownReferences: options.tolerateUnknownDesiredSkills, }); - // Runtime materialization + version selection only ever consider skills that - // actually resolve to the company library; stale keys can't be materialized. - const runtimeSkillEntries = await companySkills.listRuntimeSkillEntries(companyId, { - materializeMissing: shouldMaterializeRuntimeSkillsForAdapter(adapterType), - versionSelections: skillVersionSelectionMap(resolvedRequestedSkillEntries), - }); - const resolvedDesiredSkillEntries = resolvedRequestedSkillEntries.filter( + const requestedSkillEntries = [ + ...resolvedRequestedSkillEntries, + ...unresolvedDesiredSkillKeys.map((key) => ({ key, versionId: null })), + ].filter( (entry, index, entries) => entries.findIndex((candidate) => candidate.key === entry.key) === index, ); - // Preserve stale/unresolvable keys in the persisted desired set so they stay - // visible (and explicitly removable) instead of vanishing on the next save. - const desiredSkillEntries: AgentDesiredSkillEntry[] = [ - ...resolvedDesiredSkillEntries, - ...unresolvedDesiredSkillKeys.map((key) => ({ key, versionId: null })), - ]; + + const currentPreference = readPaperclipSkillSyncPreference(adapterConfig); + const { resolved: resolvedCurrentSkillEntries, unresolved: unresolvedCurrentSkillKeys } = + currentPreference.desiredSkillEntries.length > 0 + ? await companySkills.resolveRequestedSkillEntries( + companyId, + currentPreference.desiredSkillEntries, + { tolerateUnknownReferences: true }, + ) + : { resolved: [], unresolved: [] }; + const currentSkillEntries = [ + ...resolvedCurrentSkillEntries, + ...unresolvedCurrentSkillKeys.map((key) => ({ key, versionId: null })), + ].filter( + (entry, index, entries) => entries.findIndex((candidate) => candidate.key === entry.key) === index, + ); + + const desiredSkillEntries = mergeDesiredSkillEntries(currentSkillEntries, requestedSkillEntries, mode); const desiredSkills = desiredSkillEntries.map((entry) => entry.key); + const resolvedKeys = new Set([ + ...resolvedCurrentSkillEntries.map((entry) => entry.key), + ...resolvedRequestedSkillEntries.map((entry) => entry.key), + ]); + // Runtime materialization + version selection only ever consider final + // assignments that resolve to the company library; stale keys remain + // persisted and explicitly removable without reaching adapter runtimes. + const runtimeSkillEntries = await companySkills.listRuntimeSkillEntries(companyId, { + materializeMissing: shouldMaterializeRuntimeSkillsForAdapter(adapterType), + versionSelections: skillVersionSelectionMap( + desiredSkillEntries.filter((entry) => resolvedKeys.has(entry.key)), + ), + }); return { adapterConfig: writePaperclipSkillSyncPreference(adapterConfig, desiredSkillEntries), @@ -1992,6 +2045,7 @@ export function agentRoutes( router.post( "/agents/:id/skills/sync", + requireAgentSkillAssignmentMode, validate(agentSkillSyncSchema), async (req, res) => { const id = req.params.id as string; @@ -2010,6 +2064,7 @@ export function agentRoutes( agent.adapterType, agent.adapterConfig as Record, requestedSkills, + req.body.mode, // Toggling a resolvable skill must not fail just because the agent // already carries stale desired keys (e.g. a skill removed from the // library). Preserve those keys so they remain visible/removable. @@ -2074,6 +2129,7 @@ export function agentRoutes( adapterType: updated.adapterType, desiredSkills, desiredSkillEntries, + assignmentMode: req.body.mode, mode: snapshot.mode, supported: snapshot.supported, entryCount: snapshot.entries.length, @@ -2494,6 +2550,7 @@ export function agentRoutes( hireInput.adapterType, requestedAdapterConfig, normalizeDesiredSkillSelections(Array.isArray(requestedDesiredSkills) ? requestedDesiredSkills : undefined), + "add", ); const normalizedAdapterConfig = await normalizeMediatedAdapterConfigForPersistence({ companyId, @@ -2689,6 +2746,7 @@ export function agentRoutes( createInput.adapterType, requestedAdapterConfig, normalizeDesiredSkillSelections(Array.isArray(requestedDesiredSkills) ? requestedDesiredSkills : undefined), + "add", ); const normalizedAdapterConfig = await normalizeMediatedAdapterConfigForPersistence({ companyId, diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index a94702fb767..e11a2f4edff 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -1975,7 +1975,7 @@ registry.registerPath({ params: z.object({ id: z.string() }), body: jsonBody(agentSkillSyncSchema), }, - responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 404: r.notFound }, + responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 404: r.notFound, 422: r.unprocessable }, }); registry.registerPath({ diff --git a/server/src/services/company-portability.ts b/server/src/services/company-portability.ts index 8818598ec4a..cdf4cb56303 100644 --- a/server/src/services/company-portability.ts +++ b/server/src/services/company-portability.ts @@ -203,7 +203,7 @@ function assertInlineSourceComplete(source: CompanyPortabilityImport["source"]) } function resolveSkillConflictStrategy(mode: ImportMode, collisionStrategy: CompanyPortabilityCollisionStrategy) { - if (mode === "board_full") return "replace" as const; + if (mode === "board_full") return collisionStrategy; return collisionStrategy === "skip" ? "skip" as const : "rename" as const; } @@ -5329,7 +5329,7 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { desiredSkillRefMap.set(importedSkill.originalSlug, importedSkill.skill.key); if (importedSkill.action === "skipped") { warnings.push(`Skipped skill ${importedSkill.originalSlug}; existing skill ${importedSkill.skill.slug} was kept.`); - } else if (importedSkill.originalKey !== importedSkill.skill.key) { + } else if (importedSkill.action === "renamed") { warnings.push(`Imported skill ${importedSkill.originalSlug} as ${importedSkill.skill.slug} to avoid overwriting an existing skill.`); } } @@ -6148,6 +6148,15 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { action: companyAction, }, agents: resultAgents, + skills: importedSkills.map((result) => ({ + originalKey: result.originalKey, + originalSlug: result.originalSlug, + key: result.skill.key, + slug: result.skill.slug, + id: result.skill.id, + action: result.action, + reason: result.reason, + })), projects: resultProjects, routines: resultRoutines, envInputs: sourceManifest.envInputs ?? [], diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index 4776fadfa24..22348d87647 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -257,7 +257,7 @@ type PackageSkillConflictStrategy = "replace" | "rename" | "skip"; export type ImportPackageSkillResult = { skill: CompanySkill; - action: "created" | "updated" | "skipped"; + action: "created" | "renamed" | "replaced" | "skipped"; originalKey: string; originalSlug: string; requestedRefs: string[]; @@ -5826,6 +5826,14 @@ export function companySkillService(db: Db) { const importedSkills = readInlineSkillImports(companyId, normalizedFiles); if (importedSkills.length === 0) return []; + // Conflict handling must never bypass source and reserved-key checks. In + // particular, the safe default "skip" path still validates the incoming + // package before returning the existing skill. + for (const skill of importedSkills) { + assertImportedSkillKeyAllowed(skill); + assertImportedSkillSourceAllowed(skill); + } + for (const skill of importedSkills) { if (skill.sourceType !== "catalog") continue; const materializedDir = await materializeCatalogSkillFiles(companyId, skill, normalizedFiles); @@ -5834,7 +5842,7 @@ export function companySkillService(db: Db) { } } - const conflictStrategy = options?.onConflict ?? "replace"; + const conflictStrategy = options?.onConflict ?? "skip"; const existingSkills = await listFull(companyId); const existingByKey = new Map(existingSkills.map((skill) => [skill.key, skill])); const existingBySlug = new Map( @@ -5848,8 +5856,7 @@ export function companySkillService(db: Db) { skill: ImportedSkill; originalKey: string; originalSlug: string; - existingBefore: CompanySkill | null; - actionHint: "created" | "updated"; + actionHint: "created" | "renamed" | "replaced"; reason: string | null; }> = []; const out: ImportPackageSkillResult[] = []; @@ -5863,17 +5870,29 @@ export function companySkillService(db: Db) { const conflict = existingByIncomingKey ?? existingByIncomingSlug; if (!conflict || conflictStrategy === "replace") { - toPersist.push(importedSkill); + const skillToPersist = conflict && conflict.key !== importedSkill.key + ? { + ...importedSkill, + key: conflict.key, + slug: conflict.slug, + metadata: { + ...(importedSkill.metadata ?? {}), + skillKey: conflict.key, + importedFromSkillKey: originalKey, + importedFromSkillSlug: originalSlug, + }, + } + : importedSkill; + toPersist.push(skillToPersist); prepared.push({ - skill: importedSkill, + skill: skillToPersist, originalKey, originalSlug, - existingBefore: existingByIncomingKey, - actionHint: existingByIncomingKey ? "updated" : "created", - reason: existingByIncomingKey ? "Existing skill key matched; replace strategy." : null, + actionHint: conflict ? "replaced" : "created", + reason: conflict ? "Existing skill matched; explicit replace strategy." : null, }); - usedSlugs.add(normalizedSlug); - usedKeys.add(importedSkill.key); + usedSlugs.add(normalizeSkillSlug(skillToPersist.slug) ?? skillToPersist.slug); + usedKeys.add(skillToPersist.key); continue; } @@ -5907,8 +5926,7 @@ export function companySkillService(db: Db) { skill: renamedSkill, originalKey, originalSlug, - existingBefore: null, - actionHint: "created", + actionHint: "renamed", reason: `Existing skill matched; renamed to ${renamedSlug}.`, }); usedSlugs.add(renamedSlug); diff --git a/skills/paperclip/SKILL.md b/skills/paperclip/SKILL.md index 981afa05f26..728395f5bac 100644 --- a/skills/paperclip/SKILL.md +++ b/skills/paperclip/SKILL.md @@ -421,7 +421,7 @@ or linking cases through the agent-facing cases API. Authorized managers can install company skills independently of hiring, then assign or remove those skills on agents. - Install and inspect company skills with the company skills API. -- Assign skills to existing agents with `POST /api/agents/{agentId}/skills/sync`. +- Assign skills to existing agents with `POST /api/agents/{agentId}/skills/sync` and an explicit `add`, `remove`, or `replace` mode. Prefer `add`; `replace` overwrites the complete desired skill set. - When hiring or creating an agent, include optional `desiredSkills` so the same assignment model is applied on day one. If you are asked to install a skill for the company or an agent you MUST read: diff --git a/skills/paperclip/references/company-skills.md b/skills/paperclip/references/company-skills.md index 37ba590ab8f..696653cb9d1 100644 --- a/skills/paperclip/references/company-skills.md +++ b/skills/paperclip/references/company-skills.md @@ -188,11 +188,18 @@ curl -sS "$PAPERCLIP_API_URL/api/companies/$PAPERCLIP_COMPANY_ID/skills//skills/sync" \ -H "Authorization: Bearer $PAPERCLIP_API_KEY" \ -H "Content-Type: application/json" \ -d '{ + "mode": "add", "desiredSkills": [ "vercel-labs/agent-browser/agent-browser" ] diff --git a/ui/src/api/agents.ts b/ui/src/api/agents.ts index 45e134c3cd1..6b42b1f581f 100644 --- a/ui/src/api/agents.ts +++ b/ui/src/api/agents.ts @@ -1,6 +1,7 @@ import type { Agent, AgentDesiredSkillEntry, + AgentSkillAssignmentMode, AgentPermissions, AgentDetail, AgentInstructionsBundle, @@ -178,8 +179,12 @@ export const agentsApi = { listKeys: (id: string, companyId?: string) => api.get(agentPath(id, companyId, "/keys")), skills: (id: string, companyId?: string) => api.get(agentPath(id, companyId, "/skills")), - syncSkills: (id: string, desiredSkills: Array, companyId?: string) => - api.post(agentPath(id, companyId, "/skills/sync"), { desiredSkills }), + syncSkills: ( + id: string, + desiredSkills: Array, + mode: AgentSkillAssignmentMode, + companyId?: string, + ) => api.post(agentPath(id, companyId, "/skills/sync"), { desiredSkills, mode }), createKey: (id: string, name: string, companyId?: string, scope?: AgentApiKeyScope) => api.post(agentPath(id, companyId, "/keys"), { name, ...(scope ? { scope } : {}) }), revokeKey: (agentId: string, keyId: string, companyId?: string) => diff --git a/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx b/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx index aa88f29119b..439e0ce92c1 100644 --- a/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx +++ b/ui/src/components/skill-studio/AgentsUsingSkillDialog.tsx @@ -189,7 +189,7 @@ export function AgentsUsingSkillDialog({ if (desiredSetsEqual(currentEntries, nextEntries)) { return { agentId: agent.id, changed: false }; } - await agentsApi.syncSkills(agent.id, nextEntries, companyId); + await agentsApi.syncSkills(agent.id, nextEntries, "replace", companyId); return { agentId: agent.id, changed: true }; }, onSuccess: async ({ agentId }) => { diff --git a/ui/src/pages/CompanyImport.test.tsx b/ui/src/pages/CompanyImport.test.tsx index ac531c8ab51..dd2fefcb97b 100644 --- a/ui/src/pages/CompanyImport.test.tsx +++ b/ui/src/pages/CompanyImport.test.tsx @@ -160,6 +160,7 @@ function buildImportResult(): CompanyPortabilityImportResult { return { company: { id: "company-2", name: "Imported Test", action: "created" }, agents: [{ slug: "coder", id: "agent-1", action: "created", name: "Coder", reason: null }], + skills: [], projects: [], routines: [{ slug: "weekly-report", id: "routine-1", action: "created", title: "Weekly Report", status: "paused" }], envInputs: [], diff --git a/ui/src/pages/CompanyImport.tsx b/ui/src/pages/CompanyImport.tsx index 5aecd4a4374..3339867d6de 100644 --- a/ui/src/pages/CompanyImport.tsx +++ b/ui/src/pages/CompanyImport.tsx @@ -1455,6 +1455,7 @@ export function CompanyImport() { if (importOutcome) { const { result, dashboardPath } = importOutcome; + const skillResults = result.skills ?? []; const activationItems = importOutcome.pausedAutomations ? buildActivationItems(result) : []; const pendingCount = activationItems.filter( (item) => activationChecked.has(item.key) && !activatedKeys.has(item.key), @@ -1465,11 +1466,31 @@ export function CompanyImport() {

Import complete

{result.company.name}: {result.agents.length} agent{result.agents.length === 1 ? "" : "s"},{" "} + {skillResults.length} skill{skillResults.length === 1 ? "" : "s"},{" "} {result.projects.length} project{result.projects.length === 1 ? "" : "s"}, and{" "} {result.routines.length} routine{result.routines.length === 1 ? "" : "s"} processed.

+ {skillResults.length > 0 && ( +
+
+

Skill import results

+
+
+ {skillResults.map((skill) => ( +
+ {skill.originalSlug} + {skill.action} + {skill.slug !== skill.originalSlug && ( + as {skill.slug} + )} +
+ ))} +
+
+ )} + {result.warnings.length > 0 && (
{result.warnings.map((w) => ( diff --git a/ui/src/pages/CompanySkills.tsx b/ui/src/pages/CompanySkills.tsx index be8a9f41ad8..2bb0859f07f 100644 --- a/ui/src/pages/CompanySkills.tsx +++ b/ui/src/pages/CompanySkills.tsx @@ -4850,7 +4850,7 @@ export function CompanySkills() { const attachAgentsMutation = useMutation({ mutationFn: async (input: { agentId: string; desiredSkills: Array }) => { - return agentsApi.syncSkills(input.agentId, input.desiredSkills, selectedCompanyId ?? undefined); + return agentsApi.syncSkills(input.agentId, input.desiredSkills, "replace", selectedCompanyId ?? undefined); }, onSuccess: async () => { await Promise.all([ diff --git a/ui/src/pages/agent-skills/AgentSkillsTab.tsx b/ui/src/pages/agent-skills/AgentSkillsTab.tsx index 44adb974373..4ec4d2b82ee 100644 --- a/ui/src/pages/agent-skills/AgentSkillsTab.tsx +++ b/ui/src/pages/agent-skills/AgentSkillsTab.tsx @@ -112,7 +112,7 @@ export function AgentSkillsTab({ agent, companyId }: { agent: Agent; companyId?: const syncSkills = useMutation({ mutationFn: (desiredSkills: Array) => - agentsApi.syncSkills(agent.id, desiredSkills, companyId), + agentsApi.syncSkills(agent.id, desiredSkills, "replace", companyId), onSuccess: async (snapshot) => { queryClient.setQueryData(queryKeys.agents.skills(agent.id), snapshot); lastSavedSkillsRef.current = snapshot.desiredSkills; diff --git a/ui/storybook/stories/team-catalog.stories.tsx b/ui/storybook/stories/team-catalog.stories.tsx index c30ee006cf4..7fd0d1e4c0d 100644 --- a/ui/storybook/stories/team-catalog.stories.tsx +++ b/ui/storybook/stories/team-catalog.stories.tsx @@ -352,6 +352,7 @@ export const InstallSuccess: Story = { { slug: "cto", id: "a2", action: "created", name: "CTO", reason: null }, { slug: "cmo", id: "a3", action: "created", name: "CMO (from Core Exec Team)", reason: null }, ], + skills: [], projects: [{ slug: "launch", id: "p1", action: "created", name: "Launch", reason: null }], routines: [], envInputs: [], From 9485ffea7028584e0bd2e7ff0fd0d7b42d980964 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 7 Aug 2026 00:54:43 -0500 Subject: [PATCH 014/311] fix(config): preserve env files during managed updates (#10980) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The CLI and server both update Paperclip values in `.env` files > - The server preserved operator content, but the CLI rebuilt the complete file > - A CLI rerun could remove comments, custom values, ordering, and newline style > - Both paths need one editor with one value encoding and duplicate key policy > - The final integration also needs one regression test across the related setup and sync safety mechanisms > - This pull request moves the editor to the shared package and adds cross-cutting rerun-survival coverage > - The benefit is safe setup and worktree repair reruns that preserve operator edits ## Linked Issues or Issue Description **What happened?** The CLI rebuilt the complete `.env` file when it wrote a managed Paperclip value. This action removed comments, blank lines, custom keys, original quoting, and the original newline style. **Expected behavior** Paperclip must update only the managed assignments. It must preserve all unrelated bytes. It must skip the file replacement when all managed values are current. **Steps to reproduce** 1. Add comments, custom keys, quoted values, and CRLF newlines to the Paperclip `.env` file. 2. Run a CLI path that calls the agent JWT secret setup. 3. Observe that the old writer replaces the complete file. **Paperclip version or commit** The problem exists on `master` before this pull request. Related public context: Refs #437. ## What Changed - Add one shared line-preserving `.env` editor for the CLI and server. - Define minimal and JSON value encodings in the shared helper. - Update every stale duplicate of a managed key and preserve current duplicate encodings. - Preserve comments, ordering, blank lines, unknown keys, export prefixes, trailing comments, and newline style. - Write changed files through a same-directory temporary file and atomic rename. - Limit CLI updates to non-empty `PAPERCLIP_*` entries. - Skip the write when all managed values are current. - Add shared, CLI, and server regression coverage. - Refresh the branch after the related config, sandbox, and skill safety changes landed. - Add a cross-cutting integration test for config, env-file, managed-sandbox, and managed-instructions rerun survival. ## Verification - `pnpm exec vitest run packages/shared/src/env-file.test.ts packages/shared/src/config-schema.test.ts cli/src/__tests__/agent-jwt-env.test.ts cli/src/__tests__/config-store.test.ts server/src/__tests__/config-file.test.ts server/src/__tests__/worktree-config.test.ts` passes 39 tests. - `pnpm exec vitest run server/src/__tests__/rerun-survival.integration.test.ts` passes 4 tests. - `pnpm -r typecheck` passes on the previous head. GitHub CI reruns it on the refreshed head. - The previous head passed the complete general, serialized, workspace, and E2E matrix. GitHub CI reruns that matrix on the refreshed head. - `pnpm build` passes on the previous head. GitHub CI reruns it on the refreshed head. ## Risks - Low risk. The production change only affects managed `.env` assignments. - Existing managed assignments can keep their original quoting when their decoded values are current. - Changed CLI values keep the prior minimal encoding policy. Changed server values keep the prior JSON encoding policy. - Duplicate managed assignments now follow one explicit rule: Paperclip updates each stale occurrence. - The master refresh had one import-block conflict. The resolution keeps both the config merge imports and the env-file imports. - The added integration file is test-only. It has no database, API, or UI contract effect. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex from the GPT-5 family produced this change with reasoning, tool use, and code execution. The runtime did not expose the exact model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- cli/src/__tests__/agent-jwt-env.test.ts | 58 ++++++ cli/src/config/env.ts | 40 ++-- packages/shared/src/env-file.test.ts | 98 ++++++++++ packages/shared/src/env-file.ts | 181 +++++++++++++++++ .../rerun-survival.integration.test.ts | 183 ++++++++++++++++++ server/src/worktree-config.ts | 83 ++------ 6 files changed, 551 insertions(+), 92 deletions(-) create mode 100644 packages/shared/src/env-file.test.ts create mode 100644 packages/shared/src/env-file.ts create mode 100644 server/src/__tests__/rerun-survival.integration.test.ts diff --git a/cli/src/__tests__/agent-jwt-env.test.ts b/cli/src/__tests__/agent-jwt-env.test.ts index baf5db5128b..b26ae018df2 100644 --- a/cli/src/__tests__/agent-jwt-env.test.ts +++ b/cli/src/__tests__/agent-jwt-env.test.ts @@ -76,4 +76,62 @@ describe("agent jwt env helpers", () => { expect(contents).toContain('PAPERCLIP_WORKTREE_COLOR="#439edb"'); expect(readPaperclipEnvEntries(envPath).PAPERCLIP_WORKTREE_COLOR).toBe("#439edb"); }); + + it("preserves operator content and CRLF while updating only managed entries", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + "export PAPERCLIP_HOME = '/old path' # managed path", + "PAPERCLIP_DUPLICATE=stale", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + "", + ].join("\r\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + + mergePaperclipEnvEntries( + { + PAPERCLIP_HOME: "/new path", + PAPERCLIP_DUPLICATE: "current", + PAPERCLIP_WORKTREE_COLOR: "#439edb", + DATABASE_URL: "postgres://paperclip-must-not-overwrite", + }, + envPath, + ); + + const updated = fs.readFileSync(envPath, "utf8"); + expect(updated).toBe([ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + 'export PAPERCLIP_HOME = "/new path" # managed path', + "PAPERCLIP_DUPLICATE=current", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + 'PAPERCLIP_WORKTREE_COLOR="#439edb"', + "", + ].join("\r\n")); + expect(updated.replaceAll("\r\n", "")).not.toContain("\n"); + }); + + it("does not replace the env file when managed values are already current", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# preserve this file byte-for-byte", + "export PAPERCLIP_HOME = '/same path'", + "UNKNOWN=\"operator encoding\"", + "", + ].join("\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + const previousInode = fs.statSync(envPath).ino; + + mergePaperclipEnvEntries({ PAPERCLIP_HOME: "/same path" }, envPath); + + expect(fs.readFileSync(envPath, "utf8")).toBe(original); + expect(fs.statSync(envPath).ino).toBe(previousInode); + }); }); diff --git a/cli/src/config/env.ts b/cli/src/config/env.ts index a7266ea241c..6787ee190a8 100644 --- a/cli/src/config/env.ts +++ b/cli/src/config/env.ts @@ -2,9 +2,11 @@ import fs from "node:fs"; import path from "node:path"; import { randomBytes } from "node:crypto"; import { config as loadDotenv, parse as parseEnvFileContents } from "dotenv"; +import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file"; import { resolveConfigPath } from "./store.js"; const JWT_SECRET_ENV_KEY = "PAPERCLIP_AGENT_JWT_SECRET"; +const PAPERCLIP_OWNED_ENV_KEY_PATTERN = /^PAPERCLIP_[A-Z0-9_]+$/; function resolveEnvFilePath(configPath?: string) { return path.resolve(path.dirname(resolveConfigPath(configPath)), ".env"); } @@ -22,21 +24,20 @@ function parseEnvFile(contents: string) { } } -function formatEnvValue(value: string): string { - if (/^[A-Za-z0-9_./:@-]+$/.test(value)) { - return value; - } - return JSON.stringify(value); -} - -function renderEnvFile(entries: Record) { - const lines = [ +function emptyEnvFileContents() { + return [ "# Paperclip environment variables", "# Generated by Paperclip CLI commands", - ...Object.entries(entries).map(([key, value]) => `${key}=${formatEnvValue(value)}`), "", - ]; - return lines.join("\n"); + ].join("\n"); +} + +function paperclipOwnedEntries(entries: Record): Record { + return Object.fromEntries( + Object.entries(entries).filter( + ([key, value]) => PAPERCLIP_OWNED_ENV_KEY_PATTERN.test(key) && value.trim().length > 0, + ), + ); } export function resolvePaperclipEnvFile(configPath?: string): string { @@ -102,11 +103,11 @@ export function readPaperclipEnvEntries(filePath = resolveEnvFilePath()): Record } export function writePaperclipEnvEntries(entries: Record, filePath = resolveEnvFilePath()): void { - const dir = path.dirname(filePath); - fs.mkdirSync(dir, { recursive: true }); - fs.writeFileSync(filePath, renderEnvFile(entries), { - mode: 0o600, + const previousContents = fs.existsSync(filePath) ? fs.readFileSync(filePath, "utf8") : null; + const nextContents = updateEnvFileContents(previousContents ?? emptyEnvFileContents(), paperclipOwnedEntries(entries), { + valueEncoding: "minimal", }); + writeEnvFileAtomicallyIfChanged(filePath, previousContents, nextContents); } export function mergePaperclipEnvEntries( @@ -114,12 +115,11 @@ export function mergePaperclipEnvEntries( filePath = resolveEnvFilePath(), ): Record { const current = readPaperclipEnvEntries(filePath); + const managedEntries = paperclipOwnedEntries(entries); const next = { ...current, - ...Object.fromEntries( - Object.entries(entries).filter(([, value]) => typeof value === "string" && value.trim().length > 0), - ), + ...managedEntries, }; - writePaperclipEnvEntries(next, filePath); + writePaperclipEnvEntries(managedEntries, filePath); return next; } diff --git a/packages/shared/src/env-file.test.ts b/packages/shared/src/env-file.test.ts new file mode 100644 index 00000000000..57f2cd76173 --- /dev/null +++ b/packages/shared/src/env-file.test.ts @@ -0,0 +1,98 @@ +import { describe, expect, it } from "vitest"; +import { encodeEnvValue, updateEnvFileContents } from "./env-file.js"; + +describe("env file editor", () => { + it("pins minimal and JSON value encoding", () => { + expect(encodeEnvValue("plain-value", "minimal")).toBe("plain-value"); + expect(encodeEnvValue("#439edb", "minimal")).toBe('"#439edb"'); + expect(encodeEnvValue("plain-value", "json")).toBe('"plain-value"'); + }); + + it("preserves unrelated content and CRLF while updating every stale duplicate", () => { + const original = [ + "# operator comment", + "UNKNOWN='keep this encoding'", + "", + "export PAPERCLIP_HOME = '/old path' # managed path", + "PAPERCLIP_DUPLICATE=stale", + 'PAPERCLIP_DUPLICATE="current"', + "TRAILING=untouched", + "", + ].join("\r\n"); + + const updated = updateEnvFileContents( + original, + { + PAPERCLIP_HOME: "/new path", + PAPERCLIP_DUPLICATE: "current", + PAPERCLIP_WORKTREE_COLOR: "#439edb", + }, + { valueEncoding: "minimal" }, + ); + + expect(updated).toBe([ + "# operator comment", + "UNKNOWN='keep this encoding'", + "", + 'export PAPERCLIP_HOME = "/new path" # managed path', + "PAPERCLIP_DUPLICATE=current", + 'PAPERCLIP_DUPLICATE="current"', + "TRAILING=untouched", + 'PAPERCLIP_WORKTREE_COLOR="#439edb"', + "", + ].join("\r\n")); + expect(updated.replaceAll("\r\n", "")).not.toContain("\n"); + }); + + it("uses JSON encoding for changed values without re-encoding current assignments", () => { + const original = [ + "PAPERCLIP_CURRENT=plain-value", + "PAPERCLIP_CHANGED=old", + "UNKNOWN=\"operator value\"", + "", + ].join("\n"); + + expect( + updateEnvFileContents( + original, + { + PAPERCLIP_CURRENT: "plain-value", + PAPERCLIP_CHANGED: "new", + PAPERCLIP_ADDED: "added", + }, + { valueEncoding: "json" }, + ), + ).toBe([ + "PAPERCLIP_CURRENT=plain-value", + 'PAPERCLIP_CHANGED="new"', + 'UNKNOWN="operator value"', + 'PAPERCLIP_ADDED="added"', + "", + ].join("\n")); + }); + + it("does not treat an unquoted dotenv comment as the managed value", () => { + expect( + updateEnvFileContents( + ["PAPERCLIP_COLOR=#439edb", "PAPERCLIP_HOME=old# keep this comment"].join("\n"), + { + PAPERCLIP_COLOR: "#439edb", + PAPERCLIP_HOME: "new", + }, + { valueEncoding: "minimal" }, + ), + ).toBe( + ['PAPERCLIP_COLOR="#439edb"#439edb', "PAPERCLIP_HOME=new# keep this comment"].join("\n"), + ); + }); + + it("is a no-op when every managed duplicate is already current", () => { + const original = [ + "export PAPERCLIP_HOME = '/same path' # first", + 'PAPERCLIP_HOME="/same path"', + "UNKNOWN=value", + ].join("\n"); + + expect(updateEnvFileContents(original, { PAPERCLIP_HOME: "/same path" })).toBe(original); + }); +}); diff --git a/packages/shared/src/env-file.ts b/packages/shared/src/env-file.ts new file mode 100644 index 00000000000..ee15f8dfa66 --- /dev/null +++ b/packages/shared/src/env-file.ts @@ -0,0 +1,181 @@ +import fs from "node:fs"; +import path from "node:path"; +import { randomUUID } from "node:crypto"; + +export type EnvValueEncoding = "minimal" | "json"; + +export type UpdateEnvFileContentsOptions = { + valueEncoding?: EnvValueEncoding; +}; + +type EnvLine = { + contents: string; + ending: string; +}; + +const ENV_ASSIGNMENT_PATTERN = /^(\s*(?:export\s+)?)([A-Za-z_][A-Za-z0-9_]*)(\s*=\s*)(.*)$/; +const MINIMAL_ENV_VALUE_PATTERN = /^[A-Za-z0-9_./:@-]+$/; + +export function encodeEnvValue(value: string, encoding: EnvValueEncoding = "json"): string { + if (encoding === "minimal" && MINIMAL_ENV_VALUE_PATTERN.test(value)) { + return value; + } + return JSON.stringify(value); +} + +function trailingEnvCommentStart(rawValue: string): number | null { + let quote: "\"" | "'" | null = null; + let escaped = false; + + for (let index = 0; index < rawValue.length; index += 1) { + const character = rawValue[index]; + if (escaped) { + escaped = false; + continue; + } + if (quote === "\"" && character === "\\") { + escaped = true; + continue; + } + if (quote !== null) { + if (character === quote) quote = null; + continue; + } + if (character === "\"" || character === "'") { + quote = character; + continue; + } + // dotenv treats every unquoted # as the start of a comment, even when it + // immediately follows the assignment separator or the value. + if (character !== "#") continue; + + let commentStart = index; + while (commentStart > 0 && /\s/.test(rawValue[commentStart - 1] ?? "")) { + commentStart -= 1; + } + return commentStart; + } + + return null; +} + +function parseComparableEnvValue(rawValue: string): string | null { + const commentStart = trailingEnvCommentStart(rawValue); + const encodedValue = rawValue.slice(0, commentStart ?? rawValue.length).trim(); + + if (encodedValue === "") return ""; + if (encodedValue.startsWith("\"") && encodedValue.endsWith("\"")) { + try { + const parsed = JSON.parse(encodedValue); + return typeof parsed === "string" ? parsed : null; + } catch { + return null; + } + } + if (encodedValue.startsWith("'") && encodedValue.endsWith("'")) { + return encodedValue.slice(1, -1); + } + return encodedValue; +} + +function splitEnvLines(contents: string): EnvLine[] { + const lines: EnvLine[] = []; + const newlinePattern = /\r\n|\n|\r/g; + let lineStart = 0; + let match: RegExpExecArray | null; + + while ((match = newlinePattern.exec(contents)) !== null) { + lines.push({ + contents: contents.slice(lineStart, match.index), + ending: match[0], + }); + lineStart = newlinePattern.lastIndex; + } + + if (lineStart < contents.length || contents.length === 0) { + lines.push({ contents: contents.slice(lineStart), ending: "" }); + } + + return lines; +} + +/** + * Updates only the named assignments and preserves every unrelated byte. + * + * Duplicate managed keys are intentional: every occurrence is checked and any + * stale occurrence is updated. Occurrences that already decode to the desired + * value retain their original quoting and spacing. + */ +export function updateEnvFileContents( + contents: string, + entries: Record, + options: UpdateEnvFileContentsOptions = {}, +): string { + const valueEncoding = options.valueEncoding ?? "json"; + const missingEntries = new Map(Object.entries(entries)); + const lines = splitEnvLines(contents); + + for (const line of lines) { + const match = line.contents.match(ENV_ASSIGNMENT_PATTERN); + if (!match) continue; + + const [, prefix, key, separator, rawValue] = match; + const value = entries[key]; + if (value === undefined) continue; + + missingEntries.delete(key); + if (parseComparableEnvValue(rawValue) === value) continue; + + const commentStart = trailingEnvCommentStart(rawValue); + const trailingComment = commentStart === null ? "" : rawValue.slice(commentStart); + line.contents = `${prefix}${key}${separator}${encodeEnvValue(value, valueEncoding)}${trailingComment}`; + } + + if (missingEntries.size > 0) { + const newline = lines.find((line) => line.ending.length > 0)?.ending ?? "\n"; + const insertedLines = Array.from(missingEntries, ([key, value]) => ({ + contents: `${key}=${encodeEnvValue(value, valueEncoding)}`, + ending: newline, + })); + const trailingBlankIndex = lines.at(-1)?.contents === "" ? lines.length - 1 : lines.length; + + if (trailingBlankIndex === lines.length && lines.length > 0) { + const previousLine = lines.at(-1)!; + if (previousLine.ending === "") { + previousLine.ending = newline; + insertedLines.at(-1)!.ending = ""; + } + } + lines.splice(trailingBlankIndex, 0, ...insertedLines); + } + + return lines.map((line) => `${line.contents}${line.ending}`).join(""); +} + +/** Writes a changed env file through a same-directory temporary file and rename. */ +export function writeEnvFileAtomicallyIfChanged( + filePath: string, + previousContents: string | null, + nextContents: string, +): boolean { + if (previousContents === nextContents) return false; + + const directory = path.dirname(filePath); + fs.mkdirSync(directory, { recursive: true }); + const temporaryPath = path.join(directory, `.${path.basename(filePath)}.${process.pid}.${randomUUID()}.tmp`); + + try { + fs.writeFileSync(temporaryPath, nextContents, { encoding: "utf8", mode: 0o600, flag: "wx" }); + fs.renameSync(temporaryPath, filePath); + fs.chmodSync(filePath, 0o600); + } catch (error) { + try { + fs.unlinkSync(temporaryPath); + } catch { + // The temporary file may not have been created or may already have been renamed. + } + throw error; + } + + return true; +} diff --git a/server/src/__tests__/rerun-survival.integration.test.ts b/server/src/__tests__/rerun-survival.integration.test.ts new file mode 100644 index 00000000000..1708febeadf --- /dev/null +++ b/server/src/__tests__/rerun-survival.integration.test.ts @@ -0,0 +1,183 @@ +import { promises as fs } from "node:fs"; +import fsSync from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { mergePaperclipConfig, paperclipConfigSchema } from "@paperclipai/shared"; +import { + updateEnvFileContents, + writeEnvFileAtomicallyIfChanged, +} from "@paperclipai/shared/env-file"; +import { resourceStatus, stockHash } from "../services/managed-resource-drift.js"; +import { agentInstructionsService } from "../services/agent-instructions.js"; + +/** + * Cross-cutting setup/sync rerun survival test. + * + * Seeds hand edits across every formerly-destructive setup/sync surface, reruns the + * setup/sync mechanism, and asserts each edit survives. Each block drives the SAME + * production code the writers/reconcilers use: + * - config.json -> mergePaperclipConfig + paperclipConfigSchema (shared core + * used by BOTH writers: cli/src/config/store.ts:writeConfig + * and server/src/worktree-config.ts:writeConfigFile) + * - .env -> updateEnvFileContents + writeEnvFileAtomicallyIfChanged + * - managed sandbox env -> resourceStatus/stockHash drift gate (the `shouldWrite` + * classifier the boot reconciler and built-in sync share) + * - managed instructions -> agentInstructionsService materialize + the drift gate that + * makes built-in/plugin/portability rebuilds skip operator edits + * + * The DB-backed end-to-end variants of the sandbox-row, skills-assignment, and + * multi-caller instruction paths are additionally proven by the embedded-Postgres suites: + * server/src/__tests__/company-skills-service.test.ts (skills add/remove/replace) + * server/src/services/managed-environments.test.ts (sandbox row boot reconcile) + * server/src/__tests__/environment-service.test.ts (sandbox skip + stock update) + * server/src/__tests__/agent-instructions-service.test.ts (managed instruction drift) + */ + +const cleanupDirs = new Set(); + +async function tmp(prefix: string) { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), prefix)); + cleanupDirs.add(dir); + return dir; +} + +afterEach(async () => { + await Promise.all( + [...cleanupDirs].map(async (dir) => { + await fs.rm(dir, { recursive: true, force: true }); + cleanupDirs.delete(dir); + }), + ); + delete process.env.PAPERCLIP_HOME; + delete process.env.PAPERCLIP_INSTANCE_ID; +}); + +const BASE_CONFIG = { + $meta: { version: 1, updatedAt: "2026-08-07T00:00:00.000Z", source: "configure" }, + database: { mode: "embedded-postgres" }, + logging: { mode: "file" }, + server: {}, +} as const; + +/** Emulates a writer round-trip on disk: parse the update, merge over the parsed source. */ +function writerRoundTrip(configPath: string, update: unknown): void { + const parsedUpdate = paperclipConfigSchema.parse(update); + const source = paperclipConfigSchema.parse( + JSON.parse(fsSync.readFileSync(configPath, "utf8")), + ); + const next = paperclipConfigSchema.parse(mergePaperclipConfig(source, parsedUpdate)); + fsSync.writeFileSync(configPath, JSON.stringify(next, null, 2) + "\n"); +} + +describe("setup/sync rerun survival — cross-cutting", () => { + it("preserves unknown top-level and nested config.json keys across a writer rerun", async () => { + const dir = await tmp("pap16587-config-"); + const configPath = path.join(dir, "config.json"); + + // Seed a config that carries operator extension keys the schema does not know about. + const seeded = paperclipConfigSchema.parse({ + ...BASE_CONFIG, + customTopLevel: { keepMe: true, nested: [1, 2, 3] }, + server: { ...BASE_CONFIG.server, customServerKey: "operator-owned" }, + }); + fsSync.writeFileSync(configPath, JSON.stringify(seeded, null, 2) + "\n"); + + // Rerun setup/sync with an unrelated known-field update (flip logging mode). + writerRoundTrip(configPath, { ...BASE_CONFIG, logging: { mode: "cloud" } }); + + const after = JSON.parse(fsSync.readFileSync(configPath, "utf8")) as Record; + expect(after.customTopLevel).toEqual({ keepMe: true, nested: [1, 2, 3] }); + expect(after.server.customServerKey).toBe("operator-owned"); + expect(after.logging.mode).toBe("cloud"); // the intended update still applied + }); + + it("preserves comments, ordering, blank lines, and existing encodings in .env", async () => { + const dir = await tmp("pap16587-env-"); + const envPath = path.join(dir, ".env"); + const original = [ + "# top comment", + "", + "FIRST=one", + 'QUOTED="already json"', + "SECOND=two # trailing note", + "", + "# trailing comment", + "", + ].join("\n"); + fsSync.writeFileSync(envPath, original); + + // Rerun that only changes SECOND; everything else must be byte-preserved. + const next = updateEnvFileContents(original, { SECOND: "changed" }, { valueEncoding: "minimal" }); + const wrote = writeEnvFileAtomicallyIfChanged(envPath, original, next); + expect(wrote).toBe(true); + + const after = fsSync.readFileSync(envPath, "utf8"); + expect(after).toContain("# top comment"); + expect(after).toContain("# trailing comment"); + expect(after).toContain('QUOTED="already json"'); // encoding untouched + expect(after).toContain("SECOND=changed # trailing note"); // value changed, comment kept + expect(after.indexOf("FIRST=one")).toBeLessThan(after.indexOf("SECOND=")); // ordering kept + expect(after.split("\n").filter((l) => l === "").length).toBe(original.split("\n").filter((l) => l === "").length); + + // A no-op rerun writes nothing. + expect(writeEnvFileAtomicallyIfChanged(envPath, after, updateEnvFileContents(after, { SECOND: "changed" }, { valueEncoding: "minimal" }))).toBe(false); + }); + + it("skips an operator-modified managed sandbox row but advances an untouched stale one", () => { + const stockV1 = { image: "node:20", packages: ["git"] }; + const stockV2 = { image: "node:22", packages: ["git", "curl"] }; + const v1Hash = stockHash(stockV1); + const v2Hash = stockHash(stockV2); + + // Operator hand-edited the row away from stock -> rerun must NOT clobber it. + const operatorHash = stockHash({ image: "node:20", packages: ["git", "operator-tool"] }); + expect( + resourceStatus({ resourceId: "env-1", currentHash: operatorHash, bindingStockHash: v1Hash, latestStockHash: v2Hash }), + ).toBe("operator_modified"); + + // Untouched row still at last-synced stock, newer stock available -> update is exposed. + expect( + resourceStatus({ resourceId: "env-1", currentHash: v1Hash, bindingStockHash: v1Hash, latestStockHash: v2Hash }), + ).toBe("stock_update_available"); + + // Already current -> no-op. + expect( + resourceStatus({ resourceId: "env-1", currentHash: v2Hash, bindingStockHash: v1Hash, latestStockHash: v2Hash }), + ).toBe("stock_current"); + }); + + it("keeps operator edits and additions in a managed instructions tree across a re-materialize", async () => { + const home = await tmp("pap16587-instr-home-"); + process.env.PAPERCLIP_HOME = home; + process.env.PAPERCLIP_INSTANCE_ID = "test-instance"; + + const svc = agentInstructionsService(); + const agent = { id: "agent-1", companyId: "company-1", name: "Agent 1", adapterConfig: {} }; + const stockFiles = { "AGENTS.md": "# stock\n", "docs/TOOLS.md": "## stock tools\n" }; + + const first = await svc.materializeManagedBundle(agent, stockFiles, { entryFile: "AGENTS.md" }); + const root = first.bundle.managedRootPath!; + + // Operator hand-edits a managed file and adds a brand-new operator-only file. + await fs.writeFile(path.join(root, "AGENTS.md"), "# stock\n\noperator note\n", "utf8"); + await fs.writeFile(path.join(root, "OPERATOR.md"), "operator-added\n", "utf8"); + + // The drift gate the built-in/plugin/portability rebuilds share: classify the tree. + const currentFiles = { + "AGENTS.md": await fs.readFile(path.join(root, "AGENTS.md"), "utf8"), + "docs/TOOLS.md": await fs.readFile(path.join(root, "docs", "TOOLS.md"), "utf8"), + }; + const status = resourceStatus({ + resourceId: agent.id, + currentHash: stockHash(currentFiles), + bindingStockHash: stockHash(stockFiles), + latestStockHash: stockHash(stockFiles), + }); + expect(status).toBe("operator_modified"); // -> reconcile shouldWrite=false, tree left intact + + // Prove the on-disk edits and additions still exist after the sync decision. + await expect(fs.readFile(path.join(root, "AGENTS.md"), "utf8")).resolves.toContain("operator note"); + await expect(fs.readFile(path.join(root, "OPERATOR.md"), "utf8")).resolves.toBe("operator-added\n"); + }); +}); diff --git a/server/src/worktree-config.ts b/server/src/worktree-config.ts index de999134bdd..913451d0be6 100644 --- a/server/src/worktree-config.ts +++ b/server/src/worktree-config.ts @@ -7,6 +7,7 @@ import { paperclipConfigSchema, type PaperclipConfig, } from "@paperclipai/shared"; +import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file"; import { resolvePaperclipConfigPath, resolvePaperclipEnvPath } from "./paths.js"; function nonEmpty(value: string | null | undefined): string | null { @@ -81,73 +82,14 @@ function readEnvEntries(envPath: string): Record { return parseEnvFile(fs.readFileSync(envPath, "utf8")); } -function formatEnvEntries(entries: Record): string { +function emptyWorktreeEnvFileContents(): string { return [ "# Paperclip environment variables", "# Generated by Paperclip worktree repair", - ...Object.entries(entries).map(([key, value]) => `${key}=${JSON.stringify(value)}`), "", ].join("\n"); } -function trailingEnvComment(rawValue: string): string { - let quote: "\"" | "'" | null = null; - let escaped = false; - - for (let index = 0; index < rawValue.length; index += 1) { - const character = rawValue[index]; - if (escaped) { - escaped = false; - continue; - } - if (quote === "\"" && character === "\\") { - escaped = true; - continue; - } - if (quote !== null) { - if (character === quote) quote = null; - continue; - } - if (character === "\"" || character === "'") { - quote = character; - continue; - } - if (character !== "#" || index === 0 || !/\s/.test(rawValue[index - 1] ?? "")) continue; - - let commentStart = index; - while (commentStart > 0 && /\s/.test(rawValue[commentStart - 1] ?? "")) { - commentStart -= 1; - } - return rawValue.slice(commentStart); - } - - return ""; -} - -function updateEnvFileContents(contents: string, entries: Record): string { - const newline = contents.includes("\r\n") ? "\r\n" : "\n"; - const missingEntries = new Map(Object.entries(entries)); - const lines = contents.split(/\r?\n/).map((rawLine) => { - const match = rawLine.match(/^(\s*(?:export\s+)?)([A-Za-z_][A-Za-z0-9_]*)(\s*=\s*)(.*)$/); - if (!match) return rawLine; - - const [, prefix, key, separator, rawValue] = match; - const value = entries[key]; - if (value === undefined) return rawLine; - - missingEntries.delete(key); - return `${prefix}${key}${separator}${JSON.stringify(value)}${trailingEnvComment(rawValue)}`; - }); - - const insertionIndex = lines.at(-1) === "" ? lines.length - 1 : lines.length; - lines.splice( - insertionIndex, - 0, - ...Array.from(missingEntries, ([key, value]) => `${key}=${JSON.stringify(value)}`), - ); - return lines.join(newline); -} - function isPathInside(candidatePath: string, rootPath: string): boolean { const candidate = path.resolve(candidatePath); const root = path.resolve(rootPath); @@ -679,7 +621,6 @@ export function maybeRepairLegacyWorktreeConfigAndEnvFiles(): { } } - const existingEnvEntries = readEnvEntries(context.envPath); const managedEnvEntries: Record = { PAPERCLIP_HOME: context.homeDir, PAPERCLIP_INSTANCE_ID: context.instanceId, @@ -692,20 +633,18 @@ export function maybeRepairLegacyWorktreeConfigAndEnvFiles(): { process.env.PAPERCLIP_DB_BACKUP_ENABLED = "false"; - const repairedEnv = Object.entries(managedEnvEntries).some( - ([key, value]) => existingEnvEntries[key] !== value, + const existingContents = fs.existsSync(context.envPath) + ? fs.readFileSync(context.envPath, "utf8") + : null; + const repairedContents = updateEnvFileContents( + existingContents ?? emptyWorktreeEnvFileContents(), + managedEnvEntries, + { valueEncoding: "json" }, ); + const repairedEnv = existingContents !== repairedContents; if (repairedEnv) { - fs.mkdirSync(path.dirname(context.envPath), { recursive: true }); - const existingContents = fs.existsSync(context.envPath) - ? fs.readFileSync(context.envPath, "utf8") - : null; - const repairedContents = - existingContents === null - ? formatEnvEntries(managedEnvEntries) - : updateEnvFileContents(existingContents, managedEnvEntries); - fs.writeFileSync(context.envPath, repairedContents, { mode: 0o600 }); + writeEnvFileAtomicallyIfChanged(context.envPath, existingContents, repairedContents); } return { repairedConfig, repairedEnv }; From 4e76227f128a9e97f5848766a05d59fdb0b3ebc8 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Thu, 6 Aug 2026 22:56:21 -0700 Subject: [PATCH 015/311] feat(plugin-daytona): stream session command logs behind useLogStream (#11021) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox provider plugins let agents run commands in remote environments. > - The Daytona provider polls the exit code while it waits for command logs. > - Polling delays log delivery and does not support long-lived streamed commands. > - This pull request adds an opt-in Daytona log stream with one reconnect and a poll fallback. > - The benefit is faster log delivery while the existing default path stays unchanged. ## Linked Issues or Issue Description Refs: #10941 **Subsystem affected** packages/plugins — sandbox provider plugins. **Problem or motivation** The Daytona provider polls the command exit code every 50 milliseconds while it waits for logs. This delays output and does not support a long-lived streamed command. **Proposed solution** Add the `useLogStream` provider option. Stream stdout and stderr from the Daytona callback log form, read the exit code after the stream ends, retry the read with bounded backoff, and fall back to the existing poll path after a disconnect. **Alternatives considered** Keep polling for all commands. This keeps the current behavior but does not provide timely logs or a path for long-lived commands. **Roadmap alignment** This supports the roadmap item for cloud and sandbox agents, including Daytona. ## What Changed - Add the opt-in `useLogStream` option with a default of `false`. - Stream stdout and stderr from the Daytona callback log form. - Drop replayed log prefixes by delivered byte offset after reconnect. - Retry once after disconnect, then use the existing poll path. - Read the exit code once after a successful stream and retry when the code is not ready. - Add tests for ordered output, exit-code reads, disconnect fallback, and reconnect replay handling. ## Verification - Run `node_modules/.bin/vitest run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/plugin.test.ts`. - Confirm that the full Daytona plugin test project passes with 127 tests. - Confirm that the changed files type-check against Daytona SDK 0.203.0 types. - Review the PR against parent PR #10941 before it reaches `master`. ## Risks - The stream path changes behavior only when `useLogStream` is `true`. - A stream failure can add one reconnect attempt before the existing poll fallback. - The stream path has no command deadline because it supports long-lived commands. - No endpoint, stored data, telemetry shape, authentication rule, or result shape changes. ## Model Used OpenAI Codex, GPT-5, tool use and code execution. The context window and reasoning mode are not exposed by this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../sandbox-providers/daytona/src/manifest.ts | 6 + .../daytona/src/plugin.test.ts | 136 ++++++++++++++++ .../sandbox-providers/daytona/src/plugin.ts | 153 +++++++++++++++++- 3 files changed, 294 insertions(+), 1 deletion(-) diff --git a/packages/plugins/sandbox-providers/daytona/src/manifest.ts b/packages/plugins/sandbox-providers/daytona/src/manifest.ts index 38eedf238ac..7b52c685ee8 100644 --- a/packages/plugins/sandbox-providers/daytona/src/manifest.ts +++ b/packages/plugins/sandbox-providers/daytona/src/manifest.ts @@ -120,6 +120,12 @@ const manifest: PaperclipPluginManifestV1 = { "Whether to stop and later resume the sandbox across runs instead of deleting it on release.", default: false, }, + useLogStream: { + type: "boolean", + description: + "When true, a session command streams stdout and stderr from the Daytona callback log form and reads the exit code one time after the stream ends. When false, the command polls the exit code and reads the logs one time. Defaults to false.", + default: false, + }, }, }, }, diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index e37d2b348f1..7b2eb020c25 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -169,6 +169,7 @@ describe("Daytona sandbox provider plugin", () => { reuseLease: true, archiveOnRelease: false, useSessions: false, + useLogStream: false, }, }); }); @@ -1533,6 +1534,141 @@ describe("Daytona sandbox provider plugin", () => { expect(result).toMatchObject({ exitCode: null, timedOut: true }); expect(result!.stderr).toMatch(/timed out/); }); + + describe("log stream (useLogStream)", () => { + // A session exec params helper with the log-stream flag on. It streams + // stdout and stderr from the callback log form instead of the 50-ms poll. + const streamExecParams = (overrides: Record = {}) => + sessionExecParams({ + config: { timeoutMs: 300000, reuseLease: false, useSessions: true, useLogStream: true }, + ...overrides, + }); + + it("streams ordered stdout and stderr from the callback log form (test_log_stream_delivers_ordered_chunks)", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 0 }); + // The callback form emits stdout and stderr chunks in order. The plugin + // keeps each stream in its own arrival order. + sandbox.process.getSessionCommandLogs.mockImplementation( + async ( + _sid: string, + _cmdId: string, + onStdout?: (chunk: string) => void, + onStderr?: (chunk: string) => void, + ) => { + onStdout?.("out-1;"); + onStderr?.("err-1;"); + onStdout?.("out-2;"); + onStderr?.("err-2;"); + }, + ); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams()); + + // The callback stream form ran (four args), not the 50-ms snapshot poll. + expect(sandbox.process.getSessionCommandLogs).toHaveBeenCalledTimes(1); + const streamCall = sandbox.process.getSessionCommandLogs.mock.calls[0]!; + expect(typeof streamCall[2]).toBe("function"); + expect(typeof streamCall[3]).toBe("function"); + expect(result).toMatchObject({ + exitCode: 0, + timedOut: false, + stdout: "out-1;out-2;", + stderr: "err-1;err-2;", + }); + expect(typeof (result!.metadata as Record)?.durationMs).toBe("number"); + }); + + it("reads the exit code once after the stream ends (test_log_stream_reads_exit_code_once_after_stream_end)", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 5 }); + sandbox.process.getSessionCommandLogs.mockImplementation( + async (_sid: string, _cmdId: string, onStdout?: (chunk: string) => void) => { + onStdout?.("done"); + }, + ); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams()); + + // The exit code read runs one time after the stream promise resolves. + expect(sandbox.process.getSessionCommand).toHaveBeenCalledTimes(1); + expect(result).toMatchObject({ exitCode: 5, timedOut: false, stdout: "done" }); + }); + + it("falls back to the poll path when the stream promise rejects (test_log_stream_disconnect_rejects_and_falls_back_to_poll)", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + // The callback form disconnects and rejects. The snapshot form (no + // callbacks) serves the poll fallback read. + sandbox.process.getSessionCommandLogs.mockImplementation( + async ( + _sid: string, + _cmdId: string, + onStdout?: (chunk: string) => void, + ) => { + if (onStdout) { + onStdout("partial"); + throw new Error("socket error"); + } + return { stdout: "poll-out", stderr: "poll-err" }; + }, + ); + // The command still runs to its exit on the server, so the poll reads it. + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 9 }); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams()); + + // The poll fallback served the final result, and the command still + // yielded its exit code. + expect(result).toMatchObject({ exitCode: 9, timedOut: false, stdout: "poll-out", stderr: "poll-err" }); + // The snapshot form (two args) ran for the fallback read. + const snapshotCalls = sandbox.process.getSessionCommandLogs.mock.calls.filter( + (call) => call[2] === undefined, + ); + expect(snapshotCalls.length).toBeGreaterThanOrEqual(1); + }); + + it("drops the replayed prefix by byte offset on a reconnect (test_log_stream_reconnect_drops_replayed_prefix_by_byte_offset)", async () => { + process.env.DAYTONA_API_KEY = "host-key"; + const sandbox = createMockSandbox(); + let attempt = 0; + sandbox.process.getSessionCommandLogs.mockImplementation( + async ( + _sid: string, + _cmdId: string, + onStdout?: (chunk: string) => void, + onStderr?: (chunk: string) => void, + ) => { + attempt += 1; + if (attempt === 1) { + // First connection: deliver a prefix, then the socket drops. + onStdout?.("AAA"); + onStderr?.("EEE"); + throw new Error("socket error"); + } + // Reconnect: Daytona replays the whole log from byte 0, then the new + // tail. The plugin must drop the replayed prefix. + onStdout?.("AAA"); + onStdout?.("BBB"); + onStderr?.("EEE"); + onStderr?.("FFF"); + }, + ); + sandbox.process.getSessionCommand.mockResolvedValue({ id: "cmd-1", command: "", exitCode: 0 }); + mockGet.mockResolvedValue(sandbox); + + const result = await plugin.definition.onEnvironmentExecute?.(streamExecParams()); + + // The pre-disconnect bytes appear one time, not two. + expect(result).toMatchObject({ exitCode: 0, timedOut: false, stdout: "AAABBB", stderr: "EEEFFF" }); + expect(sandbox.process.getSessionCommandLogs).toHaveBeenCalledTimes(2); + }); + }); }); it("executes commands one-shot and returns combined output via stdout", async () => { diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 18306045b3c..b760f79b342 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -123,6 +123,7 @@ interface DaytonaDriverConfig { reuseLease: boolean; archiveOnRelease: boolean; useSessions: boolean; + useLogStream: boolean; } type WorkspaceSentinelResult = { @@ -229,6 +230,12 @@ function parseDriverConfig(raw: Record): DaytonaDriverConfig { // Daytona session per lease and dispatches every command into it. The flag // stays default off until a live leak soak passes. useSessions: raw.useSessions === true, + // Log-stream opt-in. Default OFF. When off, the session dispatch polls the + // exit code every 50 ms and then reads the logs one time. When on, the + // dispatch streams stdout and stderr from the callback log form and reads + // the exit code one time after the stream ends. The flag stays default off + // until a live soak passes. + useLogStream: raw.useLogStream === true, }; } @@ -1473,6 +1480,130 @@ function sleep(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); } +// Backoff delays for the exit-code read after the log stream ends. The live +// spike measured the exit code available within one poll (91-202 ms), so the +// first read almost always holds the code. These delays cover the rare case +// where the first read has no code yet. +const SESSION_EXIT_CODE_RETRY_DELAYS_MS = [50, 100, 200]; + +// A bounded reconnect for the log stream. A disconnect settles the stream +// promise as a rejection while the command still runs on the server. One +// reconnect replays the log from byte 0; the stream buffer drops the replayed +// prefix by byte offset. After this many reconnects the dispatch falls back to +// the poll path. +const MAX_SESSION_STREAM_RECONNECTS = 1; + +// Buffers the stdout and stderr of one session command from the callback log +// stream, and drops a replayed prefix by byte offset. +// +// The Daytona callback stream replays the whole log from byte 0 after a +// reconnect (it does not resume from an offset and does not omit earlier +// bytes). So the buffer tracks the byte count it already holds per stream and +// drops any replayed bytes that fall before that count. The dedupe runs at the +// byte level, because Daytona replays the log byte-for-byte. The SDK keeps each +// multibyte UTF-8 character whole per chunk and per stream, so the delivered +// byte count always lands on a character boundary and the byte-offset split is +// safe. +// +// The buffer stores each new tail as a separate chunk and joins the chunks one +// time at read. It does not copy the earlier output on each append, so total +// buffering work stays linear in the output size, not quadratic. +function createSessionStreamBuffer() { + const streams = { + stdout: { chunks: [] as Buffer[], length: 0, connectionBytes: 0 }, + stderr: { chunks: [] as Buffer[], length: 0, connectionBytes: 0 }, + }; + + function append( + stream: { chunks: Buffer[]; length: number; connectionBytes: number }, + chunk: string, + ): void { + const buf = Buffer.from(chunk, "utf8"); + const start = stream.connectionBytes; + stream.connectionBytes = start + buf.length; + // The whole chunk falls before the delivered byte count, so it is a replay. + if (start + buf.length <= stream.length) { + return; + } + // Keep only the new tail. When the whole chunk is new, `start >= + // stream.length` and the tail is the whole chunk. When the chunk straddles + // the delivered byte count, the tail starts after the replayed prefix. + const tail = start >= stream.length ? buf : buf.subarray(stream.length - start); + stream.chunks.push(tail); + stream.length += tail.length; + } + + return { + onStdout: (chunk: string) => append(streams.stdout, chunk), + onStderr: (chunk: string) => append(streams.stderr, chunk), + // Reset the per-connection read cursors after a reconnect, so the replayed + // prefix drops against the already-delivered byte count. + resetConnectionCursors(): void { + streams.stdout.connectionBytes = 0; + streams.stderr.connectionBytes = 0; + }, + get stdout(): string { + return Buffer.concat(streams.stdout.chunks).toString("utf8"); + }, + get stderr(): string { + return Buffer.concat(streams.stderr.chunks).toString("utf8"); + }, + }; +} + +type SessionLogStreamResult = + | { ok: true; stdout: string; stderr: string } + | { ok: false }; + +// Stream stdout and stderr of one session command from the callback log form. +// The stream buffer drops a replayed prefix by byte offset on a reconnect. A +// disconnect rejects the stream promise; the dispatch reconnects a bounded +// number of times, then reports failure so the caller falls back to the poll +// path. +async function runSessionLogStream( + sandbox: Sandbox, + sessionId: string, + commandId: string, +): Promise { + const buffer = createSessionStreamBuffer(); + let reconnects = 0; + while (true) { + try { + await sandbox.process.getSessionCommandLogs(sessionId, commandId, buffer.onStdout, buffer.onStderr); + return { ok: true, stdout: buffer.stdout, stderr: buffer.stderr }; + } catch { + if (reconnects >= MAX_SESSION_STREAM_RECONNECTS) { + return { ok: false }; + } + reconnects += 1; + buffer.resetConnectionCursors(); + } + } +} + +// Read the exit code one time after the log stream ends. The exit code is +// available within one poll, so the first read almost always holds it. Add a +// small bounded retry with backoff only for the rare case where the first read +// has no code yet. Return null when no read holds a numeric code. +async function readSessionExitCode( + sandbox: Sandbox, + sessionId: string, + commandId: string, +): Promise { + const first = await sandbox.process.getSessionCommand(sessionId, commandId); + if (typeof first.exitCode === "number") { + return first.exitCode; + } + for (const delayMs of SESSION_EXIT_CODE_RETRY_DELAYS_MS) { + await sleep(delayMs); + const status = await sandbox.process.getSessionCommand(sessionId, commandId); + if (typeof status.exitCode === "number") { + return status.exitCode; + } + } + return null; +} + // Dispatch one user command into the persistent session and return its true // stdout and stderr. // @@ -1532,9 +1663,29 @@ async function executeInSession( ); const commandId = dispatched.cmdId; + // Log-stream path (opt-in). Stream stdout and stderr from the callback log + // form, then read the exit code one time. On a stream failure, fall through + // to the poll path below, because the command still runs to its exit on the + // server. + if (config.useLogStream) { + const streamResult = await runSessionLogStream(sandbox, sessionId, commandId); + if (streamResult.ok) { + const exitCode = await readSessionExitCode(sandbox, sessionId, commandId); + const durationMs = timingNow() - execStart; + return { + exitCode, + timedOut: false, + stdout: streamResult.stdout, + stderr: streamResult.stderr, + metadata: { durationMs }, + }; + } + } + // Poll for the exit code; the SDK has no wait method. The poll deadline uses // the wall clock, separate from the injected timing clock that measures the - // reported `durationMs`. + // reported `durationMs`. The poll path is the default when the log stream is + // off, and the fallback when the log stream fails. const deadlineMs = Date.now() + effectiveTimeoutMs; let exitCode: number | null = null; while (true) { From b67c512f8228f0bd5ddcc4cfd92676ba1ab31cf7 Mon Sep 17 00:00:00 2001 From: scotttong Date: Thu, 6 Aug 2026 23:18:32 -0700 Subject: [PATCH 016/311] =?UTF-8?q?feat(ui):=20live=20run=20label=20?= =?UTF-8?q?=E2=86=92=20run=20detail,=20running=20row=20=E2=86=92=20task=20?= =?UTF-8?q?detail=20(#11034)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The agent detail page has a Dashboard tab. The Dashboard tab shows a "Live Run" section for the agent's current heartbeat. > - The "Live Run" section has two clickable pieces: the section heading and the running row. Both pieces linked to the same run detail page. > - Two controls that go to the same place waste a navigation affordance and hide the task the agent runs. > - This pull request splits the two destinations. The heading goes to the run. The running row goes to the task. > - The benefit is that a user reaches the run internals from the label and the work item from the row, in one click each. ## Linked Issues or Issue Description No public GitHub issue exists for this change. The description follows the enhancement issue template. **What existing behavior does this improve?** The "Live Run" section on the agent detail page, Dashboard tab (the `LatestRunCard` component in `ui/src/pages/AgentDetail.tsx`). **Current behavior** The "Live Run" heading and the running row both link to the run detail page (`/agents/:agentId/runs/:runId`). The row shows the run code and an invocation-source chip. There is a separate "View details →" link that also goes to the run detail page. A user cannot reach the task the run works on from this section. **Proposed behavior** The heading becomes a link to the run detail page and appends the short run code, shown as `Live Run · `. The redundant "View details →" link is removed. The running row links to the task detail page when the run's context snapshot resolves to a known issue, and the row then shows the task status glyph, the task slug, and the task title. A pure timer heartbeat with no resolvable task keeps the previous behavior: run code plus source chip, linking to the run detail page. **Reason and benefit** The heading and the row now go to distinct, intuitive destinations. A user reaches the run internals from the label and the work item from the row, each in a single click. The fallback keeps heartbeats with no task readable and avoids a blank or broken row. ## What Changed - Made the "Live Run" / "Latest Run" heading a `Link` to the run detail page and appended the short run code (`run.id.slice(0, 8)`) in a mono span, formatted `Live Run · `. Kept the pulsing live dot. - Removed the redundant "View details →" link. - Changed the running row `Link` target to the task detail page (`/issues/:identifier`) when a task resolves, falling back to the run detail page otherwise. - Resolved the task from the run context snapshot (`contextSnapshot.issueId`, falling back to `contextSnapshot.taskId`) against a `Map` of the agent's assigned issues threaded in from `AgentOverview`. - When a task resolves, replaced the run code and source chip in the row with the task status glyph (`StatusGlyph`), the task slug, and the task title. Kept the running spinner, the run status badge, and the timestamp. ## Verification - `pnpm check:token-gates` → 3/3 gates clean. - `pnpm --filter ui typecheck` → passes. - `pnpm --filter ui exec vitest run src/pages/AgentDetail.progress.test.ts src/pages/AgentDetail.instructions.test.tsx` → 10/10 pass. - Manual (needs a reviewer with a browser): open an agent detail page → Dashboard tab. - For a live issue-execution run: the heading reads `Live Run · ` and opens the run detail page; the row shows the task status icon, slug, and title and opens the task detail page. - For a pure timer heartbeat with no task: the row falls back to run code + source chip and opens the run detail page. No blank row. - Confirm both states in light and dark mode. ## Risks Low risk. The change is presentational and scoped to one component. The task lookup is defensive: it reads the context snapshot with a fallback key and only renders the task row when the issue is present in the already-loaded assigned-issue set, so an unknown or missing issue degrades to the previous run-detail behavior rather than breaking. ## Model Used - Provider: Anthropic (Claude). - Model: claude-opus-4-8 (Opus 4.8). - Context window: 200K. - Reasoning mode: extended thinking, tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Claude Opus 4.8 --- ui/src/pages/AgentDetail.liveRun.test.ts | 118 ++++++++++++++++ ui/src/pages/AgentDetail.tsx | 167 ++++++++++++++++++----- 2 files changed, 251 insertions(+), 34 deletions(-) create mode 100644 ui/src/pages/AgentDetail.liveRun.test.ts diff --git a/ui/src/pages/AgentDetail.liveRun.test.ts b/ui/src/pages/AgentDetail.liveRun.test.ts new file mode 100644 index 00000000000..0f606690b79 --- /dev/null +++ b/ui/src/pages/AgentDetail.liveRun.test.ts @@ -0,0 +1,118 @@ +import { describe, expect, it } from "vitest"; +import type { HeartbeatRun } from "@paperclipai/shared"; +import { getRunSnapshotIssueId, resolveLatestRunNavigation, type LatestRunIssue } from "./AgentDetail"; + +const AGENT_ID = "agent-1"; + +function makeRun(overrides: Partial = {}): HeartbeatRun { + return { + id: "d3bd37f9-1111-2222-3333-444455556666", + contextSnapshot: null, + ...overrides, + } as HeartbeatRun; +} + +function issuesMap(...issues: LatestRunIssue[]): Map { + return new Map(issues.map((i) => [i.id, i])); +} + +describe("resolveLatestRunNavigation", () => { + const runHref = `/agents/${AGENT_ID}/runs/d3bd37f9-1111-2222-3333-444455556666`; + + it("resolves the task from contextSnapshot.issueId and points the row at the task detail page", () => { + const issue: LatestRunIssue = { id: "issue-7", title: "Fix the thing", status: "in_progress", identifier: "PAP-7" }; + const run = makeRun({ contextSnapshot: { issueId: "issue-7" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap(issue)); + + expect(nav.task).toBe(issue); + expect(nav.runHref).toBe(runHref); + expect(nav.rowHref).toBe("/issues/PAP-7"); + }); + + it("falls back to contextSnapshot.taskId when issueId is absent (older snapshots)", () => { + const issue: LatestRunIssue = { id: "issue-9", title: "Legacy", status: "todo", identifier: "PAP-9" }; + const run = makeRun({ contextSnapshot: { taskId: "issue-9" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap(issue)); + + expect(nav.task).toBe(issue); + expect(nav.rowHref).toBe("/issues/PAP-9"); + }); + + it("prefers issueId over taskId when both are present", () => { + const wanted: LatestRunIssue = { id: "issue-a", title: "A", status: "in_progress", identifier: "PAP-A" }; + const other: LatestRunIssue = { id: "issue-b", title: "B", status: "in_progress", identifier: "PAP-B" }; + const run = makeRun({ contextSnapshot: { issueId: "issue-a", taskId: "issue-b" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap(wanted, other)); + + expect(nav.task).toBe(wanted); + expect(nav.rowHref).toBe("/issues/PAP-A"); + }); + + it("uses the issue id when the resolved task has no identifier slug", () => { + const issue: LatestRunIssue = { id: "issue-noident", title: "No slug", status: "in_progress", identifier: null }; + const run = makeRun({ contextSnapshot: { issueId: "issue-noident" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap(issue)); + + expect(nav.rowHref).toBe("/issues/issue-noident"); + }); + + it("falls back to the run detail page for a pure timer heartbeat with no snapshot", () => { + const run = makeRun({ contextSnapshot: null }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap()); + + expect(nav.task).toBeUndefined(); + expect(nav.runHref).toBe(runHref); + expect(nav.rowHref).toBe(runHref); + }); + + it("falls back to the run detail page when the snapshot issue is absent from the lookup (LatestRunCard fills this gap via a targeted fetch)", () => { + const run = makeRun({ contextSnapshot: { issueId: "issue-missing" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap()); + + expect(nav.task).toBeUndefined(); + expect(nav.rowHref).toBe(runHref); + }); + + it("resolves an issue fetched directly (outside the bounded assigned-issues page) once folded into the lookup", () => { + // Mirrors LatestRunCard folding a directly-fetched fallback issue into the + // map keyed by the snapshot id, so a live run whose task sits beyond the + // server page limit still links to the task detail page. + const fetched: LatestRunIssue = { id: "issue-far", title: "Far task", status: "in_progress", identifier: "PAP-999" }; + const run = makeRun({ contextSnapshot: { issueId: "issue-far" } as HeartbeatRun["contextSnapshot"] }); + + const nav = resolveLatestRunNavigation(run, AGENT_ID, issuesMap(fetched)); + + expect(nav.task).toBe(fetched); + expect(nav.rowHref).toBe("/issues/PAP-999"); + }); +}); + +describe("getRunSnapshotIssueId", () => { + it("reads issueId from the snapshot", () => { + expect(getRunSnapshotIssueId(makeRun({ contextSnapshot: { issueId: "issue-7" } as HeartbeatRun["contextSnapshot"] }))).toBe("issue-7"); + }); + + it("falls back to taskId for older snapshots", () => { + expect(getRunSnapshotIssueId(makeRun({ contextSnapshot: { taskId: "issue-9" } as HeartbeatRun["contextSnapshot"] }))).toBe("issue-9"); + }); + + it("prefers issueId over taskId", () => { + expect( + getRunSnapshotIssueId(makeRun({ contextSnapshot: { issueId: "issue-a", taskId: "issue-b" } as HeartbeatRun["contextSnapshot"] })), + ).toBe("issue-a"); + }); + + it("returns undefined for a pure timer heartbeat with no snapshot", () => { + expect(getRunSnapshotIssueId(makeRun({ contextSnapshot: null }))).toBeUndefined(); + }); + + it("coerces non-string ids to strings", () => { + expect(getRunSnapshotIssueId(makeRun({ contextSnapshot: { issueId: 42 } as unknown as HeartbeatRun["contextSnapshot"] }))).toBe("42"); + }); +}); diff --git a/ui/src/pages/AgentDetail.tsx b/ui/src/pages/AgentDetail.tsx index 4f7a21986c6..4e21a7d2809 100644 --- a/ui/src/pages/AgentDetail.tsx +++ b/ui/src/pages/AgentDetail.tsx @@ -41,6 +41,7 @@ import { StatusBadge } from "../components/StatusBadge"; import { MarkdownBody } from "../components/MarkdownBody"; import { CopyText } from "../components/CopyText"; import { EntityRow } from "../components/EntityRow"; +import { StatusGlyph } from "../components/StatusGlyph"; import { MembershipAction } from "../components/MembershipAction"; import { StarToggle } from "../components/StarToggle"; import { Identity } from "../components/Identity"; @@ -1429,21 +1430,81 @@ function SummaryRow({ label, children }: { label: string; children: React.ReactN ); } -function LatestRunCard({ runs, agentId }: { runs: HeartbeatRun[]; agentId: string }) { - if (runs.length === 0) return null; +export type LatestRunIssue = { id: string; title: string; status: string; identifier?: string | null }; - const sorted = [...runs].sort( - (a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime() +/** + * The id of the issue a run works on, read from its context snapshot. Newer + * snapshots use `issueId`; older ones use `taskId`. Returns undefined for pure + * timer heartbeats that carry no task reference. + */ +export function getRunSnapshotIssueId( + run: Pick, +): string | undefined { + const ctx = run.contextSnapshot as Record | null; + const issueId = ctx?.issueId ?? ctx?.taskId; + return issueId ? String(issueId) : undefined; +} + +/** + * Resolve the Live Run section's two navigation destinations and the task (if + * any) the run works on. The run→task link lives in the run's context snapshot + * (`issueId`, falling back to `taskId` for older snapshots); the `HeartbeatRun` + * itself doesn't carry the issue id. The heading always links to the run detail + * page; the running row links to the task detail page when the snapshot resolves + * to a known issue, otherwise falls back to the run detail page (pure timer + * heartbeats or an issue that can't be resolved). + */ +export function resolveLatestRunNavigation( + run: Pick, + agentId: string, + issuesById: Map, +): { task: LatestRunIssue | undefined; runHref: string; rowHref: string } { + const issueId = getRunSnapshotIssueId(run); + const task = issueId ? issuesById.get(issueId) : undefined; + const runHref = `/agents/${agentId}/runs/${run.id}`; + const rowHref = task ? `/issues/${task.identifier ?? task.id}` : runHref; + return { task, runHref, rowHref }; +} + +function LatestRunCard({ + runs, + agentId, + issuesById, +}: { + runs: HeartbeatRun[]; + agentId: string; + issuesById: Map; +}) { + const sorted = useMemo( + () => + [...runs].sort( + (a, b) => new Date(b.createdAt).getTime() - new Date(a.createdAt).getTime() + ), + [runs] ); const liveRun = sorted.find((r) => r.status === "running" || r.status === "queued"); const run = liveRun ?? sorted[0]; - const isLive = run.status === "running" || run.status === "queued"; - const statusInfo = runStatusIcons[run.status] ?? { icon: Clock, color: "text-neutral-400" }; - const StatusIcon = statusInfo.icon; - const summaryRaw = run.resultJson - ? String((run.resultJson as Record).summary ?? (run.resultJson as Record).result ?? "") - : run.error ?? ""; + + // The assigned-issues list this card resolves against is bounded (server page + // limit), so a live run can reference a valid issue that isn't on the loaded + // page. When the snapshot points at an issue we don't already have, fetch it + // directly so the running row always links to the task rather than falling + // back to run metadata. `enabled` keeps this a no-op for the common case. + const snapshotIssueId = run ? getRunSnapshotIssueId(run) : undefined; + const needsFallbackFetch = !!snapshotIssueId && !issuesById.has(snapshotIssueId); + const { data: fallbackIssue } = useQuery({ + queryKey: queryKeys.issues.detail(snapshotIssueId ?? "__none__"), + queryFn: () => issuesApi.get(snapshotIssueId as string), + enabled: needsFallbackFetch, + staleTime: 30_000, + }); + + const summaryRaw = run + ? run.resultJson + ? String((run.resultJson as Record).summary ?? (run.resultJson as Record).result ?? "") + : run.error ?? "" + : ""; // Extract a clean 2-3 line excerpt: first non-empty, non-header, non-list-mark lines const summary = useMemo(() => { @@ -1463,28 +1524,48 @@ function LatestRunCard({ runs, agentId }: { runs: HeartbeatRun[]; agentId: strin return excerpt.join(" "); }, [summaryRaw]); + if (!run) return null; + + const isLive = run.status === "running" || run.status === "queued"; + // Fold any directly-fetched fallback issue into the lookup, keyed by the same + // snapshot id used to resolve the row so it hits regardless of id-vs-slug. + const effectiveIssuesById = + fallbackIssue && snapshotIssueId + ? new Map(issuesById).set(snapshotIssueId, { + id: fallbackIssue.id, + title: fallbackIssue.title, + status: fallbackIssue.status, + identifier: fallbackIssue.identifier, + }) + : issuesById; + const { task, runHref, rowHref } = resolveLatestRunNavigation(run, agentId, effectiveIssuesById); + const statusInfo = runStatusIcons[run.status] ?? { icon: Clock, color: "text-neutral-400" }; + const StatusIcon = statusInfo.icon; + return (
-

- {isLive && ( - - - - - )} - {isLive ? "Live Run" : "Latest Run"} -

- View details → +

+ {isLive && ( + + + + + )} + {isLive ? "Live Run" : "Latest Run"} + + · {run.id.slice(0, 8)} + +

- {run.id.slice(0, 8)} - - {sourceLabels[run.invocationSource] ?? run.invocationSource} - + {task ? ( + <> + + + {task.identifier ?? task.id.slice(0, 8)} + + {task.title} + + ) : ( + <> + {run.id.slice(0, 8)} + + {sourceLabels[run.invocationSource] ?? run.invocationSource} + + + )} {relativeTime(run.createdAt)}
@@ -1533,10 +1626,16 @@ function AgentOverview({ agentId: string; agentRouteId: string; }) { + const issuesById = useMemo(() => { + const map = new Map(); + for (const issue of assignedIssues) map.set(issue.id, issue); + return map; + }, [assignedIssues]); + return (
{/* Latest Run */} - + {/* Charts */}
From 11e56654f8492f78ec56986e51be3d33f8fd6873 Mon Sep 17 00:00:00 2001 From: Tonio Date: Thu, 6 Aug 2026 23:35:05 -0700 Subject: [PATCH 017/311] feat(ui): port onboarding flow from prototype; add cloud + local variants (#10786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - First-run onboarding is the subsystem that turns a brand-new install into a working company: it creates the company, its goal, a lead agent, and that agent's first task > - The existing `OnboardingWizard` carried all of that wiring correctly, but its UI had drifted from the current design direction, and a separate design prototype (`paperclip-onboard`) existed as a standalone visual mock with no backend > - Porting the prototype's *logic* would have thrown away working, well-tested backend orchestration; leaving the two apart meant the design never shipped > - Separately, cloud and local (self-hosted) installs need meaningfully different first runs β€” local has no sign-in and must let the user pick a locally-installed CLI adapter β€” so a single linear wizard could not serve both > - This pull request rebuilds the presentational layer from the prototype on top of the existing backend orchestration, and splits it into two thin flow containers over a shared core > - The benefit is that the shipped onboarding matches the intended design, cloud and local can diverge without duplicating logic, and each can later ship to a different app version while sharing one set of step components ## Linked Issues or Issue Description No existing issue β€” describing inline (feature request). **What problem does this solve?** Onboarding is the first thing a new user sees, and the shipped wizard had drifted from the current design. In parallel, cloud and local installs need different first-run paths: local has no hosted sign-in, and its agent runs on a CLI adapter installed on the user's machine, which the cloud path never has to ask about. There was no way to express that difference without either forking the whole wizard or bolting conditionals onto a single linear flow. **Proposed solution** Extract the onboarding step views and shell into a shared core, then compose two thin flow containers (cloud and local) over it. Keep all backend orchestration in the existing `useOnboardingFlow` hook so no working logic is rewritten. **Alternatives considered** - *Single flow with a `variant` prop* β€” most DRY, but the two flows are intended to ship on different app versions, and a shared file would have to be split later anyway. - *Two fully independent copies* β€” simplest per-flow, but every shared refinement (spacing, motion, copy) would have to be made twice and would drift. ## What Changed - **Shared core** under `ui/src/components/onboarding/`: `OnboardingScaffold` owns the full-screen shell and the single `AnimatePresence` step crossfade, so both flows transition identically; step views (Start / Company / Agent / Task), `FooterNav`, `AgentPreview` and the motion constants are extracted for reuse. - **`CloudOnboardingFlow`** β€” `start β†’ company β†’ agent β†’ task`; mounted in the real app via `OnboardingWizardVariant`. Behaviour matches the retired wizard, including `previewMock` and the existing-company ("add an agent") entry point. - **`LocalOnboardingFlow`** β€” skips sign-in and adds an optional email ask (with a privacy assurance), a local model/adapter step that hires with `requireEnvProbe: true`, and a "star us on GitHub" interstitial before completing. **Harness-only for now** β€” the real app still mounts the cloud flow. - **Deleted `OnboardingWizard.tsx`** (1,786 lines); updated its Storybook stories and the `OnboardingWizardVariant` test to the new components. - **Orbiting 3D paperclip backdrop** behind the auth and welcome screens (`three`), code-split so it only downloads on those screens; honours `prefers-reduced-motion` and disposes its GL context on unmount. - **`motion`** added for step transitions and the agent-capsule choreography. - Visual values routed through design tokens per `DESIGN.md`; `Stepper` generalized to take a step total (backward compatible); `/design-guide` page and the component index updated. - **Standalone preview harness** (`ui/onboarding-preview.html`) with `?flow=` and `?step=` for backend-free review, wired as a second Vite rollup input. - **Adapter env probe bound to the adapter it ran against.** `hireLeadAgent` reused `adapterEnvResult` for any adapter, so when a hire failed and the user picked a *different* local adapter and retried, the previous adapter's verdict satisfied the `requireEnvProbe` guard while the hire posted the new adapter's config β€” hiring it unprobed. The cache is now keyed on the adapter type plus the exact config posted to the test endpoint, the config is built once and shared by probe and hire, a failed probe clears the cache, and `clearAdapterEnvResult()` (called on adapter change) stops the step displaying a stale verdict. Cloud is unaffected β€” it hires with `requireEnvProbe: false`. Reported by Greptile. - **E2E specs re-pointed at the new flow.** Four specs still drove the deleted wizard (`onboarding`, `conference-room-typing-intro`, `planning-mode-visual-verification`, `nux-phase4-screenshots`) and failed with `element(s) not found` on `"Name your company"` / `input[placeholder="Acme Corp"]`. Rather than repeat the new drive sequence four times, `tests/e2e/onboarding-flow.ts` adds one driver per step (`startCloudOnboarding`, `completeCompanyStep`, `completeAgentStep`, `completeTaskStep`, `completeCloudOnboarding`) and the specs import it, so the next flow change touches a single file. Two now-dead `**/test-environment` route stubs went with it β€” the cloud flow hires with `requireEnvProbe: false`, so that probe never fires. ## Verification - `pnpm --filter @paperclipai/ui typecheck` β€” clean. - `npx vitest run` over the onboarding suites (`OnboardingWizardVariant`, `AgentCapsule`, `onboarding-launch`, `onboarding-goal`, `onboarding-route`, `onboarding-adapter-config`) β€” 33 tests pass. - `pnpm --filter @paperclipai/ui build` β€” succeeds; the three.js chunk splits out separately (522 kB raw / 133 kB gzip) rather than entering the main bundle. - Both flows driven end-to-end in the preview harness in `previewMock` (no database writes), plus the cloud flow rendered in the real authenticated app at `/onboarding` to confirm the mount swap. - The four re-pointed e2e specs pass locally against the new flow. - New `ui/src/hooks/useOnboardingFlow.test.tsx` β€” 4 cases pinning the adapter-probe cache (switch-adapter retry, cold path, explicit clear, and the cloud flow's `requireEnvProbe: false`). Verified non-vacuous: the switch-adapter case fails against the pre-fix code. - Rebased onto current `master`; `pnpm-lock.yaml` is deliberately **not** committed β€” `.github/workflows/pr.yml` regenerates it when a manifest changes and shares it with downstream jobs as the `pr-lockfile` artifact. ## Risks - **Deleting `OnboardingWizard.tsx` is the one change that alters existing app behaviour.** The cloud flow is intended to be behaviour-equivalent, and its entry points are covered by the updated `OnboardingWizardVariant` test, but this is the area to review most closely. - **Conflict risk with open PRs that touch the old wizard**: #9900, #9501, #8982 and #6636 all modify `ui/src/components/OnboardingWizard.tsx`, which this PR removes. Whichever lands second will need its change re-applied to the new step components. Flagging so ordering can be decided deliberately. - **New dependencies**: `motion` and `three` (+ `@types/three`). `three` is large, so it is lazily imported and code-split β€” it does not affect the main bundle. Both are MIT. - The **local flow is not reachable in the app** yet (harness/canary only), so it carries no runtime risk today; wiring it up is a follow-up. - The auth screens remain **presentational only** β€” they are not wired to real auth, unchanged from before this PR. - **Pre-existing, not introduced here:** `OnboardingWizardVariant` renders outside `` in `App.tsx`, so its `useParams()` never resolves `:companyPrefix` and `/{prefix}/onboarding` opens the welcome screen instead of jumping to the agent step. `master` has the identical structure, so this PR faithfully ports existing behaviour; the working "add an agent" entry is the launcher card behind the overlay, which is what the screenshot spec drives. Worth a separate fix. ## Model Used Claude Opus 5 (`claude-opus-5`) via Claude Code, with extended thinking and tool use (repo search/edit, local test + build execution, and browser-driven visual verification of the rendered flows). Portions of the session also ran on `claude-opus-4-8` and `claude-fable-5`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge πŸ€– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 --- .claude/launch.json | 7 + .../references/component-index.md | 18 + .../e2e/conference-room-typing-intro.spec.ts | 66 +- tests/e2e/nux-phase4-screenshots.spec.ts | 141 +- tests/e2e/onboarding-flow.ts | 134 ++ tests/e2e/onboarding.spec.ts | 79 +- .../planning-mode-visual-verification.spec.ts | 53 +- ui/onboarding-preview.html | 22 + ui/package.json | 5 +- ui/src/components/AgentCapsule.tsx | 88 +- ui/src/components/OnboardingWizard.tsx | 1789 ----------------- .../OnboardingWizardVariant.test.tsx | 80 +- ui/src/components/OnboardingWizardVariant.tsx | 72 +- ui/src/components/onboarding/AgentPreview.tsx | 47 + .../onboarding/CloudOnboardingFlow.tsx | 165 ++ ui/src/components/onboarding/FooterNav.tsx | 50 + .../onboarding/GithubStarInterstitial.tsx | 44 + .../onboarding/LocalOnboardingFlow.tsx | 228 +++ .../onboarding/OnboardingAuthBackdrop.tsx | 102 + .../onboarding/OnboardingAuthScreens.tsx | 163 ++ .../onboarding/OnboardingPrimitives.tsx | 188 ++ .../onboarding/OnboardingScaffold.tsx | 42 + .../onboarding/PaperclipOrbit3D.tsx | 335 +++ .../components/onboarding/onboarding-data.ts | 79 + .../onboarding/onboarding-motion.ts | 49 + .../onboarding/steps/AdapterStep.tsx | 144 ++ .../components/onboarding/steps/AgentStep.tsx | 102 + .../onboarding/steps/CompanyStep.tsx | 87 + .../components/onboarding/steps/EmailStep.tsx | 62 + .../components/onboarding/steps/StartStep.tsx | 70 + .../components/onboarding/steps/TaskStep.tsx | 131 ++ ui/src/hooks/useOnboardingFlow.test.tsx | 162 ++ ui/src/hooks/useOnboardingFlow.ts | 483 +++++ ui/src/index.css | 72 +- ui/src/lib/onboarding-adapter-config.test.ts | 50 + ui/src/lib/onboarding-adapter-config.ts | 63 + ui/src/lib/onboarding-constants.ts | 16 + ui/src/onboarding-preview-main.tsx | 112 ++ ui/src/pages/DesignGuide.tsx | 97 + .../stories/data-viz-misc.stories.tsx | 31 +- ui/vite.config.ts | 10 +- 41 files changed, 3648 insertions(+), 2090 deletions(-) create mode 100644 tests/e2e/onboarding-flow.ts create mode 100644 ui/onboarding-preview.html delete mode 100644 ui/src/components/OnboardingWizard.tsx create mode 100644 ui/src/components/onboarding/AgentPreview.tsx create mode 100644 ui/src/components/onboarding/CloudOnboardingFlow.tsx create mode 100644 ui/src/components/onboarding/FooterNav.tsx create mode 100644 ui/src/components/onboarding/GithubStarInterstitial.tsx create mode 100644 ui/src/components/onboarding/LocalOnboardingFlow.tsx create mode 100644 ui/src/components/onboarding/OnboardingAuthBackdrop.tsx create mode 100644 ui/src/components/onboarding/OnboardingAuthScreens.tsx create mode 100644 ui/src/components/onboarding/OnboardingPrimitives.tsx create mode 100644 ui/src/components/onboarding/OnboardingScaffold.tsx create mode 100644 ui/src/components/onboarding/PaperclipOrbit3D.tsx create mode 100644 ui/src/components/onboarding/onboarding-data.ts create mode 100644 ui/src/components/onboarding/onboarding-motion.ts create mode 100644 ui/src/components/onboarding/steps/AdapterStep.tsx create mode 100644 ui/src/components/onboarding/steps/AgentStep.tsx create mode 100644 ui/src/components/onboarding/steps/CompanyStep.tsx create mode 100644 ui/src/components/onboarding/steps/EmailStep.tsx create mode 100644 ui/src/components/onboarding/steps/StartStep.tsx create mode 100644 ui/src/components/onboarding/steps/TaskStep.tsx create mode 100644 ui/src/hooks/useOnboardingFlow.test.tsx create mode 100644 ui/src/hooks/useOnboardingFlow.ts create mode 100644 ui/src/lib/onboarding-adapter-config.test.ts create mode 100644 ui/src/lib/onboarding-adapter-config.ts create mode 100644 ui/src/lib/onboarding-constants.ts create mode 100644 ui/src/onboarding-preview-main.tsx diff --git a/.claude/launch.json b/.claude/launch.json index 7cbc05c40a6..1af7d4ce098 100644 --- a/.claude/launch.json +++ b/.claude/launch.json @@ -7,6 +7,13 @@ "runtimeArgs": ["-c", "TMPDIR=/tmp pnpm dev"], "port": 3108, "autoPort": false + }, + { + "name": "ui-preview", + "runtimeExecutable": "pnpm", + "runtimeArgs": ["--filter", "@paperclipai/ui", "exec", "vite"], + "port": 5188, + "autoPort": true } ] } diff --git a/.claude/skills/design-guide/references/component-index.md b/.claude/skills/design-guide/references/component-index.md index 88e75178b31..93c43b1754a 100644 --- a/.claude/skills/design-guide/references/component-index.md +++ b/.claude/skills/design-guide/references/component-index.md @@ -188,6 +188,24 @@ Use in property rows, comment headers, assignee displays, and anywhere a user/ag **File:** `CompanySwitcher.tsx` **Usage:** Company selector dropdown in sidebar header. +### AgentCapsule + +**File:** `AgentCapsule.tsx` +**Props:** `state: "slot" | "configured" | "online"`, `gradient?: 1–10`, `size?: "sm" | "md" | "lg" | {width,height}`, `glow?: "green" | "blue"` +**Usage:** The brand "capsule is the agent" pill; evolves in place across onboarding steps. Fill uses `--agent-Na/Nb` gradient tokens; honors `prefers-reduced-motion`. + +### Onboarding primitives (OnboardingCard, OnboardingHeading, Stepper, Chip, ChoiceCard, ConnectorRow) + +**File:** `onboarding/OnboardingPrimitives.tsx` +**Usage:** Presentational pieces for the full-screen onboarding flow (`onboarding/OnboardingFlow.tsx`): the 560px card frame (`--sz-560px`), display heading + lede (text-4xl), 3-segment stepper, selectable mission chips, selectable choice cards, and connector rows. Bespoke dimensions route through verbatim `--sz-*` tokens; fields inside the flow use the shared Input/Textarea/Select/Label primitives. + +```tsx + + + + +``` + --- ## Layout Components diff --git a/tests/e2e/conference-room-typing-intro.spec.ts b/tests/e2e/conference-room-typing-intro.spec.ts index e1a7096ed6b..49364e6d8d3 100644 --- a/tests/e2e/conference-room-typing-intro.spec.ts +++ b/tests/e2e/conference-room-typing-intro.spec.ts @@ -1,32 +1,25 @@ import { test, expect } from "@playwright/test"; +import { completeCloudOnboarding, HIRING_TASK_TITLE } from "./onboarding-flow"; /** - * E2E: post-wizard onboarding launch. + * E2E: post-onboarding launch. * - * Completing the onboarding wizard now creates the first assigned task and - * lands the user on the company dashboard. The chat intro still has unit - * coverage in BoardChat tests; the wizard handoff no longer routes there. + * Completing the onboarding flow creates the first assigned task and lands the + * user on the company dashboard. The chat intro still has unit coverage in + * BoardChat tests; the onboarding handoff no longer routes there. */ const COMPANY_NAME = `E2E-TypingIntro-${Date.now()}`; -const MISSION = "Verify the dashboard launch survives the wizard handoff."; -const FIRST_TASK_TITLE = "Hire your first engineer and create a hiring plan"; +const MISSION = "Verify the dashboard launch survives the onboarding handoff."; -test.describe("Dashboard launch after onboarding wizard", () => { +test.describe("Dashboard launch after onboarding", () => { test("creates the first task and opens the dashboard", async ({ page, baseURL, }) => { - // Intercept env-test β†’ instant pass (avoid running a real CLI check). - await page.route("**/test-environment", (route) => - route.fulfill({ - contentType: "application/json", - body: JSON.stringify({ status: "pass", checks: [] }), - }), - ); - // Intercept hire β†’ perform a REAL hire server-side with an inert http - // adapter so no real agent process spawns. + // adapter so no real agent process spawns. (The cloud flow hires with + // requireEnvProbe: false, so there is no adapter-environment probe to stub.) await page.route("**/agent-hires", async (route) => { const req = route.request(); const body = JSON.parse(req.postData() || "{}"); @@ -54,38 +47,13 @@ test.describe("Dashboard launch after onboarding wizard", () => { await page.goto("/onboarding"); - // Launcher card path (existing companies) β€” enter the wizard if the - // route shows a launcher instead of opening the wizard directly. - const startBtn = page.getByRole("button", { name: /Start Onboarding/i }); - if (await startBtn.count()) await startBtn.first().click(); - - // Step 0: front door (skipped when the wizard opens on the create path). - const frontDoor = page.getByText("Build a new company"); - if (await frontDoor.count()) await frontDoor.first().click(); - - // Step 1: company name. - await page.getByPlaceholder("Acme Corp").fill(COMPANY_NAME); - await page.getByRole("button", { name: /^Next/ }).click(); - - // Step 2: mission (direct path default). - await page - .getByPlaceholder("What is your team trying to achieve?") - .fill(MISSION); - await page.getByRole("button", { name: /Confirm mission/ }).click(); - - // Step 3: lead name (prefilled) β†’ Next. - await page.waitForSelector('input[placeholder="Chief of staff"]', { - timeout: 15_000, + // Welcome β†’ company (name + mission) β†’ agent (role picker) β†’ first task. + // "Get started" on the task step creates the task and opens the dashboard. + await completeCloudOnboarding(page, { + companyName: COMPANY_NAME, + mission: MISSION, + choice: "hiring", }); - await page.getByRole("button", { name: /^Next/ }).click(); - - // Step 4: adapter (claude_local default); heartbeat is intercepted. - await page.getByRole("button", { name: /Give it a heartbeat/ }).click(); - - // Step 5: review β†’ Get started creates the first task and opens dashboard. - const getStarted = page.getByRole("button", { name: /Get started/ }); - await getStarted.waitFor({ timeout: 20_000 }); - await getStarted.click(); await expect(page).toHaveURL(/\/dashboard$/, { timeout: 30_000 }); @@ -98,8 +66,8 @@ test.describe("Dashboard launch after onboarding wizard", () => { const issuesRes = await page.request.get(`/api/companies/${company.id}/issues`); expect(issuesRes.ok()).toBe(true); const issues = await issuesRes.json(); - const firstTask = issues.find((candidate: { title: string }) => candidate.title === FIRST_TASK_TITLE); + const firstTask = issues.find((candidate: { title: string }) => candidate.title === HIRING_TASK_TITLE); expect(firstTask).toBeTruthy(); - await expect(page.getByText(FIRST_TASK_TITLE).first()).toBeVisible({ timeout: 15_000 }); + await expect(page.getByText(HIRING_TASK_TITLE).first()).toBeVisible({ timeout: 15_000 }); }); }); diff --git a/tests/e2e/nux-phase4-screenshots.spec.ts b/tests/e2e/nux-phase4-screenshots.spec.ts index 4c3a3481bea..9f788f7c688 100644 --- a/tests/e2e/nux-phase4-screenshots.spec.ts +++ b/tests/e2e/nux-phase4-screenshots.spec.ts @@ -2,6 +2,7 @@ import { test, expect } from "@playwright/test"; import fs from "node:fs"; import path from "node:path"; import { fileURLToPath } from "node:url"; +import { DEFAULT_ROLE, startCloudOnboarding } from "./onboarding-flow"; const __dirname = path.dirname(fileURLToPath(import.meta.url)); @@ -9,40 +10,43 @@ const __dirname = path.dirname(fileURLToPath(import.meta.url)); * NUX Phase 4 β€” visual QA screenshot capture. * * Boots a throwaway local_trusted instance (see playwright.config.ts webServer) - * and captures screenshots of every surface integrated by NUX Phases 1–3: - * - "Build a new company" step 1 (company name) + step 2 (mission) - * - Team-lead hire step (capsule wizard, PAP-125) - * - Onboarding front door (path picker) - * - "Add agents to your org" growth intake + * and captures screenshots of every integrated onboarding surface: + * - Welcome screen (path picker) + * - Company step (name + mission) + * - Create-your-first-agent step (role picker + capsule) + * - First-task step + * - "Add an agent to an existing company" entry (/:prefix/onboarding) * - Conference Room (BoardChat) shell + composer + activity feed * - Artifacts page * * These are structural/rendering checks β€” LLM-dependent streaming (CEO chat * responses, hiring-plan generation) is verified separately on an LLM-backed - * instance. Screenshots land in ./nux-phase4-shots for upload as evidence. + * instance. Screenshots land in ./test-results for upload as evidence. */ // Write under the gitignored test-results dir so re-runs leave no untracked // noise; screenshots are uploaded to the issue as QA evidence, not committed. const SHOT_DIR = path.join(__dirname, "test-results", "nux-phase4-shots"); +const SHOTS = [ + "01-welcome.png", + "02-company.png", + "03-agent.png", + "04-first-task.png", + "05-add-agent.png", + "06-board-chat.png", + "07-artifacts.png", +]; + function shot(name: string) { fs.mkdirSync(SHOT_DIR, { recursive: true }); return path.join(SHOT_DIR, name); } -async function openWizard(page: import("@playwright/test").Page) { - await page.goto("/onboarding"); - const startBtn = page.getByRole("button", { name: /Start Onboarding|New Company|Add Agent/ }); - if (await startBtn.count()) { - await startBtn.first().click(); - } -} - test.describe("NUX Phase 4 visual QA", () => { test("captures every integrated surface", async ({ page }) => { - // New-NUX surfaces are flag-gated default-OFF (PAP-136/137/138): turn the - // experimental flag on for this throwaway instance before driving them. + // Conference Room is flag-gated default-OFF: turn the experimental flag on + // for this throwaway instance before driving that surface (Section C). const flagRes = await page.request.patch("/api/instance/settings/experimental", { data: { enableConferenceRoomChat: true }, }); @@ -57,36 +61,43 @@ test.describe("NUX Phase 4 visual QA", () => { const baseUrl = "http://127.0.0.1:" + (process.env.PAPERCLIP_E2E_PORT ?? "3199"); - // ── Section A: create-company path (name β†’ mission β†’ hire) ──────────── - await openWizard(page); - // Front door shows when the wizard doesn't open directly on the create - // path (e.g. another spec already created a company on this instance). - const createCard = page.getByRole("button", { name: /Build a new company/ }); - if (await createCard.count()) { - await createCard.first().click(); - } + // ── Section A: the cloud flow, step by step ─────────────────────────── + await page.goto("/onboarding"); + await expect( - page.getByRole("heading", { name: "Name your company" }), + page.getByRole("heading", { name: "Welcome to Paperclip!" }), ).toBeVisible({ timeout: 15_000 }); - await page.getByPlaceholder("Acme Corp").fill("QA Robotics"); - await page.screenshot({ path: shot("02-create-name.png") }); + await page.screenshot({ path: shot("01-welcome.png") }); - await page.getByRole("button", { name: /^Next/ }).click(); + await startCloudOnboarding(page); + + // Capture the company step populated but not yet submitted, then submit. await expect( - page.getByRole("heading", { name: "Define your mission" }), - ).toBeVisible({ timeout: 10_000 }); + page.getByRole("heading", { name: "What is the name of your company or team?" }), + ).toBeVisible({ timeout: 15_000 }); + await page.locator("#onboarding-company-name").fill("QA Robotics"); await page - .getByPlaceholder("What is your team trying to achieve?") + .locator("#onboarding-mission") .fill("Build affordable home robots that handle household chores."); - await page.screenshot({ path: shot("03-create-mission.png") }); + await page.screenshot({ path: shot("02-company.png") }); + await page.getByRole("button", { name: /^Next/ }).click(); - // Step 2 advances via "Confirm mission" (creates the company + goal); - // step 3 is the team-lead naming step of the capsule wizard. - await page.getByRole("button", { name: /Confirm mission/ }).click(); - await page.waitForSelector('input[placeholder="Chief of staff"]', { - timeout: 30_000, - }); - await page.screenshot({ path: shot("04-hire-team-lead.png") }); + // Agent step: pick a role so the capsule + preview render, then capture + // before hiring. + await expect( + page.getByRole("heading", { name: "Create your first agent" }), + ).toBeVisible({ timeout: 30_000 }); + await page.locator("#onboarding-agent-role").click(); + await page.getByRole("option", { name: DEFAULT_ROLE, exact: true }).click(); + await page.screenshot({ path: shot("03-agent.png") }); + await page.getByRole("button", { name: /^Create/ }).click(); + + // First-task step: select a choice so the card's selected state is visible. + await expect( + page.getByRole("heading", { name: "Assign your agent a first task" }), + ).toBeVisible({ timeout: 30_000 }); + await page.getByRole("button", { name: /Create a hiring plan/ }).click(); + await page.screenshot({ path: shot("04-first-task.png") }); // The company just created anchors the route-scoped sections below. const companiesRes = await page.request.get(`${baseUrl}/api/companies`); @@ -95,39 +106,23 @@ test.describe("NUX Phase 4 visual QA", () => { const qaCompany = (Array.isArray(companies) ? companies : []).find( (c: { name: string }) => c.name === "QA Robotics", ); - expect(qaCompany, "wizard should have created QA Robotics").toBeTruthy(); + expect(qaCompany, "onboarding should have created QA Robotics").toBeTruthy(); const prefix: string = qaCompany.issuePrefix; - // ── Section B: front door + growth intake ───────────────────────────── + // ── Section B: "add an agent to an existing company" entry ──────────── + // OnboardingWizardVariant renders outside (App.tsx), so it never + // sees the :companyPrefix param and the company-scoped route still opens on + // the welcome screen. The real existing-company entry is the launcher card + // behind it: dismiss the overlay, then "Add Agent" opens onboarding scoped + // to this company, which skips company creation and starts at the agent step. await page.evaluate(() => window.localStorage.clear()); - await openWizard(page); - // Reach the full-screen front door (step 0): either it shows directly or - // "← Back to start" returns to it from the create step. - if (!(await page.getByRole("heading", { name: "Welcome to Paperclip" }).count())) { - await page.getByRole("button", { name: /Back to start/ }).click(); - } + await page.goto(`/${prefix}/onboarding`); + await page.getByRole("button", { name: "Close onboarding" }).click(); + await page.getByRole("button", { name: "Add Agent" }).click(); await expect( - page.getByRole("heading", { name: "Welcome to Paperclip" }), - ).toBeVisible({ timeout: 10_000 }); - await expect( - page.getByRole("heading", { name: "Build a new company" }), - ).toBeVisible(); - await expect( - page.getByRole("heading", { name: "Add agents to your org" }), - ).toBeVisible(); - await page.screenshot({ path: shot("01-front-door.png") }); - - await page.getByRole("button", { name: /Add agents to your org/ }).click(); - // The grow path shares step 1 (company name) before its step-2 intake. - await expect( - page.getByRole("heading", { name: "Name your company" }), - ).toBeVisible({ timeout: 10_000 }); - await page.getByPlaceholder("Acme Corp").fill("QA Robotics Grow"); - await page.getByRole("button", { name: /^Next/ }).click(); - await expect( - page.getByRole("heading", { name: /Tell us about your team/ }), - ).toBeVisible({ timeout: 10_000 }); - await page.screenshot({ path: shot("05-growth-intake.png") }); + page.getByRole("heading", { name: "Create your first agent" }), + ).toBeVisible({ timeout: 20_000 }); + await page.screenshot({ path: shot("05-add-agent.png") }); // ── Section C: Conference Room (BoardChat) ──────────────────────────── // Visit the company dashboard first so CompanyContext selects the company @@ -152,15 +147,7 @@ test.describe("NUX Phase 4 visual QA", () => { await page.waitForTimeout(1_000); await page.screenshot({ path: shot("07-artifacts.png") }); - for (const f of [ - "01-front-door.png", - "02-create-name.png", - "03-create-mission.png", - "04-hire-team-lead.png", - "05-growth-intake.png", - "06-board-chat.png", - "07-artifacts.png", - ]) { + for (const f of SHOTS) { const p = shot(f); expect(fs.existsSync(p), `missing ${f}`).toBe(true); expect(fs.statSync(p).size, `empty ${f}`).toBeGreaterThan(1_000); diff --git a/tests/e2e/onboarding-flow.ts b/tests/e2e/onboarding-flow.ts new file mode 100644 index 00000000000..49f4ffb60e9 --- /dev/null +++ b/tests/e2e/onboarding-flow.ts @@ -0,0 +1,134 @@ +import { expect, type Page } from "@playwright/test"; + +/** + * Shared driver for the cloud onboarding flow (ui/src/components/onboarding/). + * + * The flow replaced the retired OnboardingWizard, which had a front door, a + * separate company-name step, a separate mission step, an adapter step and a + * review step. The cloud flow is four screens: + * + * start β€” "Welcome to Paperclip!" (unnumbered) + * company β€” name + mission on one card; "Next" creates the company + goal + * agent β€” role picker (+ optional name); "Create" hires the lead agent + * task β€” first-task choice; "Get started" launches it and opens the dashboard + * + * Four specs drove the old wizard's selectors, so the step drivers live here + * rather than being copy-pasted: when the flow changes again, this is the one + * file to update. + */ + +/** Default role picked in the agent step. */ +export const DEFAULT_ROLE = "Chief of Staff"; + +/** + * Name the agent step auto-fills when DEFAULT_ROLE is picked. Selecting a role + * populates the (optional) name field with the role's acronym β€” see + * ROLE_ACRONYMS in ui/src/components/onboarding/onboarding-data.ts. + */ +export const DEFAULT_ROLE_ACRONYM = "COS"; + +/** Title of the task created by the "hiring" first-task choice. */ +export const HIRING_TASK_TITLE = "Hire your first engineer and create a hiring plan"; + +export type FirstTaskChoice = "hiring" | "strategy" | "custom"; + +/** + * Matches each first-task ChoiceCard. The cards are buttons whose accessible + * name is title + description, so these match on the title fragment only. + */ +const CHOICE_CARD: Record = { + hiring: /Create a hiring plan/, + strategy: /Write a team strategy doc/, + // Trailing ellipsis in the UI copy is deliberately not matched. + custom: /Write your own task/, +}; + +/** Step 0 β€” the welcome screen. Advances to the first numbered step. */ +export async function startCloudOnboarding(page: Page): Promise { + await expect(page.getByRole("heading", { name: "Welcome to Paperclip!" })).toBeVisible({ + timeout: 15_000, + }); + await page + .getByRole("button", { name: /Set up Paperclip for your company or team/ }) + .click(); +} + +/** + * Step 1 β€” company name + mission on a single card. "Next" persists the company + * and its company-level goal, then advances to the agent step. + */ +export async function completeCompanyStep( + page: Page, + { companyName, mission }: { companyName: string; mission: string }, +): Promise { + await expect( + page.getByRole("heading", { name: "What is the name of your company or team?" }), + ).toBeVisible({ timeout: 15_000 }); + await page.locator("#onboarding-company-name").fill(companyName); + await page.locator("#onboarding-mission").fill(mission); + await page.getByRole("button", { name: /^Next/ }).click(); +} + +/** + * Step 2 β€” role picker (a Radix select) plus an optional name. "Create" hires + * the lead agent. Pass `name` to override the acronym the role auto-fills. + */ +export async function completeAgentStep( + page: Page, + { role = DEFAULT_ROLE, name }: { role?: string; name?: string } = {}, +): Promise { + await expect(page.getByRole("heading", { name: "Create your first agent" })).toBeVisible({ + timeout: 30_000, + }); + await page.locator("#onboarding-agent-role").click(); + await page.getByRole("option", { name: role, exact: true }).click(); + if (name !== undefined) { + await page.locator("#onboarding-agent-name").fill(name); + } + await page.getByRole("button", { name: /^Create/ }).click(); +} + +/** + * Step 3 β€” pick the first task and launch it. "Get started" creates the task + * and navigates to the company dashboard. + */ +export async function completeTaskStep( + page: Page, + { choice = "hiring", customTask }: { choice?: FirstTaskChoice; customTask?: string } = {}, +): Promise { + await expect( + page.getByRole("heading", { name: "Assign your agent a first task" }), + ).toBeVisible({ timeout: 30_000 }); + await page.getByRole("button", { name: CHOICE_CARD[choice] }).click(); + if (choice === "custom") { + await page.getByPlaceholder("Describe the first task").fill(customTask ?? ""); + } + await page.getByRole("button", { name: /Get started/ }).click(); +} + +/** + * Drives the whole cloud flow from an already-loaded /onboarding route through + * to the dashboard navigation. Callers that need to assert or screenshot + * between steps should call the individual step drivers instead. + */ +export async function completeCloudOnboarding( + page: Page, + { + companyName, + mission, + role = DEFAULT_ROLE, + choice = "hiring", + customTask, + }: { + companyName: string; + mission: string; + role?: string; + choice?: FirstTaskChoice; + customTask?: string; + }, +): Promise { + await startCloudOnboarding(page); + await completeCompanyStep(page, { companyName, mission }); + await completeAgentStep(page, { role }); + await completeTaskStep(page, { choice, customTask }); +} diff --git a/tests/e2e/onboarding.spec.ts b/tests/e2e/onboarding.spec.ts index eb46e76e50a..f5a734d7772 100644 --- a/tests/e2e/onboarding.spec.ts +++ b/tests/e2e/onboarding.spec.ts @@ -1,79 +1,46 @@ import { test, expect } from "@playwright/test"; +import { completeCompanyStep, startCloudOnboarding } from "./onboarding-flow"; /** - * E2E: Onboarding wizard flow (NUX Phase 2 expanded wizard). + * E2E: cloud onboarding flow. * - * The wizard now opens on a front door (path picker) and the "Create a new - * company" path runs: - * Step 0 β€” Front door (Create a new company / Level up existing) - * Step 1a β€” Name your company - * Step 1b β€” Define your mission (direct or guided) - * Step 2 β€” Hire your team lead (adapter picker) - * Step 3+ β€” Launch celebration β†’ CEO chat β†’ hiring plan β†’ orientation + * The flow opens on a welcome screen and then runs three numbered steps: + * Step 0 β€” Welcome (unnumbered path picker) + * Step 1 β€” Company name + mission + * Step 2 β€” Create your first agent (role picker) + * Step 3 β€” Assign your agent a first task * - * This test covers the deterministic, LLM-free core: it drives the front door - * through company naming + mission definition (which creates the company and a - * company-level goal) and verifies the wizard advances to the team-lead step. + * This test covers the deterministic, LLM-free core: it drives the welcome + * screen through the company step (which creates the company and a + * company-level goal) and verifies the flow advances to the agent step. * - * The tail (CEO chat at step 4, hiring-plan generation at step 5, final - * landing) depends on a live LLM and is verified separately during manual / - * LLM-backed QA β€” see PAP-50. Surface-level rendering of every step is - * snapshotted by nux-phase4-screenshots.spec.ts. + * The tail (hiring the agent, launching the first task) is covered by + * conference-room-typing-intro.spec.ts; surface-level rendering of every step + * is snapshotted by nux-phase4-screenshots.spec.ts. */ const COMPANY_NAME = `E2E-Test-${Date.now()}`; const MISSION = "Build affordable home robots that handle household chores."; -test.describe("Onboarding wizard", () => { - test("create-company path: name + mission creates company and goal", async ({ +test.describe("Onboarding flow", () => { + test("company step: name + mission creates company and goal", async ({ page, }) => { const pageErrors: string[] = []; page.on("pageerror", (err) => pageErrors.push(err.message)); - // New-NUX surfaces are flag-gated default-OFF (PAP-136/137/138): turn the - // experimental flag on for this throwaway instance before driving them. - const flagRes = await page.request.patch("/api/instance/settings/experimental", { - data: { enableConferenceRoomChat: true }, - }); - expect(flagRes.ok()).toBe(true); - await page.goto("/onboarding"); - // The wizard may open on a launcher card or directly on the capsule - // wizard; the front door (step 0) requires a click into the create path. - const startBtn = page.getByRole("button", { - name: /Start Onboarding|New Company|Add Agent/, + await startCloudOnboarding(page); + await completeCompanyStep(page, { + companyName: COMPANY_NAME, + mission: MISSION, }); - if (await startBtn.count()) { - await startBtn.first().click(); - } - const createCard = page.getByRole("button", { name: /Build a new company/ }); - if (await createCard.count()) { - await createCard.first().click(); - } - // Step 1 β€” Name your company. + // Reaching the agent step means the company + goal writes succeeded. await expect( - page.getByRole("heading", { name: "Name your company" }), - ).toBeVisible({ timeout: 15_000 }); - await page.getByPlaceholder("Acme Corp").fill(COMPANY_NAME); - await page.getByRole("button", { name: /^Next/ }).click(); - - // Step 2 β€” Define your mission (direct entry is the default path). - await expect( - page.getByRole("heading", { name: "Define your mission" }), - ).toBeVisible({ timeout: 10_000 }); - await page - .getByPlaceholder("What is your team trying to achieve?") - .fill(MISSION); - - // "Confirm mission" creates the company + a company-level goal, then - // advances to the team-lead naming step of the capsule wizard. - await page.getByRole("button", { name: /Confirm mission/ }).click(); - await page.waitForSelector('input[placeholder="Chief of staff"]', { - timeout: 30_000, - }); + page.getByRole("heading", { name: "Create your first agent" }), + ).toBeVisible({ timeout: 30_000 }); // Verify the company + company-level goal were persisted. const baseUrl = page.url().split("/").slice(0, 3).join("/"); @@ -95,7 +62,7 @@ test.describe("Onboarding wizard", () => { ); expect(companyGoal, "a company-level goal should be created").toBeTruthy(); - // The expanded wizard must not crash the app (Rules-of-Hooks regression). + // The flow must not crash the app (Rules-of-Hooks regression). expect(pageErrors, pageErrors.join("\n")).toHaveLength(0); }); }); diff --git a/tests/e2e/planning-mode-visual-verification.spec.ts b/tests/e2e/planning-mode-visual-verification.spec.ts index 65be76312de..9692986b935 100644 --- a/tests/e2e/planning-mode-visual-verification.spec.ts +++ b/tests/e2e/planning-mode-visual-verification.spec.ts @@ -1,20 +1,21 @@ +import path from "node:path"; +import { fileURLToPath } from "node:url"; import { expect, test } from "@playwright/test"; +import { completeCloudOnboarding, HIRING_TASK_TITLE } from "./onboarding-flow"; -const AGENT_NAME = "Chief of staff"; -const TASK_TITLE = "Hire your first engineer and create a hiring plan"; +const __dirname = path.dirname(fileURLToPath(import.meta.url)); test("captures planning mode UI for desktop and mobile", async ({ page }) => { const timestamp = Date.now(); const companyName = `PAP-3413-${timestamp}`; - const screenshotDir = "test-results/planning-mode"; - - await page.route("**/test-environment", (route) => - route.fulfill({ - contentType: "application/json", - body: JSON.stringify({ status: "pass", checks: [] }), - }), - ); + // Resolve against this file, not the cwd, so screenshots land in the + // gitignored tests/e2e/test-results/ rather than an untracked dir at the + // repo root that a contributor could commit by accident. + const screenshotDir = path.join(__dirname, "test-results", "planning-mode"); + // Intercept hire β†’ perform a REAL hire server-side with an inert http adapter + // so no real agent process spawns. (The cloud flow hires with + // requireEnvProbe: false, so there is no adapter-environment probe to stub.) await page.route("**/agent-hires", async (route) => { const req = route.request(); const body = JSON.parse(req.postData() || "{}"); @@ -41,31 +42,15 @@ test("captures planning mode UI for desktop and mobile", async ({ page }) => { }); await page.goto("/onboarding"); - const startBtn = page.getByRole("button", { name: /Start Onboarding|New Company|Add Agent/ }); - if (await startBtn.count()) await startBtn.first().click(); - const createCard = page.getByRole("button", { name: /Build a new company/ }); - if (await createCard.count()) await createCard.first().click(); + // This spec only needs a company with a seeded first task to screenshot the + // planning-mode UI against; drive the whole onboarding flow to get one. + await completeCloudOnboarding(page, { + companyName, + mission: "Capture planning mode visual evidence for the graduated task UI.", + choice: "hiring", + }); - await expect(page.getByRole("heading", { name: "Name your company" })).toBeVisible({ timeout: 15_000 }); - - await page.locator('input[placeholder="Acme Corp"]').fill(companyName); - await page.getByRole("button", { name: /^Next/ }).click(); - - await expect(page.getByRole("heading", { name: "Define your mission" })).toBeVisible({ timeout: 30_000 }); - await page - .getByPlaceholder("What is your team trying to achieve?") - .fill("Capture planning mode visual evidence for the graduated task UI."); - await page.getByRole("button", { name: /Confirm mission/ }).click(); - - await page.waitForSelector('input[placeholder="Chief of staff"]', { timeout: 30_000 }); - await expect(page.locator('input[placeholder="Chief of staff"]')).toHaveValue(AGENT_NAME); - - await page.getByRole("button", { name: /^Next/ }).click(); - await page.getByRole("button", { name: /Give it a heartbeat/ }).click(); - - await expect(page.getByRole("heading", { name: "Review" })).toBeVisible({ timeout: 30_000 }); - await page.getByRole("button", { name: /Get started/ }).click(); await expect(page).toHaveURL(/\/dashboard$/, { timeout: 30_000 }); const baseOrigin = new URL(page.url()).origin; @@ -79,7 +64,7 @@ test("captures planning mode UI for desktop and mobile", async ({ page }) => { const issues = await issueRes.json(); const planningSeedIssue = issues.find( (candidate: { id: string; identifier?: string; title: string }) => - candidate.title === TASK_TITLE, + candidate.title === HIRING_TASK_TITLE, ); expect(planningSeedIssue).toBeTruthy(); diff --git a/ui/onboarding-preview.html b/ui/onboarding-preview.html new file mode 100644 index 00000000000..93a17b04b86 --- /dev/null +++ b/ui/onboarding-preview.html @@ -0,0 +1,22 @@ + + + + + + Onboarding Preview + + + + +
+ + + diff --git a/ui/package.json b/ui/package.json index ce25a72fb5b..6788beded3f 100644 --- a/ui/package.json +++ b/ui/package.json @@ -59,6 +59,7 @@ "lexical": "0.48.0", "lucide-react": "^0.577.0", "mermaid": "^11.16.0", + "motion": "^12.42.2", "radix-ui": "^1.6.4", "react": "^19.2.7", "react-dom": "^19.2.7", @@ -67,7 +68,8 @@ "react-resizable-panels": "^4.12.2", "react-router-dom": "^7.18.1", "remark-gfm": "^4.0.1", - "tailwind-merge": "^3.6.0" + "tailwind-merge": "^3.6.0", + "three": "^0.185.1" }, "devDependencies": { "@storybook/addon-a11y": "10.5.4", @@ -77,6 +79,7 @@ "@types/node": "^22.20.1", "@types/react": "^19.2.17", "@types/react-dom": "^19.2.3", + "@types/three": "^0.185.1", "@vitejs/plugin-react": "^4.3.4", "storybook": "10.5.5", "tailwindcss": "^4.3.2", diff --git a/ui/src/components/AgentCapsule.tsx b/ui/src/components/AgentCapsule.tsx index 817a9dbb76a..d671fb1ca08 100644 --- a/ui/src/components/AgentCapsule.tsx +++ b/ui/src/components/AgentCapsule.tsx @@ -1,4 +1,5 @@ import * as React from "react"; +import { motion, useReducedMotion } from "motion/react"; import { cn } from "@/lib/utils"; @@ -8,14 +9,14 @@ import { cn } from "@/lib/utils"; * A single agent is drawn as a tall pill (proportion 1:β‰₯2, radius 9999px) * that moves through three states as the agent comes to life: * - * - `slot` β€” dashed outline, gently pulsing. An empty agent slot. + * - `slot` β€” static dashed outline. An empty agent slot. * - `configured` β€” solid stroke. Agent named / model picked, not yet live. - * - `online` β€” brand agent-gradient liquid rises to fill the capsule, + * - `online` β€” brand agent-gradient fill radiates outward from the center, * which then breathes with an online-pulse ring (green by * default, or blue via `glow="blue"`). * * The three states are drawn as stacked layers (a dashed outline, a solid - * stroke, and the rising liquid) that cross-fade by opacity. Because + * stroke, and the radial fill) that cross-fade by opacity. Because * `border-style` is not animatable, the dashedβ†’solid morph is realized as the * dashed layer fading out while the solid layer fades in β€” so the SAME capsule * can evolve in place across a flow (PAP-125, Option 4 wizard). @@ -24,7 +25,7 @@ import { cn } from "@/lib/utils"; * `--agent-Na` (top) β†’ `--agent-Nb` (bottom); pick which one with `gradient` * (1–10). Size is a preset (`sm` | `md` | `lg`) or an explicit pixel pair so * the component is reusable app-wide. `prefers-reduced-motion` is honored in - * CSS β€” the liquid rise, layer cross-fade and both pulses are skipped and the + * CSS β€” the radial fill, layer cross-fade and both pulses are skipped and the * final state is rendered statically. */ @@ -60,10 +61,21 @@ export interface AgentCapsuleProps size?: AgentCapsuleSizePreset | { width: number; height: number }; /** Online-pulse colour (only applies in the `online` state). Defaults to `green`. */ glow?: AgentCapsuleGlow; + /** + * Slotβ†’configured morph as a draw-on: the solid outline is traced around the + * perimeter over the still-visible dashed outline (which fades once the draw + * completes), instead of the default border cross-fade. Honors reduced + * motion by rendering the final state instantly. + */ + strokeDraw?: boolean; /** Accessible label; defaults to a description of the state. */ "aria-label"?: string; } +/** Duration of the strokeDraw perimeter trace; the dashed layer fades after it. */ +const STROKE_DRAW_SECONDS = 0.9; +const STROKE_DRAW_EASE = [0.16, 1, 0.3, 1] as const; + /** Normalize a (possibly out-of-range) gradient index to 1…AGENT_GRADIENT_COUNT. */ function normalizeGradient(gradient: number): number { const n = Math.trunc(gradient); @@ -75,6 +87,7 @@ export function AgentCapsule({ gradient = 1, size = "md", glow = "green", + strokeDraw = false, className, style, "aria-label": ariaLabel, @@ -83,6 +96,8 @@ export function AgentCapsule({ const dims = typeof size === "string" ? SIZE_PRESETS[size] : size; const idx = normalizeGradient(gradient); const fill = `linear-gradient(to bottom, var(--agent-${idx}a), var(--agent-${idx}b))`; + const reducedMotion = useReducedMotion(); + const drawn = state === "configured" || state === "online"; return (
- {/* Dashed outline β€” an empty agent slot. Visible (and pulsing) only in - the slot state; cross-fades out as the capsule is configured. */} + {/* Dashed outline β€” an empty agent slot. Visible (static) only in the + slot state; cross-fades out as the capsule is configured. In + strokeDraw mode it instead stays put while the solid outline is + traced over it, then fades once the draw completes. */}