From 21a1e7885259eec1c83e24d18da90a282adbb83d Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 08:52:43 -0800 Subject: [PATCH] feat(environments): add managed plugin environment rows (#19490) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document > - That document cannot declare a plugin environment yet, so a managed instance cannot get a plugin-provided environment without manual setup > - The full change is split into three pull requests. This first one adds the plugin environment rows and their write floor, and nothing calls them yet > - The second pull request lets the managed config declare plugin entries and provisions them at boot. The third sets the instance default once and advertises the new managed-config capability > - The benefit is a smaller review for each step, and no instance changes behavior until the last step lands ## Linked Issues or Issue Description **Subsystem affected** Server environments (managed-config environment provisioning, environment routes) and the board UI's adapter Test target. **Problem or motivation** A managed instance cannot get a plugin environment from its managed config. Company settings cannot create one either. So each managed instance that needs one requires manual setup. **Proposed solution** Over three pull requests, let an `environments` entry in the managed config declare `driver: "plugin"`. Boot then provisions one locked row for each plugin entry, and makes one of them the instance default once on new instances. This pull request adds only the row layer: create or adopt a marked row, archive it, find it, and keep tenants from editing it. **Alternatives considered** Manual setup on each instance, and a new control-plane endpoint. Both need a step outside the deploy. The managed config already delivers the sandbox entry. **Roadmap alignment** This extends the existing managed-config `environments` section, which already provisions the managed sandbox. It adds no new product surface. ## What Changed - New `server/src/services/managed-plugin-environments.ts`, used through new `environmentService` methods (`ensureManagedPluginEnvironment`, `archiveManagedPluginEnvironment`, `findManagedPluginEnvironment`): - The ensure validates the driver config through `normalizeEnvironmentConfig` and `validatePluginEnvironmentDriverConfig`, the same path an operator-created plugin environment takes. - It marks the row `managedByPaperclip: true` and `managedPluginEnvironment: ":"`. - It adopts an existing unmarked row for the same plugin and driver instead of adding a second one, so a hand-made row keeps its agents. - A row that already matches gets no write. Changes write activity log entries. - Archive sets an active marked row to `archived`. The next ensure reactivates it. - Migration `0437`: unique index `environments_managed_plugin_environment_idx` on `metadata ->> 'managedPluginEnvironment'`. This is the only schema change of the three pull requests. Snapshot `0432` is pruned to keep the five newest. - `server/src/routes/environments.ts`: `isPlatformProvisionedEnvironment` also covers the new marker (write floor and redaction). `managedPluginEnvironment` is a reserved marker key, so a client cannot stamp it on its own row. Nothing provisions marked rows yet, so the marker-clear escape hatch stays open for them. The next pull request refuses it while the managed config declares the row's entry. - `ui/src/lib/adapter-test-environment.ts`: `resolveManagedSandboxEnvironmentId` requires `driver === "sandbox"`. The plugin row also carries `managedByPaperclip`, so the old "not local" check could take it for the managed sandbox. ## Verification - `cd server && npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres): - The ensure creates the marked row through the plugin validator. A second ensure writes nothing and logs nothing. - A hand-made row for the same plugin and driver is adopted. It keeps its id and its agents. - Archive, then ensure, reactivates the row. - The ensure refuses while the plugin has no running worker, and refuses a name that another environment holds. - The unique index allows one row per marker. - `npx vitest run src/__tests__/environment-routes.test.ts`: on a cloud-managed instance, edit, env-var edit, status change and delete of a marked plugin row are refused, including for instance admins. The marker alone keeps the floor. A client cannot stamp the marker. The marker clear is allowed. - `npx vitest run src/__tests__/environment-service.test.ts`: passes. - `cd ui && npx vitest run src/lib/adapter-test-environment.test.ts`: the plugin row is never taken for the managed sandbox. Under managed-sandbox-only, an agent on the sandbox tests in the sandbox, and an agent on the plugin environment tests there. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck` and `pnpm --filter @paperclipai/db check:migrations` pass. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. `pnpm verify` asks for broader verification because of the migration. ## Risks - Migration: one new partial unique index on a small table. No row carries the marker before this change, so the index cannot fail on existing data. - No caller uses the new ensure yet, so no instance changes behavior. - The UI change affects only lists that hold a non-sandbox row with `managedByPaperclip`. Today that is only the local row, which the old check already excluded. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../src/migrations/0437_workable_oracle.sql | 1 + ...{0432_snapshot.json => 0437_snapshot.json} | 111 ++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/environments.ts | 6 + .../src/__tests__/environment-routes.test.ts | 69 +++++ .../managed-plugin-environments.test.ts | 178 +++++++++++ server/src/routes/environments.ts | 9 +- server/src/services/environments.ts | 36 +++ .../services/managed-plugin-environments.ts | 287 ++++++++++++++++++ ui/src/lib/adapter-test-environment.test.ts | 50 +++ ui/src/lib/adapter-test-environment.ts | 8 +- 11 files changed, 754 insertions(+), 8 deletions(-) create mode 100644 packages/db/src/migrations/0437_workable_oracle.sql rename packages/db/src/migrations/meta/{0432_snapshot.json => 0437_snapshot.json} (99%) create mode 100644 server/src/__tests__/managed-plugin-environments.test.ts create mode 100644 server/src/services/managed-plugin-environments.ts diff --git a/packages/db/src/migrations/0437_workable_oracle.sql b/packages/db/src/migrations/0437_workable_oracle.sql new file mode 100644 index 00000000000..71542e50c1c --- /dev/null +++ b/packages/db/src/migrations/0437_workable_oracle.sql @@ -0,0 +1 @@ +CREATE UNIQUE INDEX "environments_managed_plugin_environment_idx" ON "environments" USING btree (("metadata" ->> 'managedPluginEnvironment')) WHERE jsonb_typeof("environments"."metadata" -> 'managedPluginEnvironment') = 'string'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0432_snapshot.json b/packages/db/src/migrations/meta/0437_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0432_snapshot.json rename to packages/db/src/migrations/meta/0437_snapshot.json index d44daa9c781..96b47269974 100644 --- a/packages/db/src/migrations/meta/0432_snapshot.json +++ b/packages/db/src/migrations/meta/0437_snapshot.json @@ -1,6 +1,6 @@ { - "id": "a87278a8-15b3-4187-94a9-f4bb79d8640d", - "prevId": "9fcf29d1-a9a6-4ead-8d43-7e087a2b2b24", + "id": "2c0cb580-67fc-45d3-9ca5-cce4148dd1a4", + "prevId": "a80d63fe-5111-49e8-a39a-7a5589b3b31a", "version": "7", "dialect": "postgresql", "tables": { @@ -4058,6 +4058,28 @@ "method": "btree", "with": {} }, + "agent_wakeup_requests_tree_hold_resume_uq": { + "name": "agent_wakeup_requests_tree_hold_resume_uq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "idempotency_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'tree-hold-resume:%' AND \"agent_wakeup_requests\".\"status\" <> 'skipped'", + "concurrently": true, + "method": "btree", + "with": {} + }, "agent_wakeup_requests_chat_completion_uq": { "name": "agent_wakeup_requests_chat_completion_uq", "columns": [ @@ -28962,6 +28984,22 @@ "method": "btree", "with": {} }, + "environments_managed_plugin_environment_idx": { + "name": "environments_managed_plugin_environment_idx", + "columns": [ + { + "expression": "(\"metadata\" ->> 'managedPluginEnvironment')", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": true, + "where": "jsonb_typeof(\"environments\".\"metadata\" -> 'managedPluginEnvironment') = 'string'", + "concurrently": false, + "method": "btree", + "with": {} + }, "environments_name_idx": { "name": "environments_name_idx", "columns": [ @@ -35053,6 +35091,28 @@ "method": "btree", "with": {} }, + "heartbeat_runs_tree_hold_suppression_idx": { + "name": "heartbeat_runs_tree_hold_suppression_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "(\"result_json\"->'treeHoldSuppression'->>'holdId')", + "asc": true, + "isExpression": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"result_json\" ? 'treeHoldSuppression'", + "concurrently": false, + "method": "btree", + "with": {} + }, "heartbeat_runs_cost_accounting_pending_idx": { "name": "heartbeat_runs_cost_accounting_pending_idx", "columns": [ @@ -42492,6 +42552,12 @@ "primaryKey": false, "notNull": false }, + "origin_issue_id": { + "name": "origin_issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, "origin_identity_context_id": { "name": "origin_identity_context_id", "type": "uuid", @@ -42935,6 +43001,27 @@ "method": "btree", "with": {} }, + "issues_company_origin_issue_idx": { + "name": "issues_company_origin_issue_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "origin_issue_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "issues_company_project_workspace_idx": { "name": "issues_company_project_workspace_idx", "columns": [ @@ -43534,6 +43621,19 @@ "onDelete": "no action", "onUpdate": "no action" }, + "issues_origin_issue_id_issues_id_fk": { + "name": "issues_origin_issue_id_issues_id_fk", + "tableFrom": "issues", + "tableTo": "issues", + "columnsFrom": [ + "origin_issue_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + }, "issues_execution_workspace_id_execution_workspaces_id_fk": { "name": "issues_execution_workspace_id_execution_workspaces_id_fk", "tableFrom": "issues", @@ -61884,6 +61984,13 @@ "type": "boolean", "primaryKey": false, "notNull": true, + "default": true + }, + "classic_workspace_enabled": { + "name": "classic_workspace_enabled", + "type": "boolean", + "primaryKey": false, + "notNull": true, "default": false }, "created_at": { diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index be513a9a67f..74008905afd 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -3046,6 +3046,13 @@ "when": 1793952565204, "tag": "0436_legal_reavers", "breakpoints": true + }, + { + "idx": 437, + "version": "7", + "when": 1793982073367, + "tag": "0437_workable_oracle", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/environments.ts b/packages/db/src/schema/environments.ts index a61c2a99ae1..2f7b4e0549a 100644 --- a/packages/db/src/schema/environments.ts +++ b/packages/db/src/schema/environments.ts @@ -25,6 +25,12 @@ export const environments = pgTable( .where( sql`${table.driver} = 'sandbox' AND (${table.metadata} ->> 'managedByPaperclip')::boolean = true`, ), + // One row per managed-config plugin entry: the marker is the entry's + // ":" pair (`ensureManagedPluginEnvironment`). + // Only string markers count; a cleared marker (`null` / `false`) is not one. + managedPluginEnvironmentIdx: uniqueIndex("environments_managed_plugin_environment_idx") + .on(sql`(${table.metadata} ->> 'managedPluginEnvironment')`) + .where(sql`jsonb_typeof(${table.metadata} -> 'managedPluginEnvironment') = 'string'`), nameIdx: uniqueIndex("environments_name_idx").on(table.name), }), ); diff --git a/server/src/__tests__/environment-routes.test.ts b/server/src/__tests__/environment-routes.test.ts index 51e82de3844..786e97d6ea2 100644 --- a/server/src/__tests__/environment-routes.test.ts +++ b/server/src/__tests__/environment-routes.test.ts @@ -1020,6 +1020,75 @@ describe("environment routes", () => { expect(mockEnvironmentService.create).not.toHaveBeenCalled(); }); + describe("managed plugin environment rows", () => { + const PLUGIN_KEY = "example.remote-host-plugin"; + const pluginRow = () => ({ + ...createPlatformSandboxEnvironment(), + id: "env-plugin-1", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + envVars: {}, + metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + + it("refuses edits, env-var edits, and deletes, including for instance admins", async () => { + mockEnvironmentService.getById.mockResolvedValue(pluginRow()); + const app = createApp(ownerAdminActor); + + for (const body of [{ name: "Renamed" }, { envVars: { A: "1" } }, { status: "archived" }]) { + const res = await request(app).patch("/api/environments/env-plugin-1").send(body); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + } + const deleted = await request(app).delete("/api/environments/env-plugin-1"); + expect(deleted.status).toBe(403); + expect(deleted.body.details).toMatchObject({ code: "environment_platform_managed" }); + expect(mockEnvironmentService.update).not.toHaveBeenCalled(); + expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled(); + }); + + it("keeps the row behind the floor on the plugin marker alone", async () => { + mockEnvironmentService.getById.mockResolvedValue({ + ...pluginRow(), + metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + const res = await request(createApp(ownerAdminActor)).patch("/api/environments/env-plugin-1").send({ name: "Renamed" }); + expect(res.status).toBe(403); + expect(res.body.details).toMatchObject({ code: "environment_platform_managed" }); + }); + + it("allows the marker clear while no managed-config entry can own the row", async () => { + // Nothing provisions marked plugin rows yet, so a marker is stale and + // stays recoverable. The managed-config plugin entry PR refuses this + // while the document declares the row's entry. + const row = pluginRow(); + mockEnvironmentService.getById.mockResolvedValue(row); + mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} }); + + const res = await request(createApp(ownerAdminActor)) + .patch("/api/environments/env-plugin-1") + .send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }); + + expect(res.status).toBe(200); + expect(mockEnvironmentService.update).toHaveBeenCalled(); + }); + + it("refuses a client that stamps the plugin marker on its own row", async () => { + const res = await request(createApp(ownerAdminActor)) + .post("/api/companies/company-1/environments") + .send({ + name: "Fake remote host", + driver: "plugin", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` }, + }); + expect(res.status).toBe(422); + expect(res.body.details).toMatchObject({ code: "environment_platform_marker_reserved" }); + expect(mockEnvironmentService.create).not.toHaveBeenCalled(); + }); + }); + it("rejects tenant patches that stamp platform markers so the row cannot become locked", async () => { const tenantEnvironment = { ...createPlatformSandboxEnvironment(), diff --git a/server/src/__tests__/managed-plugin-environments.test.ts b/server/src/__tests__/managed-plugin-environments.test.ts new file mode 100644 index 00000000000..fc8496b236d --- /dev/null +++ b/server/src/__tests__/managed-plugin-environments.test.ts @@ -0,0 +1,178 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { + activityLog, + agents, + companies, + createDb, + environments, + plugins, + type Db, +} from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { environmentService } from "../services/environments.js"; + +const PLUGIN_KEY = "example.remote-host-plugin"; +const MARKER = `${PLUGIN_KEY}:tasks`; + +const support = await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("managed plugin environment rows", () => { + let database: Awaited>; + let db: Db; + let pluginId: string; + let running: boolean; + const call = vi.fn(async (_id: string, method: string, input: { config: Record }) => + method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null); + const workerManager = { isRunning: vi.fn(() => running), call }; + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-"); + db = createDb(database.connectionString); + }, 90_000); + afterAll(async () => { await database?.cleanup(); }); + beforeEach(async () => { + running = true; + call.mockClear(); + await db.delete(activityLog); + await db.delete(agents); + await db.delete(environments); + await db.delete(plugins); + await db.delete(companies); + pluginId = randomUUID(); + await db.insert(plugins).values({ + id: pluginId, + pluginKey: PLUGIN_KEY, + packageName: "@example/remote-host-plugin", + version: "1.0.0", + status: "ready", + manifestJson: { + capabilities: ["environment.drivers.register"], + environmentDrivers: [{ driverKey: "tasks", requiresAgent: true, supportsTasks: true }], + } as never, + }); + }); + + const ensure = (overrides: Record = {}) => + environmentService(db).ensureManagedPluginEnvironment({ + name: "Remote Host", + description: "Runs the agent on a remote host", + pluginKey: PLUGIN_KEY, + driverKey: "tasks", + driverConfig: {}, + workerManager, + ...overrides, + }); + const pluginRows = () => db.select().from(environments).where(eq(environments.driver, "plugin")); + const company = async () => { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Managed", issuePrefix: `M${id.slice(0, 7)}` }); + return id; + }; + + it("creates the marked row through the plugin validator, then changes nothing", async () => { + await company(); + const first = await ensure(); + expect(first.action).toBe("added"); + expect(first.environment).toMatchObject({ + name: "Remote Host", + description: "Runs the agent on a remote host", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: { normalized: true } }, + metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER }, + }); + expect(call).toHaveBeenCalledWith(pluginId, "environmentValidateConfig", { driverKey: "tasks", config: {} }); + expect(await db.select().from(activityLog)).toHaveLength(1); + + const second = await ensure(); + const rows = await pluginRows(); + expect(second.action).toBe("unchanged"); + expect(rows).toHaveLength(1); + expect(rows[0]!.updatedAt.getTime()).toBe(first.environment.updatedAt.getTime()); + expect(await db.select().from(activityLog)).toHaveLength(1); + }); + + it("adopts a hand-made row for the same plugin and driver, keeping its agents", async () => { + const companyId = await company(); + const [handMade] = await db.insert(environments).values({ + name: "Remote Host (manual)", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { note: "made by hand" }, + }).returning(); + const agentId = randomUUID(); + await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "process", defaultEnvironmentId: handMade!.id }); + + const result = await ensure(); + const rows = await pluginRows(); + expect(result.action).toBe("adopted"); + expect(rows).toHaveLength(1); + expect(rows[0]).toMatchObject({ + id: handMade!.id, + name: "Remote Host", + metadata: { note: "made by hand", managedByPaperclip: true, managedPluginEnvironment: MARKER }, + }); + const [kept] = await db.select().from(agents).where(eq(agents.id, agentId)); + expect(kept!.defaultEnvironmentId).toBe(handMade!.id); + }); + + it("adopts a row again after its markers were cleared", async () => { + const first = await ensure(); + // What the route's marker-clear request stores. + await db.update(environments) + .set({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } }) + .where(eq(environments.id, first.environment.id)); + + const again = await ensure(); + expect(again.action).toBe("adopted"); + expect(again.environment.id).toBe(first.environment.id); + expect(await pluginRows()).toHaveLength(1); + }); + + it("lets several rows hold a cleared marker", async () => { + const config = { pluginKey: PLUGIN_KEY, driverKey: "other", driverConfig: {} }; + await db.insert(environments).values([ + { name: "Cleared one", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } }, + { name: "Cleared two", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } }, + { name: "Cleared three", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: null } }, + ]); + expect(await pluginRows()).toHaveLength(3); + }); + + it("archives the active marked row and reactivates it on the next ensure", async () => { + await ensure(); + const archived = await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }); + expect(archived?.status).toBe("archived"); + expect(await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" })).toBeNull(); + expect((await environmentService(db).findManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }))?.status) + .toBe("archived"); + + const reactivated = await ensure(); + expect(reactivated.action).toBe("updated"); + expect(reactivated.environment.status).toBe("active"); + }); + + it("refuses to ensure while the plugin has no running worker", async () => { + running = false; + await expect(ensure()).rejects.toThrow(/has no running worker/); + expect(await pluginRows()).toHaveLength(0); + }); + + it("refuses to take a name another environment already holds", async () => { + await db.insert(environments).values({ name: "Remote Host", driver: "ssh", status: "active", config: {} }); + await expect(ensure()).rejects.toThrow(/already exists with driver "ssh"/); + }); + + it("allows at most one row per marker", async () => { + await ensure(); + await expect(db.insert(environments).values({ + name: "Second remote host", + driver: "plugin", + status: "active", + config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} }, + metadata: { managedPluginEnvironment: MARKER }, + })).rejects.toThrow(); + }); +}); diff --git a/server/src/routes/environments.ts b/server/src/routes/environments.ts index eed8b648d46..9a33f1b66f9 100644 --- a/server/src/routes/environments.ts +++ b/server/src/routes/environments.ts @@ -19,6 +19,7 @@ import { import { conflict, forbidden, unprocessable } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js"; +import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js"; import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js"; import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js"; import { validate } from "../middleware/validate.js"; @@ -69,8 +70,10 @@ function isPlainRecord(value: unknown): value is Record { /** * Whether this environment row was provisioned by the platform: the - * managed-config environment provisioner marker, or the legacy managed - * Kubernetes wrapper marker (rows created by builds that predate the + * managed-config environment provisioner marker, the managed plugin + * environment marker (`":"`, stamped next to + * `managedByPaperclip` on a managed-config plugin entry's row), or the legacy + * managed Kubernetes wrapper marker (rows created by builds that predate the * generalized provisioner and have not been adopted yet). */ export function isPlatformProvisionedEnvironment(environment: { @@ -78,6 +81,7 @@ export function isPlatformProvisionedEnvironment(environment: { }): boolean { return ( environment.metadata?.managedByPaperclip === true || + typeof environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY] === "string" || environment.metadata?.managedKubernetesSandbox === true ); } @@ -151,6 +155,7 @@ function isTenantEditableManagedSandbox(environment: { const PLATFORM_PROVISIONED_MARKER_KEYS = [ "managedByPaperclip", + MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY, "managedKubernetesSandbox", ] as const; diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index ec93b17a184..26c6cae9d99 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -50,6 +50,13 @@ import { stockHash, type ManagedResourceStockStatus, } from "./managed-resource-drift.js"; +import { + archiveManagedPluginEnvironment as archiveManagedPluginEnvironmentRow, + ensureManagedPluginEnvironment as ensureManagedPluginEnvironmentRow, + findManagedPluginEnvironment as findManagedPluginEnvironmentRow, + type ManagedPluginEnvironmentInput, + type ManagedPluginEnvironmentReconcileAction, +} from "./managed-plugin-environments.js"; export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null }, attach: (tx: Db) => Promise): Promise { @@ -1062,6 +1069,35 @@ export function environmentService(db: Db) { archiveManagedSandboxEnvironment, + /** + * Ensure the instance-level row of one managed-config plugin entry, + * validating its driver config through the plugin environment validator. + * Adopts an unmarked row for the same plugin and driver (see + * `managed-plugin-environments.ts`). + */ + ensureManagedPluginEnvironment: async ( + input: ManagedPluginEnvironmentInput, + ): Promise<{ environment: Environment; action: ManagedPluginEnvironmentReconcileAction }> => { + const result = await ensureManagedPluginEnvironmentRow(db, input); + return { environment: toEnvironment(result.row), action: result.action }; + }, + + /** Archive the active row of a managed plugin entry whose plugin is not running. */ + archiveManagedPluginEnvironment: async ( + input: { pluginKey: string; driverKey: string }, + ): Promise => { + const row = await archiveManagedPluginEnvironmentRow(db, input); + return row ? toEnvironment(row) : null; + }, + + /** The marked row of a managed plugin entry, archived or not. */ + findManagedPluginEnvironment: async ( + input: { pluginKey: string; driverKey: string }, + ): Promise => { + const row = await findManagedPluginEnvironmentRow(db, input); + return row ? toEnvironment(row) : null; + }, + /** * Idempotently ensure a managed Kubernetes sandbox environment exists for * an instance, configured from instance/operator-supplied config. A thin diff --git a/server/src/services/managed-plugin-environments.ts b/server/src/services/managed-plugin-environments.ts new file mode 100644 index 00000000000..a3450722f9a --- /dev/null +++ b/server/src/services/managed-plugin-environments.ts @@ -0,0 +1,287 @@ +/** + * Instance-level plugin environment rows declared by the managed-config + * `environments` section (`driver: "plugin"` entries, see `managed-config.ts`). + * + * Each declared entry owns exactly one row, identified by the marker + * `metadata.managedPluginEnvironment = ":"` (unique + * index `environments_managed_plugin_environment_idx`) next to + * `managedByPaperclip: true`, which puts the row behind the environments-route + * write floor on cloud-managed instances. + * + * - The first ensure adopts an existing unmarked row for the same plugin and + * driver instead of adding a second one, so a hand-made row keeps the agents + * that point at it. + * - Later ensures converge the row to the declaration and write nothing when + * it already matches. + * - Archiving is only for a declared plugin that is not running; the next + * successful ensure reactivates the row. Nothing here archives a row + * because its entry left the document. + * + * The driver config always goes through the plugin environment validator + * (`normalizeEnvironmentConfig` + `validatePluginEnvironmentDriverConfig`), + * the same path an operator-created plugin environment takes. + */ + +import { and, eq, sql } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { companies, environments } from "@paperclipai/db"; +import type { PluginEnvironmentConfig } from "@paperclipai/shared"; +import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js"; +import { normalizeEnvironmentConfig } from "./environment-config.js"; +import { validatePluginEnvironmentDriverConfig } from "./plugin-environment-driver.js"; +import type { PluginWorkerManager } from "./plugin-worker-manager.js"; + +export const MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY = "managedPluginEnvironment"; + +export function managedPluginEnvironmentMarker(input: { pluginKey: string; driverKey: string }): string { + return `${input.pluginKey}:${input.driverKey}`; +} + +export interface ManagedPluginEnvironmentInput { + name: string; + description?: string; + pluginKey: string; + driverKey: string; + driverConfig?: Record; + /** Runs the plugin's own config validation (`environmentValidateConfig`). */ + workerManager: Pick; +} + +export type ManagedPluginEnvironmentReconcileAction = "added" | "adopted" | "updated" | "unchanged"; + +export type EnvironmentRow = typeof environments.$inferSelect; + +export interface ManagedPluginEnvironmentRowResult { + row: EnvironmentRow; + action: ManagedPluginEnvironmentReconcileAction; +} + +function metadataOf(row: Pick): Record { + return (row.metadata ?? {}) as Record; +} + +function stableJson(value: unknown): string { + if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`; + if (value && typeof value === "object") { + return `{${Object.keys(value as Record) + .sort() + .map((key) => `${JSON.stringify(key)}:${stableJson((value as Record)[key])}`) + .join(",")}}`; + } + return JSON.stringify(value ?? null); +} + +function servesPair(row: EnvironmentRow, input: { pluginKey: string; driverKey: string }): boolean { + const config = (row.config ?? {}) as Record; + return config.pluginKey === input.pluginKey && config.driverKey === input.driverKey; +} + +async function validatedDriverConfig(db: Db, input: ManagedPluginEnvironmentInput): Promise { + const normalized = normalizeEnvironmentConfig({ + driver: "plugin", + config: { pluginKey: input.pluginKey, driverKey: input.driverKey, driverConfig: input.driverConfig ?? {} }, + }) as unknown as PluginEnvironmentConfig; + return await validatePluginEnvironmentDriverConfig({ + db, + workerManager: input.workerManager as PluginWorkerManager, + config: normalized, + }); +} + +/** Marker value of a row, or null when it has none (absent, or cleared to `null` / `false`). */ +function markerOf(row: Pick): string | null { + const value = metadataOf(row)[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]; + return typeof value === "string" ? value : null; +} + +/** + * Record activity for every company inside the caller's transaction, so the + * entries commit (or roll back) with the row change. Live events go to + * `publications` and are published by the caller after commit. + */ +async function logForEveryCompany( + tx: Db, + publications: ActivityPublication[], + input: { action: string; environmentId: string; details: Record }, +): Promise { + const companyIds = await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id)); + for (const companyId of companyIds) { + await logActivity(tx, { + companyId, + actorType: "system", + actorId: "managed-environment-reconciler", + action: input.action, + entityType: "environment", + entityId: input.environmentId, + details: input.details, + }, publications); + } +} + +/** + * Ensure the row for one declared plugin entry. Idempotent: a row that + * already matches the declaration is returned without a write. + */ +export async function ensureManagedPluginEnvironment( + db: Db, + input: ManagedPluginEnvironmentInput, +): Promise { + const marker = managedPluginEnvironmentMarker(input); + const config = await validatedDriverConfig(db, input); + const desiredConfig = { ...config } as Record; + const description = input.description ?? null; + + const publications: ActivityPublication[] = []; + const result = await db.transaction(async (tx): Promise => { + const reconciled = await reconcile(tx); + if (reconciled.action !== "unchanged") { + await logForEveryCompany(tx as unknown as Db, publications, { + action: `environment.managed_plugin_${reconciled.action}`, + environmentId: reconciled.row.id, + details: { pluginKey: input.pluginKey, driverKey: input.driverKey, reconciliationAction: reconciled.action }, + }); + } + return reconciled; + }); + publications.forEach(publishActivity); + return result; + + async function reconcile(tx: Parameters[0]>[0]): Promise { + // Lock every plugin row: concurrent ensures (the boot pass and a worker + // recovery, or two builds in a rolling deploy) serialize here, so only one + // of them can adopt or insert. + const pluginRows = await tx.select().from(environments).where(eq(environments.driver, "plugin")).for("update"); + let row = pluginRows.find((candidate) => markerOf(candidate) === marker) ?? null; + let adopted = false; + if (!row) { + const candidates = pluginRows + // A cleared marker (`null` / `false`, from the marker-clear request) + // counts as unmarked, so a cleared row is adopted again. + .filter((candidate) => servesPair(candidate, input) && markerOf(candidate) === null) + .sort((left, right) => + Number(right.name === input.name) - Number(left.name === input.name) + || Number(right.status === "active") - Number(left.status === "active") + || left.createdAt.getTime() - right.createdAt.getTime()); + row = candidates[0] ?? null; + adopted = row !== null; + } + + const nameOwner = await tx + .select({ id: environments.id, driver: environments.driver }) + .from(environments) + .where(eq(environments.name, input.name)) + .then((rows) => rows[0] ?? null); + + if (!row) { + if (nameOwner) { + throw new Error( + `Failed to ensure managed plugin environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`, + ); + } + const now = new Date(); + const inserted = await tx + .insert(environments) + .values({ + name: input.name, + description, + driver: "plugin", + status: "active", + config: desiredConfig, + envVars: {}, + metadata: { managedByPaperclip: true, [MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker }, + createdAt: now, + updatedAt: now, + }) + .onConflictDoNothing() + .returning() + .then((rows) => rows[0] ?? null); + if (inserted) return { row: inserted, action: "added" }; + // A concurrent pass won the marker index; converge on its row. + row = await tx + .select() + .from(environments) + .where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`) + .for("update") + .then((rows) => rows[0] ?? null); + if (!row) throw new Error("Failed to ensure managed plugin environment"); + } + + // Keep the row's own name when another row already holds the declared one. + const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name; + const currentMetadata = metadataOf(row); + const desiredMetadata = { + ...currentMetadata, + managedByPaperclip: true, + [MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker, + }; + const matches = + row.name === desiredName && + (row.description ?? null) === description && + row.status === "active" && + stableJson(row.config) === stableJson(desiredConfig) && + stableJson(currentMetadata) === stableJson(desiredMetadata); + if (matches) return { row, action: "unchanged" }; + + const updated = await tx + .update(environments) + .set({ + name: desiredName, + description, + status: "active", + config: desiredConfig, + metadata: desiredMetadata, + updatedAt: new Date(), + }) + .where(eq(environments.id, row.id)) + .returning() + .then((rows) => rows[0] ?? null); + if (!updated) throw new Error("Managed plugin environment changed during reconciliation"); + return { row: updated, action: adopted ? "adopted" : "updated" }; + } +} + +/** + * Archive the marked row of a declared plugin that is not ready or has no + * running worker, so run scheduling stops selecting it. Returns null when + * there is no active marked row. + */ +export async function archiveManagedPluginEnvironment( + db: Db, + input: { pluginKey: string; driverKey: string }, +): Promise { + const marker = managedPluginEnvironmentMarker(input); + const publications: ActivityPublication[] = []; + const archived = await db.transaction(async (tx) => { + const row = await tx + .update(environments) + .set({ status: "archived", updatedAt: new Date() }) + .where(and( + eq(environments.driver, "plugin"), + eq(environments.status, "active"), + sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`, + )) + .returning() + .then((rows) => rows[0] ?? null); + if (!row) return null; + await logForEveryCompany(tx as unknown as Db, publications, { + action: "environment.managed_provider_unavailable_archived", + environmentId: row.id, + details: { pluginKey: input.pluginKey, driverKey: input.driverKey }, + }); + return row; + }); + publications.forEach(publishActivity); + return archived; +} + +/** The marked row for one plugin entry, archived or not. */ +export async function findManagedPluginEnvironment( + db: Db, + input: { pluginKey: string; driverKey: string }, +): Promise { + return await db + .select() + .from(environments) + .where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${managedPluginEnvironmentMarker(input)}`) + .then((rows) => rows[0] ?? null); +} diff --git a/ui/src/lib/adapter-test-environment.test.ts b/ui/src/lib/adapter-test-environment.test.ts index 3422e34378e..8243815f65a 100644 --- a/ui/src/lib/adapter-test-environment.test.ts +++ b/ui/src/lib/adapter-test-environment.test.ts @@ -275,6 +275,56 @@ describe("resolveManagedSandboxEnvironmentId", () => { }); }); +describe("managed plugin environments under managed-sandbox-only", () => { + const local = makeEnvironment({ + id: "local-env", + driver: "local", + metadata: { managedByPaperclip: true, defaultForInstance: true }, + }); + // Listed first, so a lookup that only excluded `local` would pick it. + const pluginEnvironment = makeEnvironment({ + id: "plugin-env", + name: "Remote Host", + driver: "plugin", + config: { pluginKey: "example.remote-host-plugin", driverKey: "tasks", driverConfig: {} }, + metadata: { + managedByPaperclip: true, + managedPluginEnvironment: "example.remote-host-plugin:tasks", + }, + }); + const sandbox = makeEnvironment({ + id: "sandbox-env", + name: "Paperclip Sandbox", + metadata: { managedByPaperclip: true, managedSandboxProvider: "daytona" }, + }); + const environments = [local, pluginEnvironment, sandbox]; + const resolve = (agentDefaultEnvironmentId: string | null, instanceDefaultEnvironmentId: string | null) => + resolveAdapterTestEnvironmentId({ + agentDefaultEnvironmentId, + instanceDefaultEnvironmentId, + localDefaultEnvironmentId: resolveLocalDefaultEnvironmentId(environments), + managedSandboxOnly: true, + managedSandboxEnvironmentId: resolveManagedSandboxEnvironmentId(environments), + visibleEnvironmentIds: [pluginEnvironment.id, sandbox.id], + }); + + it("never mistakes the plugin row for the managed sandbox", () => { + expect(resolveManagedSandboxEnvironmentId(environments)).toBe("sandbox-env"); + expect(resolveManagedSandboxEnvironmentId([local, pluginEnvironment])).toBeNull(); + }); + + it("tests an agent without the plugin environment in the sandbox", () => { + expect(resolve("sandbox-env", "plugin-env")).toBe("sandbox-env"); + // An instance that keeps local as its default: the redirect lands on the sandbox. + expect(resolve(null, null)).toBe("sandbox-env"); + }); + + it("tests an agent on the plugin environment there", () => { + expect(resolve("plugin-env", null)).toBe("plugin-env"); + expect(resolve(null, "plugin-env")).toBe("plugin-env"); + }); +}); + describe("resolveLocalDefaultEnvironmentId", () => { it("finds the local-driver instance-default environment", () => { const environments = [ diff --git a/ui/src/lib/adapter-test-environment.ts b/ui/src/lib/adapter-test-environment.ts index 7c6d57e3b45..9628be51b10 100644 --- a/ui/src/lib/adapter-test-environment.ts +++ b/ui/src/lib/adapter-test-environment.ts @@ -86,9 +86,9 @@ export function resolveAdapterTestEnvironmentId(input: { * Find the active managed sandbox environment id in an environment list. The * platform provisioner stamps the managed environment with * `metadata.managedByPaperclip` (see `isPlatformManagedEnvironment`). The - * local-default environment also carries that stamp, so this function excludes - * the `local` driver and returns the non-local managed environment. The function - * selects only an `active` environment, never an `archived` one, so the Test + * local-default environment and managed plugin environment rows also carry + * that stamp, so this function requires the `sandbox` driver and never + * mistakes a plugin row for the managed sandbox. The function selects only an `active` environment, never an `archived` one, so the Test * resolution matches the server run-time resolver that requires an active * managed sandbox (see `findManagedSandboxEnvironment`). The Test resolution * uses this id as the redirect target under the managed-sandbox-only policy. The @@ -101,7 +101,7 @@ export function resolveManagedSandboxEnvironmentId( if (!environments) return null; const managed = environments.find( (environment) => - environment.driver !== "local" && + environment.driver === "sandbox" && environment.status === "active" && environment.metadata?.managedByPaperclip === true, );