feat(environments): add managed plugin environment rows (#19490)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A managed instance gets its instance-level environments from the
`PAPERCLIP_MANAGED_CONFIG` document
> - That document cannot declare a plugin environment yet, so a managed
instance cannot get a plugin-provided environment without manual setup
> - The full change is split into three pull requests. This first one
adds the plugin environment rows and their write floor, and nothing
calls them yet
> - The second pull request lets the managed config declare plugin
entries and provisions them at boot. The third sets the instance default
once and advertises the new managed-config capability
> - The benefit is a smaller review for each step, and no instance
changes behavior until the last step lands

## Linked Issues or Issue Description

**Subsystem affected**
Server environments (managed-config environment provisioning,
environment routes) and the board UI's adapter Test target.

**Problem or motivation**
A managed instance cannot get a plugin environment from its managed
config. Company settings cannot create one either. So each managed
instance that needs one requires manual setup.

**Proposed solution**
Over three pull requests, let an `environments` entry in the managed
config declare `driver: "plugin"`. Boot then provisions one locked row
for each plugin entry, and makes one of them the instance default once
on new instances. This pull request adds only the row layer: create or
adopt a marked row, archive it, find it, and keep tenants from editing
it.

**Alternatives considered**
Manual setup on each instance, and a new control-plane endpoint. Both
need a step outside the deploy. The managed config already delivers the
sandbox entry.

**Roadmap alignment**
This extends the existing managed-config `environments` section, which
already provisions the managed sandbox. It adds no new product surface.

## What Changed

- New `server/src/services/managed-plugin-environments.ts`, used through
new `environmentService` methods (`ensureManagedPluginEnvironment`,
`archiveManagedPluginEnvironment`, `findManagedPluginEnvironment`):
- The ensure validates the driver config through
`normalizeEnvironmentConfig` and
`validatePluginEnvironmentDriverConfig`, the same path an
operator-created plugin environment takes.
- It marks the row `managedByPaperclip: true` and
`managedPluginEnvironment: "<pluginKey>:<driverKey>"`.
- It adopts an existing unmarked row for the same plugin and driver
instead of adding a second one, so a hand-made row keeps its agents.
- A row that already matches gets no write. Changes write activity log
entries.
- Archive sets an active marked row to `archived`. The next ensure
reactivates it.
- Migration `0437`: unique index
`environments_managed_plugin_environment_idx` on `metadata ->>
'managedPluginEnvironment'`. This is the only schema change of the three
pull requests. Snapshot `0432` is pruned to keep the five newest.
- `server/src/routes/environments.ts`:
`isPlatformProvisionedEnvironment` also covers the new marker (write
floor and redaction). `managedPluginEnvironment` is a reserved marker
key, so a client cannot stamp it on its own row. Nothing provisions
marked rows yet, so the marker-clear escape hatch stays open for them.
The next pull request refuses it while the managed config declares the
row's entry.
- `ui/src/lib/adapter-test-environment.ts`:
`resolveManagedSandboxEnvironmentId` requires `driver === "sandbox"`.
The plugin row also carries `managedByPaperclip`, so the old "not local"
check could take it for the managed sandbox.

## Verification

- `cd server && npx vitest run
src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres):
- The ensure creates the marked row through the plugin validator. A
second ensure writes nothing and logs nothing.
- A hand-made row for the same plugin and driver is adopted. It keeps
its id and its agents.
  - Archive, then ensure, reactivates the row.
- The ensure refuses while the plugin has no running worker, and refuses
a name that another environment holds.
  - The unique index allows one row per marker.
- `npx vitest run src/__tests__/environment-routes.test.ts`: on a
cloud-managed instance, edit, env-var edit, status change and delete of
a marked plugin row are refused, including for instance admins. The
marker alone keeps the floor. A client cannot stamp the marker. The
marker clear is allowed.
- `npx vitest run src/__tests__/environment-service.test.ts`: passes.
- `cd ui && npx vitest run src/lib/adapter-test-environment.test.ts`:
the plugin row is never taken for the managed sandbox. Under
managed-sandbox-only, an agent on the sandbox tests in the sandbox, and
an agent on the plugin environment tests there.
- `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/ui typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them. `pnpm verify` asks for broader verification
because of the migration.

## Risks

- Migration: one new partial unique index on a small table. No row
carries the marker before this change, so the index cannot fail on
existing data.
- No caller uses the new ensure yet, so no instance changes behavior.
- The UI change affects only lists that hold a non-sandbox row with
`managedByPaperclip`. Today that is only the local row, which the old
check already excluded.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (file edits, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Nicky LeachandClaude Opus 5.5 authored and GitHub committed 2026-11-06 16:52:43 +00:00
1 parent 871a9b3180
commit 21a1e78852
11 files changed
+754 -8

No files matched your search

@@ -0,0 +1 @@
CREATE UNIQUE INDEX "environments_managed_plugin_environment_idx" ON "environments" USING btree (("metadata" ->> 'managedPluginEnvironment')) WHERE jsonb_typeof("environments"."metadata" -> 'managedPluginEnvironment') = 'string';
@@ -1,6 +1,6 @@
{
"id": "a87278a8-15b3-4187-94a9-f4bb79d8640d",
"prevId": "9fcf29d1-a9a6-4ead-8d43-7e087a2b2b24",
"id": "2c0cb580-67fc-45d3-9ca5-cce4148dd1a4",
"prevId": "a80d63fe-5111-49e8-a39a-7a5589b3b31a",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -4058,6 +4058,28 @@
"method": "btree",
"with": {}
},
"agent_wakeup_requests_tree_hold_resume_uq": {
"name": "agent_wakeup_requests_tree_hold_resume_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "idempotency_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"agent_wakeup_requests\".\"idempotency_key\" LIKE 'tree-hold-resume:%' AND \"agent_wakeup_requests\".\"status\" <> 'skipped'",
"concurrently": true,
"method": "btree",
"with": {}
},
"agent_wakeup_requests_chat_completion_uq": {
"name": "agent_wakeup_requests_chat_completion_uq",
"columns": [
@@ -28962,6 +28984,22 @@
"method": "btree",
"with": {}
},
"environments_managed_plugin_environment_idx": {
"name": "environments_managed_plugin_environment_idx",
"columns": [
{
"expression": "(\"metadata\" ->> 'managedPluginEnvironment')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "jsonb_typeof(\"environments\".\"metadata\" -> 'managedPluginEnvironment') = 'string'",
"concurrently": false,
"method": "btree",
"with": {}
},
"environments_name_idx": {
"name": "environments_name_idx",
"columns": [
@@ -35053,6 +35091,28 @@
"method": "btree",
"with": {}
},
"heartbeat_runs_tree_hold_suppression_idx": {
"name": "heartbeat_runs_tree_hold_suppression_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "(\"result_json\"->'treeHoldSuppression'->>'holdId')",
"asc": true,
"isExpression": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"status\" = 'cancelled' and \"heartbeat_runs\".\"result_json\" ? 'treeHoldSuppression'",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cost_accounting_pending_idx": {
"name": "heartbeat_runs_cost_accounting_pending_idx",
"columns": [
@@ -42492,6 +42552,12 @@
"primaryKey": false,
"notNull": false
},
"origin_issue_id": {
"name": "origin_issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"origin_identity_context_id": {
"name": "origin_identity_context_id",
"type": "uuid",
@@ -42935,6 +43001,27 @@
"method": "btree",
"with": {}
},
"issues_company_origin_issue_idx": {
"name": "issues_company_origin_issue_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "origin_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"issues_company_project_workspace_idx": {
"name": "issues_company_project_workspace_idx",
"columns": [
@@ -43534,6 +43621,19 @@
"onDelete": "no action",
"onUpdate": "no action"
},
"issues_origin_issue_id_issues_id_fk": {
"name": "issues_origin_issue_id_issues_id_fk",
"tableFrom": "issues",
"tableTo": "issues",
"columnsFrom": [
"origin_issue_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
},
"issues_execution_workspace_id_execution_workspaces_id_fk": {
"name": "issues_execution_workspace_id_execution_workspaces_id_fk",
"tableFrom": "issues",
@@ -61884,6 +61984,13 @@
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": true
},
"classic_workspace_enabled": {
"name": "classic_workspace_enabled",
"type": "boolean",
"primaryKey": false,
"notNull": true,
"default": false
},
"created_at": {
+7
View File
@@ -3046,6 +3046,13 @@
"when": 1793952565204,
"tag": "0436_legal_reavers",
"breakpoints": true
},
{
"idx": 437,
"version": "7",
"when": 1793982073367,
"tag": "0437_workable_oracle",
"breakpoints": true
}
]
}
+6
View File
@@ -25,6 +25,12 @@ export const environments = pgTable(
.where(
sql`${table.driver} = 'sandbox' AND (${table.metadata} ->> 'managedByPaperclip')::boolean = true`,
),
// One row per managed-config plugin entry: the marker is the entry's
// "<pluginKey>:<driverKey>" pair (`ensureManagedPluginEnvironment`).
// Only string markers count; a cleared marker (`null` / `false`) is not one.
managedPluginEnvironmentIdx: uniqueIndex("environments_managed_plugin_environment_idx")
.on(sql`(${table.metadata} ->> 'managedPluginEnvironment')`)
.where(sql`jsonb_typeof(${table.metadata} -> 'managedPluginEnvironment') = 'string'`),
nameIdx: uniqueIndex("environments_name_idx").on(table.name),
}),
);
@@ -1020,6 +1020,75 @@ describe("environment routes", () => {
expect(mockEnvironmentService.create).not.toHaveBeenCalled();
});
describe("managed plugin environment rows", () => {
const PLUGIN_KEY = "example.remote-host-plugin";
const pluginRow = () => ({
...createPlatformSandboxEnvironment(),
id: "env-plugin-1",
name: "Remote Host",
driver: "plugin",
config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} },
envVars: {},
metadata: { managedByPaperclip: true, managedPluginEnvironment: `${PLUGIN_KEY}:tasks` },
});
it("refuses edits, env-var edits, and deletes, including for instance admins", async () => {
mockEnvironmentService.getById.mockResolvedValue(pluginRow());
const app = createApp(ownerAdminActor);
for (const body of [{ name: "Renamed" }, { envVars: { A: "1" } }, { status: "archived" }]) {
const res = await request(app).patch("/api/environments/env-plugin-1").send(body);
expect(res.status).toBe(403);
expect(res.body.details).toMatchObject({ code: "environment_platform_managed" });
}
const deleted = await request(app).delete("/api/environments/env-plugin-1");
expect(deleted.status).toBe(403);
expect(deleted.body.details).toMatchObject({ code: "environment_platform_managed" });
expect(mockEnvironmentService.update).not.toHaveBeenCalled();
expect(mockEnvironmentService.removeIfDeletable).not.toHaveBeenCalled();
});
it("keeps the row behind the floor on the plugin marker alone", async () => {
mockEnvironmentService.getById.mockResolvedValue({
...pluginRow(),
metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` },
});
const res = await request(createApp(ownerAdminActor)).patch("/api/environments/env-plugin-1").send({ name: "Renamed" });
expect(res.status).toBe(403);
expect(res.body.details).toMatchObject({ code: "environment_platform_managed" });
});
it("allows the marker clear while no managed-config entry can own the row", async () => {
// Nothing provisions marked plugin rows yet, so a marker is stale and
// stays recoverable. The managed-config plugin entry PR refuses this
// while the document declares the row's entry.
const row = pluginRow();
mockEnvironmentService.getById.mockResolvedValue(row);
mockEnvironmentService.update.mockResolvedValue({ ...row, metadata: {} });
const res = await request(createApp(ownerAdminActor))
.patch("/api/environments/env-plugin-1")
.send({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } });
expect(res.status).toBe(200);
expect(mockEnvironmentService.update).toHaveBeenCalled();
});
it("refuses a client that stamps the plugin marker on its own row", async () => {
const res = await request(createApp(ownerAdminActor))
.post("/api/companies/company-1/environments")
.send({
name: "Fake remote host",
driver: "plugin",
config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} },
metadata: { managedPluginEnvironment: `${PLUGIN_KEY}:tasks` },
});
expect(res.status).toBe(422);
expect(res.body.details).toMatchObject({ code: "environment_platform_marker_reserved" });
expect(mockEnvironmentService.create).not.toHaveBeenCalled();
});
});
it("rejects tenant patches that stamp platform markers so the row cannot become locked", async () => {
const tenantEnvironment = {
...createPlatformSandboxEnvironment(),
@@ -0,0 +1,178 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import {
activityLog,
agents,
companies,
createDb,
environments,
plugins,
type Db,
} from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { environmentService } from "../services/environments.js";
const PLUGIN_KEY = "example.remote-host-plugin";
const MARKER = `${PLUGIN_KEY}:tasks`;
const support = await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("managed plugin environment rows", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
let pluginId: string;
let running: boolean;
const call = vi.fn(async (_id: string, method: string, input: { config: Record<string, unknown> }) =>
method === "environmentValidateConfig" ? { ok: true, normalizedConfig: { ...input.config, normalized: true } } : null);
const workerManager = { isRunning: vi.fn(() => running), call };
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-managed-plugin-environment-rows-");
db = createDb(database.connectionString);
}, 90_000);
afterAll(async () => { await database?.cleanup(); });
beforeEach(async () => {
running = true;
call.mockClear();
await db.delete(activityLog);
await db.delete(agents);
await db.delete(environments);
await db.delete(plugins);
await db.delete(companies);
pluginId = randomUUID();
await db.insert(plugins).values({
id: pluginId,
pluginKey: PLUGIN_KEY,
packageName: "@example/remote-host-plugin",
version: "1.0.0",
status: "ready",
manifestJson: {
capabilities: ["environment.drivers.register"],
environmentDrivers: [{ driverKey: "tasks", requiresAgent: true, supportsTasks: true }],
} as never,
});
});
const ensure = (overrides: Record<string, unknown> = {}) =>
environmentService(db).ensureManagedPluginEnvironment({
name: "Remote Host",
description: "Runs the agent on a remote host",
pluginKey: PLUGIN_KEY,
driverKey: "tasks",
driverConfig: {},
workerManager,
...overrides,
});
const pluginRows = () => db.select().from(environments).where(eq(environments.driver, "plugin"));
const company = async () => {
const id = randomUUID();
await db.insert(companies).values({ id, name: "Managed", issuePrefix: `M${id.slice(0, 7)}` });
return id;
};
it("creates the marked row through the plugin validator, then changes nothing", async () => {
await company();
const first = await ensure();
expect(first.action).toBe("added");
expect(first.environment).toMatchObject({
name: "Remote Host",
description: "Runs the agent on a remote host",
driver: "plugin",
status: "active",
config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: { normalized: true } },
metadata: { managedByPaperclip: true, managedPluginEnvironment: MARKER },
});
expect(call).toHaveBeenCalledWith(pluginId, "environmentValidateConfig", { driverKey: "tasks", config: {} });
expect(await db.select().from(activityLog)).toHaveLength(1);
const second = await ensure();
const rows = await pluginRows();
expect(second.action).toBe("unchanged");
expect(rows).toHaveLength(1);
expect(rows[0]!.updatedAt.getTime()).toBe(first.environment.updatedAt.getTime());
expect(await db.select().from(activityLog)).toHaveLength(1);
});
it("adopts a hand-made row for the same plugin and driver, keeping its agents", async () => {
const companyId = await company();
const [handMade] = await db.insert(environments).values({
name: "Remote Host (manual)",
driver: "plugin",
status: "active",
config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} },
metadata: { note: "made by hand" },
}).returning();
const agentId = randomUUID();
await db.insert(agents).values({ id: agentId, companyId, name: "Agent", adapterType: "process", defaultEnvironmentId: handMade!.id });
const result = await ensure();
const rows = await pluginRows();
expect(result.action).toBe("adopted");
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
id: handMade!.id,
name: "Remote Host",
metadata: { note: "made by hand", managedByPaperclip: true, managedPluginEnvironment: MARKER },
});
const [kept] = await db.select().from(agents).where(eq(agents.id, agentId));
expect(kept!.defaultEnvironmentId).toBe(handMade!.id);
});
it("adopts a row again after its markers were cleared", async () => {
const first = await ensure();
// What the route's marker-clear request stores.
await db.update(environments)
.set({ metadata: { managedByPaperclip: null, managedPluginEnvironment: null } })
.where(eq(environments.id, first.environment.id));
const again = await ensure();
expect(again.action).toBe("adopted");
expect(again.environment.id).toBe(first.environment.id);
expect(await pluginRows()).toHaveLength(1);
});
it("lets several rows hold a cleared marker", async () => {
const config = { pluginKey: PLUGIN_KEY, driverKey: "other", driverConfig: {} };
await db.insert(environments).values([
{ name: "Cleared one", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } },
{ name: "Cleared two", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: false } },
{ name: "Cleared three", driver: "plugin", status: "active", config, metadata: { managedPluginEnvironment: null } },
]);
expect(await pluginRows()).toHaveLength(3);
});
it("archives the active marked row and reactivates it on the next ensure", async () => {
await ensure();
const archived = await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" });
expect(archived?.status).toBe("archived");
expect(await environmentService(db).archiveManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" })).toBeNull();
expect((await environmentService(db).findManagedPluginEnvironment({ pluginKey: PLUGIN_KEY, driverKey: "tasks" }))?.status)
.toBe("archived");
const reactivated = await ensure();
expect(reactivated.action).toBe("updated");
expect(reactivated.environment.status).toBe("active");
});
it("refuses to ensure while the plugin has no running worker", async () => {
running = false;
await expect(ensure()).rejects.toThrow(/has no running worker/);
expect(await pluginRows()).toHaveLength(0);
});
it("refuses to take a name another environment already holds", async () => {
await db.insert(environments).values({ name: "Remote Host", driver: "ssh", status: "active", config: {} });
await expect(ensure()).rejects.toThrow(/already exists with driver "ssh"/);
});
it("allows at most one row per marker", async () => {
await ensure();
await expect(db.insert(environments).values({
name: "Second remote host",
driver: "plugin",
status: "active",
config: { pluginKey: PLUGIN_KEY, driverKey: "tasks", driverConfig: {} },
metadata: { managedPluginEnvironment: MARKER },
})).rejects.toThrow();
});
});
+7 -2
View File
@@ -19,6 +19,7 @@ import {
import { conflict, forbidden, unprocessable } from "../errors.js";
import { isCloudManagedInstance } from "../services/cloud-instance.js";
import { getManagedInstanceConfig, SECRET_LIKE_CONFIG_KEY_PATTERN } from "../services/managed-config.js";
import { MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY } from "../services/managed-plugin-environments.js";
import { parseExecutionPolicyBootstrapEnv } from "../services/execution-policy-bootstrap.js";
import { isExecutionForcedToKubernetes } from "../services/execution-allowlist.js";
import { validate } from "../middleware/validate.js";
@@ -69,8 +70,10 @@ function isPlainRecord(value: unknown): value is Record<string, unknown> {
/**
* Whether this environment row was provisioned by the platform: the
* managed-config environment provisioner marker, or the legacy managed
* Kubernetes wrapper marker (rows created by builds that predate the
* managed-config environment provisioner marker, the managed plugin
* environment marker (`"<pluginKey>:<driverKey>"`, stamped next to
* `managedByPaperclip` on a managed-config plugin entry's row), or the legacy
* managed Kubernetes wrapper marker (rows created by builds that predate the
* generalized provisioner and have not been adopted yet).
*/
export function isPlatformProvisionedEnvironment(environment: {
@@ -78,6 +81,7 @@ export function isPlatformProvisionedEnvironment(environment: {
}): boolean {
return (
environment.metadata?.managedByPaperclip === true ||
typeof environment.metadata?.[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY] === "string" ||
environment.metadata?.managedKubernetesSandbox === true
);
}
@@ -151,6 +155,7 @@ function isTenantEditableManagedSandbox(environment: {
const PLATFORM_PROVISIONED_MARKER_KEYS = [
"managedByPaperclip",
MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY,
"managedKubernetesSandbox",
] as const;
+36
View File
@@ -50,6 +50,13 @@ import {
stockHash,
type ManagedResourceStockStatus,
} from "./managed-resource-drift.js";
import {
archiveManagedPluginEnvironment as archiveManagedPluginEnvironmentRow,
ensureManagedPluginEnvironment as ensureManagedPluginEnvironmentRow,
findManagedPluginEnvironment as findManagedPluginEnvironmentRow,
type ManagedPluginEnvironmentInput,
type ManagedPluginEnvironmentReconcileAction,
} from "./managed-plugin-environments.js";
export async function attachSandboxCreation(db: Db, id: string, scope: { companyId: string; environmentId: string; heartbeatRunId?: string | null },
attach: (tx: Db) => Promise<EnvironmentLease>): Promise<EnvironmentLease> {
@@ -1062,6 +1069,35 @@ export function environmentService(db: Db) {
archiveManagedSandboxEnvironment,
/**
* Ensure the instance-level row of one managed-config plugin entry,
* validating its driver config through the plugin environment validator.
* Adopts an unmarked row for the same plugin and driver (see
* `managed-plugin-environments.ts`).
*/
ensureManagedPluginEnvironment: async (
input: ManagedPluginEnvironmentInput,
): Promise<{ environment: Environment; action: ManagedPluginEnvironmentReconcileAction }> => {
const result = await ensureManagedPluginEnvironmentRow(db, input);
return { environment: toEnvironment(result.row), action: result.action };
},
/** Archive the active row of a managed plugin entry whose plugin is not running. */
archiveManagedPluginEnvironment: async (
input: { pluginKey: string; driverKey: string },
): Promise<Environment | null> => {
const row = await archiveManagedPluginEnvironmentRow(db, input);
return row ? toEnvironment(row) : null;
},
/** The marked row of a managed plugin entry, archived or not. */
findManagedPluginEnvironment: async (
input: { pluginKey: string; driverKey: string },
): Promise<Environment | null> => {
const row = await findManagedPluginEnvironmentRow(db, input);
return row ? toEnvironment(row) : null;
},
/**
* Idempotently ensure a managed Kubernetes sandbox environment exists for
* an instance, configured from instance/operator-supplied config. A thin
@@ -0,0 +1,287 @@
/**
* Instance-level plugin environment rows declared by the managed-config
* `environments` section (`driver: "plugin"` entries, see `managed-config.ts`).
*
* Each declared entry owns exactly one row, identified by the marker
* `metadata.managedPluginEnvironment = "<pluginKey>:<driverKey>"` (unique
* index `environments_managed_plugin_environment_idx`) next to
* `managedByPaperclip: true`, which puts the row behind the environments-route
* write floor on cloud-managed instances.
*
* - The first ensure adopts an existing unmarked row for the same plugin and
* driver instead of adding a second one, so a hand-made row keeps the agents
* that point at it.
* - Later ensures converge the row to the declaration and write nothing when
* it already matches.
* - Archiving is only for a declared plugin that is not running; the next
* successful ensure reactivates the row. Nothing here archives a row
* because its entry left the document.
*
* The driver config always goes through the plugin environment validator
* (`normalizeEnvironmentConfig` + `validatePluginEnvironmentDriverConfig`),
* the same path an operator-created plugin environment takes.
*/
import { and, eq, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { companies, environments } from "@paperclipai/db";
import type { PluginEnvironmentConfig } from "@paperclipai/shared";
import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js";
import { normalizeEnvironmentConfig } from "./environment-config.js";
import { validatePluginEnvironmentDriverConfig } from "./plugin-environment-driver.js";
import type { PluginWorkerManager } from "./plugin-worker-manager.js";
export const MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY = "managedPluginEnvironment";
export function managedPluginEnvironmentMarker(input: { pluginKey: string; driverKey: string }): string {
return `${input.pluginKey}:${input.driverKey}`;
}
export interface ManagedPluginEnvironmentInput {
name: string;
description?: string;
pluginKey: string;
driverKey: string;
driverConfig?: Record<string, unknown>;
/** Runs the plugin's own config validation (`environmentValidateConfig`). */
workerManager: Pick<PluginWorkerManager, "call" | "isRunning">;
}
export type ManagedPluginEnvironmentReconcileAction = "added" | "adopted" | "updated" | "unchanged";
export type EnvironmentRow = typeof environments.$inferSelect;
export interface ManagedPluginEnvironmentRowResult {
row: EnvironmentRow;
action: ManagedPluginEnvironmentReconcileAction;
}
function metadataOf(row: Pick<EnvironmentRow, "metadata">): Record<string, unknown> {
return (row.metadata ?? {}) as Record<string, unknown>;
}
function stableJson(value: unknown): string {
if (Array.isArray(value)) return `[${value.map(stableJson).join(",")}]`;
if (value && typeof value === "object") {
return `{${Object.keys(value as Record<string, unknown>)
.sort()
.map((key) => `${JSON.stringify(key)}:${stableJson((value as Record<string, unknown>)[key])}`)
.join(",")}}`;
}
return JSON.stringify(value ?? null);
}
function servesPair(row: EnvironmentRow, input: { pluginKey: string; driverKey: string }): boolean {
const config = (row.config ?? {}) as Record<string, unknown>;
return config.pluginKey === input.pluginKey && config.driverKey === input.driverKey;
}
async function validatedDriverConfig(db: Db, input: ManagedPluginEnvironmentInput): Promise<PluginEnvironmentConfig> {
const normalized = normalizeEnvironmentConfig({
driver: "plugin",
config: { pluginKey: input.pluginKey, driverKey: input.driverKey, driverConfig: input.driverConfig ?? {} },
}) as unknown as PluginEnvironmentConfig;
return await validatePluginEnvironmentDriverConfig({
db,
workerManager: input.workerManager as PluginWorkerManager,
config: normalized,
});
}
/** Marker value of a row, or null when it has none (absent, or cleared to `null` / `false`). */
function markerOf(row: Pick<EnvironmentRow, "metadata">): string | null {
const value = metadataOf(row)[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY];
return typeof value === "string" ? value : null;
}
/**
* Record activity for every company inside the caller's transaction, so the
* entries commit (or roll back) with the row change. Live events go to
* `publications` and are published by the caller after commit.
*/
async function logForEveryCompany(
tx: Db,
publications: ActivityPublication[],
input: { action: string; environmentId: string; details: Record<string, unknown> },
): Promise<void> {
const companyIds = await tx.select({ id: companies.id }).from(companies).then((rows) => rows.map((row) => row.id));
for (const companyId of companyIds) {
await logActivity(tx, {
companyId,
actorType: "system",
actorId: "managed-environment-reconciler",
action: input.action,
entityType: "environment",
entityId: input.environmentId,
details: input.details,
}, publications);
}
}
/**
* Ensure the row for one declared plugin entry. Idempotent: a row that
* already matches the declaration is returned without a write.
*/
export async function ensureManagedPluginEnvironment(
db: Db,
input: ManagedPluginEnvironmentInput,
): Promise<ManagedPluginEnvironmentRowResult> {
const marker = managedPluginEnvironmentMarker(input);
const config = await validatedDriverConfig(db, input);
const desiredConfig = { ...config } as Record<string, unknown>;
const description = input.description ?? null;
const publications: ActivityPublication[] = [];
const result = await db.transaction(async (tx): Promise<ManagedPluginEnvironmentRowResult> => {
const reconciled = await reconcile(tx);
if (reconciled.action !== "unchanged") {
await logForEveryCompany(tx as unknown as Db, publications, {
action: `environment.managed_plugin_${reconciled.action}`,
environmentId: reconciled.row.id,
details: { pluginKey: input.pluginKey, driverKey: input.driverKey, reconciliationAction: reconciled.action },
});
}
return reconciled;
});
publications.forEach(publishActivity);
return result;
async function reconcile(tx: Parameters<Parameters<Db["transaction"]>[0]>[0]): Promise<ManagedPluginEnvironmentRowResult> {
// Lock every plugin row: concurrent ensures (the boot pass and a worker
// recovery, or two builds in a rolling deploy) serialize here, so only one
// of them can adopt or insert.
const pluginRows = await tx.select().from(environments).where(eq(environments.driver, "plugin")).for("update");
let row = pluginRows.find((candidate) => markerOf(candidate) === marker) ?? null;
let adopted = false;
if (!row) {
const candidates = pluginRows
// A cleared marker (`null` / `false`, from the marker-clear request)
// counts as unmarked, so a cleared row is adopted again.
.filter((candidate) => servesPair(candidate, input) && markerOf(candidate) === null)
.sort((left, right) =>
Number(right.name === input.name) - Number(left.name === input.name)
|| Number(right.status === "active") - Number(left.status === "active")
|| left.createdAt.getTime() - right.createdAt.getTime());
row = candidates[0] ?? null;
adopted = row !== null;
}
const nameOwner = await tx
.select({ id: environments.id, driver: environments.driver })
.from(environments)
.where(eq(environments.name, input.name))
.then((rows) => rows[0] ?? null);
if (!row) {
if (nameOwner) {
throw new Error(
`Failed to ensure managed plugin environment: environment "${input.name}" already exists with driver "${nameOwner.driver}"`,
);
}
const now = new Date();
const inserted = await tx
.insert(environments)
.values({
name: input.name,
description,
driver: "plugin",
status: "active",
config: desiredConfig,
envVars: {},
metadata: { managedByPaperclip: true, [MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker },
createdAt: now,
updatedAt: now,
})
.onConflictDoNothing()
.returning()
.then((rows) => rows[0] ?? null);
if (inserted) return { row: inserted, action: "added" };
// A concurrent pass won the marker index; converge on its row.
row = await tx
.select()
.from(environments)
.where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`)
.for("update")
.then((rows) => rows[0] ?? null);
if (!row) throw new Error("Failed to ensure managed plugin environment");
}
// Keep the row's own name when another row already holds the declared one.
const desiredName = nameOwner && nameOwner.id !== row.id ? row.name : input.name;
const currentMetadata = metadataOf(row);
const desiredMetadata = {
...currentMetadata,
managedByPaperclip: true,
[MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY]: marker,
};
const matches =
row.name === desiredName &&
(row.description ?? null) === description &&
row.status === "active" &&
stableJson(row.config) === stableJson(desiredConfig) &&
stableJson(currentMetadata) === stableJson(desiredMetadata);
if (matches) return { row, action: "unchanged" };
const updated = await tx
.update(environments)
.set({
name: desiredName,
description,
status: "active",
config: desiredConfig,
metadata: desiredMetadata,
updatedAt: new Date(),
})
.where(eq(environments.id, row.id))
.returning()
.then((rows) => rows[0] ?? null);
if (!updated) throw new Error("Managed plugin environment changed during reconciliation");
return { row: updated, action: adopted ? "adopted" : "updated" };
}
}
/**
* Archive the marked row of a declared plugin that is not ready or has no
* running worker, so run scheduling stops selecting it. Returns null when
* there is no active marked row.
*/
export async function archiveManagedPluginEnvironment(
db: Db,
input: { pluginKey: string; driverKey: string },
): Promise<EnvironmentRow | null> {
const marker = managedPluginEnvironmentMarker(input);
const publications: ActivityPublication[] = [];
const archived = await db.transaction(async (tx) => {
const row = await tx
.update(environments)
.set({ status: "archived", updatedAt: new Date() })
.where(and(
eq(environments.driver, "plugin"),
eq(environments.status, "active"),
sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${marker}`,
))
.returning()
.then((rows) => rows[0] ?? null);
if (!row) return null;
await logForEveryCompany(tx as unknown as Db, publications, {
action: "environment.managed_provider_unavailable_archived",
environmentId: row.id,
details: { pluginKey: input.pluginKey, driverKey: input.driverKey },
});
return row;
});
publications.forEach(publishActivity);
return archived;
}
/** The marked row for one plugin entry, archived or not. */
export async function findManagedPluginEnvironment(
db: Db,
input: { pluginKey: string; driverKey: string },
): Promise<EnvironmentRow | null> {
return await db
.select()
.from(environments)
.where(sql`${environments.metadata} ->> ${MANAGED_PLUGIN_ENVIRONMENT_MARKER_KEY} = ${managedPluginEnvironmentMarker(input)}`)
.then((rows) => rows[0] ?? null);
}
@@ -275,6 +275,56 @@ describe("resolveManagedSandboxEnvironmentId", () => {
});
});
describe("managed plugin environments under managed-sandbox-only", () => {
const local = makeEnvironment({
id: "local-env",
driver: "local",
metadata: { managedByPaperclip: true, defaultForInstance: true },
});
// Listed first, so a lookup that only excluded `local` would pick it.
const pluginEnvironment = makeEnvironment({
id: "plugin-env",
name: "Remote Host",
driver: "plugin",
config: { pluginKey: "example.remote-host-plugin", driverKey: "tasks", driverConfig: {} },
metadata: {
managedByPaperclip: true,
managedPluginEnvironment: "example.remote-host-plugin:tasks",
},
});
const sandbox = makeEnvironment({
id: "sandbox-env",
name: "Paperclip Sandbox",
metadata: { managedByPaperclip: true, managedSandboxProvider: "daytona" },
});
const environments = [local, pluginEnvironment, sandbox];
const resolve = (agentDefaultEnvironmentId: string | null, instanceDefaultEnvironmentId: string | null) =>
resolveAdapterTestEnvironmentId({
agentDefaultEnvironmentId,
instanceDefaultEnvironmentId,
localDefaultEnvironmentId: resolveLocalDefaultEnvironmentId(environments),
managedSandboxOnly: true,
managedSandboxEnvironmentId: resolveManagedSandboxEnvironmentId(environments),
visibleEnvironmentIds: [pluginEnvironment.id, sandbox.id],
});
it("never mistakes the plugin row for the managed sandbox", () => {
expect(resolveManagedSandboxEnvironmentId(environments)).toBe("sandbox-env");
expect(resolveManagedSandboxEnvironmentId([local, pluginEnvironment])).toBeNull();
});
it("tests an agent without the plugin environment in the sandbox", () => {
expect(resolve("sandbox-env", "plugin-env")).toBe("sandbox-env");
// An instance that keeps local as its default: the redirect lands on the sandbox.
expect(resolve(null, null)).toBe("sandbox-env");
});
it("tests an agent on the plugin environment there", () => {
expect(resolve("plugin-env", null)).toBe("plugin-env");
expect(resolve(null, "plugin-env")).toBe("plugin-env");
});
});
describe("resolveLocalDefaultEnvironmentId", () => {
it("finds the local-driver instance-default environment", () => {
const environments = [
+4 -4
View File
@@ -86,9 +86,9 @@ export function resolveAdapterTestEnvironmentId(input: {
* Find the active managed sandbox environment id in an environment list. The
* platform provisioner stamps the managed environment with
* `metadata.managedByPaperclip` (see `isPlatformManagedEnvironment`). The
* local-default environment also carries that stamp, so this function excludes
* the `local` driver and returns the non-local managed environment. The function
* selects only an `active` environment, never an `archived` one, so the Test
* local-default environment and managed plugin environment rows also carry
* that stamp, so this function requires the `sandbox` driver and never
* mistakes a plugin row for the managed sandbox. The function selects only an `active` environment, never an `archived` one, so the Test
* resolution matches the server run-time resolver that requires an active
* managed sandbox (see `findManagedSandboxEnvironment`). The Test resolution
* uses this id as the redirect target under the managed-sandbox-only policy. The
@@ -101,7 +101,7 @@ export function resolveManagedSandboxEnvironmentId(
if (!environments) return null;
const managed = environments.find(
(environment) =>
environment.driver !== "local" &&
environment.driver === "sandbox" &&
environment.status === "active" &&
environment.metadata?.managedByPaperclip === true,
);