Files
PaperClipAI/server/src/services/managed-environments.test.ts
T
DottaandPaperclip f6c6452b25 fix(server): preserve managed environment drift on boot (#10979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip creates a managed sandbox environment for each company
during boot
> - Operators can change environment fields after Paperclip creates the
environment
> - The boot reconciler replaced those changes without checking for
drift
> - This pull request adds stock hashes and transactional drift
reconciliation
> - The benefit is that Paperclip can update untouched stock fields
without losing operator work

## Linked Issues or Issue Description

**What happened?**

The managed sandbox boot reconciler rewrote the stock description,
configuration, metadata, and status on every start. It did not detect
operator changes first. A restart could therefore remove an operator's
changes.

**Expected behavior**

Paperclip must preserve operator changes by default. It must update an
untouched stock environment when Paperclip ships new stock values. It
must perform each row update and stock-hash update atomically.

**Steps to reproduce**

1. Start Paperclip and let it create the managed sandbox environment.
2. Change one Paperclip-owned stock field on that environment.
3. Restart Paperclip.
4. Observe that the previous reconciler replaced the change.

**Paperclip version or commit**

This bug reproduces on `master` before this change.

**Deployment mode**

Local development and self-hosted server boot are affected.

## What Changed

- Add a shared deterministic stock-hash and drift classifier for
built-in resources.
- Track the managed sandbox stock hash with the company-scoped built-in
resource binding.
- Reconcile the environment and its stock metadata in one transaction
with a row lock.
- Preserve operator-modified and unmanaged rows and report their skipped
update state.
- Use archive ownership tokens so provider recovery reactivates only
Paperclip-archived rows and preserves later operator archive decisions.
- Keep operator-owned environment variables and unrelated metadata out
of the stock fingerprint.
- Add activity records for managed environment creation, updates,
skipped drift, tracking initialization, and archive changes.
- Add regression tests for current stock, available stock updates,
operator drift, unmanaged rows, archive and reactivation, user-owned
fields, and concurrent reconciliation.

## Verification

- `pnpm -r typecheck`
- `pnpm build`
- `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm test:run --
--mode serialized` (128 suites passed)
- Repository general server, UI, CLI, shared, skills catalog, database,
adapter, plugin SDK, and plugin creator projects passed. Two
embedded-Postgres tests exceeded the host's five-second default under
the aggregate run and passed in the complete database project with
`--testTimeout=20000`. One timing-sensitive sandbox stream test passed
on its focused retry.
- Focused managed-environment unit and integration coverage passed: 49
tests across the drift classifier, boot report, and environment service
suites.

## Risks

- The main risk is an incorrect ownership boundary in the stock
fingerprint. The fingerprint includes only Paperclip-owned stock fields.
Tests confirm that environment variables and unrelated metadata survive
reconciliation.
- Concurrent reconciliation could otherwise overwrite a late operator
edit. The implementation locks the environment row and updates the row
and hash binding in one transaction. A concurrency test covers this
path.
- There is no schema migration. Existing managed rows initialize
tracking without replacing their current values.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI GPT-5 Codex. The exact serving snapshot and context-window size
were not exposed. The model used reasoning, repository tools, code
execution, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-07 00:41:42 -05:00

464 lines
16 KiB
TypeScript

import { describe, expect, it, vi } from "vitest";
import type { Db } from "@paperclipai/db";
import { MANAGED_CONFIG_ENV_KEY, parseManagedConfigEnv } from "./managed-config.js";
import {
applyManagedEnvironments,
type ApplyManagedEnvironmentsOptions,
} from "./managed-environments.js";
const noDb = null as unknown as Db;
function parsedConfig(overrides: Record<string, unknown> = {}) {
const config = parseManagedConfigEnv({
[MANAGED_CONFIG_ENV_KEY]: JSON.stringify({
v: 1,
mode: "cloud",
catalogVersion: "2026.720.0",
features: {},
plugins: { autoInstall: ["daytona"] },
...overrides,
}),
});
if (!config) throw new Error("expected a parsed managed config");
return config;
}
function environmentRow(overrides: Record<string, unknown> = {}) {
const now = new Date();
return {
id: "env-1",
name: "Daytona",
description: null,
driver: "sandbox",
status: "active",
config: {},
envVars: {},
metadata: {},
createdAt: now,
updatedAt: now,
...overrides,
};
}
function reconciliationResult(
overrides: Record<string, unknown> = {},
) {
return {
environment: environmentRow(),
action: "added" as const,
stockStatus: "missing" as const,
updateAvailable: false,
stockHash: "sha256:stock",
...overrides,
};
}
function readyDriverResolver(status = "ready") {
return vi.fn(async () => ({
plugin: { id: "plugin-1", pluginKey: "sandbox-providers/daytona", status },
}));
}
function runningWorkerManager(running = true) {
return { isRunning: vi.fn(() => running), getWorker: vi.fn(() => undefined) };
}
type WorkerManagerSeam = NonNullable<ApplyManagedEnvironmentsOptions["workerManager"]>;
type RecoveryHandle = NonNullable<ReturnType<WorkerManagerSeam["getWorker"]>>;
/** A worker handle fake that records `ready` listeners so tests can fire them. */
function recoveryHandle() {
const readyListeners: Array<(payload: { pluginId: string }) => void> = [];
const handle: RecoveryHandle = {
on: vi.fn((_event, listener) => {
readyListeners.push(listener);
}),
off: vi.fn((_event, listener) => {
const at = readyListeners.indexOf(listener);
if (at !== -1) readyListeners.splice(at, 1);
}),
};
return {
handle,
fireReady: () => {
for (const listener of [...readyListeners]) listener({ pluginId: "plugin-1" });
},
};
}
/** Lets the fire-and-forget recovery ensure settle. */
function tick() {
return new Promise((resolve) => setImmediate(resolve));
}
type EnvironmentsSeam = NonNullable<ApplyManagedEnvironmentsOptions["environments"]>;
function environmentsSeam(overrides: Partial<EnvironmentsSeam> = {}): EnvironmentsSeam {
return {
ensureManagedSandboxEnvironment:
overrides.ensureManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(reconciliationResult()),
archiveManagedSandboxEnvironment:
overrides.archiveManagedSandboxEnvironment ?? vi.fn().mockResolvedValue(null),
};
}
describe("applyManagedEnvironments", () => {
it("no-ops for self-hosted instances and for documents without environments", async () => {
expect(await applyManagedEnvironments(noDb, null)).toBeNull();
expect(await applyManagedEnvironments(noDb, parsedConfig())).toBeNull();
});
it("refuses startup when a forced execution mode also claims the managed sandbox slot", async () => {
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
await expect(
applyManagedEnvironments(noDb, config, {
env: { PAPERCLIP_EXECUTION_MODE: "kubernetes" },
}),
).rejects.toThrow(/mutually exclusive/);
});
it("ensures each declared environment through the provider-agnostic service call", async () => {
const ensureManagedSandboxEnvironment = vi
.fn()
.mockResolvedValue(reconciliationResult());
const config = parsedConfig({
environments: [
{
name: "Daytona",
description: "Managed Daytona sandbox.",
provider: "daytona",
config: { target: "us" },
},
],
});
const resolveSandboxProviderDriver = readyDriverResolver();
const workerManager = runningWorkerManager();
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments: environmentsSeam({ ensureManagedSandboxEnvironment }),
resolveSandboxProviderDriver,
});
expect(result).toMatchObject({ ensured: 1, failed: 0, added: 1, skipped: 0 });
expect(result).toMatchObject({ removed: 0, backedUp: 0 });
expect(resolveSandboxProviderDriver).toHaveBeenCalledWith({ db: noDb, driverKey: "daytona" });
expect(workerManager.isRunning).toHaveBeenCalledWith("plugin-1");
expect(ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
expect(ensureManagedSandboxEnvironment).toHaveBeenCalledWith({
name: "Daytona",
description: "Managed Daytona sandbox.",
provider: "daytona",
config: { target: "us" },
stockVersion: "2026.720.0",
});
// The frozen parsed config must not leak into the service (the row's
// config is mutated downstream when the provider key is forced in).
const passedConfig = ensureManagedSandboxEnvironment.mock.calls[0]?.[0]?.config;
expect(Object.isFrozen(passedConfig)).toBe(false);
});
it("waits for the bundled-plugin startup pass before ensuring anything", async () => {
const ensureManagedSandboxEnvironment = vi
.fn()
.mockResolvedValue(reconciliationResult());
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
let releasePlugins!: () => void;
const pluginsReady = new Promise<void>((resolve) => {
releasePlugins = resolve;
});
const pending = applyManagedEnvironments(noDb, config, {
env: {},
pluginsReady,
workerManager: runningWorkerManager(),
environments: environmentsSeam({ ensureManagedSandboxEnvironment }),
resolveSandboxProviderDriver: readyDriverResolver(),
});
// Give the ensure every chance to (incorrectly) run early.
await new Promise((resolve) => setImmediate(resolve));
expect(ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
releasePlugins();
expect(await pending).toMatchObject({ ensured: 1, failed: 0, added: 1 });
expect(ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
});
it("reports operator-modified rows as skipped with a stock update still available", async () => {
const environments = environmentsSeam({
ensureManagedSandboxEnvironment: vi.fn().mockResolvedValue(reconciliationResult({
action: "skipped",
stockStatus: "operator_modified",
updateAvailable: true,
})),
});
const result = await applyManagedEnvironments(noDb, parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
}), {
env: {},
workerManager: runningWorkerManager(),
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
expect(result).toMatchObject({
ensured: 0,
failed: 0,
added: 0,
updated: 0,
unchanged: 0,
skipped: 1,
outcomes: [{
action: "skipped",
stockStatus: "operator_modified",
updateAvailable: true,
}],
});
});
it("skips an entry whose provider plugin is missing and archives its stale row", async () => {
const environments = environmentsSeam({
archiveManagedSandboxEnvironment: vi.fn().mockResolvedValue(environmentRow({ status: "archived" })),
});
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager: runningWorkerManager(),
environments,
resolveSandboxProviderDriver: vi.fn(async () => null),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({
provider: "daytona",
});
});
it("skips (and counts failed) an entry whose provider plugin is not ready", async () => {
const environments = environmentsSeam();
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager: runningWorkerManager(),
environments,
resolveSandboxProviderDriver: readyDriverResolver("disabled"),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({
provider: "daytona",
});
});
it("skips an entry whose plugin record is ready but whose worker is not running", async () => {
const environments = environmentsSeam();
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const workerManager = runningWorkerManager(false);
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(workerManager.isRunning).toHaveBeenCalledWith("plugin-1");
expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({
provider: "daytona",
});
// No registered handle → no respawn is coming; degraded until next boot.
expect(workerManager.getWorker).toHaveBeenCalledWith("plugin-1");
});
it("reactivates the archived environment once when the provider worker recovers", async () => {
const environments = environmentsSeam();
const config = parsedConfig({
environments: [
{ name: "Daytona", provider: "daytona", config: { target: "us" } },
],
});
const { handle, fireReady } = recoveryHandle();
const workerManager = {
isRunning: vi.fn(() => false),
getWorker: vi.fn(() => handle),
};
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
// The boot pass itself stays degraded: archived, counted failed.
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledWith({
provider: "daytona",
});
expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
// Worker manager respawns the worker → the handle re-emits `ready`.
fireReady();
fireReady();
await tick();
expect(environments.ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
expect(environments.ensureManagedSandboxEnvironment).toHaveBeenCalledWith({
name: "Daytona",
description: undefined,
provider: "daytona",
config: { target: "us" },
stockVersion: "2026.720.0",
});
expect(handle.off).toHaveBeenCalledTimes(1);
});
it("re-checks isRunning after subscribing so a recovery during the archive is not missed", async () => {
const environments = environmentsSeam();
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const { handle } = recoveryHandle();
// Down at the gate check, already back up by the time the listener is
// registered — the `ready` event has been missed.
const workerManager = {
isRunning: vi
.fn()
.mockReturnValueOnce(false)
.mockReturnValue(true),
getWorker: vi.fn(() => handle),
};
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
await tick();
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
});
it("does not subscribe for recovery when the plugin record is missing or not ready", async () => {
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
for (const resolveSandboxProviderDriver of [
vi.fn(async () => null),
readyDriverResolver("disabled"),
]) {
const workerManager = runningWorkerManager(false);
await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments: environmentsSeam(),
resolveSandboxProviderDriver,
});
expect(workerManager.getWorker).not.toHaveBeenCalled();
}
});
it("logs, not throws, when the recovery re-ensure fails", async () => {
const environments = environmentsSeam({
ensureManagedSandboxEnvironment: vi.fn().mockRejectedValue(new Error("db exploded")),
});
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const { handle, fireReady } = recoveryHandle();
const workerManager = {
isRunning: vi.fn(() => false),
getWorker: vi.fn(() => handle),
};
await applyManagedEnvironments(noDb, config, {
env: {},
workerManager,
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
fireReady();
await tick();
// The rejection is swallowed into a log line; reaching this point without
// an unhandled rejection is the assertion.
expect(environments.ensureManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
});
it("fails closed when no worker manager is provided", async () => {
const environments = environmentsSeam();
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const result = await applyManagedEnvironments(noDb, config, {
env: {},
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.ensureManagedSandboxEnvironment).not.toHaveBeenCalled();
});
it("counts a skipped entry once even when archiving its stale row fails", async () => {
const environments = environmentsSeam({
archiveManagedSandboxEnvironment: vi.fn().mockRejectedValue(new Error("db exploded")),
});
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager: runningWorkerManager(),
environments,
resolveSandboxProviderDriver: vi.fn(async () => null),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.archiveManagedSandboxEnvironment).toHaveBeenCalledTimes(1);
});
it("is fail-safe per entry: an ensure failure is counted, not thrown", async () => {
const environments = environmentsSeam({
ensureManagedSandboxEnvironment: vi.fn().mockRejectedValue(new Error("db exploded")),
});
const config = parsedConfig({
environments: [{ name: "Daytona", provider: "daytona" }],
});
const result = await applyManagedEnvironments(noDb, config, {
env: {},
workerManager: runningWorkerManager(),
environments,
resolveSandboxProviderDriver: readyDriverResolver(),
});
expect(result).toMatchObject({ ensured: 0, failed: 1 });
expect(environments.archiveManagedSandboxEnvironment).not.toHaveBeenCalled();
});
});