Files
PaperClipAI/server/src/middleware/cloud-tenant-actor.test.ts
T
Devin FoleyandPaperclip 3367b75ccc fix: fence accepted work and cleanup before idle sleep (#15522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A host can stop an idle instance to reduce unused compute.
> - Zero active runs do not prove that requests, saved work, or cleanup
are complete.
> - A client can disconnect while its request still writes, and failed
cleanup can remain only in memory or on disk.
> - This pull request holds new admission and checks accepted work,
cleanup, and persisted work under an owned drain.
> - The host gets an empty report only while those checks remain valid.

## Linked Issues or Issue Description

Refs #13413. That companion PR uses the same owned-hold vocabulary for
runtime services. This PR covers HTTP requests, scheduler work,
accounting, cleanup, and the instance work inventory. It does not
include the preview gateway changes.

**What existing behavior does this improve?**

The instance task-drain API reports process counters. It does not prove
that a host can safely stop the instance for idle sleep.

**Current behavior**

A quiet run set can coexist with an unfinished request, cleanup after a
completed run, future work, or a failed accounting write.

**Proposed behavior**

Provide a bounded owned idle hold. Block new ingress, track accepted
handler promises, inspect durable and local work, and return `none` only
when the same hold stays quiet through the checks. Keep normal
deployment drains compatible.

**Reason and benefit**

Hosts can identify eligible idle instances without treating a disconnect
or failed cleanup write as completed work.

## What Changed

- Add `purpose: "idle"`, a bounded TTL, unique owners, and owner-checked
release to task drain. Existing holds cannot be replaced by another API
request.
- Gate HTTP ingress before parsers, auth, webhooks, and MCP. Gate new
WebSocket upgrades. Track async handlers in nested Express routers and
error middleware until they settle, even after the response or client
disconnect.
- Count accepted live-event WebSocket authentication through settlement,
even after disconnects. Count detached built-in agent, managed-home and
runtime-service startup reconciliation after readiness.
- Keep health and control mutations tracked. Count control-request
authentication separately from the read-only report, including
concurrent user/company/membership writes.
- Pause new scheduler admissions during idle holds. Count work already
in flight, including database backups, and reject scans whose work
generation changes. Periodic backups block sleep without a host wake
schedule.
- Inspect accounting and orphan-cleanup spool directories without
skipping temporary or malformed entries. Retain orphan tokens through
queue splices, flush failures, and buffer overflow. Keep failed usage
capture counted until its database failure fence is written.
- Check persisted work across companies in a bounded read-only
transaction. Include deferred agent-file cleanup and saved watchdogs
whose watched issues are complete. Enabled plugins and unsupported
retained work remain blockers.
- Require the exact idle owner and completed startup tracking before
returning an empty report. Keep reports free of tenant details.
- Document the hosting protocol, retry behavior, conservative blockers,
and the remaining external provider-stop race.

## Verification

- Current head: `8d9a599b732c2047b8c671d4799065d7c90a3567`, rebased on
master `941a3fa991aeb97eb1ac390c65b7973b5f6de1ad`. Both heartbeat helper
extractions are preserved. GitHub confirms no merge conflicts.
- Focused heartbeat renderer/run-log, drain, control-auth, admission,
route and PostgreSQL inventory checks: 234 tests passed in 10 suites
after the rebase.
- The POST task-drain contract includes the expected `409` conflict
response. The 84 OpenAPI and instance-settings route tests and server
typecheck passed after that final documentation fix.
- Final accepted-upgrade/startup regression run: 104 tests passed in
five suites, including success and failure after readiness or
disconnect. Final server typecheck also passed.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- `pnpm install --frozen-lockfile` and the PR diff secret scan passed.
No dependency or lockfile changes are added by this PR.
- The prior verification covered HTTP disconnects and early responses,
async error handlers, concurrent authentication writes, signed
bootstrap, saved watchdogs, backup promises, orphan cleanup, spools and
failed accounting fences. Those tests passed.
- The last full local `pnpm test:run` stopped in the general-server
phase with 16,658 passing tests and five skill/connector fixture
failures caused by an ancestor workspace skill directory. That full
local run preceded this rebase and has not been repeated for the import
conflict. Full current-head CI passed: 53 successful checks and two
conditional skips, with no failures.
- All eight review findings are fixed and their threads resolved,
including accepted upgrade authentication, detached startup writes and
the POST conflict contract. Current-head Apex review is 5/5 with no new
findings; all eight review threads remain resolved.
- No live provider stop or production change was performed.

## Risks

- The hosting controller must use the owned protocol and hold external
admission through its final validation and provider stop. A legacy drain
cannot authorize idle sleep. During an idle hold, new requests receive
503 with `Retry-After: 1`; the host must handle queueing or retry before
enabling this path.
- This is a single-process protocol. An unexpected restart after the
last validation can race an external provider stop. The host must
serialize deploy/wake/sleep operations and bind the validation to the
instance it stops. Multiple replicas need shared fencing.
- Some retained state conservatively prevents sleep, including every
enabled plugin. This PR does not promise that every inactive instance
becomes eligible.
- The HTTP adapter uses Express 5 router layers. Real Express tests
cover nested routes, errors and disconnects. New routes must register
before tracking is installed. Detached work must have durable state or
explicit work tracking.
- Unrecoverable in-memory cleanup debt keeps the instance awake until
reconciliation. This change does not make such debt survive an unplanned
process crash. No schema migration or provider configuration change is
included.

## Model Used

OpenAI GPT-6 via Codex, with reasoning, repository inspection, tool use
and code execution. The exact deployment variant/model ID and context
window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the full local
fixture limitation and clean full CI result are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 08:35:45 -07:00

380 lines
17 KiB
TypeScript

import { afterEach, beforeEach, describe, expect, it } from "vitest";
import express, { type Request } from "express";
import request from "supertest";
import { and, eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { authUsers, companies, companyMemberships, instanceSettings, instanceUserRoles } from "@paperclipai/db";
import { actorMiddleware, cloudActorHeaderSourceFromHeaders, resolveCloudTenantActor } from "./auth.js";
import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./idle-admission.js";
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain } from "../services/task-admission.js";
import { readIdleSleepSafety } from "../services/idle-sleep-safety.js";
// Minimal fake Drizzle Db: records every table passed to .insert() / .delete() and
// supports the chained call shapes used by resolveCloudTenantActor (values /
// onConflictDo* / returning().then() / delete().where()), plus the
// select().from(table).where() reads: instanceSettings for the owner-elevation
// flag resolution through instanceSettingsService, and companyMemberships for
// the user's own membership rows (rows configurable via membershipQueryRows,
// where-conditions captured in selectWheres). The chain is awaitable so
// directly-awaited statements resolve.
function createFakeDb(options: {
membershipRow?: { companyId: string; membershipRole: string; status: string };
membershipQueryRows?: Array<{ companyId: string; membershipRole: string | null; status: string }>;
settingsRow?: Record<string, unknown> | null;
selectThrows?: boolean;
beforeWrite?: () => Promise<void>;
} = {}) {
const membershipRow =
options.membershipRow ?? { companyId: "company-x", membershipRole: "owner", status: "active" };
const settingsRow =
options.settingsRow === undefined
? {
id: "00000000-0000-0000-0000-000000000001",
singletonKey: "default",
defaultEnvironmentId: null,
general: {},
experimental: {},
createdAt: new Date(),
updatedAt: new Date(),
}
: options.settingsRow;
const insertedTables: unknown[] = [];
const deletedTables: unknown[] = [];
const selectWheres: Array<{ table: unknown; condition: unknown }> = [];
const chain: Record<string, unknown> = {};
chain.values = () => chain;
chain.onConflictDoUpdate = () => chain;
chain.onConflictDoNothing = () => chain;
chain.where = () => chain;
chain.returning = async () => { await options.beforeWrite?.(); return [membershipRow]; };
chain.then = (resolve: (v: unknown) => unknown, reject?: (error: unknown) => unknown) =>
Promise.resolve().then(options.beforeWrite).then(resolve, reject);
const db = {
insert: (table: unknown) => {
insertedTables.push(table);
return chain;
},
delete: (table: unknown) => {
deletedTables.push(table);
return chain;
},
select: () => {
if (options.selectThrows) throw new Error("select unavailable");
return {
from: (table: unknown) => ({
where: (condition: unknown) => {
selectWheres.push({ table, condition });
const rows =
table === instanceSettings && settingsRow
? [settingsRow]
: table === companyMemberships
? (options.membershipQueryRows ?? [])
: [];
return {
then: (resolve: (v: unknown) => unknown) => Promise.resolve(rows).then(resolve),
};
},
}),
};
},
} as unknown as Db;
return { db, insertedTables, deletedTables, selectWheres };
}
function settingsRowWith(experimental: Record<string, unknown>) {
return {
id: "00000000-0000-0000-0000-000000000001",
singletonKey: "default",
defaultEnvironmentId: null,
general: {},
experimental,
createdAt: new Date(),
updatedAt: new Date(),
};
}
function fakeReq(headers: Record<string, string>): Request {
const lower: Record<string, string> = {};
for (const [k, v] of Object.entries(headers)) lower[k.toLowerCase()] = v;
return { header: (name: string) => lower[name.toLowerCase()] } as unknown as Request;
}
const VALID_HEADERS = {
"x-paperclip-cloud-tenant-token": "test-server-token",
"x-paperclip-cloud-user-id": "user-123",
"x-paperclip-cloud-user-email": "Owner@Example.com",
"x-paperclip-cloud-stack-id": "stack-abc",
"x-paperclip-cloud-stack-role": "owner",
};
const MANAGED_CONFIG_FLAG_ON = JSON.stringify({
v: 1,
mode: "cloud",
catalogVersion: "test",
features: { enableOwnerInstanceAdmin: true },
plugins: { autoInstall: [] },
});
const MANAGED_CONFIG_FLAG_OFF = JSON.stringify({
v: 1,
mode: "cloud",
catalogVersion: "test",
features: { enableOwnerInstanceAdmin: false },
plugins: { autoInstall: [] },
});
describe("resolveCloudTenantActor (shared-pool hardening)", () => {
beforeEach(() => {
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "test-server-token";
});
afterEach(() => {
stopTaskDrain();
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
delete process.env.PAPERCLIP_MANAGED_CONFIG;
});
it("does not grant instance admin by default (flag off)", async () => {
const { db, insertedTables } = createFakeDb();
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor).not.toBeNull();
expect(actor!.isInstanceAdmin).toBe(false);
expect(insertedTables).not.toContain(instanceUserRoles);
});
it("stays scoped to exactly the stack's company when the user has no other membership rows", async () => {
const { db } = createFakeDb();
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.companyIds).toHaveLength(1);
expect(actor!.memberships).toHaveLength(1);
expect(actor?.memberships?.[0]?.companyId).toBe(actor?.companyIds?.[0]);
expect(actor?.memberships?.[0]?.membershipRole).toBe("owner");
expect(actor!.source).toBe("cloud_tenant");
});
it("purges stale instance_admin rows left by pre-hardening deployments", async () => {
const { db, deletedTables } = createFakeDb();
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor).not.toBeNull();
expect(deletedTables).toContain(instanceUserRoles);
});
it("still upserts the user, company, and membership", async () => {
const { db, insertedTables } = createFakeDb();
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(insertedTables).toContain(authUsers);
expect(insertedTables).toContain(companies);
expect(insertedTables).toContain(companyMemberships);
});
it.each(["/api/health", "/api/instance/task-drain"])("counts unfinished authentication on exempt %s requests", async (path) => {
let enterWrite!: () => void, finishWrite!: () => void;
const entered = new Promise<void>(resolve => { enterWrite = resolve; });
const pending = new Promise<void>(resolve => { finishWrite = resolve; });
const { db, insertedTables } = createFakeDb({ beforeWrite: async () => { enterWrite(); await pending; } });
const server = express();
server.use(idleAdmissionMiddleware);
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
const hold = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
const emptyDb = { transaction: async (run: (tx: unknown) => Promise<unknown>) =>
run({ execute: async () => [{ blocked: false }] }) } as unknown as Db;
server.get("/api/health", (_req, res) => res.sendStatus(204));
server.get("/api/instance/task-drain", async (_req, res) => {
res.json(await readIdleSleepSafety(emptyDb, () => ({ ...hold, draining: true, activeRuns: 0, pendingWakes: 0 }),
Date.now, hold.ownerId, async () => "none"));
});
trackIdleRequestHandlers(server);
const accepted = request(server).get(path).set(VALID_HEADERS).then(response => response);
try {
await entered;
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
const during = await request(server).get("/api/instance/task-drain").expect(200);
expect(during.body.backgroundWork).toBe("unknown");
} finally {
finishWrite();
await accepted;
}
expect(insertedTables).toEqual(expect.arrayContaining([authUsers, companies, companyMemberships]));
expect(idleWorkSnapshot().active).toBe(0);
const after = await request(server).get("/api/instance/task-drain").expect(200);
expect(after.body.backgroundWork).toBe("none");
});
it("releases failed control authentication and preserves Express error handling", async () => {
const { db } = createFakeDb({ beforeWrite: async () => { throw new Error("fixture write failed"); } });
const server = express();
server.use(idleAdmissionMiddleware);
server.use(actorMiddleware(db, { deploymentMode: "authenticated", resolveSession: async () => null }));
server.get("/api/health", (_req, res) => res.sendStatus(204));
server.use((_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => res.sendStatus(500));
trackIdleRequestHandlers(server);
await request(server).get("/api/health").set(VALID_HEADERS).expect(500);
expect(idleWorkSnapshot().active).toBe(0);
await request(server).get("/api/health").expect(204);
});
it("resyncs an A to B to A context transition inside the debounce window", async () => {
const { db, insertedTables } = createFakeDb();
const contextA = VALID_HEADERS;
const contextB = { ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" };
await resolveCloudTenantActor(db, fakeReq(contextA));
await resolveCloudTenantActor(db, fakeReq(contextB));
await resolveCloudTenantActor(db, fakeReq(contextA));
expect(insertedTables.filter((table) => table === authUsers)).toHaveLength(3);
expect(insertedTables.filter((table) => table === companyMemberships)).toHaveLength(3);
});
it("returns null when the server token is unset", async () => {
delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN;
const { db } = createFakeDb();
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor).toBeNull();
});
it("resolves identically from a raw upgrade-request header map via the shim", async () => {
// Websocket upgrades hand us IncomingMessage.headers (lowercased keys,
// possibly string[] values), not an Express Request. The shim must feed
// resolveCloudTenantActor the same way Express header() does.
const { db } = createFakeDb();
const rawHeaders: Record<string, string | string[] | undefined> = {};
for (const [k, v] of Object.entries(VALID_HEADERS)) rawHeaders[k.toLowerCase()] = v;
rawHeaders["x-paperclip-cloud-user-name"] = ["Cloud Owner", "ignored-duplicate"];
const actor = await resolveCloudTenantActor(db, cloudActorHeaderSourceFromHeaders(rawHeaders));
expect(actor).not.toBeNull();
expect(actor!.userId).toBe("user-123");
expect(actor!.userName).toBe("Cloud Owner");
expect(actor!.companyIds).toHaveLength(1);
});
it("maps a non-owner stack role through to the membership without elevating", async () => {
const { db } = createFakeDb({
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
});
const actor = await resolveCloudTenantActor(
db,
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": "member" }),
);
expect(actor!.isInstanceAdmin).toBe(false);
expect(actor?.memberships?.[0]?.membershipRole).toBe("member");
});
describe("company membership union", () => {
// The primary company id is derived from the stack id; resolve it once
// through the same code path instead of duplicating the hash here.
async function resolvePrimaryCompanyId() {
const { db } = createFakeDb();
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
return actor!.companyIds![0]!;
}
it("unions the pinned primary with the user's own active membership rows, primary first and deduped", async () => {
const primaryCompanyId = await resolvePrimaryCompanyId();
const { db } = createFakeDb({
membershipQueryRows: [
// The user's own row for the primary company comes back from the
// query too — it must not appear twice.
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
{ companyId: "company-imported", membershipRole: "member", status: "active" },
],
});
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.companyIds).toEqual([primaryCompanyId, "company-imported"]);
expect(actor!.memberships).toEqual([
{ companyId: primaryCompanyId, membershipRole: "owner", status: "active" },
{ companyId: "company-imported", membershipRole: "member", status: "active" },
]);
});
it("loads memberships with the session path's exact own-user active-status filter", async () => {
const { db, selectWheres } = createFakeDb();
await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
const membershipSelects = selectWheres.filter((entry) => entry.table === companyMemberships);
expect(membershipSelects).toHaveLength(1);
// Rows for other users and rows in any non-active status are excluded
// by the query itself: the filter binds exactly this user id and the
// "active" status — the same condition the session path applies.
expect(membershipSelects[0]!.condition).toEqual(
and(
eq(companyMemberships.principalType, "user"),
eq(companyMemberships.principalId, "user-123"),
eq(companyMemberships.status, "active"),
),
);
});
it("degrades to the primary company when the membership read fails", async () => {
const { db } = createFakeDb({ selectThrows: true });
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor).not.toBeNull();
expect(actor!.companyIds).toHaveLength(1);
expect(actor!.memberships).toHaveLength(1);
});
});
describe("owner instance-admin elevation (enableOwnerInstanceAdmin)", () => {
it("elevates the owner while the flag is enabled, still without any role row", async () => {
const { db, insertedTables, deletedTables } = createFakeDb({
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
});
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.isInstanceAdmin).toBe(true);
expect(actor!.source).toBe("cloud_tenant");
// Elevation is computed, never persisted: no instance_user_roles insert,
// and the stale-row purge still runs on every authentication.
expect(insertedTables).not.toContain(instanceUserRoles);
expect(deletedTables).toContain(instanceUserRoles);
});
it("does not elevate the owner while the flag is disabled", async () => {
const { db } = createFakeDb({
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
});
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.isInstanceAdmin).toBe(false);
});
it.each(["member", "admin", "support"] as const)(
"never elevates the %s stack role even with the flag enabled",
async (stackRole) => {
const { db } = createFakeDb({
membershipRow: { companyId: "company-y", membershipRole: "member", status: "active" },
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
});
const actor = await resolveCloudTenantActor(
db,
fakeReq({ ...VALID_HEADERS, "x-paperclip-cloud-stack-role": stackRole }),
);
expect(actor).not.toBeNull();
expect(actor!.isInstanceAdmin).toBe(false);
},
);
it("resolves the flag through the managed overlay: overlay on elevates over a DB value of off", async () => {
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_ON;
const { db } = createFakeDb({
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: false }),
});
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.isInstanceAdmin).toBe(true);
});
it("resolves the flag through the managed overlay: overlay off wins over a DB value of on", async () => {
process.env.PAPERCLIP_MANAGED_CONFIG = MANAGED_CONFIG_FLAG_OFF;
const { db } = createFakeDb({
settingsRow: settingsRowWith({ enableOwnerInstanceAdmin: true }),
});
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor!.isInstanceAdmin).toBe(false);
});
it("fails closed when the settings read errors: actor resolves without elevation", async () => {
const { db, deletedTables } = createFakeDb({ selectThrows: true });
const actor = await resolveCloudTenantActor(db, fakeReq(VALID_HEADERS));
expect(actor).not.toBeNull();
expect(actor!.isInstanceAdmin).toBe(false);
expect(deletedTables).toContain(instanceUserRoles);
});
});
});