mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 01:27:09 +01:00
Own agent hiring and termination in a lifecycle module (#15631)
## Thinking Path > - Paperclip manages AI agents and their work. > - Agent creation, approval, pause, and termination change whether an agent can work. > - These changes currently use separate service and database paths. > - Some agents need setup after their record exists and before their harness can run. > - This PR adds one module that owns those state changes and their transactions. > - The module records progress and retries incomplete work after a server restart. ## Linked Issues or Issue Description **Problem or motivation** A hire can become active before its execution environment is ready. Direct agent state writes also bypass one common lifecycle boundary. Termination does not have a durable step for resource cleanup. **Proposed solution** Use agent lifecycle commands for hiring, approval, pause, resume, and termination. Keep proposed hires pending until approval. Prepare each approved hire, test its saved harness configuration, and then permit work. Wait for required plugin work before completing a step. **Alternatives considered** A general database commit hook would expose transaction details to callers. A timer alone would delay each new hire. This module owns each command transaction and starts its worker after commit. A periodic scan recovers work after a process stops. **Roadmap alignment** This change adds the lifecycle boundary for the existing agent and plugin systems. It does not add a user interface or a provider implementation. It replaces the lifecycle delivery approach in #15391. Related proposals include #11475 and #13099; this change does not add their external executor protocols. ## What Changed - Add durable approval, preparation, verification, pause, resume, stop, and cleanup states. - Keep agent creation, lifecycle state, configuration writes, and agent record deletion behind `modules/agent-lifecycle`. Keep validation, credential rules, reads, onboarding, invitations, general approvals, and company data deletion in their services. - Commit a hire before the remaining onboarding or invitation work. Serialize requests with the existing dedicated database connection helper. Reuse the hire after a failed attempt. - Keep accepted work in the run queue during setup. Require execution and cleanup evidence before completing pause or termination. - Retain the original credential owner when approving migrated hires. - Start lifecycle work after each command commits. Use database leases and revision checks for retries and late results. - Add a generic required plugin method and capability. Store the required plugin IDs for each agent. Track host and plugin completion separately. - Keep harness verification and credential rules in services. Supply their results to lifecycle through explicit ports. Reject failed logins and incomplete probes. - Remove reverse imports from lifecycle adapters into services. Share record queries and validation below both layers. - Use one hire decision path for agent IDs and approval IDs. Resolve the approval and agent change in one transaction. Expose policy reconciliation on the command object. - Save configuration and verification invalidation through one module operation. Ordinary commands own their transaction. Restrict the shared transaction integration to its listed callers. Reject identity, company, accounting, timestamp, and unknown fields. - Require completed termination before deleting agent records. The company deletion service calls the module through a separate deletion interface. - Save an approved secret before applying its agent binding. A failed binding leaves the secret approved. A retry reuses it. - Remove old permission and metadata export files. Move hire configuration rules out of lifecycle persistence. - Report failed worker steps and scans with operation context and known error codes. Omit provider messages, stacks, and configuration values. Document ordinary application use and extension rules. - Recover expired native owners only after execution stops. Retry cleanup of failed retained leases. Wait for setup in the live smoke script. - Keep manual, budget, and company pause reasons separate. Guard execution status writes with the ready state. - Add a migration, boundary checks, regression tests, and a lifecycle guide. Keep the current UI. - Limit periodic policy checks to 100 eligible agents per minute. Continue from the last agent ID. Exclude pending hires and agents in termination or a final state. Keep explicit company and budget updates immediate. ## Verification - Latest head `ccc3c86b0` aligns the shared status update schema with the PATCH route. All 15 shared tests, 104 server route/OpenAPI tests, 2 CLI tests, and shared/server/CLI TypeScript checks pass. The Greptile comment is fixed. CI is running; no new Apex review was requested for this small fix. - Rebase `75708ac94` rebases onto master at `2eb5187e4`. The backup test uses the shared 90-second timeout from master; the older timeout commit was removed. The affected backup test passes. - Agent budget edits now pass the target agent ID through enforcement. All 50 affected tests, server TypeScript, and 11 boundary tests pass. Both configuration update paths have regression coverage. CI is running on the latest head. No new Apex review was requested for these small fixes. - Rebase `9b568666a` preserves the run-control extraction and delivery-work startup changes from master. All 559 affected tests, server TypeScript, and 11 boundary tests pass. CI and Apex review are running on the final head. - The policy scan fix passes 56 lifecycle and diagnostic tests, 24 policy and caller tests, and 11 boundary tests. A regression test checks the page limit, interval, excluded states, and cursor wraparound. - Rebase `fd4698750` passes all 611 tests in eight affected lifecycle, route, and heartbeat suites. Server TypeScript and all 11 module boundary tests pass. The company deletion rollback and concurrent hire tests are included. - Follow-up `00aac7646` fixes the failed decision deletion test and the public update schema. All 126 affected server tests, 8 shared tests, 2 CLI tests, and 11 boundary tests pass. Shared, server, CLI, and UI TypeScript checks pass. - Server TypeScript passes: `pnpm --filter @paperclipai/server exec tsc --noEmit`. - All 11 boundary tests pass: `node --test scripts/check-module-boundaries.test.mjs`. The repository boundary scan also passes. - All 68 lifecycle, failure diagnostic, and harness verification tests pass. Tests reject forbidden fields through the root command, combined transition command, and restricted transaction integration. They also check invalid fields returned by configuration preparation. - Failure tests cover host work, plugin work, claims, renewals, deferrals, and scans. Log tests check known error codes and exclusion of private error contents. - All 355 tests in 10 existing caller suites pass. These tests cover secret approval, pairing, pending hires, credentials, company import, built-in agents, managed agents, instructions, and skills. - The branch is rebased onto `origin/master` at `2eb5187e4`. The queue guards and cancellation lock fix use the extracted heartbeat queue. Agent cancellation uses the extracted run-control file. The source diff scan found no secrets or private URLs. - Earlier checks cover approval retries, rollback, late verification results, policy holds, deletion rollback, concurrent company deletion, and queue admission during setup. - Before the rebase, CI passed on `00aac7646` and Greptile Apex rated it 5/5 with no open comments. CI and a new Apex review are pending for the rebased commit. The branch has no merge conflicts. - The full local build and native Dot suite were not run. The host has no Cargo. No live provider test was run. ## Risks - Background policy recovery takes more than one minute when more than 100 agents are eligible. Explicit company and budget updates still reconcile immediately. - The restricted configuration integration keeps an agent reference and its dependent records in one transaction. It cannot change a budget or invoke a lifecycle transition. A periodic scan starts pending setup after that transaction commits. - Secret approval and binding use separate transactions. A failed binding can leave an approved secret with no agent binding. - New approved hires remain unavailable until setup and verification complete. Existing clients see `paused` during these steps. This PR does not change the UI. - Migration `0320_daily_onslaught.sql` maps existing statuses and pause reasons. It does not create external resources or claim that an existing harness passed verification. - Use a coordinated server update for this migration. Older server versions can write legacy status without the new lifecycle checks. - A failed onboarding or invitation request can leave a committed hire. A retry reuses that hire and completes the remaining records. - Deletion requires completed termination. Company deletion can require another request while cleanup is pending. - Required plugins must make repeated calls safe and reject older revisions. A disabled or removed required plugin blocks completion until it is restored. - A new hire can run a second harness test after the existing UI test. Some adapters can charge for this test. - No live provider test was run. Provider implementation and resource backfill remain separate work. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code tools, and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in this PR with the feature request field labels - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] Targeted local tests pass - [x] All Paperclip CI gates are green on the new commit - [x] Greptile Apex is 5/5 with no open comments on the new commit - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky LeachandPaperclip authored and GitHub committed
1 parent
ab8dcab809
commit
2dd9811e8b
152 files changed
+6790
-2758
No files matched your search
@@ -0,0 +1,68 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { createLifecycleWorker } from "../application/worker.js";
|
||||
import type { LifecycleAgent, LifecycleStore, LifecycleDriver } from "../application/ports.js";
|
||||
import { startAgentLifecycle } from "../index.js";
|
||||
import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
|
||||
import { logger } from "../../../middleware/logger.js";
|
||||
|
||||
const agent: LifecycleAgent = { id: "agent", companyId: "company", lifecycleState: "verifying", lifecycleVersion: 3,
|
||||
lifecycleRequiredPluginIds: null, lifecycleError: null,
|
||||
lifecycleOperation: { id: "operation", hostComplete: false, completedPluginIds: [], attempts: 1 } };
|
||||
function fixture() {
|
||||
const store = { get: vi.fn(), claim: vi.fn().mockResolvedValue(agent), renew: vi.fn().mockResolvedValue(true),
|
||||
setRequiredPlugins: vi.fn().mockResolvedValue(true), completeHost: vi.fn().mockResolvedValue(true),
|
||||
completePlugin: vi.fn().mockResolvedValue(true), defer: vi.fn().mockResolvedValue(true),
|
||||
change: vi.fn().mockResolvedValue({ ...agent, lifecycleState: "ready" }), pending: vi.fn().mockResolvedValue([agent.id]) } satisfies LifecycleStore;
|
||||
const driver = { requiredPluginIds: vi.fn().mockResolvedValue(["plugin"]), runHost: vi.fn().mockResolvedValue("complete"),
|
||||
runPlugin: vi.fn().mockResolvedValue("complete") } satisfies LifecycleDriver;
|
||||
const report = vi.fn();
|
||||
return { store, driver, report, worker: createLifecycleWorker(store, driver, report) };
|
||||
}
|
||||
afterEach(() => { vi.restoreAllMocks(); vi.useRealTimers(); });
|
||||
|
||||
describe("lifecycle failure diagnostics", () => {
|
||||
it.each(["host", "plugin", "claim", "renew", "scan", "defer"] as const)("reports %s failures with the available context", async stage => {
|
||||
const { store, driver, report, worker } = fixture();
|
||||
const error = new Error("private provider detail");
|
||||
if (stage === "host") driver.runHost.mockRejectedValue(error);
|
||||
else if (stage === "plugin") driver.runPlugin.mockRejectedValue(error);
|
||||
else if (stage === "scan") store.pending.mockRejectedValue(error);
|
||||
else if (stage === "defer") { driver.runHost.mockResolvedValue("pending"); store.defer.mockRejectedValue(error); }
|
||||
else store[stage].mockRejectedValue(error);
|
||||
const work = stage === "scan" ? worker.sweep() : worker.process(agent.id);
|
||||
if (["claim", "scan", "defer"].includes(stage)) await expect(work).rejects.toThrow(error);
|
||||
else await work;
|
||||
expect(report).toHaveBeenCalledWith(expect.objectContaining({ stage }), error);
|
||||
if (!["claim", "scan"].includes(stage)) expect(report).toHaveBeenCalledWith(expect.objectContaining({
|
||||
agentId: agent.id, companyId: agent.companyId, phase: "verifying", operationId: "operation", version: 3,
|
||||
...(stage === "plugin" ? { pluginId: "plugin" } : {}),
|
||||
}), error);
|
||||
await worker.stop();
|
||||
});
|
||||
|
||||
it("reports periodic policy scan failures", async () => {
|
||||
vi.useFakeTimers();
|
||||
const error = new Error("private database detail");
|
||||
const db = { select: () => { throw error; } } as unknown as Db;
|
||||
const effects = { ...createAgentLifecycleEffects(), reportFailure: vi.fn() };
|
||||
const work = startAgentLifecycle(db, effects, fixture().driver, () => true);
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(5_000);
|
||||
expect(effects.reportFailure).toHaveBeenCalledWith({ stage: "policy_scan" }, error);
|
||||
} finally { await work.stop(); }
|
||||
});
|
||||
|
||||
it.each([
|
||||
[{ code: "required_plugin_unavailable" }, "required_plugin_unavailable"],
|
||||
[{ code: "harness_test_failed" }, "harness_test_failed"],
|
||||
[{ cause: { code: "40P01", message: "private query" } }, "40P01"],
|
||||
[{ code: "private credential value" }, "unclassified"],
|
||||
])("logs known codes without provider error contents", (details, failureCode) => {
|
||||
const log = vi.spyOn(logger, "warn").mockImplementation(() => {});
|
||||
const context = { stage: "host" as const, agentId: "agent", phase: "verifying" as const, operationId: "operation", version: 3 };
|
||||
createAgentLifecycleEffects().reportFailure(context, Object.assign(new Error("private provider detail"), details));
|
||||
expect(log).toHaveBeenCalledExactlyOnceWith({ ...context, failureCode }, "Agent lifecycle operation failed");
|
||||
expect(JSON.stringify(log.mock.calls)).not.toContain("private");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,673 @@
|
||||
import { createAgentLifecycle as createLifecycleCommands } from "../index.js";
|
||||
import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
|
||||
import { deleteCompany } from "../../../services/company-deletion.js";
|
||||
import { agentLifecycleCompanyDeletion } from "../company-deletion.js";
|
||||
import { requireServerAdapter } from "../../../adapters/index.js";
|
||||
import { agentExecutionsHaveStopped } from "../../../services/agent-execution-stop.js";
|
||||
import { remoteTerminationReceipt } from "../../../services/remote-execution-termination.js";
|
||||
import { approvalService } from "../../../services/approvals.js";
|
||||
import { updateAgentConfigurationInTransaction } from "../../../services/agent-configuration-transaction.js";
|
||||
import { agentService as agentConfiguration } from "../../../services/agents.js";
|
||||
import { readFileSync } from "node:fs";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { agentRoutes } from "../../../routes/agents.js";
|
||||
import { errorHandler } from "../../../middleware/error-handler.js";
|
||||
import { createLifecycleDriver } from "../../../services/agent-lifecycle-driver.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { agents, approvals, projects, issues, heartbeatRuns, nativeRunFinalizations, budgetPolicies, environmentLeases, agentApiKeys, activityLog, agentConfigRevisions, userCompanyPreferences, companies, companyMemberships, principalPermissionGrants, plugins, pluginCompanySettings, createDb, type Db } from "@paperclipai/db";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
|
||||
import { createAgentLifecycle, configureAgentLifecycle } from "../../../services/agent-lifecycle.js";
|
||||
import { createLifecycleStore as createStore } from "../adapters/postgres.js";
|
||||
import { transition } from "../domain/policy.js";
|
||||
import type { LifecycleAgent } from "../application/ports.js";
|
||||
import { budgetService } from "../../../services/budgets.js";
|
||||
|
||||
const createLifecycleStore = (db: Db) => createStore(db, createAgentLifecycleEffects());
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("agent lifecycle commands", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
let companyId: string;
|
||||
const workers: ReturnType<typeof configureAgentLifecycle>[] = [];
|
||||
beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("paperclip-agent-lifecycle-"); db = createDb(database.connectionString); }, 90_000);
|
||||
afterAll(async () => { for (const worker of workers) await worker.stop(); await database?.cleanup(); });
|
||||
beforeEach(async () => {
|
||||
for (const work of workers) await work.stop();
|
||||
companyId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Lifecycle test", issuePrefix: `L${companyId.slice(0, 7)}` });
|
||||
});
|
||||
function worker(runHost: (agent: LifecycleAgent) => Promise<"complete" | "pending"> = async () => "complete") {
|
||||
const value = configureAgentLifecycle(db, { requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost });
|
||||
workers.push(value); return value;
|
||||
}
|
||||
async function hire(status = "idle") { return createAgentLifecycle(db).requestHire(companyId, { name: "Agent", status, adapterType: "process" }); }
|
||||
async function current(id: string) { return (await db.select().from(agents).where(eq(agents.id, id)))[0]!; }
|
||||
|
||||
it("maps existing states without starting setup or clearing pause reasons", async () => {
|
||||
const rows = await db.insert(agents).values(["idle", "running", "error", "paused", "pending_approval", "terminated"].map(status => ({
|
||||
companyId, name: status, status, pauseReason: status === "paused" ? "budget" : null,
|
||||
}))).returning();
|
||||
const migration = readFileSync(new URL("../../../../../packages/db/src/migrations/0320_daily_onslaught.sql", import.meta.url), "utf8");
|
||||
await db.execute(sql.raw(migration.slice(migration.indexOf('UPDATE "agents"'))));
|
||||
for (const row of rows) {
|
||||
const mapped = await current(row.id);
|
||||
expect(mapped.lifecycleState).toBe(["paused", "pending_approval", "terminated"].includes(row.status) ? row.status : "ready");
|
||||
expect(mapped.lifecycleOperation).toBeNull();
|
||||
expect(mapped.lifecycleRequiredPluginIds).toBeNull();
|
||||
expect(mapped.lifecycleHolds).toEqual(row.status === "paused" ? ["budget"] : []);
|
||||
}
|
||||
});
|
||||
|
||||
it("commits the record before preparation and verifies before admission", async () => {
|
||||
const observed: string[] = [];
|
||||
const work = worker(async agent => {
|
||||
expect((await current(agent.id)).lifecycleState).toBe(agent.lifecycleState);
|
||||
observed.push(agent.lifecycleState); return "complete";
|
||||
});
|
||||
const agent = await hire();
|
||||
expect(agent).toMatchObject({ lifecycleState: "preparing", status: "paused" });
|
||||
await work.process(agent.id);
|
||||
expect(observed).toEqual(["preparing", "verifying"]);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle" });
|
||||
expect(agent).not.toHaveProperty("lifecycleOperation");
|
||||
});
|
||||
|
||||
it("rolls back a hire and its grants when hire initialization fails", async () => {
|
||||
const effects = createAgentLifecycleEffects();
|
||||
const lifecycle = createLifecycleCommands(db, { ...effects,
|
||||
initializeHire: async (tx, agent, options) => {
|
||||
await effects.initializeHire(tx, agent, options);
|
||||
throw new Error("Hire initialization failed");
|
||||
},
|
||||
});
|
||||
await expect(lifecycle.requestHire(companyId, { name: "Rollback hire", adapterType: "process" }))
|
||||
.rejects.toThrow("Hire initialization failed");
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, companyId))).toHaveLength(0);
|
||||
expect(await db.select().from(principalPermissionGrants).where(eq(principalPermissionGrants.companyId, companyId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("runs the saved harness test and keeps a failed configuration out of ready", async () => {
|
||||
const driver = createLifecycleDriver(db, {} as never);
|
||||
const work = worker(driver.runHost);
|
||||
const agent = await hire();
|
||||
await work.process(agent.id);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "verifying", status: "paused" });
|
||||
await agentConfiguration(db).update(agent.id, { adapterConfig: { command: process.execPath } });
|
||||
await work.process(agent.id);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", status: "idle", lifecycleError: null });
|
||||
});
|
||||
|
||||
it.each(["claude_local", "gemini_local"])("keeps %s authentication warnings out of ready", async adapterType => {
|
||||
const adapter = requireServerAdapter(adapterType);
|
||||
const probe = vi.spyOn(adapter, "testEnvironment").mockResolvedValue({ adapterType, status: "warn",
|
||||
testedAt: new Date().toISOString(), checks: [{ code: `${adapterType.split("_")[0]}_hello_probe_auth_required`, level: "warn", message: "Login required" }] });
|
||||
try {
|
||||
const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "Login test", adapterType });
|
||||
const work = worker(createLifecycleDriver(db, {} as never).runHost);
|
||||
await work.process(agent.id);
|
||||
expect(probe).toHaveBeenCalled();
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "verifying", lifecycleError: "The lifecycle step failed. Retry the operation." });
|
||||
} finally { probe.mockRestore(); }
|
||||
});
|
||||
|
||||
it("recovers an expired native owner only after the cancelled run and controller stop", async () => {
|
||||
const agent = await hire();
|
||||
const [issue] = await db.insert(issues).values({ companyId, title: "Cancelled native run" }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, nativeIssueId: issue.id,
|
||||
status: "cancelled", invocationSource: "on_demand", runtimeMode: "native",
|
||||
startedAt: new Date(), finishedAt: new Date(), processPid: process.pid }).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ companyId, issueId: issue.id, runId: run.id,
|
||||
phase: "executing", leaseOwner: "stopped-server", leaseExpiresAt: new Date(0), controllerPid: process.pid });
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
|
||||
await db.update(nativeRunFinalizations).set({ controllerPid: null }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ processPid: 2_000_000_000 }).where(eq(heartbeatRuns.id, run.id));
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(true);
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)))[0]).toMatchObject({ leaseOwner: null, leaseExpiresAt: null });
|
||||
});
|
||||
|
||||
it("does not advance termination while a cancelled process still runs", async () => {
|
||||
const agent = await hire();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id,
|
||||
status: "cancelled", invocationSource: "on_demand", runtimeMode: "legacy",
|
||||
startedAt: new Date(), finishedAt: new Date(), processPid: process.pid }).returning();
|
||||
const driver = createLifecycleDriver(db, {} as never);
|
||||
const work = worker(driver.runHost);
|
||||
await createAgentLifecycle(db).terminateAgent(agent.id);
|
||||
await work.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("terminating");
|
||||
await db.update(heartbeatRuns).set({ processPid: null,
|
||||
resultJson: { executionCancellation: { state: "acknowledged" } } }).where(eq(heartbeatRuns.id, run.id));
|
||||
await createAgentLifecycle(db).retry(agent.id);
|
||||
await work.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("terminated");
|
||||
});
|
||||
|
||||
it("does not cancel a reporting agent when its manager pauses", async () => {
|
||||
const parent = await hire();
|
||||
const [child] = await db.insert(agents).values({ companyId, name: "Report", reportsTo: parent.id,
|
||||
status: "idle", lifecycleState: "ready" }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: child.id,
|
||||
status: "queued", invocationSource: "on_demand" }).returning();
|
||||
const work = worker(createLifecycleDriver(db, {} as never).runHost);
|
||||
await createAgentLifecycle(db).pauseAgent(parent.id);
|
||||
await work.process(parent.id);
|
||||
expect((await current(parent.id)).lifecycleState).toBe("paused");
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0]!.status).toBe("queued");
|
||||
});
|
||||
|
||||
it("retains company data until every agent finishes termination", async () => {
|
||||
const agent = await hire();
|
||||
const [project] = await db.insert(projects).values({ companyId, name: "Keep until cleanup", leadAgentId: agent.id }).returning();
|
||||
await db.insert(issues).values({ companyId, projectId: project.id, title: "Keep this task", assigneeAgentId: agent.id });
|
||||
const pending = worker(async () => "pending");
|
||||
await expect(deleteCompany(db, companyId)).rejects.toThrow("Complete termination");
|
||||
expect(await db.select().from(projects).where(eq(projects.id, project.id))).toHaveLength(1);
|
||||
expect(await db.select().from(issues).where(eq(issues.companyId, companyId))).toHaveLength(1);
|
||||
await pending.stop();
|
||||
const complete = worker();
|
||||
await createAgentLifecycle(db).retry(agent.id);
|
||||
await complete.process(agent.id);
|
||||
await deleteCompany(db, companyId);
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, companyId))).toHaveLength(0);
|
||||
expect(await db.select().from(companies).where(eq(companies.id, companyId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("requires a transaction for the company deletion integration", async () => {
|
||||
await expect(agentLifecycleCompanyDeletion.deleteCompanyData(db as never, companyId))
|
||||
.rejects.toThrow("Company deletion requires a database transaction");
|
||||
});
|
||||
|
||||
it("rejects individual deletion before cleanup and rolls back failed cleanup", async () => {
|
||||
const agent = await hire();
|
||||
const effects = createAgentLifecycleEffects();
|
||||
const cleanup = vi.fn(effects.deleteDependencies);
|
||||
const lifecycle = createLifecycleCommands(db, { ...effects, deleteDependencies: cleanup });
|
||||
await expect(lifecycle.purgeAgent(agent.id)).rejects.toThrow("Complete termination");
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
|
||||
await db.update(agents).set({ lifecycleState: "terminated", status: "terminated" }).where(eq(agents.id, agent.id));
|
||||
const [key] = await db.insert(agentApiKeys).values({ companyId, agentId: agent.id, name: "Keep on rollback", keyHash: randomUUID() }).returning();
|
||||
cleanup.mockImplementation(async (tx, companyId, id) => {
|
||||
await effects.deleteDependencies(tx, companyId, id);
|
||||
throw new Error("Dependency cleanup failed");
|
||||
});
|
||||
await expect(lifecycle.purgeAgent(agent.id)).rejects.toThrow("Dependency cleanup failed");
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "terminated" });
|
||||
expect(await db.select().from(agentApiKeys).where(eq(agentApiKeys.id, key.id))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("deletes terminated and rejected agents only in the requested company", async () => {
|
||||
await db.insert(agents).values((["terminated", "rejected"] as const).map(lifecycleState => ({
|
||||
companyId, name: lifecycleState, status: "terminated", lifecycleState,
|
||||
})));
|
||||
const [otherCompany] = await db.insert(companies).values({ name: "Keep this company", issuePrefix: `K${companyId.slice(0, 7)}` }).returning();
|
||||
const [otherAgent] = await db.insert(agents).values({ companyId: otherCompany.id, name: "Keep this agent",
|
||||
status: "terminated", lifecycleState: "terminated" }).returning();
|
||||
|
||||
await deleteCompany(db, companyId);
|
||||
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, companyId))).toHaveLength(0);
|
||||
expect(await current(otherAgent.id)).toMatchObject({ companyId: otherCompany.id });
|
||||
expect(await db.select().from(companies).where(eq(companies.id, otherCompany.id))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("rolls back agent and company data deletion when a later operation fails", async () => {
|
||||
const agent = await hire("terminated");
|
||||
const [project] = await db.insert(projects).values({ companyId, name: "Keep on rollback" }).returning();
|
||||
const removeAgents = agentLifecycleCompanyDeletion.deleteCompanyData;
|
||||
const failure = vi.spyOn(agentLifecycleCompanyDeletion, "deleteCompanyData").mockImplementation(async (tx, id) => {
|
||||
await removeAgents(tx, id);
|
||||
expect(await tx.select().from(agents).where(eq(agents.companyId, id))).toHaveLength(0);
|
||||
throw new Error("Later deletion failed");
|
||||
});
|
||||
try {
|
||||
await expect(deleteCompany(db, companyId)).rejects.toThrow("Later deletion failed");
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "terminated" });
|
||||
expect(await db.select().from(projects).where(eq(projects.id, project.id))).toHaveLength(1);
|
||||
expect(await db.select().from(companies).where(eq(companies.id, companyId))).toHaveLength(1);
|
||||
} finally { failure.mockRestore(); }
|
||||
});
|
||||
|
||||
it("blocks a concurrent hire until company deletion commits", async () => {
|
||||
let reportLock!: (pid: number) => void;
|
||||
let rejectLock!: (error: unknown) => void;
|
||||
const locked = new Promise<number>((resolve, reject) => { reportLock = resolve; rejectLock = reject; });
|
||||
let release!: () => void;
|
||||
const released = new Promise<void>(resolve => { release = resolve; });
|
||||
const removeAgents = agentLifecycleCompanyDeletion.deleteCompanyData;
|
||||
const pause = vi.spyOn(agentLifecycleCompanyDeletion, "deleteCompanyData").mockImplementation(async (tx, id) => {
|
||||
const [backend] = await tx.execute<{ pid: number }>(sql`select pg_backend_pid() as pid`);
|
||||
reportLock(backend!.pid);
|
||||
await released;
|
||||
await removeAgents(tx, id);
|
||||
});
|
||||
const deletion = deleteCompany(db, companyId);
|
||||
void deletion.catch(rejectLock);
|
||||
let hiring: Promise<unknown> | undefined;
|
||||
try {
|
||||
const pid = await locked;
|
||||
hiring = hire().then(() => "created", error => error.message);
|
||||
await vi.waitFor(async () => {
|
||||
const blocked = await db.execute(sql`select 1 from pg_stat_activity
|
||||
where datname = current_database() and ${pid} = any(pg_blocking_pids(pid))`);
|
||||
expect(blocked).toHaveLength(1);
|
||||
});
|
||||
release();
|
||||
await deletion;
|
||||
expect(await hiring).toBe("Company not found");
|
||||
expect(await db.select().from(agents).where(eq(agents.companyId, companyId))).toHaveLength(0);
|
||||
} finally {
|
||||
release();
|
||||
await Promise.allSettled([deletion, hiring]);
|
||||
pause.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("requires cleanup proof after a remote run is already cancelled", async () => {
|
||||
const agent = await hire();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id,
|
||||
status: "cancelled", invocationSource: "on_demand", runtimeMode: "native",
|
||||
startedAt: new Date(), finishedAt: new Date() }).returning();
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId, heartbeatRunId: run.id,
|
||||
provider: "test", providerLeaseId: "test-lease", status: "pending_cleanup",
|
||||
cleanupStatus: "failed", releasedAt: new Date() }).returning();
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
|
||||
await db.update(environmentLeases).set({ status: "released", cleanupStatus: "success" })
|
||||
.where(eq(environmentLeases.id, lease.id));
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(false);
|
||||
await db.update(environmentLeases).set({ metadata: {
|
||||
remoteExecutionTermination: remoteTerminationReceipt(lease, { providerLeaseId: lease.providerLeaseId, state: "destroyed" }),
|
||||
} }).where(eq(environmentLeases.id, lease.id));
|
||||
expect(await agentExecutionsHaveStopped(db, [agent.id])).toBe(true);
|
||||
});
|
||||
|
||||
it("does not prepare a proposed hire until approval", async () => {
|
||||
const observed: string[] = [];
|
||||
const work = worker(async agent => { observed.push(agent.lifecycleState); return "complete"; });
|
||||
const agent = await hire("pending_approval");
|
||||
await work.process(agent.id);
|
||||
expect(observed).toEqual([]);
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent", payload: { agentId: agent.id }, status: "pending" });
|
||||
await approvalService(db).approve(approval.id, "board");
|
||||
await vi.waitFor(async () => expect((await current(agent.id)).lifecycleState).toBe("ready"));
|
||||
expect(observed).toEqual(["preparing", "verifying"]);
|
||||
});
|
||||
|
||||
it.each(["approved", "rejected"] as const)("uses the same %s hire decision for an agent and its approval", async decision => {
|
||||
const agent = await hire("pending_approval");
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent",
|
||||
requestedByUserId: "requester", payload: { agentId: agent.id }, status: "pending" });
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
const decide = decision === "approved" ? lifecycle.approveHire : lifecycle.rejectHire;
|
||||
const first = await decide(agent.id, "board");
|
||||
expect(first).toMatchObject({ applied: true, approval: { id: approval.id, status: decision },
|
||||
agent: { lifecycleState: decision === "approved" ? "preparing" : "rejected" } });
|
||||
const version = (await current(agent.id)).lifecycleVersion;
|
||||
expect(await decide({ approvalId: approval.id }, "board")).toMatchObject({ applied: false, hireApprovedAgentId: null });
|
||||
expect((await current(agent.id)).lifecycleVersion).toBe(version);
|
||||
});
|
||||
|
||||
it("rolls back the approval and agent together when activation fails", async () => {
|
||||
const agent = await hire("pending_approval");
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent",
|
||||
payload: { agentId: agent.id }, status: "pending" });
|
||||
const lifecycle = createLifecycleCommands(db, { ...createAgentLifecycleEffects(),
|
||||
recordCreation: async () => { throw new Error("Creation event failed"); } });
|
||||
await expect(lifecycle.approveHire(agent.id, "board")).rejects.toThrow("Creation event failed");
|
||||
expect((await current(agent.id)).lifecycleState).toBe("pending_approval");
|
||||
expect((await db.select().from(approvals).where(eq(approvals.id, approval.id)))[0].status).toBe("pending");
|
||||
});
|
||||
|
||||
it("keeps the hire credential owner and recovers it for migrated approvals", async () => {
|
||||
for (const savedOwner of [null, "original-owner"]) {
|
||||
const [agent] = await db.insert(agents).values({ companyId, name: "Migrated hire", status: "pending_approval",
|
||||
lifecycleState: "pending_approval", lifecycleOperation: savedOwner ? {
|
||||
id: randomUUID(), hostComplete: false, completedPluginIds: [], attempts: 0, responsibleUserId: savedOwner,
|
||||
} : null }).returning();
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent",
|
||||
requestedByUserId: "requesting-member", payload: { agentId: agent.id }, status: "pending" });
|
||||
await approvalService(db).approve(approval.id, "approving-member");
|
||||
expect((await current(agent.id)).lifecycleOperation?.responsibleUserId).toBe(savedOwner ?? "requesting-member");
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects transaction injection and generic state writes", async () => {
|
||||
const agent = await hire("pending_approval");
|
||||
await expect(db.transaction(async tx => createAgentLifecycle(tx as unknown as Db))).rejects.toThrow("root database");
|
||||
await expect(agentConfiguration(db).update(agent.id, { status: "idle" } as never)).rejects.toThrow("lifecycle command");
|
||||
await expect(createAgentLifecycle(db).requestHire(companyId, { name: "Bypass", lifecycleRequiredPluginIds: [] } as never)).rejects.toThrow("Lifecycle fields");
|
||||
});
|
||||
|
||||
it("clears a rejected hire from the user's primary selection", async () => {
|
||||
const agent = await createAgentLifecycle(db).requestHire(companyId, { name: "Proposed", status: "pending_approval" }, { createdByUserId: "board" });
|
||||
const preference = () => db.select().from(userCompanyPreferences).where(eq(userCompanyPreferences.companyId, companyId));
|
||||
expect((await preference())[0]!.primaryAgentId).toBe(agent.id);
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent", payload: { agentId: agent.id }, status: "pending" });
|
||||
await approvalService(db).reject(approval.id, "board");
|
||||
expect((await preference())[0]!.primaryAgentId).toBeNull();
|
||||
expect((await current(agent.id)).lifecycleState).toBe("rejected");
|
||||
});
|
||||
|
||||
it("rolls back configuration and its revision when the requested transition fails", async () => {
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
const agent = await hire("terminated");
|
||||
await expect(lifecycle.updateAndTransition(agent.id, "resume", { name: "Must not persist" },
|
||||
{ recordRevision: { createdByUserId: "board", source: "patch" } })).rejects.toThrow("Cannot resume");
|
||||
expect((await current(agent.id)).name).toBe("Agent");
|
||||
expect(await db.select().from(agentConfigRevisions).where(eq(agentConfigRevisions.agentId, agent.id))).toEqual([]);
|
||||
const active = await hire();
|
||||
expect(await lifecycle.updateAndTransition(active.id, "pause", { name: "Paused agent" })).toMatchObject({ name: "Paused agent", lifecycleState: "pausing" });
|
||||
});
|
||||
|
||||
it("preserves a disabled positive budget at approval and applies explicit configuration changes", async () => {
|
||||
const agent = await hire("pending_approval");
|
||||
const budgets = budgetService(db);
|
||||
await budgets.upsertPolicy(companyId, { scopeType: "agent", scopeId: agent.id, amount: 1000, isActive: false }, null);
|
||||
const approval = await approvalService(db).create(companyId, { type: "hire_agent",
|
||||
payload: { agentId: agent.id, budgetMonthlyCents: 1000 }, status: "pending" });
|
||||
const policy = async () => (await db.select().from(budgetPolicies).where(eq(budgetPolicies.scopeId, agent.id)))[0]!;
|
||||
await approvalService(db).approve(approval.id, "board");
|
||||
expect(await policy()).toMatchObject({ amount: 1000, isActive: false });
|
||||
await agentConfiguration(db).update(agent.id, { budgetMonthlyCents: 2000 });
|
||||
expect(await policy()).toMatchObject({ amount: 2000, isActive: true });
|
||||
await agentConfiguration(db).update(agent.id, { budgetMonthlyCents: 0 });
|
||||
expect(await policy()).toMatchObject({ amount: 0, isActive: false });
|
||||
});
|
||||
|
||||
it.each(["updateConfiguration", "updateAndTransition"] as const)("limits %s budget reconciliation to the edited agent", async command => {
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
const one = await hire();
|
||||
const two = await lifecycle.requestHire(companyId, { name: "Other budget target" });
|
||||
await db.update(companies).set({ status: "paused" }).where(eq(companies.id, companyId));
|
||||
if (command === "updateConfiguration") await lifecycle.updateConfiguration(one.id, { budgetMonthlyCents: 1000 });
|
||||
else await lifecycle.updateAndTransition(one.id, "pause", { budgetMonthlyCents: 1000 });
|
||||
expect((await current(one.id)).lifecycleHolds).toContain("company_paused");
|
||||
expect((await current(two.id)).lifecycleHolds).toEqual([]);
|
||||
});
|
||||
|
||||
it("limits invocation policy checks to the requested agent", async () => {
|
||||
const one = await hire();
|
||||
const two = await createAgentLifecycle(db).requestHire(companyId, { name: "Another agent" });
|
||||
await db.update(companies).set({ status: "paused" }).where(eq(companies.id, companyId));
|
||||
await budgetService(db).getInvocationBlock(companyId, one.id);
|
||||
expect((await current(one.id)).lifecycleHolds).toContain("company_paused");
|
||||
expect((await current(two.id)).lifecycleHolds).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps a manual hold when company policy changes", async () => {
|
||||
const work = worker(); const lifecycle = createAgentLifecycle(db); const agent = await hire(); await work.process(agent.id);
|
||||
await lifecycle.pauseAgent(agent.id); await work.process(agent.id);
|
||||
await db.update(companies).set({ status: "archived" }).where(eq(companies.id, companyId));
|
||||
await createAgentLifecycle(db).reconcilePolicyHolds(companyId); await work.process(agent.id);
|
||||
await db.update(companies).set({ status: "active" }).where(eq(companies.id, companyId));
|
||||
await createAgentLifecycle(db).reconcilePolicyHolds(companyId); await work.process(agent.id);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "paused", lifecycleHolds: ["manual"] });
|
||||
await lifecycle.resumeAgent(agent.id); await work.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("ready");
|
||||
});
|
||||
|
||||
it("defers automatic policy changes while a task holds the agent lock", async () => {
|
||||
const agent = await hire("paused");
|
||||
await db.transaction(async tx => {
|
||||
await tx.select().from(agents).where(eq(agents.id, agent.id)).for("update");
|
||||
expect(await createLifecycleStore(db).change(agent.id, "reconcile")).toBeNull();
|
||||
});
|
||||
expect(await createLifecycleStore(db).change(agent.id, "reconcile")).not.toBeNull();
|
||||
});
|
||||
|
||||
it("retains the failed phase and the same operation for retry", async () => {
|
||||
let fail = true;
|
||||
const work = worker(async agent => { if (agent.lifecycleState === "verifying" && fail) throw new Error("private detail"); return "complete"; });
|
||||
const agent = await hire(); await work.process(agent.id);
|
||||
const failed = await current(agent.id);
|
||||
expect(failed.lifecycleState).toBe("verifying");
|
||||
expect(failed.lifecycleError).not.toContain("private detail");
|
||||
fail = false;
|
||||
await createAgentLifecycle(db).retry(agent.id); await work.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("ready");
|
||||
});
|
||||
|
||||
it("fences a late completion when termination replaces preparation", async () => {
|
||||
const agent = await hire("paused");
|
||||
const store = createLifecycleStore(db);
|
||||
await createAgentLifecycle(db).resumeAgent(agent.id);
|
||||
const claimed = await store.claim(agent.id, "old", new Date());
|
||||
expect(claimed).not.toBeNull();
|
||||
await store.setRequiredPlugins(claimed!, "old", []);
|
||||
await createAgentLifecycle(db).terminateAgent(agent.id);
|
||||
expect(await store.completeHost(claimed!, "old")).toBe(false);
|
||||
expect(await store.change(agent.id, "complete", { owner: "old", version: claimed!.lifecycleVersion })).toBeNull();
|
||||
expect((await current(agent.id)).lifecycleState).toBe("terminating");
|
||||
});
|
||||
|
||||
it("revokes keys at termination and waits for cleanup before deletion", async () => {
|
||||
let pending = true;
|
||||
const phases: string[] = [];
|
||||
const work = worker(async agent => { phases.push(agent.lifecycleState); return agent.lifecycleState === "cleaning_up" && pending ? "pending" : "complete"; });
|
||||
const lifecycle = createAgentLifecycle(db); const agent = await hire(); await work.process(agent.id);
|
||||
await db.insert(agentApiKeys).values({ agentId: agent.id, companyId, name: "Test", keyHash: randomUUID() });
|
||||
await lifecycle.terminateAgent(agent.id); await work.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("cleaning_up");
|
||||
expect((await db.select().from(agentApiKeys).where(eq(agentApiKeys.agentId, agent.id)))[0].revokedAt).not.toBeNull();
|
||||
await expect(agentConfiguration(db).remove(agent.id)).rejects.toThrow("Complete termination");
|
||||
pending = false; await lifecycle.retry(agent.id); await work.process(agent.id);
|
||||
expect(phases).toContain("terminating");
|
||||
expect((await current(agent.id)).lifecycleState).toBe("terminated");
|
||||
await agentConfiguration(db).remove(agent.id);
|
||||
expect(await current(agent.id)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("retains required plugin IDs and checks the operation and company on each reply", async () => {
|
||||
const agent = await hire("paused");
|
||||
const pluginId = randomUUID();
|
||||
await db.insert(plugins).values({ id: pluginId, pluginKey: pluginId, packageName: "lifecycle-test", version: "1.0.0", status: "ready",
|
||||
manifestJson: { agentLifecycle: true, capabilities: ["agents.lifecycle.manage"] } as never });
|
||||
const call = vi.fn(async (_id, _method, input) => ({ operationId: input.operationId, version: input.version, status: "complete" }));
|
||||
const driver = createLifecycleDriver(db, { call } as never);
|
||||
const snapshot = (await createLifecycleStore(db).get(agent.id))!;
|
||||
expect(await driver.requiredPluginIds(snapshot)).toContain(pluginId);
|
||||
await db.insert(pluginCompanySettings).values({ companyId, pluginId, enabled: false });
|
||||
expect(await driver.requiredPluginIds(snapshot)).not.toContain(pluginId);
|
||||
expect(await driver.requiredPluginIds({ ...snapshot, lifecycleRequiredPluginIds: [pluginId] })).toContain(pluginId);
|
||||
await expect(driver.runPlugin(snapshot, pluginId)).rejects.toThrow("unavailable");
|
||||
await db.update(pluginCompanySettings).set({ enabled: true }).where(eq(pluginCompanySettings.pluginId, pluginId));
|
||||
expect(await driver.runPlugin(snapshot, pluginId)).toBe("complete");
|
||||
expect(call).toHaveBeenCalledWith(pluginId, "agentLifecycle", { companyId, agentId: agent.id,
|
||||
operationId: snapshot.lifecycleOperation!.id, version: snapshot.lifecycleVersion, phase: "paused" }, 30_000);
|
||||
call.mockResolvedValueOnce({ operationId: "stale", version: snapshot.lifecycleVersion, status: "complete" });
|
||||
await expect(driver.runPlugin(snapshot, pluginId)).rejects.toThrow("Invalid lifecycle result");
|
||||
});
|
||||
|
||||
it("keeps host completion and required plugin results across a worker restart", async () => {
|
||||
const pluginIds = [randomUUID(), randomUUID()];
|
||||
const runHost = vi.fn(async () => "complete" as const);
|
||||
const firstPlugin = vi.fn(async (_agent: LifecycleAgent, pluginId: string) =>
|
||||
pluginId === pluginIds[0] ? "complete" as const : "pending" as const);
|
||||
const first = configureAgentLifecycle(db, { requiredPluginIds: async () => pluginIds, runHost, runPlugin: firstPlugin });
|
||||
workers.push(first);
|
||||
const agent = await hire();
|
||||
await first.process(agent.id);
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "preparing", lifecycleRequiredPluginIds: pluginIds,
|
||||
lifecycleOperation: { hostComplete: true, completedPluginIds: [pluginIds[0]] } });
|
||||
await first.stop();
|
||||
|
||||
const discover = vi.fn(async () => []);
|
||||
const resumedPlugin = vi.fn(async (_agent: LifecycleAgent, _pluginId: string) => "complete" as const);
|
||||
const resumed = configureAgentLifecycle(db, { requiredPluginIds: discover, runHost, runPlugin: resumedPlugin });
|
||||
workers.push(resumed);
|
||||
await createAgentLifecycle(db).retry(agent.id);
|
||||
await resumed.process(agent.id);
|
||||
expect((await current(agent.id)).lifecycleState).toBe("ready");
|
||||
expect(discover).not.toHaveBeenCalled();
|
||||
expect(runHost).toHaveBeenCalledTimes(2);
|
||||
expect(resumedPlugin.mock.calls.map(call => call[1])).toEqual([pluginIds[1], ...pluginIds]);
|
||||
});
|
||||
|
||||
it("requires host and plugin completion and freezes the selected plugin IDs", async () => {
|
||||
const agent = await hire();
|
||||
const store = createLifecycleStore(db);
|
||||
const claimed = (await store.claim(agent.id, "test", new Date()))!;
|
||||
const pluginId = randomUUID();
|
||||
expect(await store.setRequiredPlugins(claimed, "test", [pluginId])).toBe(true);
|
||||
expect(await store.setRequiredPlugins(claimed, "test", [])).toBe(false);
|
||||
expect(await store.completePlugin(claimed, "test", randomUUID())).toBe(false);
|
||||
const complete = () => store.change(agent.id, "complete", { owner: "test", version: claimed.lifecycleVersion });
|
||||
expect(await complete()).toBeNull();
|
||||
expect(await store.completePlugin(claimed, "test", pluginId)).toBe(true);
|
||||
expect(await complete()).toBeNull();
|
||||
expect(await store.completeHost(claimed, "test")).toBe(true);
|
||||
expect(await complete()).toMatchObject({ lifecycleState: "verifying" });
|
||||
expect(await store.completePlugin(claimed, "test", pluginId)).toBe(false);
|
||||
});
|
||||
|
||||
it("fences verification when the saved configuration changes", async () => {
|
||||
const lifecycle = createAgentLifecycle(db); const store = createLifecycleStore(db);
|
||||
const agent = await hire();
|
||||
const claimed = (await store.claim(agent.id, "configuration-test", new Date()))!;
|
||||
await store.setRequiredPlugins(claimed, "configuration-test", []);
|
||||
await agentConfiguration(db).update(agent.id, { adapterConfig: { command: "echo" } });
|
||||
expect(await store.completeHost(claimed, "configuration-test")).toBe(false);
|
||||
expect((await lifecycle.get(agent.id))!.lifecycleVersion).toBeGreaterThan(claimed.lifecycleVersion);
|
||||
});
|
||||
|
||||
it("invalidates only changed execution configuration during setup", async () => {
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
const agent = await hire();
|
||||
const before = await current(agent.id);
|
||||
await lifecycle.updateConfiguration(agent.id, { name: "Profile only" });
|
||||
await lifecycle.updateConfiguration(agent.id, { adapterConfig: before.adapterConfig });
|
||||
expect((await current(agent.id)).lifecycleVersion).toBe(before.lifecycleVersion);
|
||||
await lifecycle.updateConfiguration(agent.id, { runtimeConfig: { heartbeat: { enabled: false } } });
|
||||
expect((await current(agent.id)).lifecycleVersion).toBe(before.lifecycleVersion + 1);
|
||||
const work = worker();
|
||||
await work.process(agent.id);
|
||||
const ready = await current(agent.id);
|
||||
expect(ready.lifecycleState).toBe("ready");
|
||||
await lifecycle.updateConfiguration(agent.id, { adapterConfig: { command: "echo" } });
|
||||
expect(await current(agent.id)).toMatchObject({ lifecycleState: "ready", lifecycleVersion: ready.lifecycleVersion });
|
||||
});
|
||||
|
||||
it("rejects identity, accounting, and unknown fields through every configuration entry point", async () => {
|
||||
const agent = await hire();
|
||||
const lifecycle = createAgentLifecycle(db);
|
||||
const before = await current(agent.id);
|
||||
for (const patch of [{ id: randomUUID() }, { companyId: randomUUID() }, { spentMonthlyCents: 0 },
|
||||
{ spendMonthUtc: "2099-01" }, { lastHeartbeatAt: new Date() }, { errorReason: null },
|
||||
{ createdAt: new Date() }, { updatedAt: new Date() }, { lifecycleState: "ready" }, { futureColumn: true }]) {
|
||||
await expect(lifecycle.updateConfiguration(agent.id, patch as never)).rejects.toThrow("Unsupported configuration field");
|
||||
await expect(lifecycle.updateAndTransition(agent.id, "pause", patch as never)).rejects.toThrow("Unsupported configuration field");
|
||||
await expect(db.transaction(tx => updateAgentConfigurationInTransaction(tx as unknown as Db, agent.id, patch as never)))
|
||||
.rejects.toThrow("Unsupported configuration field");
|
||||
}
|
||||
expect(await current(agent.id)).toEqual(before);
|
||||
const effects = createAgentLifecycleEffects();
|
||||
const malformed = createLifecycleCommands(db, { ...effects, prepareConfiguration: async () => ({ id: randomUUID() }) as never });
|
||||
await expect(malformed.updateConfiguration(agent.id, { name: "Changed" })).rejects.toThrow("Unsupported configuration field");
|
||||
expect(await current(agent.id)).toEqual(before);
|
||||
});
|
||||
|
||||
it("records only the changed configuration fields for a legacy appearance", async () => {
|
||||
const [agent] = await db.insert(agents).values({ companyId, name: "Legacy", appearance: null }).returning();
|
||||
await createAgentLifecycle(db).updateConfiguration(agent.id, { name: "Renamed" },
|
||||
{ recordRevision: { source: "test", createdByUserId: "board" } });
|
||||
const revisions = await db.select().from(agentConfigRevisions).where(eq(agentConfigRevisions.agentId, agent.id));
|
||||
expect(revisions).toHaveLength(1);
|
||||
expect(revisions[0].changedKeys).toEqual(["name"]);
|
||||
expect(revisions[0].beforeConfig.appearance).toEqual(revisions[0].afterConfig.appearance);
|
||||
});
|
||||
|
||||
it("accepts the unchanged resolved appearance of a pending hire", async () => {
|
||||
const [agent] = await db.insert(agents).values({ companyId, name: "Pending legacy", status: "pending_approval", appearance: null }).returning();
|
||||
const resolved = await agentConfiguration(db).getById(agent.id);
|
||||
await expect(createAgentLifecycle(db).updateConfiguration(agent.id, { appearance: resolved!.appearance }))
|
||||
.resolves.toMatchObject({ id: agent.id });
|
||||
await expect(createAgentLifecycle(db).updateConfiguration(agent.id, { name: "Changed" }))
|
||||
.rejects.toThrow("Pending approval agent configuration cannot be changed");
|
||||
});
|
||||
|
||||
it("rolls back a root configuration command when dependent records fail", async () => {
|
||||
const agent = await hire();
|
||||
const before = await current(agent.id);
|
||||
const effects = createAgentLifecycleEffects();
|
||||
const lifecycle = createLifecycleCommands(db, { ...effects, completeConfiguration: async (...args) => {
|
||||
await effects.completeConfiguration(...args);
|
||||
throw new Error("Configuration completion failed");
|
||||
} });
|
||||
await expect(lifecycle.updateConfiguration(agent.id, { adapterConfig: { command: "echo" } },
|
||||
{ recordRevision: { source: "test", createdByUserId: "board" } })).rejects.toThrow("Configuration completion failed");
|
||||
expect(await current(agent.id)).toMatchObject({ adapterConfig: before.adapterConfig,
|
||||
lifecycleVersion: before.lifecycleVersion, lifecycleOperation: before.lifecycleOperation });
|
||||
expect(await db.select().from(agentConfigRevisions).where(eq(agentConfigRevisions.agentId, agent.id))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("rolls back configuration and verification invalidation in the binding transaction", async () => {
|
||||
const agent = await hire();
|
||||
const before = await current(agent.id);
|
||||
await expect(updateAgentConfigurationInTransaction(db, agent.id, { name: "Uncommitted" }))
|
||||
.rejects.toThrow("requires a database transaction");
|
||||
expect(() => updateAgentConfigurationInTransaction(db, agent.id, { budgetMonthlyCents: 100 } as never))
|
||||
.toThrow("Budget changes require a root lifecycle command");
|
||||
await expect(db.transaction(async tx => agentConfiguration(tx as unknown as Db).update(agent.id, { name: "Uncommitted" })))
|
||||
.rejects.toThrow("root database");
|
||||
await expect(db.transaction(async tx => {
|
||||
await updateAgentConfigurationInTransaction(tx as unknown as Db, agent.id, { adapterConfig: { command: "echo" } });
|
||||
expect((await tx.select().from(agents).where(eq(agents.id, agent.id)))[0].lifecycleVersion).toBe(before.lifecycleVersion + 1);
|
||||
throw new Error("Binding change failed");
|
||||
})).rejects.toThrow("Binding change failed");
|
||||
expect(await current(agent.id)).toMatchObject({ adapterConfig: before.adapterConfig,
|
||||
lifecycleVersion: before.lifecycleVersion, lifecycleOperation: before.lifecycleOperation });
|
||||
});
|
||||
|
||||
it("permits a board retry and rejects an agent or a different company", async () => {
|
||||
const agent = await hire();
|
||||
function app(actor: Express.Request["actor"]) {
|
||||
const app = express(); app.use(express.json());
|
||||
app.use((req, _res, next) => { req.actor = actor; next(); });
|
||||
app.use("/api", agentRoutes(db)); app.use(errorHandler); return app;
|
||||
}
|
||||
const board: Express.Request["actor"] = { type: "board", source: "session", userId: "board", companyIds: [companyId],
|
||||
memberships: [{ companyId, membershipRole: "admin", status: "active" }] };
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: "board", membershipRole: "admin" });
|
||||
await db.insert(principalPermissionGrants).values(["agents:create", "agents:configure"].map(permissionKey => ({ companyId, principalType: "user", principalId: "board", permissionKey })));
|
||||
const url = `/api/agents/${agent.id}/lifecycle/retry`;
|
||||
const response = await request(app(board)).post(url).send({});
|
||||
expect(response.status, JSON.stringify(response.body)).toBe(200);
|
||||
expect(response.body.lifecycleState).toBe("preparing");
|
||||
expect(response.body).not.toHaveProperty("lifecycleOperation");
|
||||
expect(await db.select().from(activityLog).where(and(eq(activityLog.entityId, agent.id), eq(activityLog.action, "agent.lifecycle_retried"))))
|
||||
.toEqual([expect.objectContaining({ actorType: "user", actorId: "board" })]);
|
||||
await request(app({ type: "agent", source: "agent_key", companyId, agentId: agent.id })).post(url).send({}).expect(403);
|
||||
const denied = await request(app({ ...board, companyIds: [], memberships: [] })).post(url).send({});
|
||||
expect([403, 404]).toContain(denied.status);
|
||||
});
|
||||
|
||||
it("recovers an expired lease and never accepts an expired completion", async () => {
|
||||
const agent = await hire("paused"); const store = createLifecycleStore(db);
|
||||
await createAgentLifecycle(db).resumeAgent(agent.id);
|
||||
const claim = await store.claim(agent.id, "first", new Date(Date.now() - 180_000));
|
||||
expect(claim).not.toBeNull();
|
||||
await store.setRequiredPlugins(claim!, "first", []);
|
||||
await store.completeHost(claim!, "first");
|
||||
expect(await store.change(agent.id, "complete", { owner: "first", version: claim!.lifecycleVersion })).toBeNull();
|
||||
expect(await store.claim(agent.id, "second", new Date())).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("lifecycle transition rules", () => {
|
||||
it("does not bypass authorization, verification, or cleanup", () => {
|
||||
expect(() => transition("pending_approval", "resume")).toThrow();
|
||||
expect(transition("preparing", "complete")).toBe("verifying");
|
||||
expect(transition("terminating", "complete")).toBe("cleaning_up");
|
||||
expect(transition("cleaning_up", "complete")).toBe("terminated");
|
||||
expect(() => transition("terminated", "resume")).toThrow();
|
||||
expect(transition("terminating", "terminate")).toBe("terminating");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,63 @@
|
||||
import { agents, companies, createDb, type Db } from "@paperclipai/db";
|
||||
import { asc, eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { startAgentLifecycle } from "../index.js";
|
||||
import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
(support.supported ? describe : describe.skip)("lifecycle policy sweep", () => {
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: Db;
|
||||
beforeAll(async () => {
|
||||
database = await startEmbeddedPostgresTestDatabase("paperclip-lifecycle-policy-");
|
||||
db = createDb(database.connectionString);
|
||||
}, 90_000);
|
||||
afterAll(async () => { await database?.cleanup(); });
|
||||
|
||||
it("caps each policy page, skips ineligible agents, and returns to the first page", async () => {
|
||||
const [company] = await db.insert(companies).values({ name: "Policy pages", issuePrefix: "POL" }).returning();
|
||||
const eligible = await db.insert(agents).values(Array.from({ length: 101 }, (_, index) => ({
|
||||
id: `00000000-0000-4000-8000-${String(index + 1).padStart(12, "0")}`,
|
||||
companyId: company.id, name: `Agent ${index}`, status: "idle", lifecycleState: "ready" as const,
|
||||
lifecycleHolds: ["company_paused"], pauseReason: "company_paused",
|
||||
}))).returning();
|
||||
const excluded = await db.insert(agents).values((["pending_approval", "terminating", "cleaning_up", "terminated", "rejected"] as const).map(lifecycleState => ({
|
||||
companyId: company.id, name: lifecycleState, status: "terminated", lifecycleState,
|
||||
lifecycleHolds: ["company_paused"], pauseReason: "company_paused",
|
||||
}))).returning();
|
||||
const effects = createAgentLifecycleEffects();
|
||||
const transaction = vi.spyOn(effects, "transaction");
|
||||
const now = Date.now();
|
||||
const clock = vi.spyOn(Date, "now").mockReturnValue(now);
|
||||
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => ({ unref() {} }) as ReturnType<typeof setInterval>);
|
||||
const clearTimer = vi.spyOn(globalThis, "clearInterval").mockImplementation(() => {});
|
||||
const work = startAgentLifecycle(db, effects, {
|
||||
requiredPluginIds: async () => [], runHost: async () => "pending", runPlugin: async () => "pending",
|
||||
}, () => true);
|
||||
const rows = () => db.select().from(agents).where(eq(agents.companyId, company.id)).orderBy(asc(agents.id));
|
||||
try {
|
||||
await work.sweep();
|
||||
expect(transaction).toHaveBeenCalledTimes(100);
|
||||
expect((await rows()).filter(row => row.lifecycleHolds.length === 0)).toHaveLength(100);
|
||||
await work.sweep();
|
||||
expect(transaction).toHaveBeenCalledTimes(100);
|
||||
clock.mockReturnValue(now + 60_001);
|
||||
await work.sweep();
|
||||
expect(transaction).toHaveBeenCalledTimes(101);
|
||||
expect((await rows()).filter(row => row.lifecycleHolds.length > 0).map(row => row.id).sort())
|
||||
.toEqual(excluded.map(row => row.id).sort());
|
||||
await db.update(agents).set({ lifecycleHolds: ["company_paused"] }).where(eq(agents.id, eligible[0].id));
|
||||
clock.mockReturnValue(now + 120_002);
|
||||
await work.sweep();
|
||||
expect(transaction).toHaveBeenCalledTimes(201);
|
||||
expect((await rows())[0].lifecycleHolds).toEqual([]);
|
||||
} finally {
|
||||
await work.stop();
|
||||
transaction.mockRestore();
|
||||
clock.mockRestore();
|
||||
timer.mockRestore();
|
||||
clearTimer.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user