Files
PaperClipAI/server/src/modules/agent-lifecycle/adapters/postgres.ts
T
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:54:23 -06:00

326 lines
22 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, asc, eq, inArray, sql } from "drizzle-orm";
import { agents, agentApiKeys, companies, budgetPolicies, type Db } from "@paperclipai/db";
import type { AgentLifecycleOperation } from "@paperclipai/shared";
import { AgentLifecycleConflict, canConfigureAgentConnection, compatibilityStatus, isLifecycleWorkPending, transition, type LifecycleCommand } from "../domain/policy.js";
import type { LifecycleEffects } from "./effects.js";
import type { LifecycleStore } from "../application/ports.js";
export function assertRootDatabase(db: Db) {
if ("nestedIndex" in db) throw new AgentLifecycleConflict("Lifecycle commands require the root database connection");
}
type LifecycleRow = typeof agents.$inferSelect;
const LIFECYCLE_PENDING_BASE_MS = 2_000;
const LIFECYCLE_PENDING_MAX_MS = 60_000;
export function lifecyclePendingRetryDelayMs(passes: number) {
return Math.min(LIFECYCLE_PENDING_MAX_MS, LIFECYCLE_PENDING_BASE_MS * 2 ** Math.max(0, passes - 1));
}
type ChangeOptions = { version?: number; owner?: string; reason?: string };
// Prepared drafts wait for the wizard; they have no background verification obligation.
export const pendingLifecycleWork = and(
inArray(agents.lifecycleState, ["preparing", "verifying", "pausing", "resuming", "terminating", "cleaning_up"]),
sql`not (${agents.lifecycleState} = 'verifying' and ${agents.lifecycleHolds} @> '["setup"]'::jsonb)`,
sql`not (${agents.lifecycleState} in ('verifying', 'resuming') and coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') = 'true')`,
sql`not (${agents.lifecycleState} = 'verifying' and coalesce(${agents.lifecycleOperation}->>'waitingForSignal', 'false') = 'true')`,
);
/** A bounded startup pass rechecks this saved wait once, without a timer. */
export const lifecycleSignalWait = and(
eq(agents.lifecycleState, "verifying"),
sql`coalesce(${agents.lifecycleOperation}->>'waitingForSignal', 'false') = 'true'`,
sql`coalesce(${agents.lifecycleOperation}->>'retryRequired', 'false') <> 'true'`,
);
/** The producer owns this outer transaction. Register its intent before either
* the recheck write or the corresponding external fact commits. */
/**
* Releases a lifecycle lease whose holder process restart recovery proved gone.
* Compare-and-swap on that exact holder and lease owner; the lifecycle state,
* operation and progress are untouched, so the worker claims the step again.
*/
export async function releaseAgentLifecycleLeaseOfHolderInTransaction(
tx: Pick<Db, "select" | "update">,
input: { agentId: string; holder: { bootId: string | null; pid: number | null } },
notify: () => Promise<void>,
): Promise<{ released: false; reason: string } | { released: true; agent: LifecycleRow; leaseUntil: string }> {
if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Lifecycle lease release requires the caller transaction");
const [agent] = await tx.select().from(agents).where(eq(agents.id, input.agentId)).for("update");
const operation = agent?.lifecycleOperation;
if (!agent || !operation?.leaseOwner || !operation.leaseUntil) return { released: false, reason: "lease_no_longer_held" };
if (operation.leaseHolder?.bootId !== input.holder.bootId || operation.leaseHolder?.pid !== input.holder.pid) {
return { released: false, reason: "owner_changed_since_proof" };
}
// Register the wake intent before the write so drain and sleep checks see it.
await notify();
const [row] = await tx.update(agents).set({
lifecycleOperation: sql`${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder'`,
}).where(and(eq(agents.id, agent.id), sql`${agents.lifecycleOperation}->>'leaseOwner' = ${operation.leaseOwner}`))
.returning({ id: agents.id });
return row ? { released: true, agent, leaseUntil: operation.leaseUntil } : { released: false, reason: "lease_no_longer_held" };
}
export async function requestAgentLifecycleRecheckInTransaction(
tx: Pick<Db, "select" | "update">,
input: { companyId: string; agentId: string; expectedVersion?: number; onlyIfWaiting?: boolean; configurationRepaired?: boolean },
notify: () => Promise<void>,
): Promise<boolean> {
if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Lifecycle rechecks require the producer transaction");
const [agent] = await tx.select().from(agents).where(and(
eq(agents.id, input.agentId), eq(agents.companyId, input.companyId),
)).for("update");
if (!agent || !["preparing", "verifying", "resuming"].includes(agent.lifecycleState)
|| agent.lifecycleOperation?.retryRequired && !input.configurationRepaired
|| input.expectedVersion !== undefined && agent.lifecycleVersion !== input.expectedVersion
|| input.onlyIfWaiting && !agent.lifecycleOperation?.waitingForSignal) return false;
await notify();
// Expiry permits a replacement claim; only an owner change fences a late park.
const owned = Boolean(agent.lifecycleOperation?.leaseOwner);
await tx.update(agents).set({ lifecycleError: null,
lifecycleOperation: owned
? sql`${agents.lifecycleOperation} || '{"recheckRequested":true}'::jsonb`
: input.configurationRepaired
? sql`${agents.lifecycleOperation} - 'waitingForSignal' - 'recheckRequested' - 'retryAt' - 'retryRequired' - 'verificationFailures'`
: sql`${agents.lifecycleOperation} - 'waitingForSignal' - 'recheckRequested' - 'retryAt'`,
updatedAt: new Date(),
}).where(and(eq(agents.id, agent.id), eq(agents.companyId, input.companyId), eq(agents.lifecycleVersion, agent.lifecycleVersion)));
return true;
}
export async function wakeStoppingAgentLifecycleInTransaction(
tx: Pick<Db, "select" | "update">,
agentId: string,
notify: () => Promise<void>,
): Promise<boolean> {
if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Lifecycle wakes require the producer transaction");
const [agent] = await tx.select({ lifecycleState: agents.lifecycleState, lifecycleOperation: agents.lifecycleOperation })
.from(agents).where(eq(agents.id, agentId)).for("update");
const operation = agent?.lifecycleOperation;
if (!agent || !operation || !["pausing", "terminating", "cleaning_up"].includes(agent.lifecycleState)
|| (!operation.retryAt && !operation.leaseOwner)) return false;
await notify();
await tx.update(agents).set({ lifecycleOperation: sql`(${agents.lifecycleOperation} - 'retryAt')
|| case when ${agents.lifecycleOperation} ? 'leaseOwner' then '{"recheckRequested":true}'::jsonb else '{}'::jsonb end` })
.where(eq(agents.id, agentId));
return true;
}
/** The caller owns the accounting transaction and has locked this agent. */
async function changeLockedAgent(tx: Db, effects: LifecycleEffects, agent: LifecycleRow,
command: LifecycleCommand, options: ChangeOptions = {}, policyHolds?: string[], restartPause = false) {
if (options.version !== undefined && options.version !== agent.lifecycleVersion) return null;
if (command === "resume" && agent.lifecycleHolds.includes("setup")) throw new AgentLifecycleConflict("Finish agent setup before resuming");
if (command === "complete" && (!options.owner || !agent.lifecycleOperation?.leaseUntil || agent.lifecycleOperation?.leaseOwner !== options.owner ||
Date.parse(agent.lifecycleOperation?.leaseUntil ?? "") <= Date.now() ||
!agent.lifecycleOperation.hostComplete || agent.lifecycleRequiredPluginIds === null ||
agent.lifecycleRequiredPluginIds.some(id => !agent.lifecycleOperation!.completedPluginIds.includes(id)))) return null;
if (command === "retry") {
transition(agent.lifecycleState, command);
if (agent.lifecycleOperation?.leaseUntil && Date.parse(agent.lifecycleOperation.leaseUntil) > Date.now()) {
await effects.notifyWork(tx);
const [requested] = await tx.update(agents).set({ lifecycleOperation:
sql`${agents.lifecycleOperation} || '{"recheckRequested":true}'::jsonb`, updatedAt: new Date(),
}).where(eq(agents.id, agent.id)).returning();
return requested;
}
await effects.notifyWork(tx);
const [updated] = await tx.update(agents).set({ lifecycleError: null,
lifecycleOperation: sql`${agents.lifecycleOperation} - 'retryAt' - 'retryRequired' - 'verificationFailures' - 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'waitingForSignal' - 'recheckRequested'`, updatedAt: new Date() }).where(eq(agents.id, agent.id)).returning();
return updated;
}
if (command === "resume" && !["ready", "paused", "pausing", "resuming"].includes(agent.lifecycleState)) transition(agent.lifecycleState, command);
let holds = policyHolds ?? agent.lifecycleHolds;
const reason = options.reason ?? "manual";
if (command === "reconcile" && policyHolds === undefined) {
if (["pending_approval", "terminating", "cleaning_up", "terminated", "rejected"].includes(agent.lifecycleState)) return agent;
const [company] = await tx.select().from(companies).where(eq(companies.id, agent.companyId));
const policies = await tx.select().from(budgetPolicies).where(and(eq(budgetPolicies.companyId, agent.companyId),
eq(budgetPolicies.scopeType, "agent"), eq(budgetPolicies.scopeId, agent.id)));
let budgetBlocked = false;
for (const policy of policies) if (await effects.policyBlocks(tx, policy)) budgetBlocked = true;
holds = holds.filter(hold => hold !== "budget" && hold !== "company_archived" && hold !== "company_paused");
if (budgetBlocked) holds.push("budget");
if (company.status === "archived") holds.push("company_archived");
if (company.status === "paused") holds.push("company_paused");
}
if (command === "pause") holds = [...new Set([...holds, reason])];
if (command === "resume") holds = holds.filter(hold => reason === "user" ? ["budget", "company_archived", "company_paused"].includes(hold) : hold !== reason);
const blockingHolds = holds.filter(hold => hold !== "setup" || !["preparing", "verifying"].includes(agent.lifecycleState));
let next = command === "reconcile"
? blockingHolds.length && !["pausing", "paused"].includes(agent.lifecycleState) ? "pausing" as const
: !holds.length && agent.lifecycleState === "paused" ? "resuming" as const : agent.lifecycleState
: command === "resume" && (holds.length || agent.lifecycleState === "pausing") ? agent.lifecycleState
: transition(agent.lifecycleState, command, agent.lifecycleOperation?.resumeState);
if (restartPause && next === "paused") next = "pausing";
if (command === "complete" && next === "ready" && holds.length) next = "pausing";
if (command === "complete" && next === "paused" && !holds.length) next = "resuming";
if (["complete", "reconcile"].includes(command) && next === "paused" && holds.length === 1 && holds[0] === "setup"
&& await effects.setupRequiresPreparation(tx, agent.companyId, agent.defaultEnvironmentId, agent)) next = "preparing";
if (next === agent.lifecycleState) {
if (JSON.stringify(holds) !== JSON.stringify(agent.lifecycleHolds) || agent.pauseReason !== (holds[0] ?? null)) {
const [updated] = await tx.update(agents).set({ lifecycleHolds: holds, pauseReason: holds[0] ?? null }).where(eq(agents.id, agent.id)).returning();
return updated;
}
return agent;
}
const operation: AgentLifecycleOperation = {
responsibleUserId: agent.lifecycleOperation?.responsibleUserId,
id: randomUUID(), hostComplete: false, completedPluginIds: [], attempts: 0,
resumeState: next === "pausing"
? agent.lifecycleState === "preparing" || agent.lifecycleState === "verifying" ? agent.lifecycleState : agent.lifecycleOperation?.resumeState ?? "ready"
: agent.lifecycleOperation?.resumeState,
};
if (next === "ready") operation.resumeState = "ready";
if (next === "pausing") {
const [company] = await tx.select({ pauseReason: companies.pauseReason }).from(companies)
.where(eq(companies.id, agent.companyId));
operation.pauseCause = { holds, companyPauseReason: company?.pauseReason ?? null };
}
const status = compatibilityStatus(next, agent.status);
if(!canConfigureAgentConnection(agent)&&canConfigureAgentConnection({lifecycleState:next,status}))await effects.beforeConnectionEligibilityRestore?.(tx,agent.companyId,agent.id);
if (status !== agent.status) await effects.beforeStatusChange(tx, agent.companyId, agent.id, agent.status, status);
if (isLifecycleWorkPending(next)) await effects.notifyWork(tx);
const [updated] = await tx.update(agents).set({
lifecycleHolds: holds, lifecycleState: next, lifecycleVersion: agent.lifecycleVersion + 1, lifecycleError: null,
lifecycleOperation: operation, status,
pauseReason: holds[0] ?? null,
pausedAt: next === "pausing" ? new Date() : next === "ready" ? null : agent.pausedAt,
updatedAt: new Date(),
}).where(eq(agents.id, agent.id)).returning();
if (next === "terminating" || next === "rejected") {
await tx.update(agentApiKeys).set({ revokedAt: new Date() }).where(eq(agentApiKeys.agentId, agent.id));
await effects.clearPrimary(tx, agent.companyId, agent.id);
}
if (command === "approve") await effects.recordCreation(tx, agent.companyId, agent.id);
if (next === "pausing" || next === "resuming" || next === "terminating" || next === "rejected") {
await effects.recordStatus(tx, agent.companyId, agent.id,
next === "resuming" ? "paused" : "idle",
next === "resuming" ? "idle" : next === "pausing" ? "paused" : "terminated");
}
return updated;
}
/** Apply a computed hold without resolving policies or opening another transaction. */
export async function applyAgentPolicyHoldsInTransaction(tx: Db, effects: LifecycleEffects,
companyId: string, agentId: string, patch: { budget?: boolean; companyStatus?: string; restartPause?: boolean }, lockedAgent?: LifecycleRow) {
if (!("nestedIndex" in tx)) throw new AgentLifecycleConflict("Policy holds require the caller's accounting transaction");
const agent = lockedAgent ?? (await tx.select().from(agents).where(and(
eq(agents.id, agentId), eq(agents.companyId, companyId),
)).for("update"))[0];
if (!agent || agent.companyId !== companyId || ["pending_approval", "terminating", "cleaning_up", "terminated", "rejected"].includes(agent.lifecycleState)) return null;
let holds = [...agent.lifecycleHolds];
if (patch.budget !== undefined) {
holds = holds.filter(hold => hold !== "budget");
if (patch.budget) holds.push("budget");
}
if (patch.companyStatus !== undefined) {
holds = holds.filter(hold => hold !== "company_paused" && hold !== "company_archived");
if (patch.companyStatus === "paused") holds.push("company_paused");
if (patch.companyStatus === "archived") holds.push("company_archived");
}
return changeLockedAgent(tx, effects, agent, "reconcile", {}, holds, patch.restartPause);
}
export function createLifecycleStore(db: Db, effects: LifecycleEffects): LifecycleStore {
const get = (id: string) => db.select().from(agents).where(eq(agents.id, id)).then(rows => rows[0] ?? null);
return {
get,
async change(id, command, options = {}) {
const existing = await get(id);
if (!existing) return null;
return effects.transaction(db, existing.companyId, async tx => {
const [agent] = await tx.select().from(agents).where(eq(agents.id, id)).for("update", command === "reconcile" ? { skipLocked: true } : undefined);
if (!agent) return null;
return changeLockedAgent(tx, effects, agent, command, options);
});
},
async claim(id, owner, now) {
// leaseHolder names the claiming process, so restart recovery settles a dead holder's lease at once.
const leaseHolder = effects.leaseHolder?.() ?? null;
const [agent] = await db.update(agents).set({
lifecycleOperation: sql`(coalesce(${agents.lifecycleOperation}, '{"id":"", "hostComplete":false, "completedPluginIds":[], "attempts":0}'::jsonb) - 'recheckRequested')
|| jsonb_build_object('leaseOwner', ${owner}::text, 'leaseUntil', ${(new Date(now.getTime() + 120_000)).toISOString()}::text,
'attempts', coalesce((${agents.lifecycleOperation}->>'attempts')::integer, 0) + 1)
|| ${JSON.stringify(leaseHolder ? { leaseHolder } : {})}::jsonb`,
}).where(and(eq(agents.id, id), pendingLifecycleWork,
sql`(${agents.lifecycleState} in ('pausing', 'terminating', 'cleaning_up') or exists (
select 1 from ${companies} where ${companies.id} = ${agents.companyId} and ${companies.status} = 'active'))`,
sql`coalesce((${agents.lifecycleOperation}->>'leaseUntil')::timestamptz, '-infinity') <= ${now.toISOString()}::timestamptz`,
sql`coalesce((${agents.lifecycleOperation}->>'retryAt')::timestamptz, '-infinity') <= ${now.toISOString()}::timestamptz`,
)).returning();
return agent ?? null;
},
async renew(agent, owner, now) {
const rows = await db.update(agents).set({ lifecycleOperation: sql`jsonb_set(${agents.lifecycleOperation}, '{leaseUntil}', ${JSON.stringify(new Date(now.getTime() + 120_000).toISOString())}::jsonb)` })
.where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion), sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
},
async setRequiredPlugins(agent, owner, pluginIds) {
const rows = await db.update(agents).set({ lifecycleRequiredPluginIds: pluginIds })
.where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
agent.lifecycleRequiredPluginIds === null ? sql`${agents.lifecycleRequiredPluginIds} is null`
: sql`${agents.lifecycleRequiredPluginIds} = ${JSON.stringify(agent.lifecycleRequiredPluginIds)}::jsonb`,
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
},
async completeHost(agent, owner) {
const rows = await db.update(agents).set({
lifecycleOperation: sql`jsonb_set(${agents.lifecycleOperation}, '{hostComplete}', 'true'::jsonb)`,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
},
async completePlugin(agent, owner, pluginId) {
const rows = await db.update(agents).set({
lifecycleOperation: sql`jsonb_set(${agents.lifecycleOperation}, '{completedPluginIds}',
coalesce(${agents.lifecycleOperation}->'completedPluginIds', '[]'::jsonb) || ${JSON.stringify([pluginId])}::jsonb)`,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
sql`${agents.lifecycleRequiredPluginIds} @> ${JSON.stringify([pluginId])}::jsonb`,
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
},
async defer(agent, owner, error, now, options) {
if (options?.waitForSignal && agent.lifecycleState === "verifying") {
const rows = await db.update(agents).set({ lifecycleError: null,
lifecycleOperation: sql`(${agents.lifecycleOperation} - 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'retryAt' - 'waitingForSignal' - 'recheckRequested')
|| case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true'
then '{}'::jsonb else '{"waitingForSignal":true}'::jsonb end`,
updatedAt: now,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
}
const previousPasses = (agent.lifecycleOperation as { pendingPasses?: unknown } | null)?.pendingPasses;
const pendingPasses = options?.pendingReason ? (typeof previousPasses === "number" ? previousPasses : 0) + 1 : undefined;
const retry: Partial<AgentLifecycleOperation> = options?.retryRequired && ["verifying", "resuming"].includes(agent.lifecycleState)
? { retryRequired: true }
: { retryAt: new Date(now.getTime() + (error ? 60_000 : pendingPasses ? lifecyclePendingRetryDelayMs(pendingPasses) : 2_000)).toISOString() };
if (options?.pendingReason && pendingPasses) Object.assign(retry, { pendingReason: options.pendingReason, pendingPasses });
if (options?.verificationFailures !== undefined && agent.lifecycleState === "verifying") {
retry.verificationFailures = options.verificationFailures;
}
const rows = await db.update(agents).set({
lifecycleError: sql`case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true' then null else ${error}::text end`,
lifecycleOperation: sql`((case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true'
then ${agents.lifecycleOperation} - 'verificationFailures' else ${agents.lifecycleOperation} end)
- 'leaseOwner' - 'leaseUntil' - 'leaseHolder' - 'retryAt' - 'retryRequired' - 'waitingForSignal' - 'recheckRequested')
|| case when coalesce(${agents.lifecycleOperation}->>'recheckRequested', 'false') = 'true'
then '{}'::jsonb else ${JSON.stringify(retry)}::jsonb end`,
updatedAt: now,
}).where(and(eq(agents.id, agent.id), eq(agents.lifecycleVersion, agent.lifecycleVersion),
sql`${agents.lifecycleOperation}->>'leaseOwner' = ${owner}`)).returning({ id: agents.id });
return rows.length > 0;
},
async pending(limit, now) {
return db.select({ id: agents.id }).from(agents).where(and(
pendingLifecycleWork,
sql`coalesce((${agents.lifecycleOperation}->>'leaseUntil')::timestamptz, '-infinity') <= ${now.toISOString()}::timestamptz`,
sql`coalesce((${agents.lifecycleOperation}->>'retryAt')::timestamptz, '-infinity') <= ${now.toISOString()}::timestamptz`,
)).orderBy(asc(agents.updatedAt), asc(agents.id)).limit(limit).then(rows => rows.map(row => row.id));
},
};
}