mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-12 08:19:54 +02:00
## Thinking Path > - Paperclip controls agent work across service deployments. > - A run can provision a remote sandbox before a process or invocation event exists. > - Each container previously treated its own missing process handle as proof that the run was orphaned. > - Overlapping deployments could therefore fail a run owned by another container. > - This pull request records and renews a controller lease before provisioning. > - A recovery worker must revoke an expired owner before it finalizes the run. ## Linked Issues or Issue Description Merged PR #13272 records startup adapter identity and restores explicit user continuation. This PR adds controller ownership on top of current master. Refs #7997 and #10442 for related replica and ownership problems. Related #13138 addresses silence and detached local processes; this change does not infer death from silence. **What happened?** During an overlapping hosted service deployment, a new container reaped a legacy conversation run that another container was provisioning. The run had no PID or adapter invocation yet. **Expected behavior** A live controller keeps its run. After controller loss, one recovery worker takes cleanup authority and the old controller cannot dispatch further work. **Steps to reproduce** Claim a legacy run in controller A. Start controller B against the same database before A finishes provisioning. Run the startup reaper in B. **Paperclip version or commit** Observed on `663c44cb2b9c28336d38d0b4a6971f4f1964bce6` in a hosted Railway deployment with a Daytona environment. ## What Changed - Add nullable controller boot ID, lease deadline, and execution stage columns. Claim ownership in the queued-to-running update. - Renew ownership independently of run output. Abort and reject dispatch if renewal fails. - Serialize reaper revocation against renewal. Let unfinished recovery claims expire after a restart. - Restrict graceful shutdown to legacy runs owned by the current controller. - Hand ownership back to the existing native coordinator when runtime selection becomes native. - Add twelve database regressions and document the lease contract. Update the task-drain regression to require controller expiry before reaping. ## Verification - `pnpm exec vitest run server/src/services/legacy-controller-lease.test.ts server/src/__tests__/heartbeat-task-drain-admission-release.test.ts`: 14 passed after rebasing onto master (`f12b647ae`). - Queue-interruption regressions in `heartbeat-process-recovery.test.ts`: 2 passed after preserving the new cleanup promotion from #13275. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passed after rebuilding runner TypeScript outputs for the updated master. Broad local tests are omitted at the maintainer’s request; CI owns broad coverage. - Latest-head CI passed on `f255e8e4d5ab2b24b12638a02434e6aa8a2285c5`: [run 34658248569](https://github.com/paperclipai/paperclip/actions/runs/34658248569). All test shards, browser suites, typecheck, build, canary, and security checks passed. Greptile is 5/5 with no unresolved review threads. ## Risks - Additive, idempotent migration; historical rows retain the previous recovery behavior. - Database unavailability aborts new dispatch rather than permitting an unfenced controller to continue. - Lease expiry is permission to clean up, not evidence that remote inference stopped. Follow-up PRs add persistent cleanup and automatic continuation. - Mixed-version deployment still includes old binaries whose reapers do not understand controller leases. ## Model Used OpenAI GPT-6 through Codex, using reasoning, repository inspection, code execution, and test tools. The precise backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
112 lines
4.6 KiB
TypeScript
112 lines
4.6 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, gt, lte, sql } from "drizzle-orm";
|
|
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
|
|
|
// A boot UUID has meaning across containers; a numeric PID does not.
|
|
export const legacyControllerBootId = randomUUID();
|
|
export const LEGACY_CONTROLLER_LEASE_MS = 60_000;
|
|
export const LEGACY_CONTROLLER_RENEW_MS = 10_000;
|
|
|
|
type Run = typeof heartbeatRuns.$inferSelect;
|
|
|
|
/** Commit these fields in the same UPDATE that claims a queued run. */
|
|
export function legacyControllerClaim(runtimeMode: string) {
|
|
if (runtimeMode === "native") return {};
|
|
return {
|
|
controllerBootId: legacyControllerBootId,
|
|
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
|
|
executionStage: "preparing",
|
|
};
|
|
}
|
|
|
|
export async function renewLegacyControllerLease(
|
|
db: Db,
|
|
run: Pick<Run, "id" | "companyId" | "controllerBootId">,
|
|
stage?: "dispatching",
|
|
): Promise<boolean> {
|
|
const [renewed] = await db.update(heartbeatRuns).set({
|
|
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
|
|
...(stage ? { executionStage: stage } : {}),
|
|
}).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.runtimeMode, "legacy"), eq(heartbeatRuns.status, "running"),
|
|
eq(heartbeatRuns.controllerBootId, legacyControllerBootId),
|
|
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
|
|
)).returning({ id: heartbeatRuns.id });
|
|
return Boolean(renewed);
|
|
}
|
|
|
|
export async function hasLiveLegacyController(db: Db, run: Run): Promise<boolean> {
|
|
if (run.runtimeMode === "native" || !run.controllerBootId) return false;
|
|
const [owner] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
gt(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
|
|
));
|
|
return Boolean(owner);
|
|
}
|
|
|
|
/** Atomically revoke an expired controller. Renewal and revocation serialize on
|
|
* the run row. Expiry permits cleanup, never dispatch of a replacement agent. */
|
|
export async function revokeExpiredLegacyController(db: Db, run: Run): Promise<boolean> {
|
|
if (run.runtimeMode === "native" || !run.controllerBootId) return true;
|
|
const [revoked] = await db.update(heartbeatRuns).set({
|
|
controllerBootId: randomUUID(),
|
|
controllerLeaseExpiresAt: sql`clock_timestamp() + interval '60 seconds'`,
|
|
}).where(and(
|
|
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
|
|
eq(heartbeatRuns.status, "running"),
|
|
eq(heartbeatRuns.controllerBootId, run.controllerBootId),
|
|
lte(heartbeatRuns.controllerLeaseExpiresAt, sql`clock_timestamp()`),
|
|
)).returning({ id: heartbeatRuns.id });
|
|
return Boolean(revoked);
|
|
}
|
|
|
|
/** Abort the adapter if the controller cannot renew. Bound each check by the
|
|
* lease duration even when the database connection never settles. */
|
|
export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortController) {
|
|
if (run.runtimeMode === "native" || !run.controllerBootId) {
|
|
return { stop() {}, async assertOwned(_stage?: "dispatching") {} };
|
|
}
|
|
let stopped = false;
|
|
let pending = false;
|
|
const lost = () => { if (!stopped) controller.abort(new Error("Legacy controller lease lost")); };
|
|
let deadline = setTimeout(lost, Math.max(0,
|
|
(run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now()));
|
|
deadline.unref();
|
|
const assertOwned = async (stage?: "dispatching") => {
|
|
if (stopped) return;
|
|
controller.signal.throwIfAborted();
|
|
const startedAt = Date.now();
|
|
let onAbort!: () => void;
|
|
const aborted = new Promise<never>((_, reject) => {
|
|
onAbort = () => reject(controller.signal.reason);
|
|
controller.signal.addEventListener("abort", onAbort, { once: true });
|
|
});
|
|
let renewed: boolean;
|
|
try {
|
|
renewed = await Promise.race([renewLegacyControllerLease(db, run, stage), aborted]);
|
|
} finally {
|
|
controller.signal.removeEventListener("abort", onAbort);
|
|
}
|
|
if (stopped) return;
|
|
if (!renewed) {
|
|
lost();
|
|
controller.signal.throwIfAborted();
|
|
}
|
|
controller.signal.throwIfAborted();
|
|
if (!stopped) {
|
|
clearTimeout(deadline);
|
|
deadline = setTimeout(lost, Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - startedAt)));
|
|
deadline.unref();
|
|
}
|
|
};
|
|
const timer = setInterval(() => {
|
|
if (pending || stopped) return;
|
|
pending = true;
|
|
void assertOwned().catch(lost).finally(() => { pending = false; });
|
|
}, LEGACY_CONTROLLER_RENEW_MS);
|
|
timer.unref();
|
|
return { assertOwned, stop() { stopped = true; clearInterval(timer); clearTimeout(deadline); } };
|
|
}
|