diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index a85a79356f5..bc9747378cd 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -1,36 +1,62 @@ # Zero-downtime web deployments -Date: 2026-10-31. Status (2026-11-01 14:05 UTC): existing-volume staging enrollment -is complete and its read/save/file-integrity checks pass. Active-agent recovery, -natural database idling and the ten-deploy staging campaign remain unqualified. -No production canary yet. +Date: 2026-10-31. Status (2026-11-01 14:54 UTC): existing-volume staging enrollment +and manual sleep/wake preserve saved files and read/save behavior. The full local +active-agent journey, natural database idling and ten-deploy staging campaign +remain unqualified. No production canary yet. -Latest critical path: Core #18543 includes runtime admission, immutable-cache -retirement, remote agent-file projection, and adoption without rematerializing -launch assets. The live test at `67714af9ff` transferred the same runner/session to -controller generation two after deleting A's entire home, then failed in the full -backend constructor: composing system instructions still read A's retired bundle. -All three owned Daytona sandboxes were independently confirmed absent; the child -credential was revoked and the test object bucket deleted. This remains failed -evidence, not a successful deployment qualification. +Core #18543 merged as `7751fc3107` after 55 successful CI checks, Greptile 5/5 +and no unresolved threads. It supplies runtime admission, immutable-cache +retirement, remote agent-file projection, and adoption using validated admitted +Runner instructions without rematerializing retired launch assets. The earlier +`67714af9ff` deleted-bundle failure is reproduced and fixed; the `cd690de4d3` +attempt was interrupted because the unrelated probe task blocked the business +all-tasks-done oracle. The probe now has a separate disposable company; the oracle +is unchanged. All test credentials, buckets and provider sandboxes were cleaned. -The backend now accepts admitted instructions from a validated same-run Runner -checkpoint. Runner owns checkpoint/template inspection; Core binds it to the -current run, normalized session and context digest. Staged attachments select the -validated target template. Fresh runs and ordinary later continuations still -compose current instructions. The deleted-bundle reproduction failed before the -fix, then passed with real backend session recovery and no new thread/turn start. -All 59 backend factory tests and the focused checkpoint validation tests pass. -At `cd690de4d3`, the real Codex/Daytona run survives removal of A's entire home, -reattaches the same runner/session under generation two, and succeeds. Its next -revision also succeeds. The handoff records 372 reads, 38 saves and 38 verified -saves with zero failures. Full qualification still fails: the supervisor created -an unrelated open probe task in the business company, so the unchanged all-tasks- -done oracle could never advance to the second replacement. The attempt was -interrupted and retained as incomplete. A regression reproduces this harness -scope defect; probes now use their own disposable company. The oracle is unchanged. -Exact-head review of the recovery fix is 5/5 with no unresolved threads; new -fixture checks and another bounded live attempt are next. +The complete retry at `f5915fdc33` exercised both replacements with 523 reads, +53 saves and 53 verified saves, zero probe failures. The original remote execution +survived with the same runner/session and completed once. Saved project tests, +file-reference continuity, and the later useful answer passed. The workflow still +FAILED: its later admission rotated native/provider sessions, and all four public +run records lacked the native terminal contract. These failures remain acceptance +blockers, not successful qualification. + +The next slice owns those gaps at existing boundaries: instruction authority +exposes the committed entry digest without reading a disposable cache; heartbeat +session compatibility identifies immutable skill versions rather than their host +paths; the native receipt transaction publishes its accepted terminal contract +without depending on the originating adapter returning. Both regressions failed +before the changes. All 249 affected workspace/session, durable-file and control- +plane persistence tests pass, as does server typecheck. Assembled admission suites +ran 59 passing cases, but native resumption teardown closed every client on the +shared database host, including its fixture's admin connection. This reproduces +on clean master; limiting teardown to the owned database makes all 16 native +resumption cases pass. PR #18552 has a 5/5 review for the initial fix. + +The next real attempt at `d2165dec5d` failed before B became ready: startup +recovery tried to lock a run still owned by A, hit a one-second PostgreSQL lock +timeout, and aborted startup. Its credential and bucket were cleaned; provider +cleanup is independently checked after the asynchronous stop/delete transition. +A three-case regression reproduces this for locked issue, run and coordinator +rows. The fix skips those locked rows, schedules the same concrete run through +the existing recovery retry, and avoids a diagnostic write that would wait on the +same lock. All 74 enabled restart recovery tests pass, including the assembled startup +caller; eight Docker-specific cases were not enabled in that run. Real content, skill-version, +model, credential and task boundaries remain compatibility checks. + +The real retry at `aae8a53d01` reached a different startup failure: the readiness +gate treated a run still preparing its skill selection as an unmigrated skill +installation. Saved rows show committed version-backed skills and one admitted +run whose exact pins had not yet been published. The new PostgreSQL regression +fails before the fix. Startup now checks installation durability independently +of active run pinning; maintenance eligibility, migration and cache eviction +keep their conservative pin requirements. Missing durable versions still refuse +readiness. All 33 readiness, managed-skill and durable-file tests pass, with server +typecheck. The failed attempt remains failed evidence; its child credential, +bucket and all three provider sandboxes were cleaned and checked. Review also +replaced the whole agent-file manifest read with a scalar SQL digest projection. +The next live attempt must exercise both replacements and later continuation. Review also identified save quotas incorrectly applied during cache recovery. Recovery uses the existing inbound remote projection and preserves path/link @@ -49,9 +75,9 @@ The original browser tab reconnected and persisted a task title. Server logs lat showed old Agents chunks returned 404 and navigation recovered via a page reload: this is migration continuity evidence, not uninterrupted old-tab asset loading. -Two eight-minute natural database-idle observations failed. A separate bounded SQL +Three eight-minute natural database-idle observations failed. A separate bounded SQL attribution window saw no other app-database sessions; it is diagnostic stimulus, -not idle proof, and does not establish the source of provider activity. Automatic +not idle proof, and does not establish the source of provider activity. The third window, bracketed by closed SQL counter snapshots, recorded 36 transactions and small read deltas on environment setup, leases, environments and plugins; no app-row writes. The source is not yet attributed. Automatic idle sleep was correctly refused because this development fixture is not eligible for the free/demo policy. Manual sleep with force disabled passed: the registry reports sleeping, Railway has no active web deployment or attached volume, and diff --git a/server/src/__tests__/agent-file-durability.test.ts b/server/src/__tests__/agent-file-durability.test.ts index 6348f2c60a7..7c79a267ebe 100644 --- a/server/src/__tests__/agent-file-durability.test.ts +++ b/server/src/__tests__/agent-file-durability.test.ts @@ -11,7 +11,7 @@ import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.j import { agentFileStore, adoptAgentFiles, fileHash } from "../services/agent-file-store.js"; import { invalidateAgentFileProjection, persistDurableAgentFiles } from "../services/agent-file-durability.js"; import { materializeInstructionBytes } from "../services/agent-instruction-files.js"; -import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js"; +import { agentInstructionsService, resolveManagedInstructionsRoot } from "../services/agent-instructions.js"; import { AGENT_FILE_OBJECT_GRACE_MS, collectAgentFileObjects, migrateAgentFiles } from "../services/agent-file-maintenance.js"; import { getStorageService } from "../storage/index.js"; @@ -89,6 +89,28 @@ describe("durable agent file authority", () => { root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "File owner", adapterConfig: {} }); } + it("reads instruction compatibility from the committed head after deleting the entire home", async () => { + await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Original")); + await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root, + instructionsFilePath: path.join(root, "AGENTS.md"), instructionsEntryFile: "AGENTS.md" } }).where(eq(agents.id, agentId)); + const [agent] = await db.select().from(agents).where(eq(agents.id, agentId)); + const instructions = agentInstructionsService(db); + expect(await instructions.getManagedEntryFingerprint(agent)).toBeNull(); + await store().read(companyId, agentId, "AGENTS.md", actor()); + const expected = { entryFile: "AGENTS.md", contentHash: `sha256:${fileHash(Buffer.from("Original"))}` }; + expect(await instructions.getManagedEntryFingerprint(agent)).toEqual(expected); + await replaceFilesystem(); + expect(await instructions.getManagedEntryFingerprint(agent)).toEqual(expected); + await expect(fs.stat(root)).rejects.toMatchObject({ code: "ENOENT" }); + expect(await instructions.getManagedEntryFingerprint({ ...agent, companyId: randomUUID() })).toBeNull(); + expect(await instructions.getManagedEntryFingerprint({ ...agent, adapterConfig: { + ...agent.adapterConfig, instructionsBundleMode: "external", + } })).toBeNull(); + expect(await instructions.getManagedEntryFingerprint({ ...agent, adapterConfig: { + ...agent.adapterConfig, instructionsEntryFile: "missing.md", + } })).toEqual({ entryFile: "missing.md", contentHash: null }); + }); + it("imports legacy bytes, modes and empty entries, then restores without any old filesystem", async () => { const binary = Buffer.alloc(2 * 1024 * 1024, 0x81); await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Original")); diff --git a/server/src/__tests__/ephemeral-server-data.test.ts b/server/src/__tests__/ephemeral-server-data.test.ts index 6c42c6de952..5ce5b67098d 100644 --- a/server/src/__tests__/ephemeral-server-data.test.ts +++ b/server/src/__tests__/ephemeral-server-data.test.ts @@ -1,9 +1,9 @@ import { randomUUID } from "node:crypto"; import { beforeAll, afterAll, afterEach, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; -import { agentFileRoots, agents, companies, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db"; +import { agentFileRoots, agents, companies, companySkills, companySkillVersions, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; -import { assertEphemeralServerData } from "../services/ephemeral-server-contract.js"; +import { assertEphemeralServerData, readEphemeralServerDataEligibility } from "../services/ephemeral-server-contract.js"; import { initializeDurableAgentFiles } from "../services/agent-file-durability.js"; vi.mock("../storage/index.js", () => ({ getStorageService: () => ({ provider: "s3" }) })); let database: Awaited>; @@ -40,3 +40,28 @@ it("refuses volume-dependent rows before readiness and accepts their durable rep })).rejects.toThrow("abort hire"); expect(await db.select().from(agentFileRoots)).toHaveLength(1); }); + +it("keeps replacement readiness independent of a serving controller's unfinished skill selection", async () => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true"); + const companyId = randomUUID(), agentId = randomUUID(), skillId = randomUUID(), versionId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Overlapping skill preparation", issuePrefix: "PINS" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Preparing agent" }); + await db.transaction(tx => initializeDurableAgentFiles(tx, { id: agentId, companyId })); + await db.insert(companySkills).values({ id: skillId, companyId, key: "overlap", slug: "overlap", name: "Overlap", + sourceType: "managed", markdown: "# Overlap", fileInventory: [{ path: "SKILL.md", kind: "entrypoint" }] }); + await db.insert(companySkillVersions).values({ id: versionId, companyId, companySkillId: skillId, revisionNumber: 1, + fileInventory: [{ path: "SKILL.md", kind: "entrypoint", content: "# Overlap" }] }); + await db.update(companySkills).set({ currentVersionId: versionId }).where(eq(companySkills.id, skillId)); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + // Admission marks the run running before preparation publishes its skill pins. + // That prevents cache eviction/migration, but does not make durable bytes local. + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", + status: "running", contextSnapshot: {} }).returning(); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + expect((await readEphemeralServerDataEligibility(db)).blockers).toContain("skills_require_migration"); + await db.update(heartbeatRuns).set({ contextSnapshot: { skillVersions: { overlap: versionId } } }).where(eq(heartbeatRuns.id, run.id)); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + expect((await readEphemeralServerDataEligibility(db)).eligible).toBe(true); + await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skillId)); + await expect(assertEphemeralServerData(db)).rejects.toThrow("skills_require_migration"); +}); diff --git a/server/src/__tests__/heartbeat-workspace-session.test.ts b/server/src/__tests__/heartbeat-workspace-session.test.ts index 154a7ff0600..62196235945 100644 --- a/server/src/__tests__/heartbeat-workspace-session.test.ts +++ b/server/src/__tests__/heartbeat-workspace-session.test.ts @@ -2703,6 +2703,31 @@ describe("effective run session config freshness", () => { expect(next.fingerprints.sessionFingerprint.canonicalJson).not.toContain("Version two instructions"); }); + it("retains a session across relocated durable instruction and versioned skill caches", async () => { + const metadata = (home: string, contentHash = "sha256:entry-v1", versionId = "skill-version-1") => { + const runtimeSkills = [{ key: "paperclip", runtimeName: "paperclip", versionId, + currentVersionId: versionId, source: `${home}/packages/skill`, sourceStatus: "available" }]; + return buildSessionConfigMetadata({ + adapterType: "paperclip_runner", + effectiveAdapterConfig: { model: "gpt-5.4-mini", instructionsBundleMode: "managed", + instructionsRootPath: `${home}/instructions`, instructionsFilePath: `${home}/instructions/AGENTS.md`, + instructionsEntryFile: "AGENTS.md", paperclipRuntimeSkills: runtimeSkills }, + managedInstructions: { entryFile: "AGENTS.md", contentHash }, + runtimeSkills, + }); + }; + const before = await metadata("/retired-controller"); + const after = await metadata("/replacement-controller"); + const freshness = (next: SessionConfigMetadata) => resolveTaskSessionConfigFreshness({ + hasTaskSession: true, configuredModel: "gpt-5.4-mini", + taskSessionParams: sessionParamsWithConfigMetadata(before), configMetadata: next, + }); + expect(freshness(after).reset).toBe(false); + expect(after.fingerprints.sessionFingerprint.canonicalJson).not.toContain("/replacement-controller"); + expect(freshness(await metadata("/replacement-controller", "sha256:entry-v2")).changedCategories).toContain("instructions"); + expect(freshness(await metadata("/replacement-controller", "sha256:entry-v1", "skill-version-2")).changedCategories).toContain("runtimeSkills"); + }); + it("does not read unbounded legacy instructions paths for config fingerprints", async () => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-session-fingerprint-")); const instructionsPath = path.join(root, "AGENTS.md"); diff --git a/server/src/__tests__/native-session-resumption.test.ts b/server/src/__tests__/native-session-resumption.test.ts index 47b8e01dfdc..264bc8327a3 100644 --- a/server/src/__tests__/native-session-resumption.test.ts +++ b/server/src/__tests__/native-session-resumption.test.ts @@ -1530,7 +1530,7 @@ describe("goal-less native task-session publication", () => { } }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { - await closeRegisteredClients(process.env.PAPERCLIP_TEST_DATABASE_URL ?? database!.connectionString); + await closeRegisteredClients(process.env.PAPERCLIP_TEST_DATABASE_URL ?? database!.connectionString, { scope: "database" }); await database?.cleanup(); }); diff --git a/server/src/services/agent-instructions.ts b/server/src/services/agent-instructions.ts index d86bd0c8d54..1afabe9d2e0 100644 --- a/server/src/services/agent-instructions.ts +++ b/server/src/services/agent-instructions.ts @@ -2,8 +2,8 @@ import { adoptAgentFiles, inspectAgentFile, fileHash, agentFilePath, MAX_AGENT_F import { invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles } from "./agent-file-durability.js"; import fs from "node:fs/promises"; import path from "node:path"; -import { and, eq } from "drizzle-orm"; -import { agentInstructionHeads, agents, fileStorageDb, type Db } from "@paperclipai/db"; +import { and, eq, sql } from "drizzle-orm"; +import { agentFileRoots, agentInstructionHeads, agents, fileStorageDb, type Db } from "@paperclipai/db"; import { instructionPath, assertInstructionPathSafe, instructionBytes, readInstructionBytes } from "./agent-instruction-files.js"; import { notFound, unprocessable } from "../errors.js"; import { resolveHomeAwarePath, resolvePaperclipInstanceRoot } from "../home-paths.js"; @@ -506,6 +506,18 @@ export function syncInstructionsBundleConfigFromFilePath( export function agentInstructionsService(db?: Db) { if (db) db = fileStorageDb(db); + async function getManagedEntryFingerprint(agent: AgentLike): Promise<{ entryFile: string; contentHash: string | null } | null> { + if (!db || agentInstructionsBundleMode(agent) !== "managed") return null; + const state = deriveBundleState(agent); + const [head] = await db.select({ sha256: sql`jsonb_path_query_first( + ${agentFileRoots.entries}, '$[*] ? (@.path == $entry && @.kind == "file")', + jsonb_build_object('entry', ${state.entryFile}::text)) ->> 'sha256'` }).from(agentFileRoots) + .where(and(eq(agentFileRoots.companyId, agent.companyId), eq(agentFileRoots.agentId, agent.id))); + if (!head) return null; + // This head is the file authority. A missing or retired local projection + // must not make identical instructions appear to change at admission. + return { entryFile: state.entryFile, contentHash: head.sha256 ? `sha256:${head.sha256}` : null }; + } async function assertUnversionedEntry(agent: AgentLike, entryFile: string, connection: Db | Parameters[0]>[0] | undefined = db) { if (!connection) throw unprocessable("Bundle initialization requires the database-backed instructions service"); const [head] = await connection.select().from(agentInstructionHeads).where(and(eq(agentInstructionHeads.companyId, agent.companyId), @@ -871,6 +883,7 @@ export function agentInstructionsService(db?: Db) { } return { + getManagedEntryFingerprint, getBundleLocation: (agent: AgentLike) => withProjection(agent, false, getBundleLocation), getBundle: (agent: AgentLike) => withProjection(agent, false, getBundle), readFile: (agent: AgentLike, relativePath: string) => withProjection(agent, false, current => readFile(current, relativePath)), diff --git a/server/src/services/ephemeral-server-contract.ts b/server/src/services/ephemeral-server-contract.ts index f210837f954..396885b3f57 100644 --- a/server/src/services/ephemeral-server-contract.ts +++ b/server/src/services/ephemeral-server-contract.ts @@ -37,11 +37,11 @@ export function assertEphemeralServerConfig(config: Pick skill.status !== "version_backed")) { blockers.push("skills_require_migration"); break; } if (page.inspected < 50 || !page.after) break; after = page.after; diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index bc74f44a984..c72ff89214b 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1012,6 +1012,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD if (disposition.kind === "awaiting_evidence" && disposition.retryAt) { scheduleNativeSessionResumeDispatch(disposition.runId, disposition.retryAt, "restart"); } + // No ownership decision was made for a locked row. Writing an event + // would wait on that same row and defeat the non-blocking startup scan. + if (disposition.kind === "awaiting_evidence" && disposition.reason === "recovery_rows_busy") continue; const run = await getRun(disposition.runId); if (run) { const isClaim = diff --git a/server/src/services/heartbeat/session-preparation.ts b/server/src/services/heartbeat/session-preparation.ts index 0179e26c90a..832c5ebcefd 100644 --- a/server/src/services/heartbeat/session-preparation.ts +++ b/server/src/services/heartbeat/session-preparation.ts @@ -5,6 +5,7 @@ import { getServerAdapter } from "../../adapters/index.js"; import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js"; import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "../ai-connection-session.js"; import { createNativeSessionHandoffLoader } from "../native-runtime/native-session-handoff.js"; +import { agentInstructionsService } from "../agent-instructions.js"; import { describeSessionResetReason, shouldResetTaskSessionForWake, isCanonicalSessionIdForAdapter, normalizeResumeParamsForAdapter, normalizeSessionParams, requiresCanonicalSessionIds, @@ -87,6 +88,7 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre agentIdentityKeyId: agentIdentity?.keyId, adapterType: agent.adapterType, effectiveAdapterConfig: runtimeConfig, + managedInstructions: await agentInstructionsService(db).getManagedEntryFingerprint({ ...agent, adapterConfig: runtimeConfig }), managedAiHome: managedAiRuntime?.home, agentRuntimeConfig: agent.runtimeConfig, issueOverrides: issueAssigneeOverrides, diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 6b036657a8c..9e8c39c9cab 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -2838,6 +2838,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: { adapterType: string; effectiveAdapterConfig: Record; managedAiHome?: string; + managedInstructions?: { entryFile: string; contentHash: string | null } | null; agentRuntimeConfig: unknown; issueOverrides: unknown; workspaceConfig: unknown; @@ -2851,12 +2852,30 @@ export async function buildEffectiveRunSessionConfigMetadata(input: { agentIdentityKeyId?: string; }): Promise { const secretManifest = input.secretManifest ?? []; - const instructions = await resolveInstructionsConfigFingerprintMetadata( - input.effectiveAdapterConfig, - ); + const instructions = input.managedInstructions ? { + configured: true, bundleMode: "managed", entryFile: input.managedInstructions.entryFile, + pathKind: "absolute", readPolicy: "root_bound", readable: input.managedInstructions.contentHash !== null, + ...(input.managedInstructions.contentHash !== null ? { contentHash: input.managedInstructions.contentHash } : {}), + } : await resolveInstructionsConfigFingerprintMetadata(input.effectiveAdapterConfig); + const effectiveAdapterConfig = { ...managedAiSessionFingerprintConfig(input.effectiveAdapterConfig, input.managedAiHome) }; + if (input.managedInstructions) { + delete effectiveAdapterConfig.instructionsRootPath; + delete effectiveAdapterConfig.instructionsFilePath; + } + // A selected immutable skill version is its identity; source is merely its + // disposable projection. Unversioned/local skills retain their path boundary. + const versionedSkills = (value: unknown) => Array.isArray(value) ? value.map(entry => { + const skill = parseObject(entry); + return readNonEmptyString(skill.versionId) + ? { key: skill.key, runtimeName: skill.runtimeName, versionId: skill.versionId } + : entry; + }) : value; + if (effectiveAdapterConfig.paperclipRuntimeSkills !== undefined) { + effectiveAdapterConfig.paperclipRuntimeSkills = versionedSkills(effectiveAdapterConfig.paperclipRuntimeSkills); + } const categoryValues = buildSessionConfigCategoryValues({ adapterType: input.adapterType, - effectiveAdapterConfig: managedAiSessionFingerprintConfig(input.effectiveAdapterConfig, input.managedAiHome), + effectiveAdapterConfig, agentRuntimeConfig: input.agentRuntimeConfig, instructions, issueOverrides: input.issueOverrides, @@ -2866,7 +2885,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: { projectEnv: input.projectEnv, routineEnv: input.routineEnv, secretManifest, - runtimeSkills: input.runtimeSkills, + runtimeSkills: versionedSkills(input.runtimeSkills), agentConfigRevision: input.agentConfigRevision ?? null, agentIdentityKeyId: input.agentIdentityKeyId, }); diff --git a/server/src/services/native-runtime/native-restart-recovery.ts b/server/src/services/native-runtime/native-restart-recovery.ts index af9878f9907..96a4112e5a6 100644 --- a/server/src/services/native-runtime/native-restart-recovery.ts +++ b/server/src/services/native-runtime/native-restart-recovery.ts @@ -560,11 +560,17 @@ export async function claimNativeRestartRecoveries(input: { await tx.execute( sql`select set_config('statement_timeout', '15000', true), set_config('lock_timeout', '1000', true)`, ); - await tx + const busy = () => ({ kind: "awaiting_evidence" as const, runId: candidate.runId, + reason: "recovery_rows_busy", retryAt: new Date(now.getTime() + 1_000) }); + // During overlap the serving controller can be committing this run. + // Defer that concrete recovery obligation instead of failing startup + // or waiting while holding a lock the serving controller needs. + const [issue] = await tx .select({ id: issues.id }) .from(issues) .where(eq(issues.id, candidate.issueId)) - .for("update"); + .for("update", { skipLocked: true }); + if (!issue) return busy(); const row = await tx .select({ run: heartbeatRuns, @@ -586,15 +592,11 @@ export async function claimNativeRestartRecoveries(input: { ), ) .where(eq(heartbeatRuns.id, candidate.runId)) - .for("update") + .for("update", { skipLocked: true }) .limit(1) .then((rows) => rows[0] ?? null); if (!row) { - return { - kind: "blocked", - runId: candidate.runId, - reason: "recovery_rows_missing", - } as const; + return busy(); } if (isNativeRunnerOwnershipHeld(row.run)) { return { diff --git a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts index 6711939a601..82b90639a47 100644 --- a/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts +++ b/server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts @@ -36,6 +36,7 @@ import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, } from "@paperclipai/db"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { heartbeatService } from "../heartbeat.js"; import { canonicalNativeRuntimeContextDigest, @@ -322,6 +323,41 @@ describeEmbeddedPostgres("native runner restart recovery with real processes", ( }; } + it.each(["issue", "run", "coordinator"] as const)("defers a busy %s row without blocking replacement startup", async kind => { + const fixture = await seedRun(`OVERLAP-BUSY-${kind}`); + let acquired!: () => void, release!: () => void; + const locked = new Promise(resolve => { acquired = resolve; }); + const released = new Promise(resolve => { release = resolve; }); + const holder = fixture.db.transaction(async tx => { + if (kind === "issue") await tx.select().from(issues).where(eq(issues.id, fixture.issueId)).for("update"); + else if (kind === "run") await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId)).for("update"); + else await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)).for("update"); + acquired(); await released; + }); + await locked; + const now = new Date(); + const heartbeat = heartbeatService(fixture.db); + try { + const dispositions = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId], now }); + expect(dispositions).toEqual([{ kind: "awaiting_evidence", runId: fixture.runId, + reason: "recovery_rows_busy", retryAt: new Date(now.getTime() + 1_000) }]); + const [coordinator] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId)); + expect(coordinator.controllerGeneration).toBe(0); + let deadline: ReturnType | undefined; + try { + const startup = await Promise.race([ + heartbeat.recoverNativeRunsAfterRestart(now, [fixture.runId]), + new Promise((_resolve, reject) => { deadline = setTimeout(() => reject(new Error("startup blocked on the serving controller")), 2_000); }), + ]); + expect(startup).toMatchObject({ claims: [], awaitingEvidenceRunIds: [fixture.runId] }); + } finally { clearTimeout(deadline); } + } finally { await heartbeat.drainActiveRunExecutions(); release(); await holder; } + const [next] = await claimNativeRestartRecoveries({ db: fixture.db, controller: successor, + restartKind: "hard", runIds: [fixture.runId] }); + expect(next.kind).toBe("bootstrap_incomplete"); + }); + it.skipIf(!process.env.PAPERCLIP_TEST_RUNNER_CONTAINER).each(["warm", "hard", "legacy", "changed", "same-server", "diskless", "diskless-per-turn", "diskless-per-turn-read-retry"])("preserves remote warm continuity across completed turns: %s", async (scenario) => { const hardRestart = !["warm", "same-server"].includes(scenario); const diskless = scenario.startsWith("diskless"); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index ec45dca6437..92aefdd9ad7 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -733,6 +733,10 @@ describe("PaperclipControlPlanePort conformance", () => { .where(eq(nativeRunFinalizations.runId, identity.runId))).resolves.toEqual([ expect.objectContaining({ phase: "committed" }), ]); + // Completion and restart reconciliation must retain the contract even if + // the originating heartbeat never returns from its adapter completion. + const [completedRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, identity.runId)); + expect(completedRun.resultJson?.nativeTerminal).toEqual(CONTROL_PLANE_CONFORMANCE_TERMINAL); await expect(db.select().from(issues).where(eq(issues.id, identity.issueId))).resolves.toEqual([ expect.objectContaining({ status: "in_progress", statusVersion: 1 }), ]); diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.ts b/server/src/services/native-runtime/paperclip-control-plane-port.ts index f7bac1622d3..4e85a8ef843 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.ts @@ -568,6 +568,9 @@ export class PaperclipControlPlanePort implements ControlPlanePort { nativePhaseUpdatedAt: new Date(), resultJson: { ...record(run.resultJson), + // Publish the accepted contract with its durable receipt. Recovery + // may settle this run without the original adapter ever returning. + nativeTerminal: value.terminal, prpTurnTerminalState: value.terminal.turnTerminalState, prpRunTerminalState: value.terminal.runTerminalState, prpReportedWorkDisposition: value.terminal.reportedWorkDisposition, diff --git a/server/src/services/skill-installation-maintenance.ts b/server/src/services/skill-installation-maintenance.ts index d283a42af34..9b9f7218ac0 100644 --- a/server/src/services/skill-installation-maintenance.ts +++ b/server/src/services/skill-installation-maintenance.ts @@ -158,13 +158,16 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { return { inspected: skills.length, after: skills.at(-1)?.id ?? null, results }; } - async function eligibility(options: Page = {}) { + async function eligibility(options: Page & { purpose?: "migration" | "startup" } = {}) { const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options)); const results = []; for (const skill of skills) { - const pins = await protectedVersions(db, skill.companyId); + // Migration/eviction must wait for exact pins. Replacement readiness only + // inspects installation durability: a serving controller may legitimately + // be between run admission and publishing its immutable skill selection. + const pins = options.purpose === "startup" ? undefined : await protectedVersions(db, skill.companyId); let status = linkedLocal(skill) ? "linked_local" : legacyManaged(skill) ? "legacy_managed" : "version_backed"; - if (pins.versions === null) status = "unsupported_active_run"; + if (pins?.versions === null) status = "unsupported_active_run"; else if (!linkedLocal(skill) && !skill.currentVersionId) status = "missing_current_version"; else if (skill.sourceType === "catalog" && !skill.metadata?.originVersionId) status = "legacy_catalog_origin"; else if (status === "version_backed" && skill.currentVersionId) {