Files
PaperClipAI/server/src/services/ephemeral-server-contract.ts
T
DottaandPaperclip c0550e7010 fix(server): preserve sessions and defer busy overlap recovery (#18552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server replacement must preserve remote execution and later
task continuation.
> - The active-run handoff now survives deletion of the departing server
home.
> - The next admission still compared disposable instruction and skill
paths as configuration.
> - A native receipt could also settle without the old adapter returning
its terminal metadata.
> - A replacement can also collide with a run row locked by the serving
controller.
> - This change preserves durable session identity and terminal
evidence, and defers locked recovery work so overlap can reach
readiness.

## Linked Issues or Issue Description

Refs #18543, #18366, #9977, #18118.

**What happened?**

A real Codex/Daytona journey replaced both control-plane servers and
deleted their homes. Reads, saves, saved files and active execution
passed. A later useful answer used a new provider session. All four run
records lacked the native terminal contract. The whole qualification
therefore failed. A later attempt also exposed startup failure: the
replacement waited for a row lock held by the serving controller and
aborted on its one-second lock timeout.

**Expected behavior**

An unchanged managed instruction entry and immutable skill version
retain the session across server cache relocation. Actual content and
configuration changes still reset it. A committed native receipt retains
its terminal contract even when the originating adapter never returns. A
busy recovery row defers through the existing scheduler while the
replacement reaches readiness.

**Steps to reproduce**

Run the `rolling-web-deployment.runner-codex.daytona.recover-controller`
Product E2E cell with separate controller homes and shared
PostgreSQL/S3. The focused regressions reproduce the failures without a
model call.

## What Changed

- Read the managed instruction entry digest from the company-scoped
durable file head. Do not restore or read the cache for this check.
- Omit disposable instruction paths only when that durable authority
exists. Fingerprint selected immutable skill versions in both runtime
skill inputs.
- Preserve the existing local and external instruction path checks, and
unversioned skill identity.
- Publish the accepted native terminal contract in the same transaction
as the canonical result receipt.
- Skip locked recovery rows and schedule the same run for retry. Avoid
an audit write that would block on the same row. No ownership is
transferred while busy.
- Keep replacement readiness independent of an in-progress skill
selection; migration and cache eviction still require exact pins.
Missing durable skill versions still refuse startup.
- Project only the instruction digest in SQL instead of loading the
entire agent-file manifest per admission.
- Add real PostgreSQL lock contention tests for the issue, run and
coordinator, including the heartbeat startup caller and later claim.
- Limit native test teardown to its owned database on a shared test
host. The old behavior also fails on clean master.
- Add cache-loss, scope, changed-content, changed-version and
finalization regressions. Update the deployment qualification record.

## Verification

- Both focused regressions failed before the source changes.
- All 249 tests pass across `heartbeat-workspace-session`,
`agent-file-durability`, and `paperclip-control-plane-port`.
- Server typecheck and exact-revision build pass.
- Assembled admission ran 59 passing cases but failed during
native-suite teardown. The teardown failure reproduced on clean master.
All 16 native resumption cases and teardown pass after the
database-scope fix.
- All three row-lock regressions fail with the pre-fix lock timeout and
pass with deferred recovery. The full recovery suite passes 74 cases,
including the startup caller; eight Docker-specific cases were not
enabled.
- The real attempt at `d2165dec5d` failed when B aborted on the run
lock. Its credential and bucket were removed; all three provider
sandboxes were independently confirmed absent. The next real attempt at
`aae8a53d01` failed a different readiness gate while a run was preparing
skill pins. That race now has a failing-before regression; all 33
readiness, durable-file and managed-skill tests pass after the fix. Its
owned resources are cleaned. The `a15a12651c` live attempt passed both
web replacements: 765 reads, 78 saves and 78 verified saves with zero
HTTP probe failures. The first three turns preserved the provider
session and native terminal contract. The fourth continuation failed
because a safely suspended remote Runner was treated as live/idle; that
distinct recovery gap is being handled in a follow-up. All three owned
sandboxes were independently confirmed absent. No claim of full Railway
qualification yet.

## Risks

- The new fingerprint shape may reset a session once when upgrading from
older fingerprint metadata. It is stable across subsequent cache
replacements.
- A versioned skill is identified by its immutable selected version;
materialization and admission still enforce availability. Unversioned
skills retain their existing path boundary.
- A native terminal describes the accepted execution result. Workspace
and issue settlement remain separate authorities.
- This does not yet prove the ten-deploy staging campaign, natural
database autosuspension or the production canary.

## Model Used

OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 09:14:34 -06:00

77 lines
5.1 KiB
TypeScript

import { and, eq, isNotNull, isNull, sql } from "drizzle-orm";
import { agentFileRoots, agents, executionWorkspaces, heartbeatRuns, providerTraceRecords, type Db } from "@paperclipai/db";
import type { Config } from "../config.js";
import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js";
import { agentInstructionsBundleMode } from "./agent-instructions.js";
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true";
/** Validate before consumers can generate a replacement local identity. Error
* messages name settings only, never credential values or connection URLs. */
export function assertEphemeralServerConfig(config: Pick<Config, "databaseUrl" | "storageProvider">, env = process.env): string | null {
if (env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return null;
const refuse = (reason: string): never => { throw new Error(`Volume-free server configuration: ${reason}`); };
if (config.storageProvider !== "s3" || env.PAPERCLIP_DURABLE_AGENT_FILES !== "true" || env.PAPERCLIP_SHARED_UI_ASSETS !== "true") {
refuse("shared S3 storage, durable agent files and shared UI assets are required");
}
if (!env.PAPERCLIP_SECRETS_MASTER_KEY || !isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY)) {
refuse("a stable PAPERCLIP_SECRETS_MASTER_KEY is required");
}
if ((env.PAPERCLIP_DECISION_SIGNING_SECRET?.trim().length ?? 0) < 32) {
refuse("a stable PAPERCLIP_DECISION_SIGNING_SECRET of at least 32 characters is required; preserve the existing decision key during migration");
}
let runtime: URL, listener: URL;
try { runtime = new URL(config.databaseUrl ?? ""); listener = new URL(env.PAPERCLIP_DATABASE_LISTEN_URL ?? ""); }
catch { return refuse("external DATABASE_URL and PAPERCLIP_DATABASE_LISTEN_URL are required"); }
if (![runtime, listener].every(url => ["postgres:", "postgresql:"].includes(url.protocol))
|| runtime.username !== listener.username || runtime.password !== listener.password || runtime.pathname !== listener.pathname
|| runtime.hostname.replace(/-pooler(?=\.)/, "") !== listener.hostname
|| /pooler|pgbouncer/i.test(listener.hostname)) {
refuse("PAPERCLIP_DATABASE_LISTEN_URL must be a direct connection to the same database using the runtime role");
}
return listener.href;
}
/** A readiness gate, not a periodic scan or an implicit migration. Existing
* volume data is adopted by the owning maintenance commands before removal. */
export async function assertEphemeralServerData(db: Db) {
if (!ephemeralServerEnabled()) return;
const { blockers } = await readEphemeralServerDataEligibility(db, { purpose: "startup" });
if (blockers.length) throw new Error(`Volume-free server data is not eligible: ${blockers.join(", ")}`);
}
export async function readEphemeralServerDataEligibility(db: Db, options: { purpose?: "migration" | "startup" } = {}) {
const blockers: string[] = [];
const [missingFiles] = await db.select({ id: agents.id }).from(agents).leftJoin(agentFileRoots,
and(eq(agentFileRoots.companyId, agents.companyId), eq(agentFileRoots.agentId, agents.id)))
.where(isNull(agentFileRoots.agentId)).limit(1);
if (missingFiles) blockers.push("agent_files_require_migration");
// Only inspect configuration here; filesystem reads must never adopt an
// empty directory on the replacement machine as the old server's content.
const configured = await db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, adapterConfig: agents.adapterConfig }).from(agents);
if (configured.some(agent => agentInstructionsBundleMode(agent) === "external")) blockers.push("external_agent_instructions");
const [localWorkspace] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
isNotNull(executionWorkspaces.cwd), sql`${executionWorkspaces.providerType} <> 'plugin'`,
sql`coalesce(${executionWorkspaces.metadata}->'remoteWorkspace'->>'schema', '') <> 'paperclip.remote-workspace.v1'`,
)).limit(1);
if (localWorkspace) blockers.push("workspaces_require_migration");
const [localLog] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.logStore, "local_file"), isNotNull(heartbeatRuns.logRef),
)).limit(1);
if (localLog) blockers.push("run_logs_require_migration");
const [localTrace] = await db.select({ id: providerTraceRecords.id }).from(providerTraceRecords).where(and(
isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`,
)).limit(1);
if (localTrace) blockers.push("provider_traces_require_migration");
const skills = skillInstallationMaintenance(db);
let after: string | undefined;
for (;;) {
const page = await skills.eligibility({ after, limit: 50, purpose: options.purpose });
if (page.results.some(skill => skill.status !== "version_backed")) { blockers.push("skills_require_migration"); break; }
if (page.inspected < 50 || !page.after) break;
after = page.after;
}
return { eligible: blockers.length === 0, blockers };
}