mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description **Bug description** After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. **Expected behavior** - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
87 lines
5.9 KiB
TypeScript
87 lines
5.9 KiB
TypeScript
import { and, eq, isNotNull, isNull, sql } from "drizzle-orm";
|
|
import { agentFileRoots, agents, executionWorkspaces, heartbeatRuns, providerTraceRecords, type Db } from "@paperclipai/db";
|
|
import type { Config } from "../config.js";
|
|
import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js";
|
|
import { agentInstructionsBundleMode } from "./agent-instructions.js";
|
|
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
|
|
import { hasLocalNativeHarnessObligations } from "./native-runtime/native-harness-backup-migration.js";
|
|
import { hasLocalNativeControllerObligations } from "./native-runtime/native-controller-migration.js";
|
|
import { hasUnsettledLocalInstructionCopies } from "./agent-instruction-copy-preservation.js";
|
|
import { hasLocalRuntimeSkillObligations } from "./native-runtime/runtime-skill-preservation.js";
|
|
|
|
import { ephemeralServerEnabled } from "./ephemeral-server-mode.js";
|
|
|
|
export { ephemeralServerEnabled };
|
|
|
|
/** Validate before consumers can generate a replacement local identity. Error
|
|
* messages name settings only, never credential values or connection URLs. */
|
|
export function assertEphemeralServerConfig(config: Pick<Config, "databaseUrl" | "storageProvider">, env = process.env): string | null {
|
|
if (env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return null;
|
|
const refuse = (reason: string): never => { throw new Error(`Volume-free server configuration: ${reason}`); };
|
|
if (config.storageProvider !== "s3" || env.PAPERCLIP_DURABLE_AGENT_FILES !== "true" || env.PAPERCLIP_SHARED_UI_ASSETS !== "true") {
|
|
refuse("shared S3 storage, durable agent files and shared UI assets are required");
|
|
}
|
|
if (!env.PAPERCLIP_SECRETS_MASTER_KEY || !isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY)) {
|
|
refuse("a stable PAPERCLIP_SECRETS_MASTER_KEY is required");
|
|
}
|
|
if ((env.PAPERCLIP_DECISION_SIGNING_SECRET?.trim().length ?? 0) < 32) {
|
|
refuse("a stable PAPERCLIP_DECISION_SIGNING_SECRET of at least 32 characters is required; preserve the existing decision key during migration");
|
|
}
|
|
let runtime: URL, listener: URL;
|
|
try { runtime = new URL(config.databaseUrl ?? ""); listener = new URL(env.PAPERCLIP_DATABASE_LISTEN_URL ?? ""); }
|
|
catch { return refuse("external DATABASE_URL and PAPERCLIP_DATABASE_LISTEN_URL are required"); }
|
|
if (![runtime, listener].every(url => ["postgres:", "postgresql:"].includes(url.protocol))
|
|
|| runtime.username !== listener.username || runtime.password !== listener.password || runtime.pathname !== listener.pathname
|
|
|| runtime.hostname.replace(/-pooler(?=\.)/, "") !== listener.hostname
|
|
|| /pooler|pgbouncer/i.test(listener.hostname)) {
|
|
refuse("PAPERCLIP_DATABASE_LISTEN_URL must be a direct connection to the same database using the runtime role");
|
|
}
|
|
return listener.href;
|
|
}
|
|
|
|
/** A readiness gate, not a periodic scan or an implicit migration. Existing
|
|
* volume data is adopted by the owning maintenance commands before removal. */
|
|
export async function assertEphemeralServerData(db: Db) {
|
|
if (!ephemeralServerEnabled()) return;
|
|
const { blockers } = await readEphemeralServerDataEligibility(db, { purpose: "startup" });
|
|
if (blockers.length) throw new Error(`Volume-free server data is not eligible: ${blockers.join(", ")}`);
|
|
}
|
|
|
|
export async function readEphemeralServerDataEligibility(db: Db, options: { purpose?: "migration" | "startup" } = {}) {
|
|
const blockers: string[] = [];
|
|
const [missingFiles] = await db.select({ id: agents.id }).from(agents).leftJoin(agentFileRoots,
|
|
and(eq(agentFileRoots.companyId, agents.companyId), eq(agentFileRoots.agentId, agents.id)))
|
|
.where(isNull(agentFileRoots.agentId)).limit(1);
|
|
if (missingFiles) blockers.push("agent_files_require_migration");
|
|
if (await hasUnsettledLocalInstructionCopies(db)) blockers.push("local_instruction_copies_require_settlement");
|
|
// Only inspect configuration here; filesystem reads must never adopt an
|
|
// empty directory on the replacement machine as the old server's content.
|
|
const configured = await db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, adapterConfig: agents.adapterConfig }).from(agents);
|
|
if (configured.some(agent => agentInstructionsBundleMode(agent) === "external")) blockers.push("external_agent_instructions");
|
|
const [localWorkspace] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
|
|
isNotNull(executionWorkspaces.cwd), sql`${executionWorkspaces.providerType} <> 'plugin'`,
|
|
sql`coalesce(${executionWorkspaces.metadata}->'remoteWorkspace'->>'schema', '') <> 'paperclip.remote-workspace.v1'`,
|
|
)).limit(1);
|
|
if (localWorkspace) blockers.push("workspaces_require_migration");
|
|
const [localLog] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.logStore, "local_file"), isNotNull(heartbeatRuns.logRef),
|
|
)).limit(1);
|
|
if (localLog) blockers.push("run_logs_require_migration");
|
|
const [localTrace] = await db.select({ id: providerTraceRecords.id }).from(providerTraceRecords).where(and(
|
|
isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`,
|
|
)).limit(1);
|
|
if (localTrace) blockers.push("provider_traces_require_migration");
|
|
if (await hasLocalNativeHarnessObligations(db)) blockers.push("native_harness_backups_require_migration");
|
|
if (await hasLocalNativeControllerObligations(db)) blockers.push("native_controllers_require_migration");
|
|
if (await hasLocalRuntimeSkillObligations(db)) blockers.push("runtime_skill_assets_require_migration");
|
|
const skills = skillInstallationMaintenance(db);
|
|
let after: string | undefined;
|
|
for (;;) {
|
|
const page = await skills.eligibility({ after, limit: 50, purpose: options.purpose });
|
|
if (page.results.some(skill => skill.status !== "version_backed")) { blockers.push("skills_require_migration"); break; }
|
|
if (page.inspected < 50 || !page.after) break;
|
|
after = page.after;
|
|
}
|
|
return { eligible: blockers.length === 0, blockers };
|
|
}
|