mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 19:35:04 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server can preserve eligible agent runs during a controlled hot restart. > - A path change moved restart state from the Paperclip home root to the instance root. > - A staged update can therefore make the old server and the new server read different intent files. > - The old server then drains live runs, while the new server can start without a shutdown snapshot. > - This pull request adds a correlated compatibility handoff and records the live preflight set. > - It also verifies the target process instance on Linux, macOS, and Windows. > - The benefit is complete and safe run classification across the path upgrade. ## Linked Issues or Issue Description No public GitHub issue covers this defect. **What happened?** A staged hot restart can run an older server that reads `hot-restart-intent.json` from the Paperclip home root and a new server that writes the file under the instance root. The old server misses the request and uses graceful drain. The new server later finds its marker without a shutdown snapshot. Before this change, that state could produce an empty loss list even when live runs existed before restart. **Expected behavior** The old server must receive the PID-targeted restart request at its legacy path. The new server must correlate the legacy shutdown snapshot with its instance-scoped request. Every run that was live during preflight must appear as adopted, finalized while down, or lost. A reused PID must not let a stale marker claim a different process instance. **Steps to reproduce** 1. Start a server version from before the instance-root marker change. 2. Keep one or more local-agent heartbeat runs active. 3. Stage a current build and request a hot restart from that build. 4. Observe that the old server reads only the home-root path while the staged build writes only the instance-root path. 5. Observe graceful drain and a new-server intent that has no shutdown snapshot. **Paperclip version or commit** The path transition entered `master` in #10045. The hot-restart adoption flow came from #9647. This fix targets current `master` and compatibility with the immediately preceding home-root behavior. **Deployment mode** Self-hosted server built from source with controlled service hot restarts. Related work: #9628 is the original broader hot-restart feature PR. #10556 addresses embedded PostgreSQL lifecycle behavior and does not address marker-path compatibility. ## What Changed - Write an authoritative instance-scoped intent and a correlated legacy home-root handoff marker. - Merge a legacy shutdown snapshot only when immutable request identity fields match. - Prevent a non-default instance from consuming an uncorrelated legacy-only marker. - Record preflight running heartbeat IDs and reconcile snapshot omissions from current database state. - Serialize marker claims, snapshot writes, stale recovery, and matching cleanup with recoverable per-path filesystem leases. - Read process start identity on Linux, macOS, and Windows to distinguish a reused PID from the original server. - Require identity for new restart requests and fail closed when a supported platform cannot provide it. - Classify older markers by comparing the replacement server boot time or operating-system process start time with the request time. - Close the preflight database client explicitly and use a root-safe SQL query. - Add focused unit, platform-branch, database-backed, and CI regression coverage. - Document the compatibility handoff, process identity probes, and instance-scoped report path. ## Verification - `pnpm exec vitest run server/src/services/hot-restart.test.ts server/src/__tests__/heartbeat-process-recovery.test.ts -t "hot-restart|old-server legacy|preflight live|preflight run|spawn identity before hot restart"` — 24 tests passed and 90 tests were skipped across 2 files. - `pnpm exec vitest run server/src/services/hot-restart.test.ts` — 17 tests passed. - `pnpm exec vitest run server/src/__tests__/issue-watchdogs-routes.test.ts -t "restarts a stalled claimed run"` — 1 test passed and 10 tests were skipped. - `pnpm exec vitest run server/src/__tests__/agent-action-audit-routes.test.ts -t "allows an agent with issue:delegate"` — 1 test passed and 7 tests were skipped. - `pnpm --filter @paperclipai/server typecheck` — passed. - `git diff --check` — passed. - GitHub Actions — 26 of 26 checks passed at `55a79cb029be8b1dc89926d9d89ccd2181266d5c`. - Greptile — 5/5 at the same head with no unresolved current-head review threads. ## Risks - The legacy handoff path is shared across instances. Exclusive claims and per-path leases prevent overwrite and match-before-delete races. - Process identity uses platform commands as a fallback when the health endpoint has no identity. Linux reads `/proc`, macOS and BSD use `ps`, and Windows uses PowerShell. - A supported-platform identity probe failure aborts the restart. This fails closed instead of replacing an unknown live process. - Older intent files do not contain process identity. The server compares the replacement boot or process start time with the request time when those values are available. - A preflight database read can fail before the marker is written. The command fails closed instead of claiming a restart whose live-run set is unknown. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5. The exact deployment model ID and context-window size were not exposed by this runtime. Reasoning, repository editing, shell execution, test execution, GitHub CLI, and Paperclip API capabilities were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
121 lines
3.9 KiB
JavaScript
121 lines
3.9 KiB
JavaScript
#!/usr/bin/env -S node --import tsx
|
|
import { createDb } from "../packages/db/src/index.js";
|
|
import { loadConfig } from "../server/src/config.js";
|
|
import {
|
|
resolveHotRestartIntentPath,
|
|
writeHotRestartIntent,
|
|
} from "../server/src/services/hot-restart.js";
|
|
|
|
function usage(): never {
|
|
console.error([
|
|
"Usage: tsx scripts/request-hot-restart.ts --server-pid <pid> [--drain-required]",
|
|
"",
|
|
"Writes an instance-scoped hot-restart intent plus a legacy home-root handoff marker.",
|
|
].join("\n"));
|
|
process.exit(2);
|
|
}
|
|
|
|
function readArgs(argv: string[]) {
|
|
let serverPid: number | null = null;
|
|
let drainRequired = false;
|
|
|
|
for (let index = 0; index < argv.length; index += 1) {
|
|
const arg = argv[index];
|
|
if (arg === "--server-pid") {
|
|
const raw = argv[index + 1];
|
|
if (!raw) usage();
|
|
const parsed = Number(raw);
|
|
if (!Number.isInteger(parsed) || parsed <= 0) usage();
|
|
serverPid = parsed;
|
|
index += 1;
|
|
continue;
|
|
}
|
|
if (arg === "--drain-required") {
|
|
drainRequired = true;
|
|
continue;
|
|
}
|
|
if (arg === "--help" || arg === "-h") usage();
|
|
console.error(`Unknown argument: ${arg}`);
|
|
usage();
|
|
}
|
|
|
|
if (!serverPid) usage();
|
|
return { serverPid, drainRequired };
|
|
}
|
|
|
|
function normalizeApiBase(raw: string | undefined) {
|
|
const trimmed = raw?.trim();
|
|
if (!trimmed) return null;
|
|
return trimmed.replace(/\/+$/, "").replace(/\/api$/, "");
|
|
}
|
|
|
|
async function readPreviousServerInfo() {
|
|
const apiBase = normalizeApiBase(process.env.PAPERCLIP_API_URL);
|
|
if (!apiBase) return { version: null, identity: null };
|
|
try {
|
|
const apiKey = process.env.PAPERCLIP_API_KEY?.trim();
|
|
const response = await fetch(`${apiBase}/api/health`, {
|
|
headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : undefined,
|
|
signal: AbortSignal.timeout(2_000),
|
|
});
|
|
if (!response.ok) return { version: null, identity: null };
|
|
const body = await response.json() as Record<string, unknown>;
|
|
const serverInfo = body.serverInfo && typeof body.serverInfo === "object"
|
|
? body.serverInfo as Record<string, unknown>
|
|
: null;
|
|
return {
|
|
version: typeof body.serverVersion === "string"
|
|
? body.serverVersion
|
|
: typeof body.version === "string"
|
|
? body.version
|
|
: null,
|
|
identity: typeof serverInfo?.processStartedAt === "string"
|
|
? serverInfo.processStartedAt
|
|
: null,
|
|
};
|
|
} catch {
|
|
return { version: null, identity: null };
|
|
}
|
|
}
|
|
|
|
async function readPreflightActiveRunIds() {
|
|
const config = loadConfig();
|
|
const dbUrl = process.env.DATABASE_URL?.trim()
|
|
|| config.databaseUrl
|
|
|| `postgres://paperclip:paperclip@127.0.0.1:${config.embeddedPostgresPort}/paperclip`;
|
|
const db = createDb(dbUrl);
|
|
try {
|
|
const rows = await db.$client<{ id: string }[]>`
|
|
SELECT id
|
|
FROM heartbeat_runs
|
|
WHERE status = 'running'
|
|
`;
|
|
return rows.map((row) => row.id);
|
|
} finally {
|
|
await db.$client.end({ timeout: 1 });
|
|
}
|
|
}
|
|
|
|
const { serverPid, drainRequired } = readArgs(process.argv.slice(2));
|
|
const preflightActiveRunIds = drainRequired ? [] : await readPreflightActiveRunIds();
|
|
const previousServerInfo = await readPreviousServerInfo();
|
|
const intent = await writeHotRestartIntent({
|
|
previousServerPid: serverPid,
|
|
previousServerIdentity: previousServerInfo.identity,
|
|
previousServerVersion: previousServerInfo.version,
|
|
drainRequired,
|
|
requestedByRunId: process.env.PAPERCLIP_RUN_ID?.trim() || null,
|
|
preflightActiveRunIds,
|
|
});
|
|
|
|
console.log(JSON.stringify({
|
|
status: "hot_restart_intent_written",
|
|
intentPath: resolveHotRestartIntentPath(),
|
|
previousServerPid: intent.previousServerPid,
|
|
previousServerIdentity: intent.previousServerIdentity,
|
|
previousServerStartedAt: intent.previousServerStartedAt,
|
|
previousServerVersion: intent.previousServerVersion,
|
|
drainRequired: intent.drainRequired,
|
|
preflightActiveRunIds: intent.preflightActiveRunIds,
|
|
}, null, 2));
|