Files
PaperClipAI/scripts/request-hot-restart.ts
DottaandPaperclip b163c6c473 fix(server): preserve hot restart intent across path upgrade (#10593)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can preserve eligible agent runs during a controlled hot
restart.
> - A path change moved restart state from the Paperclip home root to
the instance root.
> - A staged update can therefore make the old server and the new server
read different intent files.
> - The old server then drains live runs, while the new server can start
without a shutdown snapshot.
> - This pull request adds a correlated compatibility handoff and
records the live preflight set.
> - It also verifies the target process instance on Linux, macOS, and
Windows.
> - The benefit is complete and safe run classification across the path
upgrade.

## Linked Issues or Issue Description

No public GitHub issue covers this defect.

**What happened?**

A staged hot restart can run an older server that reads
`hot-restart-intent.json` from the Paperclip home root and a new server
that writes the file under the instance root. The old server misses the
request and uses graceful drain. The new server later finds its marker
without a shutdown snapshot. Before this change, that state could
produce an empty loss list even when live runs existed before restart.

**Expected behavior**

The old server must receive the PID-targeted restart request at its
legacy path. The new server must correlate the legacy shutdown snapshot
with its instance-scoped request. Every run that was live during
preflight must appear as adopted, finalized while down, or lost. A
reused PID must not let a stale marker claim a different process
instance.

**Steps to reproduce**

1. Start a server version from before the instance-root marker change.
2. Keep one or more local-agent heartbeat runs active.
3. Stage a current build and request a hot restart from that build.
4. Observe that the old server reads only the home-root path while the
staged build writes only the instance-root path.
5. Observe graceful drain and a new-server intent that has no shutdown
snapshot.

**Paperclip version or commit**

The path transition entered `master` in #10045. The hot-restart adoption
flow came from #9647. This fix targets current `master` and
compatibility with the immediately preceding home-root behavior.

**Deployment mode**

Self-hosted server built from source with controlled service hot
restarts.

Related work: #9628 is the original broader hot-restart feature PR.
#10556 addresses embedded PostgreSQL lifecycle behavior and does not
address marker-path compatibility.

## What Changed

- Write an authoritative instance-scoped intent and a correlated legacy
home-root handoff marker.
- Merge a legacy shutdown snapshot only when immutable request identity
fields match.
- Prevent a non-default instance from consuming an uncorrelated
legacy-only marker.
- Record preflight running heartbeat IDs and reconcile snapshot
omissions from current database state.
- Serialize marker claims, snapshot writes, stale recovery, and matching
cleanup with recoverable per-path filesystem leases.
- Read process start identity on Linux, macOS, and Windows to
distinguish a reused PID from the original server.
- Require identity for new restart requests and fail closed when a
supported platform cannot provide it.
- Classify older markers by comparing the replacement server boot time
or operating-system process start time with the request time.
- Close the preflight database client explicitly and use a root-safe SQL
query.
- Add focused unit, platform-branch, database-backed, and CI regression
coverage.
- Document the compatibility handoff, process identity probes, and
instance-scoped report path.

## Verification

- `pnpm exec vitest run server/src/services/hot-restart.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts -t
"hot-restart|old-server legacy|preflight live|preflight run|spawn
identity before hot restart"` — 24 tests passed and 90 tests were
skipped across 2 files.
- `pnpm exec vitest run server/src/services/hot-restart.test.ts` — 17
tests passed.
- `pnpm exec vitest run
server/src/__tests__/issue-watchdogs-routes.test.ts -t "restarts a
stalled claimed run"` — 1 test passed and 10 tests were skipped.
- `pnpm exec vitest run
server/src/__tests__/agent-action-audit-routes.test.ts -t "allows an
agent with issue:delegate"` — 1 test passed and 7 tests were skipped.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `git diff --check` — passed.
- GitHub Actions — 26 of 26 checks passed at
`55a79cb029be8b1dc89926d9d89ccd2181266d5c`.
- Greptile — 5/5 at the same head with no unresolved current-head review
threads.

## Risks

- The legacy handoff path is shared across instances. Exclusive claims
and per-path leases prevent overwrite and match-before-delete races.
- Process identity uses platform commands as a fallback when the health
endpoint has no identity. Linux reads `/proc`, macOS and BSD use `ps`,
and Windows uses PowerShell.
- A supported-platform identity probe failure aborts the restart. This
fails closed instead of replacing an unknown live process.
- Older intent files do not contain process identity. The server
compares the replacement boot or process start time with the request
time when those values are available.
- A preflight database read can fail before the marker is written. The
command fails closed instead of claiming a restart whose live-run set is
unknown.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5. The exact deployment model ID and
context-window size were not exposed by this runtime. Reasoning,
repository editing, shell execution, test execution, GitHub CLI, and
Paperclip API capabilities were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-31 21:55:59 -07:00

121 lines
3.9 KiB
JavaScript

#!/usr/bin/env -S node --import tsx
import { createDb } from "../packages/db/src/index.js";
import { loadConfig } from "../server/src/config.js";
import {
resolveHotRestartIntentPath,
writeHotRestartIntent,
} from "../server/src/services/hot-restart.js";
function usage(): never {
console.error([
"Usage: tsx scripts/request-hot-restart.ts --server-pid <pid> [--drain-required]",
"",
"Writes an instance-scoped hot-restart intent plus a legacy home-root handoff marker.",
].join("\n"));
process.exit(2);
}
function readArgs(argv: string[]) {
let serverPid: number | null = null;
let drainRequired = false;
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--server-pid") {
const raw = argv[index + 1];
if (!raw) usage();
const parsed = Number(raw);
if (!Number.isInteger(parsed) || parsed <= 0) usage();
serverPid = parsed;
index += 1;
continue;
}
if (arg === "--drain-required") {
drainRequired = true;
continue;
}
if (arg === "--help" || arg === "-h") usage();
console.error(`Unknown argument: ${arg}`);
usage();
}
if (!serverPid) usage();
return { serverPid, drainRequired };
}
function normalizeApiBase(raw: string | undefined) {
const trimmed = raw?.trim();
if (!trimmed) return null;
return trimmed.replace(/\/+$/, "").replace(/\/api$/, "");
}
async function readPreviousServerInfo() {
const apiBase = normalizeApiBase(process.env.PAPERCLIP_API_URL);
if (!apiBase) return { version: null, identity: null };
try {
const apiKey = process.env.PAPERCLIP_API_KEY?.trim();
const response = await fetch(`${apiBase}/api/health`, {
headers: apiKey ? { Authorization: `Bearer ${apiKey}` } : undefined,
signal: AbortSignal.timeout(2_000),
});
if (!response.ok) return { version: null, identity: null };
const body = await response.json() as Record<string, unknown>;
const serverInfo = body.serverInfo && typeof body.serverInfo === "object"
? body.serverInfo as Record<string, unknown>
: null;
return {
version: typeof body.serverVersion === "string"
? body.serverVersion
: typeof body.version === "string"
? body.version
: null,
identity: typeof serverInfo?.processStartedAt === "string"
? serverInfo.processStartedAt
: null,
};
} catch {
return { version: null, identity: null };
}
}
async function readPreflightActiveRunIds() {
const config = loadConfig();
const dbUrl = process.env.DATABASE_URL?.trim()
|| config.databaseUrl
|| `postgres://paperclip:paperclip@127.0.0.1:${config.embeddedPostgresPort}/paperclip`;
const db = createDb(dbUrl);
try {
const rows = await db.$client<{ id: string }[]>`
SELECT id
FROM heartbeat_runs
WHERE status = 'running'
`;
return rows.map((row) => row.id);
} finally {
await db.$client.end({ timeout: 1 });
}
}
const { serverPid, drainRequired } = readArgs(process.argv.slice(2));
const preflightActiveRunIds = drainRequired ? [] : await readPreflightActiveRunIds();
const previousServerInfo = await readPreviousServerInfo();
const intent = await writeHotRestartIntent({
previousServerPid: serverPid,
previousServerIdentity: previousServerInfo.identity,
previousServerVersion: previousServerInfo.version,
drainRequired,
requestedByRunId: process.env.PAPERCLIP_RUN_ID?.trim() || null,
preflightActiveRunIds,
});
console.log(JSON.stringify({
status: "hot_restart_intent_written",
intentPath: resolveHotRestartIntentPath(),
previousServerPid: intent.previousServerPid,
previousServerIdentity: intent.previousServerIdentity,
previousServerStartedAt: intent.previousServerStartedAt,
previousServerVersion: intent.previousServerVersion,
drainRequired: intent.drainRequired,
preflightActiveRunIds: intent.preflightActiveRunIds,
}, null, 2));