Files
PaperClipAI/server/src/shutdown.ts
T
DottaandPaperclip ca6416da81 fix(server): preserve hot-restart shutdown snapshots (#10815)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A guarded hot restart must preserve or finalize every active agent
run.
> - The server uses embedded PostgreSQL when `DATABASE_URL` is not set.
> - The database dependency installs signal handlers before Paperclip
installs its coordinated shutdown handler.
> - Those handlers can stop PostgreSQL before Paperclip writes the
shutdown snapshot.
> - ACP runs also use server-owned stdio and cannot be adopted after
that server exits.
> - This pull request keeps PostgreSQL available through snapshot and
drain, then uses the existing ordered stop.
> - The benefit is a complete restart report with no false adoption and
no missing snapshot loss.

## Linked Issues or Issue Description

**What happened?**

A guarded hot restart with a valid marker can report a live preflight
run as lost with reason `missing_shutdown_snapshot`. The
`embedded-postgres` package imports `async-exit-hook`. That package
registers `SIGINT` and `SIGTERM` listeners before Paperclip registers
its own shutdown listener. The dependency can close PostgreSQL while
Paperclip queries active heartbeat runs and writes the snapshot.

**Expected behavior**

Paperclip must keep its database available until it persists the
shutdown snapshot and completes any required run drain. A detached CLI
run must remain eligible for adoption. An ACP run must finish as
interrupted and queue a retry because its server-owned stdio cannot
survive the server.

**Steps to reproduce**

1. Run Paperclip from source with embedded PostgreSQL.
2. Start a local ACP-backed agent run.
3. Write a valid hot-restart marker for the current server process.
4. send `SIGTERM` through the service manager.
5. Inspect the restart report and server log.
6. Observe that PostgreSQL can close before the shutdown snapshot query
completes.

**Paperclip version or commit**

The defect reproduces on `2ab797dcbed0031c45c7335a0f497fea2a20bd9a`.

**Deployment mode**

Self-hosted server built from source, with embedded PostgreSQL and a
systemd service.

Related work: #9628 introduced hot-restart continuity. #10556 explores a
broader database ownership transfer. #10775 addresses ACP continuity
after replacement startup. This pull request uses a smaller path: it
keeps the current database owner alive through snapshot and drain, then
performs the existing explicit database stop.

## What Changed

- Remove only the `SIGINT` and `SIGTERM` listeners added by the embedded
PostgreSQL import.
- Preserve Paperclip's existing ordered database stop after heartbeat
snapshot and drain.
- Detect active ACP and server-stdio local runs before shutdown.
- Persist their complete snapshot before changing the marker to an ACP
drain request.
- Drain only ACP runs to an interrupted terminal state and queue their
retry.
- Keep detached CLI runs eligible for adoption in the same mixed
restart.
- Quiesce already-running scheduler queue claims before capturing the
snapshot and selective drain set.
- Report a selected ACP run as lost if process termination succeeds but
its terminal database write does not persist.
- Add the drain reason to the restart report.
- Document the normal path and the one-time recovery path across an
older affected build.

## Verification

- `PAPERCLIP_TEST_DATABASE_MODE=native pnpm --filter @paperclipai/server
exec vitest run src/__tests__/heartbeat-process-recovery.test.ts` — 100
passed.
- `pnpm exec vitest run server/src/shutdown.test.ts
server/src/services/hot-restart.test.ts` — 24 passed.
- The shutdown suite imports the real `embedded-postgres` package and
verifies that its eager signal listeners are absent after the guarded
import.
- The embedded PostgreSQL recovery suite verifies snapshot, pre-snapshot
scheduler quiescence, selective ACP drain, detached CLI adoption, queued
retry, original-run finalization, `lostRunIds=[]` in a mixed restart,
and fail-closed reporting when terminal persistence fails.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `git diff --check` — passed.
- A full workspace typecheck reached the UI and stopped because the
shared local install does not contain its declared `@base-ui/react`
dependency. All server and preceding package checks passed. CI uses a
clean install and remains the authoritative full gate.

## Risks

- Low to moderate risk. This changes shutdown signal ownership and local
run behavior during guarded restarts.
- Paperclip already stops its managed embedded database explicitly. The
change removes only the dependency listeners that race the coordinated
path.
- ACP runs now retry instead of receiving an unsafe bare-process
adoption. Detached CLI runs keep their existing adoption behavior.
- The report adds one field. There is no schema migration or breaking
API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5. The runtime did not expose a more specific
model revision or context-window size. Reasoning, repository editing,
shell execution, and test execution were enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-04 11:44:43 -05:00

84 lines
2.6 KiB
TypeScript

type HotRestartShutdownPreparation = {
skipDrain: boolean;
};
const COORDINATED_SHUTDOWN_SIGNALS = ["SIGINT", "SIGTERM"] as const;
type ShutdownSignalTarget = {
rawListeners(eventName: string): Function[];
removeListener(eventName: string, listener: (...args: any[]) => void): unknown;
};
/**
* Some dependencies eagerly install process signal handlers as an import side
* effect. Paperclip must remain the sole owner of SIGINT/SIGTERM ordering: its
* handler first snapshots live heartbeat runs and only then stops embedded
* infrastructure. Remove only listeners added by the supplied import, while
* preserving every listener that was already registered.
*/
export async function loadWithoutCoordinatedShutdownSignalHooks<T>(
load: () => Promise<T>,
signalTarget: ShutdownSignalTarget = process,
) {
const listenersBeforeLoad = new Map(
COORDINATED_SHUTDOWN_SIGNALS.map((signal) => [
signal,
signalTarget.rawListeners(signal),
]),
);
let loaded: T;
try {
loaded = await load();
} finally {
for (const signal of COORDINATED_SHUTDOWN_SIGNALS) {
const remainingBeforeLoad = [...(listenersBeforeLoad.get(signal) ?? [])];
for (const listener of signalTarget.rawListeners(signal)) {
const existingIndex = remainingBeforeLoad.indexOf(listener);
if (existingIndex >= 0) {
remainingBeforeLoad.splice(existingIndex, 1);
continue;
}
signalTarget.removeListener(signal, listener as (...args: any[]) => void);
}
}
}
return loaded;
}
export async function coordinateHeartbeatSchedulerShutdown<
TPreparation extends HotRestartShutdownPreparation,
>(input: {
signal: "SIGINT" | "SIGTERM";
prepareHotRestartShutdown: ((signal: "SIGINT" | "SIGTERM") => Promise<TPreparation>) | null;
waitForHeartbeatSchedulerIdle: () => Promise<void>;
}): Promise<{
hotRestart: TPreparation | null;
preparationError: unknown;
waitedForSchedulerIdle: boolean;
}> {
let hotRestart: TPreparation | null = null;
let preparationError: unknown = null;
// The signal handler stops the scheduler before entering this coordinator.
// Quiesce any callback that was already in flight before querying running
// rows for the shutdown snapshot, otherwise a late queue claim can create a
// run that is absent from both the snapshot and the selective drain set.
await input.waitForHeartbeatSchedulerIdle();
if (input.prepareHotRestartShutdown) {
try {
hotRestart = await input.prepareHotRestartShutdown(input.signal);
} catch (err) {
preparationError = err;
}
}
return {
hotRestart,
preparationError,
waitedForSchedulerIdle: true,
};
}