From 9b624a110e281badc54cbdf37ddb792cdc1690ff Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 8 Oct 2026 15:58:55 -0700 Subject: [PATCH] fix: checkpoint database backups before idle sleep (#15629) ## Thinking Path > - Paperclip coordinates agent work and preserves company state. > - Hosted instances can stop compute when admission and all work sources are quiet. > - The periodic database backup timer currently blocks every automatic sleep. > - Disabling backups would remove recovery points. > - This change offers an explicit final-backup checkpoint before sleep. > - A verified archive and restart marker preserve recovery while compute is stopped. ## Linked Issues or Issue Description **What happened?** An otherwise idle instance with automatic backups enabled always reports background work. Operators cannot reclaim idle compute without disabling those backups. The work inventory also queries the database before checking known local blockers. **Expected behavior** An operator can enable checkpoint mode. The server may authorize sleep only after all other work is quiet and a fresh backup is verified under the same owned hold. Backups resume on restart with the existing retention policy. **Steps to reproduce** Acquire a bounded idle drain on an instance with no application work and automatic backups enabled. Read its owned idle safety report. It reports present even when all other work is clear. With this change and `PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1`, a successful final backup can permit sleep; backup failures still refuse it. **Paperclip version or commit** Reproduced from master at 4265cb3a2b. **Deployment mode** Hosted single-process instances with persistent backup storage and serialized sleep/wake operations. Searched open backup and sleep PRs and issues. Related: #15522 and #15599 establish owned idle holds and plugin drains. #15358 proposes general backup health/retention validation; this change verifies only opted-in sleep checkpoints and their restart catch-up. ## What Changed - Add an off-by-default checkpoint flag. Keep automatic backups enabled. - Reject known local blockers before database work. - Finish a new backup during each owned safety check, including final revalidation. - Verify the complete gzip archive and sync its file and directory before pruning older recovery points. - Use unique checkpoint filenames so repeated checks cannot replace an earlier verified archive. - Keep backup work counted through dump, validation and persistence. Changed owners, expired holds, concurrent work and errors refuse sleep. - Persist a restart marker before the dump. Attempt a fresh backup on restart before idle eligibility, then resume the normal timer. Only a verified fresh backup clears the marker. - Restrict managed checkpoint reads to verified Cloud control actors. Tenant instance-admin elevation does not grant backup authority. - Keep archive verification outside engine fallback so failures retain their original cause. - Document that recurring duplicate archives pause while asleep. The final checkpoint and historical archives remain on persistent storage; retention does not prune while asleep. ## Verification - Focused idle, spool, backup and route checks: 137 tests passed, including restoration into a separate PostgreSQL database. - Backup library and checkpoint tests: 15 passed, including rejection before historical backups can be pruned. - `pnpm -r typecheck` and `pnpm build` passed. Follow-up server typecheck and database build cover the verification callback. - Review follow-up: 51 backup/restore/startup tests, 122 route/safety tests, 32 signed-control tests, and server typecheck passed. Startup cases exercise both flag states and retry after failure. The final test-cleanup adjustment passes all 35 startup tests. - The full local run hit four ancestor-directory skill-fixture failures and one tool-route failure. All five passed from an isolated checkout at 949210 (3 targeted chat/tool checks and the complete 39-test email suite). The redundant local run was stopped after final-head hosted CI passed. No full local pass is claimed. - Added-line secret/private-reference review and `git diff --check` passed. - [Final-head hosted CI](https://github.com/paperclipai/paperclip/actions/runs/37853380365) passed, including typecheck, build, all general and serialized test shards, all eight E2E shards and the canary dry run. Apex is 5/5 on 5b1de388ef, with no new findings and all review threads resolved. The branch is conflict-free. Live staging activation awaits merge and a canonical image; no production configuration or deployment changed. ## Risks - The flag defaults off. This is for a single process whose host controls every writer and preserves its backup volume across sleep. Independent database writers require a different backup owner. - Safety reads perform a backup only after a valid owned drain and all other checks pass. Slow backups can outlive the caller timeout; they remain counted until their work settles and cannot authorize sleep after hold expiry. - Initial and final reads each take a fresh backup. Hosts must allow sufficient request time while retaining shutdown headroom. - Retention settings are unchanged. Sleeping instances stop producing duplicate hourly archives and retain their last checkpoint. Filesystem data, encryption keys and off-provider disaster recovery remain separate responsibilities. - No schema migration. Disable the checkpoint flag to restore the blanket backup blocker; automatic backups and an existing restart marker remain active. ## Model Used OpenAI Codex, GPT-6. Used reasoning, repository inspection, code editing and command execution. The runtime did not expose an exact model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal identifier - [x] Focused and isolated local checks pass; final-head hosted CI is green (local full-run limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 3 + doc/idle-sleep-safety.md | 43 ++++++++- packages/db/src/backup-lib.test.ts | 34 ++++++- packages/db/src/backup-lib.ts | 29 ++++-- .../__tests__/idle-database-backup.test.ts | 92 +++++++++++++++++++ server/src/__tests__/idle-local-work.test.ts | 2 + .../src/__tests__/idle-sleep-safety.test.ts | 75 ++++++++++++++- .../instance-settings-routes.test.ts | 44 ++++++++- .../server-startup-feedback-export.test.ts | 68 +++++++++++++- server/src/app.ts | 3 +- server/src/index.ts | 20 +++- .../src/routes/instance-database-backups.ts | 2 +- server/src/routes/instance-settings.ts | 16 +++- server/src/services/idle-database-backup.ts | 52 +++++++++++ server/src/services/idle-local-work.ts | 6 +- server/src/services/idle-sleep-safety.ts | 17 +++- 16 files changed, 476 insertions(+), 30 deletions(-) create mode 100644 server/src/__tests__/idle-database-backup.test.ts create mode 100644 server/src/services/idle-database-backup.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 9736a2e7be1..84dfd41b1e5 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1687,6 +1687,9 @@ Environment overrides: - `PAPERCLIP_DB_BACKUP_ENABLED=true|false` - `PAPERCLIP_DB_BACKUP_INTERVAL_MINUTES=` +- `PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1` enables verified final backups + for owned idle sleep, with restart catch-up. Off by default; see + [idle sleep safety](idle-sleep-safety.md) for the hosting and storage contract. - `PAPERCLIP_DB_BACKUP_RETENTION_DAYS=` - `PAPERCLIP_DB_BACKUP_DIR=/absolute/or/~/path` - `PAPERCLIP_DB_BACKUP_MAX_AGE_HOURS=` controls the `/api/health` diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 61f0977f044..71bd5a549e2 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -52,11 +52,50 @@ counted after readiness until its writes settle. Scheduler work already in flight remains counted until its promise settles. Idle holds pause new scheduler admissions. Database backup promises remain -counted through success or failure. Configured periodic backups also block -sleep until a host owns a durable wake schedule for them. A generation counter invalidates a +counted through success or failure. Configured periodic backups block sleep by +default. The opt-in checkpoint mode below preserves a final backup instead of +running the timer while asleep. A generation counter invalidates a scan if tracked work both starts and finishes during inspection. Reports remain unknown until startup recovery and HTTP tracking are installed. +### Backup checkpoints for idle instances + +`PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED=1` opts an instance with automatic +backups enabled into this mode. Keep its database and persistent backup volume +attached across sleep. All writers must use the instance's admission protocol; +this mode cannot cover independent writers to the same database. +On managed instances, checkpoint-enabled safety reads require a verified signed +Cloud control actor. Tenant instance-admin elevation does not grant platform +backup authority. Self-hosted instance administrators retain operator access. + +Each owned safety read first checks local work, plugin drains and durable work. +Known local blockers return before opening a database transaction. If every +other check passes, the server takes a fresh backup with the existing backup +engine and the retention policy from Instance Settings. It reads the full gzip +archive, checks its trailer and nonempty content, and syncs the archive and +directory before returning `none`. The final pre-stop revalidation takes another +fresh backup. Reads can therefore take longer; the host must bound its request +and still leave enough lease time for a confirmed provider stop. + +There is no cached backup exemption. Failed, concurrent or untracked backups, +other work during the scan, changed owners and expired holds cannot authorize +sleep. Client timeouts do not clear the backup's in-flight token. Slow backups +may retain an awake instance until a later sleep attempt. + +A durable `.idle-backup-wake-required` marker is written before the dump. +No recurring duplicate archives are produced while the instance is asleep. +The last pre-sleep archive and existing history stay on persistent storage; +no retention pruning runs during sleep. On restart, the server attempts a fresh +backup before enabling idle eligibility, then resumes the normal interval. +Only a verified new backup clears the marker. Failure keeps it for the next +scheduled retry. Disabling automatic backups explicitly still disables this +startup backup. Disabling only checkpoint mode restores the sleep blocker and +continues to honor an existing restart marker. + +This does not replace filesystem backups, encryption-key recovery or off-provider +disaster recovery. Validate restoration of a staging checkpoint before enabling +the flag on customer instances. + The report inspects accounting and sandbox-cleanup spools directly. Any entry, including malformed JSON, temporary files and failed write probes, blocks sleep. Read errors and invalid spool paths return unknown. Recovery readers that skip diff --git a/packages/db/src/backup-lib.test.ts b/packages/db/src/backup-lib.test.ts index b6851534991..de3166714cc 100644 --- a/packages/db/src/backup-lib.test.ts +++ b/packages/db/src/backup-lib.test.ts @@ -1,7 +1,7 @@ import fs from "node:fs"; import os from "node:os"; import path from "node:path"; -import { gunzipSync } from "node:zlib"; +import { gunzipSync, gzipSync } from "node:zlib"; import { afterEach, describe, expect, it } from "vitest"; import postgres from "postgres"; import { createBufferedTextFileWriter, runDatabaseBackup, runDatabaseRestore } from "./backup-lib.js"; @@ -75,6 +75,38 @@ describe("createBufferedTextFileWriter", () => { }); describeEmbeddedPostgres("runDatabaseBackup", () => { + it("preserves a post-pg_dump verification error without falling back or pruning", async () => { + const connectionString = await createTempDatabase(); + const backupDir = createTempDir("checkpoint-pg-dump-"); + const executable = path.join(backupDir, "pg-dump-fixture"); + fs.writeFileSync(executable, "#!/bin/sh\nprintf 'SELECT 1;\\n'\n", { mode: 0o700 }); + const history = path.join(backupDir, "paperclip-history.sql.gz"); + const historyBytes = gzipSync("SELECT 'history';"); + fs.writeFileSync(history, historyBytes); + fs.utimesSync(history, new Date(0), new Date(0)); + const previous = process.env.PAPERCLIP_PG_DUMP_PATH; + process.env.PAPERCLIP_PG_DUMP_PATH = executable; + const failure = new Error("fixture archive fsync failed"); + let verified = 0; + try { + await expect(runDatabaseBackup({ + connectionString, backupDir, + retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }, + verifyBeforePrune: async archive => { + verified++; + expect(gunzipSync(fs.readFileSync(archive)).toString()).toBe("SELECT 1;\n"); + throw failure; + }, + })).rejects.toBe(failure); + expect(verified).toBe(1); + expect(fs.readFileSync(history)).toEqual(historyBytes); + expect(fs.readdirSync(backupDir).filter(name => name.endsWith(".sql.gz"))).toEqual(["paperclip-history.sql.gz"]); + } finally { + if (previous === undefined) delete process.env.PAPERCLIP_PG_DUMP_PATH; + else process.env.PAPERCLIP_PG_DUMP_PATH = previous; + } + }); + it("preserves identity generation, sequence options and progress in JavaScript backups", async () => { const source = await createTempDatabase(); const target = await createSiblingDatabase(source, "identity_restore_target"); diff --git a/packages/db/src/backup-lib.ts b/packages/db/src/backup-lib.ts index a95ac3f1be8..fdf849de266 100644 --- a/packages/db/src/backup-lib.ts +++ b/packages/db/src/backup-lib.ts @@ -6,6 +6,7 @@ import { open as openFile } from "node:fs/promises"; import { pipeline } from "node:stream/promises"; import { createGunzip, createGzip } from "node:zlib"; import postgres from "postgres"; +import { randomUUID } from "node:crypto"; export type BackupRetentionPolicy = { dailyDays: number; @@ -28,6 +29,8 @@ export type RunDatabaseBackupOptions = { excludeTables?: string[]; nullifyColumns?: Record; backupEngine?: "auto" | "pg_dump" | "javascript"; + /** Optional durability gate. A rejected archive must not prune history. */ + verifyBeforePrune?: (backupFile: string) => Promise; }; export type RunDatabaseBackupResult = { @@ -542,13 +545,17 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise await sql.end(); }; mkdirSync(opts.backupDir, { recursive: true }); - const sqlFile = resolve(opts.backupDir, `${filenamePrefix}-${timestamp()}.sql`); + // Initial and final sleep checks can run within the same second. A failed + // second checkpoint must never overwrite the first verified recovery point. + const checkpointSuffix = opts.verifyBeforePrune ? `-${randomUUID()}` : ""; + const sqlFile = resolve(opts.backupDir, `${filenamePrefix}-${timestamp()}${checkpointSuffix}.sql`); const backupFile = `${sqlFile}.gz`; const writer = createBufferedTextFileWriter(sqlFile); try { if (backupEngine === "pg_dump" || (backupEngine === "auto" && canUsePgDump)) { await sql`SELECT 1`; + let dumped = false; try { await closeSql(); await runPgDumpBackup({ @@ -556,14 +563,7 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise backupFile, connectTimeout, }); - await writer.abort(); - const sizeBytes = statSync(backupFile).size; - const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); - return { - backupFile, - sizeBytes, - prunedCount, - }; + dumped = true; } catch (error) { if (existsSync(backupFile)) { try { unlinkSync(backupFile); } catch { /* ignore */ } @@ -575,6 +575,16 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise sql = postgres(opts.connectionString, { max: 1, connect_timeout: connectTimeout }); sqlClosed = false; } + if (dumped) { + // Only an engine failure can select the JavaScript fallback. Once + // pg_dump succeeds, archive verification or persistence failures + // must retain their original error and must not reuse a closed writer. + await writer.abort(); + const sizeBytes = statSync(backupFile).size; + await opts.verifyBeforePrune?.(backupFile); + const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); + return { backupFile, sizeBytes, prunedCount }; + } } await sql`SELECT 1`; @@ -1056,6 +1066,7 @@ export async function runDatabaseBackup(opts: RunDatabaseBackupOptions): Promise unlinkSync(sqlFile); const sizeBytes = statSync(backupFile).size; + await opts.verifyBeforePrune?.(backupFile); const prunedCount = pruneOldBackups(opts.backupDir, retention, filenamePrefix); return { diff --git a/server/src/__tests__/idle-database-backup.test.ts b/server/src/__tests__/idle-database-backup.test.ts new file mode 100644 index 00000000000..07e794c9205 --- /dev/null +++ b/server/src/__tests__/idle-database-backup.test.ts @@ -0,0 +1,92 @@ +import { mkdtemp, readFile, rm, symlink, utimes, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { gzipSync } from "node:zlib"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { sql } from "drizzle-orm"; +import { createDb, ensurePostgresDatabase, getEmbeddedPostgresTestSupport, runDatabaseBackup, runDatabaseRestore, startEmbeddedPostgresTestDatabase } from "@paperclipai/db"; +import { clearIdleBackupWakeRequired, idleBackupWakeMarker, markIdleBackupWakeRequired, verifyIdleDatabaseBackup } from "../services/idle-database-backup.js"; + +let directory: string; +beforeEach(async () => { directory = await mkdtemp(path.join(os.tmpdir(), "idle-backup-")); }); +afterEach(async () => { await rm(directory, { recursive: true, force: true }); }); + +describe("idle database backup persistence", () => { + it("keeps a restart marker until a fresh valid backup is complete", async () => { + const archive = path.join(directory, "backup.sql.gz"); + await markIdleBackupWakeRequired(directory); + await writeFile(archive, gzipSync("CREATE TABLE fixture (id integer);\n")); + await verifyIdleDatabaseBackup(archive); + // Verification on the sleep path must not remove the wake obligation. + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + await clearIdleBackupWakeRequired(directory); + await expect(readFile(idleBackupWakeMarker(directory))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readFile(archive)).toEqual(gzipSync("CREATE TABLE fixture (id integer);\n")); + }); + + it.each([ + ["empty file", Buffer.alloc(0)], ["empty SQL", gzipSync("")], + ["truncated gzip", gzipSync("SELECT 1;").subarray(0, -4)], + ["plain text", Buffer.from("SELECT 1;")], + ])("rejects %s and retains the restart obligation", async (_label, contents) => { + await markIdleBackupWakeRequired(directory); + const archive = path.join(directory, "invalid.sql.gz"); + await writeFile(archive, contents); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + }); + + it("rejects a missing archive and symlinked archive or marker", async () => { + const archive = path.join(directory, "backup.sql.gz"); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + const target = path.join(directory, "target"); + await writeFile(target, gzipSync("SELECT 1;")); + await symlink(target, archive); + await expect(verifyIdleDatabaseBackup(archive)).rejects.toThrow(); + await symlink(target, idleBackupWakeMarker(directory)); + await expect(markIdleBackupWakeRequired(directory)).rejects.toThrow(); + expect(await readFile(target)).toEqual(gzipSync("SELECT 1;")); + }); +}); + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("idle backup restore", () => { + it("restores the last pre-sleep state into a separate database", async () => { + const source = await startEmbeddedPostgresTestDatabase("idle-backup-restore-"); + const sourceDb = createDb(source.connectionString); + const targetUrl = new URL(source.connectionString); + targetUrl.pathname = "/idle_restore"; + try { + await sourceDb.execute(sql`CREATE TABLE backup_fixture (id integer PRIMARY KEY, value text NOT NULL)`); + await sourceDb.execute(sql`INSERT INTO backup_fixture VALUES (1, 'last committed state')`); + await markIdleBackupWakeRequired(directory); + const result = await runDatabaseBackup({ + connectionString: source.connectionString, backupDir: directory, + retention: { dailyDays: 14, weeklyWeeks: 4, monthlyMonths: 6 }, + verifyBeforePrune: verifyIdleDatabaseBackup, + }); + await verifyIdleDatabaseBackup(result.backupFile); + await ensurePostgresDatabase(source.connectionString, "idle_restore"); + await runDatabaseRestore({ connectionString: targetUrl.toString(), backupFile: result.backupFile }); + const target = createDb(targetUrl.toString()); + expect(await target.execute(sql`SELECT id, value FROM backup_fixture`)).toEqual([{ id: 1, value: "last committed state" }]); + const checkpointBytes = await readFile(result.backupFile); + expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts"); + // A failed verification must not delete existing recovery points, even + // when their age would make them eligible for the configured pruning. + const history = path.join(directory, "paperclip-history.sql.gz"); + const historyBytes = gzipSync("SELECT 'historical recovery point';"); + await writeFile(history, historyBytes); + await utimes(history, new Date(0), new Date(0)); + await expect(runDatabaseBackup({ + connectionString: source.connectionString, backupDir: directory, + backupEngine: "javascript", retention: { dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }, + verifyBeforePrune: async () => { throw new Error("verification failed"); }, + })).rejects.toThrow("verification failed"); + expect(await readFile(history)).toEqual(historyBytes); + expect(await readFile(result.backupFile)).toEqual(checkpointBytes); + } finally { + await source.cleanup(); + } + }, 60_000); +}); diff --git a/server/src/__tests__/idle-local-work.test.ts b/server/src/__tests__/idle-local-work.test.ts index 51e9dae011f..666c58d0b20 100644 --- a/server/src/__tests__/idle-local-work.test.ts +++ b/server/src/__tests__/idle-local-work.test.ts @@ -40,10 +40,12 @@ describe("idle cleanup inspection", () => { expect(await readIdleLocalWork()).toBe("none"); markIdleStartupComplete({ scheduledBackups: true }); expect(await readIdleLocalWork()).toBe("present"); + expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("none"); markIdleStartupComplete({ scheduledBackups: false }); await fs.mkdir(custom); await fs.writeFile(path.join(custom, "receipt.tmp"), "incomplete"); expect(await readIdleLocalWork()).toBe("present"); + expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("present"); await fs.rm(custom, { recursive: true }); expect(await readIdleLocalWork()).toBe("none"); }); diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index 832673aca58..66a023dd9b9 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -1,4 +1,4 @@ -import { beginIdleTrackedWork } from "../services/task-admission.js"; +import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -22,6 +22,73 @@ const ownerId = "d0b833f4-4098-42de-8420-1907f3aa4895"; const owned = () => ({ ...held(), ownerId }); const emptyLocal = async () => "none" as const; +describe("idle backup checkpoint", () => { + const database = (blocked = false) => { + const transaction = vi.fn(async (run: (tx: unknown) => Promise) => run({ execute: async () => [{ blocked }] })); + return { db: { transaction } as unknown as Db, transaction }; + }; + const backup = (work: () => Promise = async () => true) => vi.fn(async () => { + const finish = beginIdleTrackedWork(); + try { return await work(); } finally { finish(); } + }); + it("backs up after durable inspection and rechecks local work before authorizing sleep", async () => { + const { db, transaction } = database(); + const local = vi.fn(emptyLocal); + const checkpoint = backup(async () => { + expect(transaction).toHaveBeenCalledOnce(); + expect(idleWorkSnapshot().active).toBe(1); + return true; + }); + expect(await readIdleSleepSafety(db, owned, () => now, ownerId, local, undefined, checkpoint)).toEqual(none); + expect(checkpoint).toHaveBeenCalledOnce(); + expect(local).toHaveBeenCalledTimes(2); + }); + it.each(["present", "unknown"] as const)("avoids DB probes and backups for known local %s work", async state => { + const { db, transaction } = database(); + const checkpoint = backup(); + const plugins = vi.fn(); + expect(await readIdleSleepSafety(db, owned, () => now, ownerId, async () => state, plugins, checkpoint)) + .toEqual({ version: 1, backgroundWork: state }); + expect(transaction).not.toHaveBeenCalled(); + expect(plugins).not.toHaveBeenCalled(); + expect(checkpoint).not.toHaveBeenCalled(); + }); + it("does not start a backup for durable work or an unowned hold", async () => { + const checkpoint = backup(); + expect(await readIdleSleepSafety(database(true).db, owned, () => now, ownerId, emptyLocal, undefined, checkpoint)).toEqual(present); + expect(await readIdleSleepSafety(database().db, owned, () => now, "stale", emptyLocal, undefined, checkpoint)).toEqual(unknown); + expect(checkpoint).not.toHaveBeenCalled(); + }); + it.each(["failed", "threw", "untracked", "concurrent", "expired", "replaced"])("keeps the instance awake when the backup is %s", async kind => { + let clock = now; + let status = owned(); + const work = async () => { + if (kind === "threw") throw new Error("private archive path"); + if (kind === "concurrent") { const finish = beginIdleTrackedWork(); finish(); } + if (kind === "expired") clock += 60_000; + if (kind === "replaced") status = { ...status, ownerId: "replacement" }; + return kind !== "failed"; + }; + const checkpoint = kind === "untracked" ? work : backup(work); + expect(await readIdleSleepSafety(database().db, () => status, () => clock, ownerId, emptyLocal, undefined, checkpoint)).toEqual(unknown); + expect(idleWorkSnapshot().active).toBe(0); + }); + it("retains actual backup work after the owning hold expires", async () => { + let finishBackup!: () => void; + const pending = new Promise(resolve => { finishBackup = resolve; }); + const checkpoint = backup(async () => { await pending; return true; }); + let clock = now; + const scan = readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal, undefined, checkpoint); + await vi.waitFor(() => expect(checkpoint).toHaveBeenCalledOnce()); + clock += 60_000; + expect(idleWorkSnapshot().active).toBe(1); + expect(await readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal)).toEqual(unknown); + finishBackup(); + expect(await scan).toEqual(unknown); + expect(idleWorkSnapshot().active).toBe(0); + }); +}); + describe("idle sleep safety failure boundaries", () => { it.each([ @@ -36,7 +103,7 @@ describe("idle sleep safety failure boundaries", () => { it("fails closed when the database is unavailable", async () => { const db = { transaction: vi.fn().mockRejectedValue(new Error("private connection detail")) } as unknown as Db; - expect(await readIdleSleepSafety(db, held, () => now)).toEqual(unknown); + expect(await readIdleSleepSafety(db, held, () => now, undefined, emptyLocal)).toEqual(unknown); }); it.each([ @@ -46,14 +113,14 @@ describe("idle sleep safety failure boundaries", () => { const getStatus = vi.fn().mockReturnValueOnce(held()).mockReturnValue({ ...held(), ...change }); const execute = vi.fn().mockResolvedValue([{ blocked: true }]); const transaction = vi.fn(async (fn: (tx: unknown) => Promise) => fn({ execute })); - expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now)).toEqual(unknown); + expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now, undefined, emptyLocal)).toEqual(unknown); expect(transaction).toHaveBeenCalledWith(expect.any(Function), { isolationLevel: "repeatable read", accessMode: "read only" }); }); it("reports persisted work when the admission hold remains unchanged", async () => { const execute = vi.fn().mockResolvedValue([{ blocked: true }]); const transaction = vi.fn(async (fn: (tx: unknown) => Promise) => fn({ execute })); - expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now)).toEqual(present); + expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now, undefined, emptyLocal)).toEqual(present); }); }); diff --git a/server/src/__tests__/instance-settings-routes.test.ts b/server/src/__tests__/instance-settings-routes.test.ts index ffee8171aaf..5d6cf9d0345 100644 --- a/server/src/__tests__/instance-settings-routes.test.ts +++ b/server/src/__tests__/instance-settings-routes.test.ts @@ -77,7 +77,7 @@ describe("instance settings routes", () => { return { errorHandler, instanceSettingsRoutes }; }); - function createApp(actor: any, pluginWorkers?: PluginWorkerManager) { + function createApp(actor: any, pluginWorkers?: PluginWorkerManager, prepareBackup?: () => Promise) { const { errorHandler, instanceSettingsRoutes } = routeModules.value; const app = express(); app.use(express.json()); @@ -85,7 +85,7 @@ describe("instance settings routes", () => { req.actor = actor; next(); }); - app.use("/api", instanceSettingsRoutes(mockDb as any, pluginWorkers)); + app.use("/api", instanceSettingsRoutes(mockDb as any, pluginWorkers, prepareBackup)); app.use(errorHandler); return app; } @@ -973,7 +973,7 @@ describe("instance settings routes", () => { await request(createApp(adminActor)).post("/api/instance/task-drain").send({ purpose: "idle", ttlMs: 60_000 }).expect(200); expect(mockHeartbeatService.computeTaskDrain).toHaveBeenCalledWith({ purpose: "idle", ttlMs: 60_000 }); await request(createApp(adminActor)).get(`/api/instance/task-drain?idleSleepSafety=1&ownerId=${ownerId}`).expect(200); - expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, expect.any(Function), undefined, undefined); }); it.each([{}, { ttlMs: null }, { ttlMs: 1 }, { ttlMs: 300_001 }])("rejects an unbounded or invalid idle TTL: %j", async body => { @@ -1000,7 +1000,7 @@ describe("instance settings routes", () => { const res = await request(createApp(adminActor)).get("/api/instance/task-drain?idleSleepSafety=1"); expect(res.status).toBe(200); expect(res.body).toEqual({ ...idleStatus, idleSleepSafety: report }); - expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, expect.any(Function), undefined, undefined); expect(mockReadIdleSleepSafety.mock.calls[0][1]()).toEqual(idleStatus); }); @@ -1028,6 +1028,42 @@ describe("instance settings routes", () => { expect(release).toHaveBeenCalledOnce(); }); + it("passes the operator backup hook only through the protected safety report", async () => { + const backup = vi.fn(async () => true); + mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus); + const app = createApp(adminActor, undefined, backup); + await request(app).get("/api/instance/task-drain").expect(200); + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + await request(app).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, + "fixture-owner", expect.any(Function), undefined, backup); + mockReadIdleSleepSafety.mockClear(); + await request(createApp(nonAdminActor, undefined, backup)).get("/api/instance/task-drain?idleSleepSafety=1").expect(403); + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + expect(backup).not.toHaveBeenCalled(); + }); + + it("reserves managed checkpoint creation for verified Cloud control actors", async () => { + const backup = vi.fn(async () => true); + process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "fixture-managed-signal"; + mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus); + try { + for (const source of ["session", "cloud_tenant", "local_implicit"]) { + const tenantAdmin = createApp({ ...adminActor, source }, undefined, backup); + await request(tenantAdmin).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner") + .set("x-paperclip-cloud-control", "unverified-header").expect(403); + } + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + expect(backup).not.toHaveBeenCalled(); + const control = createApp({ ...adminActor, source: "cloud_control" }, undefined, backup); + await request(control).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, + "fixture-owner", expect.any(Function), undefined, backup); + } finally { + delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN; + } + }); + it.each([ ["company member", nonAdminActor], ["agent", { type: "agent", agentId: "agent-1", companyId: "company-1" }], diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 7b89d44dd9a..449d8d5b5dd 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,9 +1,12 @@ import { idleWorkSnapshot } from "../services/task-admission.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import * as idleLocalWork from "../services/idle-local-work.js"; +import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js"; import { reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; import { runDatabaseBackup } from "@paperclipai/db"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { chmodSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { gzipSync } from "node:zlib"; import { tmpdir } from "node:os"; import path from "node:path"; @@ -441,6 +444,69 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true].flatMap(checkpointMode => [false, true].map(fails => ({ checkpointMode, fails })))) + ("finishes restart backup before idle eligibility (checkpoint=$checkpointMode, failure=$fails)", async ({ checkpointMode, fails }) => { + const directory = mkdtempSync(path.join(tmpdir(), "startup-idle-backup-")); + const previousFlag = process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED; + process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = checkpointMode ? "1" : "0"; + loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: true, databaseBackupDir: directory })); + const eligible = vi.spyOn(idleLocalWork, "markIdleStartupComplete"); + let retry: (() => void) | undefined; + const timers = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, ms: number) => { + if (ms === 60 * 60 * 1000) retry = callback; + return 1 as unknown as ReturnType; + }) as typeof setInterval); + let finish!: () => void; + const pending = new Promise(resolve => { finish = resolve; }); + const writeBackup = async (options: Parameters[0]) => { + const backupFile = path.join(directory, "paperclip-restart.sql.gz"); + const bytes = gzipSync("SELECT 'restart recovery';\n"); + writeFileSync(backupFile, bytes); + expect(options.retention).toEqual({ dailyDays: 7, weeklyWeeks: 4, monthlyMonths: 1 }); + expect(options.verifyBeforePrune).toEqual(expect.any(Function)); + await options.verifyBeforePrune!(backupFile); + return { backupFile, sizeBytes: bytes.length, prunedCount: 0 }; + }; + vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { + await pending; + if (fails) throw new Error("fixture restart backup failed"); + return writeBackup(options); + }); + let starting: ReturnType | undefined; + try { + await markIdleBackupWakeRequired(directory); + starting = startServer(); + await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce()); + expect(eligible).not.toHaveBeenCalled(); + expect(idleWorkSnapshot().active).toBeGreaterThan(0); + finish(); + await starting; + expect(eligible).toHaveBeenCalledWith({ scheduledBackups: true }); + const options = createAppMock.mock.calls[0]?.[1] as unknown as { prepareIdleDatabaseBackup?: unknown }; + expect(Boolean(options.prepareIdleDatabaseBackup)).toBe(checkpointMode); + expect(existsSync(idleBackupWakeMarker(directory))).toBe(fails); + if (fails) { + vi.mocked(runDatabaseBackup).mockImplementationOnce(writeBackup); + expect(retry).toEqual(expect.any(Function)); + const activeBeforeRetry = idleWorkSnapshot().active; + retry!(); + await vi.waitFor(() => { + expect(existsSync(idleBackupWakeMarker(directory))).toBe(false); + // Marker removal precedes directory fsync and release of the work receipt. + expect(idleWorkSnapshot().active).toBe(activeBeforeRetry); + }); + expect(runDatabaseBackup).toHaveBeenCalledTimes(2); + } + } finally { + finish(); + await starting; + timers.mockRestore(); eligible.mockRestore(); + if (previousFlag === undefined) delete process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED; + else process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = previousFlag; + rmSync(directory, { recursive: true, force: true }); + } + }); + it.each(([ ["built-in agents", reconcileBuiltInAgentsOnStartup], ["managed homes", reconcileCodexLocalManagedHomesOnStartup], diff --git a/server/src/app.ts b/server/src/app.ts index db931861787..bda8217716a 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -491,6 +491,7 @@ export async function createApp( }): Promise; }; databaseBackupService?: InstanceDatabaseBackupService; + prepareIdleDatabaseBackup?: () => Promise; databaseBackupHealth?: InspectDatabaseBackupHealthOptions; deploymentMode: DeploymentMode; deploymentExposure: DeploymentExposure; @@ -836,7 +837,7 @@ export async function createApp( api.use(resourceMembershipRoutes(db)); api.use(primaryAgentRoutes(db)); api.use(inboxDismissalRoutes(db)); - api.use(instanceSettingsRoutes(db, workerManager)); + api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup)); if (opts.databaseBackupService) { api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService)); } diff --git a/server/src/index.ts b/server/src/index.ts index 0c79e723e62..ce72014df01 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,5 +1,6 @@ import { isIdleTaskDrainActive, beginIdleTrackedWork, trackIdleWork } from "./services/task-admission.js"; import { markIdleStartupComplete } from "./services/idle-local-work.js"; +import { clearIdleBackupWakeRequired, idleBackupWakeMarker, markIdleBackupWakeRequired, verifyIdleDatabaseBackup } from "./services/idle-database-backup.js"; import { cloudWarmStandbyServerOptions } from "./middleware/cloud-warm-standby.js"; import { createCloudWarmStandby } from "./services/cloud-warm-standby.js"; import { subscribeAllCompanyLiveEvents } from "./services/live-events.js"; @@ -819,6 +820,8 @@ async function startServerWithDatabaseTeardown( resolve(config.databaseBackupDir, "..", "db-backup-to-s3.failure"), ]; let databaseBackupInFlight = false; + const idleDatabaseBackupEnabled = config.databaseBackupEnabled + && process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED === "1"; const runServerDatabaseBackup = async ( trigger: InstanceDatabaseBackupTrigger, ): Promise => { @@ -835,19 +838,25 @@ async function startServerWithDatabaseTeardown( const finishIdleBackup = beginIdleTrackedWork(); const startedAt = new Date(); const startedAtMs = Date.now(); - const label = trigger === "scheduled" ? "Automatic" : "Manual"; + const label = trigger === "manual" ? "Manual" : trigger === "idle" ? "Pre-sleep" : "Automatic"; try { + if (trigger === "idle") await markIdleBackupWakeRequired(config.databaseBackupDir); logger.info({ backupDir: config.databaseBackupDir, trigger }, `${label} database backup starting`); // Read retention from Instance Settings (DB) so changes take effect without restart. const generalSettings = await backupSettingsSvc.getGeneral(); const retention = generalSettings.backupRetention; + const verifyCheckpoint = trigger === "idle" || existsSync(idleBackupWakeMarker(config.databaseBackupDir)); const result = await runDatabaseBackup({ connectionString: activeDatabaseConnectionString, backupDir: config.databaseBackupDir, retention, filenamePrefix: "paperclip", + verifyBeforePrune: verifyCheckpoint ? verifyIdleDatabaseBackup : undefined, }); + if (verifyCheckpoint) { + if (trigger !== "idle") await clearIdleBackupWakeRequired(config.databaseBackupDir); + } const finishedAt = new Date(); const response: InstanceDatabaseBackupRunResult = { ...result, @@ -892,6 +901,9 @@ async function startServerWithDatabaseTeardown( // self-hosted: createApp falls back to its built-in kubernetes-only default. const managedPluginAutoInstall = managedConfig?.plugins.autoInstall ?? null; const app = await createApp(db as any, { + prepareIdleDatabaseBackup: idleDatabaseBackupEnabled + ? async () => (await runServerDatabaseBackup("idle")) !== null + : undefined, cloudWarmStandby: isWarmStandby, uiMode, serverPort: listenPort, @@ -1872,6 +1884,12 @@ async function startServerWithDatabaseTeardown( }, "Automatic database backups enabled", ); + // The final pre-sleep dump remains on the volume throughout sleep. Take + // a fresh backup on restart before idle eligibility is enabled, including + // after rollback of the opt-in flag. Failure retains the marker for retry. + if (!isWarmStandby() && existsSync(idleBackupWakeMarker(config.databaseBackupDir))) { + await runServerDatabaseBackup("scheduled").catch(() => {}); + } setInterval(() => { if (isWarmStandby() || isIdleTaskDrainActive()) return; void runServerDatabaseBackup("scheduled").catch(() => { diff --git a/server/src/routes/instance-database-backups.ts b/server/src/routes/instance-database-backups.ts index f05e3361c2c..9635a1cb31b 100644 --- a/server/src/routes/instance-database-backups.ts +++ b/server/src/routes/instance-database-backups.ts @@ -4,7 +4,7 @@ import { forbidden } from "../errors.js"; import { isCloudManagedInstance } from "../services/cloud-instance.js"; import { assertInstanceAdmin } from "./authz.js"; -export type InstanceDatabaseBackupTrigger = "manual" | "scheduled"; +export type InstanceDatabaseBackupTrigger = "manual" | "scheduled" | "idle"; export type InstanceDatabaseBackupRunResult = RunDatabaseBackupResult & { trigger: InstanceDatabaseBackupTrigger; diff --git a/server/src/routes/instance-settings.ts b/server/src/routes/instance-settings.ts index ef578bb7fda..0c126b729e1 100644 --- a/server/src/routes/instance-settings.ts +++ b/server/src/routes/instance-settings.ts @@ -15,6 +15,7 @@ import { } from "../services/cloud-instance.js"; import { getHiddenSettings } from "../services/settings-visibility.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import { readIdleLocalWork } from "../services/idle-local-work.js"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { validate } from "../middleware/validate.js"; import { logger } from "../middleware/logger.js"; @@ -120,7 +121,7 @@ function withTaskDrainTransition(run: () => Promise): Promise { return turn; } -export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager) { +export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager, prepareIdleDatabaseBackup?: () => Promise) { const router = Router(); const svc = instanceSettingsService(db); const environments = environmentService(db); @@ -304,9 +305,18 @@ export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManag // The report covers every company in this process. Ordinary company // members may read process counters, but not instance-wide work state. assertCanManageInstanceSettings(req); + // A checkpoint performs platform-owned backup work. Tenant owners can + // have instance-admin elevation, but that must not grant the manual + // backup authority which managed instances deliberately withhold. + if (prepareIdleDatabaseBackup && isCloudManagedInstance() && req.actor.source !== "cloud_control") { + throw forbidden("Backup checkpoints are platform-managed on cloud-managed instances", { + code: "database_backups_platform_managed", + }); + } const idleSleepSafety = await readIdleSleepSafety(db, () => heartbeat.getTaskDrainStatus(), Date.now, - typeof req.query.ownerId === "string" ? req.query.ownerId : undefined, undefined, - pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers)); + typeof req.query.ownerId === "string" ? req.query.ownerId : undefined, + () => readIdleLocalWork({ backupCheckpoint: Boolean(prepareIdleDatabaseBackup) }), + pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers), prepareIdleDatabaseBackup); res.json({ ...heartbeat.getTaskDrainStatus(), idleSleepSafety }); return; } diff --git a/server/src/services/idle-database-backup.ts b/server/src/services/idle-database-backup.ts new file mode 100644 index 00000000000..271750d118c --- /dev/null +++ b/server/src/services/idle-database-backup.ts @@ -0,0 +1,52 @@ +import { constants, createReadStream } from "node:fs"; +import { mkdir, open, unlink } from "node:fs/promises"; +import path from "node:path"; +import { Writable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import { createGunzip } from "node:zlib"; + +// The marker survives process replacement on persistent instance storage. +// It contains no credentials or tenant data, and never substitutes for a dump. +export const idleBackupWakeMarker = (directory: string) => path.join(directory, ".idle-backup-wake-required"); + +async function syncDirectory(directory: string): Promise { + const handle = await open(directory, "r"); + try { await handle.sync(); } finally { await handle.close(); } +} + +export async function markIdleBackupWakeRequired(directory: string): Promise { + await mkdir(directory, { recursive: true }); + const marker = await open(idleBackupWakeMarker(directory), constants.O_WRONLY | constants.O_CREAT | constants.O_NOFOLLOW, 0o600); + try { + await marker.writeFile("Take a fresh database backup after this process restarts.\n"); + await marker.sync(); + } finally { await marker.close(); } + await syncDirectory(directory); +} + +/** Read the entire archive to verify the gzip trailer, then persist its bytes + * and directory entry before allowing compute to stop. The backup runner + * retains its in-flight receipt until this function settles. */ +export async function verifyIdleDatabaseBackup(backupFile: string): Promise { + const handle = await open(backupFile, constants.O_RDONLY | constants.O_NOFOLLOW); + try { + const stat = await handle.stat(); + if (!stat.isFile() || stat.size === 0) throw new Error("Idle backup archive is empty or invalid"); + let bytes = 0; + await pipeline( + createReadStream(backupFile, { fd: handle.fd, autoClose: false }), + createGunzip(), + new Writable({ write(chunk, _encoding, done) { bytes += chunk.length; done(); } }), + ); + if (bytes === 0) throw new Error("Idle backup archive contains no SQL"); + await handle.sync(); + } finally { await handle.close(); } + await syncDirectory(path.dirname(backupFile)); +} + +export async function clearIdleBackupWakeRequired(directory: string): Promise { + await unlink(idleBackupWakeMarker(directory)).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + }); + await syncDirectory(directory); +} diff --git a/server/src/services/idle-local-work.ts b/server/src/services/idle-local-work.ts index 1095c298f80..2d54ad90309 100644 --- a/server/src/services/idle-local-work.ts +++ b/server/src/services/idle-local-work.ts @@ -33,9 +33,11 @@ export async function inspectIdleSpool(directory: string): Promise { +export async function readIdleLocalWork(options: { backupCheckpoint?: boolean } = {}): Promise { if (!startupComplete || !ingressTracked) return "unknown"; - if (scheduledBackups || idleWorkSnapshot().active !== 0) return "present"; + // This exemption belongs only to a scan which will finish a new backup + // under the same admission hold before it can return an empty report. + if ((scheduledBackups && !options.backupCheckpoint) || idleWorkSnapshot().active !== 0) return "present"; const directories = new Set([...spoolDirectories, idleAccountingSpoolPath(), idleOrphanSpoolPath()]); const results = await Promise.all([...directories].map(inspectIdleSpool)); if (results.includes("unknown")) return "unknown"; diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index ad24e3037d1..a8c7a80e13b 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -97,12 +97,17 @@ export async function readIdleSleepSafety( ownerId?: string, inspectLocalWork: () => Promise = readIdleLocalWork, inspectPlugins?: InspectIdlePlugins, + prepareBackup?: () => Promise, ): Promise { const unknown: IdleSleepSafety = { version: 1, backgroundWork: "unknown" }; const before = getDrainStatus(); const localBefore = idleWorkSnapshot(); if (!sameQuietHold(before, before, now())) return unknown; try { + // Reject known local blockers before querying (and potentially waking) + // the tenant database. Keep the final inspection below as a second fence. + const localFirst = await inspectLocalWork(); + if (localFirst !== "none") return { version: 1, backgroundWork: localFirst }; let quietPlugins: string[] = []; if (inspectPlugins) { if (!ownerId || ownerId !== before.ownerId || before.expiresAt === null || localBefore.active !== 0) return unknown; @@ -124,11 +129,21 @@ export async function readIdleSleepSafety( if (blocked === undefined || !sameQuietHold(before, getDrainStatus(), now())) return unknown; if (blocked) return { version: 1, backgroundWork: "present" }; if (!ownerId || ownerId !== before.ownerId || before.expiresAt === null || localBefore.active !== 0) return unknown; + let expectedGeneration = localBefore.generation; + if (prepareBackup) { + if (!sameQuietHold(before, getDrainStatus(), now()) || + idleWorkSnapshot().generation !== expectedGeneration || idleWorkSnapshot().active !== 0) return unknown; + // The runner owns exactly one tracked-work receipt through dump, + // archive verification and fsync. Its start/finish are the only work + // allowed during this scan. Any concurrent work invalidates the dump. + if (!(await prepareBackup())) return unknown; + expectedGeneration += 2; + } const local = await inspectLocalWork(); const after = getDrainStatus(); const localAfter = idleWorkSnapshot(); if (!sameQuietHold(before, after, now()) || localAfter.active !== 0 || - localBefore.generation !== localAfter.generation) return unknown; + expectedGeneration !== localAfter.generation) return unknown; return { version: 1, backgroundWork: local }; } catch {