fix(server): schedule automatic backups through shared lifecycle (#17836)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/39/changes/paperclip/39
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agent work and its durable state.
> - Automatic database backups protect that state on a configured
schedule.
> - Their independent interval is outside the shared deadline and
shutdown lifecycle.
> - An enabled backup schedule owes real work even when no queue
changed.
> - The existing coordinator can own this deadline without another table
or notification protocol.
> - This change preserves backup policy and moves scheduling into that
shared lifecycle.

## Linked Issues or Issue Description

Refs: #17835. Related work: #11756 (abortable backups), #3549
(stale-backup catch-up), #2120 (interval unref), #16364 (bounded
shutdown steps). This slice changes scheduling; it does not implement
those separate features.

**What existing behavior does this improve?**

Automatic database backup scheduling and shutdown ownership.

**Current behavior**

A standalone interval invokes backups, skips drain and warm standby, and
is not registered with the shared work scheduler. Its future deadline is
invisible there. The interval is not cleared or joined by the delivery
coordinator during shutdown.

**Proposed behavior**

Register the configured backup deadline with the shared coordinator.
Call the existing backup owner only when due. Keep the startup-relative
phase, skip missed slots, retain the next interval after failure or a
busy-owner skip, and join active scheduled work on shutdown.

**Reason and benefit**

One scheduling lifecycle owns overlap, admission, deadlines and
shutdown. Waiting does not query the database. This is a true periodic
obligation, so enabled backups still recur; it is not an empty-queue
polling removal or a claim of measured savings.

This existing slice now follows the merged import-spool parent #17835
and targets master. Merge authorization and exact-head CI/review gates
are tracked separately.

## What Changed

- Add a small backup scheduling owner using the existing coordinator and
scheduler.
- Replace the automatic-backup interval in startup. Register only when
backups are enabled, independently of heartbeat scheduling.
- Preserve the existing backup executor, manual serialization, retention
reads, pre-sleep verification, restart marker and host-sleep gates.
- Test deadlines, drain/standby, failures, busy-owner skips, slow
attempts, restart and shutdown. Update startup-marker tests to exercise
the registered deadline.
- Record the audit and update inventory item 15. No schema or backup
format changes.

## Verification

- Current source acceptance: **130 passing cases in four complete
files**, no skips: backup worker, startup checkpoints, scoped deadline
work and delivery coordinator. Database build, production server
typechecking/build, module boundaries, target migration order and Runner
profile identity pass.
- Runtime and all four selected test sources are byte-identical to
accepted source `11775eee7ed795b989cbb31052e69643c468c3cf` after normal
parent reconciliation. The parent carries the tool-access test-order
repair; it changes no backup behavior.
- Historical broader checks and the real JavaScript backup/restore
probe, including their intermediate failures and limits, remain in [the
backup audit](doc/plans/2026-10-26-database-backup-sleep-schedule.md).
They are not claimed as repeated current-head acceptance.
- No full local repository-suite, deployed suspend/wake, pg_dump, live
provider or cost-savings claim. Final-head CI, security, Greptile 5/5
and any actionable Ponytail findings remain separate merge gates.

## Risks

- Enabled backups still require periodic database and disk work. This
change centralizes the deadline; it does not reduce backup frequency or
establish deployed cost savings.
- The backup engine has no AbortSignal contract. Coordinator stop joins
an active scheduled backup, so a stalled backup can delay graceful
shutdown. Bounded backup cancellation remains separate work.
- Restart still starts a new ordinary interval; it does not replay
historical slots. The existing pre-sleep marker still requests an
immediate verified restart backup and persists on failure.
- A due backup waits during drain or standby and retries on the next
coordinator attempt after admission reopens, within one configured
interval.
- The internal deadline is not a persisted Cloud alarm. Existing
scheduled-backup and checkpoint sleep gates remain unchanged. The real
probe qualifies the JavaScript engine only, not pg_dump or deployed
shutdown latency.

## Model Used

OpenAI Codex (GPT-6-based coding agent). Repository inspection, ordinary
Git merges, source-composition proofs and local lifecycle tests were
used for this reconciliation. No subagents were used in the current
merge campaign.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
authored and GitHub committed 2026-10-29 07:55:53 -05:00
1 parent 1bceae0c91
commit fa5e70faf4
7 files changed
+362 -15

No files matched your search

@@ -30,7 +30,7 @@ Do not mark an item done from a timer removal or a green check alone.
| 12 | Plugin scheduled jobs | Registration/update/removal and actual job deadlines | #17518 published; see latest-head gate records below |
| 13 | Routine, agent-timer and issue-monitor scheduling | Remove periodic wake backstops after complete producer coverage; routine execution-status reconciliation separately | Pending |
| 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending |
| 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending |
| 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Slice 60 implemented; remote acceptance pending; see `2026-10-26-database-backup-sleep-schedule.md` |
| 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending |
| 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | [#17490](https://github.com/paperclipai/paperclip/pull/17490) merged after exact-head 5/5 reviews and passing CI |
| 18 | Abandoned import transfer spools | Creation/completion events, existing retention deadline, startup recovery | [#17835](https://github.com/paperclipai/paperclip/pull/17835), implemented; remote acceptance pending |
@@ -3471,3 +3471,21 @@ and fixture failures during lifecycle testing are recorded in
[the producer and lifecycle audit](2026-10-26-import-spool-cleanup-sleep.md).
Workspace and remote checks remain separate acceptance gates. This is one
remaining original duty, not a declaration that the inventory or stack is done.
## Slice 60: automatic database backup scheduling
The ordinary backup interval now registers its real deadline with the shared
delivery coordinator. Backup execution, serialization, retention, manual calls,
and the pre-sleep/restart-marker protocol remain with their existing owner.
The coordinator prevents overlap, leaves passive drain/standby waits without
SQL, and joins a scheduled backup during shutdown. Enabled backups still owe
periodic work; this does not claim they become event-only or that Cloud persists
this deadline. Existing host-sleep gates remain unchanged.
Nine worker lifecycle cases and the complete affected six-suite set passed
(152 cases), as did workspace typechecking and the full build. A real PostgreSQL
backup/restore probe also passed after correcting two harness-only errors.
Full local test:run and remote acceptance remain outstanding. See the dedicated
[backup scheduling audit](2026-10-26-database-backup-sleep-schedule.md) for limits.
Skill retention, native harness expiration, and the other original inventory
duties remain open; this is not completion of the full stack request.
@@ -0,0 +1,133 @@
# Automatic database backup scheduling
## Scope and ownership
Inventory item 15: automatic database backups. This is slice 60, based on the
import-spool cleanup branch (PR #17835). The server used an independent interval
for this duty. It neither exposed its next deadline through the shared work
scheduler nor joined the delivery coordinator's shutdown.
An enabled backup schedule is a real recurring obligation. This change does not
turn it off after an empty read, reduce its frequency, or claim that scheduled
backups become event-only work. No backup job table or database notification
producer is needed: startup configuration creates the obligation.
The new `database-backup-work.ts` owns cadence. The existing
`runServerDatabaseBackup` callback continues to own backup serialization,
settings reads, archive creation, retention, checkpoint verification, error
logging, and idle-work tracking. Manual and pre-sleep calls keep that same owner.
`index.ts` registers the schedule only when automatic backups are enabled.
The existing shared coordinator owns timers, admission gates, coalescing,
failure reporting, and stopping. Other workers' contracts are unchanged.
## Deadline and lifecycle contract
- The first ordinary backup is due one configured interval after registration,
after any existing restart-marker backup. This preserves the previous
startup-relative cadence. There is no new persisted wall-clock schedule.
- Discovery and early wakes perform no backup, settings query, filesystem read,
or database transaction. `hasPending` reads no external state. The configured
future deadline remains visible through the shared scheduler.
- A due callback invokes the existing backup owner once. It preserves the
owner's `null` result when a manual or pre-sleep backup is already active.
- Success, failure, and a busy-owner skip all advance to the next future slot
on the original cadence. A slow backup skips elapsed slots rather than
creating a catch-up backlog. Failures remain logged and are attempted again
at the next configured interval, as before.
- Drain and warm standby prevent dispatch. Passive waiting does not read the
database or hold an idle receipt. The due obligation remains in memory;
after admission reopens it runs on the next coordinator attempt, within one
configured interval. It does not replay each missed slot.
- Shutdown stops the shared timer and waits for an already-running scheduled
backup. The existing backup implementation has no AbortSignal contract, so
this is a join, not cancellation. A stalled backup can delay graceful shutdown;
adding bounded backup cancellation is separate work (see PR #11756).
- Restart retains the existing behavior: a fresh ordinary interval begins. A
pre-sleep marker still requests the existing immediate verified startup backup.
A failed startup backup leaves that marker for the scheduled retry. This slice
does not add stale-backup catch-up (see PR #3549).
## Host sleep boundary
`markIdleStartupComplete({ scheduledBackups: ... })`, the backup checkpoint
opt-in, wake marker, `readIdleLocalWork` gate, and archive verification are
unchanged. Without the existing checkpoint proof, scheduled backups remain a
host-sleep blocker. With that proof, the existing restart-marker protocol still
owns wake recovery. A registered in-process deadline is not a persisted Cloud
alarm. This slice proves scheduling and lifecycle behavior, not host suspension,
external wake delivery, or fleet savings.
## Verification
The complete six-suite run passed 152 tests, without skips:
```sh
pnpm exec vitest run \
server/src/__tests__/database-backup-work.test.ts \
server/src/__tests__/server-startup-feedback-export.test.ts \
server/src/__tests__/delivery-work-coordinator.test.ts \
server/src/__tests__/work-scheduler.test.ts \
server/src/__tests__/idle-local-work.test.ts \
server/src/__tests__/instance-database-backups-routes.test.ts
pnpm -r typecheck
pnpm build
```
The nine new worker cases exercise the real coordinator and count both backup
calls and database-driver calls. They cover no early work, drain/standby and
startup during drain, failure before drain, busy-owner skips, slow work with
concurrent wakes, stop while active, and startup-relative restart behavior.
Startup integration covers enabled/disabled registration independently of the
heartbeat scheduler. Existing checkpoint tests now invoke the registered backup
deadline, retaining their assertions about marker persistence, startup readiness,
verification failures, and idle receipts.
A separate real PostgreSQL probe used two fresh disposable test databases. It
created a row, waited for the new worker to call the actual JavaScript backup
engine, verified the archive, restored it into the second database, and compared
the committed row. It also checked no early invocation and no invocation after
coordinator stop. The final probe passed. Its first launch referenced a nonexistent
cleanup export; the next probe restored the correct row but rejected the
PostgreSQL `Result` subclass when compared to a plain array. The harness was
corrected to compare row contents and rerun with fresh databases. Neither failure
required a production change. This probe does not qualify the pg_dump engine or
prove deployed shutdown latency.
Workspace typechecking and the full build passed. The repository-wide local
`test:run` remains unverified under the previously recorded embedded PostgreSQL
capacity limitation; targeted suites are not a substitute claim. Remote CI and
review acceptance are separate and remain pending until checked at the PR head.
## October 29 master reconciliation and current acceptance
The current merge campaign carries this existing backup slice through the
reviewed import-spool parent in #17835 and targets master after that parent lands.
The normal parent merge preserved the backup worker, production registration and
all four selected test files byte for byte. Parent changes since qualification
are the import-spool audit and its inherited tool-access concurrency test repair;
they do not change backup execution or scheduling. No packages, schema, backup
format, interval policy, or sleep gate are added by this reconciliation.
Qualification at source `11775eee7ed795b989cbb31052e69643c468c3cf` passed
**130 cases in four complete files**, without skips, in 7.24 seconds:
`database-backup-work`, `server-startup-feedback-export`, `scoped-deadline-work`
and `delivery-work-coordinator`. The tests cover deadline replacement, no work
before a deadline, drain and standby admission, retry after failure, busy-owner
skips, slow callbacks, coordinator stop and startup checkpoint-marker behavior.
Database package build, production server typechecking and build, module
boundaries, target migration order and Runner profile identity also passed.
The reviewed runtime and these four test sources remain identical to that
accepted source; a documentation-only publication does not claim another run.
The package tree remains the already qualified profile-80 parent tree. Historical
six-suite and real JavaScript backup/restore probe results above remain evidence
for their original revisions. This current four-file acceptance does not claim
a repeated physical backup/restore probe, a pg_dump qualification, a full local
repository-suite pass, live provider acceptance, measured host suspension or cost
savings. Fresh CI, security and review results must be checked on the final
published head separately.
Automatic backups remain a genuine scheduled obligation. The shared deadline is
visible internally, but it is not a persisted external Cloud wake alarm. Existing
checkpoint and host-sleep gates still govern whether an instance may suspend.
Shutdown still joins an active backup whose executor has no AbortSignal contract.
@@ -0,0 +1,155 @@
import { afterEach, describe, expect, it, vi } from "vitest";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { registerDatabaseBackupWork } from "../services/database-backup-work.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
const coordinators: ReturnType<typeof createDeliveryWorkCoordinator>[] = [];
afterEach(async () => {
for (const coordinator of coordinators.splice(0)) await coordinator.stop();
vi.useRealTimers();
});
function setup() {
vi.useFakeTimers();
vi.setSystemTime(new Date("2026-10-26T12:00:00Z"));
const execute = vi.fn(async () => []);
const owner = installDatabaseWorkSignals({ execute, transaction: vi.fn() });
const canRun = vi.fn(() => true), canReconcile = vi.fn(() => true), onError = vi.fn();
const coordinator = createDeliveryWorkCoordinator({ owner, canRun, canReconcile, onError });
coordinators.push(coordinator);
const run = vi.fn(async (): Promise<unknown> => ({}));
return { coordinator, execute, canRun, canReconcile, onError, run };
}
const intervalMs = 60_000;
describe("automatic database backup scheduling", () => {
it("waits for the actual backup deadline without querying or holding idle work", async () => {
const s = setup(), now = Date.now();
const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run });
await worker.ready;
expect(s.coordinator.nextWakeAt()).toBe(now + intervalMs);
await vi.advanceTimersByTimeAsync(intervalMs - 1);
expect(s.run).not.toHaveBeenCalled();
expect(s.execute).not.toHaveBeenCalled();
expect(idleWorkSnapshot().active).toBe(0);
await vi.advanceTimersByTimeAsync(1);
expect(s.run).toHaveBeenCalledOnce();
expect(s.coordinator.nextWakeAt()).toBe(now + 2 * intervalMs);
worker.wake();
await vi.advanceTimersByTimeAsync(1);
expect(s.run).toHaveBeenCalledOnce();
expect(s.execute).not.toHaveBeenCalled();
});
it.each(["drain", "standby"])("retains a due backup through %s without passive SQL", async mode => {
const s = setup();
const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run });
await worker.ready;
s.canRun.mockReturnValue(false);
s.canReconcile.mockReturnValue(mode === "drain");
await vi.advanceTimersByTimeAsync(3 * intervalMs);
expect(s.run).not.toHaveBeenCalled();
expect(s.execute).not.toHaveBeenCalled();
expect(idleWorkSnapshot().active).toBe(0);
s.canRun.mockReturnValue(true);
s.canReconcile.mockReturnValue(true);
await vi.advanceTimersByTimeAsync(intervalMs);
expect(s.run).toHaveBeenCalledOnce();
expect(s.coordinator.nextWakeAt()).toBe(Date.now() + intervalMs);
});
it("startup while draining preserves the first due backup", async () => {
const s = setup();
s.canRun.mockReturnValue(false);
await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready;
await vi.advanceTimersByTimeAsync(2 * intervalMs);
expect(s.run).not.toHaveBeenCalled();
s.canRun.mockReturnValue(true);
await vi.advanceTimersByTimeAsync(intervalMs);
expect(s.run).toHaveBeenCalledOnce();
expect(s.execute).not.toHaveBeenCalled();
});
it("reports a failed attempt and retries at the next scheduled slot", async () => {
const s = setup(), now = Date.now();
s.run.mockRejectedValueOnce(new Error("backup storage unavailable"));
await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready;
await vi.advanceTimersByTimeAsync(intervalMs);
expect(s.onError).toHaveBeenCalledOnce();
expect(s.coordinator.nextWakeAt()).toBe(now + 2 * intervalMs);
s.canRun.mockReturnValue(false);
await vi.advanceTimersByTimeAsync(intervalMs);
expect(s.run).toHaveBeenCalledOnce();
expect(s.execute).not.toHaveBeenCalled();
s.canRun.mockReturnValue(true);
await vi.advanceTimersByTimeAsync(intervalMs);
expect(s.run).toHaveBeenCalledTimes(2);
expect(s.coordinator.nextWakeAt()).toBe(now + 4 * intervalMs);
});
it("keeps the schedule after the backup owner skips an overlapping manual backup", async () => {
const s = setup();
s.run.mockResolvedValueOnce(null);
await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready;
await vi.advanceTimersByTimeAsync(2 * intervalMs);
expect(s.run).toHaveBeenCalledTimes(2);
expect(s.onError).not.toHaveBeenCalled();
});
it("coalesces wakes during a slow backup and skips missed slots", async () => {
const s = setup(), now = Date.now();
let finish!: () => void;
s.run.mockImplementationOnce(() => new Promise<void>(resolve => { finish = resolve; }));
const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run });
await worker.ready;
await vi.advanceTimersByTimeAsync(intervalMs);
expect(idleWorkSnapshot().active).toBe(1);
worker.wake(); worker.wake();
await vi.advanceTimersByTimeAsync(2 * intervalMs + 100);
expect(s.run).toHaveBeenCalledOnce();
finish();
await vi.advanceTimersByTimeAsync(1);
expect(s.run).toHaveBeenCalledOnce();
expect(s.coordinator.nextWakeAt()).toBe(now + 4 * intervalMs);
expect(idleWorkSnapshot().active).toBe(0);
await vi.advanceTimersByTimeAsync(intervalMs - 101);
expect(s.run).toHaveBeenCalledTimes(2);
});
it("shutdown waits for the active backup and cannot start another", async () => {
const s = setup();
let finish!: () => void;
s.run.mockImplementationOnce(() => new Promise<void>(resolve => { finish = resolve; }));
const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run });
await worker.ready;
await vi.advanceTimersByTimeAsync(intervalMs);
let stopped = false;
const stopping = s.coordinator.stop().then(() => { stopped = true; });
worker.wake();
await vi.advanceTimersByTimeAsync(3 * intervalMs);
expect(stopped).toBe(false);
expect(s.run).toHaveBeenCalledOnce();
finish();
await stopping;
expect(s.coordinator.nextWakeAt()).toBeNull();
expect(idleWorkSnapshot().active).toBe(0);
await vi.advanceTimersByTimeAsync(3 * intervalMs);
expect(s.run).toHaveBeenCalledOnce();
});
it("restart establishes a fresh interval without replaying historical slots", async () => {
const s = setup();
await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready;
await vi.advanceTimersByTimeAsync(intervalMs / 2);
await s.coordinator.stop();
await vi.advanceTimersByTimeAsync(4 * intervalMs);
const next = setup();
const now = Date.now();
await registerDatabaseBackupWork(next.coordinator, { intervalMs, run: next.run }).ready;
expect(next.coordinator.nextWakeAt()).toBe(now + intervalMs);
expect(next.run).not.toHaveBeenCalled();
await vi.advanceTimersByTimeAsync(intervalMs);
expect(next.run).toHaveBeenCalledOnce();
expect(s.run).not.toHaveBeenCalled();
});
});
@@ -62,6 +62,7 @@ const {
const deliveryWorkErrors: Array<{ queue: string; error: unknown }> = [];
const deliveryWorkRegisterMock = vi.fn((queue: string, task: {
retryMs?: number;
nextRunAt?: () => number | null;
run: (signal?: AbortSignal) => Promise<unknown>;
hasPending?: () => Promise<boolean>;
}) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}) }));
@@ -800,6 +801,19 @@ describe("startServer feedback export wiring", () => {
expect(caseHandoffDeliveryServiceMock.hasPending).toHaveBeenCalledOnce();
});
it.each([false, true])("registers automatic backups independently of heartbeat scheduling when enabled=%s", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: enabled, heartbeatSchedulerEnabled: false }));
await startServer();
const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.databaseBackups);
expect(Boolean(registration)).toBe(enabled);
expect(runDatabaseBackup).not.toHaveBeenCalled();
if (registration) {
expect(registration[1].nextRunAt!()).toBeGreaterThan(Date.now());
await registration[1].run(new AbortController().signal);
expect(runDatabaseBackup).not.toHaveBeenCalled();
}
});
it.each([false, true].flatMap(checkpointMode => [false, true].map(fails => ({ checkpointMode, fails }))))
("finishes restart backup before idle eligibility (checkpoint=$checkpointMode, failure=$fails)", async ({ checkpointMode, fails }) => {
const directory = mkdtempSync(path.join(tmpdir(), "startup-idle-backup-"));
@@ -807,11 +821,6 @@ describe("startServer feedback export wiring", () => {
process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = checkpointMode ? "1" : "0";
loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: true, databaseBackupDir: directory }));
const eligible = vi.spyOn(idleLocalWork, "markIdleStartupComplete");
let retry: (() => void) | undefined;
const timers = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, ms: number) => {
if (ms === 60 * 60 * 1000) retry = callback;
return 1 as unknown as ReturnType<typeof setInterval>;
}) as typeof setInterval);
let finish!: () => void;
const pending = new Promise<void>(resolve => { finish = resolve; });
const writeBackup = async (options: Parameters<typeof runDatabaseBackup>[0]) => {
@@ -843,9 +852,12 @@ describe("startServer feedback export wiring", () => {
expect(existsSync(idleBackupWakeMarker(directory))).toBe(fails);
if (fails) {
vi.mocked(runDatabaseBackup).mockImplementationOnce(writeBackup);
expect(retry).toEqual(expect.any(Function));
const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.databaseBackups);
expect(registration).toBeDefined();
const activeBeforeRetry = idleWorkSnapshot().active;
retry!();
const clock = vi.spyOn(Date, "now").mockReturnValue(registration![1].nextRunAt!()!);
try { await registration![1].run(new AbortController().signal); }
finally { clock.mockRestore(); }
await vi.waitFor(() => {
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
// Marker removal precedes directory fsync and release of the work receipt.
@@ -856,7 +868,7 @@ describe("startServer feedback export wiring", () => {
} finally {
finish();
await starting;
timers.mockRestore(); eligible.mockRestore();
eligible.mockRestore();
if (previousFlag === undefined) delete process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED;
else process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = previousFlag;
rmSync(directory, { recursive: true, force: true });
+5 -6
View File
@@ -1,3 +1,4 @@
import { registerDatabaseBackupWork } from "./services/database-backup-work.js";
import { registerToolConnectionHealthWork } from "./services/tool-connection-health-work.js";
import { approvalQueueSeedingService, registerApprovalQueueSeedingWork } from "./services/approval-queue-seeding-work.js";
import { joinRequestRetentionService, registerJoinRequestRetentionWork } from "./services/join-request-retention.js";
@@ -2127,12 +2128,10 @@ async function startServerWithDatabaseTeardown(
if (!isWarmStandby() && existsSync(idleBackupWakeMarker(config.databaseBackupDir))) {
await runServerDatabaseBackup("scheduled").catch(() => {});
}
setInterval(() => {
if (isWarmStandby() || isIdleTaskDrainActive()) return;
void runServerDatabaseBackup("scheduled").catch(() => {
// runServerDatabaseBackup already logs the failure with context.
});
}, backupIntervalMs);
await registerDatabaseBackupWork(deliveryWork, {
intervalMs: backupIntervalMs,
run: () => runServerDatabaseBackup("scheduled"),
}).ready;
}
// Wait for external adapters to finish loading before accepting requests.
@@ -0,0 +1,29 @@
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
export function registerDatabaseBackupWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
options: { intervalMs: number; run: () => Promise<unknown> },
) {
const { intervalMs } = options;
let nextAt = Date.now() + intervalMs;
return coordinator.register(DELIVERY_QUEUES.databaseBackups, {
retryMs: intervalMs,
pausePassiveMaintenance: true,
hasPending: async () => false,
nextRunAt: () => nextAt,
run: async () => {
if (Date.now() < nextAt) return;
try {
await options.run();
} finally {
nextAt += Math.max(1, Math.floor((Date.now() - nextAt) / intervalMs) + 1) * intervalMs;
}
},
});
}
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
databaseBackups: "database-backups",
importSpoolCleanup: "import-spool-cleanup",
externalConversationSettlement: "external-conversation-settlement",
nativeRecoveryNotices: "native-recovery-notices",