diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index f9b92a5fedd..6310c54c46e 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -30,7 +30,7 @@ Do not mark an item done from a timer removal or a green check alone. | 12 | Plugin scheduled jobs | Registration/update/removal and actual job deadlines | #17518 published; see latest-head gate records below | | 13 | Routine, agent-timer and issue-monitor scheduling | Remove periodic wake backstops after complete producer coverage; routine execution-status reconciliation separately | Pending | | 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending | -| 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending | +| 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Slice 60 implemented; remote acceptance pending; see `2026-10-26-database-backup-sleep-schedule.md` | | 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending | | 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | [#17490](https://github.com/paperclipai/paperclip/pull/17490) merged after exact-head 5/5 reviews and passing CI | | 18 | Abandoned import transfer spools | Creation/completion events, existing retention deadline, startup recovery | [#17835](https://github.com/paperclipai/paperclip/pull/17835), implemented; remote acceptance pending | @@ -3471,3 +3471,21 @@ and fixture failures during lifecycle testing are recorded in [the producer and lifecycle audit](2026-10-26-import-spool-cleanup-sleep.md). Workspace and remote checks remain separate acceptance gates. This is one remaining original duty, not a declaration that the inventory or stack is done. + +## Slice 60: automatic database backup scheduling + +The ordinary backup interval now registers its real deadline with the shared +delivery coordinator. Backup execution, serialization, retention, manual calls, +and the pre-sleep/restart-marker protocol remain with their existing owner. +The coordinator prevents overlap, leaves passive drain/standby waits without +SQL, and joins a scheduled backup during shutdown. Enabled backups still owe +periodic work; this does not claim they become event-only or that Cloud persists +this deadline. Existing host-sleep gates remain unchanged. + +Nine worker lifecycle cases and the complete affected six-suite set passed +(152 cases), as did workspace typechecking and the full build. A real PostgreSQL +backup/restore probe also passed after correcting two harness-only errors. +Full local test:run and remote acceptance remain outstanding. See the dedicated +[backup scheduling audit](2026-10-26-database-backup-sleep-schedule.md) for limits. +Skill retention, native harness expiration, and the other original inventory +duties remain open; this is not completion of the full stack request. diff --git a/doc/plans/2026-10-26-database-backup-sleep-schedule.md b/doc/plans/2026-10-26-database-backup-sleep-schedule.md new file mode 100644 index 00000000000..22272b8270c --- /dev/null +++ b/doc/plans/2026-10-26-database-backup-sleep-schedule.md @@ -0,0 +1,133 @@ +# Automatic database backup scheduling + +## Scope and ownership + +Inventory item 15: automatic database backups. This is slice 60, based on the +import-spool cleanup branch (PR #17835). The server used an independent interval +for this duty. It neither exposed its next deadline through the shared work +scheduler nor joined the delivery coordinator's shutdown. + +An enabled backup schedule is a real recurring obligation. This change does not +turn it off after an empty read, reduce its frequency, or claim that scheduled +backups become event-only work. No backup job table or database notification +producer is needed: startup configuration creates the obligation. + +The new `database-backup-work.ts` owns cadence. The existing +`runServerDatabaseBackup` callback continues to own backup serialization, +settings reads, archive creation, retention, checkpoint verification, error +logging, and idle-work tracking. Manual and pre-sleep calls keep that same owner. +`index.ts` registers the schedule only when automatic backups are enabled. +The existing shared coordinator owns timers, admission gates, coalescing, +failure reporting, and stopping. Other workers' contracts are unchanged. + +## Deadline and lifecycle contract + +- The first ordinary backup is due one configured interval after registration, + after any existing restart-marker backup. This preserves the previous + startup-relative cadence. There is no new persisted wall-clock schedule. +- Discovery and early wakes perform no backup, settings query, filesystem read, + or database transaction. `hasPending` reads no external state. The configured + future deadline remains visible through the shared scheduler. +- A due callback invokes the existing backup owner once. It preserves the + owner's `null` result when a manual or pre-sleep backup is already active. +- Success, failure, and a busy-owner skip all advance to the next future slot + on the original cadence. A slow backup skips elapsed slots rather than + creating a catch-up backlog. Failures remain logged and are attempted again + at the next configured interval, as before. +- Drain and warm standby prevent dispatch. Passive waiting does not read the + database or hold an idle receipt. The due obligation remains in memory; + after admission reopens it runs on the next coordinator attempt, within one + configured interval. It does not replay each missed slot. +- Shutdown stops the shared timer and waits for an already-running scheduled + backup. The existing backup implementation has no AbortSignal contract, so + this is a join, not cancellation. A stalled backup can delay graceful shutdown; + adding bounded backup cancellation is separate work (see PR #11756). +- Restart retains the existing behavior: a fresh ordinary interval begins. A + pre-sleep marker still requests the existing immediate verified startup backup. + A failed startup backup leaves that marker for the scheduled retry. This slice + does not add stale-backup catch-up (see PR #3549). + +## Host sleep boundary + +`markIdleStartupComplete({ scheduledBackups: ... })`, the backup checkpoint +opt-in, wake marker, `readIdleLocalWork` gate, and archive verification are +unchanged. Without the existing checkpoint proof, scheduled backups remain a +host-sleep blocker. With that proof, the existing restart-marker protocol still +owns wake recovery. A registered in-process deadline is not a persisted Cloud +alarm. This slice proves scheduling and lifecycle behavior, not host suspension, +external wake delivery, or fleet savings. + +## Verification + +The complete six-suite run passed 152 tests, without skips: + +```sh +pnpm exec vitest run \ + server/src/__tests__/database-backup-work.test.ts \ + server/src/__tests__/server-startup-feedback-export.test.ts \ + server/src/__tests__/delivery-work-coordinator.test.ts \ + server/src/__tests__/work-scheduler.test.ts \ + server/src/__tests__/idle-local-work.test.ts \ + server/src/__tests__/instance-database-backups-routes.test.ts +pnpm -r typecheck +pnpm build +``` + +The nine new worker cases exercise the real coordinator and count both backup +calls and database-driver calls. They cover no early work, drain/standby and +startup during drain, failure before drain, busy-owner skips, slow work with +concurrent wakes, stop while active, and startup-relative restart behavior. +Startup integration covers enabled/disabled registration independently of the +heartbeat scheduler. Existing checkpoint tests now invoke the registered backup +deadline, retaining their assertions about marker persistence, startup readiness, +verification failures, and idle receipts. + +A separate real PostgreSQL probe used two fresh disposable test databases. It +created a row, waited for the new worker to call the actual JavaScript backup +engine, verified the archive, restored it into the second database, and compared +the committed row. It also checked no early invocation and no invocation after +coordinator stop. The final probe passed. Its first launch referenced a nonexistent +cleanup export; the next probe restored the correct row but rejected the +PostgreSQL `Result` subclass when compared to a plain array. The harness was +corrected to compare row contents and rerun with fresh databases. Neither failure +required a production change. This probe does not qualify the pg_dump engine or +prove deployed shutdown latency. + +Workspace typechecking and the full build passed. The repository-wide local +`test:run` remains unverified under the previously recorded embedded PostgreSQL +capacity limitation; targeted suites are not a substitute claim. Remote CI and +review acceptance are separate and remain pending until checked at the PR head. + +## October 29 master reconciliation and current acceptance + +The current merge campaign carries this existing backup slice through the +reviewed import-spool parent in #17835 and targets master after that parent lands. +The normal parent merge preserved the backup worker, production registration and +all four selected test files byte for byte. Parent changes since qualification +are the import-spool audit and its inherited tool-access concurrency test repair; +they do not change backup execution or scheduling. No packages, schema, backup +format, interval policy, or sleep gate are added by this reconciliation. + +Qualification at source `11775eee7ed795b989cbb31052e69643c468c3cf` passed +**130 cases in four complete files**, without skips, in 7.24 seconds: +`database-backup-work`, `server-startup-feedback-export`, `scoped-deadline-work` +and `delivery-work-coordinator`. The tests cover deadline replacement, no work +before a deadline, drain and standby admission, retry after failure, busy-owner +skips, slow callbacks, coordinator stop and startup checkpoint-marker behavior. +Database package build, production server typechecking and build, module +boundaries, target migration order and Runner profile identity also passed. +The reviewed runtime and these four test sources remain identical to that +accepted source; a documentation-only publication does not claim another run. + +The package tree remains the already qualified profile-80 parent tree. Historical +six-suite and real JavaScript backup/restore probe results above remain evidence +for their original revisions. This current four-file acceptance does not claim +a repeated physical backup/restore probe, a pg_dump qualification, a full local +repository-suite pass, live provider acceptance, measured host suspension or cost +savings. Fresh CI, security and review results must be checked on the final +published head separately. + +Automatic backups remain a genuine scheduled obligation. The shared deadline is +visible internally, but it is not a persisted external Cloud wake alarm. Existing +checkpoint and host-sleep gates still govern whether an instance may suspend. +Shutdown still joins an active backup whose executor has no AbortSignal contract. diff --git a/server/src/__tests__/database-backup-work.test.ts b/server/src/__tests__/database-backup-work.test.ts new file mode 100644 index 00000000000..d1350189a4e --- /dev/null +++ b/server/src/__tests__/database-backup-work.test.ts @@ -0,0 +1,155 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerDatabaseBackupWork } from "../services/database-backup-work.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const coordinators: ReturnType[] = []; +afterEach(async () => { + for (const coordinator of coordinators.splice(0)) await coordinator.stop(); + vi.useRealTimers(); +}); +function setup() { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2026-10-26T12:00:00Z")); + const execute = vi.fn(async () => []); + const owner = installDatabaseWorkSignals({ execute, transaction: vi.fn() }); + const canRun = vi.fn(() => true), canReconcile = vi.fn(() => true), onError = vi.fn(); + const coordinator = createDeliveryWorkCoordinator({ owner, canRun, canReconcile, onError }); + coordinators.push(coordinator); + const run = vi.fn(async (): Promise => ({})); + return { coordinator, execute, canRun, canReconcile, onError, run }; +} +const intervalMs = 60_000; + +describe("automatic database backup scheduling", () => { + it("waits for the actual backup deadline without querying or holding idle work", async () => { + const s = setup(), now = Date.now(); + const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }); + await worker.ready; + expect(s.coordinator.nextWakeAt()).toBe(now + intervalMs); + await vi.advanceTimersByTimeAsync(intervalMs - 1); + expect(s.run).not.toHaveBeenCalled(); + expect(s.execute).not.toHaveBeenCalled(); + expect(idleWorkSnapshot().active).toBe(0); + await vi.advanceTimersByTimeAsync(1); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.coordinator.nextWakeAt()).toBe(now + 2 * intervalMs); + worker.wake(); + await vi.advanceTimersByTimeAsync(1); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.execute).not.toHaveBeenCalled(); + }); + + it.each(["drain", "standby"])("retains a due backup through %s without passive SQL", async mode => { + const s = setup(); + const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }); + await worker.ready; + s.canRun.mockReturnValue(false); + s.canReconcile.mockReturnValue(mode === "drain"); + await vi.advanceTimersByTimeAsync(3 * intervalMs); + expect(s.run).not.toHaveBeenCalled(); + expect(s.execute).not.toHaveBeenCalled(); + expect(idleWorkSnapshot().active).toBe(0); + s.canRun.mockReturnValue(true); + s.canReconcile.mockReturnValue(true); + await vi.advanceTimersByTimeAsync(intervalMs); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.coordinator.nextWakeAt()).toBe(Date.now() + intervalMs); + }); + + it("startup while draining preserves the first due backup", async () => { + const s = setup(); + s.canRun.mockReturnValue(false); + await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready; + await vi.advanceTimersByTimeAsync(2 * intervalMs); + expect(s.run).not.toHaveBeenCalled(); + s.canRun.mockReturnValue(true); + await vi.advanceTimersByTimeAsync(intervalMs); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.execute).not.toHaveBeenCalled(); + }); + + it("reports a failed attempt and retries at the next scheduled slot", async () => { + const s = setup(), now = Date.now(); + s.run.mockRejectedValueOnce(new Error("backup storage unavailable")); + await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready; + await vi.advanceTimersByTimeAsync(intervalMs); + expect(s.onError).toHaveBeenCalledOnce(); + expect(s.coordinator.nextWakeAt()).toBe(now + 2 * intervalMs); + s.canRun.mockReturnValue(false); + await vi.advanceTimersByTimeAsync(intervalMs); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.execute).not.toHaveBeenCalled(); + s.canRun.mockReturnValue(true); + await vi.advanceTimersByTimeAsync(intervalMs); + expect(s.run).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBe(now + 4 * intervalMs); + }); + + it("keeps the schedule after the backup owner skips an overlapping manual backup", async () => { + const s = setup(); + s.run.mockResolvedValueOnce(null); + await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready; + await vi.advanceTimersByTimeAsync(2 * intervalMs); + expect(s.run).toHaveBeenCalledTimes(2); + expect(s.onError).not.toHaveBeenCalled(); + }); + + it("coalesces wakes during a slow backup and skips missed slots", async () => { + const s = setup(), now = Date.now(); + let finish!: () => void; + s.run.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }); + await worker.ready; + await vi.advanceTimersByTimeAsync(intervalMs); + expect(idleWorkSnapshot().active).toBe(1); + worker.wake(); worker.wake(); + await vi.advanceTimersByTimeAsync(2 * intervalMs + 100); + expect(s.run).toHaveBeenCalledOnce(); + finish(); + await vi.advanceTimersByTimeAsync(1); + expect(s.run).toHaveBeenCalledOnce(); + expect(s.coordinator.nextWakeAt()).toBe(now + 4 * intervalMs); + expect(idleWorkSnapshot().active).toBe(0); + await vi.advanceTimersByTimeAsync(intervalMs - 101); + expect(s.run).toHaveBeenCalledTimes(2); + }); + + it("shutdown waits for the active backup and cannot start another", async () => { + const s = setup(); + let finish!: () => void; + s.run.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + const worker = registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }); + await worker.ready; + await vi.advanceTimersByTimeAsync(intervalMs); + let stopped = false; + const stopping = s.coordinator.stop().then(() => { stopped = true; }); + worker.wake(); + await vi.advanceTimersByTimeAsync(3 * intervalMs); + expect(stopped).toBe(false); + expect(s.run).toHaveBeenCalledOnce(); + finish(); + await stopping; + expect(s.coordinator.nextWakeAt()).toBeNull(); + expect(idleWorkSnapshot().active).toBe(0); + await vi.advanceTimersByTimeAsync(3 * intervalMs); + expect(s.run).toHaveBeenCalledOnce(); + }); + + it("restart establishes a fresh interval without replaying historical slots", async () => { + const s = setup(); + await registerDatabaseBackupWork(s.coordinator, { intervalMs, run: s.run }).ready; + await vi.advanceTimersByTimeAsync(intervalMs / 2); + await s.coordinator.stop(); + await vi.advanceTimersByTimeAsync(4 * intervalMs); + const next = setup(); + const now = Date.now(); + await registerDatabaseBackupWork(next.coordinator, { intervalMs, run: next.run }).ready; + expect(next.coordinator.nextWakeAt()).toBe(now + intervalMs); + expect(next.run).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(intervalMs); + expect(next.run).toHaveBeenCalledOnce(); + expect(s.run).not.toHaveBeenCalled(); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 4cc764a5e9a..94f22012fe9 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -62,6 +62,7 @@ const { const deliveryWorkErrors: Array<{ queue: string; error: unknown }> = []; const deliveryWorkRegisterMock = vi.fn((queue: string, task: { retryMs?: number; + nextRunAt?: () => number | null; run: (signal?: AbortSignal) => Promise; hasPending?: () => Promise; }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}) })); @@ -800,6 +801,19 @@ describe("startServer feedback export wiring", () => { expect(caseHandoffDeliveryServiceMock.hasPending).toHaveBeenCalledOnce(); }); + it.each([false, true])("registers automatic backups independently of heartbeat scheduling when enabled=%s", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: enabled, heartbeatSchedulerEnabled: false })); + await startServer(); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.databaseBackups); + expect(Boolean(registration)).toBe(enabled); + expect(runDatabaseBackup).not.toHaveBeenCalled(); + if (registration) { + expect(registration[1].nextRunAt!()).toBeGreaterThan(Date.now()); + await registration[1].run(new AbortController().signal); + expect(runDatabaseBackup).not.toHaveBeenCalled(); + } + }); + it.each([false, true].flatMap(checkpointMode => [false, true].map(fails => ({ checkpointMode, fails })))) ("finishes restart backup before idle eligibility (checkpoint=$checkpointMode, failure=$fails)", async ({ checkpointMode, fails }) => { const directory = mkdtempSync(path.join(tmpdir(), "startup-idle-backup-")); @@ -807,11 +821,6 @@ describe("startServer feedback export wiring", () => { process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = checkpointMode ? "1" : "0"; loadConfigMock.mockReturnValue(buildTestConfig({ databaseBackupEnabled: true, databaseBackupDir: directory })); const eligible = vi.spyOn(idleLocalWork, "markIdleStartupComplete"); - let retry: (() => void) | undefined; - const timers = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, ms: number) => { - if (ms === 60 * 60 * 1000) retry = callback; - return 1 as unknown as ReturnType; - }) as typeof setInterval); let finish!: () => void; const pending = new Promise(resolve => { finish = resolve; }); const writeBackup = async (options: Parameters[0]) => { @@ -843,9 +852,12 @@ describe("startServer feedback export wiring", () => { expect(existsSync(idleBackupWakeMarker(directory))).toBe(fails); if (fails) { vi.mocked(runDatabaseBackup).mockImplementationOnce(writeBackup); - expect(retry).toEqual(expect.any(Function)); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.databaseBackups); + expect(registration).toBeDefined(); const activeBeforeRetry = idleWorkSnapshot().active; - retry!(); + const clock = vi.spyOn(Date, "now").mockReturnValue(registration![1].nextRunAt!()!); + try { await registration![1].run(new AbortController().signal); } + finally { clock.mockRestore(); } await vi.waitFor(() => { expect(existsSync(idleBackupWakeMarker(directory))).toBe(false); // Marker removal precedes directory fsync and release of the work receipt. @@ -856,7 +868,7 @@ describe("startServer feedback export wiring", () => { } finally { finish(); await starting; - timers.mockRestore(); eligible.mockRestore(); + eligible.mockRestore(); if (previousFlag === undefined) delete process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED; else process.env.PAPERCLIP_DB_BACKUP_IDLE_CHECKPOINT_ENABLED = previousFlag; rmSync(directory, { recursive: true, force: true }); diff --git a/server/src/index.ts b/server/src/index.ts index ee3da8b6e60..d4112fe24b7 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerDatabaseBackupWork } from "./services/database-backup-work.js"; import { registerToolConnectionHealthWork } from "./services/tool-connection-health-work.js"; import { approvalQueueSeedingService, registerApprovalQueueSeedingWork } from "./services/approval-queue-seeding-work.js"; import { joinRequestRetentionService, registerJoinRequestRetentionWork } from "./services/join-request-retention.js"; @@ -2127,12 +2128,10 @@ async function startServerWithDatabaseTeardown( if (!isWarmStandby() && existsSync(idleBackupWakeMarker(config.databaseBackupDir))) { await runServerDatabaseBackup("scheduled").catch(() => {}); } - setInterval(() => { - if (isWarmStandby() || isIdleTaskDrainActive()) return; - void runServerDatabaseBackup("scheduled").catch(() => { - // runServerDatabaseBackup already logs the failure with context. - }); - }, backupIntervalMs); + await registerDatabaseBackupWork(deliveryWork, { + intervalMs: backupIntervalMs, + run: () => runServerDatabaseBackup("scheduled"), + }).ready; } // Wait for external adapters to finish loading before accepting requests. diff --git a/server/src/services/database-backup-work.ts b/server/src/services/database-backup-work.ts new file mode 100644 index 00000000000..c4989e603a7 --- /dev/null +++ b/server/src/services/database-backup-work.ts @@ -0,0 +1,29 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; + + +export function registerDatabaseBackupWork( + coordinator: ReturnType, + options: { intervalMs: number; run: () => Promise }, +) { + const { intervalMs } = options; + + + let nextAt = Date.now() + intervalMs; + return coordinator.register(DELIVERY_QUEUES.databaseBackups, { + retryMs: intervalMs, + pausePassiveMaintenance: true, + hasPending: async () => false, + nextRunAt: () => nextAt, + run: async () => { + if (Date.now() < nextAt) return; + try { + await options.run(); + } finally { + + + nextAt += Math.max(1, Math.floor((Date.now() - nextAt) / intervalMs) + 1) * intervalMs; + } + }, + }); +} diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index f087fccba55..f5e68e69690 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + databaseBackups: "database-backups", importSpoolCleanup: "import-spool-cleanup", externalConversationSettlement: "external-conversation-settlement", nativeRecoveryNotices: "native-recovery-notices",