Files
PaperClipAI/server/src/__tests__/database-backup-owner.test.ts
T
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00

134 lines
7.8 KiB
TypeScript

import { existsSync } from "node:fs";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { gzipSync } from "node:zlib";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { runDatabaseBackup } from "@paperclipai/db";
import { createDatabaseBackupOwner } from "../services/database-backup-owner.js";
import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js";
import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js";
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
import type { Db } from "@paperclipai/db";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { registerDatabaseBackupWork } from "../services/database-backup-work.js";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
vi.mock("@paperclipai/db", async importOriginal => ({ ...await importOriginal<typeof import("@paperclipai/db")>(), runDatabaseBackup: vi.fn() }));
let directory: string;
const policy = { dailyDays: 14, weeklyWeeks: 4, monthlyMonths: 6 };
const logger = { info: vi.fn(), warn: vi.fn(), error: vi.fn() };
function owner(getRetention = async () => policy) {
return createDatabaseBackupOwner({ connectionString: "unused", backupDir: directory, getRetention, logger });
}
async function validArchive(options: Parameters<typeof runDatabaseBackup>[0]) {
const backupFile = path.join(directory, "fixture.sql.gz");
const bytes = gzipSync("CREATE TABLE fixture (id integer);\n");
await writeFile(backupFile, bytes);
await options.verifyBeforePrune?.(backupFile);
return { backupFile, sizeBytes: bytes.length, prunedCount: 0 };
}
beforeEach(async () => {
directory = await mkdtemp(path.join(os.tmpdir(), "backup-owner-"));
vi.mocked(runDatabaseBackup).mockReset().mockImplementation(validArchive);
});
afterEach(async () => { await rm(directory, { recursive: true, force: true }); vi.restoreAllMocks(); });
describe("database backup owner", () => {
it("marks before retention access and retains the marker until a verified wake backup", async () => {
const run = owner(async () => {
expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts");
expect(idleWorkSnapshot().active).toBe(1);
return policy;
});
const before = idleWorkSnapshot();
const checkpoint = await run("idle");
expect(checkpoint).toMatchObject({ trigger: "idle", retention: policy, backupDir: directory });
expect(idleWorkSnapshot()).toEqual({ active: before.active, generation: before.generation + 2 });
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
await owner()("scheduled");
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
expect(runDatabaseBackup).toHaveBeenCalledWith(expect.objectContaining({ retention: policy, verifyBeforePrune: expect.any(Function) }));
});
it("keeps one receipt and single-flight exclusion until the actual operation settles", async () => {
let finish!: () => void;
const gate = new Promise<void>(resolve => { finish = resolve; });
vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { await gate; return validArchive(options); });
const run = owner(); const backup = run("idle");
await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(1);
expect(await run("scheduled")).toBeNull();
await expect(run("manual")).rejects.toMatchObject({ status: 409 });
await expect(run("idle")).rejects.toMatchObject({ status: 409 });
finish(); await backup;
expect(idleWorkSnapshot().active).toBe(0);
});
it.each(["dump", "integrity", "retention"])("retains the marker and releases the receipt after %s failure", async failure => {
await markIdleBackupWakeRequired(directory);
if (failure === "dump") vi.mocked(runDatabaseBackup).mockRejectedValueOnce(new Error("dump failed"));
if (failure === "integrity") vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => {
const backupFile = path.join(directory, "invalid.sql.gz"); await writeFile(backupFile, "corrupt");
await options.verifyBeforePrune!(backupFile); throw new Error("unreachable");
});
const run = owner(async () => { if (failure === "retention") throw new Error("retention unavailable"); return policy; });
await expect(run("scheduled")).rejects.toThrow();
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
expect(idleWorkSnapshot().active).toBe(0);
await owner()("scheduled");
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
});
it.each(["expiry", "owner", "generation"])("rejects sleep after %s changes during the production owner's backup", async change => {
let now = Date.now();
let hold: IdleSleepDrainStatus = { ownerId: "owner", draining: true, startedAt: new Date(now - 1), expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 };
let finish!: () => void; const gate = new Promise<void>(resolve => { finish = resolve; });
vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { await gate; return validArchive(options); });
const db = { transaction: async (fn: (tx: unknown) => unknown) => fn({ select: () => ({ from: () => ({ where: async () => [] }) }), execute: async () => [{ blocked: false }] }) } as unknown as Db;
const run = owner();
const safety = readIdleSleepSafety(db, () => hold, () => now, "owner", async () => "none", undefined, async () => Boolean(await run("idle")));
await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce());
expect(idleWorkSnapshot().active).toBe(1);
if (change === "expiry") now += 60_000;
if (change === "owner") hold = { ...hold, ownerId: "replacement" };
if (change === "generation") beginIdleTrackedWork()();
finish();
expect(await safety).toEqual({ version: 1, backgroundWork: "unknown" });
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
expect(idleWorkSnapshot().active).toBe(0);
});
it("resumes the normal interval after a verified wake backup and retains a due slot through drain", async () => {
await markIdleBackupWakeRequired(directory);
const run = owner(); await run("scheduled");
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
vi.useFakeTimers();
const execute = vi.fn(async () => []);
let canRun = true;
const coordinator = createDeliveryWorkCoordinator({ owner: installDatabaseWorkSignals({ execute, transaction: vi.fn() }), canRun: () => canRun, onError: error => { throw error; } });
try {
await registerDatabaseBackupWork(coordinator, { intervalMs: 1000, run: () => run("scheduled") }).ready;
await vi.advanceTimersByTimeAsync(999);
expect(runDatabaseBackup).toHaveBeenCalledOnce();
expect(execute).not.toHaveBeenCalled();
canRun = false;
await vi.advanceTimersByTimeAsync(3001);
expect(runDatabaseBackup).toHaveBeenCalledOnce();
expect(execute).not.toHaveBeenCalled();
canRun = true;
await vi.advanceTimersByTimeAsync(1000);
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
expect(runDatabaseBackup).toHaveBeenCalledTimes(2);
expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now());
} finally { await coordinator.stop(); vi.useRealTimers(); }
});
it("fails closed if the storage path is no longer a directory", async () => {
await rm(directory, { recursive: true }); await writeFile(directory, "mount unavailable");
await expect(owner()("idle")).rejects.toThrow();
expect(runDatabaseBackup).not.toHaveBeenCalled();
expect(idleWorkSnapshot().active).toBe(0);
});
});