mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected apps can retain unfinished provider work through MCP Tasks. > - Managed defaults can enable Tasks without a stored experimental flag. > - The sleep check used the stored flag and could miss an active provider task. > - Enabling Tasks also failed to notify an owner that had become idle. > - This pull request uses the effective setting and wakes the existing owner after the setting commits. > - Pending work keeps its recovery path. Completed history can become quiet. ## Linked Issues or Issue Description Refs #21263, which added durable provider Tasks. The current roadmap covers the governed MCP Tool Gateway. This change repairs two existing paths. **What happened?** The gateway could enable MCP Tasks through managed defaults while the idle check reported no pending work. After an operator disabled and enabled Tasks, an existing task could stay idle until a separate task write woke its owner. **Expected behavior** The gateway and sleep check must use the same effective setting. Enabling Tasks must notify the existing owner. It must retain the task's original deadline and backoff. **Steps to reproduce** 1. Retain a task with `next_poll_at`. Use managed Tasks defaults without a stored true flag. Read the normal sleep projection. 2. Disable Tasks, retain the same task, then enable Tasks through the settings service without writing the task again. ## What Changed - Read the effective Tasks flag inside the same read-only sleep transaction. Bind that value to the existing pending-task query. - Register the existing Tasks queue notification before the settings transaction writes. The normal commit protocol owns the wake. - Add two real-PostgreSQL regressions to the existing nine-case suite. Extend eight mock transactions with the read projection they now need. Keep their assertions unchanged. - Publish the Tailscale enrollment fixture state by rename so its CLI watcher cannot parse a truncated JSON file. Existing enrollment assertions and time limits remain unchanged. ## Verification The two regressions fail against the preceding current-master products: the pending-work projection reports `none`, and re-enable does not rearm the owner within the original two-second oracle. After the fix, all nine PostgreSQL cases pass at the recorded source, including completed history, disabled Tasks, credential debt and fail-closed query errors. At current `72bc32f`, the normal DB build and migrations, all nine PostgreSQL cases, 155 affected mock cases, and the complete normal Server typecheck/planner pass. The mock run includes three selected safety families and six complete files; it explicitly excludes 196 existing durable-PG cases. All original case deadlines and assertions remain. The startup file's earlier 133-case pass is limited to its unchanged mock contract; a new hot-restart implementation is not relabeled as tested. Root independently verified all current outputs and process closure. The MCP product and regression files remain byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale test fixture repair below. Exact current-head CI passes: 55 successful checks, four explicit skips and one neutral result; Snyk succeeds. Fresh Greptile is 5/5 for this exact head. The complete latest edited summary and all paginated threads were inspected, with no open findings. An earlier additional whole-eight-file run reached its 180-second aggregate watchdog while the existing mixed PostgreSQL suite was still running. It produced no individual failing assertion. Its startup file passed 133 cases. The original failure remains recorded. The follow-up selects the three affected mock families and runs the other affected mock files whole; it does not claim that the interrupted PostgreSQL file passed. No test assertion or case deadline changed. Full required CI remains a merge gate. The first remote run at `72bc32f` passed 51 checks but failed one Tailscale enrollment case and an interrupted browser shard; two aggregate gates then failed. The browser log records a runner shutdown signal; its underlying cause is unknown, and one specific-job recovery was requested. The Tailscale case failed its status CLI on both original CI attempts. A local control then held publication until the original file watcher observed an empty state: the CLI threw `Unexpected end of JSON input`, and the existing enrollment assertion failed. Publishing the state atomically passed the same control. All nine original enrollment cases now pass in 4.393 seconds with the fixture repair. The recorded remote interleaving remains unknown; the original logs and the initial local non-reproduction are retained. No retry count, assertion, timeout or product behavior was weakened. The later exact-head run passes all required checks. A subsequent server shard was interrupted by a runner shutdown and later recovered at the same source; the interruption and unknown underlying cause remain recorded. Use Node24/pnpm9, one worker, an owned external PostgreSQL test database and the existing fixture/admin URL options. The two new regressions are in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`. These checks do not establish deployed suspension or a live provider's availability. ## Risks - The read adds the normal effective-settings projection to the existing bounded snapshot. Query errors must still block sleep. - A settings update now notifies the existing task owner. Transaction rollback must not publish that notification. - This adds no schema, provider protocol, timer or replacement task framework. Existing active task polling remains required until its work settles. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix. GPT-6-based Codex also reviewed and integrated it. The host does not expose every supervising model ID or context-window size. The agents used source editing, Git, code execution and regression tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (the focused scope above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the existing pending-task contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
134 lines
7.8 KiB
TypeScript
134 lines
7.8 KiB
TypeScript
import { existsSync } from "node:fs";
|
|
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { gzipSync } from "node:zlib";
|
|
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { runDatabaseBackup } from "@paperclipai/db";
|
|
import { createDatabaseBackupOwner } from "../services/database-backup-owner.js";
|
|
import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js";
|
|
import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js";
|
|
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
|
import { registerDatabaseBackupWork } from "../services/database-backup-work.js";
|
|
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
|
|
|
|
vi.mock("@paperclipai/db", async importOriginal => ({ ...await importOriginal<typeof import("@paperclipai/db")>(), runDatabaseBackup: vi.fn() }));
|
|
let directory: string;
|
|
const policy = { dailyDays: 14, weeklyWeeks: 4, monthlyMonths: 6 };
|
|
const logger = { info: vi.fn(), warn: vi.fn(), error: vi.fn() };
|
|
function owner(getRetention = async () => policy) {
|
|
return createDatabaseBackupOwner({ connectionString: "unused", backupDir: directory, getRetention, logger });
|
|
}
|
|
async function validArchive(options: Parameters<typeof runDatabaseBackup>[0]) {
|
|
const backupFile = path.join(directory, "fixture.sql.gz");
|
|
const bytes = gzipSync("CREATE TABLE fixture (id integer);\n");
|
|
await writeFile(backupFile, bytes);
|
|
await options.verifyBeforePrune?.(backupFile);
|
|
return { backupFile, sizeBytes: bytes.length, prunedCount: 0 };
|
|
}
|
|
beforeEach(async () => {
|
|
directory = await mkdtemp(path.join(os.tmpdir(), "backup-owner-"));
|
|
vi.mocked(runDatabaseBackup).mockReset().mockImplementation(validArchive);
|
|
});
|
|
afterEach(async () => { await rm(directory, { recursive: true, force: true }); vi.restoreAllMocks(); });
|
|
|
|
describe("database backup owner", () => {
|
|
it("marks before retention access and retains the marker until a verified wake backup", async () => {
|
|
const run = owner(async () => {
|
|
expect(await readFile(idleBackupWakeMarker(directory), "utf8")).toContain("restarts");
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
return policy;
|
|
});
|
|
const before = idleWorkSnapshot();
|
|
const checkpoint = await run("idle");
|
|
expect(checkpoint).toMatchObject({ trigger: "idle", retention: policy, backupDir: directory });
|
|
expect(idleWorkSnapshot()).toEqual({ active: before.active, generation: before.generation + 2 });
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
|
|
await owner()("scheduled");
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
|
|
expect(runDatabaseBackup).toHaveBeenCalledWith(expect.objectContaining({ retention: policy, verifyBeforePrune: expect.any(Function) }));
|
|
});
|
|
|
|
it("keeps one receipt and single-flight exclusion until the actual operation settles", async () => {
|
|
let finish!: () => void;
|
|
const gate = new Promise<void>(resolve => { finish = resolve; });
|
|
vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { await gate; return validArchive(options); });
|
|
const run = owner(); const backup = run("idle");
|
|
await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce());
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
expect(await run("scheduled")).toBeNull();
|
|
await expect(run("manual")).rejects.toMatchObject({ status: 409 });
|
|
await expect(run("idle")).rejects.toMatchObject({ status: 409 });
|
|
finish(); await backup;
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it.each(["dump", "integrity", "retention"])("retains the marker and releases the receipt after %s failure", async failure => {
|
|
await markIdleBackupWakeRequired(directory);
|
|
if (failure === "dump") vi.mocked(runDatabaseBackup).mockRejectedValueOnce(new Error("dump failed"));
|
|
if (failure === "integrity") vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => {
|
|
const backupFile = path.join(directory, "invalid.sql.gz"); await writeFile(backupFile, "corrupt");
|
|
await options.verifyBeforePrune!(backupFile); throw new Error("unreachable");
|
|
});
|
|
const run = owner(async () => { if (failure === "retention") throw new Error("retention unavailable"); return policy; });
|
|
await expect(run("scheduled")).rejects.toThrow();
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
await owner()("scheduled");
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
|
|
});
|
|
|
|
it.each(["expiry", "owner", "generation"])("rejects sleep after %s changes during the production owner's backup", async change => {
|
|
let now = Date.now();
|
|
let hold: IdleSleepDrainStatus = { ownerId: "owner", draining: true, startedAt: new Date(now - 1), expiresAt: new Date(now + 60_000), activeRuns: 0, pendingWakes: 0 };
|
|
let finish!: () => void; const gate = new Promise<void>(resolve => { finish = resolve; });
|
|
vi.mocked(runDatabaseBackup).mockImplementationOnce(async options => { await gate; return validArchive(options); });
|
|
const db = { transaction: async (fn: (tx: unknown) => unknown) => fn({ select: () => ({ from: () => ({ where: async () => [] }) }), execute: async () => [{ blocked: false }] }) } as unknown as Db;
|
|
const run = owner();
|
|
const safety = readIdleSleepSafety(db, () => hold, () => now, "owner", async () => "none", undefined, async () => Boolean(await run("idle")));
|
|
await vi.waitFor(() => expect(runDatabaseBackup).toHaveBeenCalledOnce());
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
if (change === "expiry") now += 60_000;
|
|
if (change === "owner") hold = { ...hold, ownerId: "replacement" };
|
|
if (change === "generation") beginIdleTrackedWork()();
|
|
finish();
|
|
expect(await safety).toEqual({ version: 1, backgroundWork: "unknown" });
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(true);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it("resumes the normal interval after a verified wake backup and retains a due slot through drain", async () => {
|
|
await markIdleBackupWakeRequired(directory);
|
|
const run = owner(); await run("scheduled");
|
|
expect(existsSync(idleBackupWakeMarker(directory))).toBe(false);
|
|
vi.useFakeTimers();
|
|
const execute = vi.fn(async () => []);
|
|
let canRun = true;
|
|
const coordinator = createDeliveryWorkCoordinator({ owner: installDatabaseWorkSignals({ execute, transaction: vi.fn() }), canRun: () => canRun, onError: error => { throw error; } });
|
|
try {
|
|
await registerDatabaseBackupWork(coordinator, { intervalMs: 1000, run: () => run("scheduled") }).ready;
|
|
await vi.advanceTimersByTimeAsync(999);
|
|
expect(runDatabaseBackup).toHaveBeenCalledOnce();
|
|
expect(execute).not.toHaveBeenCalled();
|
|
canRun = false;
|
|
await vi.advanceTimersByTimeAsync(3001);
|
|
expect(runDatabaseBackup).toHaveBeenCalledOnce();
|
|
expect(execute).not.toHaveBeenCalled();
|
|
canRun = true;
|
|
await vi.advanceTimersByTimeAsync(1000);
|
|
await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0));
|
|
expect(runDatabaseBackup).toHaveBeenCalledTimes(2);
|
|
expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now());
|
|
} finally { await coordinator.stop(); vi.useRealTimers(); }
|
|
});
|
|
|
|
it("fails closed if the storage path is no longer a directory", async () => {
|
|
await rm(directory, { recursive: true }); await writeFile(directory, "mount unavailable");
|
|
await expect(owner()("idle")).rejects.toThrow();
|
|
expect(runDatabaseBackup).not.toHaveBeenCalled();
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
});
|