Files
PaperClipAI/server/src/__tests__/native-workspace-sync-retention.test.ts
T
DottaandClaude Opus 5.5 beec70ad9f fix(native): judge sync-retention run processes with the shared run-process prover (#20025)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run syncs its workspace into a local folder. A retention
job deletes old folders, but never one whose run's adapter process may
still write it
> - That job had its own liveness check. It probed the recorded PID in
this PID space, whatever host recorded it, and compared process start
times to the millisecond
> - So a dead PID here could delete the synced files of a process still
running on another host. And a live process whose recorded start
differed by under a second (`ps lstart` has 1 s precision) counted as a
recycled PID, so its folder could be deleted too
> - The durable ownership plan proves owners by identity with one shared
prover. The lost-process reapers already use it for exactly this run
identity (#20004)
> - This pull request makes retention use that prover, so anything but a
proof the process is gone keeps the folder
> - The benefit is that preserved files are no longer deleted on a
wrong-host or sub-second-skew judgment

## Linked Issues or Issue Description

Refs #19831 (plan, Phase 2), #19838 (inventory: "One fact, many
proofs"). Phase 2's native agent flagged `runProcessMayBeLive` in
`native-workspace-sync-retention.ts` as still bespoke after #19997.

**What happened?**
`runProcessMayBeLive` ignored `process_host_hash` and required an exact
start-time match.

**Expected behavior**
The same verdict as the lost-process reapers (`proveRunProcessGone`):
- the PID is judged only in its recorded PID space;
- rows saved before `process_host_hash` keep the one-release local
fallback on single-host servers;
- a reused PID needs a definitely later start;
- EPERM is alive.

## What Changed

- `native-workspace-sync-retention.ts`: `runProcessMayBeLive` calls
`proveRunProcessGone` (reading `process_host_hash`). The injectable
retention probe is adapted to the shared probe for tests. Remote-only
runs keep their existing shortcut.

## Verification

- `native-workspace-sync-retention.test.ts`: 59 pass. Two new tests fail
on `master` and pass here:
- a run recorded in another PID space is kept, even if that PID is dead
here;
  - a live PID whose start time is within the 1 s precision is kept.

The new "deletes a terminal run whose process, recorded in this PID
space, has exited" and all existing retention cases (recycled PID, dead
process, destroyed sandbox, live group) pass.
- The server package typecheck passes.

## Risks

- Folders of runs recorded on another host are now kept until their
other owners settle, rather than deleted on a local PID probe. That is
the safe direction; the folders are bounded by the existing retention
limits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 05:32:46 -06:00

686 lines
29 KiB
TypeScript

import { currentProcessSpaceHash } from "../services/controller-process-identity.js";
import { randomUUID } from "node:crypto";
import os from "node:os";
import path from "node:path";
import { promises as fs } from "node:fs";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { sql } from "drizzle-orm";
import {
agents,
companies,
createDb,
environmentLeases,
heartbeatRuns,
issueRecoveryActions,
issues,
nativeRunFinalizations,
} from "@paperclipai/db";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./helpers/embedded-postgres.js";
import { nativeWorkspaceFinalizationLockKey } from "../services/native-runtime/native-workspace-finalization-ownership.ts";
import {
NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS,
nativeWorkspaceSyncRetentionService,
registerNativeWorkspaceSyncRetentionWork,
} from "../services/native-runtime/native-workspace-sync-retention.ts";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.ts";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.ts";
import {
nativeWorkspaceSyncInternals,
nativeWorkspaceSyncStateRoot,
} from "../services/native-runtime/native-workspace-sync.ts";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
console.warn(
`Skipping embedded Postgres native workspace sync retention tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
}
const DAY_MS = 24 * 60 * 60 * 1000;
async function exists(target: string) {
return fs.lstat(target).then(() => true, () => false);
}
type SeedInput = {
runStatus?: string;
finishedDaysAgo?: number | null;
folderAgeDays?: number;
reference?: boolean;
owner?: boolean;
processPid?: number | null;
processGroupId?: number | null;
processStartedAt?: Date | null;
processHostHash?: string | null;
leaseProvider?: string | null;
controllerLeaseDaysFromNow?: number | null;
finalization?: {
phase: string;
recoveryState?: string | null;
leaseOwner?: string | null;
nextAttemptDaysFromNow?: number | null;
} | null;
leaseMetadata?: Record<string, unknown>;
leasePolicy?: string;
recoveryAction?: { cause?: string; status?: string; evidence?: Record<string, unknown> };
resultJson?: Record<string, unknown>;
};
describeEmbeddedPostgres("native workspace sync retention", () => {
let db!: ReturnType<typeof createDb>;
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
let stateRoot = "";
const now = new Date();
beforeAll(async () => {
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-sync-retention-");
db = createDb(tempDb.connectionString);
}, 20_000);
afterEach(async () => {
livePids = new Map();
await db.delete(issueRecoveryActions);
await db.delete(nativeRunFinalizations);
await db.delete(environmentLeases);
await db.delete(heartbeatRuns);
await db.delete(issues);
await db.delete(agents);
await db.delete(companies);
if (stateRoot) await fs.rm(stateRoot, { recursive: true, force: true });
stateRoot = "";
});
afterAll(async () => {
await tempDb?.cleanup();
});
async function root() {
if (!stateRoot) stateRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-native-workspace-sync-"));
return stateRoot;
}
async function writeFolder(name: string, ageDays: number) {
const folder = path.join(await root(), name);
await fs.mkdir(folder, { recursive: true });
const files = ["descriptor.json", "manifest.sqlite", "workspace-seed.tar", "git-seed.tar"];
for (const file of files) await fs.writeFile(path.join(folder, file), "x".repeat(4096));
const at = new Date(now.getTime() - ageDays * DAY_MS);
for (const file of files) await fs.utimes(path.join(folder, file), at, at);
await fs.utimes(folder, at, at);
return folder;
}
async function seed(input: SeedInput = {}) {
const companyId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Sync", issuePrefix: `S${companyId.slice(0, 6)}` });
const [agent] = await db.insert(agents).values({ companyId, name: "Coder" }).returning();
const [issue] = await db.insert(issues).values({ companyId, title: "Work", status: "done" }).returning();
const runId = randomUUID();
const leaseId = randomUUID();
const runStatus = input.runStatus ?? "succeeded";
const finishedDaysAgo = input.finishedDaysAgo === undefined ? 10 : input.finishedDaysAgo;
const profile: Record<string, unknown> = {};
if (input.reference ?? true) {
profile.nativeWorkspaceSync = {
schema: "paperclip.native-workspace-sync/v3",
state: "finalized",
descriptorSha256: "a".repeat(64),
baselineSha256: "b".repeat(64),
finalHostSha256: null,
workspaceId: randomUUID(),
leaseId,
providerLeaseId: "provider-lease",
remoteCwd: "/workspace",
};
}
if (input.owner) profile.nativeWorkspaceFinalizationOwner = { token: randomUUID() };
await db.insert(heartbeatRuns).values({
id: runId,
companyId,
agentId: agent!.id,
status: runStatus,
issueId: issue!.id,
nativeIssueId: issue!.id,
finishedAt: finishedDaysAgo === null ? null : new Date(now.getTime() - finishedDaysAgo * DAY_MS),
processPid: input.processPid ?? null,
processGroupId: input.processGroupId ?? null,
processStartedAt: input.processStartedAt ?? null,
processHostHash: input.processHostHash ?? null,
controllerLeaseExpiresAt: input.controllerLeaseDaysFromNow == null
? null
: new Date(now.getTime() + input.controllerLeaseDaysFromNow * DAY_MS),
runnerProfileJson: profile,
resultJson: input.resultJson ?? null,
});
const finalization = input.finalization === undefined ? { phase: "committed" } : input.finalization;
if (finalization) {
await db.insert(nativeRunFinalizations).values({
runId,
companyId,
issueId: issue!.id,
phase: finalization.phase,
recoveryState: finalization.recoveryState ?? null,
leaseOwner: finalization.leaseOwner ?? null,
nextAttemptAt: finalization.nextAttemptDaysFromNow == null
? null
: new Date(now.getTime() + finalization.nextAttemptDaysFromNow * DAY_MS),
});
}
if (input.leaseMetadata || input.leasePolicy || input.leaseProvider !== undefined) {
await db.insert(environmentLeases).values({
id: leaseId,
companyId,
heartbeatRunId: runId,
status: "released",
provider: input.leaseProvider ?? null,
leasePolicy: input.leasePolicy ?? "ephemeral",
metadata: input.leaseMetadata ?? {},
});
}
if (input.recoveryAction) {
await db.insert(issueRecoveryActions).values({
companyId,
sourceIssueId: issue!.id,
kind: "active_run_watchdog",
ownerType: "system",
cause: input.recoveryAction.cause ?? "native_continuation_requires_reconciliation",
fingerprint: randomUUID(),
status: input.recoveryAction.status ?? "active",
nextAction: "Recover",
evidence: { runId, systemRecovery: { state: "complete" }, ...input.recoveryAction.evidence },
});
}
const folder = await writeFolder(runId, input.folderAgeDays ?? 9);
return { companyId, runId, folder };
}
const processStart = new Date(now.getTime() - 20 * DAY_MS);
let livePids = new Map<number, string | null>();
const processProbe = {
mayBeAlive: (pid: number) => livePids.has(pid),
startedAt: async (pid: number) => livePids.get(pid) ?? null,
};
function service() {
return nativeWorkspaceSyncRetentionService(db, { stateRoot, now: () => now, processProbe });
}
async function expectKept(input: SeedInput, reason: string) {
const { folder } = await seed(input);
const result = await service().tick();
expect(result.expired).toBe(0);
expect(result.skipped[reason]).toBe(1);
expect(await exists(folder)).toBe(true);
}
it("deletes a settled committed run's folder after the retention window", async () => {
const { folder } = await seed();
const result = await service().tick();
expect(result).toMatchObject({ checked: 1, candidates: 1, expired: 1, failed: 0 });
expect(result.reclaimedBytes).toBeGreaterThan(0);
expect(await exists(folder)).toBe(false);
expect((await fs.readdir(stateRoot)).filter((name) => name.startsWith(".expired-"))).toEqual([]);
});
it("deletes committed runs whose recovery state was left behind without a lease", async () => {
const { folder } = await seed({ finalization: { phase: "committed", recoveryState: "resuming_session" } });
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("deletes blocked terminal failures with no recovery owner", async () => {
const { folder } = await seed({
runStatus: "failed",
finalization: { phase: "terminal_failure", recoveryState: "blocked" },
recoveryAction: { status: "resolved", cause: "native_workspace_sync_out_unrecoverable" },
});
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("deletes orphaned folders and unreferenced settled runs", async () => {
const orphan = await writeFolder(randomUUID(), 9);
const { folder } = await seed({ reference: false, finalization: { phase: "observed" } });
const result = await service().tick();
expect(result.expired).toBe(2);
expect(await exists(orphan)).toBe(false);
expect(await exists(folder)).toBe(false);
});
it("keeps folders with recent files or recently finished runs", async () => {
const recentFiles = await seed({ folderAgeDays: 2 });
const recentRun = await seed({ finishedDaysAgo: 3 });
const result = await service().tick();
expect(result.expired).toBe(0);
expect(result.skipped).toMatchObject({ recent_files: 1, recently_finished: 1 });
expect(await exists(recentFiles.folder)).toBe(true);
expect(await exists(recentRun.folder)).toBe(true);
});
it("keeps runs that are not settled", async () => {
await expectKept({ runStatus: "running", finishedDaysAgo: null }, "run_unsettled");
});
it("keeps runs whose local process is still alive", async () => {
livePids.set(1234, processStart.toISOString());
await expectKept({ processPid: 1234, processStartedAt: processStart, leaseProvider: "local" }, "process_live");
});
it("keeps runs whose local process is alive with an unreadable start time", async () => {
livePids.set(1234, null);
await expectKept({ processPid: 1234, processStartedAt: processStart }, "process_live");
});
it("keeps runs whose local process group is still alive", async () => {
livePids.set(-4321, null);
await expectKept({ processPid: 4321, processGroupId: 4321 }, "process_live");
});
it("keeps a running run that still records a process", async () => {
livePids.set(1234, processStart.toISOString());
await expectKept({ runStatus: "running", finishedDaysAgo: null, processPid: 1234 }, "run_unsettled");
});
it("deletes a terminal run whose recorded local process has exited", async () => {
const { folder } = await seed({
runStatus: "failed",
finalization: { phase: "terminal_failure" },
processPid: 1234,
processGroupId: 1234,
processStartedAt: processStart,
leaseProvider: "local",
});
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("keeps a run whose process was recorded in another PID space, even if that pid is dead here", async () => {
await expectKept({ processPid: 1234, processStartedAt: processStart, processHostHash: "another-host-pid-space",
leaseProvider: "local" }, "process_live");
});
it("deletes a terminal run whose process, recorded in this PID space, has exited", async () => {
const { folder } = await seed({ processPid: 1234, processStartedAt: processStart,
processHostHash: currentProcessSpaceHash(), leaseProvider: "local" });
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("keeps a run whose live pid's start time matches within the 1 s start-time precision", async () => {
livePids.set(1234, new Date(processStart.getTime() + 400).toISOString());
await expectKept({ processPid: 1234, processStartedAt: processStart, leaseProvider: "local" }, "process_live");
});
it("deletes a terminal run whose recorded pid was recycled by another process", async () => {
livePids.set(1234, new Date(now.getTime() - DAY_MS).toISOString());
const { folder } = await seed({ processPid: 1234, processStartedAt: processStart });
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("deletes a terminal run whose stale pid came from a destroyed sandbox", async () => {
livePids.set(1234, processStart.toISOString());
livePids.set(-1234, null);
const { folder } = await seed({
runStatus: "cancelled",
finalization: { phase: "terminal_failure" },
processPid: 1234,
processGroupId: 1234,
processStartedAt: processStart,
leaseProvider: "daytona",
leaseMetadata: { remoteExecutionTermination: { state: "destroyed" } },
});
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("keeps runs with a live controller lease", async () => {
await expectKept({ controllerLeaseDaysFromNow: 1 }, "controller_live");
});
it("keeps runs with a finalization owner receipt, even without a reference", async () => {
await expectKept({ owner: true, reference: false }, "finalization_owner");
});
it("keeps unsettled finalization phases", async () => {
await expectKept({ finalization: { phase: "observed" } }, "finalization_unsettled");
});
it("keeps retryable failures without a result", async () => {
await expectKept({ finalization: { phase: "retryable_failure" } }, "finalization_unsettled");
});
it("keeps finalizations with a live lease owner", async () => {
await expectKept({ finalization: { phase: "committed", leaseOwner: "controller" } }, "finalization_leased");
});
it("keeps finalizations with a scheduled retry", async () => {
await expectKept(
{ finalization: { phase: "terminal_failure", nextAttemptDaysFromNow: 1 } },
"finalization_retry_scheduled",
);
});
it.each(["awaiting_evidence", "awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"])(
"keeps terminal failures in recovery state %s",
async (recoveryState) => {
await expectKept({ finalization: { phase: "terminal_failure", recoveryState } }, "finalization_recovering");
},
);
it.each(["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"])(
"keeps runs with an open %s recovery action",
async (cause) => {
await expectKept({ recoveryAction: { cause, status: "escalated" } }, "export_recovery");
},
);
it("keeps runs with an open owner-unverified recovery action", async () => {
await expectKept(
{ recoveryAction: { cause: "native_workspace_finalization_owner_unverified" } },
"owner_unverified",
);
});
it("keeps runs whose export resume is recorded on the lease", async () => {
await expectKept({ leaseMetadata: { nativeWorkspaceExportResume: { attempt: 1 } } }, "export_resume");
});
it("keeps runs whose lease cleanup is in flight", async () => {
await expectKept({ leaseMetadata: { pendingCleanupInFlight: true } }, "export_resume");
});
it.each([
{ systemRecovery: { state: "scheduled" } },
{ systemRecovery: { state: "running" } },
{ systemRecovery: { state: "awaiting_evidence" } },
{ workspaceRecovery: { status: "queued" } },
{ workspaceRecovery: { status: "recovering" } },
])("keeps retained recovery %j on a lease that was not destroyed", async (evidence) => {
await expectKept(
{ leaseMetadata: { systemRecoveryAdoption: {} }, recoveryAction: { status: "resolved", evidence } },
"retained_recovery",
);
});
it("deletes when the retained lease was destroyed", async () => {
const { folder } = await seed({
leaseMetadata: { remoteExecutionTermination: { state: "destroyed" } },
recoveryAction: { status: "resolved", evidence: { systemRecovery: { state: "scheduled" } } },
});
expect((await service().tick()).expired).toBe(1);
expect(await exists(folder)).toBe(false);
});
it("keeps runs that historical recovery adoption has not reached", async () => {
await expectKept({ leasePolicy: "retain_on_failure" }, "recovery_backfill_pending");
});
it("keeps runs with an unadopted export recovery action", async () => {
const { runId, folder } = await seed({
recoveryAction: { status: "resolved", cause: "native_workspace_sync_out_unrecoverable" },
});
await db.execute(sql`update issue_recovery_actions set evidence = evidence - 'systemRecovery'
where evidence->>'runId' = ${runId}`);
const result = await service().tick();
expect(result.skipped.recovery_backfill_pending).toBe(1);
expect(await exists(folder)).toBe(true);
});
it("never touches generations, non-UUID names, symlinks, or files", async () => {
const base = await root();
const generations = await writeFolder("generations", 30);
const other = await writeFolder("not-a-run", 30);
const target = await writeFolder(".outside", 30);
const linkName = randomUUID();
await fs.symlink(target, path.join(base, linkName));
const fileName = randomUUID();
await fs.writeFile(path.join(base, fileName), "x");
const old = new Date(now.getTime() - 30 * DAY_MS);
await fs.utimes(path.join(base, fileName), old, old);
const result = await service().tick();
expect(result.expired).toBe(0);
expect(result.skipped.unsafe_path).toBe(2);
for (const entry of [generations, other, target, path.join(base, linkName), path.join(base, fileName)]) {
expect(await exists(entry)).toBe(true);
}
});
it("deletes nothing during a dry run", async () => {
const { folder } = await seed();
const leftover = await writeFolder(`.expired-${randomUUID()}`, 1);
const result = await nativeWorkspaceSyncRetentionService(db, { stateRoot, now: () => now, dryRun: true }).tick();
expect(result).toMatchObject({ candidates: 1, expired: 0, leftoversRemoved: 0 });
expect(await exists(folder)).toBe(true);
expect(await exists(leftover)).toBe(true);
});
it("bounds leftover removal by the batch limits", async () => {
const leftovers: string[] = [];
for (let index = 0; index < 3; index++) leftovers.push(await writeFolder(`.expired-${randomUUID()}`, 0));
const folder = await writeFolder(randomUUID(), 9);
expect(await service().tick({ shouldContinue: () => false })).toMatchObject({ leftoversRemoved: 0, truncated: true });
const retention = service();
const first = await retention.tick({ maxRemovals: 2 });
expect(first).toMatchObject({ leftoversRemoved: 2, expired: 1, truncated: true });
expect(await exists(folder)).toBe(false);
const rest = await retention.tick({ maxRemovals: 2 });
expect(rest).toMatchObject({ leftoversRemoved: 1, expired: 0, truncated: false });
for (const leftover of leftovers) expect(await exists(leftover)).toBe(false);
});
it("keeps expiring run folders and moves past leftovers that cannot be removed", async () => {
const stuck: string[] = [];
for (let index = 0; index < 2; index++) {
const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
await fs.chmod(leftover, 0o500);
stuck.push(leftover);
}
const folder = await writeFolder(randomUUID(), 9);
const retention = service();
try {
const first = await retention.tick({ maxRemovals: 1 });
expect(first).toMatchObject({ failed: 1, expired: 1, truncated: true });
expect(await exists(folder)).toBe(false);
const second = await retention.tick({ maxRemovals: 1 });
expect(second).toMatchObject({ failed: 1, truncated: false });
} finally {
for (const leftover of stuck) await fs.chmod(leftover, 0o700);
}
});
it("removes leftover expired folders", async () => {
const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
const retiring = await writeFolder(`.retiring-${randomUUID()}`, 30);
const result = await service().tick();
expect(result.leftoversRemoved).toBe(1);
expect(await exists(leftover)).toBe(false);
expect(await exists(retiring)).toBe(true);
});
it("skips a run while its finalization ownership lock is held", async () => {
const { companyId, runId, folder } = await seed();
let result: Awaited<ReturnType<ReturnType<typeof service>["tick"]>> | undefined;
await db.transaction(async (tx) => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(companyId, runId)}, 0))`);
result = await service().tick();
});
expect(result?.skipped.finalization_locked).toBe(1);
expect(await exists(folder)).toBe(true);
expect((await service().tick()).expired).toBe(1);
});
it("bounds each tick and resumes from its cursor", async () => {
const first = await writeFolder(randomUUID(), 9);
const second = await writeFolder(randomUUID(), 9);
const retention = service();
const initial = await retention.tick({ maxRemovals: 1 });
expect(initial).toMatchObject({ expired: 1, truncated: true });
const rest = await retention.tick({ maxRemovals: 1 });
expect(rest).toMatchObject({ expired: 1, truncated: false });
expect(await exists(first)).toBe(false);
expect(await exists(second)).toBe(false);
});
it("stops when asked to", async () => {
const folder = await writeFolder(randomUUID(), 9);
const result = await service().tick({ shouldContinue: () => false });
expect(result).toMatchObject({ checked: 0, truncated: true });
expect(await exists(folder)).toBe(true);
});
describe("scheduled worker", () => {
function register() {
let queue = "";
let task!: { run: (signal: AbortSignal) => Promise<unknown>; nextRunAt?: () => number | null };
const coordinator = {
register: (registeredQueue: string, registered: typeof task) => {
queue = registeredQueue;
task = registered;
return { ready: Promise.resolve(), wake: () => {} };
},
};
const onResult = vi.fn();
registerNativeWorkspaceSyncRetentionWork(coordinator as never, service(), { onResult });
return { queue: () => queue, task: () => task, onResult };
}
it("goes quiet with no folders and schedules the next real deadline", async () => {
await root();
const worker = register();
expect(worker.queue()).toBe(DELIVERY_QUEUES.nativeWorkspaceSyncRetention);
await worker.task().run(new AbortController().signal);
expect(worker.task().nextRunAt?.()).toBeNull();
await seed({ finishedDaysAgo: 3, folderAgeDays: 4 });
await worker.task().run(new AbortController().signal);
expect(worker.task().nextRunAt?.()).toBe(now.getTime() + 4 * DAY_MS);
expect(worker.onResult).toHaveBeenCalledTimes(2);
});
it("waits the retry delay when a leftover cannot be removed", async () => {
const base = await root();
const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
await fs.writeFile(path.join(leftover, "nested"), "x");
await fs.chmod(leftover, 0o500);
try {
const worker = register();
await worker.task().run(new AbortController().signal);
expect(worker.onResult.mock.calls[0]![0]).toMatchObject({ failed: 1 });
expect(worker.task().nextRunAt?.()).toBeGreaterThanOrEqual(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
} finally {
await fs.chmod(leftover, 0o700);
}
expect(await exists(base)).toBe(true);
});
it("wakes the quiet coordinator when a run creates its folder", async () => {
const home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-sync-home-"));
const originalHome = process.env.PAPERCLIP_HOME;
const originalInstance = process.env.PAPERCLIP_INSTANCE_ID;
process.env.PAPERCLIP_HOME = home;
process.env.PAPERCLIP_INSTANCE_ID = "retention";
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} });
const onResult = vi.fn();
const worker = registerNativeWorkspaceSyncRetentionWork(
coordinator,
nativeWorkspaceSyncRetentionService(db, { stateRoot: nativeWorkspaceSyncStateRoot() }),
{ onResult },
);
try {
await worker.ready;
expect(onResult).toHaveBeenCalledTimes(1);
expect(coordinator.nextWakeAt()).toBeNull();
await nativeWorkspaceSyncInternals.writeDescriptor({
schema: "paperclip.native-workspace-sync/v3",
state: "prepared",
baselineSha256: "b".repeat(64),
finalHostSha256: null,
binding: { runId: randomUUID(), workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "lease", remoteCwd: "/workspace" },
} as never);
await vi.waitFor(() => expect(onResult).toHaveBeenCalledTimes(2));
await vi.waitFor(() => expect(coordinator.nextWakeAt()).not.toBeNull());
const wakeAt = coordinator.nextWakeAt()!;
expect(wakeAt).toBeGreaterThan(Date.now() + 6 * DAY_MS);
} finally {
worker.unsubscribe();
await coordinator.stop();
if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = originalHome;
if (originalInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
else process.env.PAPERCLIP_INSTANCE_ID = originalInstance;
await fs.rm(home, { recursive: true, force: true });
}
});
it("retries soon when the finalization lock was held", async () => {
const { companyId, runId } = await seed();
const worker = register();
const before = Date.now();
await db.transaction(async (tx) => {
await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(companyId, runId)}, 0))`);
await worker.task().run(new AbortController().signal);
});
const next = worker.task().nextRunAt?.() ?? 0;
expect(next).toBeGreaterThanOrEqual(before + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
expect(next).toBeLessThanOrEqual(Date.now() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
});
});
describe("nextDeadlineAt", () => {
it("is null without folders", async () => {
await root();
await writeFolder("generations", 0);
expect(await service().nextDeadlineAt()).toBeNull();
});
it("is null when the state root does not exist", async () => {
expect(await nativeWorkspaceSyncRetentionService(db, {
stateRoot: path.join(os.tmpdir(), `missing-${randomUUID()}`),
now: () => now,
}).nextDeadlineAt()).toBeNull();
});
it("retries leftover expired folders after the retry delay", async () => {
await writeFolder(`.expired-${randomUUID()}`, 0);
expect(await service().nextDeadlineAt()).toBe(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
});
it("stops scanning when asked to and retries later", async () => {
await writeFolder(randomUUID(), 1);
expect(await service().nextDeadlineAt({ shouldContinue: () => false }))
.toBe(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
});
it("carries the earliest deadline across batches", async () => {
for (let index = 0; index < 205; index++) await writeFolder(randomUUID(), 1);
await writeFolder("ffffffff-ffff-4fff-8fff-ffffffffffff", 6);
const deadline = await service().nextDeadlineAt();
expect(Math.abs(deadline! - (now.getTime() + DAY_MS))).toBeLessThan(1000);
});
it("is the earliest settled run finish plus the retention window", async () => {
await seed({ finishedDaysAgo: 3, folderAgeDays: 4 });
await seed({ finishedDaysAgo: 2, folderAgeDays: 4 });
expect(await service().nextDeadlineAt()).toBe(now.getTime() + 4 * DAY_MS);
});
it("uses the folder's newest file for orphans", async () => {
await writeFolder(randomUUID(), 5);
const deadline = await service().nextDeadlineAt();
expect(Math.abs(deadline! - (now.getTime() + 2 * DAY_MS))).toBeLessThan(1000);
});
it("rechecks unsettled or kept folders one retention window later", async () => {
await seed({ runStatus: "running", finishedDaysAgo: null, folderAgeDays: 30 });
await seed({ finalization: { phase: "observed" } });
expect(await service().nextDeadlineAt()).toBe(now.getTime() + 7 * DAY_MS);
});
});
});