Files
PaperClipAI/server/src/__tests__/idle-local-work.test.ts
T
DottaandPaperclip 7c016c049b perf(workspaces): wake terminal cleanup from committed changes (#16282)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use execution workspaces to retain their files and Git
branches.
> - A reaper closes eligible workspaces after their task tree is
terminal.
> - The reaper scanned every heartbeat tick and read Git before cheap
eligibility checks.
> - Permanent scans also query idle databases when no cleanup work
exists.
> - This pull request wakes cleanup after transaction settlement and
restores bounded work at startup.
> - The benefit is fewer scans while fresh cleanup guards continue to
protect retained work.

## Linked Issues or Issue Description

Related work: Refs #13619, Refs #11753. Those PRs bound scans or back
off unchanged candidates. This PR uses the transaction work coordinator
to remove the permanent reaper tick. It overlaps their early
terminal-state filtering but does not change their runtime restart
behavior or introduce a second Git scheduler cache.

**What existing behavior does this improve?**

Terminal workspace cleanup and Git inspection for workspace detail
reads.

**Subsystem affected**

Server orchestration, execution workspaces, and idle sleep admission.

**Current behavior**

The heartbeat tick scans workspace candidates every thirty seconds. It
runs Git before rejecting nonterminal trees, cooldowns, and active runs.
Display inspections expire after five seconds.

**Proposed behavior**

Recover workspace rows in bounded startup pages. Wake after relevant
issue, workspace, and pull-request work-product transactions settle.
Schedule the terminal tree cooldown or a five-minute eligibility retry.
Back off unchanged candidates across unrelated wakes, and invalidate
their backoff after relevant committed changes. Park empty and
nonterminal-only lanes. Reuse display inspections for thirty seconds.
Read fresh Git before destructive cleanup.

**Reason and benefit**

Avoid recurring database and Git work when no cleanup is possible.
Retain transaction admission holds, restart recovery, and fresh deletion
checks. Retention eligibility can wait for the next host wake; active
and failed recovery work still blocks sleep.

**Breaking changes**

No schema or API shape changes. Display Git state can lag by thirty
seconds. Cleanup eligibility retries can take five minutes.

## What Changed

- Add a terminal-workspace worker to the existing transaction work
coordinator and scheduler.
- Signal relevant writes before mutation on the owning transaction.
Outer transaction settlement wakes cleanup.
- Derive cooldown deadlines from the full company-scoped terminal task
tree. Continue bounded recovery pages without a permanent sweep.
- Bound unchanged candidate inspections to five minutes, retain at most
256 entries, and invalidate affected entries with scoped transaction
hints. Repeated wakes preserve the earliest inspection deadline.
- Collect deadlines from task trees already read by bounded pages.
Restore the cursor after a failed page and rescan commits received
during a rotation.
- Check terminal state, cooldown, and active runs before Git. Preserve
lifecycle locks, generation fences, and fresh final checks.
- Keep failed or incomplete recovery visible to idle admission. Preserve
the passive retention exception.
- Extend the bounded display cache to thirty seconds and document the
scheduling and wake limits.

## Verification

- Focused workspace, coordinator, cache, and local sleep suites: 111
tests passed with real local PostgreSQL and Git worktrees where
applicable.
- Final worker, display cache, and local sleep suites: 28 tests passed.
- The regressions prove three ineligible workspaces invoke zero Git
scans, cached clean display state cannot remove new untracked files, and
real producer notifications wait for outer transaction settlement.
- `pnpm -r typecheck` and `pnpm build` passed locally. Map and
module-boundary checks passed.
- Work-product and sleep-safety suites: 221 tests passed. Reopen and
issue-tree control suites: 46 tests passed. Final backoff, workspace,
worker, and work-product suites: 87 tests passed.
- Final workspace and worker suites: 81 tests passed. Deadline selection
performs zero database reads. A failed full page retries its unread
terminal workspace before recovery can park.
- Full CI coverage passed on head `2c397970f7`: general tests,
serialized server suites, runner verification, E2E shards, typecheck,
build, and the final verification gate. The duplicate serial local `pnpm
test:run` was stopped after full CI coverage passed; it is not reported
as a completed local pass.
- Greptile: 5/5 with both findings resolved. The security scan also
passed. No failing or pending PR checks remain.

## Risks

- Display state can be stale for thirty seconds. Destructive guards
bypass the display cache.
- Undelivered or otherwise ineligible terminal work retries after five
minutes instead of each heartbeat tick.
- Transaction signals cover the owning `createDb` scope. Direct SQL and
independent server processes require startup recovery.
- Local cooldown timers are maintenance hints. They do not establish an
external wake alarm or whole-instance sleep safety. Existing blockers
remain in force; failed bounded recovery keeps the host awake.

## Model Used

OpenAI GPT-6, model ID `gpt-6`, through Codex. Reasoning, repository
inspection, shell execution, and browser tools assisted the change. The
runtime does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 08:28:45 -05:00

121 lines
6.5 KiB
TypeScript

import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { inspectIdleSpool, inspectIdleQuotaRecovery, idleQuotaRecoveryPath, readIdleLocalWork, markIdleIngressTracked, markIdleStartupComplete, registerIdleSpoolDirectory, registerIdleLocalWorkProbe } from "../services/idle-local-work.js";
let root: string;
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "idle-spool-")); });
afterEach(async () => { vi.restoreAllMocks(); vi.unstubAllEnvs(); await fs.rm(root, { recursive: true, force: true }); });
describe("idle cleanup inspection", () => {
it("refuses to authorize sleep until startup and ingress tracking are ready", async () => {
expect(await readIdleLocalWork()).toBe("unknown");
});
it("accepts absent and empty directories", async () => {
expect(await inspectIdleSpool(path.join(root, "absent"))).toBe("none");
expect(await inspectIdleSpool(root)).toBe("none");
});
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])("retains every spool entry: %s", async name => {
await fs.writeFile(path.join(root, name), "not valid JSON");
expect(await inspectIdleSpool(root)).toBe("present");
await fs.rm(path.join(root, name));
expect(await inspectIdleSpool(root)).toBe("none");
});
it("fails closed on a non-directory, dangling symlink, or read failure", async () => {
const file = path.join(root, "file"), link = path.join(root, "link");
await fs.writeFile(file, "");
await fs.symlink(path.join(root, "missing"), link);
expect(await inspectIdleSpool(file)).toBe("unknown");
expect(await inspectIdleSpool(path.join(file, "nested"))).toBe("unknown");
expect(await inspectIdleSpool(link)).toBe("unknown");
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
expect(await inspectIdleSpool(root)).toBe("unknown");
});
it("retains failed local recovery and fails closed on probe errors", async () => {
vi.stubEnv("PAPERCLIP_HOME", root);
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
let pending = true;
const unregister = registerIdleLocalWorkProbe(() => pending ? "present" : "none");
try {
expect(await readIdleLocalWork()).toBe("present");
pending = false;
expect(await readIdleLocalWork()).toBe("none");
} finally { unregister(); }
const removeBroken = registerIdleLocalWorkProbe(() => { throw new Error("unknown deadline"); });
try { expect(await readIdleLocalWork()).toBe("unknown"); }
finally { removeBroken(); }
expect(await readIdleLocalWork()).toBe("none");
});
it("checks custom spool directories as well as the default paths", async () => {
vi.stubEnv("PAPERCLIP_HOME", root);
vi.stubEnv("SANDBOX_ORPHAN_CLEANUP_SPOOL_DIR", path.join(root, "orphans"));
const custom = path.join(root, "custom-accounting");
registerIdleSpoolDirectory(custom);
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
expect(await readIdleLocalWork()).toBe("none");
markIdleStartupComplete({ scheduledBackups: true });
expect(await readIdleLocalWork()).toBe("present");
expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("none");
markIdleStartupComplete({ scheduledBackups: false });
await fs.mkdir(custom);
await fs.writeFile(path.join(custom, "receipt.tmp"), "incomplete");
expect(await readIdleLocalWork()).toBe("present");
expect(await readIdleLocalWork({ backupCheckpoint: true })).toBe("present");
await fs.rm(custom, { recursive: true });
expect(await readIdleLocalWork()).toBe("none");
});
});
describe("idle credential recovery inspection", () => {
const company = "7238bc31-f3ca-42ba-b487-4122e77b8867";
it("allows missing roots and empty company directories after successful recovery", async () => {
expect(await inspectIdleQuotaRecovery(path.join(root, "missing"))).toBe("none");
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
await fs.mkdir(path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
});
it.each(["receipt.json", "malformed.json", "partial.tmp", "write.probe", "unexpected"])(
"retains every pending company entry without reading its contents: %s", async name => {
await fs.mkdir(path.join(root, company));
const file = path.join(root, company, name);
await fs.writeFile(file, "unreadable recovery fixture");
const read = vi.spyOn(fs, "readFile");
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
expect(read).not.toHaveBeenCalled();
await fs.rm(file);
expect(await inspectIdleQuotaRecovery(root)).toBe("none");
},
);
it("retains unexpected root entries, nested directories and directory symlinks", async () => {
await fs.mkdir(path.join(root, "unexpected"));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, "unexpected"), { recursive: true });
await fs.symlink(path.join(root, "missing"), path.join(root, company));
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
await fs.rm(path.join(root, company));
await fs.mkdir(path.join(root, company, "nested"), { recursive: true });
expect(await inspectIdleQuotaRecovery(root)).toBe("present");
});
it("fails closed on invalid roots and directory read errors", async () => {
const file = path.join(root, "file"), link = path.join(root, "link");
await fs.writeFile(file, ""); await fs.symlink(root, link);
expect(await inspectIdleQuotaRecovery(file)).toBe("unknown");
expect(await inspectIdleQuotaRecovery(link)).toBe("unknown");
vi.spyOn(fs, "opendir").mockRejectedValue(Object.assign(new Error("private path"), { code: "EACCES" }));
expect(await inspectIdleQuotaRecovery(root)).toBe("unknown");
});
it("keeps persisted credential recovery awake after the request and process work have ended", async () => {
vi.stubEnv("PAPERCLIP_HOME", root);
markIdleIngressTracked(); markIdleStartupComplete({ scheduledBackups: false });
const directory = path.join(idleQuotaRecoveryPath(), company);
await fs.mkdir(directory, { recursive: true });
const file = path.join(directory, "pending.json");
await fs.writeFile(file, "encrypted journal fixture");
expect(await readIdleLocalWork()).toBe("present");
await fs.rm(file);
expect(await readIdleLocalWork()).toBe("none");
});
});