mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
test(recovery): stop the idle-backstop suite starting real runs and bound every wait (#21924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The task recovery suites drive real recovery workers against an embedded database with a faked clock > - `issue-recovery-idle-backstop.test.ts` seeds 50 agents that take on-demand wakes, so recovery starts real `process` runs while timers are faked. A run then waits on timers that only move when the test advances, and the test waits for the run's passes with no time limit > - On master's readiness run forb4fa1c7fb1(run 43861616674, attempt 2), server shard 6/10 went silent inside this file and was cancelled at the 20-minute job limit. Locally the same test timed out at 30 minutes in larger runs, and then the next test's setup hook timed out at the database truncate > - This pull request stops the file's agents from starting runs, as #21867 did for the agent-row census, and puts a time limit on every wait, with a message that names what it waited for > - The benefit is that this suite can no longer hang a CI shard silently; a stuck step fails within 20 seconds and says why ## Linked Issues or Issue Description Refs #21867, #21872, #21905 **What happened?** Server shard 6/10 of the readiness run for masterb4fa1c7fb1(run 43861616674, attempt 2, job 133284921313) printed "running 199 of 1988 suites", logged a completion timing line for 197 files, and was then silent from 12:15:05 until it was cancelled at 12:24:24. `issue-recovery-idle-backstop.test.ts` started at 12:12:47 and never logged its completion line. Its last output was at 12:13:34, in "recovers a task whose hint was never sent within the backstop bound". Locally, the same test hit its 30-minute test timeout in two larger multi-file runs, and the following tests' `beforeEach` (`TRUNCATE companies CASCADE`) hit the 30-second hook timeout. The logs of those runs show real runs executing inside this test: a process run starting ("running without injected PAPERCLIP_API_KEY") and `LegacyControllerLeaseLostError` raised from a faked-timer deadline in `legacy-controller-lease.ts`. Cause: `seedRecoveryPopulation` creates agents with `wakeOnDemand: true`, and the held agent in this file too. When recovery finds a stranded task it wakes the agent, and the wake starts a real run at once (`startNextQueuedRunForAgent`). The run's legacy controller lease and other waits use timers that the test fakes, while lease checks use the database clock. So the run either loses its lease when the fake clock jumps, or waits on a frozen timer. Meanwhile `advance()` waited for every delivery pass with no time limit. Whether a pass ends up waiting on such a run depends on timing, which is why the hang is rare. **Expected behavior** No fixture agent in this suite can start a run, and every wait fails with a named message within a fixed time. **Steps to reproduce** Run `issue-recovery-idle-backstop.test.ts` and grep its output for "running without injected PAPERCLIP_API_KEY" or "Legacy controller lease lost": 11 to 13 occurrences per run on master. The hang itself is rare (see Verification). **Paperclip version or commit** `master` at9de343b3fe. ## What Changed - `helpers/recovery-population.ts`: optional `wakeOnDemand` (default `true`, so the other suites are unchanged). - `issue-recovery-idle-backstop.test.ts`: - The population and the held agent take no on-demand wakes. - The two tests that asserted a run was created now assert that recovery asked for a wake of the task. The wake queue records each one as skipped (`heartbeat.wakeOnDemand.disabled`), so the assertion still proves recovery found the task and acted. - Every wait has a 20-second real-time limit with a message naming the step: seeding, worker start, each simulated second, each set of pending delivery passes (listed by queue), each write and read. Per-test limit 2 minutes, down from 30. - `issue-recovery.test-fixture.ts`: records the queue of each pending pass (`pendingPassQueues()`), so a timed-out wait names the queues it was still waiting for. Not a product bug: no product code kept a transaction open after shutdown. The truncate timeouts came after the previous test timed out with real runs still in flight. ## Verification - Real runs started during one run of the file: 11 to 13 on master (two runs), 0 after. - The file alone: 4 passed (98 s). - Loop after the change: 15 runs (5 rounds of 3 in parallel): 15 passed, 0 hung or failed, 0 real run starts. - Before the change, my local reproduction rate was low: 3 failing multi-file runs out of about 10 large runs today (one silent hang, two with this test's 30-minute timeout). 0 of 14 targeted runs (alone, alone under load, and the 4-file combination) failed. So the loop shows the change holds; it cannot show a before/after hang count. The deterministic difference is the run starts above. - Master shard 6/10 forb4fa1c7fb1: see "What happened?". My local replay of the sharding script for that commit gave 201 of 2004 suites, against CI's 199 of 1988. The CI environment excludes some suites, so I identified the file from the job log, not from the replay. ## Risks - The two tests now prove that a wake was asked for, not that a run was created. Creating a run from a wake is covered by the wake-queue and dispatch suites, and these tests are about recovery finding the task. - Test-only change. No product code changed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3452639a81
commit
d216e413f3
3 files changed
+59
-30
No files matched your search
@@ -15,9 +15,11 @@ export type RecoveryPopulation = {
|
||||
|
||||
export async function seedRecoveryPopulation(db: Db, companyId: string, counts: {
|
||||
agents: number; inReview: number; inProgress?: number; todo?: number; recoveryActions?: number; terminated?: number;
|
||||
/** False keeps every agent from starting a run, for suites that fake timers. */
|
||||
wakeOnDemand?: boolean;
|
||||
}): Promise<RecoveryPopulation> {
|
||||
const config = { adapterType: "process", adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
|
||||
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } }, role: "engineer" };
|
||||
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: counts.wakeOnDemand ?? true } }, role: "engineer" };
|
||||
const [ceo] = await db.insert(agents).values({ ...config, companyId, name: "Chief", status: "idle" }).returning({ id: agents.id });
|
||||
const managers = await db.insert(agents).values(Array.from({ length: 6 }, (_, index) => ({ ...config, companyId,
|
||||
name: `Manager ${index}`, status: "idle", reportsTo: ceo!.id }))).returning({ id: agents.id });
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { setImmediate as immediate } from "node:timers/promises";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import { agents, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { agentWakeupRequests, agents, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { workSignalDebug } from "@paperclipai/db/work-signal-debug";
|
||||
import { ISSUE_RECOVERY_BACKSTOP_MS, ISSUE_RECOVERY_FULL_BACKSTOP_MS, notifyIssueGraphRecovery }
|
||||
from "../services/issue-recovery-notifications.js";
|
||||
@@ -14,16 +14,32 @@ import { seedRecoveryPopulation } from "./helpers/recovery-population.js";
|
||||
vi.hoisted(() => { process.env.PAPERCLIP_DEBUG_WORK_SIGNALS = "stranded-assignments:1"; });
|
||||
|
||||
const MINUTE = 60_000;
|
||||
const STEP_TIMEOUT_MS = 20_000;
|
||||
const TEST_TIMEOUT_MS = 120_000;
|
||||
const realSetTimeout = globalThis.setTimeout;
|
||||
const realClearTimeout = globalThis.clearTimeout;
|
||||
type Recorded = { at: number; name: string; fields: Record<string, unknown> };
|
||||
|
||||
/** Fails with a message naming what it waited for instead of hanging. */
|
||||
function bounded<T>(work: Promise<T>, waitingFor: () => string): Promise<T> {
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const expired = new Promise<never>((_, reject) => {
|
||||
timer = realSetTimeout(() => reject(new Error(`Timed out after ${STEP_TIMEOUT_MS} ms waiting for ${waitingFor()}`)), STEP_TIMEOUT_MS);
|
||||
});
|
||||
return Promise.race([work, expired]).finally(() => realClearTimeout(timer));
|
||||
}
|
||||
|
||||
describeIssueRecovery("stranded-task checks at idle for a staging-shaped company", () => {
|
||||
const state = issueRecoveryFixture({ meter: true });
|
||||
|
||||
async function advance(ms: number) {
|
||||
const until = Date.now() + ms;
|
||||
while (Date.now() < until) {
|
||||
await vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now()));
|
||||
await Promise.allSettled([...state.passes]); await immediate();
|
||||
const at = new Date(Date.now()).toISOString();
|
||||
await bounded(vi.advanceTimersByTimeAsync(Math.min(1000, until - Date.now())), () => `simulated time to advance past ${at}`);
|
||||
await bounded(Promise.allSettled([...state.passes]),
|
||||
() => `delivery passes started by simulated ${at} to settle: ${state.pendingPassQueues().join(", ") || "none"}`);
|
||||
await immediate();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -53,21 +69,29 @@ describeIssueRecovery("stranded-task checks at idle for a staging-shaped company
|
||||
const checkedAt = (events: Recorded[], issueId: string) =>
|
||||
events.find(event => event.name === "check" && event.fields.issueId === issueId)?.at ?? null;
|
||||
|
||||
/** The wakes recovery asked for on a task. No fixture agent takes on-demand
|
||||
* wakes, so each is recorded as skipped and no real run starts under the
|
||||
* faked clock (a run there could wait forever on a frozen timer). */
|
||||
const wakesFor = (companyId: string, issueId: string) => bounded(state.db.select({ id: agentWakeupRequests.id })
|
||||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`)), () => `the wake requests of task ${issueId} to be read`);
|
||||
|
||||
async function seed() {
|
||||
const f = await state.fixture();
|
||||
const f = await bounded(state.fixture(), () => "the company to be created");
|
||||
const population = await seedRecoveryPopulation(state.db, f.companyId, { agents: 50, inReview: 780, inProgress: 40, todo: 30,
|
||||
recoveryActions: 12, terminated: 3 });
|
||||
recoveryActions: 12, terminated: 3, wakeOnDemand: false });
|
||||
const [side] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Finished side task", status: "done" })
|
||||
.returning({ id: issues.id });
|
||||
const activity = () => state.db.transaction(tx => notifyIssueGraphRecovery(tx, { companyId: f.companyId, issueId: side!.id }));
|
||||
const activity = () => bounded(state.db.transaction(tx => notifyIssueGraphRecovery(tx, { companyId: f.companyId, issueId: side!.id })),
|
||||
() => "the unrelated activity hint to commit");
|
||||
return { ...f, population, activity };
|
||||
}
|
||||
|
||||
async function boot() {
|
||||
const f = await seed();
|
||||
const f = await bounded(seed(), () => "the staging-shaped population to be seeded");
|
||||
const events = record();
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"] });
|
||||
await state.start({});
|
||||
await bounded(state.start({}), () => "the recovery workers to start");
|
||||
await advance(10 * MINUTE);
|
||||
return { ...f, events, boot: summarize(events.splice(0)) };
|
||||
}
|
||||
@@ -77,7 +101,7 @@ describeIssueRecovery("stranded-task checks at idle for a staging-shaped company
|
||||
const [held] = await state.db.insert(agents).values({ companyId: f.companyId, name: "Held worker", role: "engineer", status: "paused",
|
||||
lifecycleHolds: ["manual"], adapterType: "process", reportsTo: f.population.agentIds[1], createdAt: old, updatedAt: old,
|
||||
adapterConfig: { command: process.execPath, args: ["-e", "0"], cwd: process.cwd() },
|
||||
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: true } } }).returning({ id: agents.id });
|
||||
runtimeConfig: { heartbeat: { enabled: false, intervalSec: 0, wakeOnDemand: false } } }).returning({ id: agents.id });
|
||||
const [waiting] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Waits for its agent", status: "todo",
|
||||
assigneeAgentId: held!.id, createdAt: old, updatedAt: old }).returning({ id: issues.id });
|
||||
return { held: held!, waiting: waiting! };
|
||||
@@ -103,37 +127,34 @@ describeIssueRecovery("stranded-task checks at idle for a staging-shaped company
|
||||
expect(between.totalChecks).toBe(0);
|
||||
expect(between.statements).toBeLessThan(2_000);
|
||||
expect(hour.checks.backstop ?? 0).toBeLessThanOrEqual(f.population.inReview.length + 100);
|
||||
}, 1_800_000);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
it("recovers a task whose hint was never sent within the backstop bound", async () => {
|
||||
const f = await boot();
|
||||
const worker = f.population.agentIds[7]!;
|
||||
const now = new Date();
|
||||
const [missed] = await state.db.insert(issues).values({ companyId: f.companyId, title: "Created without a hint", status: "todo",
|
||||
assigneeAgentId: worker, createdAt: now, updatedAt: now }).returning({ id: issues.id });
|
||||
const [missed] = await bounded(state.db.insert(issues).values({ companyId: f.companyId, title: "Created without a hint", status: "todo",
|
||||
assigneeAgentId: worker, createdAt: now, updatedAt: now }).returning({ id: issues.id }), () => "the task without a hint to be created");
|
||||
const window = await trickle(f, Math.ceil((ISSUE_RECOVERY_BACKSTOP_MS + MINUTE) / MINUTE) + 1);
|
||||
const checked = checkedAt(window.raw, missed!.id);
|
||||
expect(checked).not.toBeNull();
|
||||
expect(checked! - now.getTime()).toBeLessThanOrEqual(ISSUE_RECOVERY_BACKSTOP_MS + 2 * MINUTE);
|
||||
const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId),
|
||||
sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${missed!.id}`));
|
||||
expect(runs.length).toBeGreaterThan(0);
|
||||
}, 1_800_000);
|
||||
expect(await wakesFor(f.companyId, missed!.id), "recovery did not ask for a wake of the missed task").not.toHaveLength(0);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
it("re-checks a task whose only change is outside its own rows within the hourly full pass", async () => {
|
||||
const f = await boot();
|
||||
const { held, waiting } = await heldTask(f);
|
||||
await state.db.execute(sql`update agents set status = 'idle', lifecycle_holds = '[]'::jsonb where id = ${held.id}`);
|
||||
const { held, waiting } = await bounded(heldTask(f), () => "the held agent and its waiting task to be seeded");
|
||||
await bounded(state.db.execute(sql`update agents set status = 'idle', lifecycle_holds = '[]'::jsonb where id = ${held.id}`),
|
||||
() => "the hold release without a hint to commit");
|
||||
const changedAt = Date.now();
|
||||
const window = await trickle(f, Math.ceil((ISSUE_RECOVERY_FULL_BACKSTOP_MS + 2 * MINUTE) / MINUTE));
|
||||
const checks = window.raw.filter(event => event.name === "check" && event.fields.issueId === waiting.id);
|
||||
expect(checks.length).toBeGreaterThan(0);
|
||||
expect(checks[0]!.at - changedAt).toBeGreaterThan(2 * ISSUE_RECOVERY_BACKSTOP_MS);
|
||||
expect(checks[0]!.at - changedAt).toBeLessThanOrEqual(ISSUE_RECOVERY_FULL_BACKSTOP_MS + 2 * MINUTE);
|
||||
const runs = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${waiting.id}`));
|
||||
expect(runs.length).toBeGreaterThan(0);
|
||||
}, 1_800_000);
|
||||
expect(await wakesFor(f.companyId, waiting.id), "recovery did not ask for a wake of the waiting task").not.toHaveLength(0);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
|
||||
it("picks up a finished run, a user message, a raised budget and a resumed agent within a minute", async () => {
|
||||
const f = await boot();
|
||||
@@ -142,25 +163,25 @@ describeIssueRecovery("stranded-task checks at idle for a staging-shaped company
|
||||
const within = async (issueId: string, act: () => Promise<unknown>) => {
|
||||
f.events.length = 0;
|
||||
const at = Date.now();
|
||||
await act();
|
||||
await bounded(act(), () => `the change to task ${issueId} to commit`);
|
||||
await advance(60_000);
|
||||
const checked = checkedAt(f.events, issueId);
|
||||
expect(checked, issueId).not.toBeNull();
|
||||
expect(checked! - at).toBeLessThanOrEqual(60_000);
|
||||
};
|
||||
const [run] = await state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${target!}`).limit(1);
|
||||
const [run] = await bounded(state.db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
|
||||
.where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${target!}`).limit(1), () => "the finished run to be read");
|
||||
await within(target!, () => state.db.transaction(tx => notifyNativeStatusReassessment(tx, { companyId: f.companyId, runId: run!.id })));
|
||||
await advance(31_000);
|
||||
await within(messaged!, () => issueService(state.db).addComment(messaged!, "Please look again", { userId: "recovery-user" }));
|
||||
await advance(31_000);
|
||||
await within(target!, () => state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted" })));
|
||||
await advance(31_000);
|
||||
const { held, waiting } = await heldTask(f);
|
||||
const { held, waiting } = await bounded(heldTask(f), () => "the held agent and its waiting task to be seeded");
|
||||
await advance(31_000);
|
||||
await within(waiting.id, async () => {
|
||||
await state.db.update(agents).set({ status: "idle", lifecycleHolds: [], updatedAt: new Date() }).where(eq(agents.id, held.id));
|
||||
await state.db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted" }));
|
||||
});
|
||||
}, 1_800_000);
|
||||
}, TEST_TIMEOUT_MS);
|
||||
});
|
||||
@@ -32,6 +32,9 @@ export function issueRecoveryFixture(options: { meter?: boolean } = {}) {
|
||||
const tasks = new Map<string, { nextRunAt?: () => number | null; hasPending: () => Promise<boolean> }>();
|
||||
const errors: unknown[] = [];
|
||||
const passes = new Set<Promise<unknown>>();
|
||||
// The queue each pending pass belongs to, so a bounded wait can say which
|
||||
// queue it was still waiting for.
|
||||
const passQueues = new Map<Promise<unknown>, string>();
|
||||
const extraStops: Array<() => Promise<void>> = [];
|
||||
const extraDeadlines: Array<() => number | null> = [];
|
||||
beforeAll(async () => {
|
||||
@@ -61,8 +64,9 @@ export function issueRecoveryFixture(options: { meter?: boolean } = {}) {
|
||||
coordinator.register = (queue, task) => {
|
||||
tasks.set(queue, task);
|
||||
return register(queue, { ...task, run: signal => {
|
||||
const work = task.run(signal); passes.add(work);
|
||||
void work.then(() => passes.delete(work), () => passes.delete(work));
|
||||
const work = task.run(signal); passes.add(work); passQueues.set(work, queue);
|
||||
const done = () => { passes.delete(work); passQueues.delete(work); };
|
||||
void work.then(done, done);
|
||||
return work;
|
||||
} });
|
||||
};
|
||||
@@ -254,5 +258,7 @@ export function issueRecoveryFixture(options: { meter?: boolean } = {}) {
|
||||
set draining(value: typeof draining) { draining = value; },
|
||||
get errors() { return errors; },
|
||||
get passes() { return passes; },
|
||||
/** The queues of the passes still pending, for a bounded wait's message. */
|
||||
pendingPassQueues: () => [...passes].map(pass => passQueues.get(pass) ?? "?"),
|
||||
};
|
||||
}
|
||||
Reference in new issue
Block a user