Files
PaperClipAI/server/src/services/heartbeat/scheduling.test.ts
T
DottaandPaperclip 21137db805 fix: make Check now work for retained monitors on blocked tasks (#18061)
## Thinking Path

> - Paperclip lets people manage agents and inspect their work.
> - Task monitors schedule checks of work that is still pending.
> - A task can retain a monitor after it becomes blocked.
> - The UI still offered Check now, but the server rejected the task
status.
> - The UI also claimed that an automatic check would run on the next
tick.
> - This change admits an explicit human check through the normal
execution gates and shows that automatic checks are paused.
> - People can request the check without changing task status, and
skipped checks no longer report success.

## Linked Issues or Issue Description

**What happened?**

A blocked task with a retained monitor showed Check now and an overdue
countdown. Clicking the button failed because manual checks required an
in-progress or in-review task. The endpoint also returned success when
admission skipped a check.

**Expected behavior**

A human can request one check through normal admission. Automatic checks
stay paused while the task is blocked. A check that cannot start reports
a conflict.

**Steps to reproduce**

1. Schedule a monitor on an agent-owned task.
2. Let the task become blocked while it retains the schedule.
3. Open the task and click Check now.

Related: #14909 addresses scheduled-retry promotion. This PR addresses
retained task monitors.

**Paperclip version or commit**

Base: 7f10cc0585.

## What Changed

- Allow a human to check a retained monitor on a blocked task. Keep
agent and automatic checks limited to active task statuses.
- Keep the existing admission path and status guard. Do not change task
status to start a check.
- Return 409 when admission skips the check. Preserve the schedule when
a dependency or recovery hold prevents dispatch. Keep a successful
result when a new schedule replaces the consumed schedule after
admission.
- Show Automatic checks paused in the banner, composer strip and
properties. Hide Check now on backlog and todo tasks. Provider-quota
recovery stays paused on blocked tasks because it uses a direct retry
path; its monitor is retained until the task resumes.
- Add database, process, route and UI regression coverage, a Storybook
state, API metadata, and behavior documentation.

## Verification

- `pnpm -r typecheck` passed.
- `pnpm build` passed.
- Server scheduling, process admission and route suites: 94 tests
passed.
- UI monitor, properties and time-formatting suites: 142 tests passed.
- Additional admission checks passed: an explicit check starts a process
on a blocked task; an unresolved dependency prevents dispatch and
preserves the monitor.
- Token gates, map checks, Storybook catalog checks and 30 map/catalog
tests passed.
- The full local `pnpm test:run` attempt reported 1 timeout among 1,621
chat integration tests: “keeps an ambiguous github text part blocked
across restart and retries only that part and its tail” exceeded 15
seconds. The exact case passed in isolation on the final head (1 passed,
1,620 skipped). The remaining full-run groups were stopped to isolate
that failure. This does not establish a full local pass or explain the
initial timeout. Final-head CI is green, including the full chat lanes
and the successful single server-shard retry.
- Browser inspection passed for the blocked-task banner, composer strip
and properties row. Clicking Check now set both visible buttons to
disabled Checking state. The preview uses isolated fixtures.
- CI server shard 7/12 failed one decision-expiry deadline assertion.
The entire unchanged decision-expiry suite passed locally on the final
head (20/20). The single retry of that shard passed after the first
workflow attempt completed. The initial failure remains recorded here.
Neither that test nor its production path changed before retry; the
cause is not established by a passing retry.
- Greptile reviewed final head ed936f7433
at 5/5. Both prior review threads are resolved.
- The first CI chat job stopped after its runner received a shutdown
signal. The final head completed a fresh CI run.
- The process fixture uses a local no-op adapter. It proves admission
and execution, not a live provider's response. The Storybook state
simulates dispatch.

## Risks

- An explicit human check can start work on a blocked task with no
unresolved execution gate. This is intentional. Dependencies, recovery,
native finalization, ownership, pause and budgets retain their existing
checks.
- Skipped manual checks now return 409 instead of 200. The UI already
displays API errors in context.
- No schema change or migration.

## Model Used

OpenAI Codex, based on GPT-6. The session does not expose a more
specific served model ID or context-window size. Used repository tools,
code execution, and browser inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 13:01:11 -05:00

486 lines
30 KiB
TypeScript

import { randomUUID } from "node:crypto";
import type { Db } from "@paperclipai/db";
import {
activityLog, agents, agentSessionGoalActions, agentTaskSessions, companies, createDb,
completionContracts, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults,
} from "@paperclipai/db";
import { eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { conflict } from "../../errors.js";
import { createHeartbeatRunPreparation } from "./run-preparation.js";
import { createHeartbeatRunState } from "./run-state.js";
import { createHeartbeatScheduling, type HeartbeatSchedulingDependencies } from "./scheduling.js";
import { readIdleSleepSafety } from "../idle-sleep-safety.js";
import { PaperclipRunnerToolAuthority } from "../native-runtime/paperclip-runner-tool-authority.js";
import { finalizeNativeRun } from "../native-runtime/native-run-finalizer.js";
import { nativeCompletionFeedback } from "../native-runtime/native-completion-feedback.js";
import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() }));
type Run = typeof heartbeatRuns.$inferSelect;
const now = new Date("2026-10-09T12:00:00.000Z");
const dueAt = new Date(now.getTime() - 60_000);
const admitted = { id: "admitted-run" } as Run;
const timerPolicy = {
enabled: false, intervalSec: 60, wakeOnDemand: true, maxConcurrentRuns: 1,
skipTimerWhenNoActionableWork: false, maxDailyRuns: null, maxDailyCostCents: null,
};
function callbacks(db: Db) {
const state = createHeartbeatRunState(db);
const preparation = createHeartbeatRunPreparation(db);
return {
getAgent: state.getAgent,
getRun: state.getRun,
toAgentOrgRow: preparation.toAgentOrgRow,
groupAgentOrgRowsByCompany: preparation.groupAgentOrgRowsByCompany,
issuesSvc: {
listReviewAttention: vi.fn<HeartbeatSchedulingDependencies["issuesSvc"]["listReviewAttention"]>(async () => new Map()),
create: vi.fn<HeartbeatSchedulingDependencies["issuesSvc"]["create"]>(),
},
enqueueWakeup: vi.fn<HeartbeatSchedulingDependencies["enqueueWakeup"]>(async () => admitted),
scheduleBoundedRetryForRun: vi.fn<HeartbeatSchedulingDependencies["scheduleBoundedRetryForRun"]>(),
getSchedulingSuppression: vi.fn(async () => ({ suppressed: false })),
getWorktreeExecutionCutoff: vi.fn<HeartbeatSchedulingDependencies["getWorktreeExecutionCutoff"]>(async () => null),
parseHeartbeatPolicy: vi.fn<HeartbeatSchedulingDependencies["parseHeartbeatPolicy"]>(() => timerPolicy),
claimDueTimerHeartbeat: vi.fn<HeartbeatSchedulingDependencies["claimDueTimerHeartbeat"]>(async () => ({ wasFirstHeartbeat: true })),
} satisfies HeartbeatSchedulingDependencies;
}
describe("heartbeat scheduling module boundary", () => {
function guardedDatabase() {
const access = vi.fn(() => { throw new Error("Unexpected scheduling database access"); });
return { db: new Proxy({}, { get: access }) as Db, access };
}
it("constructs without querying, admitting work, or starting timers", () => {
const database = guardedDatabase();
const deps = callbacks(database.db);
createHeartbeatScheduling(database.db, deps);
expect(database.access).not.toHaveBeenCalled();
for (const callback of Object.values(deps)) if (vi.isMockFunction(callback)) expect(callback).not.toHaveBeenCalled();
});
it.each(["tickTimers", "recoverActiveSessionGoals", "recoverPendingSessionGoalActions"] as const)(
"honors suppression before %s reads or admits work", async (operation) => {
const database = guardedDatabase();
const deps = callbacks(database.db);
deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true });
const result = await createHeartbeatScheduling(database.db, deps)[operation]();
expect(Object.values(result).every(value => value === 0)).toBe(true);
expect(database.access).not.toHaveBeenCalled();
expect(deps.enqueueWakeup).not.toHaveBeenCalled();
expect(deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
},
);
});
const postgres = await getEmbeddedPostgresTestSupport();
describe.skipIf(!postgres.supported)("heartbeat scheduling database boundary", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-scheduling-module");
db = createDb(database.connectionString);
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
afterAll(async () => {
await db?.$client.end();
await database?.cleanup();
});
beforeEach(async () => {
await db.execute(sql`truncate table companies restart identity cascade`);
});
async function fixture() {
const [company] = await db.insert(companies).values({ name: "Scheduling", issuePrefix: "SCH" }).returning();
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Scheduled agent", adapterType: "paperclip_runner", createdAt: dueAt }).returning();
const [issue] = await db.insert(issues).values({
companyId: company.id, title: "Check deployment", status: "in_progress", assigneeAgentId: agent.id,
monitorNextCheckAt: dueAt, monitorAttemptCount: 0, createdAt: dueAt,
executionPolicy: { mode: "normal", commentRequired: true, stages: [], monitor: { nextCheckAt: dueAt.toISOString() } },
}).returning();
const deps = callbacks(db);
return { company, agent, issue, deps, scheduler: createHeartbeatScheduling(db, deps) };
}
async function readIssue(id: string) {
return db.select().from(issues).where(eq(issues.id, id)).then(rows => rows[0]!);
}
async function session(f: Awaited<ReturnType<typeof fixture>>, patch: Partial<typeof agentTaskSessions.$inferInsert> = {}) {
return db.insert(agentTaskSessions).values({
companyId: f.company.id, agentId: f.agent.id, adapterType: f.agent.adapterType, taskKey: f.issue.id,
goalDesiredState: "active", goalStatus: "active", goalRevision: 7, ...patch,
}).returning().then(rows => rows[0]!);
}
function enableTimer(deps: ReturnType<typeof callbacks>) {
deps.parseHeartbeatPolicy.mockReturnValue({ ...timerPolicy, enabled: true });
deps.parseHeartbeatPolicy.mockClear();
}
it("shares database monitor claims across scheduler instances", async () => {
const f = await fixture();
let release!: () => void;
let entered!: () => void;
const started = new Promise<void>(resolve => { entered = resolve; });
const hold = new Promise<void>(resolve => { release = resolve; });
f.deps.enqueueWakeup.mockImplementationOnce(async () => { entered(); await hold; return admitted; });
const first = f.scheduler.tickTimers(now);
try {
await started;
const second = createHeartbeatScheduling(db, f.deps);
expect(await second.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
await expect(second.triggerIssueMonitor(f.issue.id, { now })).rejects.toThrow("already in progress");
} finally {
release();
await first;
}
expect(f.deps.enqueueWakeup).toHaveBeenCalledTimes(1);
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null, monitorAttemptCount: 1 });
});
it("restores the reported future deadline and claims monitor delivery exactly once", async () => {
const f = await fixture();
const at = new Date(now.getTime() + 600_000);
await db.update(issues).set({ monitorNextCheckAt: at,
executionPolicy: { monitor: { nextCheckAt: at.toISOString() } } }).where(eq(issues.id, f.issue.id));
const ownerId = randomUUID();
const hold = () => ({ ownerId, draining: true, activeRuns: 0, pendingWakes: 0,
startedAt: now, expiresAt: new Date(now.getTime() + 120_000) });
const report = await readIdleSleepSafety(db, hold, () => now.getTime(), ownerId,
async () => "none", undefined, undefined, true, true);
expect(report).toMatchObject({ version: 2, backgroundWork: "none", scheduledWake: {
kind: "routines_and_issue_monitors", nextWakeAt: at.toISOString() } });
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
const restored = createHeartbeatScheduling(db, f.deps);
const deadline = new Date(report.scheduledWake!.nextWakeAt);
await Promise.all([restored.tickTimers(deadline), f.scheduler.tickTimers(deadline)]);
expect(f.deps.enqueueWakeup).toHaveBeenCalledOnce();
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({ reason: "issue_monitor_due" }));
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null, monitorAttemptCount: 1 });
});
it("preserves a replacement schedule written during wake admission", async () => {
const f = await fixture();
const replacement = new Date(now.getTime() + 60_000);
f.deps.enqueueWakeup.mockImplementationOnce(async () => {
await db.update(issues).set({ monitorNextCheckAt: replacement, monitorWakeRequestedAt: null }).where(eq(issues.id, f.issue.id));
return admitted;
});
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: replacement, monitorAttemptCount: 0 });
});
it("retains a deferred monitor and releases its claim for a later tick", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
const calls = f.deps.enqueueWakeup.mock.calls;
expect(calls[0]![1]!.idempotencyKey).toBe(calls[1]![1]!.idempotencyKey);
expect(calls[0]![1]!.issueStateGuard).toMatchObject({
assigneeAgentId: f.agent.id, statuses: ["in_progress"], monitorNextCheckAt: dueAt.toISOString(), monitorWakeRequestedAt: now.toISOString(),
});
});
it("keeps manual error recovery distinct from scheduled client-error handling", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockRejectedValue(conflict("Agent is paused"));
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType: "user", actorId: "board-user" })).rejects.toThrow("Agent is paused");
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
expect(f.deps.enqueueWakeup).toHaveBeenLastCalledWith(f.agent.id, expect.objectContaining({
source: "on_demand", requestedByActorType: "user", requestedByActorId: "board-user",
}));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null });
});
it("lets a human check a blocked task without resuming automatic checks", async () => {
const f = await fixture();
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issue.id));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType: "user", actorId: "board-user" }))
.resolves.toMatchObject({ outcome: "triggered" });
expect(f.deps.enqueueWakeup).toHaveBeenCalledExactlyOnceWith(f.agent.id, expect.objectContaining({
source: "on_demand", requestedByActorType: "user", requestedByActorId: "board-user",
issueStateGuard: expect.objectContaining({ statuses: ["blocked"], assigneeAgentId: f.agent.id }),
}));
expect(await readIssue(f.issue.id)).toMatchObject({ status: "blocked", monitorNextCheckAt: null, monitorAttemptCount: 1 });
});
it.each(["agent", "system"] as const)("does not let a %s check a blocked task", async (actorType) => {
const f = await fixture();
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issue.id));
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType, actorId: "caller" })).rejects.toThrow();
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
expect(await readIssue(f.issue.id)).toMatchObject({ status: "blocked", monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
});
it("retains monitors after transient admission failures", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockRejectedValueOnce(new Error("Temporary admission failure"));
await f.scheduler.tickTimers(now);
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
});
it("records bounded dispatch failure diagnostics while retaining the schedule", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockRejectedValue(new Error("PRIVATE_PROVIDER_FAILURE"));
await f.scheduler.tickTimers(now);
await f.scheduler.tickTimers(new Date(now.getTime() + 30_000));
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
const events = await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id));
expect(events.filter(event => event.action === "issue.monitor_dispatch_failed")).toHaveLength(1);
expect(events[0]?.details).toMatchObject({ reason: "monitor_dispatch_failed", nextCheckAt: dueAt.toISOString(), attemptCount: 0 });
expect(JSON.stringify(events)).not.toContain("PRIVATE_PROVIDER_FAILURE");
});
it("repeats a persistent diagnostic after its bounded quiet window", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockResolvedValue(null);
await f.scheduler.tickTimers(now);
await f.scheduler.tickTimers(new Date(now.getTime() + 30_000));
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(1);
await db.update(activityLog).set({ createdAt: new Date(Date.now() - 6 * 60_000) }).where(eq(activityLog.entityId, f.issue.id));
await f.scheduler.tickTimers(new Date(now.getTime() + 60_000));
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(2);
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorAttemptCount: 0 });
});
it("does not copy an unrecognized blocker cause into monitor activity", async () => {
const f = await fixture();
f.deps.enqueueWakeup.mockImplementationOnce(async (_agentId, options) => {
options?.issueStateGuard?.onExecutionDeferred?.("PRIVATE_BLOCKER_CAUSE");
return null;
});
await f.scheduler.tickTimers(now);
const events = await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id));
expect(events).toMatchObject([{ action: "issue.monitor_deferred", details: { reason: "execution_reconciliation_required" } }]);
expect(JSON.stringify(events)).not.toContain("PRIVATE_BLOCKER_CAUSE");
});
it("releases a monitor claim when diagnostic storage fails", async () => {
const f = await fixture();
await db.execute(sql`create function fail_monitor_diagnostic_test() returns trigger language plpgsql as $$
begin if new.action = 'issue.monitor_deferred' then raise exception 'Synthetic diagnostic storage failure'; end if; return new; end; $$`);
await db.execute(sql`create trigger fail_monitor_diagnostic_test before insert on activity_log
for each row execute function fail_monitor_diagnostic_test()`);
try {
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
} finally {
await db.execute(sql`drop trigger fail_monitor_diagnostic_test on activity_log`);
await db.execute(sql`drop function fail_monitor_diagnostic_test()`);
}
});
it("does not attach old dispatch diagnostics to a replacement monitor", async () => {
const f = await fixture();
const replacement = new Date(now.getTime() + 60_000);
f.deps.enqueueWakeup.mockImplementationOnce(async () => {
await db.update(issues).set({ monitorNextCheckAt: replacement, monitorWakeRequestedAt: null }).where(eq(issues.id, f.issue.id));
return null;
});
await f.scheduler.tickTimers(now);
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: replacement });
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(0);
});
it("leaves monitors alone while a native run owns the issue", async () => {
const f = await fixture();
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id, runtimeMode: "native", status: "running", contextSnapshot: { issueId: f.issue.id } }).returning();
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
});
it("keeps a due monitor until a terminal native run finishes finalization", async () => {
const f = await fixture();
const [run] = await db.insert(heartbeatRuns).values({
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: f.issue.id },
}).returning();
await db.insert(nativeRunFinalizations).values({
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, phase: "retryable_failure",
});
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now })).rejects.toThrow("waiting for the previous run to settle");
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, run!.id));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
expect(f.deps.enqueueWakeup).toHaveBeenCalledTimes(1);
});
it.each(["scheduled", "manual"] as const)("applies expired monitor recovery after workspace settlement without another turn (%s)", async (dispatch) => {
const f = await fixture();
const start = new Date();
const nextCheckAt = new Date(start.getTime() + 60_000).toISOString();
const timeoutAt = new Date(start.getTime() + 120_000).toISOString();
const [contract] = await db.insert(completionContracts).values({
companyId: f.company.id, issueId: f.issue.id, revision: 1,
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(),
createdByActorType: "system", createdByActorId: "test",
}).returning();
const [run] = await db.insert(heartbeatRuns).values({
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
runtimeMode: "native", status: "running", completionContractId: contract!.id,
contextSnapshot: { issueId: f.issue.id },
}).returning();
await db.update(issues).set({ executionRunId: run!.id, checkoutRunId: run!.id }).where(eq(issues.id, f.issue.id));
const authority = new PaperclipRunnerToolAuthority(db, { companyId: f.company.id,
issueId: f.issue.id, agentId: f.agent.id, runId: run!.id });
await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: {
idempotencyKey: "expires-during-workspace-save",
monitor: { nextCheckAt, timeoutAt, notes: "Check once saved", recoveryPolicy: "escalate_to_board" },
} });
const result: PrpStructuredRunResult = {
schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Wait for the check.",
completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [],
remainingWork: [{ description: "Check after saving", blocksCompletion: true }] },
continuation: { kind: "monitor", summary: "Wait for the saved check", idempotencyKey: "wait" },
evidence: [], verification: [], attentionRequests: [], artifacts: [],
};
await expect(nativeCompletionFeedback(db, run!.id, result)).resolves.toContain("issue_monitor_due");
const [saved] = await db.insert(nativeRunResults).values({
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, completionContractId: contract!.id,
serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted",
resultJson: { result, terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded",
turnTerminalState: "completed", reportedWorkDisposition: "yielded" } },
}).returning();
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
await db.insert(nativeRunFinalizations).values({ companyId: f.company.id, issueId: f.issue.id,
runId: run!.id, phase: "result_persisted", resultId: saved!.id });
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "failed" });
const expiredAt = new Date(start.getTime() + 180_000);
vi.useFakeTimers({ toFake: ["Date"] });
vi.setSystemTime(expiredAt);
try {
await expect(nativeCompletionFeedback(db, run!.id, result)).rejects.toThrow("persisted, eligible monitor");
expect(await f.scheduler.tickTimers(expiredAt)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt })).rejects.toThrow("waiting for the previous run to settle");
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run!.id)))[0])
.toMatchObject({ phase: "committed", failureCode: null });
expect(await readIssue(f.issue.id)).toMatchObject({ executionRunId: null, checkoutRunId: null,
monitorNextCheckAt: new Date(nextCheckAt) });
if (dispatch === "manual") await f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt });
else await f.scheduler.tickTimers(expiredAt);
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null,
executionState: { monitor: { status: "cleared", clearReason: "timeout_exceeded" } } });
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
await f.scheduler.tickTimers(expiredAt);
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, saved!.id)))[0]!.resultJson).toEqual(saved!.resultJson);
} finally {
vi.useRealTimers();
}
});
it("skips archived companies for both timers and monitors", async () => {
const f = await fixture();
enableTimer(f.deps);
await db.update(companies).set({ status: "archived" }).where(eq(companies.id, f.company.id));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt });
});
it("combines timer and monitor counts and passes the timer claim receipt to admission", async () => {
const f = await fixture();
enableTimer(f.deps);
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 2, enqueued: 2, skipped: 0 });
expect(f.deps.claimDueTimerHeartbeat).toHaveBeenCalledWith(expect.objectContaining({ id: f.agent.id }), now, 60);
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
source: "timer", reason: "heartbeat_timer", contextSnapshot: expect.objectContaining({ timerClaimWasFirstHeartbeat: true }),
}));
});
it("does not enqueue a timer after losing its due claim", async () => {
const f = await fixture();
enableTimer(f.deps);
await db.update(issues).set({ monitorNextCheckAt: null }).where(eq(issues.id, f.issue.id));
f.deps.claimDueTimerHeartbeat.mockResolvedValue(null);
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
});
it("enforces worktree cutoffs and agent invokability before timer claims", async () => {
const f = await fixture();
enableTimer(f.deps);
await db.update(issues).set({ monitorNextCheckAt: null }).where(eq(issues.id, f.issue.id));
f.deps.getWorktreeExecutionCutoff.mockResolvedValue(now);
await f.scheduler.tickTimers(now);
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
await db.update(issues).set({ createdAt: now }).where(eq(issues.id, f.issue.id));
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id));
await f.scheduler.tickTimers(now);
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
await db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agent.id));
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
});
it("recovers active goals with stable revision-based admission keys", async () => {
const f = await fixture();
const active = await session(f);
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 1, enqueued: 1 });
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
idempotencyKey: `goal_recovery:${active.id}:7`, contextSnapshot: { issueId: f.issue.id, taskKey: f.issue.id, resumeSessionGoalHeartbeat: true, skipIssueComment: true },
}));
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 1, enqueued: 0 });
expect(f.deps.enqueueWakeup.mock.calls[1]![1]!.idempotencyKey).toBe(f.deps.enqueueWakeup.mock.calls[0]![1]!.idempotencyKey);
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issue.id));
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
});
it("excludes goals after reassignment or with a mismatched company binding", async () => {
const f = await fixture();
const [other] = await db.insert(companies).values({ name: "Other company", issuePrefix: "OTH" }).returning();
const active = await session(f, { companyId: other!.id });
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
await db.update(agentTaskSessions).set({ companyId: f.company.id }).where(eq(agentTaskSessions.id, active.id));
await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issue.id));
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
});
it.each(["queued", "scheduled_retry", "running"])("deduplicates a pending goal action against a %s run", async status => {
const f = await fixture();
const active = await session(f);
const requestId = randomUUID();
const [action] = await db.insert(agentSessionGoalActions).values({ companyId: f.company.id, sessionId: active.id, requestId, action: "pause", payloadJson: { requestId, action: "pause" } }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, status, contextSnapshot: { goalControlRequestId: requestId } }).returning();
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 0, alreadyQueued: 1, invalid: 0 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issue.id));
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 1, alreadyQueued: 0, invalid: 0 });
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
idempotencyKey: `goal_control_recovery:${action!.id}`, contextSnapshot: expect.objectContaining({ resumeIntent: true, goalControlRequestId: requestId, runnerGoalControl: { requestId, action: "pause" } }),
}));
});
it("persists invalid goal actions as failures without wake admission", async () => {
const f = await fixture();
const active = await session(f);
const [action] = await db.insert(agentSessionGoalActions).values({ companyId: f.company.id, sessionId: active.id, requestId: "expected", action: "pause", payloadJson: { requestId: "different", action: "pause" } }).returning();
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 0, alreadyQueued: 0, invalid: 1 });
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
const failed = await db.select().from(agentSessionGoalActions).where(eq(agentSessionGoalActions.id, action!.id));
expect(failed[0]).toMatchObject({ status: "failed", error: "session_goal_control_payload_invalid" });
});
});