mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 21:07:40 +01:00
## Thinking Path > - Paperclip lets people manage agents and inspect their work. > - Task monitors schedule checks of work that is still pending. > - A task can retain a monitor after it becomes blocked. > - The UI still offered Check now, but the server rejected the task status. > - The UI also claimed that an automatic check would run on the next tick. > - This change admits an explicit human check through the normal execution gates and shows that automatic checks are paused. > - People can request the check without changing task status, and skipped checks no longer report success. ## Linked Issues or Issue Description **What happened?** A blocked task with a retained monitor showed Check now and an overdue countdown. Clicking the button failed because manual checks required an in-progress or in-review task. The endpoint also returned success when admission skipped a check. **Expected behavior** A human can request one check through normal admission. Automatic checks stay paused while the task is blocked. A check that cannot start reports a conflict. **Steps to reproduce** 1. Schedule a monitor on an agent-owned task. 2. Let the task become blocked while it retains the schedule. 3. Open the task and click Check now. Related: #14909 addresses scheduled-retry promotion. This PR addresses retained task monitors. **Paperclip version or commit** Base:7f10cc0585. ## What Changed - Allow a human to check a retained monitor on a blocked task. Keep agent and automatic checks limited to active task statuses. - Keep the existing admission path and status guard. Do not change task status to start a check. - Return 409 when admission skips the check. Preserve the schedule when a dependency or recovery hold prevents dispatch. Keep a successful result when a new schedule replaces the consumed schedule after admission. - Show Automatic checks paused in the banner, composer strip and properties. Hide Check now on backlog and todo tasks. Provider-quota recovery stays paused on blocked tasks because it uses a direct retry path; its monitor is retained until the task resumes. - Add database, process, route and UI regression coverage, a Storybook state, API metadata, and behavior documentation. ## Verification - `pnpm -r typecheck` passed. - `pnpm build` passed. - Server scheduling, process admission and route suites: 94 tests passed. - UI monitor, properties and time-formatting suites: 142 tests passed. - Additional admission checks passed: an explicit check starts a process on a blocked task; an unresolved dependency prevents dispatch and preserves the monitor. - Token gates, map checks, Storybook catalog checks and 30 map/catalog tests passed. - The full local `pnpm test:run` attempt reported 1 timeout among 1,621 chat integration tests: “keeps an ambiguous github text part blocked across restart and retries only that part and its tail” exceeded 15 seconds. The exact case passed in isolation on the final head (1 passed, 1,620 skipped). The remaining full-run groups were stopped to isolate that failure. This does not establish a full local pass or explain the initial timeout. Final-head CI is green, including the full chat lanes and the successful single server-shard retry. - Browser inspection passed for the blocked-task banner, composer strip and properties row. Clicking Check now set both visible buttons to disabled Checking state. The preview uses isolated fixtures. - CI server shard 7/12 failed one decision-expiry deadline assertion. The entire unchanged decision-expiry suite passed locally on the final head (20/20). The single retry of that shard passed after the first workflow attempt completed. The initial failure remains recorded here. Neither that test nor its production path changed before retry; the cause is not established by a passing retry. - Greptile reviewed final headed936f7433at 5/5. Both prior review threads are resolved. - The first CI chat job stopped after its runner received a shutdown signal. The final head completed a fresh CI run. - The process fixture uses a local no-op adapter. It proves admission and execution, not a live provider's response. The Storybook state simulates dispatch. ## Risks - An explicit human check can start work on a blocked task with no unresolved execution gate. This is intentional. Dependencies, recovery, native finalization, ownership, pause and budgets retain their existing checks. - Skipped manual checks now return 409 instead of 200. The UI already displays API errors in context. - No schema change or migration. ## Model Used OpenAI Codex, based on GPT-6. The session does not expose a more specific served model ID or context-window size. Used repository tools, code execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
486 lines
30 KiB
TypeScript
486 lines
30 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
activityLog, agents, agentSessionGoalActions, agentTaskSessions, companies, createDb,
|
|
completionContracts, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults,
|
|
} from "@paperclipai/db";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { conflict } from "../../errors.js";
|
|
import { createHeartbeatRunPreparation } from "./run-preparation.js";
|
|
import { createHeartbeatRunState } from "./run-state.js";
|
|
import { createHeartbeatScheduling, type HeartbeatSchedulingDependencies } from "./scheduling.js";
|
|
import { readIdleSleepSafety } from "../idle-sleep-safety.js";
|
|
import { PaperclipRunnerToolAuthority } from "../native-runtime/paperclip-runner-tool-authority.js";
|
|
import { finalizeNativeRun } from "../native-runtime/native-run-finalizer.js";
|
|
import { nativeCompletionFeedback } from "../native-runtime/native-completion-feedback.js";
|
|
import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() }));
|
|
|
|
type Run = typeof heartbeatRuns.$inferSelect;
|
|
const now = new Date("2026-10-09T12:00:00.000Z");
|
|
const dueAt = new Date(now.getTime() - 60_000);
|
|
const admitted = { id: "admitted-run" } as Run;
|
|
const timerPolicy = {
|
|
enabled: false, intervalSec: 60, wakeOnDemand: true, maxConcurrentRuns: 1,
|
|
skipTimerWhenNoActionableWork: false, maxDailyRuns: null, maxDailyCostCents: null,
|
|
};
|
|
|
|
function callbacks(db: Db) {
|
|
const state = createHeartbeatRunState(db);
|
|
const preparation = createHeartbeatRunPreparation(db);
|
|
return {
|
|
getAgent: state.getAgent,
|
|
getRun: state.getRun,
|
|
toAgentOrgRow: preparation.toAgentOrgRow,
|
|
groupAgentOrgRowsByCompany: preparation.groupAgentOrgRowsByCompany,
|
|
issuesSvc: {
|
|
listReviewAttention: vi.fn<HeartbeatSchedulingDependencies["issuesSvc"]["listReviewAttention"]>(async () => new Map()),
|
|
create: vi.fn<HeartbeatSchedulingDependencies["issuesSvc"]["create"]>(),
|
|
},
|
|
enqueueWakeup: vi.fn<HeartbeatSchedulingDependencies["enqueueWakeup"]>(async () => admitted),
|
|
scheduleBoundedRetryForRun: vi.fn<HeartbeatSchedulingDependencies["scheduleBoundedRetryForRun"]>(),
|
|
getSchedulingSuppression: vi.fn(async () => ({ suppressed: false })),
|
|
getWorktreeExecutionCutoff: vi.fn<HeartbeatSchedulingDependencies["getWorktreeExecutionCutoff"]>(async () => null),
|
|
parseHeartbeatPolicy: vi.fn<HeartbeatSchedulingDependencies["parseHeartbeatPolicy"]>(() => timerPolicy),
|
|
claimDueTimerHeartbeat: vi.fn<HeartbeatSchedulingDependencies["claimDueTimerHeartbeat"]>(async () => ({ wasFirstHeartbeat: true })),
|
|
} satisfies HeartbeatSchedulingDependencies;
|
|
}
|
|
|
|
describe("heartbeat scheduling module boundary", () => {
|
|
function guardedDatabase() {
|
|
const access = vi.fn(() => { throw new Error("Unexpected scheduling database access"); });
|
|
return { db: new Proxy({}, { get: access }) as Db, access };
|
|
}
|
|
|
|
it("constructs without querying, admitting work, or starting timers", () => {
|
|
const database = guardedDatabase();
|
|
const deps = callbacks(database.db);
|
|
createHeartbeatScheduling(database.db, deps);
|
|
expect(database.access).not.toHaveBeenCalled();
|
|
for (const callback of Object.values(deps)) if (vi.isMockFunction(callback)) expect(callback).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["tickTimers", "recoverActiveSessionGoals", "recoverPendingSessionGoalActions"] as const)(
|
|
"honors suppression before %s reads or admits work", async (operation) => {
|
|
const database = guardedDatabase();
|
|
const deps = callbacks(database.db);
|
|
deps.getSchedulingSuppression.mockResolvedValue({ suppressed: true });
|
|
const result = await createHeartbeatScheduling(database.db, deps)[operation]();
|
|
expect(Object.values(result).every(value => value === 0)).toBe(true);
|
|
expect(database.access).not.toHaveBeenCalled();
|
|
expect(deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
|
|
},
|
|
);
|
|
});
|
|
|
|
const postgres = await getEmbeddedPostgresTestSupport();
|
|
describe.skipIf(!postgres.supported)("heartbeat scheduling database boundary", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
beforeAll(async () => {
|
|
database = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-scheduling-module");
|
|
db = createDb(database.connectionString);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => {
|
|
await db?.$client.end();
|
|
await database?.cleanup();
|
|
});
|
|
beforeEach(async () => {
|
|
await db.execute(sql`truncate table companies restart identity cascade`);
|
|
});
|
|
|
|
async function fixture() {
|
|
const [company] = await db.insert(companies).values({ name: "Scheduling", issuePrefix: "SCH" }).returning();
|
|
const [agent] = await db.insert(agents).values({ companyId: company.id, name: "Scheduled agent", adapterType: "paperclip_runner", createdAt: dueAt }).returning();
|
|
const [issue] = await db.insert(issues).values({
|
|
companyId: company.id, title: "Check deployment", status: "in_progress", assigneeAgentId: agent.id,
|
|
monitorNextCheckAt: dueAt, monitorAttemptCount: 0, createdAt: dueAt,
|
|
executionPolicy: { mode: "normal", commentRequired: true, stages: [], monitor: { nextCheckAt: dueAt.toISOString() } },
|
|
}).returning();
|
|
const deps = callbacks(db);
|
|
return { company, agent, issue, deps, scheduler: createHeartbeatScheduling(db, deps) };
|
|
}
|
|
async function readIssue(id: string) {
|
|
return db.select().from(issues).where(eq(issues.id, id)).then(rows => rows[0]!);
|
|
}
|
|
async function session(f: Awaited<ReturnType<typeof fixture>>, patch: Partial<typeof agentTaskSessions.$inferInsert> = {}) {
|
|
return db.insert(agentTaskSessions).values({
|
|
companyId: f.company.id, agentId: f.agent.id, adapterType: f.agent.adapterType, taskKey: f.issue.id,
|
|
goalDesiredState: "active", goalStatus: "active", goalRevision: 7, ...patch,
|
|
}).returning().then(rows => rows[0]!);
|
|
}
|
|
function enableTimer(deps: ReturnType<typeof callbacks>) {
|
|
deps.parseHeartbeatPolicy.mockReturnValue({ ...timerPolicy, enabled: true });
|
|
deps.parseHeartbeatPolicy.mockClear();
|
|
}
|
|
|
|
it("shares database monitor claims across scheduler instances", async () => {
|
|
const f = await fixture();
|
|
let release!: () => void;
|
|
let entered!: () => void;
|
|
const started = new Promise<void>(resolve => { entered = resolve; });
|
|
const hold = new Promise<void>(resolve => { release = resolve; });
|
|
f.deps.enqueueWakeup.mockImplementationOnce(async () => { entered(); await hold; return admitted; });
|
|
const first = f.scheduler.tickTimers(now);
|
|
try {
|
|
await started;
|
|
const second = createHeartbeatScheduling(db, f.deps);
|
|
expect(await second.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
await expect(second.triggerIssueMonitor(f.issue.id, { now })).rejects.toThrow("already in progress");
|
|
} finally {
|
|
release();
|
|
await first;
|
|
}
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledTimes(1);
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null, monitorAttemptCount: 1 });
|
|
});
|
|
|
|
it("restores the reported future deadline and claims monitor delivery exactly once", async () => {
|
|
const f = await fixture();
|
|
const at = new Date(now.getTime() + 600_000);
|
|
await db.update(issues).set({ monitorNextCheckAt: at,
|
|
executionPolicy: { monitor: { nextCheckAt: at.toISOString() } } }).where(eq(issues.id, f.issue.id));
|
|
const ownerId = randomUUID();
|
|
const hold = () => ({ ownerId, draining: true, activeRuns: 0, pendingWakes: 0,
|
|
startedAt: now, expiresAt: new Date(now.getTime() + 120_000) });
|
|
const report = await readIdleSleepSafety(db, hold, () => now.getTime(), ownerId,
|
|
async () => "none", undefined, undefined, true, true);
|
|
expect(report).toMatchObject({ version: 2, backgroundWork: "none", scheduledWake: {
|
|
kind: "routines_and_issue_monitors", nextWakeAt: at.toISOString() } });
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
const restored = createHeartbeatScheduling(db, f.deps);
|
|
const deadline = new Date(report.scheduledWake!.nextWakeAt);
|
|
await Promise.all([restored.tickTimers(deadline), f.scheduler.tickTimers(deadline)]);
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledOnce();
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({ reason: "issue_monitor_due" }));
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null, monitorAttemptCount: 1 });
|
|
});
|
|
|
|
it("preserves a replacement schedule written during wake admission", async () => {
|
|
const f = await fixture();
|
|
const replacement = new Date(now.getTime() + 60_000);
|
|
f.deps.enqueueWakeup.mockImplementationOnce(async () => {
|
|
await db.update(issues).set({ monitorNextCheckAt: replacement, monitorWakeRequestedAt: null }).where(eq(issues.id, f.issue.id));
|
|
return admitted;
|
|
});
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: replacement, monitorAttemptCount: 0 });
|
|
});
|
|
|
|
it("retains a deferred monitor and releases its claim for a later tick", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
const calls = f.deps.enqueueWakeup.mock.calls;
|
|
expect(calls[0]![1]!.idempotencyKey).toBe(calls[1]![1]!.idempotencyKey);
|
|
expect(calls[0]![1]!.issueStateGuard).toMatchObject({
|
|
assigneeAgentId: f.agent.id, statuses: ["in_progress"], monitorNextCheckAt: dueAt.toISOString(), monitorWakeRequestedAt: now.toISOString(),
|
|
});
|
|
});
|
|
|
|
it("keeps manual error recovery distinct from scheduled client-error handling", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockRejectedValue(conflict("Agent is paused"));
|
|
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType: "user", actorId: "board-user" })).rejects.toThrow("Agent is paused");
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
|
|
expect(f.deps.enqueueWakeup).toHaveBeenLastCalledWith(f.agent.id, expect.objectContaining({
|
|
source: "on_demand", requestedByActorType: "user", requestedByActorId: "board-user",
|
|
}));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null });
|
|
});
|
|
|
|
it("lets a human check a blocked task without resuming automatic checks", async () => {
|
|
const f = await fixture();
|
|
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issue.id));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType: "user", actorId: "board-user" }))
|
|
.resolves.toMatchObject({ outcome: "triggered" });
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledExactlyOnceWith(f.agent.id, expect.objectContaining({
|
|
source: "on_demand", requestedByActorType: "user", requestedByActorId: "board-user",
|
|
issueStateGuard: expect.objectContaining({ statuses: ["blocked"], assigneeAgentId: f.agent.id }),
|
|
}));
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ status: "blocked", monitorNextCheckAt: null, monitorAttemptCount: 1 });
|
|
});
|
|
|
|
it.each(["agent", "system"] as const)("does not let a %s check a blocked task", async (actorType) => {
|
|
const f = await fixture();
|
|
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, f.issue.id));
|
|
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now, actorType, actorId: "caller" })).rejects.toThrow();
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ status: "blocked", monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
|
|
});
|
|
|
|
it("retains monitors after transient admission failures", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockRejectedValueOnce(new Error("Temporary admission failure"));
|
|
await f.scheduler.tickTimers(now);
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
});
|
|
|
|
it("records bounded dispatch failure diagnostics while retaining the schedule", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockRejectedValue(new Error("PRIVATE_PROVIDER_FAILURE"));
|
|
await f.scheduler.tickTimers(now);
|
|
await f.scheduler.tickTimers(new Date(now.getTime() + 30_000));
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
|
|
const events = await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id));
|
|
expect(events.filter(event => event.action === "issue.monitor_dispatch_failed")).toHaveLength(1);
|
|
expect(events[0]?.details).toMatchObject({ reason: "monitor_dispatch_failed", nextCheckAt: dueAt.toISOString(), attemptCount: 0 });
|
|
expect(JSON.stringify(events)).not.toContain("PRIVATE_PROVIDER_FAILURE");
|
|
});
|
|
|
|
it("repeats a persistent diagnostic after its bounded quiet window", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockResolvedValue(null);
|
|
await f.scheduler.tickTimers(now);
|
|
await f.scheduler.tickTimers(new Date(now.getTime() + 30_000));
|
|
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(1);
|
|
await db.update(activityLog).set({ createdAt: new Date(Date.now() - 6 * 60_000) }).where(eq(activityLog.entityId, f.issue.id));
|
|
await f.scheduler.tickTimers(new Date(now.getTime() + 60_000));
|
|
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(2);
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorAttemptCount: 0 });
|
|
});
|
|
|
|
it("does not copy an unrecognized blocker cause into monitor activity", async () => {
|
|
const f = await fixture();
|
|
f.deps.enqueueWakeup.mockImplementationOnce(async (_agentId, options) => {
|
|
options?.issueStateGuard?.onExecutionDeferred?.("PRIVATE_BLOCKER_CAUSE");
|
|
return null;
|
|
});
|
|
await f.scheduler.tickTimers(now);
|
|
const events = await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id));
|
|
expect(events).toMatchObject([{ action: "issue.monitor_deferred", details: { reason: "execution_reconciliation_required" } }]);
|
|
expect(JSON.stringify(events)).not.toContain("PRIVATE_BLOCKER_CAUSE");
|
|
});
|
|
|
|
it("releases a monitor claim when diagnostic storage fails", async () => {
|
|
const f = await fixture();
|
|
await db.execute(sql`create function fail_monitor_diagnostic_test() returns trigger language plpgsql as $$
|
|
begin if new.action = 'issue.monitor_deferred' then raise exception 'Synthetic diagnostic storage failure'; end if; return new; end; $$`);
|
|
await db.execute(sql`create trigger fail_monitor_diagnostic_test before insert on activity_log
|
|
for each row execute function fail_monitor_diagnostic_test()`);
|
|
try {
|
|
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 1 });
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null, monitorAttemptCount: 0 });
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
} finally {
|
|
await db.execute(sql`drop trigger fail_monitor_diagnostic_test on activity_log`);
|
|
await db.execute(sql`drop function fail_monitor_diagnostic_test()`);
|
|
}
|
|
});
|
|
|
|
it("does not attach old dispatch diagnostics to a replacement monitor", async () => {
|
|
const f = await fixture();
|
|
const replacement = new Date(now.getTime() + 60_000);
|
|
f.deps.enqueueWakeup.mockImplementationOnce(async () => {
|
|
await db.update(issues).set({ monitorNextCheckAt: replacement, monitorWakeRequestedAt: null }).where(eq(issues.id, f.issue.id));
|
|
return null;
|
|
});
|
|
await f.scheduler.tickTimers(now);
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: replacement });
|
|
expect(await db.select().from(activityLog).where(eq(activityLog.entityId, f.issue.id))).toHaveLength(0);
|
|
});
|
|
|
|
it("leaves monitors alone while a native run owns the issue", async () => {
|
|
const f = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id, runtimeMode: "native", status: "running", contextSnapshot: { issueId: f.issue.id } }).returning();
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
});
|
|
|
|
it("keeps a due monitor until a terminal native run finishes finalization", async () => {
|
|
const f = await fixture();
|
|
const [run] = await db.insert(heartbeatRuns).values({
|
|
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
|
|
runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: f.issue.id },
|
|
}).returning();
|
|
await db.insert(nativeRunFinalizations).values({
|
|
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, phase: "retryable_failure",
|
|
});
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now })).rejects.toThrow("waiting for the previous run to settle");
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
|
|
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, run!.id));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledTimes(1);
|
|
});
|
|
|
|
it.each(["scheduled", "manual"] as const)("applies expired monitor recovery after workspace settlement without another turn (%s)", async (dispatch) => {
|
|
const f = await fixture();
|
|
const start = new Date();
|
|
const nextCheckAt = new Date(start.getTime() + 60_000).toISOString();
|
|
const timeoutAt = new Date(start.getTime() + 120_000).toISOString();
|
|
const [contract] = await db.insert(completionContracts).values({
|
|
companyId: f.company.id, issueId: f.issue.id, revision: 1,
|
|
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
|
|
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
|
contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(),
|
|
createdByActorType: "system", createdByActorId: "test",
|
|
}).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({
|
|
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
|
|
runtimeMode: "native", status: "running", completionContractId: contract!.id,
|
|
contextSnapshot: { issueId: f.issue.id },
|
|
}).returning();
|
|
await db.update(issues).set({ executionRunId: run!.id, checkoutRunId: run!.id }).where(eq(issues.id, f.issue.id));
|
|
const authority = new PaperclipRunnerToolAuthority(db, { companyId: f.company.id,
|
|
issueId: f.issue.id, agentId: f.agent.id, runId: run!.id });
|
|
await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: {
|
|
idempotencyKey: "expires-during-workspace-save",
|
|
monitor: { nextCheckAt, timeoutAt, notes: "Check once saved", recoveryPolicy: "escalate_to_board" },
|
|
} });
|
|
const result: PrpStructuredRunResult = {
|
|
schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Wait for the check.",
|
|
completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [],
|
|
remainingWork: [{ description: "Check after saving", blocksCompletion: true }] },
|
|
continuation: { kind: "monitor", summary: "Wait for the saved check", idempotencyKey: "wait" },
|
|
evidence: [], verification: [], attentionRequests: [], artifacts: [],
|
|
};
|
|
await expect(nativeCompletionFeedback(db, run!.id, result)).resolves.toContain("issue_monitor_due");
|
|
const [saved] = await db.insert(nativeRunResults).values({
|
|
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, completionContractId: contract!.id,
|
|
serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted",
|
|
resultJson: { result, terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded",
|
|
turnTerminalState: "completed", reportedWorkDisposition: "yielded" } },
|
|
}).returning();
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
|
|
await db.insert(nativeRunFinalizations).values({ companyId: f.company.id, issueId: f.issue.id,
|
|
runId: run!.id, phase: "result_persisted", resultId: saved!.id });
|
|
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "failed" });
|
|
const expiredAt = new Date(start.getTime() + 180_000);
|
|
vi.useFakeTimers({ toFake: ["Date"] });
|
|
vi.setSystemTime(expiredAt);
|
|
try {
|
|
await expect(nativeCompletionFeedback(db, run!.id, result)).rejects.toThrow("persisted, eligible monitor");
|
|
expect(await f.scheduler.tickTimers(expiredAt)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt })).rejects.toThrow("waiting for the previous run to settle");
|
|
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
|
|
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run!.id)))[0])
|
|
.toMatchObject({ phase: "committed", failureCode: null });
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ executionRunId: null, checkoutRunId: null,
|
|
monitorNextCheckAt: new Date(nextCheckAt) });
|
|
if (dispatch === "manual") await f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt });
|
|
else await f.scheduler.tickTimers(expiredAt);
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null,
|
|
executionState: { monitor: { status: "cleared", clearReason: "timeout_exceeded" } } });
|
|
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
|
|
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
|
|
await f.scheduler.tickTimers(expiredAt);
|
|
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, saved!.id)))[0]!.resultJson).toEqual(saved!.resultJson);
|
|
} finally {
|
|
vi.useRealTimers();
|
|
}
|
|
});
|
|
|
|
it("skips archived companies for both timers and monitors", async () => {
|
|
const f = await fixture();
|
|
enableTimer(f.deps);
|
|
await db.update(companies).set({ status: "archived" }).where(eq(companies.id, f.company.id));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
|
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt });
|
|
});
|
|
|
|
it("combines timer and monitor counts and passes the timer claim receipt to admission", async () => {
|
|
const f = await fixture();
|
|
enableTimer(f.deps);
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 2, enqueued: 2, skipped: 0 });
|
|
expect(f.deps.claimDueTimerHeartbeat).toHaveBeenCalledWith(expect.objectContaining({ id: f.agent.id }), now, 60);
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
|
|
source: "timer", reason: "heartbeat_timer", contextSnapshot: expect.objectContaining({ timerClaimWasFirstHeartbeat: true }),
|
|
}));
|
|
});
|
|
|
|
it("does not enqueue a timer after losing its due claim", async () => {
|
|
const f = await fixture();
|
|
enableTimer(f.deps);
|
|
await db.update(issues).set({ monitorNextCheckAt: null }).where(eq(issues.id, f.issue.id));
|
|
f.deps.claimDueTimerHeartbeat.mockResolvedValue(null);
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 0, skipped: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("enforces worktree cutoffs and agent invokability before timer claims", async () => {
|
|
const f = await fixture();
|
|
enableTimer(f.deps);
|
|
await db.update(issues).set({ monitorNextCheckAt: null }).where(eq(issues.id, f.issue.id));
|
|
f.deps.getWorktreeExecutionCutoff.mockResolvedValue(now);
|
|
await f.scheduler.tickTimers(now);
|
|
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
|
|
await db.update(issues).set({ createdAt: now }).where(eq(issues.id, f.issue.id));
|
|
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agent.id));
|
|
await f.scheduler.tickTimers(now);
|
|
expect(f.deps.claimDueTimerHeartbeat).not.toHaveBeenCalled();
|
|
await db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agent.id));
|
|
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
|
});
|
|
|
|
it("recovers active goals with stable revision-based admission keys", async () => {
|
|
const f = await fixture();
|
|
const active = await session(f);
|
|
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 1, enqueued: 1 });
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
|
|
idempotencyKey: `goal_recovery:${active.id}:7`, contextSnapshot: { issueId: f.issue.id, taskKey: f.issue.id, resumeSessionGoalHeartbeat: true, skipIssueComment: true },
|
|
}));
|
|
f.deps.enqueueWakeup.mockResolvedValueOnce(null);
|
|
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 1, enqueued: 0 });
|
|
expect(f.deps.enqueueWakeup.mock.calls[1]![1]!.idempotencyKey).toBe(f.deps.enqueueWakeup.mock.calls[0]![1]!.idempotencyKey);
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issue.id));
|
|
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
|
|
});
|
|
|
|
it("excludes goals after reassignment or with a mismatched company binding", async () => {
|
|
const f = await fixture();
|
|
const [other] = await db.insert(companies).values({ name: "Other company", issuePrefix: "OTH" }).returning();
|
|
const active = await session(f, { companyId: other!.id });
|
|
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
|
|
await db.update(agentTaskSessions).set({ companyId: f.company.id }).where(eq(agentTaskSessions.id, active.id));
|
|
await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issue.id));
|
|
expect(await f.scheduler.recoverActiveSessionGoals()).toEqual({ scanned: 0, enqueued: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it.each(["queued", "scheduled_retry", "running"])("deduplicates a pending goal action against a %s run", async status => {
|
|
const f = await fixture();
|
|
const active = await session(f);
|
|
const requestId = randomUUID();
|
|
const [action] = await db.insert(agentSessionGoalActions).values({ companyId: f.company.id, sessionId: active.id, requestId, action: "pause", payloadJson: { requestId, action: "pause" } }).returning();
|
|
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.company.id, agentId: f.agent.id, status, contextSnapshot: { goalControlRequestId: requestId } }).returning();
|
|
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 0, alreadyQueued: 1, invalid: 0 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, f.issue.id));
|
|
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 1, alreadyQueued: 0, invalid: 0 });
|
|
expect(f.deps.enqueueWakeup).toHaveBeenCalledWith(f.agent.id, expect.objectContaining({
|
|
idempotencyKey: `goal_control_recovery:${action!.id}`, contextSnapshot: expect.objectContaining({ resumeIntent: true, goalControlRequestId: requestId, runnerGoalControl: { requestId, action: "pause" } }),
|
|
}));
|
|
});
|
|
|
|
it("persists invalid goal actions as failures without wake admission", async () => {
|
|
const f = await fixture();
|
|
const active = await session(f);
|
|
const [action] = await db.insert(agentSessionGoalActions).values({ companyId: f.company.id, sessionId: active.id, requestId: "expected", action: "pause", payloadJson: { requestId: "different", action: "pause" } }).returning();
|
|
expect(await f.scheduler.recoverPendingSessionGoalActions()).toEqual({ scanned: 1, enqueued: 0, alreadyQueued: 0, invalid: 1 });
|
|
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
|
const failed = await db.select().from(agentSessionGoalActions).where(eq(agentSessionGoalActions.id, action!.id));
|
|
expect(failed[0]).toMatchObject({ status: "failed", error: "session_goal_control_payload_invalid" });
|
|
});
|
|
});
|