mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 08:40:01 +02:00
Fix native monitor dispatch and finalization recovery (#16975)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs persist their results and workspace before they release execution. > - A task monitor must wait for that settlement before it starts the next run. > - The scheduler checked provider status but did not check unfinished finalization. > - A timer could dispatch too early, then its old finalizer could no longer find the consumed schedule. > - This pull request delays dispatch and reconciles prior delivery from server records. > - The task can continue without repeating the provider turn or removing a newer run's ownership. ## Linked Issues or Issue Description **What happened?** A native run saved a yielded monitor result while workspace finalization needed recovery. Its timer dispatched after the provider stopped. The follow-up failed because the earlier run still held workspace recovery. The old finalizer then retried `monitor_wait_authority_lost` indefinitely. **Expected behavior** Timer dispatch waits for native finalization. If an older timer already dispatched, the finalizer uses its saved scheduling receipt and admitted successor to settle the old result. It preserves current task state and ownership. **Steps to reproduce** 1. Schedule a task monitor from a native run and save a yielded result. 2. Leave native finalization retryable after the provider reaches a terminal state. 3. Let the monitor become due. Before this change, the scheduler consumes it. 4. Restore workspace persistence and retry the old finalizer. Before this change, the missing schedule leaves it retryable. **Paperclip version or commit** Reproduced against master `34be1b831bcaba070264ae8d214ceffe6a8f9eab`. **Deployment mode** Self-hosted server built from source, with PostgreSQL and the native runner. Related recent recovery work: #16962 and #16963. No duplicate repair found for this failure. ## What Changed - Scheduled and manual monitor dispatch wait for unfinished native finalization. If the monitor expires during workspace recovery, the accepted result can settle into the persisted schedule, and the scheduler then applies its existing expiry policy. New monitor-wait results still require an unexpired, unexhausted schedule. - A narrow reader joins the run's server-issued monitor receipt to its durable wake and admitted successor. - The finalizer and locked status commit verify that evidence and preserve current task ownership. They do not create another wake. - Monitor delivery participates in assessment identity, so recovery can replace a prior error decision without violating the decision ledger's uniqueness constraints. - PostgreSQL regressions cover saved-result replay, active successors, wrong task or agent, missing receipts, unadmitted wakes, ownership races, and failed workspace persistence. - The execution contract and verification maps describe the recovery boundary. ## Verification - Red: the saved monitor regression remained `retryable_failure` instead of `committed` on unpatched code. - Red: the scheduler admitted a wake while terminal-run finalization remained retryable. - Red/green: two PostgreSQL cases reproduce timeout during workspace recovery for scheduled and manual checks. Before the review repair, both remain `retryable_failure` with `monitor_wait_authority_lost`. Afterward, both commit, apply board escalation once, preserve the saved result, and enqueue zero provider wakes. New completion still rejects the expired schedule. - Typecheck: `pnpm -r typecheck` passed. - Build: `pnpm build` passed. - Maps: `node scripts/check-maps.mjs` passed. - Review: Greptile 5/5 and Ponytail Paperclip Review 5/5 on the current head, with no unresolved findings. Both reviews are static; execution evidence is recorded separately above and below. - Linux staging: the original repair passed 170 tests across monitor integration, scheduling, status arbitration, and workspace finalization. The expanded suite passes all 172 tests, including both new timeout regressions. - Live acceptance: deployed the original repair to the authorized Linux ARM64 self-hosted instance. The stranded saved result finalized automatically; the original task then completed through normal admission, with its final answer persisted after browser reload. The other original task was already complete and its external delivery was verified without repeating it. - The initial Mac regression run passed 68 tests, but three suites failed during embedded PostgreSQL startup. A subsequent focused run confirmed the host had exhausted its PostgreSQL semaphore allowance. Linux verification uses isolated databases; the local failure logs remain preserved. - The local full test suite is not claimed as passed. It is blocked by the same PostgreSQL startup limitation; full-suite CI passed on head `8bc29d1b7783446f397b858e558083570a96bb66` ([run](https://github.com/paperclipai/paperclip/actions/runs/39447913466)). ## Risks - A delivery match must not grant new execution authority. The lookup requires the original run's server receipt and a matching company, agent, task, wake, and admitted successor. The locked commit rejects old-run checkout ownership and permits no status effects. - Workspace persistence must succeed before settlement. Its existing failure path remains effective. - The scheduler waits while a native finalizer is unfinished. Workspace recovery must settle before dispatch; elapsed monitor limits transfer to the existing scheduler recovery policy instead of stranding the result. Neither settlement path renews a deadline or changes budget, approval, pause, or provider authority. - No schema, migration, credential, or provider configuration change. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
24758b9da1
commit
98d17da1e2
12 files changed
+348
-16
No files matched your search
@@ -453,6 +453,19 @@ and record `issue.monitor_dispatch_failed`, with no raw provider error. Repeated
|
||||
identical diagnostics for one schedule are limited to one per five minutes.
|
||||
Diagnostics never authorize execution or consume a monitor attempt.
|
||||
|
||||
A native run's terminal provider status does not release its timer while
|
||||
finalization is unfinished. Scheduled and manual monitor dispatch wait for that
|
||||
finalization to settle. For an already-dispatched timer, recovery can settle the
|
||||
old saved monitor result when its server-issued scheduling receipt matches a
|
||||
durable wake and an admitted successor run. The old run must no longer own the
|
||||
task's checkout or execution. This settlement preserves the task and successor
|
||||
ownership, and neither re-arms the timer nor starts another provider turn.
|
||||
Workspace persistence must still succeed first.
|
||||
If a saved monitor expires during workspace recovery, settlement releases the
|
||||
old run into the still-persisted schedule. The scheduler then applies its timeout
|
||||
or attempt-limit recovery policy. Expiration still rejects new monitor-wait
|
||||
results; finalizing accepted work neither renews the deadline nor starts a turn.
|
||||
|
||||
Native `set_task_monitor` arguments put scheduling fields inside `monitor`:
|
||||
`{ "idempotencyKey": "check-ci", "monitor": { "nextCheckAt": "<future UTC timestamp>", "notes": "Check pending CI." } }`.
|
||||
Use `monitor: null` to clear a schedule. A runner schema rejection executes no
|
||||
|
||||
@@ -149,6 +149,14 @@ for the safe blocker cause; unexpected dispatch failures record
|
||||
five minutes. The scheduler regression proves that a held source does not replay
|
||||
and that clearing the hold permits one later wake. This is local PostgreSQL
|
||||
evidence, not live provider recovery.
|
||||
Native timer dispatch also waits for unfinished finalization after a terminal
|
||||
provider run. Recovery may use that run's scheduling receipt and the matching
|
||||
admitted successor as historical delivery evidence. It does not grant new
|
||||
execution authority or clear a successor's ownership. Missing receipts,
|
||||
unadmitted wakes, and unfinished workspace persistence remain held.
|
||||
An accepted result may settle into a schedule that expired during recovery;
|
||||
the scheduler applies its existing expiry policy afterward. This does not
|
||||
authorize a new provider result to wait on an expired or exhausted monitor.
|
||||
|
||||
Gaps: Dedicated tools differ in scope and fields. Generic `call_api` forwards
|
||||
status, assignment and dependency fields to existing route authorization.
|
||||
|
||||
@@ -78,6 +78,15 @@ schema feedback supports a corrected call before a wait is claimed.
|
||||
For a due monitor held by execution recovery, confirm that the schedule remains
|
||||
and task activity explains the deferral. Repeated checks must not replay the held
|
||||
work. Resolve the hold through its existing recovery action, then verify one wake.
|
||||
Hold native finalization after the provider has stopped, and verify that both
|
||||
scheduled and manual checks retain the due monitor. After settlement, verify one
|
||||
wake. If its timeout passes during workspace recovery, verify settlement followed
|
||||
by the configured expiry recovery, with no ordinary monitor wake or duplicate
|
||||
escalation. For a timer already dispatched before recovery, verify that the saved
|
||||
result settles without changing the task, removing a successor's checkout, or
|
||||
creating another wake. The [monitor regression](../../server/src/services/native-runtime/task-monitor.integration.test.ts)
|
||||
and [scheduler regression](../../server/src/services/heartbeat/scheduling.test.ts)
|
||||
cover these boundaries with PostgreSQL.
|
||||
|
||||
Manual: Add an ordinary Board comment while a task retry is queued or running.
|
||||
Confirm that the same execution keeps working and receives the saved message,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import {
|
||||
activityLog, agents, agentSessionGoalActions, agentTaskSessions, companies, createDb,
|
||||
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues,
|
||||
completionContracts, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults,
|
||||
} from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
@@ -12,6 +12,10 @@ import { createHeartbeatRunPreparation } from "./run-preparation.js";
|
||||
import { createHeartbeatRunState } from "./run-state.js";
|
||||
import { createHeartbeatScheduling, type HeartbeatSchedulingDependencies } from "./scheduling.js";
|
||||
import { readIdleSleepSafety } from "../idle-sleep-safety.js";
|
||||
import { PaperclipRunnerToolAuthority } from "../native-runtime/paperclip-runner-tool-authority.js";
|
||||
import { finalizeNativeRun } from "../native-runtime/native-run-finalizer.js";
|
||||
import { nativeCompletionFeedback } from "../native-runtime/native-completion-feedback.js";
|
||||
import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
vi.mock("../live-events.js", () => ({ publishLiveEvent: vi.fn() }));
|
||||
|
||||
@@ -275,6 +279,93 @@ describe.skipIf(!postgres.supported)("heartbeat scheduling database boundary", (
|
||||
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
||||
});
|
||||
|
||||
it("keeps a due monitor until a terminal native run finishes finalization", async () => {
|
||||
const f = await fixture();
|
||||
const [run] = await db.insert(heartbeatRuns).values({
|
||||
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
|
||||
runtimeMode: "native", status: "succeeded", contextSnapshot: { issueId: f.issue.id },
|
||||
}).returning();
|
||||
await db.insert(nativeRunFinalizations).values({
|
||||
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, phase: "retryable_failure",
|
||||
});
|
||||
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
||||
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now })).rejects.toThrow("waiting for the previous run to settle");
|
||||
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
||||
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: dueAt, monitorWakeRequestedAt: null });
|
||||
await db.update(nativeRunFinalizations).set({ phase: "committed" }).where(eq(nativeRunFinalizations.runId, run!.id));
|
||||
expect(await f.scheduler.tickTimers(now)).toEqual({ checked: 1, enqueued: 1, skipped: 0 });
|
||||
expect(f.deps.enqueueWakeup).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it.each(["scheduled", "manual"] as const)("applies expired monitor recovery after workspace settlement without another turn (%s)", async (dispatch) => {
|
||||
const f = await fixture();
|
||||
const start = new Date();
|
||||
const nextCheckAt = new Date(start.getTime() + 60_000).toISOString();
|
||||
const timeoutAt = new Date(start.getTime() + 120_000).toISOString();
|
||||
const [contract] = await db.insert(completionContracts).values({
|
||||
companyId: f.company.id, issueId: f.issue.id, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
|
||||
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(),
|
||||
createdByActorType: "system", createdByActorId: "test",
|
||||
}).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({
|
||||
companyId: f.company.id, agentId: f.agent.id, nativeIssueId: f.issue.id,
|
||||
runtimeMode: "native", status: "running", completionContractId: contract!.id,
|
||||
contextSnapshot: { issueId: f.issue.id },
|
||||
}).returning();
|
||||
await db.update(issues).set({ executionRunId: run!.id, checkoutRunId: run!.id }).where(eq(issues.id, f.issue.id));
|
||||
const authority = new PaperclipRunnerToolAuthority(db, { companyId: f.company.id,
|
||||
issueId: f.issue.id, agentId: f.agent.id, runId: run!.id });
|
||||
await authority.execute({ tool: "set_task_monitor", callId: randomUUID(), arguments: {
|
||||
idempotencyKey: "expires-during-workspace-save",
|
||||
monitor: { nextCheckAt, timeoutAt, notes: "Check once saved", recoveryPolicy: "escalate_to_board" },
|
||||
} });
|
||||
const result: PrpStructuredRunResult = {
|
||||
schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Wait for the check.",
|
||||
completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [],
|
||||
remainingWork: [{ description: "Check after saving", blocksCompletion: true }] },
|
||||
continuation: { kind: "monitor", summary: "Wait for the saved check", idempotencyKey: "wait" },
|
||||
evidence: [], verification: [], attentionRequests: [], artifacts: [],
|
||||
};
|
||||
await expect(nativeCompletionFeedback(db, run!.id, result)).resolves.toContain("issue_monitor_due");
|
||||
const [saved] = await db.insert(nativeRunResults).values({
|
||||
companyId: f.company.id, issueId: f.issue.id, runId: run!.id, completionContractId: contract!.id,
|
||||
serverFingerprint: randomUUID(), canonicalSha256: randomUUID(), schemaStatus: "accepted",
|
||||
resultJson: { result, terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded",
|
||||
turnTerminalState: "completed", reportedWorkDisposition: "yielded" } },
|
||||
}).returning();
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
|
||||
await db.insert(nativeRunFinalizations).values({ companyId: f.company.id, issueId: f.issue.id,
|
||||
runId: run!.id, phase: "result_persisted", resultId: saved!.id });
|
||||
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "failed" });
|
||||
const expiredAt = new Date(start.getTime() + 180_000);
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
vi.setSystemTime(expiredAt);
|
||||
try {
|
||||
await expect(nativeCompletionFeedback(db, run!.id, result)).rejects.toThrow("persisted, eligible monitor");
|
||||
expect(await f.scheduler.tickTimers(expiredAt)).toEqual({ checked: 0, enqueued: 0, skipped: 0 });
|
||||
await expect(f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt })).rejects.toThrow("waiting for the previous run to settle");
|
||||
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run!.id)))[0])
|
||||
.toMatchObject({ phase: "committed", failureCode: null });
|
||||
expect(await readIssue(f.issue.id)).toMatchObject({ executionRunId: null, checkoutRunId: null,
|
||||
monitorNextCheckAt: new Date(nextCheckAt) });
|
||||
if (dispatch === "manual") await f.scheduler.triggerIssueMonitor(f.issue.id, { now: expiredAt });
|
||||
else await f.scheduler.tickTimers(expiredAt);
|
||||
expect(await readIssue(f.issue.id)).toMatchObject({ monitorNextCheckAt: null,
|
||||
executionState: { monitor: { status: "cleared", clearReason: "timeout_exceeded" } } });
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
|
||||
await finalizeNativeRun({ db, runId: run!.id, workspaceFinalizeStatus: "succeeded" });
|
||||
await f.scheduler.tickTimers(expiredAt);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, f.issue.id))).toHaveLength(1);
|
||||
expect(f.deps.enqueueWakeup).not.toHaveBeenCalled();
|
||||
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, saved!.id)))[0]!.resultJson).toEqual(saved!.resultJson);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("skips archived companies for both timers and monitors", async () => {
|
||||
const f = await fixture();
|
||||
enableTimer(f.deps);
|
||||
|
||||
@@ -2,6 +2,7 @@ import {
|
||||
issues,
|
||||
issueComments,
|
||||
heartbeatRuns,
|
||||
nativeRunFinalizations,
|
||||
companies,
|
||||
agentTaskSessions,
|
||||
agentSessionGoalActions,
|
||||
@@ -864,7 +865,11 @@ export function createHeartbeatScheduling(
|
||||
where monitor_run.company_id = ${issues.companyId}
|
||||
and monitor_run.native_issue_id = ${issues.id}
|
||||
and monitor_run.runtime_mode = 'native'
|
||||
and monitor_run.status in ('queued', 'running', 'scheduled_retry'))`;
|
||||
and (monitor_run.status in ('queued', 'running', 'scheduled_retry')
|
||||
or exists (select 1 from ${nativeRunFinalizations} monitor_finalization
|
||||
where monitor_finalization.company_id = monitor_run.company_id
|
||||
and monitor_finalization.run_id = monitor_run.id
|
||||
and monitor_finalization.phase not in ('committed', 'terminal_failure'))))`;
|
||||
}
|
||||
|
||||
async function triggerIssueMonitor(
|
||||
@@ -918,6 +923,7 @@ export function createHeartbeatScheduling(
|
||||
.where(
|
||||
and(
|
||||
eq(issues.id, issueId),
|
||||
noActiveNativeMonitorRun(),
|
||||
sql`${issues.monitorNextCheckAt} is not null`,
|
||||
isNull(issues.assigneeUserId),
|
||||
sql`${issues.assigneeAgentId} is not null`,
|
||||
@@ -933,7 +939,7 @@ export function createHeartbeatScheduling(
|
||||
});
|
||||
|
||||
if (!claimed) {
|
||||
throw conflict("Issue monitor check is already in progress");
|
||||
throw conflict("Issue monitor check is already in progress or waiting for the previous run to settle");
|
||||
}
|
||||
|
||||
return dispatchClaimedIssueMonitor(claimed, {
|
||||
|
||||
@@ -149,18 +149,30 @@ export function prepareIssueMonitorUpdate(
|
||||
return { ...transition.patch, executionPolicy: Object.keys(storedPolicy).length ? storedPolicy : null } as Partial<typeof issues.$inferInsert>;
|
||||
}
|
||||
|
||||
/** Timestamp presence alone is not authority to park a native task. */
|
||||
export function eligibleIssueMonitorWait(
|
||||
/** A saved result can settle into this schedule even if its limits elapsed
|
||||
* during workspace recovery. The scheduler owns applying those limits next. */
|
||||
export function scheduledIssueMonitorWait(
|
||||
issue: typeof issues.$inferSelect,
|
||||
agentId: string,
|
||||
now = new Date(),
|
||||
): string | null {
|
||||
if (issue.assigneeAgentId !== agentId || issue.assigneeUserId ||
|
||||
!["in_progress", "in_review"].includes(issue.status) || !issue.monitorNextCheckAt) return null;
|
||||
const monitor = normalizeIssueExecutionPolicy(issue.executionPolicy)?.monitor;
|
||||
if (!monitor || monitor.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME ||
|
||||
Date.parse(monitor.nextCheckAt) !== issue.monitorNextCheckAt.getTime() ||
|
||||
(monitor.timeoutAt && Date.parse(monitor.timeoutAt) <= now.getTime()) ||
|
||||
(monitor.maxAttempts != null && issue.monitorAttemptCount >= monitor.maxAttempts)) return null;
|
||||
Date.parse(monitor.nextCheckAt) !== issue.monitorNextCheckAt.getTime()) return null;
|
||||
return issue.monitorNextCheckAt.toISOString();
|
||||
}
|
||||
|
||||
/** New provider results still need an unexpired, unexhausted monitor. */
|
||||
export function eligibleIssueMonitorWait(
|
||||
issue: typeof issues.$inferSelect,
|
||||
agentId: string,
|
||||
now = new Date(),
|
||||
): string | null {
|
||||
const nextCheckAt = scheduledIssueMonitorWait(issue, agentId);
|
||||
if (!nextCheckAt) return null;
|
||||
const monitor = normalizeIssueExecutionPolicy(issue.executionPolicy)!.monitor!;
|
||||
if ((monitor.timeoutAt && Date.parse(monitor.timeoutAt) <= now.getTime()) ||
|
||||
(monitor.maxAttempts != null && issue.monitorAttemptCount >= monitor.maxAttempts)) return null;
|
||||
return nextCheckAt;
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
import { agentWakeupRequests, heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { and, desc, eq, inArray, ne, sql } from "drizzle-orm";
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? value as Record<string, unknown> : {};
|
||||
}
|
||||
|
||||
export type NativeMonitorDelivery = { wakeId: string; successorRunId: string };
|
||||
|
||||
/** A dispatched timer is historical delivery evidence, never authority to run again. */
|
||||
export async function readNativeMonitorDelivery(
|
||||
db: Db,
|
||||
binding: { companyId: string; issueId: string; runId: string },
|
||||
): Promise<NativeMonitorDelivery | null> {
|
||||
const [run] = await db.select({
|
||||
agentId: heartbeatRuns.agentId, resultJson: heartbeatRuns.resultJson,
|
||||
}).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, binding.runId),
|
||||
eq(heartbeatRuns.companyId, binding.companyId),
|
||||
eq(heartbeatRuns.nativeIssueId, binding.issueId),
|
||||
eq(heartbeatRuns.runtimeMode, "native"),
|
||||
));
|
||||
if (!run) return null;
|
||||
const keys = Object.values(record(run.resultJson?.semanticToolReceipts)).flatMap(value => {
|
||||
const receipt = record(value);
|
||||
const result = record(receipt.result), monitor = record(result.monitor);
|
||||
if (receipt.operationId !== "set_task_monitor" || result.taskId !== binding.issueId ||
|
||||
monitor.status !== "scheduled" || typeof monitor.nextCheckAt !== "string") return [];
|
||||
return [`issue-monitor:${binding.issueId}:${monitor.nextCheckAt}`];
|
||||
});
|
||||
if (!keys.length) return null;
|
||||
// The issue index bounds this lookup to one task. Joining the admitted run
|
||||
// excludes queued, forged, and cross-company/task delivery references.
|
||||
const [delivery] = await db.select({
|
||||
wakeId: agentWakeupRequests.id, successorRunId: heartbeatRuns.id,
|
||||
}).from(agentWakeupRequests).innerJoin(heartbeatRuns, and(
|
||||
eq(heartbeatRuns.id, agentWakeupRequests.runId),
|
||||
eq(heartbeatRuns.companyId, binding.companyId),
|
||||
eq(heartbeatRuns.agentId, run.agentId),
|
||||
ne(heartbeatRuns.id, binding.runId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId',
|
||||
${heartbeatRuns.contextSnapshot}->>'taskId') = ${binding.issueId}`,
|
||||
)).where(and(
|
||||
eq(agentWakeupRequests.companyId, binding.companyId),
|
||||
eq(agentWakeupRequests.agentId, run.agentId),
|
||||
eq(agentWakeupRequests.reason, "issue_monitor_due"),
|
||||
sql`${agentWakeupRequests.payload}->>'issueId' = ${binding.issueId}`,
|
||||
inArray(agentWakeupRequests.idempotencyKey, keys),
|
||||
)).orderBy(desc(agentWakeupRequests.createdAt), desc(agentWakeupRequests.id)).limit(1);
|
||||
return delivery ?? null;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js";
|
||||
import { eligibleIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { readNativeMonitorDelivery } from "./native-monitor-delivery.js";
|
||||
import { isNativePlanWaitResult, readNativePlanWait } from "./native-plan-wait.js";
|
||||
import { activeIssueInteractionCondition } from "../issue-question-context.js";
|
||||
import { hasPendingNativeChildCompletion } from "./native-child-completion-delivery.js";
|
||||
@@ -1366,8 +1367,14 @@ export async function finalizeNativeRun(input: {
|
||||
};
|
||||
const hasPendingChildCompletion = !reviewContext &&
|
||||
await hasPendingNativeChildCompletion(input.db, childCompletionRecipient);
|
||||
const monitorWaitAt = eligibleIssueMonitorWait(authoritativeIssue, run.agentId);
|
||||
const monitorWaitAt = scheduledIssueMonitorWait(authoritativeIssue, run.agentId);
|
||||
const monitorDelivery = assessment.reportedDisposition === "yielded" &&
|
||||
assessment.continuation?.kind === "monitor" &&
|
||||
authoritativeIssue.executionRunId !== run.id && authoritativeIssue.checkoutRunId !== run.id
|
||||
? await readNativeMonitorDelivery(input.db, { companyId: run.companyId, issueId: coordinator.issueId, runId: run.id })
|
||||
: null;
|
||||
const proposedDecision = resolveNativeFinalizerStatus({
|
||||
monitorWaitDispatched: monitorDelivery !== null,
|
||||
monitorWaitAuthorized: authoritativeIssue.workMode === "standard" && authoritativeIssue.executionRunId === run.id && monitorWaitAt !== null,
|
||||
planWaitAuthorized: planWait !== null,
|
||||
hasPendingChildCompletion,
|
||||
@@ -1423,6 +1430,8 @@ export async function finalizeNativeRun(input: {
|
||||
assessment,
|
||||
supersedesAssessmentId,
|
||||
workspaceFinalizeStatus: input.workspaceFinalizeStatus,
|
||||
monitorDelivery,
|
||||
monitorWaitAt: assessment.continuation?.kind === "monitor" ? monitorWaitAt : null,
|
||||
});
|
||||
await input.db
|
||||
.update(nativeRunFinalizations)
|
||||
@@ -1459,6 +1468,8 @@ export async function finalizeNativeRun(input: {
|
||||
decision,
|
||||
requireMonitorWait: decision.reasonCode === "scheduled_monitor_waiting" && monitorWaitAt
|
||||
? { agentId: run.agentId, nextCheckAt: monitorWaitAt } : undefined,
|
||||
requireMonitorDelivery: decision.reasonCode === "scheduled_monitor_dispatched"
|
||||
? monitorDelivery ?? undefined : undefined,
|
||||
requirePlanWaitSource:
|
||||
decision.reasonCode === "native_plan_accepted_waiting_for_continuation"
|
||||
? planWait?.source : undefined,
|
||||
|
||||
@@ -114,6 +114,8 @@ export function arbitrateNativeStatus(input: {
|
||||
/** Server-verified accepted plan request and normal provider terminal. */
|
||||
planWaitAuthorized?: boolean;
|
||||
monitorWaitAuthorized?: boolean;
|
||||
/** The saved timer already admitted another run and this run no longer owns checkout. */
|
||||
monitorWaitDispatched?: boolean;
|
||||
boardResponseWaitAuthorized?: boolean;
|
||||
boardResponseWaitOrigin?: boolean;
|
||||
isConversation?: boolean;
|
||||
@@ -463,6 +465,16 @@ export function arbitrateNativeStatus(input: {
|
||||
};
|
||||
}
|
||||
if (input.assessment.reportedDisposition === "yielded" && input.assessment.continuation?.kind === "monitor") {
|
||||
if (input.monitorWaitDispatched) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
statusAction: "preserve", toStatus: input.priorIssueStatus,
|
||||
reasonCode: "scheduled_monitor_dispatched", unblockDescriptor: null,
|
||||
// Delivery already happened. Preserve successor ownership and never
|
||||
// re-arm its timer or repeat the completed provider turn.
|
||||
effects: [],
|
||||
};
|
||||
}
|
||||
if (input.monitorWaitAuthorized) {
|
||||
return {
|
||||
policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { eligibleIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { readNativeMonitorDelivery, type NativeMonitorDelivery } from "./native-monitor-delivery.js";
|
||||
import { readNativePlanWait, type NativePlanWaitSource } from "./native-plan-wait.js";
|
||||
import { activeIssueInteractionCondition } from "../issue-question-context.js";
|
||||
import { isConversation } from "../agent-conversations.js";
|
||||
@@ -1556,6 +1557,7 @@ export async function commitNativeStatusDecision(input: {
|
||||
requireExternalChatResponseWaitAuthorization?: { agentId: string };
|
||||
requirePlanWaitSource?: NativePlanWaitSource;
|
||||
requireMonitorWait?: { agentId: string; nextCheckAt: string };
|
||||
requireMonitorDelivery?: NativeMonitorDelivery;
|
||||
requireNoPendingChildCompletion?: NativeChildCompletionRecipient;
|
||||
requireModelRejectionOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null };
|
||||
requireProviderFailureOwner?: { agentId: string; reviewContext: NativeReviewAssignmentContext | null };
|
||||
@@ -1651,10 +1653,19 @@ export async function commitNativeStatusDecision(input: {
|
||||
if (!expected || issue.workMode !== "standard" || issue.executionRunId !== input.runId ||
|
||||
input.decision.statusAction !== "preserve" || input.decision.effects.length !== 1 ||
|
||||
input.decision.effects[0]?.kind !== "release_checkout" ||
|
||||
eligibleIssueMonitorWait(issue, expected.agentId) !== expected.nextCheckAt) {
|
||||
scheduledIssueMonitorWait(issue, expected.agentId) !== expected.nextCheckAt) {
|
||||
throw new NativeStatusRaceError();
|
||||
}
|
||||
}
|
||||
if (reasonCode === "scheduled_monitor_dispatched") {
|
||||
const expected = input.requireMonitorDelivery;
|
||||
const actual = expected ? await readNativeMonitorDelivery(tx as unknown as Db, input) : null;
|
||||
if (!expected || !actual || actual.wakeId !== expected.wakeId ||
|
||||
actual.successorRunId !== expected.successorRunId ||
|
||||
issue.executionRunId === input.runId || issue.checkoutRunId === input.runId ||
|
||||
input.decision.statusAction !== "preserve" || input.decision.toStatus !== issue.status ||
|
||||
input.decision.effects.length !== 0) throw new NativeStatusRaceError();
|
||||
}
|
||||
// Recheck under the parent status lock: a child can finish between the
|
||||
// finalizer's read and this commit. Re-arbitrate instead of discarding its wake.
|
||||
if (input.requireNoPendingChildCompletion && await hasPendingNativeChildCompletion(
|
||||
@@ -1872,6 +1883,7 @@ export async function commitNativeStatusDecision(input: {
|
||||
: {}),
|
||||
...(boardResponseWaitOrigin ? { boardResponseWaitOrigin } : {}),
|
||||
...(planWait ? { planWait: planWait.source, planWaitResult } : {}),
|
||||
...(reasonCode === "scheduled_monitor_dispatched" ? { monitorDelivery: input.requireMonitorDelivery } : {}),
|
||||
priorStatusVersion: input.priorStatusVersion,
|
||||
projectedStatusVersion:
|
||||
input.decision.statusAction === "preserve"
|
||||
|
||||
@@ -2,13 +2,15 @@ import { subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { activityLog, agents, authUsers, companyMemberships, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { activityLog, agents, agentWakeupRequests, authUsers, companyMemberships, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions } from "@paperclipai/db";
|
||||
import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
import { buildIssueMonitorTriggeredPatch, normalizeIssueExecutionPolicy } from "../issue-execution-policy.js";
|
||||
import { nativeCompletionFeedback } from "./native-completion-feedback.js";
|
||||
import { finalizeNativeRun } from "./native-run-finalizer.js";
|
||||
import * as monitorDelivery from "./native-monitor-delivery.js";
|
||||
import type { PrpStructuredRunResult } from "../../vendor/paperclip-runner/index.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
@@ -248,4 +250,99 @@ describeEmbeddedPostgres("native task monitors", () => {
|
||||
...value.monitor, serviceName: "Provider usage dashboard",
|
||||
} })).resolves.toMatchObject({ monitor: { status: "scheduled" }, replayed: false });
|
||||
});
|
||||
|
||||
it.each(["failed_successor", "running_successor", "no_receipt", "other_task", "other_agent", "unadmitted", "still_owned", "ownership_race", "workspace_failed"] as const)(
|
||||
"settles a saved monitor result only with admitted delivery and settled ownership (%s)", async (state) => {
|
||||
const value = await fixture();
|
||||
await value.call({ idempotencyKey: "schedule", monitor: value.monitor });
|
||||
const contractId = randomUUID(), resultId = randomUUID(), successorId = randomUUID();
|
||||
await db.insert(completionContracts).values({
|
||||
id: contractId, companyId: value.companyId, issueId: value.issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test",
|
||||
risk: "standard", completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: { revision: "test", criteria: [] }, canonicalSha256: randomUUID(),
|
||||
createdByActorType: "system", createdByActorId: "test",
|
||||
});
|
||||
const resultJson = {
|
||||
result: {
|
||||
schema: "paperclip.run_result.v1", reportedWorkDisposition: "yielded", summary: "Continue in the selected workspace.",
|
||||
completionClaim: { contractRevision: "test", objectiveSatisfied: false, criteria: [],
|
||||
remainingWork: [{ description: "Measure the archive", blocksCompletion: true }] },
|
||||
continuation: { kind: "monitor", summary: "Wait for the saved check", idempotencyKey: "provider-selected-key" },
|
||||
evidence: [], verification: [], attentionRequests: [], artifacts: [],
|
||||
},
|
||||
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "yielded" },
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", completionContractId: contractId })
|
||||
.where(eq(heartbeatRuns.id, value.runId));
|
||||
await db.insert(nativeRunResults).values({
|
||||
id: resultId, companyId: value.companyId, issueId: value.issueId, runId: value.runId,
|
||||
completionContractId: contractId, serverFingerprint: randomUUID(), canonicalSha256: randomUUID(),
|
||||
schemaStatus: "accepted", resultJson,
|
||||
});
|
||||
await db.insert(nativeRunFinalizations).values({
|
||||
companyId: value.companyId, issueId: value.issueId, runId: value.runId,
|
||||
phase: "retryable_failure", resultId, failureCode: "monitor_wait_authority_lost",
|
||||
});
|
||||
const successorAgentId = state === "other_agent" ? randomUUID() : value.agentId;
|
||||
if (state === "other_agent") await db.insert(agents).values({
|
||||
id: successorAgentId, companyId: value.companyId, name: "Another agent", adapterType: "paperclip_runner",
|
||||
});
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: successorId, companyId: value.companyId, agentId: successorAgentId,
|
||||
status: state === "running_successor" ? "running" : "failed",
|
||||
contextSnapshot: { issueId: state === "other_task" ? randomUUID() : value.issueId },
|
||||
error: "Previous task run still owns workspace recovery",
|
||||
});
|
||||
const scheduled = await value.read();
|
||||
let executionRunId = state === "still_owned" ? value.runId : state === "running_successor" ? successorId : null;
|
||||
await db.update(issues).set({
|
||||
...buildIssueMonitorTriggeredPatch({ issue: scheduled, policy: normalizeIssueExecutionPolicy(scheduled.executionPolicy), triggeredAt: new Date() }),
|
||||
executionRunId, checkoutRunId: executionRunId,
|
||||
}).where(eq(issues.id, value.issueId));
|
||||
// Persist the pre-repair error decision, then retry the same accepted result.
|
||||
await finalizeNativeRun({ db, runId: value.runId, workspaceFinalizeStatus: "succeeded" });
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, value.runId)))[0])
|
||||
.toMatchObject({ phase: "retryable_failure", failureCode: "monitor_wait_authority_lost" });
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({
|
||||
companyId: value.companyId, agentId: value.agentId, source: "automation", reason: "issue_monitor_due",
|
||||
status: state === "unadmitted" ? "queued" : state === "running_successor" ? "claimed" : "failed",
|
||||
runId: state === "unadmitted" ? null : successorId,
|
||||
idempotencyKey: `issue-monitor:${value.issueId}:${value.monitor.nextCheckAt}`,
|
||||
payload: { issueId: value.issueId, nextCheckAt: value.monitor.nextCheckAt },
|
||||
}).returning();
|
||||
if (state === "no_receipt") await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, value.runId));
|
||||
|
||||
const workspaceFinalizeStatus = state === "workspace_failed" ? "failed" : "succeeded";
|
||||
const settles = state === "failed_successor" || state === "running_successor";
|
||||
if (settles) expect(await monitorDelivery.readNativeMonitorDelivery(db, value.binding))
|
||||
.toEqual({ wakeId: wake!.id, successorRunId: successorId });
|
||||
const original = monitorDelivery.readNativeMonitorDelivery;
|
||||
const race = state === "ownership_race"
|
||||
? vi.spyOn(monitorDelivery, "readNativeMonitorDelivery").mockImplementationOnce(async (...args) => {
|
||||
const delivery = await original(...args);
|
||||
expect(delivery).not.toBeNull();
|
||||
executionRunId = value.runId;
|
||||
await db.update(issues).set({ executionRunId, checkoutRunId: executionRunId }).where(eq(issues.id, value.issueId));
|
||||
return delivery;
|
||||
}) : null;
|
||||
try {
|
||||
await finalizeNativeRun({ db, runId: value.runId, workspaceFinalizeStatus });
|
||||
} finally {
|
||||
race?.mockRestore();
|
||||
}
|
||||
const [finalization] = await db.select({
|
||||
phase: nativeRunFinalizations.phase, code: nativeRunFinalizations.failureCode, detail: nativeRunFinalizations.failureDetail,
|
||||
}).from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, value.runId));
|
||||
expect(finalization, JSON.stringify(finalization)).toMatchObject({ phase: settles ? "committed" : "retryable_failure" });
|
||||
expect(await value.read()).toMatchObject({ status: "in_progress", executionRunId, checkoutRunId: executionRunId, monitorNextCheckAt: null });
|
||||
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, resultId)))[0].resultJson).toEqual(resultJson);
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, value.runId));
|
||||
expect(decisions.some(decision => decision.reasonCode === "scheduled_monitor_dispatched")).toBe(settles);
|
||||
if (settles) {
|
||||
await finalizeNativeRun({ db, runId: value.runId, workspaceFinalizeStatus });
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.runId, value.runId))).toEqual(decisions);
|
||||
}
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, value.companyId))).toEqual([wake]);
|
||||
});
|
||||
});
|
||||
@@ -3,6 +3,7 @@ import type { Db } from "@paperclipai/db";
|
||||
import { workAssessments } from "@paperclipai/db";
|
||||
import type { NativeEvidenceAssessment } from "./evidence-classifier.js";
|
||||
import { nativeSha256 } from "./canonical.js";
|
||||
import type { NativeMonitorDelivery } from "./native-monitor-delivery.js";
|
||||
|
||||
export async function recordNativeWorkAssessment(input: {
|
||||
db: Db;
|
||||
@@ -21,9 +22,17 @@ export async function recordNativeWorkAssessment(input: {
|
||||
assessment: NativeEvidenceAssessment;
|
||||
/** Workspace repair changes the status decision even when work evidence is unchanged. */
|
||||
workspaceFinalizeStatus?: "succeeded" | "failed";
|
||||
/** Timer delivery can settle an old wait without changing the work evidence. */
|
||||
monitorDelivery?: NativeMonitorDelivery | null;
|
||||
/** A persisted schedule can regain settlement authority after a policy repair. */
|
||||
monitorWaitAt?: string | null;
|
||||
supersedesAssessmentId?: string | null;
|
||||
}) {
|
||||
const assessmentJson = input.assessment as unknown as Record<string, unknown>;
|
||||
const assessmentJson = {
|
||||
...input.assessment,
|
||||
...(input.monitorDelivery ? { monitorDelivery: input.monitorDelivery } : {}),
|
||||
...(input.monitorWaitAt ? { monitorWaitAt: input.monitorWaitAt } : {}),
|
||||
} as unknown as Record<string, unknown>;
|
||||
const inputDigest = nativeSha256({
|
||||
contractId: input.contractId,
|
||||
contractCanonicalSha256: input.contractCanonicalSha256,
|
||||
|
||||
Reference in new issue
Block a user