mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
fix(recovery): continue Retry after completed file preservation (#18845)
Reuse completed file preservation for task Retry and show bounded admission explanations, including newer saved messages and identifier routes. Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
519b345d69
commit
a80eec1164
13 files changed
+276
-18
No files matched your search
@@ -966,6 +966,18 @@ A new user message can continue a terminal native run whose process fields were
|
||||
|
||||
The task thread exposes the guarded Retry action for failed or timed-out conversation runs and native preparation cancelled before provider startup when the never-started proof is verified. The server projects this eligibility on the recovery notice and rechecks it on Retry. Cleanup quarantine retains its inspection path, and non-conversation reconciliation gates remain enforced. Pending decisions, active execution, pause, budget, dependency, and ownership gates remain in force. A refused Retry reports its reason inline and remains available for another attempt. When a newer linked-human GitHub message was saved but its admission was skipped by a budget or agent-availability gate, ordinary task Retry selects that unstarted message before considering the older failed run. The existing chat task-Retry owner snapshots the complete pending batch, including eligible skipped input, into a new admission generation bound to the canonical processed deliveries, comments, principal, session generation, admitted instruction snapshots, and task Retry intent. It retains the original skipped receipt and failed result unchanged. A terminal failed publication of that old result also remains history for a distinct selected request; pending or uncertain publication still holds work, and the old output is never replayed. Board Retry refusals expose only stable allowlisted authority categories, never source payloads, identifiers, or credential fingerprints. Admission rechecks source integrity and current access, then uses the normal new-user-turn continuation proof: a committed failed result can remain history only after controller, successor, process, workspace, and exact remote-termination guards pass. Unknown ownership stays deferred; Retry cannot certify it. The existing chat retry queue owns delivery and bounded infrastructure retries; a committed receipt settles a lost acknowledgment without another execution. An accepted saved-input receipt discharges only its matching unadmitted-input gate, allowing later genuine conversation or tree-resume wakes without replaying the skipped request.
|
||||
|
||||
A deliberate Retry also binds an already completed file-preservation record to
|
||||
that intent. Other recovery holds for the same source run can then use its saved
|
||||
snapshot when selecting a fresh workspace. This binding preserves the recovery
|
||||
generation, attempt budget, saved files, and stop checks; it does not repeat file
|
||||
recovery or replay the failed turn. An accepted Retry that remains deferred shows
|
||||
its persisted admission reason through a fixed public message. The task view
|
||||
performs bounded follow-up reads after the click so asynchronous admission can
|
||||
explain its wait without a page reload or idle polling.
|
||||
Newer saved input keeps its own receipt while the existing Retry explanation stays
|
||||
visible. Terminal waits direct the user to send new input and hide the Retry action
|
||||
that cannot continue that attempt.
|
||||
|
||||
A continuation cancelled by the queued-run gate while waiting for review did not start a provider. Its matching dispatch-gate receipt and unclaimed execution fields keep this deliberate wait out of unknown-action reconciliation. Pending review and decision gates still apply. If an older server already created a recovery hold for this exact state, Retry or a new user message can request one fresh turn after the server verifies there is no provider, coordinator, or unfinished cleanup. Saved user input is reconsidered through the same admission path; the cancelled turn is not replayed.
|
||||
|
||||
Other legacy queued-run gate cancellations also stay out of unknown-action reconciliation when the terminal run has no retained execution authority, session, workspace recovery, resources, or execution events. The same proof removes the effect of an old false recovery hold only for its exact company, task, and run; the audit row remains. Started or conflicting work retains its hold. Review waits retain their guarded Retry path. New work still passes the current approval, dependency, budget, pause, and ownership gates.
|
||||
|
||||
@@ -101,6 +101,8 @@ export interface ExecutionReconciliation {
|
||||
export interface TaskInterruptionPresentation {
|
||||
providerFailure?: import("../provider-failure.generated.js").ProviderFailure;
|
||||
workspaceFailure?: { message: string };
|
||||
/** A persisted Retry admission wait, normalized to public text by the server. */
|
||||
retryWait?: { reason: string; message: string };
|
||||
intentId: string | null;
|
||||
responsibility: "system" | "user";
|
||||
state: "working" | "retry_scheduled" | "failed" | "waiting_for_user" | "resolved";
|
||||
|
||||
@@ -7,6 +7,28 @@ const run = { id: "run", agentId: "agent", status: "failed", scheduledRetryAt: n
|
||||
resultJson: { projectAccessDenied: { projectId: "22222222-2222-4222-8222-222222222222" } } };
|
||||
const agent = { status: "idle", pauseReason: null };
|
||||
|
||||
it("explains a persisted Retry wait without exposing the saved payload", () => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
const wait = { id: "retry", requestedAt: new Date(), sourceRunId: failed.id,
|
||||
executionWaitReason: "workspace_repair_required" };
|
||||
const presentation = projectTaskInterruption(issue, { run: failed, agent, wait, paused: false, approval: false });
|
||||
expect(presentation).toMatchObject({ state: "failed", canRetry: true,
|
||||
retryWait: { reason: "workspace_repair_required", message: "Retry is waiting for Paperclip to recover the saved files. Your message is saved." } });
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, sourceRunId: "another-run" }, paused: false, approval: false }))
|
||||
.not.toHaveProperty("retryWait");
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, wait: { ...wait, executionWaitReason: "secret-payload" }, paused: false, approval: false }))
|
||||
.toMatchObject({ retryWait: { reason: "execution_recovery", message: "Retry is waiting for execution checks. Your message is saved." } });
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, wait, paused: true, approval: false }))
|
||||
.toMatchObject({ state: "waiting_for_user", userAction: "resume" });
|
||||
});
|
||||
|
||||
it.each(["cleanup_quarantined", "cancelled_by_operator"])("does not offer Retry for terminal Retry waits: %s", reason => {
|
||||
const failed = { ...run, errorCode: "native_provider_terminal_failed", resultJson: null };
|
||||
expect(projectTaskInterruption(issue, { run: failed, agent, paused: false, approval: false,
|
||||
wait: { id: "retry", requestedAt: new Date(), sourceRunId: failed.id, executionWaitReason: reason } }))
|
||||
.toMatchObject({ state: "failed", canRetry: false, retryWait: { reason } });
|
||||
});
|
||||
|
||||
it("preserves the actual workspace preparation error without changing retry authority", () => {
|
||||
const failedRun = { ...run, errorCode: "workspace_validation_failed",
|
||||
error: "The selected project has no repository source for a remote workspace.", resultJson: null };
|
||||
|
||||
@@ -19,10 +19,12 @@ type ProjectionIssue = Pick<typeof issues.$inferSelect, "id" | "companyId" | "st
|
||||
type ProjectionRun = Pick<typeof heartbeatRuns.$inferSelect,
|
||||
"id" | "status" | "scheduledRetryAt" | "errorCode" | "livenessState" | "resultJson"> &
|
||||
Partial<Pick<typeof heartbeatRuns.$inferSelect, "agentId" | "error" | "companyId" | "nativeIssueId" | "runtimeMode">>;
|
||||
type ProjectionWait = { id: string; requestedAt: Date; sourceRunId?: string | null; executionWaitReason?: string | null };
|
||||
type ProjectionState = {
|
||||
run?: ProjectionRun;
|
||||
action?: typeof issueRecoveryActions.$inferSelect;
|
||||
wait?: { id: string; requestedAt: Date };
|
||||
wait?: ProjectionWait;
|
||||
retryWait?: ProjectionWait;
|
||||
agent?: Pick<typeof agents.$inferSelect, "status" | "pauseReason">;
|
||||
projectAccessAllowed?: boolean;
|
||||
projectAccessProjectId?: string | null;
|
||||
@@ -34,9 +36,27 @@ type ProjectionState = {
|
||||
};
|
||||
|
||||
function boundedJsonString(value: SQL, limit: number) {
|
||||
return sql`case when jsonb_typeof(${value}) = 'string' then left(${value} #>> '{}', ${limit}) else null end`;
|
||||
return sql<string | null>`case when jsonb_typeof(${value}) = 'string' then left(${value} #>> '{}', ${limit}) else null end`;
|
||||
}
|
||||
|
||||
// Persisted wait messages can contain diagnostics. Ordinary task views expose
|
||||
// only these stable categories, never arbitrary payloads or source identifiers.
|
||||
const retryWaitMessages: Record<string, string> = {
|
||||
workspace_repair_required: "Retry is waiting for Paperclip to recover the saved files. Your message is saved.",
|
||||
remote_cleanup: "Retry is waiting for the previous environment to stop. Your message is saved.",
|
||||
local_cleanup: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
|
||||
cleanup_pending: "Retry is waiting for the previous execution to finish cleanup. Your message is saved.",
|
||||
execution_settling: "Retry is waiting for the previous execution to stop. Your message is saved.",
|
||||
controller_settling: "Retry is waiting for the previous execution to finish recovery. Your message is saved.",
|
||||
process_identity_missing: "Retry is waiting for Paperclip to verify that the previous execution stopped. Your message is saved.",
|
||||
process_running: "Retry is waiting for the previous execution to stop. Your message is saved.",
|
||||
execution_active: "Retry is waiting for the current run to finish. Your message is saved.",
|
||||
decision_pending: "Retry is waiting for a pending approval or question to be resolved. Your message is saved.",
|
||||
finalizing_saved_result: "Paperclip is finishing the saved result before Retry can continue.",
|
||||
cleanup_quarantined: "Retry cannot continue this attempt. Send a new message after the previous execution has stopped.",
|
||||
cancelled_by_operator: "This attempt was stopped. Send a new message to continue from saved work.",
|
||||
};
|
||||
|
||||
/** Recognize old persisted failures without displaying arbitrary run error text. */
|
||||
function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string | null | undefined {
|
||||
const historical = run?.errorCode === "setup_failed" && run.error === "Project is outside this low-trust boundary.";
|
||||
@@ -51,6 +71,7 @@ function deniedProjectId(issue: ProjectionIssue, run?: ProjectionRun): string |
|
||||
* locks. The same projection is used in detail, lists and attention surfaces. */
|
||||
export function projectTaskInterruption(issue: ProjectionIssue, input: ProjectionState, now = Date.now()): TaskInterruptionPresentation | null {
|
||||
const { run, wait, action, agent, project, company } = input;
|
||||
const retryWait = input.retryWait ?? wait;
|
||||
if (!run) return null;
|
||||
const system = readSystemRecovery(action?.evidence);
|
||||
const base: TaskInterruptionPresentation = { intentId: wait?.id ?? null, responsibility: "system",
|
||||
@@ -90,6 +111,13 @@ export function projectTaskInterruption(issue: ProjectionIssue, input: Projectio
|
||||
const activeObservation = action?.evidence.systemRecoveryActiveAt;
|
||||
if (system?.state === "running" && typeof activeObservation === "number" &&
|
||||
activeObservation > now - 45_000) return { ...base, state: "working" };
|
||||
if (retryWait?.sourceRunId === run.id && retryWait.executionWaitReason) {
|
||||
const reason = Object.hasOwn(retryWaitMessages, retryWait.executionWaitReason) ? retryWait.executionWaitReason : "execution_recovery";
|
||||
// Keep the latest input receipt so message submission can settle independently.
|
||||
return { ...base, state: "failed",
|
||||
canRetry: !["cleanup_quarantined", "cancelled_by_operator"].includes(reason), retryWait: { reason,
|
||||
message: retryWaitMessages[reason] ?? "Retry is waiting for execution checks. Your message is saved." } };
|
||||
}
|
||||
if (wait && wait.requestedAt.getTime() > now - 10_000 && system?.state !== "exhausted")
|
||||
return { ...base, state: "working" };
|
||||
return { ...base, state: "failed", canRetry: true, ...(projectAccess ? { projectAccess } : {}) };
|
||||
@@ -118,6 +146,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const latestRuns = db.select({ id: candidates.id }).from(candidates)
|
||||
.orderBy(desc(candidates.createdAt), desc(candidates.id)).limit(1).as("latest_interruption_runs");
|
||||
const wakeIssue = sql<string>`${agentWakeupRequests.payload}->>'issueId'`;
|
||||
const isRetryWait = sql<boolean>`coalesce(jsonb_typeof(${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}') = 'string', false)`;
|
||||
const agentIds = selected.flatMap(issue => issue.assigneeAgentId ? [issue.assigneeAgentId] : []);
|
||||
const projectIds = selected.flatMap(issue => issue.projectId ? [issue.projectId] : []);
|
||||
const [runs, waits, agentRows, projectRows, pauseRows, approvalRows, companyRows] = await Promise.all([
|
||||
@@ -148,10 +177,14 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
}).from(requested).innerJoinLateral(latestRuns, sql`true`).innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, latestRuns.id)))
|
||||
.leftJoin(agents, and(eq(agents.companyId, heartbeatRuns.companyId), eq(agents.id, heartbeatRuns.agentId)))
|
||||
.orderBy(requestedIssueId),
|
||||
db.selectDistinctOn([wakeIssue], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt })
|
||||
// Keep at most two candidates per task: latest Retry and latest other input.
|
||||
// A new message must not hide an existing Retry's admission explanation.
|
||||
db.selectDistinctOn([wakeIssue, isRetryWait], { issueId: wakeIssue, id: agentWakeupRequests.id, requestedAt: agentWakeupRequests.requestedAt,
|
||||
sourceRunId: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{taskRetry,sourceRunId}'`, 160),
|
||||
executionWaitReason: boundedJsonString(sql`${agentWakeupRequests.payload}#>'{executionWait,reason}'`, 80) })
|
||||
.from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, companyId), inArray(wakeIssue, ids),
|
||||
eq(agentWakeupRequests.status, "deferred_issue_execution"), eq(agentWakeupRequests.requestedByActorType, "user")))
|
||||
.orderBy(wakeIssue, desc(agentWakeupRequests.requestedAt)),
|
||||
.orderBy(wakeIssue, isRetryWait, desc(agentWakeupRequests.requestedAt), desc(agentWakeupRequests.id)),
|
||||
agentIds.length ? db.select({ id: agents.id, status: agents.status, pauseReason: agents.pauseReason }).from(agents)
|
||||
.where(and(eq(agents.companyId, companyId), inArray(agents.id, agentIds))) : [],
|
||||
projectIds.length ? db.select({ id: projects.id, pausedAt: projects.pausedAt, pauseReason: projects.pauseReason }).from(projects)
|
||||
@@ -173,7 +206,13 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
sql`${issueRecoveryActions.evidence} ? 'systemRecovery'`,
|
||||
)).orderBy(issueRecoveryActions.sourceIssueId, desc(issueRecoveryActions.createdAt)) : [];
|
||||
const byIssue = <T extends { issueId: string }>(rows: T[]) => new Map(rows.map(row => [row.issueId, row]));
|
||||
const runMap = byIssue(runs), waitMap = byIssue(waits);
|
||||
const runMap = byIssue(runs), waitMap = new Map<string, typeof waits[number]>();
|
||||
const retryWaitMap = byIssue(waits.filter(wait => wait.sourceRunId));
|
||||
for (const wait of waits) {
|
||||
const current = waitMap.get(wait.issueId);
|
||||
if (!current || wait.requestedAt > current.requestedAt ||
|
||||
(wait.requestedAt.getTime() === current.requestedAt.getTime() && wait.id > current.id)) waitMap.set(wait.issueId, wait);
|
||||
}
|
||||
// Recheck current authority, including standard trust, root-issue grants and
|
||||
// project policy. Do not reuse the failed run's frozen policy for a new retry.
|
||||
// Only denied tasks need these reads; deduplicate projects and batch per agent.
|
||||
@@ -205,7 +244,7 @@ export async function taskInterruptionPresentations(db: Db, companyId: string, i
|
||||
const required = requirements.get(issue.id);
|
||||
const denied = required?.projectIds.filter(projectId => currentAccess.get(`${issue.assigneeAgentId}:${projectId}`) !== true);
|
||||
const presentation = projectTaskInterruption(issue, {
|
||||
run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id),
|
||||
run: runMap.get(issue.id), action: actionMap.get(issue.id), wait: waitMap.get(issue.id), retryWait: retryWaitMap.get(issue.id),
|
||||
projectAccessAllowed: denied?.length === 0,
|
||||
projectAccessProjectId: denied?.includes(issue.projectId ?? "") ? issue.projectId : null,
|
||||
workspaceSelectionAvailable: required?.available ?? false,
|
||||
|
||||
@@ -331,8 +331,11 @@ export async function requestTaskRetry(input: {
|
||||
if (!busy) {
|
||||
for (const action of records) {
|
||||
const previous = readSystemRecovery(action.evidence);
|
||||
if (!previous || previous.state === "complete") continue;
|
||||
if (previous.state === "running") {
|
||||
if (!previous) continue;
|
||||
// A completed preservation still owns its snapshot evidence. Bind the
|
||||
// new intent so sibling holds can select that same continuation, without
|
||||
// repeating recovery or renewing its attempt budget.
|
||||
if (previous.state === "complete" || previous.state === "running") {
|
||||
await updateSystemRecovery(db, { companyId: issue.companyId, issueId: issue.id, runId: latest.id,
|
||||
actionId: action.id }, previous, { ...previous, intentId: active?.id ?? id });
|
||||
continue;
|
||||
|
||||
@@ -27,7 +27,7 @@ import { environmentRunOrchestrator } from "./environment-run-orchestrator.js";
|
||||
import { executionWorkspaceService } from "./execution-workspaces.js";
|
||||
import { workspaceOperationService } from "./workspace-operations.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
import { projectTaskInterruption } from "./task-interruption-presentation.js";
|
||||
import { projectTaskInterruption, taskInterruptionPresentation } from "./task-interruption-presentation.js";
|
||||
import * as nativeWorkspaceSync from "./native-runtime/native-workspace-sync.js";
|
||||
import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
@@ -1674,6 +1674,66 @@ describe("automatic retained workspace recovery", () => {
|
||||
}
|
||||
}, 30_000);
|
||||
|
||||
it.each(["stopped", "unverified_stop"])("retries a recovered snapshot with sibling holds once: %s", async state => {
|
||||
const f = await seed("native", false);
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeWorkspaceSync: {
|
||||
schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64),
|
||||
baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: f.issueId,
|
||||
leaseId: f.lease.id, providerLeaseId: f.providerLeaseId, remoteCwd: f.remote, resourceDisposition: "keep_running",
|
||||
} } }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", status: "resolved", ownerType: "board", outcome: "blocked",
|
||||
cause: "provider_failure_meaning_unverified", fingerprint: randomUUID(), nextAction: "Inspect original failure",
|
||||
evidence: { runId: f.runId, automaticRecovery: { replay: "blocked" } } });
|
||||
const source = await run(f.runId);
|
||||
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: source.agentId, status: "running" });
|
||||
await recoverRetainedWorkspace(f.input);
|
||||
const recovered = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
expect(recovered.state).toBe("complete");
|
||||
expect(recovered.intentId).toBeNull();
|
||||
if (state === "unverified_stop") await db.update(environmentLeases).set({ metadata: f.lease.metadata })
|
||||
.where(eq(environmentLeases.id, f.lease.id));
|
||||
const receipt = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
actorId: "board", actorSource: "local_implicit", idempotencyKey: randomUUID(), observedRunId: f.runId });
|
||||
const service = heartbeatService(db);
|
||||
await Promise.all([service.resumeTaskRetry(f.companyId, receipt.requestId!), service.resumeTaskRetry(f.companyId, receipt.requestId!)]);
|
||||
const turns = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
if (state === "unverified_stop") {
|
||||
expect(turns).toHaveLength(0);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
expect(wake.payload?.executionWait).toMatchObject({ reason: "remote_cleanup" });
|
||||
const [task] = await db.select().from(issues).where(eq(issues.id, f.issueId));
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ state: "failed", canRetry: true,
|
||||
retryWait: { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." } });
|
||||
const commentId = randomUUID(), messageWakeId = randomUUID();
|
||||
await db.insert(issueComments).values({ id: commentId, companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: "board", body: "Keep my saved message while the previous environment stops." });
|
||||
await db.insert(agentWakeupRequests).values({ id: messageWakeId, companyId: f.companyId, agentId: wake.agentId,
|
||||
source: "automation", reason: "issue_commented", status: "deferred_issue_execution",
|
||||
requestedByActorType: "user", requestedByActorId: "board", requestedAt: new Date(wake.requestedAt.getTime() + 1),
|
||||
payload: { issueId: f.issueId, commentId } });
|
||||
expect(await taskInterruptionPresentation(db, task)).toMatchObject({ intentId: messageWakeId,
|
||||
retryWait: { reason: "remote_cleanup" } });
|
||||
await db.update(agentWakeupRequests).set({ status: "coalesced" }).where(eq(agentWakeupRequests.id, receipt.requestId!));
|
||||
const newInput = await taskInterruptionPresentation(db, task);
|
||||
expect(newInput).toMatchObject({ state: "working", intentId: messageWakeId });
|
||||
expect(newInput).not.toHaveProperty("retryWait");
|
||||
return;
|
||||
}
|
||||
expect(turns).toHaveLength(1);
|
||||
expect(turns[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.runId, taskRetryRequestId: receipt.requestId });
|
||||
expect(await readFreshWorkspaceReadiness(db, turns[0])).toMatchObject({ mode: "fresh_isolated_workspace", sourceRunId: f.runId });
|
||||
const fresh = path.join(root, randomUUID()); await mkdir(fresh);
|
||||
const attached = await attachPreservedWorkspaceSnapshots({ db, companyId: f.companyId, issueId: f.issueId,
|
||||
sourceRunIds: [turns[0].id], cwd: fresh });
|
||||
expect(await readFile(path.join(attached[0].path, "files", "saved.txt"), "utf8")).toBe("finished work\n");
|
||||
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
|
||||
const final = readSystemRecovery((await action(f.actionId)).evidence)!;
|
||||
expect(final).toMatchObject({ generationId: recovered.generationId, attempts: recovered.attempts,
|
||||
retriesUsed: recovered.retriesUsed, preservationOperationIds: recovered.preservationOperationIds, intentId: receipt.requestId });
|
||||
expect((await run(f.runId)).status).toBe("failed");
|
||||
}, 30_000);
|
||||
|
||||
it.each(["copyback_unknown", "source_unavailable", "executor_unknown"])("ordinary task Retry evaluates fresh isolation independently: %s", async failure => {
|
||||
const f = await seed("legacy", false);
|
||||
const source = await run(f.runId);
|
||||
|
||||
@@ -70,6 +70,17 @@ describe("ordinary task interruption", () => {
|
||||
await render({ ...failed, workspaceFailure, state: "working" });
|
||||
expect(container.textContent).toBe("");
|
||||
});
|
||||
it("shows why an accepted Retry is waiting and keeps another attempt available", async () => {
|
||||
const retryWait = { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." };
|
||||
await render({ ...failed, retryWait });
|
||||
expect(container.textContent).toContain(retryWait.message);
|
||||
expect(container.textContent).not.toContain("Couldn’t finish this attempt.");
|
||||
await act(async () => container.querySelector<HTMLButtonElement>("button")!.click());
|
||||
expect(retry).toHaveBeenCalledOnce();
|
||||
await render({ ...failed, retryWait }, { canRetry: false });
|
||||
expect(container.textContent).toContain(retryWait.message);
|
||||
expect(container.querySelector("button")).toBeNull();
|
||||
});
|
||||
it("shows an unknown provider's original explanation as text and preserves read-only access", async () => {
|
||||
const providerFailure = { schema: "paperclip.provider-failure.v1" as const, provider: "new-provider", phase: "startup" as const,
|
||||
category: "unknown" as const, retryability: "unknown" as const, httpStatus: 429,
|
||||
|
||||
@@ -25,7 +25,7 @@ export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimi
|
||||
canRetry={canRetry && presentation.canRetry} onRetry={onRetry} error={error} className={className} />;
|
||||
if (error && !optimistic && !takingLonger) return <div role="alert" className={cn("flex items-center justify-between gap-2 text-sm text-destructive", className)}><span>{error.message}</span>{canRetry && <Button variant="outline" size="sm" onClick={onRetry}>Retry</Button>}</div>;
|
||||
if (takingLonger) return <div role="status" className={cn("flex items-center justify-between gap-2 text-sm text-muted-foreground", className)}><span>Taking longer than usual…</span>{canRetry && <Button variant="outline" size="sm" onClick={onRetry}>Retry</Button>}</div>;
|
||||
if (!optimistic && presentation?.providerFailure && ["failed", "retry_scheduled", "waiting_for_user"].includes(presentation.state)) {
|
||||
if (!optimistic && !presentation?.retryWait && presentation?.providerFailure && ["failed", "retry_scheduled", "waiting_for_user"].includes(presentation.state)) {
|
||||
return <ProviderFailureNotice failure={presentation.providerFailure} className={className}
|
||||
actions={canRetry && presentation.canRetry ? <Button variant="outline" size="sm" onClick={onRetry}>Retry</Button> : undefined}>
|
||||
{presentation.state === "retry_scheduled" && <p>Paperclip will retry automatically.</p>}
|
||||
@@ -43,6 +43,7 @@ export function ExecutionBlockerNotice({ presentation, canRetry, onRetry, optimi
|
||||
{presentation.state === "failed" && <AlertCircle className="size-5 shrink-0 text-destructive" aria-hidden="true" />}
|
||||
<span className="min-w-0 flex-1">{presentation.state === "retry_scheduled"
|
||||
? `Trying again in ${seconds > 60 ? `${Math.ceil(seconds / 60)}m` : `${seconds}s`}…`
|
||||
: presentation.retryWait ? presentation.retryWait.message
|
||||
: presentation.workspaceFailure
|
||||
? <>Couldn’t prepare the task folders. {presentation.workspaceFailure.message}</>
|
||||
: "Couldn’t finish this attempt. Your message is saved."}</span>
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
// @vitest-environment jsdom
|
||||
import { act } from "react";
|
||||
import { createRoot } from "react-dom/client";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { TaskInterruptionPresentation } from "@paperclipai/shared";
|
||||
import type { IssueChatComment } from "../lib/issue-chat-messages";
|
||||
import { ExecutionBlockerNotice } from "../components/ExecutionBlockerNotice";
|
||||
import { useRunnerSubmission } from "./useRunnerSubmission";
|
||||
|
||||
it("settles a newer saved message while the same Retry explanation remains visible", async () => {
|
||||
vi.useFakeTimers();
|
||||
const container = document.createElement("div"); document.body.append(container);
|
||||
const root = createRoot(container);
|
||||
const wait: TaskInterruptionPresentation = { intentId: "retry", responsibility: "system", state: "failed",
|
||||
canRetry: true, nextRetryAt: null, userAction: null, observedRunId: "failed-run", sourceRunIds: ["failed-run"],
|
||||
retryWait: { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." } };
|
||||
function View({ interruption = wait, comments = [] }: { interruption?: TaskInterruptionPresentation; comments?: IssueChatComment[] }) {
|
||||
const submission = useRunnerSubmission({ issueId: "task", agentId: "agent", agent: { status: "idle" },
|
||||
issueStatus: "blocked", runId: null, comments, interruption });
|
||||
return <><button onClick={() => submission.begin("message")}>Send</button>
|
||||
{submission.pending && <span>{submission.takingLonger ? "Taking longer than usual…" : "Thinking"}</span>}
|
||||
<ExecutionBlockerNotice presentation={interruption} canRetry onRetry={() => {}} /></>;
|
||||
}
|
||||
try {
|
||||
await act(async () => root.render(<View />));
|
||||
await act(async () => container.querySelector("button")!.click());
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(10_000); });
|
||||
expect(container.textContent).toContain("Taking longer than usual…");
|
||||
const comments: IssueChatComment[] = [{ id: "comment", clientRequestId: "message", companyId: "company", issueId: "task",
|
||||
authorType: "user", authorAgentId: null, authorUserId: "board", body: "Keep my saved message.",
|
||||
presentation: null, metadata: null, createdAt: new Date(), updatedAt: new Date() }];
|
||||
await act(async () => root.render(<View comments={comments} />));
|
||||
expect(container.textContent).toContain("Taking longer than usual…");
|
||||
await act(async () => root.render(<View comments={comments} interruption={{ ...wait, intentId: "new-message-receipt" }} />));
|
||||
expect(container.textContent).not.toContain("Taking longer than usual…");
|
||||
expect(container.textContent).not.toContain("Thinking");
|
||||
expect(container.textContent).toContain(wait.retryWait!.message);
|
||||
} finally {
|
||||
await act(async () => root.unmount()); container.remove(); vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
@@ -1,13 +1,14 @@
|
||||
// @vitest-environment jsdom
|
||||
import { act } from "react";
|
||||
import { createRoot, type Root } from "react-dom/client";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { QueryClient, QueryClientProvider, useQuery } from "@tanstack/react-query";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Issue } from "@paperclipai/shared";
|
||||
import { useTaskRetry } from "./useTaskRetry";
|
||||
import { ExecutionBlockerNotice } from "../components/ExecutionBlockerNotice";
|
||||
import { ApiError } from "../api/client";
|
||||
import { issuesApi } from "../api/issues";
|
||||
import { queryKeys } from "../lib/queryKeys";
|
||||
vi.mock("../api/issues", () => ({ issuesApi: { retryTask: vi.fn() } }));
|
||||
const issue = { id: "task", interruption: { observedRunId: "failed-run" } } as Issue;
|
||||
function View({ task = issue }: { task?: Issue }) {
|
||||
@@ -67,4 +68,33 @@ describe("optimistic task Retry", () => {
|
||||
await act(async () => settlements[0]({ requestId: "old", accepted: true, stale: false, presentation: null }));
|
||||
expect(container.textContent).toContain("Thinking");
|
||||
});
|
||||
it.each(["id", "identifier"])("reads a deferred Retry reason on the %s route and stops checking once explained", async route => {
|
||||
vi.useFakeTimers();
|
||||
const presentation = { intentId: "intent", responsibility: "system" as const, state: "failed" as const,
|
||||
canRetry: true, nextRetryAt: null, userAction: null, sourceRunIds: ["failed-run"], observedRunId: "failed-run" };
|
||||
let saved = { ...issue, identifier: "RTA-1", interruption: presentation } as Issue;
|
||||
const queryFn = vi.fn(async () => saved);
|
||||
const detailKey = queryKeys.issues.detail(route === "identifier" ? saved.identifier! : issue.id);
|
||||
client.setQueryData(detailKey, saved);
|
||||
function Observed() {
|
||||
const { data } = useQuery({ queryKey: detailKey, queryFn, staleTime: Infinity });
|
||||
const retry = useTaskRetry(data);
|
||||
return <ExecutionBlockerNotice presentation={data?.interruption} canRetry onRetry={retry.retry}
|
||||
optimistic={retry.optimistic} takingLonger={retry.takingLonger} error={retry.error} />;
|
||||
}
|
||||
vi.mocked(issuesApi.retryTask).mockResolvedValue({ requestId: "intent", accepted: true, stale: false, presentation });
|
||||
await act(async () => root.render(<QueryClientProvider client={client}><Observed /></QueryClientProvider>));
|
||||
await act(async () => container.querySelector("button")!.click());
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(1); });
|
||||
saved = { ...saved, interruption: { ...presentation, retryWait: {
|
||||
reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved.",
|
||||
} } };
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(1_000); });
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(1); });
|
||||
expect(container.textContent).toContain(saved.interruption!.retryWait!.message);
|
||||
const reads = queryFn.mock.calls.length;
|
||||
await act(async () => { await vi.advanceTimersByTimeAsync(60_000); });
|
||||
expect(queryFn).toHaveBeenCalledTimes(reads);
|
||||
expect(issuesApi.retryTask).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
@@ -5,22 +5,27 @@ import { ApiError } from "../api/client";
|
||||
import { issuesApi } from "../api/issues";
|
||||
import { queryKeys } from "../lib/queryKeys";
|
||||
|
||||
type RetryRequest = { issueId: string; identifier?: string | null; idempotencyKey: string; observedRunId: string | null };
|
||||
|
||||
export function useTaskRetry(issue: Issue | undefined) {
|
||||
const queryClient = useQueryClient();
|
||||
const request = useRef<{ issueId: string; idempotencyKey: string; observedRunId: string | null } | null>(null);
|
||||
const request = useRef<RetryRequest | null>(null);
|
||||
const [optimistic, setOptimistic] = useState(false);
|
||||
const [takingLonger, setTakingLonger] = useState(false);
|
||||
const [reconcile, setReconcile] = useState<{ refs: string[]; sourceRunId: string | null } | null>(null);
|
||||
const mutation = useMutation({
|
||||
mutationFn: ({ issueId, ...body }: { issueId: string; idempotencyKey: string; observedRunId: string | null }) =>
|
||||
mutationFn: ({ issueId, identifier: _identifier, ...body }: RetryRequest) =>
|
||||
issuesApi.retryTask(issueId, body),
|
||||
onSuccess: async (result, variables) => {
|
||||
const isCurrent = request.current?.idempotencyKey === variables.idempotencyKey;
|
||||
if (isCurrent) request.current = null;
|
||||
const refs = [...new Set([variables.issueId, variables.identifier].filter((ref): ref is string => Boolean(ref)))];
|
||||
if (isCurrent && result.accepted) setReconcile({ refs, sourceRunId: variables.observedRunId });
|
||||
{
|
||||
queryClient.setQueryData<Issue>(queryKeys.issues.detail(variables.issueId), current =>
|
||||
for (const ref of refs) queryClient.setQueryData<Issue>(queryKeys.issues.detail(ref), current =>
|
||||
current ? { ...current, interruption: result.presentation } : current);
|
||||
await Promise.all([queryKeys.issues.detail(variables.issueId), queryKeys.issues.runs(variables.issueId),
|
||||
queryKeys.issues.liveRuns(variables.issueId), queryKeys.issues.activeRun(variables.issueId)].map(queryKey =>
|
||||
await Promise.all(refs.flatMap(ref => [queryKeys.issues.detail(ref), queryKeys.issues.runs(ref),
|
||||
queryKeys.issues.liveRuns(ref), queryKeys.issues.activeRun(ref)]).map(queryKey =>
|
||||
queryClient.invalidateQueries({ queryKey })));
|
||||
}
|
||||
if (isCurrent && !request.current) { setOptimistic(false); setTakingLonger(false); }
|
||||
@@ -36,6 +41,17 @@ export function useTaskRetry(issue: Issue | undefined) {
|
||||
}
|
||||
},
|
||||
});
|
||||
useEffect(() => {
|
||||
if (!reconcile || !issue || !reconcile.refs.includes(issue.id) || issue.interruption?.retryWait ||
|
||||
issue.interruption?.observedRunId !== reconcile.sourceRunId) return;
|
||||
// Dispatch follows the committed HTTP receipt. Read its admission reason
|
||||
// after that handoff, including the existing ten-second presentation grace.
|
||||
// These bounded reads end with this click; idle tasks do not poll.
|
||||
const timers = [1_000, 10_000].map(delay => window.setTimeout(() => {
|
||||
for (const ref of reconcile.refs) void queryClient.invalidateQueries({ queryKey: queryKeys.issues.detail(ref) });
|
||||
}, delay));
|
||||
return () => timers.forEach(timer => window.clearTimeout(timer));
|
||||
}, [reconcile, issue?.id, issue?.interruption?.observedRunId, issue?.interruption?.retryWait, queryClient]);
|
||||
useEffect(() => {
|
||||
if (!optimistic) return;
|
||||
const timer = window.setTimeout(() => {
|
||||
@@ -50,12 +66,14 @@ export function useTaskRetry(issue: Issue | undefined) {
|
||||
request.current = null;
|
||||
setOptimistic(false);
|
||||
setTakingLonger(false);
|
||||
setReconcile(null);
|
||||
mutation.reset();
|
||||
}, [issue?.id]);
|
||||
return { retry: () => {
|
||||
if (!issue) return;
|
||||
setReconcile(null);
|
||||
mutation.reset();
|
||||
request.current ??= { issueId: issue.id, idempotencyKey: crypto.randomUUID(),
|
||||
request.current ??= { issueId: issue.id, identifier: issue.identifier, idempotencyKey: crypto.randomUUID(),
|
||||
observedRunId: issue.interruption?.observedRunId ?? issue.executionBlocker?.runId ?? null };
|
||||
setOptimistic(true);
|
||||
setTakingLonger(false);
|
||||
|
||||
@@ -2131,6 +2131,23 @@ describe("IssueDetail", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.each([false, true])("refreshes admission with latest comments (compact chat: %s)", async compact => {
|
||||
const interruption: NonNullable<Issue["interruption"]> = { intentId: "retry", responsibility: "system", state: "failed",
|
||||
nextRetryAt: null, canRetry: true, userAction: null, sourceRunIds: ["failed-run"], observedRunId: "failed-run",
|
||||
retryWait: { reason: "remote_cleanup", message: "Retry is waiting for the previous environment to stop. Your message is saved." } };
|
||||
let saved = createIssue({ status: "blocked", assigneeAgentId: "agent-1", interruption,
|
||||
...(compact ? { conversationAgentId: "agent-1", conversationUserId: "user-1" } : {}) });
|
||||
mockIssuesApi.get.mockImplementation(async () => saved);
|
||||
await act(async () => renderIssueView());
|
||||
const props = () => mockIssueChatThreadRender.mock.calls.at(-1)?.[0];
|
||||
await waitForAssertion(() => expect(props()?.interruption).toMatchObject({ intentId: "retry" }));
|
||||
saved = { ...saved, interruption: { ...interruption, intentId: "new-message-receipt" } };
|
||||
await act(async () => { await props().onRefreshLatestComments(); });
|
||||
await waitForAssertion(() => expect(props()?.interruption).toMatchObject({
|
||||
intentId: "new-message-receipt", retryWait: interruption.retryWait,
|
||||
}));
|
||||
});
|
||||
|
||||
it("opens chats compactly and pages backward without expanding all history", async () => {
|
||||
queryClient.setQueryData(["issues", "history-expanded", "PAP-1"], false);
|
||||
const chat = createIssue({ conversationAgentId: "agent-1", conversationUserId: "user-1" });
|
||||
|
||||
@@ -6758,6 +6758,8 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
|
||||
const loadOlderComments = useCallback(() => paginatedConversation
|
||||
? earlierChatComments.load() : fetchOlderComments(), [paginatedConversation, earlierChatComments.load, fetchOlderComments]);
|
||||
const refetchLatestComments = useCallback(async () => {
|
||||
// Submission reconciliation needs the current admission receipt as well as comments.
|
||||
void refetchIssueDetail();
|
||||
if (paginatedConversation) {
|
||||
await recentComments.refetch();
|
||||
return;
|
||||
@@ -6791,7 +6793,7 @@ export function TaskDetailSurface({ conversation, tasksTab }: { tasksTab?: TaskS
|
||||
}
|
||||
window.requestAnimationFrame(() => resolve());
|
||||
});
|
||||
}, [issueId, queryClient, refetchComments, paginatedConversation, recentComments.refetch]);
|
||||
}, [issueId, queryClient, refetchIssueDetail, refetchComments, paginatedConversation, recentComments.refetch]);
|
||||
useEffect(() => {
|
||||
if (
|
||||
!shouldPrefetchOlderComments &&
|
||||
|
||||
Reference in new issue
Block a user