mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 01:57:31 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task board shows which recovery actions are active. > - A native run can stop while a person must repair its workspace. > - The board previously called that state “Recovery in progress.” > - The label implied that work would continue without operator action. > - This pull request derives the label from the recovery owner and live continuation. > - Operators can distinguish scheduled recovery from a repair that needs attention. ## Linked Issues or Issue Description **What happened?** A blocked task showed “Recovery in progress” after native finalization stopped and no automatic continuation remained. **Expected behavior** Show “Recovery needed” for an idle board repair or when no live recovery path exists. Show “Recovery in progress” while the recorded continuation can run, including an explicitly admitted export whose exact callback is executing even if the old recovery action remains board-owned. **Steps to reproduce** 1. Complete a native run whose workspace export cannot be recovered automatically. 2. Inspect the task recovery action and badge. 3. Compare the board-owned action with the old “Recovery in progress” label. Related work: #14314 serializes native workspace finalization and fences stale recovery outcomes. This change reports the recovery action that currently owns the task. ## What Changed - Show “Recovery needed” for board-owned active-run recovery unless the exact native export callback is positively verified as executing. - Require the native continuation run and a live or future continuation before showing progress. - Project native activity from the exact company, source issue, and run. Include active workspace export while the original heartbeat remains failed. - Require an executing callback before using a running export row as evidence. Preserve activity for long exports and clear it when the callback joins. - Give native resume its own card explanation. Preserve ordinary watchdog observation behavior. - Remove the redundant ownership sentence from all six recovery-card explanations that used it. - Document the labels and add regression cases for stopped, scheduled, and active recovery. ## Verification - Copy-only follow-up (`c68aef04c`): all 167 focused recovery UI tests and token gates pass. No UI occurrence of the removed sentence remains. `pnpm -r typecheck`, `pnpm build`, and current-head CI pass (54 successful checks, two optional Storybook checks skipped). Greptile is 5/5 with no open review threads. The duplicate local `pnpm test:run` was stopped after the full CI suite passed; it did not complete locally. - Original regressions: seven failures before the change, then 52 focused cases pass. - Review regressions: seven UI failures and nine database failures before the follow-up. All 75 database/API recovery tests, 167 UI tests, and 18 workspace lifecycle/finalizer tests pass. A further three RED cases cover explicit board retry activity; one RED case rejects orphaned running export rows after controller loss. Wrong company, issue, run, service, phase, and completed-operation cases remain inactive. - Final recursive typecheck, production build, token gates, and complete local suite coverage pass. Embedded PostgreSQL startup/socket failures passed in isolated retries with the canonical test environment; no expected behavior was weakened. The recovery regression added during the earlier full run passed in its final complete 75-case file. - Before the copy-only follow-up, all 56 CI checks passed on `c2f84cd89c905cda85c53aaf5bb83b7250900fe6`; Greptile is 5/5 with no unresolved review threads. The final native-activity staging repeat passed on integrated source `2bedd0f23bf4698b1f8b818f6796900647030427`. - Verified on a separate staging instance: a real failed Daytona workspace export retains its board-owned repair action and displays “Recovery needed” in the task list. The repair card remains actionable without starting another provider turn. - Real staged export-only repair: the actual task list showed “Recovery in progress” while the original run had a positively identified running export operation, then Done after exact copyback of all 20,000 nonce-bound files. The accepted result and full provider session/turn/terminal envelopes remained unchanged. All 17 independent final checks passed; the browser downloaded the exact 19-byte result. The separate fixture was cleaned up with independent provider-absence verification. A control transport process restarted during repair; it did not submit another provider turn. - Final deployed-source repeat on `d884e1ab046cc76004e35e6091e9e6e2c918c9eb`: explicit per-turn ephemeral Daytona allocation, actual browser export repair, and a saved full activity projection referencing the exact executing export operation with no scheduled retry. The task list showed recovery in progress, then Done; all 21 final checks passed, including 20,000 exact host files, unchanged provider provenance, and provider deletion only after committed copyback. The downloaded 19-byte result matched independently. This integrates #14334; no source change was required here. ## Risks - The label depends on the persisted recovery action. A separate runtime defect can still stop work; this change makes that condition visible. - Future recovery kinds must supply a valid continuation path before they can display progress. ## Model Used OpenAI Codex, based on GPT-6, with code execution, browser testing, and subagent tool use. The runtime does not expose an exact serving model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
244 lines
11 KiB
TypeScript
244 lines
11 KiB
TypeScript
import type { IssueRecoveryAction, IssueScheduledRetry } from "@paperclipai/shared";
|
|
import { formatMonitorOffset } from "./issue-monitor";
|
|
|
|
/**
|
|
* Which bounded retry budget the server is currently spending on a recovery action.
|
|
*
|
|
* The owner-sticky contract keeps one recovery-action row per source issue and rewrites
|
|
* its `wakePolicy` as the action moves between budgets: the original owner's repair
|
|
* attempts, then a manager's path-repair attempts, then the board. The policy type is
|
|
* therefore the canonical lane signal — every surface reads the same stored value rather
|
|
* than re-deriving liveness on its own.
|
|
*/
|
|
export type RecoveryRetryLane = "source_owner" | "recovery_owner" | "native_run" | "board";
|
|
|
|
export interface RecoveryRetryLineage {
|
|
lane: RecoveryRetryLane;
|
|
/** Attempts already spent in the current lane. 0 before the first attempt is scheduled. */
|
|
attempt: number;
|
|
maxAttempts: number | null;
|
|
attemptsRemaining: number | null;
|
|
/** The lane has no attempts left (always true once the board owns the action). */
|
|
exhausted: boolean;
|
|
/** When the stored next attempt is due, as an ISO string. Null once a lane is exhausted. */
|
|
nextRetryAt: string | null;
|
|
/**
|
|
* The stored attempt came due and nothing picked it up. The row still reports the time it
|
|
* was due — an expired attempt is a missed promise worth showing, not a fact to hide — but
|
|
* it no longer counts as a path anyone is waiting on.
|
|
*/
|
|
retryExpired: boolean;
|
|
/** The run the server parked for the next attempt, when it recorded one. */
|
|
scheduledRunId: string | null;
|
|
/**
|
|
* The parked run, once a caller has confirmed it is genuinely in flight. A bare
|
|
* `scheduledRunId` is only a record that a run was once created, so it is never enough on
|
|
* its own; this field is set only from a liveness signal the caller passed in.
|
|
*/
|
|
liveRunId: string | null;
|
|
/** The agent the stored attempt will wake. */
|
|
retryAgentId: string | null;
|
|
/** The server recorded that this lane does not move the source deliverable. */
|
|
preservesSourceAssignee: boolean;
|
|
/** Attempts the original owner spent before the manager lane opened. */
|
|
sourceAttempt: number | null;
|
|
sourceMaxAttempts: number | null;
|
|
/**
|
|
* Something will still move this lane forward without anyone intervening: either a stored
|
|
* attempt that is still ahead of us, or a parked run confirmed to be in flight. This — not
|
|
* the mere existence of an open recovery action, and not a due time that already passed —
|
|
* is what keeps a surface quiet.
|
|
*/
|
|
hasDurablePath: boolean;
|
|
}
|
|
|
|
const SOURCE_LANE_POLICY = "bounded_owner_disposition_repair";
|
|
const RECOVERY_LANE_POLICY = "bounded_recovery_owner";
|
|
const BOARD_LANE_POLICY = "board_escalation";
|
|
const NATIVE_RESUME_POLICY = "resume_native_run";
|
|
|
|
/** Scheduled-run states that mean the parked attempt is actually in flight right now. */
|
|
const LIVE_SCHEDULED_RUN_STATUSES = new Set(["queued", "running"]);
|
|
|
|
/**
|
|
* How far past its due time a stored attempt may sit before it counts as missed.
|
|
*
|
|
* `formatMonitorOffset` collapses anything inside a minute-rounded zero to "now", so reusing
|
|
* that same band keeps the label and the verdict from ever contradicting each other: while a
|
|
* surface still reads "next try now", the lane is still treated as durable. Past that, the
|
|
* scheduler had its chance and did not take it.
|
|
*/
|
|
const RETRY_EXPIRY_GRACE_MS = 30_000;
|
|
|
|
export type RecoveryLineageInput = Pick<IssueRecoveryAction, "wakePolicy"> &
|
|
Partial<Pick<IssueRecoveryAction, "evidence" | "attemptCount" | "maxAttempts" | "timeoutAt" | "nativeRunActivity">>;
|
|
|
|
/**
|
|
* The source issue's scheduled-retry record, which is what lets a surface tell a run that is
|
|
* really executing from a run id the server wrote down once and never started.
|
|
*/
|
|
export type RecoveryScheduledRetryInput = Partial<Pick<IssueScheduledRetry, "runId" | "status">>;
|
|
|
|
/**
|
|
* Everything outside the stored action that decides whether a lane is still live. Callers
|
|
* that cannot observe the scheduled run leave `scheduledRetry` unset and get the conservative
|
|
* answer, so a surface can under-promise but never over-promise.
|
|
*/
|
|
export interface RecoveryLivenessContext {
|
|
/** Epoch ms to treat as now. Defaults to the wall clock. */
|
|
now?: number;
|
|
scheduledRetry?: RecoveryScheduledRetryInput | null;
|
|
}
|
|
|
|
function asRecord(value: unknown): Record<string, unknown> | null {
|
|
return value && typeof value === "object" && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: null;
|
|
}
|
|
|
|
function asNonEmptyString(value: unknown): string | null {
|
|
if (typeof value !== "string") return null;
|
|
const trimmed = value.trim();
|
|
return trimmed.length > 0 ? trimmed : null;
|
|
}
|
|
|
|
function asCount(value: unknown): number | null {
|
|
if (typeof value !== "number" || !Number.isFinite(value)) return null;
|
|
const floored = Math.floor(value);
|
|
return floored >= 0 ? floored : null;
|
|
}
|
|
|
|
function asIsoDate(value: unknown): string | null {
|
|
if (!value) return null;
|
|
if (!(value instanceof Date) && typeof value !== "string") return null;
|
|
const date = value instanceof Date ? value : new Date(value);
|
|
return Number.isNaN(date.getTime()) ? null : date.toISOString();
|
|
}
|
|
|
|
/**
|
|
* Confirm the parked run is in flight. When the policy named a specific run, only that run
|
|
* counts — a different live run on the issue is someone else's work and says nothing about
|
|
* whether this retry will happen.
|
|
*/
|
|
function resolveLiveRunId(
|
|
scheduledRunId: string | null,
|
|
scheduledRetry: RecoveryScheduledRetryInput | null | undefined,
|
|
): string | null {
|
|
if (!scheduledRetry) return null;
|
|
const runId = asNonEmptyString(scheduledRetry.runId);
|
|
const status = asNonEmptyString(scheduledRetry.status);
|
|
if (!runId || !status || !LIVE_SCHEDULED_RUN_STATUSES.has(status)) return null;
|
|
if (scheduledRunId !== null && scheduledRunId !== runId) return null;
|
|
return runId;
|
|
}
|
|
|
|
/**
|
|
* Read the bounded retry lineage the server stored on a recovery action, or null when the
|
|
* action does not carry one (older kinds, or a board escalation that is not part of the
|
|
* owner-sticky contract).
|
|
*/
|
|
export function readRecoveryRetryLineage(
|
|
action: RecoveryLineageInput,
|
|
context?: RecoveryLivenessContext,
|
|
): RecoveryRetryLineage | null {
|
|
const policy = asRecord(action.wakePolicy);
|
|
if (!policy) return null;
|
|
const type = asNonEmptyString(policy.type);
|
|
const nativeResume = policy.kind === NATIVE_RESUME_POLICY;
|
|
const preservesSourceAssignee = policy.preservesSourceAssignee === true;
|
|
const evidence = asRecord(action.evidence) ?? {};
|
|
const evidenceSourceAttempt = asCount(evidence.sourceAttemptCount);
|
|
const evidenceSourceMaxAttempts = asCount(evidence.sourceMaxAttempts);
|
|
|
|
let lane: RecoveryRetryLane;
|
|
if (nativeResume) lane = "native_run";
|
|
else if (type === SOURCE_LANE_POLICY) lane = "source_owner";
|
|
else if (type === RECOVERY_LANE_POLICY) lane = "recovery_owner";
|
|
else if (
|
|
type === BOARD_LANE_POLICY &&
|
|
(preservesSourceAssignee || evidenceSourceMaxAttempts !== null)
|
|
) {
|
|
lane = "board";
|
|
} else return null;
|
|
|
|
const attempt = asCount(policy.attempt) ?? asCount(action.attemptCount) ?? 0;
|
|
const maxAttempts = asCount(policy.maxAttempts) ?? asCount(action.maxAttempts);
|
|
const scheduledRunId = asNonEmptyString(nativeResume ? policy.runId : policy.scheduledRunId);
|
|
// Native recovery resumes an existing run; its scheduling fields
|
|
// differ from an agent wake. A retry without that run identity is not a path.
|
|
if (nativeResume && scheduledRunId === null) return null;
|
|
const storedRetryAt = nativeResume
|
|
? asIsoDate(policy.notBefore)
|
|
: asIsoDate(policy.retryAt) ?? asIsoDate(action.timeoutAt);
|
|
const exhausted = lane === "board" || (maxAttempts !== null && attempt >= maxAttempts);
|
|
const nextRetryAt = exhausted ? null : storedRetryAt;
|
|
const liveRunId = nativeResume
|
|
? resolveLiveRunId(scheduledRunId, action.nativeRunActivity)
|
|
: resolveLiveRunId(scheduledRunId, context?.scheduledRetry);
|
|
// A run that is already executing is the reason its due time is behind us, so a confirmed
|
|
// live run is never "missed" — it is the attempt happening.
|
|
const retryExpired =
|
|
!exhausted &&
|
|
liveRunId === null &&
|
|
nextRetryAt !== null &&
|
|
Date.parse(nextRetryAt) + RETRY_EXPIRY_GRACE_MS < (context?.now ?? Date.now());
|
|
|
|
return {
|
|
lane,
|
|
attempt,
|
|
maxAttempts,
|
|
attemptsRemaining: maxAttempts === null ? null : Math.max(0, maxAttempts - attempt),
|
|
exhausted,
|
|
nextRetryAt,
|
|
retryExpired,
|
|
scheduledRunId,
|
|
liveRunId,
|
|
retryAgentId: asNonEmptyString(policy.retryAgentId) ?? asNonEmptyString(policy.ownerAgentId),
|
|
preservesSourceAssignee: preservesSourceAssignee || lane === "source_owner" || nativeResume,
|
|
sourceAttempt: nativeResume ? null : lane === "source_owner" ? attempt : evidenceSourceAttempt,
|
|
sourceMaxAttempts: nativeResume ? null : lane === "source_owner" ? maxAttempts : evidenceSourceMaxAttempts,
|
|
hasDurablePath:
|
|
(nativeResume && liveRunId !== null) ||
|
|
(!exhausted && (liveRunId !== null || (nextRetryAt !== null && !retryExpired))),
|
|
};
|
|
}
|
|
|
|
/** "Attempt 2 of 5", or null when the server did not record a bounded budget. */
|
|
export function formatRecoveryAttemptLabel(lineage: RecoveryRetryLineage): string | null {
|
|
if (lineage.maxAttempts === null) return null;
|
|
return `Attempt ${Math.min(lineage.attempt, lineage.maxAttempts)} of ${lineage.maxAttempts}`;
|
|
}
|
|
|
|
/** "in 3m" / "now" / "3m ago" for the stored next attempt, or null when none is stored. */
|
|
export function formatRecoveryRetryOffset(lineage: RecoveryRetryLineage): string | null {
|
|
if (!lineage.nextRetryAt) return null;
|
|
try {
|
|
return formatMonitorOffset(lineage.nextRetryAt);
|
|
} catch {
|
|
return null;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* One compact sentence fragment shared by the card and the blocker chips so every surface
|
|
* reports the same attempt count and due time.
|
|
*/
|
|
export function formatRecoveryLineageSummary(lineage: RecoveryRetryLineage): string | null {
|
|
const parts: string[] = [];
|
|
const attempt = formatRecoveryAttemptLabel(lineage);
|
|
if (attempt) parts.push(attempt);
|
|
const offset = formatRecoveryRetryOffset(lineage);
|
|
if (lineage.liveRunId) {
|
|
parts.push("attempt running now");
|
|
} else if (lineage.retryExpired) {
|
|
// Never "next try 5m ago": a due time in the past is a missed attempt, and phrasing it as
|
|
// an upcoming one is exactly the false healthy state this helper exists to prevent.
|
|
parts.push(offset ? `retry missed ${offset}` : "retry missed");
|
|
} else if (offset) {
|
|
parts.push(offset === "now" ? "next try now" : `next try ${offset}`);
|
|
} else if (lineage.exhausted) {
|
|
parts.push("retries used up");
|
|
}
|
|
return parts.length > 0 ? parts.join(" · ") : null;
|
|
}
|