Files
PaperClipAI/ui/src/components/ActivityCharts.tsx
T
DottaandPaperclip 555391fed7 fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in heartbeat runs orchestrated by the server; run
lifecycle, retry scheduling, and the dashboard's run-activity metrics
are the subsystems involved
> - A spike in "failed" tasks traced to three causes: server restarts
killing in-flight runs and mislabeling them as failures, deterministic
workspace-validation loops when a worktree's branch diverged, and
provider quota/usage-limit errors being classified as generic transient
failures (putting agents into error state and polluting metrics)
> - Killed-then-recovered runs and quota waits are not product failures,
so both the runtime behavior and the reporting needed to distinguish
them
> - This pull request drains runs gracefully on shutdown with idempotent
restart retries, self-heals workspace branch mismatches, adds a
quota-aware failure class with reset-time retry, separates recovered
restart kills from true failures on the dashboard, and documents restart
hygiene for operators
> - The benefit is fewer spurious failures, automatic recovery instead
of manual repair, and dashboard metrics that reflect real failure rates

## Linked Issues or Issue Description

No public GitHub issue exists; describing the bug inline per the
bug-report template:

**What happened?**

In-flight heartbeat runs are marked `failed` when the server restarts,
even though a retry later succeeds. Worktrees whose checked-out branch
diverges from the issue branch fail workspace validation on every
subsequent run with no recovery path. Provider quota/usage-limit
responses are treated as generic transient upstream errors, putting
agents into an error state and retrying before the quota window resets.
The dashboard counts all of these as true failures, inflating failure
metrics.

**Expected behavior**

Graceful shutdown should interrupt (not fail) running runs and chain
exactly one recovery retry. Workspace validation should repair
recoverable branch mismatches automatically. Quota errors should get
their own error class with the retry scheduled at the provider reset
time and the agent left idle. The dashboard should report recovered
restart kills separately from true failures.

**Steps to reproduce**

1. Start a heartbeat run, then restart the server (SIGTERM) while it is
in flight — the run lands as `failed` with a process-loss error code
even when its retry succeeds
2. Check out an issue whose worktree branch has diverged (e.g. after a
force-moved branch) — every subsequent run fails
`workspace_validation_failed` deterministically
3. Drive an agent into a provider usage-limit window — the run fails as
a generic transient upstream error and the agent enters an error state
instead of idling until the reset time

**Paperclip version or commit**

master (base c07e650cd)

**Deployment mode**

Self-hosted dev plane (Linux, node server + embedded Postgres)

## What Changed

- Graceful shutdown (SIGTERM hook) now marks in-flight runs
`interrupted` instead of `failed` and enqueues an idempotent
process-loss retry (pre-insert existence check on `retryOfRunId`
prevents duplicates; bursts chain exactly one retry per interrupted run)
- Run-liveness classification routes `interrupted` to `needs_followup`
rather than `failed`
- Workspace validation self-heals branch mismatch / missing-branch
states instead of failing deterministically on every run
- New `provider_quota` error class: session/usage-limit responses
schedule the retry at the provider reset time and leave the agent idle
(not errored); fixes a case where a quota-terminated run with subtype
`success` was misclassified as failed; HTTP 529 remains transient
- Dashboard run-activity query separates recovered restart kills from
true failures via a recursive CTE over `retry_of_run_id` (ancestors of a
succeeded retry count as recovered), adds a per-day failed-by-error-code
breakdown, and binds the window start as a timestamptz string
- Activity charts UI: amber "Recovered" segment with legend and per-day
error-code tooltip; success-rate chart counts recovered runs as
successes
- New ops runbook: `docs/deploy/dev-plane-restart-hygiene.md`

## Verification

- Greptile follow-up fixes on `992705edd`: `pnpm exec vitest run
server/src/__tests__/server-startup-feedback-export.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` (209
tests); `pnpm --filter @paperclipai/server typecheck`; `git diff
--check`
- Post-rebase CI fixes: `pnpm exec vitest run
server/src/__tests__/heartbeat-retry-scheduling.test.ts`; `pnpm exec
vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/workspace-runtime.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts` (227 tests);
`pnpm --filter @paperclipai/server typecheck`
- `npm test` server suites covering the changes:
`heartbeat-process-recovery`, `heartbeat-stop-metadata`,
`heartbeat-retry-scheduling` (95 tests), quota parse +
execute/retry-scheduling suites (100 tests), workspace self-heal suites
(200 tests), dashboard run-activity tests (3 tests) — all green,
typecheck exit 0
- Dashboard CTE cross-checked against a real development database: two
restart-burst days moved from 17 to 8 and 19 to 6 true failures once
recovered kills were separated, matching manual retry-chain inspection
- Screenshot verification of the real ActivityCharts component
(recovered segment + tooltip) during QA

## Risks

- Behavioral shift: runs killed by a restart no longer surface as
`failed`; anyone consuming raw run statuses will see `interrupted` (new
status value) — dashboards/queries in this repo were updated accordingly
- Retry chaining on repeated restarts is bounded (one chained retry per
interruption) but a pathological restart loop still delays work rather
than failing it; the runbook covers operator hygiene for that case
- Dashboard query adds a recursive CTE; cost is bounded by the
day-window row count and was verified against production-sized data
- No schema migrations; low migration risk

## Model Used

- Claude (Anthropic) — claude-fable-5 via Claude Code / Paperclip agent
harness, extended thinking with tool use; implementation commits also
produced with Codex CLI (GPT-5 class) agents under the same harness

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-07 20:30:09 -05:00

328 lines
13 KiB
TypeScript

import type { DashboardRunActivityDay, HeartbeatRun } from "@paperclipai/shared";
/* ---- Utilities ---- */
export function getLast14Days(): string[] {
return Array.from({ length: 14 }, (_, i) => {
const d = new Date();
d.setDate(d.getDate() - (13 - i));
return d.toISOString().slice(0, 10);
});
}
function formatDayLabel(dateStr: string): string {
const d = new Date(dateStr + "T12:00:00");
return `${d.getMonth() + 1}/${d.getDate()}`;
}
function emptyRunDay(date: string): DashboardRunActivityDay {
return { date, succeeded: 0, failed: 0, recovered: 0, other: 0, total: 0, failedByErrorCode: {} };
}
const runSegmentColors = {
succeeded: "#10b981",
recovered: "#f59e0b",
failed: "#ef4444",
other: "#737373",
} as const;
// Compact per-day tooltip that also attributes failures to their error class.
function runDayTooltip(entry: DashboardRunActivityDay): string {
const lines = [`${entry.date}: ${entry.total} run${entry.total === 1 ? "" : "s"}`];
if (entry.succeeded > 0) lines.push(` succeeded: ${entry.succeeded}`);
if (entry.recovered > 0) lines.push(` recovered: ${entry.recovered} (retry succeeded)`);
if (entry.failed > 0) {
lines.push(` failed: ${entry.failed}`);
const codes = Object.entries(entry.failedByErrorCode ?? {}).sort((a, b) => b[1] - a[1]);
for (const [code, count] of codes) lines.push(` ${code}: ${count}`);
}
if (entry.other > 0) lines.push(` other: ${entry.other}`);
return lines.join("\n");
}
/* ---- Sub-components ---- */
function DateLabels({ days }: { days: string[] }) {
return (
<div className="flex gap-(--sz-3px) mt-1.5">
{days.map((day, i) => (
<div key={day} className="flex-1 text-center">
{(i === 0 || i === 6 || i === 13) ? (
<span className="text-(length:--text-nano) text-muted-foreground tabular-nums">{formatDayLabel(day)}</span>
) : null}
</div>
))}
</div>
);
}
function ChartLegend({ items }: { items: { color: string; label: string }[] }) {
return (
<div className="flex flex-wrap gap-x-2.5 gap-y-0.5 mt-2">
{items.map(item => (
<span key={item.label} className="flex items-center gap-1 text-(length:--text-nano) text-muted-foreground">
<span className="h-1.5 w-1.5 rounded-full shrink-0" style={{ backgroundColor: item.color }} />
{item.label}
</span>
))}
</div>
);
}
export function ChartCard({ title, subtitle, children }: { title: string; subtitle?: string; children: React.ReactNode }) {
return (
<div className="border border-border rounded-lg p-4 space-y-3">
<div>
<h3 className="text-xs font-medium text-muted-foreground">{title}</h3>
{subtitle && <span className="text-(length:--text-nano) text-muted-foreground/60">{subtitle}</span>}
</div>
{children}
</div>
);
}
/* ---- Chart Components ---- */
type RunChartProps =
| { activity?: DashboardRunActivityDay[] | null; runs?: never }
| { runs?: HeartbeatRun[] | null; activity?: never };
function aggregateRuns(runs: readonly HeartbeatRun[] = []): DashboardRunActivityDay[] {
const days = getLast14Days();
const grouped = new Map<string, DashboardRunActivityDay>();
for (const day of days) grouped.set(day, emptyRunDay(day));
for (const run of runs) {
const day = new Date(run.createdAt).toISOString().slice(0, 10);
const entry = grouped.get(day);
if (!entry) continue;
if (run.status === "succeeded") {
entry.succeeded++;
} else if (run.status === "failed" || run.status === "timed_out") {
// A flat run list has no retry-chain linkage, so recovery can't be derived
// here (the company dashboard computes it server-side). Attribute the
// failure to its error class so the breakdown still renders.
entry.failed++;
const code = run.errorCode && run.errorCode.length > 0 ? run.errorCode : "unknown";
entry.failedByErrorCode[code] = (entry.failedByErrorCode[code] ?? 0) + 1;
} else {
entry.other++;
}
entry.total++;
}
return Array.from(grouped.values());
}
function resolveRunActivity(props: RunChartProps): DashboardRunActivityDay[] {
if (Array.isArray(props.activity)) return props.activity;
if (Array.isArray(props.runs)) return aggregateRuns(props.runs);
return [];
}
export function RunActivityChart(props: RunChartProps) {
const activity = resolveRunActivity(props);
const days = activity.length > 0 ? activity.map((day) => day.date) : getLast14Days();
const grouped = new Map(activity.map((day) => [day.date, day]));
const maxValue = Math.max(...activity.map(v => v.total), 1);
const hasData = activity.some(v => v.total > 0);
const hasRecovered = activity.some(v => v.recovered > 0);
if (!hasData) return <p className="text-xs text-muted-foreground">No runs yet</p>;
const legendItems = [
{ color: runSegmentColors.succeeded, label: "Succeeded" },
...(hasRecovered ? [{ color: runSegmentColors.recovered, label: "Recovered" }] : []),
{ color: runSegmentColors.failed, label: "Failed" },
{ color: runSegmentColors.other, label: "Other" },
];
return (
<div>
<div className="flex items-end gap-(--sz-3px) h-20">
{days.map(day => {
const entry = grouped.get(day) ?? emptyRunDay(day);
const total = entry.total;
const heightPct = (total / maxValue) * 100;
return (
<div key={day} className="flex-1 h-full flex flex-col justify-end" title={runDayTooltip(entry)}>
{total > 0 ? (
<div className="flex flex-col-reverse gap-px overflow-hidden" style={{ height: `${heightPct}%`, minHeight: 2 }}>
{entry.succeeded > 0 && <div style={{ flex: entry.succeeded, backgroundColor: runSegmentColors.succeeded }} />}
{entry.recovered > 0 && <div style={{ flex: entry.recovered, backgroundColor: runSegmentColors.recovered }} />}
{entry.failed > 0 && <div style={{ flex: entry.failed, backgroundColor: runSegmentColors.failed }} />}
{entry.other > 0 && <div style={{ flex: entry.other, backgroundColor: runSegmentColors.other }} />}
</div>
) : (
<div className="bg-muted/30 rounded-sm" style={{ height: 2 }} />
)}
</div>
);
})}
</div>
<DateLabels days={days} />
<ChartLegend items={legendItems} />
</div>
);
}
const priorityColors: Record<string, string> = {
critical: "var(--hex-ef4444)",
high: "var(--hex-f97316)",
medium: "var(--hex-eab308)",
low: "var(--hex-6b7280)",
};
const priorityOrder = ["critical", "high", "medium", "low"] as const;
export function PriorityChart({ issues }: { issues: { priority: string; createdAt: Date }[] }) {
const days = getLast14Days();
const grouped = new Map<string, Record<string, number>>();
for (const day of days) grouped.set(day, { critical: 0, high: 0, medium: 0, low: 0 });
for (const issue of issues) {
const day = new Date(issue.createdAt).toISOString().slice(0, 10);
const entry = grouped.get(day);
if (!entry) continue;
if (issue.priority in entry) entry[issue.priority]++;
}
const maxValue = Math.max(...Array.from(grouped.values()).map(v => Object.values(v).reduce((a, b) => a + b, 0)), 1);
const hasData = Array.from(grouped.values()).some(v => Object.values(v).reduce((a, b) => a + b, 0) > 0);
if (!hasData) return <p className="text-xs text-muted-foreground">No tasks</p>;
return (
<div>
<div className="flex items-end gap-(--sz-3px) h-20">
{days.map(day => {
const entry = grouped.get(day)!;
const total = Object.values(entry).reduce((a, b) => a + b, 0);
const heightPct = (total / maxValue) * 100;
return (
<div key={day} className="flex-1 h-full flex flex-col justify-end" title={`${day}: ${total} issues`}>
{total > 0 ? (
<div className="flex flex-col-reverse gap-px overflow-hidden" style={{ height: `${heightPct}%`, minHeight: 2 }}>
{priorityOrder.map(p => entry[p] > 0 ? (
<div key={p} style={{ flex: entry[p], backgroundColor: priorityColors[p] }} />
) : null)}
</div>
) : (
<div className="bg-muted/30 rounded-sm" style={{ height: 2 }} />
)}
</div>
);
})}
</div>
<DateLabels days={days} />
<ChartLegend items={priorityOrder.map(p => ({ color: priorityColors[p], label: p.charAt(0).toUpperCase() + p.slice(1) }))} />
</div>
);
}
// DECISION-SHEET.md B5: chart status colors re-pointed at the canonical
// --status-task-* system (DESIGN.md principle 5 — an operator learns one
// status vocabulary; badge, row, chart, and log agree). Previously an
// independent palette (todo blue, in_progress violet, etc.). `backlog`
// deliberately keeps --project-none (pre-B5, per user ruling); the
// priority series and success-rate tints below are not status hues and
// are left alone.
const statusColors: Record<string, string> = {
todo: "var(--status-task-todo)",
in_progress: "var(--status-task-in_progress)",
in_review: "var(--status-task-in_review)",
done: "var(--status-task-done)",
blocked: "var(--status-task-blocked)",
cancelled: "var(--status-task-cancelled)",
backlog: "var(--project-none)",
};
const statusLabels: Record<string, string> = {
todo: "To Do",
in_progress: "In Progress",
in_review: "In Review",
done: "Done",
blocked: "Blocked",
cancelled: "Cancelled",
backlog: "Backlog",
};
export function IssueStatusChart({ issues }: { issues: { status: string; createdAt: Date }[] }) {
const days = getLast14Days();
const allStatuses = new Set<string>();
const grouped = new Map<string, Record<string, number>>();
for (const day of days) grouped.set(day, {});
for (const issue of issues) {
const day = new Date(issue.createdAt).toISOString().slice(0, 10);
const entry = grouped.get(day);
if (!entry) continue;
entry[issue.status] = (entry[issue.status] ?? 0) + 1;
allStatuses.add(issue.status);
}
const statusOrder = ["todo", "in_progress", "in_review", "done", "blocked", "cancelled", "backlog"].filter(s => allStatuses.has(s));
const maxValue = Math.max(...Array.from(grouped.values()).map(v => Object.values(v).reduce((a, b) => a + b, 0)), 1);
const hasData = allStatuses.size > 0;
if (!hasData) return <p className="text-xs text-muted-foreground">No tasks</p>;
return (
<div>
<div className="flex items-end gap-(--sz-3px) h-20">
{days.map(day => {
const entry = grouped.get(day)!;
const total = Object.values(entry).reduce((a, b) => a + b, 0);
const heightPct = (total / maxValue) * 100;
return (
<div key={day} className="flex-1 h-full flex flex-col justify-end" title={`${day}: ${total} issues`}>
{total > 0 ? (
<div className="flex flex-col-reverse gap-px overflow-hidden" style={{ height: `${heightPct}%`, minHeight: 2 }}>
{statusOrder.map(s => (entry[s] ?? 0) > 0 ? (
<div key={s} style={{ flex: entry[s], backgroundColor: statusColors[s] ?? "var(--hex-6b7280)" }} />
) : null)}
</div>
) : (
<div className="bg-muted/30 rounded-sm" style={{ height: 2 }} />
)}
</div>
);
})}
</div>
<DateLabels days={days} />
<ChartLegend items={statusOrder.map(s => ({ color: statusColors[s] ?? "var(--hex-6b7280)", label: statusLabels[s] ?? s }))} />
</div>
);
}
export function SuccessRateChart(props: RunChartProps) {
const activity = resolveRunActivity(props);
const days = activity.length > 0 ? activity.map((day) => day.date) : getLast14Days();
const grouped = new Map(activity.map((day) => [day.date, day]));
const hasData = activity.some(v => v.total > 0);
if (!hasData) return <p className="text-xs text-muted-foreground">No runs yet</p>;
return (
<div>
<div className="flex items-end gap-(--sz-3px) h-20">
{days.map(day => {
const entry = grouped.get(day) ?? emptyRunDay(day);
// Recovered runs ultimately succeeded, so they count toward the rate
// rather than dragging it down as failures.
const effectiveSucceeded = entry.succeeded + entry.recovered;
const rate = entry.total > 0 ? effectiveSucceeded / entry.total : 0;
const color = entry.total === 0 ? undefined : rate >= 0.8 ? "var(--hex-10b981)" : rate >= 0.5 ? "var(--hex-eab308)" : "var(--hex-ef4444)";
return (
<div key={day} className="flex-1 h-full flex flex-col justify-end" title={`${day}: ${entry.total > 0 ? Math.round(rate * 100) : 0}% (${effectiveSucceeded}/${entry.total})`}>
{entry.total > 0 ? (
<div style={{ height: `${rate * 100}%`, minHeight: 2, backgroundColor: color }} />
) : (
<div className="bg-muted/30 rounded-sm" style={{ height: 2 }} />
)}
</div>
);
})}
</div>
<DateLabels days={days} />
</div>
);
}