From ccc77c5b5ef95032b8407a4ea663492ba33ad80f Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Wed, 14 Oct 2026 19:52:23 -0700 Subject: [PATCH] fix(server): record legacy controller lease-loss timings (#16359) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat runner executes legacy agent runs, and a controller lease watchdog keeps durable ownership of each run > - The watchdog renews the lease every 10 seconds and stops the run when a renewal does not finish within the 60-second lease > - A staging run failed in setup with `LegacyControllerLeaseLostError` and the `renewal_deadline` cause, after about eight minutes of preparation and with no server restart > - The setup failure path did not record the cause, and no evidence showed why the renewal did not finish: a stalled database query or a blocked event loop > - This pull request records bounded watchdog timings with each lease loss, on the setup failure path and on the cancellation path > - The benefit is that the next event shows which of the two causes stops renewal, which is the missing fact to fix the root cause ## Linked Issues or Issue Description Refs #16084 (added the closed lease-loss cause for cancelled runs) Refs #16355 (retries a run that a restart stopped before dispatch; a different path) **What happened?** A legacy `paperclip_runner` run on staging failed with `setup_failed` and the message "Legacy controller lease lost". The stack shows the watchdog deadline timer (`renewal_deadline`). The run was in `executionStage: "preparing"` for about 467 seconds. The server started about 11 minutes before the run, so this was not a restart. The run record and the Sentry event do not show the lease-loss cause, because the setup failure path does not read it. Nothing shows if a renewal query was in flight at the deadline, or if the deadline callback ran late. **Expected behavior** When the watchdog loses the lease, the run records the closed cause and the watchdog timings. This applies when the loss fails setup and when it stops the adapter. The failure report contains only the closed cause and bounded numbers. **Steps to reproduce** 1. Start a legacy run with a controller lease. 2. During setup, make a lease renewal query hang for more than 60 seconds, or block the event loop past the deadline. 3. Let the watchdog abort the run. 4. Read `resultJson` on the failed run. Before this change, it has no lease-loss evidence. ## What Changed - `legacy-controller-lease.ts`: The watchdog tracks the last successful renewal, the renewals in flight, the scheduled deadline, and the renewal count. `LegacyControllerLeaseLostError` now carries `leaseLossTiming`: `sinceRenewalMs`, `renewalPendingMs` (or `null`), `deadlineLateMs` (only for `renewal_deadline`), and `renewalCount`. Ownership behavior does not change. - `run-cancellation.ts`: Add the `LegacyControllerLeaseLossTiming` type and `readControllerLeaseLoss()`. The reader accepts only a closed cause and non-negative finite numbers, and clamps each value to 24 hours. - `heartbeat/run-completion.ts`: Persist `resultJson.controllerLeaseLoss` with the cause and timings. This applies to a lease loss that stops the adapter and to a lease loss that fails setup (the thrown abort reason). - `run-failure-diagnostics.ts`: Report `controllerLeaseLossCause` and the four bounded timing fields from `controllerLeaseLoss`. A setup failure now gets the cause label too. - `doc/run-log-events.md`: Document the new record and how to read the timings. - Tests: watchdog timings for a stalled query, a blocked event loop, and a failing query; completion records the evidence for an adapter stop and for a setup failure; the failure report clamps values, drops unknown keys, and drops a record without a closed cause. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/run-completion.test.ts src/services/__tests__/run-failure-report.test.ts src/services/run-cancellation.test.ts` - I could not run these tests locally. The sandbox has a 4 GB memory limit, and `pnpm install` was stopped by the out-of-memory killer during lockfile resolution. CI is the test run for this change. - After deployment, a new lease loss shows `controllerLeaseRenewalPendingMs` and `controllerLeaseDeadlineLateMs` in the Sentry execution context. A pending time near 50 seconds or more points at the database. A large deadline lateness points at the event loop. ## Risks - Low risk. This change adds diagnostics. It does not change when the watchdog stops a run, and it does not prevent the lease loss. - The new `resultJson.controllerLeaseLoss` field is server-written. Readers ignore unknown values. Old rows do not have it. - The failure report gets five closed or numeric fields. It gets no reason text, IDs, or user content. - The event-loop test blocks the loop for about 80 ms to make the deadline late. It uses real timers, so a very slow CI host makes the lateness larger, not smaller. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Sentry and GitHub connectors). The exact context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (not possible in this sandbox; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 --- doc/run-log-events.md | 8 ++++ .../__tests__/run-failure-report.test.ts | 21 +++++++++ .../services/heartbeat/run-completion.test.ts | 20 +++++++++ .../src/services/heartbeat/run-completion.ts | 15 ++++++- .../services/legacy-controller-lease.test.ts | 40 ++++++++++++++++- .../src/services/legacy-controller-lease.ts | 43 +++++++++++++++---- server/src/services/run-cancellation.ts | 30 +++++++++++++ .../src/services/run-failure-diagnostics.ts | 10 ++++- 8 files changed, 175 insertions(+), 12 deletions(-) diff --git a/doc/run-log-events.md b/doc/run-log-events.md index 10912e00640..d71bbe9283e 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -334,6 +334,14 @@ The local lifecycle event includes this evidence. Started cancellations without an expected stop are also reported to Sentry; its cancellation diagnostics contain only source, expectedness, initiator type, and the closed lease-loss cause, never initiator IDs or reason text. +A lease loss also records `resultJson.controllerLeaseLoss`, including a loss +during setup that fails the run before adapter dispatch. It holds the closed +cause and the watchdog timings in milliseconds: `sinceRenewalMs` (since the +last successful renewal), `renewalPendingMs` (age of a renewal query still in +flight, or `null`), `deadlineLateMs` (how late the deadline callback ran, for +`renewal_deadline` only), and `renewalCount`. A long `renewalPendingMs` points +at a stalled database query; a long `deadlineLateMs` points at a blocked event +loop. Failure reports carry these bounded numbers and the cause label only. Historical ambiguous cancellations stay `unknown` and do not authorize replay. Provider tool-definition validation failures use diff --git a/server/src/services/__tests__/run-failure-report.test.ts b/server/src/services/__tests__/run-failure-report.test.ts index 038f2b6fd32..6feb238bc71 100644 --- a/server/src/services/__tests__/run-failure-report.test.ts +++ b/server/src/services/__tests__/run-failure-report.test.ts @@ -939,6 +939,27 @@ describeEmbeddedPostgres("reportRunFailure", () => { cancellationSource: "controller_lease", cancellationExpected: false, controllerLeaseLossCause: "renewal_deadline" } } }); expect(JSON.stringify(mockCaptureRunFailure.mock.calls)).not.toMatch(/Legacy controller lease lost/); }); + it("reports bounded lease-loss timings for a setup failure", async () => { + await seedCompanyAndAgent(); + await reportRunFailure(db, buildRun({ status: "failed", errorCode: "setup_failed", startedAt: new Date(0), + finishedAt: new Date(1000), resultJson: { controllerLeaseLoss: { cause: "renewal_deadline", + sinceRenewalMs: 61_000.4, renewalPendingMs: 59_000, deadlineLateMs: 1e12, renewalCount: -1, note: "private-note" } } })); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + const execution = mockCaptureRunFailure.mock.calls[0][0].diagnostics.execution; + expect(execution).toMatchObject({ controllerLeaseLossCause: "renewal_deadline", controllerLeaseSinceRenewalMs: 61_000, + controllerLeaseRenewalPendingMs: 59_000, controllerLeaseDeadlineLateMs: 86_400_000 }); + expect(execution).not.toHaveProperty("controllerLeaseRenewalCount"); + expect(JSON.stringify(mockCaptureRunFailure.mock.calls)).not.toMatch(/private-note/); + }); + it("drops lease-loss timings without a closed cause", async () => { + await seedCompanyAndAgent(); + await reportRunFailure(db, buildRun({ status: "failed", errorCode: "setup_failed", startedAt: new Date(0), + finishedAt: new Date(1000), resultJson: { controllerLeaseLoss: { cause: "forged", sinceRenewalMs: 1 } } })); + expect(mockCaptureRunFailure).toHaveBeenCalledOnce(); + const execution = mockCaptureRunFailure.mock.calls[0][0].diagnostics.execution; + expect(execution).not.toHaveProperty("controllerLeaseLossCause"); + expect(execution).not.toHaveProperty("controllerLeaseSinceRenewalMs"); + }); it("does not report an operator's Stop as a failure", async () => { await seedCompanyAndAgent(); await reportRunFailure(db, buildRun({ status: "cancelled", startedAt: new Date(0), resultJson: { diff --git a/server/src/services/heartbeat/run-completion.test.ts b/server/src/services/heartbeat/run-completion.test.ts index beb1c545801..bb1e74005c3 100644 --- a/server/src/services/heartbeat/run-completion.test.ts +++ b/server/src/services/heartbeat/run-completion.test.ts @@ -175,6 +175,26 @@ describe.skipIf(!support.supported)("heartbeat run completion boundary", () => { cancellation: { source: "controller_lease", expected: false, reason: "Legacy controller lease lost (renewal_rejected)" } } }); }); + it("records the watchdog timings with a lease loss that stops the adapter", async () => { + const f = await fixture(); + Object.assign(f.input.adapterResult, { exitCode: null, errorCode: "cancelled", errorMessage: "Grok execution was cancelled" }); + f.controller.abort(new LegacyControllerLeaseLostError("renewal_deadline", + { sinceRenewalMs: 61_000, renewalPendingMs: 58_000, deadlineLateMs: 2, renewalCount: 41 })); + await f.completion.completeRun(f.input); + expect((await read(f.run.id)).resultJson).toMatchObject({ controllerLeaseLoss: { + cause: "renewal_deadline", sinceRenewalMs: 61_000, renewalPendingMs: 58_000, deadlineLateMs: 2, renewalCount: 41 } }); + }); + + it("records the lease-loss cause and timings when setup fails on the abort reason", async () => { + const f = await fixture(); + await f.completion.failRunSetup(f.setup(new LegacyControllerLeaseLostError("renewal_deadline", + { sinceRenewalMs: 60_004, renewalPendingMs: null, deadlineLateMs: 55_000, renewalCount: 3 }))); + expect(await read(f.run.id)).toMatchObject({ status: "failed", errorCode: "setup_failed", resultJson: { + executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, + controllerLeaseLoss: { cause: "renewal_deadline", sinceRenewalMs: 60_004, renewalPendingMs: null, + deadlineLateMs: 55_000, renewalCount: 3 } } }); + }); + it("keeps a stop requested before the lease loss when the adapter throws", async () => { const f = await fixture(); const requested = requestedRunCancellation({ cancelledByActorType: "user", cancelledByUserId: "operator-1" }, "Stopped from the board"); diff --git a/server/src/services/heartbeat/run-completion.ts b/server/src/services/heartbeat/run-completion.ts index 970acf1881f..83767f3e780 100644 --- a/server/src/services/heartbeat/run-completion.ts +++ b/server/src/services/heartbeat/run-completion.ts @@ -317,11 +317,22 @@ function readNonEmptyString(value: unknown): string | null { return typeof value === "string" && value.trim().length > 0 ? value : null; } +// Keep the watchdog's closed cause and timings so the run and its failure +// report show whether renewal stalled in the database or the event loop. +function controllerLeaseLossEvidence(reason: unknown) { + return reason instanceof LegacyControllerLeaseLostError + ? { controllerLeaseLoss: { cause: reason.leaseLossCause, ...reason.leaseLossTiming } } + : {}; +} + // Attribute a stop the lease watchdog caused, whether the adapter returned // or threw. A requested stop recorded earlier still wins in cancellationResultJson. function controllerLeaseLossResultJson(signal: AbortSignal) { return signal.reason instanceof LegacyControllerLeaseLostError - ? { cancellation: controllerLeaseLossCancellation(signal.reason.leaseLossCause) } + ? { + cancellation: controllerLeaseLossCancellation(signal.reason.leaseLossCause), + ...controllerLeaseLossEvidence(signal.reason), + } : {}; } @@ -1330,6 +1341,8 @@ export function createHeartbeatRunCompletion(db: Db, dependencies: HeartbeatRunC const setupFailureResultJson = { ...setupFailureDetails, ...(connectionFailure ? { connectionFailure } : {}), + // A lease lost during setup surfaces as the thrown abort reason. + ...controllerLeaseLossEvidence(outerErr), executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, }; const setupFailureWrite = await setRunStatusIfRunning(runId, "failed", { diff --git a/server/src/services/legacy-controller-lease.test.ts b/server/src/services/legacy-controller-lease.test.ts index 2280f70dd0e..354d88923f0 100644 --- a/server/src/services/legacy-controller-lease.test.ts +++ b/server/src/services/legacy-controller-lease.test.ts @@ -4,7 +4,7 @@ import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; import { heartbeatService } from "./heartbeat.js"; -import { hasLiveLegacyController, LEGACY_CONTROLLER_RENEW_MS, legacyControllerBootId, legacyControllerClaim, LegacyControllerLeaseLostError, +import { hasLiveLegacyController, LEGACY_CONTROLLER_LEASE_MS, LEGACY_CONTROLLER_RENEW_MS, legacyControllerBootId, legacyControllerClaim, LegacyControllerLeaseLostError, renewLegacyControllerLease, revokeExpiredLegacyController, revokeExpiredLegacyControllerWithState, watchLegacyControllerLease } from "./legacy-controller-lease.js"; @@ -147,6 +147,44 @@ describe("legacy controller lease loss attribution", () => { await vi.advanceTimersByTimeAsync(LEGACY_CONTROLLER_RENEW_MS + 1); expect(controller.signal.reason).toBeInstanceOf(LegacyControllerLeaseLostError); expect(controller.signal.reason.leaseLossCause).toBe("renewal_error"); + expect(controller.signal.reason.leaseLossTiming).toMatchObject({ renewalPendingMs: null, deadlineLateMs: null, renewalCount: 0 }); } finally { watch.stop(); vi.useRealTimers(); } }); + + type WatchArgs = Parameters; + const watchedRun = (leaseMs: number) => ({ id: randomUUID(), companyId: randomUUID(), runtimeMode: "legacy", + controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: new Date(Date.now() + leaseMs) }) as WatchArgs[1]; + + it("records a renewal still in flight when a stalled query reaches the deadline", async () => { + let calls = 0; + const stallingDb = { update: () => ({ set: () => ({ where: () => ({ returning: () => { + calls += 1; + return calls <= 2 ? Promise.resolve([{ id: "run" }]) : new Promise(() => {}); + } }) }) }) } as unknown as WatchArgs[0]; + vi.useFakeTimers(); + const controller = new AbortController(); + const watch = watchLegacyControllerLease(stallingDb, watchedRun(LEGACY_CONTROLLER_LEASE_MS), controller); + try { + await vi.advanceTimersByTimeAsync(2 * LEGACY_CONTROLLER_RENEW_MS); + expect(controller.signal.aborted).toBe(false); + await vi.advanceTimersByTimeAsync(LEGACY_CONTROLLER_LEASE_MS + 1); + expect(controller.signal.reason.leaseLossCause).toBe("renewal_deadline"); + expect(controller.signal.reason.leaseLossTiming).toEqual({ sinceRenewalMs: LEGACY_CONTROLLER_LEASE_MS, + renewalPendingMs: LEGACY_CONTROLLER_LEASE_MS - LEGACY_CONTROLLER_RENEW_MS, deadlineLateMs: 0, renewalCount: 2 }); + } finally { watch.stop(); vi.useRealTimers(); } + }); + + it("records how late the deadline ran when the event loop was blocked", async () => { + const idleDb = { update: () => ({ set: () => ({ where: () => ({ returning: () => new Promise(() => {}) }) }) }) } as unknown as WatchArgs[0]; + const controller = new AbortController(); + const watch = watchLegacyControllerLease(idleDb, watchedRun(10), controller); + try { + const blockedUntil = Date.now() + 80; + while (Date.now() < blockedUntil) { /* hold the event loop past the deadline */ } + await new Promise((resolve) => setTimeout(resolve, 5)); + expect(controller.signal.reason.leaseLossCause).toBe("renewal_deadline"); + expect(controller.signal.reason.leaseLossTiming.deadlineLateMs).toBeGreaterThanOrEqual(60); + expect(controller.signal.reason.leaseLossTiming.renewalPendingMs).toBeNull(); + } finally { watch.stop(); } + }); }); diff --git a/server/src/services/legacy-controller-lease.ts b/server/src/services/legacy-controller-lease.ts index e13e2ee2308..10ee785e4ea 100644 --- a/server/src/services/legacy-controller-lease.ts +++ b/server/src/services/legacy-controller-lease.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; import { and, eq, gt, lte, sql } from "drizzle-orm"; import { heartbeatRuns, type Db } from "@paperclipai/db"; -import type { LegacyControllerLeaseLossCause } from "./run-cancellation.js"; +import type { LegacyControllerLeaseLossCause, LegacyControllerLeaseLossTiming } from "./run-cancellation.js"; // A boot UUID has meaning across containers; a numeric PID does not. export const legacyControllerBootId = randomUUID(); @@ -12,7 +12,10 @@ type Run = typeof heartbeatRuns.$inferSelect; /** Abort reason for a lost lease, so completion can attribute the stop. */ export class LegacyControllerLeaseLostError extends Error { - constructor(readonly leaseLossCause: LegacyControllerLeaseLossCause) { + constructor( + readonly leaseLossCause: LegacyControllerLeaseLossCause, + readonly leaseLossTiming?: LegacyControllerLeaseLossTiming, + ) { super("Legacy controller lease lost"); this.name = "LegacyControllerLeaseLostError"; } @@ -97,17 +100,38 @@ export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortCo } let stopped = false; let pending = false; + // Timing evidence for a loss: a renewal still in flight at the deadline + // points at the database; a late deadline callback points at the event loop. + let lastRenewedAt = Date.now(); + let renewalCount = 0; + const renewalsInFlight = new Set<{ startedAt: number }>(); + let deadlineDueAt = 0; + let deadline: ReturnType | undefined; const lost = (cause: LegacyControllerLeaseLossCause) => { - if (!stopped) controller.abort(new LegacyControllerLeaseLostError(cause)); + if (stopped || controller.signal.aborted) return; + const now = Date.now(); + const oldestRenewal = Math.min(...[...renewalsInFlight].map((renewal) => renewal.startedAt)); + controller.abort(new LegacyControllerLeaseLostError(cause, { + sinceRenewalMs: now - lastRenewedAt, + renewalPendingMs: renewalsInFlight.size > 0 ? now - oldestRenewal : null, + deadlineLateMs: cause === "renewal_deadline" ? Math.max(0, now - deadlineDueAt) : null, + renewalCount, + })); }; const deadlinePassed = () => lost("renewal_deadline"); - let deadline = setTimeout(deadlinePassed, Math.max(0, - (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now())); - deadline.unref(); + const armDeadline = (delayMs: number) => { + clearTimeout(deadline); + deadlineDueAt = Date.now() + delayMs; + deadline = setTimeout(deadlinePassed, delayMs); + deadline.unref(); + }; + armDeadline(Math.max(0, (run.controllerLeaseExpiresAt?.getTime() ?? 0) - Date.now())); const assertOwned = async (stage?: "dispatching") => { if (stopped) return; controller.signal.throwIfAborted(); const startedAt = Date.now(); + const renewal = { startedAt }; + renewalsInFlight.add(renewal); let onAbort!: () => void; const aborted = new Promise((_, reject) => { onAbort = () => reject(controller.signal.reason); @@ -117,6 +141,7 @@ export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortCo try { renewed = await Promise.race([renewLegacyControllerLease(db, run, stage), aborted]); } finally { + renewalsInFlight.delete(renewal); controller.signal.removeEventListener("abort", onAbort); } if (stopped) return; @@ -126,9 +151,9 @@ export function watchLegacyControllerLease(db: Db, run: Run, controller: AbortCo } controller.signal.throwIfAborted(); if (!stopped) { - clearTimeout(deadline); - deadline = setTimeout(deadlinePassed, Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - startedAt))); - deadline.unref(); + lastRenewedAt = Date.now(); + renewalCount += 1; + armDeadline(Math.max(0, LEGACY_CONTROLLER_LEASE_MS - (Date.now() - startedAt))); } }; const timer = setInterval(() => { diff --git a/server/src/services/run-cancellation.ts b/server/src/services/run-cancellation.ts index 95a37aabfb4..89932ecd4df 100644 --- a/server/src/services/run-cancellation.ts +++ b/server/src/services/run-cancellation.ts @@ -55,6 +55,36 @@ export function readControllerLeaseLossCause(cancellation: RunCancellation): Leg return LEGACY_CONTROLLER_LEASE_LOSS_CAUSES.find(cause => cancellation.reason === controllerLeaseLossReason(cause)) ?? null; } +/** Watchdog timings at the moment the lease was lost, in milliseconds. */ +export interface LegacyControllerLeaseLossTiming { + sinceRenewalMs: number; + renewalPendingMs: number | null; + deadlineLateMs: number | null; + renewalCount: number; +} +/** Read-side record: a `null` timing is dropped, so every present timing is a number. */ +export type LegacyControllerLeaseLoss = { cause: LegacyControllerLeaseLossCause } + & Partial>; + +const MAX_LEASE_LOSS_TIMING = 86_400_000; +const leaseLossTimingValue = (value: unknown) => + typeof value === "number" && Number.isFinite(value) && value >= 0 + ? Math.min(Math.round(value), MAX_LEASE_LOSS_TIMING) : undefined; + +/** Closed cause and bounded counters only; unknown keys and values are dropped. */ +export function readControllerLeaseLoss(value: unknown): LegacyControllerLeaseLoss | null { + if (!value || typeof value !== "object") return null; + const record = value as Record; + const cause = LEGACY_CONTROLLER_LEASE_LOSS_CAUSES.find(label => label === record.cause); + if (!cause) return null; + const loss: LegacyControllerLeaseLoss = { cause }; + for (const field of ["sinceRenewalMs", "renewalPendingMs", "deadlineLateMs", "renewalCount"] as const) { + const timing = leaseLossTimingValue(record[field]); + if (timing !== undefined) loss[field] = timing; + } + return loss; +} + /** Preserve recorded stop intent when adapter completion replaces resultJson. */ export function cancellationResultJson(run: Run, status: string, patch?: Run["resultJson"], errorCode?: string | null, error?: string | null): Run["resultJson"] { if (status !== "cancelled" && status !== "interrupted") return patch ?? run.resultJson; diff --git a/server/src/services/run-failure-diagnostics.ts b/server/src/services/run-failure-diagnostics.ts index 291a9503b77..f03bf4c2043 100644 --- a/server/src/services/run-failure-diagnostics.ts +++ b/server/src/services/run-failure-diagnostics.ts @@ -1,7 +1,7 @@ import { readWorkspaceBaseRefDiagnostic } from "./workspace-base-ref-diagnostics.js"; import { getEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; import type { heartbeatRuns } from "@paperclipai/db"; -import { readControllerLeaseLossCause, readRunCancellation } from "./run-cancellation.js"; +import { readControllerLeaseLoss, readControllerLeaseLossCause, readRunCancellation } from "./run-cancellation.js"; import { readProcessLossDiagnostic } from "./process-loss-diagnostics.js"; import { WORKSPACE_RESTORE_FAILURE_CODES } from "@paperclipai/shared"; import { redactDiagnosticText } from "@paperclipai/adapter-utils/command-redaction"; @@ -290,6 +290,14 @@ export function collectRunFailureDiagnostics(run: Run, options: RunFailureReport execution.cancellationSource = "unknown"; execution.cancellationExpected = false; } + const leaseLoss = readControllerLeaseLoss(read(result, "controllerLeaseLoss")); + if (leaseLoss) { + execution.controllerLeaseLossCause = leaseLoss.cause; + if (leaseLoss.sinceRenewalMs !== undefined) execution.controllerLeaseSinceRenewalMs = leaseLoss.sinceRenewalMs; + if (leaseLoss.renewalPendingMs !== undefined) execution.controllerLeaseRenewalPendingMs = leaseLoss.renewalPendingMs; + if (leaseLoss.deadlineLateMs !== undefined) execution.controllerLeaseDeadlineLateMs = leaseLoss.deadlineLateMs; + if (leaseLoss.renewalCount !== undefined) execution.controllerLeaseRenewalCount = leaseLoss.renewalCount; + } Object.assign(execution, scalars(result, [ "mode", "stopReason", "timeoutFired", "timeoutSource", "timeoutConfigured", "effectiveTimeoutSec", "errorFamily",