Files
PaperClipAI/server/src/services/activity.ts
T
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00

714 lines
31 KiB
TypeScript

import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { skillAccessService } from "./skill-access.js";
import { credentialAccessConnectionNameSql } from "./credential-access-visibility.js";
import { executionProjectionsForRuns } from "./execution-projection.js";
import { and, asc, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
activityLog,
agents,
documentRevisions,
documents,
environmentLeases,
environments,
heartbeatRunEvents,
heartbeatRuns,
workspaceRestoreFailureCodesSql,
issueComments,
issueDocuments,
issues,
issueWorkProducts,
workspaceOperations,
} from "@paperclipai/db";
import { hasWorkspaceRestoreFailure, safeWorkspaceRestorePath, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, normalizeRunFailureCause, RUN_FAILURE_CAUSE_MAX_LENGTH } from "@paperclipai/shared";
import { logger } from "../middleware/logger.js";
import { visibleIssueCondition } from "./issue-visibility.js";
import { classifyRunLiveness } from "./run-liveness.js";
import { missingRunLivenessCandidatesQuery } from "./run-liveness-backfill-query.js";
export interface ActivityFilters {
companyId: string;
agentId?: string;
entityType?: string;
entityId?: string;
limit?: number;
readCondition?: SQL<boolean>;
caseReadCondition?: SQL<boolean>;
}
const DEFAULT_ACTIVITY_LIMIT = 100;
const MAX_ACTIVITY_LIMIT = 500;
export function normalizeActivityLimit(limit: number | undefined) {
if (!Number.isFinite(limit)) return DEFAULT_ACTIVITY_LIMIT;
return Math.max(1, Math.min(MAX_ACTIVITY_LIMIT, Math.floor(limit ?? DEFAULT_ACTIVITY_LIMIT)));
}
const RUN_LIVENESS_BACKFILL_BATCH = 20;
// Runs finalize with a liveness classification. The backfill only repairs rows
// that missed it (older rows or a failed finalization), so a delay is safe.
const RUN_LIVENESS_CLEAN_TTL_MS = 10 * 60_000;
const RUN_LIVENESS_CLEAN_MAX_ENTRIES = 10_000;
export function activityService(db: Db) {
const scheduledLivenessBackfills = new Set<string>();
// Issues whose last backfill pass left no missing liveness, keyed like
// `scheduledLivenessBackfills`. Values are the time the skip window ends.
// The map keeps insertion order, so the first entry is the oldest.
const livenessCleanUntil = new Map<string, number>();
function rememberLivenessClean(key: string) {
livenessCleanUntil.delete(key);
livenessCleanUntil.set(key, Date.now() + RUN_LIVENESS_CLEAN_TTL_MS);
while (livenessCleanUntil.size > RUN_LIVENESS_CLEAN_MAX_ENTRIES) {
const oldest = livenessCleanUntil.keys().next().value;
if (oldest === undefined) break;
livenessCleanUntil.delete(oldest);
}
}
const issueIdAsText = sql<string>`${issues.id}::text`;
const summarizedUsageJson = sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.usageJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'inputTokens', coalesce(${heartbeatRuns.usageJson} -> 'inputTokens', ${heartbeatRuns.usageJson} -> 'input_tokens'),
'input_tokens', coalesce(${heartbeatRuns.usageJson} -> 'input_tokens', ${heartbeatRuns.usageJson} -> 'inputTokens'),
'outputTokens', coalesce(${heartbeatRuns.usageJson} -> 'outputTokens', ${heartbeatRuns.usageJson} -> 'output_tokens'),
'output_tokens', coalesce(${heartbeatRuns.usageJson} -> 'output_tokens', ${heartbeatRuns.usageJson} -> 'outputTokens'),
'cachedInputTokens', coalesce(
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cached_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cache_read_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens'
),
'billingType', coalesce(${heartbeatRuns.usageJson} -> 'billingType', ${heartbeatRuns.usageJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.usageJson} -> 'billing_type', ${heartbeatRuns.usageJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'total_cost_usd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd'
)
))
end
`.as("usageJson");
// A denied user needs the selected connection's name to repair their task.
// Other readers need the connection's existing human sharing permission.
// A malformed present Stop receipt must stay unconfirmed, not disappear.
const summarizedResultJson = (viewerUserId: string | null) => sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.resultJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'conversationReset', ${heartbeatRuns.resultJson} -> 'conversationReset',
'executionCancellation', case when ${heartbeatRuns.resultJson} ? 'executionCancellation'
then jsonb_build_object('state', case
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'executionCancellation') = 'object'
and ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' in ('requested', 'acknowledged')
then ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' else 'unknown' end) end,
'nativeCancellation', case
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'nativeCancellation') = 'object'
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,schema}' = 'paperclip.native-cancellation.v1'
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,runId}' = ${heartbeatRuns.id}::text
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,companyId}' = ${heartbeatRuns.companyId}::text
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}' in ('pending', 'acknowledged')
then jsonb_build_object('schema', 'paperclip.native-cancellation.v1',
'runId', ${heartbeatRuns.id}, 'companyId', ${heartbeatRuns.companyId},
'dispatchState', ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}') end,
'configurationIncomplete', case
when ${heartbeatRuns.resultJson} -> 'configurationIncomplete' ->> 'reason' = 'workspace_base_ref_unresolved'
then jsonb_build_object('reason', 'workspace_base_ref_unresolved')
when ${heartbeatRuns.errorCode} = 'configuration_incomplete'
and (${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
or ${heartbeatRuns.error} = 'This credential is not shared with the responsible user')
then jsonb_build_object(
'selectionFailure', case when ${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
then 'ai_connection_credential_not_shared' end,
'credentialAccess', jsonb_build_object(
'connectionName', ${credentialAccessConnectionNameSql(viewerUserId)}
)
) end,
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
in (${workspaceRestoreFailureCodesSql})
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
'workspaceRestorePath', case when length(${heartbeatRuns.resultJson} ->> 'workspaceRestorePath') <= 180
then ${heartbeatRuns.resultJson} -> 'workspaceRestorePath' end,
'finalResponseRecorded', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'finalResponseRecorded') = 'boolean'
then ${heartbeatRuns.resultJson} -> 'finalResponseRecorded' end,
'billingType', coalesce(${heartbeatRuns.resultJson} -> 'billingType', ${heartbeatRuns.resultJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.resultJson} -> 'billing_type', ${heartbeatRuns.resultJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'total_cost_usd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd'
),
'stopReason', ${heartbeatRuns.resultJson} -> 'stopReason',
'effectiveTimeoutSec', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutSec',
'effectiveTimeoutMs', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutMs',
'timeoutConfigured', ${heartbeatRuns.resultJson} -> 'timeoutConfigured',
'timeoutSource', ${heartbeatRuns.resultJson} -> 'timeoutSource',
'timeoutFired', ${heartbeatRuns.resultJson} -> 'timeoutFired'
))
end
`.as("resultJson");
function countValue(value: unknown) {
const parsed = Number(value ?? 0);
return Number.isFinite(parsed) ? Math.max(0, Math.floor(parsed)) : 0;
}
function dateValue(value: unknown) {
if (value instanceof Date) return Number.isNaN(value.getTime()) ? null : value;
if (typeof value === "string" || typeof value === "number") {
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? null : parsed;
}
return null;
}
function latestDate(...values: unknown[]) {
let latest: Date | null = null;
for (const value of values) {
const parsed = dateValue(value);
if (!parsed) continue;
if (!latest || parsed.getTime() > latest.getTime()) latest = parsed;
}
return latest;
}
function asRecord(value: unknown): Record<string, unknown> | null {
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
return value as Record<string, unknown>;
}
function readNumber(value: unknown) {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
async function backfillMissingRunLivenessForIssue(companyId: string, issueId: string) {
const runs = await missingRunLivenessCandidatesQuery(db, companyId, issueId, RUN_LIVENESS_BACKFILL_BATCH);
if (runs.length === 0) return 0;
const issue = await db
.select({
status: issues.status,
title: issues.title,
description: issues.description,
workMode: issues.workMode,
})
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
.then((rows) => rows[0] ?? null);
for (const run of runs) {
const context = asRecord(run.contextSnapshot);
const continuationAttempt =
readNumber(context?.continuationAttempt) ??
readNumber(context?.livenessContinuationAttempt) ??
run.continuationAttempt ??
0;
const [commentStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueComments.createdAt})`,
})
.from(issueComments)
.where(
and(
eq(issueComments.companyId, companyId),
eq(issueComments.issueId, issueId),
eq(issueComments.createdByRunId, run.id),
),
);
const [documentStats] = await db
.select({
count: sql<number>`count(*)::int`,
planCount: sql<number>`count(*) filter (where ${issueDocuments.key} = 'plan')::int`,
latestAt: sql<Date | null>`max(${documentRevisions.createdAt})`,
})
.from(documentRevisions)
.innerJoin(issueDocuments, eq(documentRevisions.documentId, issueDocuments.documentId))
.where(
and(
eq(documentRevisions.companyId, companyId),
eq(documentRevisions.createdByRunId, run.id),
eq(issueDocuments.companyId, companyId),
eq(issueDocuments.issueId, issueId),
sql`${issueDocuments.key} != ${ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY}`,
),
);
const [workProductStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueWorkProducts.createdAt})`,
})
.from(issueWorkProducts)
.where(
and(
eq(issueWorkProducts.companyId, companyId),
eq(issueWorkProducts.issueId, issueId),
eq(issueWorkProducts.createdByRunId, run.id),
),
);
const [workspaceOperationStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${workspaceOperations.startedAt})`,
})
.from(workspaceOperations)
.where(and(eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.heartbeatRunId, run.id)));
const [activityStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${activityLog.createdAt})`,
})
.from(activityLog)
.where(and(eq(activityLog.companyId, companyId), eq(activityLog.runId, run.id)));
const [eventStats] = await db
.select({
count: sql<number>`count(*) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))::int`,
latestAt: sql<Date | null>`max(${heartbeatRunEvents.createdAt}) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))`,
})
.from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, companyId), eq(heartbeatRunEvents.runId, run.id)));
const classification = classifyRunLiveness({
runStatus: run.status,
issue,
resultJson: asRecord(run.resultJson),
stdoutExcerpt: run.stdoutExcerpt,
stderrExcerpt: run.stderrExcerpt,
error: run.error,
errorCode: run.errorCode,
continuationAttempt,
evidence: {
issueCommentsCreated: countValue(commentStats?.count),
documentRevisionsCreated: countValue(documentStats?.count),
planDocumentRevisionsCreated: countValue(documentStats?.planCount),
workProductsCreated: countValue(workProductStats?.count),
workspaceOperationsCreated: countValue(workspaceOperationStats?.count),
activityEventsCreated: countValue(activityStats?.count),
toolOrActionEventsCreated: countValue(eventStats?.count),
latestEvidenceAt: latestDate(
commentStats?.latestAt,
documentStats?.latestAt,
workProductStats?.latestAt,
workspaceOperationStats?.latestAt,
activityStats?.latestAt,
eventStats?.latestAt,
),
},
});
await db
.update(heartbeatRuns)
.set({
livenessState: classification.livenessState,
livenessReason: classification.livenessReason,
continuationAttempt: classification.continuationAttempt,
lastUsefulActionAt: classification.lastUsefulActionAt,
nextAction: classification.nextAction,
updatedAt: new Date(),
})
.where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.livenessState)));
}
return runs.length;
}
function scheduleRunLivenessBackfill(companyId: string, issueId: string) {
const key = `${companyId}:${issueId}`;
if (scheduledLivenessBackfills.has(key)) return;
const cleanUntil = livenessCleanUntil.get(key);
if (cleanUntil !== undefined) {
if (cleanUntil > Date.now()) return;
livenessCleanUntil.delete(key);
}
scheduledLivenessBackfills.add(key);
void backfillMissingRunLivenessForIssue(companyId, issueId)
.then((found) => {
// A partial batch means this pass classified every remaining candidate.
// Skip this issue for a while, so page reads do not repeat the lookup.
if (found < RUN_LIVENESS_BACKFILL_BATCH) rememberLivenessClean(key);
})
.catch((err: unknown) => {
logger.warn({ err, companyId, issueId }, "run liveness backfill failed");
})
.finally(() => {
scheduledLivenessBackfills.delete(key);
});
}
return {
list: async (filters: ActivityFilters) => {
const conditions = [eq(activityLog.companyId, filters.companyId), await skillAccessService(db).activityCondition(filters.companyId),
filters.caseReadCondition ?? sql<boolean>`${activityLog.entityType} <> 'case'`];
const limit = normalizeActivityLimit(filters.limit);
if (filters.agentId) {
conditions.push(eq(activityLog.agentId, filters.agentId));
}
if (filters.entityType) {
conditions.push(eq(activityLog.entityType, filters.entityType));
}
if (filters.entityId) {
conditions.push(eq(activityLog.entityId, filters.entityId));
}
return db
.select({ activityLog })
.from(activityLog)
.leftJoin(
issues,
and(
eq(activityLog.entityType, sql`'issue'`),
eq(activityLog.entityId, issueIdAsText),
),
)
.where(
and(
...conditions,
or(
sql`${activityLog.entityType} != 'issue'`,
and(visibleIssueCondition(), filters.readCondition ?? sql<boolean>`true`),
),
),
)
.orderBy(desc(activityLog.createdAt))
.limit(limit)
.then((rows) => rows.map((r) => r.activityLog));
},
inputFactsForIssue: async (companyId: string, issueId: string, commentIds: string[]) => {
if (!commentIds.length) return [];
const commentId = sql<string>`${activityLog.details}->>'commentId'`;
return db.selectDistinctOn([commentId, activityLog.action])
.from(activityLog)
.where(and(
eq(activityLog.companyId, companyId),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, issueId),
inArray(commentId, commentIds),
inArray(activityLog.action, ["issue.queued_comment_steered", "issue.comment_added"]),
sql`${activityLog.details}->>'duplicate' is distinct from 'true'`,
))
.orderBy(commentId, activityLog.action, asc(activityLog.createdAt), asc(activityLog.id));
},
forIssue: (issueId: string, limit?: number) => {
const query = db
.select()
.from(activityLog)
.where(
or(
and(eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issueId)),
and(or(eq(activityLog.action, "project.created"), eq(activityLog.action, "company.skill_created")), sql`${activityLog.details}->>'sourceIssueId' = ${issueId}`,
sql`${activityLog.companyId} = (select company_id from issues where id = ${issueId})`),
),
)
.orderBy(desc(activityLog.createdAt), desc(activityLog.id));
return limit === undefined ? query : query.limit(limit);
},
runsForIssue: async (
companyId: string,
issueId: string,
viewerUserId: string | null = null,
limit?: number,
selectedRunIds?: string[],
// Read-only callers (the chat bootstrap) skip the background liveness
// backfill: it writes to runs and outlives the request's own limits.
// The task page's own `/runs` read still schedules it.
options: { skipLivenessBackfill?: boolean } = {},
) => {
if (selectedRunIds && !selectedRunIds.length) return [];
if (!options.skipLivenessBackfill) scheduleRunLivenessBackfill(companyId, issueId);
const runsQuery = db
.select({
runId: heartbeatRuns.id,
runtimeMode: heartbeatRuns.runtimeMode,
wakeupRequestId: heartbeatRuns.wakeupRequestId,
status: heartbeatRuns.status,
agentId: heartbeatRuns.agentId,
adapterType: agents.adapterType,
startedAt: heartbeatRuns.startedAt,
finishedAt: heartbeatRuns.finishedAt,
createdAt: heartbeatRuns.createdAt,
invocationSource: heartbeatRuns.invocationSource,
responsibleUserId: heartbeatRuns.responsibleUserId,
errorCode: heartbeatRuns.errorCode,
error: sql<string | null>`case when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'native_provider_model_rejected'
then left(${heartbeatRuns.error}, 2000)
when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.error} = 'This credential is not shared with the responsible user'
then 'This credential is not shared with the responsible user' else null end`,
usageJson: summarizedUsageJson,
resultJson: summarizedResultJson(viewerUserId),
// Bounded cause of a generic failure code; validated below, legacy rows have none.
failureCause: sql<string | null>`case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'failureCause') = 'string'
then left(${heartbeatRuns.resultJson} ->> 'failureCause', ${RUN_FAILURE_CAUSE_MAX_LENGTH + 1}) end`,
logBytes: heartbeatRuns.logBytes,
retryOfRunId: heartbeatRuns.retryOfRunId,
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
livenessState: heartbeatRuns.livenessState,
livenessReason: heartbeatRuns.livenessReason,
continuationAttempt: heartbeatRuns.continuationAttempt,
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
nextAction: heartbeatRuns.nextAction,
wakeCommentIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'wakeCommentIds'`,
queuedCommentSourceWakeIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'queuedCommentSourceWakeIds'`,
wakeCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'wakeCommentId'`,
contextCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'commentId'`,
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`,
})
.from(heartbeatRuns)
.innerJoin(
agents,
and(
eq(agents.id, heartbeatRuns.agentId),
eq(agents.companyId, heartbeatRuns.companyId),
),
)
.where(
and(
eq(heartbeatRuns.companyId, companyId),
sql`(not exists (select 1 from ${issues} where ${issues.id} = ${issueId}
and ${issues.companyId} = ${companyId} and ${issues.conversationAgentId} is not null)
or coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
${heartbeatRuns.contextSnapshot} ->> 'issueId', ${heartbeatRuns.contextSnapshot} ->> 'taskId') = ${issueId})`,
...(selectedRunIds ? [inArray(heartbeatRuns.id, selectedRunIds)] : []),
or(
eq(heartbeatRuns.issueId, issueId),
sql`exists (
select 1
from ${activityLog}
where ${activityLog.companyId} = ${companyId}
and ${activityLog.entityType} = 'issue'
and ${activityLog.entityId} = ${issueId}
and ${activityLog.runId} = ${heartbeatRuns.id}
)`,
),
),
)
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id));
const runs = await (limit === undefined ? runsQuery : runsQuery.limit(limit));
if (runs.length === 0) return runs;
const runIds = runs.map((run) => run.runId);
if (runIds.length === 0) return runs;
const exhaustionRowsQuery = db
.select({
runId: heartbeatRunEvents.runId,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
),
)
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id));
const leaseRowsQuery = db
.select({
lease: environmentLeases,
environment: {
id: environments.id,
name: environments.name,
driver: environments.driver,
},
})
.from(environmentLeases)
.innerJoin(environments, eq(environmentLeases.environmentId, environments.id))
.where(
and(
eq(environmentLeases.companyId, companyId),
inArray(environmentLeases.heartbeatRunId, runIds),
),
)
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
// Only stored, current plan revisions can support a saved-plan link.
// Do not trust an adapter's claim that it wrote a document.
const savedPlanQuery = runs.some((run) => hasWorkspaceRestoreFailure(run.resultJson))
? db.select({ revisionId: documentRevisions.id, runId: documentRevisions.createdByRunId })
.from(issueDocuments)
.innerJoin(documents, and(eq(documents.id, issueDocuments.documentId), eq(documents.companyId, companyId)))
.innerJoin(documentRevisions, and(eq(documentRevisions.id, documents.latestRevisionId), eq(documentRevisions.documentId, documents.id), eq(documentRevisions.companyId, companyId)))
.where(and(eq(issueDocuments.companyId, companyId), eq(issueDocuments.issueId, issueId), eq(issueDocuments.key, "plan")))
.limit(1)
: Promise.resolve([]);
const [exhaustionRows, leaseRows, executionByRunId, [savedPlan]] = await Promise.all([
exhaustionRowsQuery,
leaseRowsQuery,
executionProjectionsForRuns(db, companyId, runIds, new Date(), { retryDatabaseReads: true }),
savedPlanQuery,
]);
const retryExhaustedReasonByRunId = new Map<string, string>();
for (const row of exhaustionRows) {
if (!row.message || retryExhaustedReasonByRunId.has(row.runId)) continue;
retryExhaustedReasonByRunId.set(row.runId, row.message);
}
const leaseByRunId = new Map<string, (typeof leaseRows)[number]>();
for (const row of leaseRows) {
if (row.lease.heartbeatRunId && !leaseByRunId.has(row.lease.heartbeatRunId)) {
leaseByRunId.set(row.lease.heartbeatRunId, row);
}
}
return runs.map((run) => {
const leaseRow = leaseByRunId.get(run.runId);
const leaseMetadata = leaseRow?.lease.metadata ?? null;
const workspacePath =
typeof leaseMetadata?.remoteCwd === "string" && leaseMetadata.remoteCwd.trim().length > 0
? leaseMetadata.remoteCwd
: typeof leaseMetadata?.remoteWorkspacePath === "string" && leaseMetadata.remoteWorkspacePath.trim().length > 0
? leaseMetadata.remoteWorkspacePath
: null;
return {
...run,
failureCause: normalizeRunFailureCause(run.failureCause),
resultJson: run.resultJson ? {
...run.resultJson,
...(Object.hasOwn(run.resultJson, "workspaceRestorePath") ? {
workspaceRestorePath: safeWorkspaceRestorePath(run.resultJson.workspaceRestorePath),
} : {}),
...(hasWorkspaceRestoreFailure(run.resultJson) ? {
...(savedPlan?.runId === run.runId ? { savedPlanRevisionId: savedPlan.revisionId } : {}),
} : {}),
} : null,
execution: executionByRunId.get(run.runId) ?? null,
environment: leaseRow
? {
id: leaseRow.environment.id,
name: leaseRow.environment.name,
driver: leaseRow.environment.driver,
}
: null,
environmentLease: leaseRow
? {
id: leaseRow.lease.id,
status: leaseRow.lease.status,
leasePolicy: leaseRow.lease.leasePolicy,
provider: leaseRow.lease.provider,
providerLeaseId: leaseRow.lease.providerLeaseId,
executionWorkspaceId: leaseRow.lease.executionWorkspaceId,
workspacePath,
failureReason: leaseRow.lease.failureReason,
cleanupStatus: leaseRow.lease.cleanupStatus,
acquiredAt: leaseRow.lease.acquiredAt,
releasedAt: leaseRow.lease.releasedAt,
}
: null,
retryExhaustedReason: retryExhaustedReasonByRunId.get(run.runId) ?? null,
};
});
},
issuesForRun: async (runId: string, readCondition: SQL<boolean>) => {
const run = await db
.select({
companyId: heartbeatRuns.companyId,
issueId: heartbeatRuns.issueId,
})
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.then((rows) => rows[0] ?? null);
if (!run) return [];
const fromActivity = await db
.selectDistinctOn([issueIdAsText], {
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(activityLog)
.innerJoin(issues, eq(activityLog.entityId, issueIdAsText))
.where(
and(
eq(activityLog.companyId, run.companyId),
eq(activityLog.runId, runId),
eq(activityLog.entityType, "issue"),
eq(issues.companyId, run.companyId),
visibleIssueCondition(),
readCondition,
),
)
.orderBy(issueIdAsText);
const contextIssueId = run.issueId;
if (!contextIssueId) return fromActivity;
if (fromActivity.some((issue) => issue.issueId === contextIssueId)) return fromActivity;
const fromContext = await db
.select({
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(issues)
.where(
and(
eq(issues.companyId, run.companyId),
eq(issues.id, contextIssueId),
visibleIssueCondition(),
readCondition,
),
)
.then((rows) => rows[0] ?? null);
if (!fromContext) return fromActivity;
return [fromContext, ...fromActivity];
},
create: (data: typeof activityLog.$inferInsert) => db.transaction(async tx => {
if (data.entityType === "issue") await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, `task:${data.entityId}`);
const [row] = await tx.insert(activityLog).values(data).returning();
return row;
}),
};
}