Files
PaperClipAI/server/src/services/activity.ts
T
DottaandClaude Opus 5.5 ee6922f705 perf(tasks): open recent tasks without waiting on a slow ten-task preload (#21939)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A person opens a task from the sidebar's Recent Tasks, the Inbox or
the Tasks list, and the task chat shows its first frame from a
`chat-bootstrap` read
> - On staging, a task opened from Recent Tasks soon after a page load
took 15.3 s to show its chat. The same task from the Inbox took 0.19 s
> - Two defects cause this. Every task's bootstrap paid a fixed ~1.8 s
for its latest runs. Recent Tasks preloaded ten tasks in one request
(17.7 s), and an open waited for that whole request
> - Recent Tasks also warmed its tasks only once per page load. Unread
cache entries are dropped after five minutes, so its tasks were cold for
the rest of the session
> - This pull request makes the runs read use indexes, lets an open read
its own task instead of waiting for a ten-task preload, and re-warms
Recent Tasks on navigation
> - The benefit is that a task from Recent Tasks opens as fast as from a
task list

## Linked Issues or Issue Description

Refs #19818, Refs #16713 (earlier proposals of the same `runsForIssue`
rewrite; this PR adds a query-plan regression test and the client
fixes).

**What happened?**
On staging, a click on a Recent Tasks link about 6 s after a page load
took 15.3 s until the chat thread showed. The Inbox opened the same task
in 0.19 s.

**Steps to reproduce**
1. Load a page with a populated Recent Tasks sidebar.
2. Within a few seconds, click a recent task.
3. The chat stays on its loading state for many seconds.

**Expected behavior**
The task opens about as fast as from the Inbox or the Tasks list.

**Root cause (measured on staging)**
- `GET /issues/:id/runs?view=recent` took 1.7–1.9 s of server time for
tasks with 1 to 24 runs. Every other first-frame read took 80–350 ms.
The query filtered with `issue_id = $1 OR EXISTS (activity_log … run_id
= heartbeat_runs.id)`, so Postgres walked every run of the company.
- The bootstrap reads two tasks at a time, so the Recents preload of ten
tasks took 17.7 s. Each task in a batch settles only with the whole
request, and the page's reads wait for it (up to the 5 s share deadline,
then read again under load).
- `useRecentTaskOpenWarmup` remembered preloaded task ids for the life
of the page and never preloaded them again.

## What Changed

- `activityService.runsForIssue`: select the task's runs by `id IN (runs
with issue_id = task UNION run ids from the task's activity)`. Each half
uses its own index. The results are the same.
- `startIssueChatBootstrap`: when the only request for a task is a
background batch that is still loading, send a request for that task
alone. All refs of the task point to the new request. The later batch
answer only fills keys that are still empty or older.
- `loadIssueChatBootstraps` (background): a task whose answer did not
arrive is not held back for five minutes, so the next preload can retry
it.
- `useRecentTaskOpenWarmup`: use the background preload (five-minute
repeat, the same as the cache GC time) and run again after each
navigation (`navigationKey`). It no longer keeps a per-page-load set.
`SidebarRecentTasks` and `CoordinatorSidebar` pass the pathname.

## Verification

- `server/src/__tests__/issue-runs-read-cost.test.ts` (new): a company
with 20,003 runs, and a task with 3 runs. EXPLAIN ANALYZE of the runs
reads examined **40,011** run rows before the fix and **15** after
(red/green). The test also checks the run order and the run that is
linked only through activity.
- `ui/src/lib/issue-chat-bootstrap.test.ts`: a new test opens a task
while a ten-task preload is still loading. Before the fix it timed out
waiting on the batch. After the fix it reads its own request. A second
new test checks that a single-task request is still shared.
- `ui/src/hooks/useRecentTaskOpenWarmup.test.tsx`: a new test re-warms
Recents on a navigation after five minutes (red on the old hook).
Another new test sends nothing for navigations inside five minutes.
- Existing suites pass: activity service/routes, task cold open, agent
conversations (197 tests); sidebar, Recents, navigation prefetch, issue
detail cache, IssueDetail (341 tests).
- `tsc --noEmit` is clean for `server` and `ui` (apart from the
pre-existing Storybook index errors).
- `pnpm verify` exits 2 (an opaque server setup dependency), so CI
supplies the broad coverage.
- After deploy: on staging, time a Recent Tasks click a few seconds
after a cold load, and time `runs?view=recent`.

## Risks

- Low risk. The SQL change keeps the same filter. It changes only how
Postgres finds the candidate runs. The existing activity tests cover
retry, credential and conversation cases.
- An open during a background preload now sends one more bootstrap
request for that task. This happens only on an open or a resting hover,
never for each row.
- Recents now sends at most one preload per five minutes while the
person navigates. It sends nothing while the tab is idle or hidden.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser measurement on staging, code execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 15:26:22 -06:00

807 lines
37 KiB
TypeScript

import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { skillAccessService } from "./skill-access.js";
import { credentialAccessConnectionNameSql } from "./credential-access-visibility.js";
import { acceptedFirstTurnCommentIdsSql } from "./queued-comment-delivery.js";
import { executionProjectionsForRuns } from "./execution-projection.js";
import { and, asc, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
activityLog,
agentWakeupRequests,
agents,
authUsers,
documentRevisions,
documents,
environmentLeases,
environments,
heartbeatRunEvents,
heartbeatRuns,
workspaceRestoreFailureCodesSql,
issueComments,
issueDocuments,
issues,
issueWorkProducts,
workspaceOperations,
} from "@paperclipai/db";
import { hasWorkspaceRestoreFailure, safeWorkspaceRestorePath, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, normalizeRunFailureCause, RUN_FAILURE_CAUSE_MAX_LENGTH } from "@paperclipai/shared";
import { logger } from "../middleware/logger.js";
import { visibleIssueCondition } from "./issue-visibility.js";
import { classifyRunLiveness } from "./run-liveness.js";
import { missingRunLivenessCandidatesQuery } from "./run-liveness-backfill-query.js";
export interface ActivityFilters {
companyId: string;
agentId?: string;
entityType?: string;
entityId?: string;
limit?: number;
readCondition?: SQL<boolean>;
caseReadCondition?: SQL<boolean>;
}
const DEFAULT_ACTIVITY_LIMIT = 100;
const MAX_ACTIVITY_LIMIT = 500;
export function normalizeActivityLimit(limit: number | undefined) {
if (!Number.isFinite(limit)) return DEFAULT_ACTIVITY_LIMIT;
return Math.max(1, Math.min(MAX_ACTIVITY_LIMIT, Math.floor(limit ?? DEFAULT_ACTIVITY_LIMIT)));
}
const RUN_LIVENESS_BACKFILL_BATCH = 20;
// Runs finalize with a liveness classification. The backfill only repairs rows
// that missed it (older rows or a failed finalization), so a delay is safe.
const RUN_LIVENESS_CLEAN_TTL_MS = 10 * 60_000;
const RUN_LIVENESS_CLEAN_MAX_ENTRIES = 10_000;
const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
/** A person-facing name for an activity actor, so readers never see a raw id such as local-board. */
export function activityActorFallbackName(actorType: string, actorId: string) {
if (actorType === "user") return actorId === "local-board" ? "Board" : "A board member";
if (actorType === "agent") return "An agent";
if (actorType === "plugin") return "A plugin";
return "Paperclip";
}
/** Wake reasons recorded when a person presses Retry on a failed run or task. */
const PERSON_RETRY_WAKE_REASONS = new Set(["retry_failed_run", "task_retry"]);
export type RunRetriedBy = { kind: "person"; personName: string; summary: string } | { kind: "automatic"; summary: string };
/**
* Who started a retry run, in plain words. A run's own invocation fields do
* not say this: a person's Retry is an on-demand run whose responsible user
* may also own automatic retries. The run's wake request records whether a
* person pressed Retry; automatic recovery creates an automation run. Returns
* null for a run that is not a retry or whose starter is not recorded, rather
* than guessing.
*/
export function describeRunRetry(
run: { retryOfRunId: string | null; taskRetryRequestId?: string | null; invocationSource: string | null },
wake: { reason: string | null; requestedByActorType: string | null } | null,
personName: string | null,
): RunRetriedBy | null {
if (!run.retryOfRunId && !run.taskRetryRequestId) return null;
// The run's own start decides first: automatic recovery after a person's
// Task Retry inherits that Retry's request, but the person did not start it.
if (run.invocationSource === "automation") {
return { kind: "automatic", summary: "Paperclip retried the failed run automatically; no person pressed Retry." };
}
if (wake?.requestedByActorType === "user" && wake.reason && PERSON_RETRY_WAKE_REASONS.has(wake.reason)) {
const name = personName?.trim() || "A board member";
return { kind: "person", personName: name, summary: `${name} pressed Retry on the failed run. Paperclip did not retry it automatically.` };
}
return null;
}
export function activityService(db: Db) {
const scheduledLivenessBackfills = new Set<string>();
// Issues whose last backfill pass left no missing liveness, keyed like
// `scheduledLivenessBackfills`. Values are the time the skip window ends.
// The map keeps insertion order, so the first entry is the oldest.
const livenessCleanUntil = new Map<string, number>();
function rememberLivenessClean(key: string) {
livenessCleanUntil.delete(key);
livenessCleanUntil.set(key, Date.now() + RUN_LIVENESS_CLEAN_TTL_MS);
while (livenessCleanUntil.size > RUN_LIVENESS_CLEAN_MAX_ENTRIES) {
const oldest = livenessCleanUntil.keys().next().value;
if (oldest === undefined) break;
livenessCleanUntil.delete(oldest);
}
}
const issueIdAsText = sql<string>`${issues.id}::text`;
const summarizedUsageJson = sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.usageJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'inputTokens', coalesce(${heartbeatRuns.usageJson} -> 'inputTokens', ${heartbeatRuns.usageJson} -> 'input_tokens'),
'input_tokens', coalesce(${heartbeatRuns.usageJson} -> 'input_tokens', ${heartbeatRuns.usageJson} -> 'inputTokens'),
'outputTokens', coalesce(${heartbeatRuns.usageJson} -> 'outputTokens', ${heartbeatRuns.usageJson} -> 'output_tokens'),
'output_tokens', coalesce(${heartbeatRuns.usageJson} -> 'output_tokens', ${heartbeatRuns.usageJson} -> 'outputTokens'),
'cachedInputTokens', coalesce(
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cached_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cache_read_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens'
),
'billingType', coalesce(${heartbeatRuns.usageJson} -> 'billingType', ${heartbeatRuns.usageJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.usageJson} -> 'billing_type', ${heartbeatRuns.usageJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'total_cost_usd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd'
)
))
end
`.as("usageJson");
// A denied user needs the selected connection's name to repair their task.
// Other readers need the connection's existing human sharing permission.
// A malformed present Stop receipt must stay unconfirmed, not disappear.
const summarizedResultJson = (viewerUserId: string | null) => sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.resultJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'conversationReset', ${heartbeatRuns.resultJson} -> 'conversationReset',
'executionCancellation', case when ${heartbeatRuns.resultJson} ? 'executionCancellation'
then jsonb_build_object('state', case
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'executionCancellation') = 'object'
and ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' in ('requested', 'acknowledged')
then ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' else 'unknown' end) end,
'nativeCancellation', case
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'nativeCancellation') = 'object'
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,schema}' = 'paperclip.native-cancellation.v1'
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,runId}' = ${heartbeatRuns.id}::text
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,companyId}' = ${heartbeatRuns.companyId}::text
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}' in ('pending', 'acknowledged')
then jsonb_build_object('schema', 'paperclip.native-cancellation.v1',
'runId', ${heartbeatRuns.id}, 'companyId', ${heartbeatRuns.companyId},
'dispatchState', ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}') end,
'configurationIncomplete', case
when ${heartbeatRuns.resultJson} -> 'configurationIncomplete' ->> 'reason' = 'workspace_base_ref_unresolved'
then jsonb_build_object('reason', 'workspace_base_ref_unresolved')
when ${heartbeatRuns.errorCode} = 'configuration_incomplete'
and (${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
or ${heartbeatRuns.error} = 'This credential is not shared with the responsible user')
then jsonb_build_object(
'selectionFailure', case when ${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
then 'ai_connection_credential_not_shared' end,
'credentialAccess', jsonb_build_object(
'connectionName', ${credentialAccessConnectionNameSql(viewerUserId)}
)
) end,
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
in (${workspaceRestoreFailureCodesSql})
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
'workspaceRestorePath', case when length(${heartbeatRuns.resultJson} ->> 'workspaceRestorePath') <= 180
then ${heartbeatRuns.resultJson} -> 'workspaceRestorePath' end,
'finalResponseRecorded', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'finalResponseRecorded') = 'boolean'
then ${heartbeatRuns.resultJson} -> 'finalResponseRecorded' end,
'billingType', coalesce(${heartbeatRuns.resultJson} -> 'billingType', ${heartbeatRuns.resultJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.resultJson} -> 'billing_type', ${heartbeatRuns.resultJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'total_cost_usd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd'
),
'stopReason', ${heartbeatRuns.resultJson} -> 'stopReason',
'effectiveTimeoutSec', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutSec',
'effectiveTimeoutMs', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutMs',
'timeoutConfigured', ${heartbeatRuns.resultJson} -> 'timeoutConfigured',
'timeoutSource', ${heartbeatRuns.resultJson} -> 'timeoutSource',
'timeoutFired', ${heartbeatRuns.resultJson} -> 'timeoutFired'
))
end
`.as("resultJson");
function countValue(value: unknown) {
const parsed = Number(value ?? 0);
return Number.isFinite(parsed) ? Math.max(0, Math.floor(parsed)) : 0;
}
function dateValue(value: unknown) {
if (value instanceof Date) return Number.isNaN(value.getTime()) ? null : value;
if (typeof value === "string" || typeof value === "number") {
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? null : parsed;
}
return null;
}
function latestDate(...values: unknown[]) {
let latest: Date | null = null;
for (const value of values) {
const parsed = dateValue(value);
if (!parsed) continue;
if (!latest || parsed.getTime() > latest.getTime()) latest = parsed;
}
return latest;
}
function asRecord(value: unknown): Record<string, unknown> | null {
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
return value as Record<string, unknown>;
}
function readNumber(value: unknown) {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
async function backfillMissingRunLivenessForIssue(companyId: string, issueId: string) {
const runs = await missingRunLivenessCandidatesQuery(db, companyId, issueId, RUN_LIVENESS_BACKFILL_BATCH);
if (runs.length === 0) return 0;
const issue = await db
.select({
status: issues.status,
title: issues.title,
description: issues.description,
workMode: issues.workMode,
})
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
.then((rows) => rows[0] ?? null);
for (const run of runs) {
const context = asRecord(run.contextSnapshot);
const continuationAttempt =
readNumber(context?.continuationAttempt) ??
readNumber(context?.livenessContinuationAttempt) ??
run.continuationAttempt ??
0;
const [commentStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueComments.createdAt})`,
})
.from(issueComments)
.where(
and(
eq(issueComments.companyId, companyId),
eq(issueComments.issueId, issueId),
eq(issueComments.createdByRunId, run.id),
),
);
const [documentStats] = await db
.select({
count: sql<number>`count(*)::int`,
planCount: sql<number>`count(*) filter (where ${issueDocuments.key} = 'plan')::int`,
latestAt: sql<Date | null>`max(${documentRevisions.createdAt})`,
})
.from(documentRevisions)
.innerJoin(issueDocuments, eq(documentRevisions.documentId, issueDocuments.documentId))
.where(
and(
eq(documentRevisions.companyId, companyId),
eq(documentRevisions.createdByRunId, run.id),
eq(issueDocuments.companyId, companyId),
eq(issueDocuments.issueId, issueId),
sql`${issueDocuments.key} != ${ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY}`,
),
);
const [workProductStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueWorkProducts.createdAt})`,
})
.from(issueWorkProducts)
.where(
and(
eq(issueWorkProducts.companyId, companyId),
eq(issueWorkProducts.issueId, issueId),
eq(issueWorkProducts.createdByRunId, run.id),
),
);
const [workspaceOperationStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${workspaceOperations.startedAt})`,
})
.from(workspaceOperations)
.where(and(eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.heartbeatRunId, run.id)));
const [activityStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${activityLog.createdAt})`,
})
.from(activityLog)
.where(and(eq(activityLog.companyId, companyId), eq(activityLog.runId, run.id)));
const [eventStats] = await db
.select({
count: sql<number>`count(*) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))::int`,
latestAt: sql<Date | null>`max(${heartbeatRunEvents.createdAt}) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))`,
})
.from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, companyId), eq(heartbeatRunEvents.runId, run.id)));
const classification = classifyRunLiveness({
runStatus: run.status,
issue,
resultJson: asRecord(run.resultJson),
stdoutExcerpt: run.stdoutExcerpt,
stderrExcerpt: run.stderrExcerpt,
error: run.error,
errorCode: run.errorCode,
continuationAttempt,
evidence: {
issueCommentsCreated: countValue(commentStats?.count),
documentRevisionsCreated: countValue(documentStats?.count),
planDocumentRevisionsCreated: countValue(documentStats?.planCount),
workProductsCreated: countValue(workProductStats?.count),
workspaceOperationsCreated: countValue(workspaceOperationStats?.count),
activityEventsCreated: countValue(activityStats?.count),
toolOrActionEventsCreated: countValue(eventStats?.count),
latestEvidenceAt: latestDate(
commentStats?.latestAt,
documentStats?.latestAt,
workProductStats?.latestAt,
workspaceOperationStats?.latestAt,
activityStats?.latestAt,
eventStats?.latestAt,
),
},
});
await db
.update(heartbeatRuns)
.set({
livenessState: classification.livenessState,
livenessReason: classification.livenessReason,
continuationAttempt: classification.continuationAttempt,
lastUsefulActionAt: classification.lastUsefulActionAt,
nextAction: classification.nextAction,
updatedAt: new Date(),
})
.where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.livenessState)));
}
return runs.length;
}
function scheduleRunLivenessBackfill(companyId: string, issueId: string) {
const key = `${companyId}:${issueId}`;
if (scheduledLivenessBackfills.has(key)) return;
const cleanUntil = livenessCleanUntil.get(key);
if (cleanUntil !== undefined) {
if (cleanUntil > Date.now()) return;
livenessCleanUntil.delete(key);
}
scheduledLivenessBackfills.add(key);
void backfillMissingRunLivenessForIssue(companyId, issueId)
.then((found) => {
// A partial batch means this pass classified every remaining candidate.
// Skip this issue for a while, so page reads do not repeat the lookup.
if (found < RUN_LIVENESS_BACKFILL_BATCH) rememberLivenessClean(key);
})
.catch((err: unknown) => {
logger.warn({ err, companyId, issueId }, "run liveness backfill failed");
})
.finally(() => {
scheduledLivenessBackfills.delete(key);
});
}
return {
list: async (filters: ActivityFilters) => {
const conditions = [eq(activityLog.companyId, filters.companyId), await skillAccessService(db).activityCondition(filters.companyId),
filters.caseReadCondition ?? sql<boolean>`${activityLog.entityType} <> 'case'`];
const limit = normalizeActivityLimit(filters.limit);
if (filters.agentId) {
conditions.push(eq(activityLog.agentId, filters.agentId));
}
if (filters.entityType) {
conditions.push(eq(activityLog.entityType, filters.entityType));
}
if (filters.entityId) {
conditions.push(eq(activityLog.entityId, filters.entityId));
}
return db
.select({ activityLog })
.from(activityLog)
.leftJoin(
issues,
and(
eq(activityLog.entityType, sql`'issue'`),
eq(activityLog.entityId, issueIdAsText),
),
)
.where(
and(
...conditions,
or(
sql`${activityLog.entityType} != 'issue'`,
and(visibleIssueCondition(), filters.readCondition ?? sql<boolean>`true`),
),
),
)
.orderBy(desc(activityLog.createdAt))
.limit(limit)
.then((rows) => rows.map((r) => r.activityLog));
},
inputFactsForIssue: async (companyId: string, issueId: string, commentIds: string[]) => {
if (!commentIds.length) return [];
const commentId = sql<string>`${activityLog.details}->>'commentId'`;
return db.selectDistinctOn([commentId, activityLog.action])
.from(activityLog)
.where(and(
eq(activityLog.companyId, companyId),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, issueId),
inArray(commentId, commentIds),
inArray(activityLog.action, ["issue.queued_comment_steered", "issue.comment_added"]),
sql`${activityLog.details}->>'duplicate' is distinct from 'true'`,
))
.orderBy(commentId, activityLog.action, asc(activityLog.createdAt), asc(activityLog.id));
},
/** Adds actorName to each row; agents and API readers name the actor without its raw id. */
withActorNames: async <T extends { actorType: string; actorId: string; companyId: string }>(rows: T[]): Promise<Array<T & { actorName: string }>> => {
// actorId is free text; only a UUID can name a row in agents.
const agentIds = [...new Set(rows.filter(row => row.actorType === "agent" && UUID_PATTERN.test(row.actorId)).map(row => row.actorId))];
const userIds = [...new Set(rows.filter(row => row.actorType === "user" && row.actorId !== "local-board").map(row => row.actorId))];
const [agentRows, userRows] = await Promise.all([
agentIds.length ? db.select({ id: agents.id, companyId: agents.companyId, name: agents.name }).from(agents).where(inArray(agents.id, agentIds)) : [],
userIds.length ? db.select({ id: authUsers.id, name: authUsers.name }).from(authUsers).where(inArray(authUsers.id, userIds)) : [],
]);
const agentNames = new Map(agentRows.map(row => [`${row.companyId}:${row.id}`, row.name]));
const userNames = new Map(userRows.filter(row => row.name?.trim()).map(row => [row.id, row.name!.trim()]));
return rows.map(row => ({ ...row, actorName: (row.actorType === "agent" ? agentNames.get(`${row.companyId}:${row.actorId}`)
: row.actorType === "user" ? userNames.get(row.actorId) : undefined) ?? activityActorFallbackName(row.actorType, row.actorId) }));
},
forIssue: (issueId: string, limit?: number) => {
const query = db
.select()
.from(activityLog)
.where(
or(
and(eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issueId)),
and(or(eq(activityLog.action, "project.created"), eq(activityLog.action, "company.skill_created")), sql`${activityLog.details}->>'sourceIssueId' = ${issueId}`,
sql`${activityLog.companyId} = (select company_id from issues where id = ${issueId})`),
),
)
.orderBy(desc(activityLog.createdAt), desc(activityLog.id));
return limit === undefined ? query : query.limit(limit);
},
runsForIssue: async (
companyId: string,
issueId: string,
viewerUserId: string | null = null,
limit?: number,
selectedRunIds?: string[],
// Read-only callers (the chat bootstrap) skip the background liveness
// backfill: it writes to runs and outlives the request's own limits.
// The task page's own `/runs` read still schedules it.
options: { skipLivenessBackfill?: boolean } = {},
) => {
if (selectedRunIds && !selectedRunIds.length) return [];
if (!options.skipLivenessBackfill) scheduleRunLivenessBackfill(companyId, issueId);
const runsQuery = db
.select({
runId: heartbeatRuns.id,
runtimeMode: heartbeatRuns.runtimeMode,
wakeupRequestId: heartbeatRuns.wakeupRequestId,
status: heartbeatRuns.status,
agentId: heartbeatRuns.agentId,
adapterType: agents.adapterType,
startedAt: heartbeatRuns.startedAt,
finishedAt: heartbeatRuns.finishedAt,
createdAt: heartbeatRuns.createdAt,
invocationSource: heartbeatRuns.invocationSource,
responsibleUserId: heartbeatRuns.responsibleUserId,
errorCode: heartbeatRuns.errorCode,
error: sql<string | null>`case when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'native_provider_model_rejected'
then left(${heartbeatRuns.error}, 2000)
when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.error} = 'This credential is not shared with the responsible user'
then 'This credential is not shared with the responsible user' else null end`,
usageJson: summarizedUsageJson,
resultJson: summarizedResultJson(viewerUserId),
// Bounded cause of a generic failure code; validated below, legacy rows have none.
failureCause: sql<string | null>`case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'failureCause') = 'string'
then left(${heartbeatRuns.resultJson} ->> 'failureCause', ${RUN_FAILURE_CAUSE_MAX_LENGTH + 1}) end`,
logBytes: heartbeatRuns.logBytes,
retryOfRunId: heartbeatRuns.retryOfRunId,
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
livenessState: heartbeatRuns.livenessState,
livenessReason: heartbeatRuns.livenessReason,
continuationAttempt: heartbeatRuns.continuationAttempt,
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
nextAction: heartbeatRuns.nextAction,
wakeCommentIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'wakeCommentIds'`,
// Queued messages this run's own accepted first turn took.
firstTurnCommentIds: acceptedFirstTurnCommentIdsSql(heartbeatRuns),
queuedCommentSourceWakeIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'queuedCommentSourceWakeIds'`,
wakeCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'wakeCommentId'`,
contextCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'commentId'`,
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`,
// A person's Task Retry links its saved request here instead of retryOfRunId.
taskRetryRequestId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'taskRetryRequestId'`,
})
.from(heartbeatRuns)
.innerJoin(
agents,
and(
eq(agents.id, heartbeatRuns.agentId),
eq(agents.companyId, heartbeatRuns.companyId),
),
)
.where(
and(
eq(heartbeatRuns.companyId, companyId),
sql`(not exists (select 1 from ${issues} where ${issues.id} = ${issueId}
and ${issues.companyId} = ${companyId} and ${issues.conversationAgentId} is not null)
or coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
${heartbeatRuns.contextSnapshot} ->> 'issueId', ${heartbeatRuns.contextSnapshot} ->> 'taskId') = ${issueId})`,
...(selectedRunIds ? [inArray(heartbeatRuns.id, selectedRunIds)] : []),
// The task's runs: linked directly, or through the task's activity.
// Collect the candidate ids first, each half by its own index. An
// OR with a per-run EXISTS walks every run of the company instead.
sql`${heartbeatRuns.id} in (
select hr.id from ${heartbeatRuns} hr
where hr.company_id = ${companyId} and hr.issue_id = ${issueId}
union
select al.run_id from ${activityLog} al
where al.company_id = ${companyId}
and al.entity_type = 'issue'
and al.entity_id = ${issueId}
and al.run_id is not null
)`,
),
)
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id));
const runs = await (limit === undefined ? runsQuery : runsQuery.limit(limit));
if (runs.length === 0) return [];
const runIds = runs.map((run) => run.runId);
if (runIds.length === 0) return [];
const exhaustionRowsQuery = db
.select({
runId: heartbeatRunEvents.runId,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
),
)
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id));
const leaseRowsQuery = db
.select({
lease: environmentLeases,
environment: {
id: environments.id,
name: environments.name,
driver: environments.driver,
},
})
.from(environmentLeases)
.innerJoin(environments, eq(environmentLeases.environmentId, environments.id))
.where(
and(
eq(environmentLeases.companyId, companyId),
inArray(environmentLeases.heartbeatRunId, runIds),
),
)
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
// Only stored, current plan revisions can support a saved-plan link.
// Do not trust an adapter's claim that it wrote a document.
const savedPlanQuery = runs.some((run) => hasWorkspaceRestoreFailure(run.resultJson))
? db.select({ revisionId: documentRevisions.id, runId: documentRevisions.createdByRunId })
.from(issueDocuments)
.innerJoin(documents, and(eq(documents.id, issueDocuments.documentId), eq(documents.companyId, companyId)))
.innerJoin(documentRevisions, and(eq(documentRevisions.id, documents.latestRevisionId), eq(documentRevisions.documentId, documents.id), eq(documentRevisions.companyId, companyId)))
.where(and(eq(issueDocuments.companyId, companyId), eq(issueDocuments.issueId, issueId), eq(issueDocuments.key, "plan")))
.limit(1)
: Promise.resolve([]);
const retryWakeId = (run: (typeof runs)[number]) => run.taskRetryRequestId && UUID_PATTERN.test(run.taskRetryRequestId)
? run.taskRetryRequestId : run.retryOfRunId ? run.wakeupRequestId : null;
const retryWakeIds = [...new Set(runs.flatMap(run => { const id = retryWakeId(run); return id ? [id] : []; }))];
const retryWakesQuery = retryWakeIds.length
? db.select({
id: agentWakeupRequests.id,
reason: agentWakeupRequests.reason,
requestedByActorType: agentWakeupRequests.requestedByActorType,
requestedByActorId: agentWakeupRequests.requestedByActorId,
}).from(agentWakeupRequests)
.where(and(eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.id, retryWakeIds)))
: Promise.resolve([]);
const [exhaustionRows, leaseRows, executionByRunId, [savedPlan], retryWakes] = await Promise.all([
exhaustionRowsQuery,
leaseRowsQuery,
executionProjectionsForRuns(db, companyId, runIds, new Date(), { retryDatabaseReads: true }),
savedPlanQuery,
retryWakesQuery,
]);
const retryWakeById = new Map(retryWakes.map(wake => [wake.id, wake]));
const retryPersonIds = [...new Set(retryWakes.flatMap(wake =>
wake.requestedByActorType === "user" && wake.requestedByActorId && wake.requestedByActorId !== "local-board" ? [wake.requestedByActorId] : []))];
const retryPersonNames = new Map((retryPersonIds.length
? await db.select({ id: authUsers.id, name: authUsers.name }).from(authUsers).where(inArray(authUsers.id, retryPersonIds))
: []).filter(row => row.name?.trim()).map(row => [row.id, row.name!.trim()]));
const retriedBy = (run: (typeof runs)[number]) => {
const wakeId = retryWakeId(run);
const wake = wakeId ? retryWakeById.get(wakeId) ?? null : null;
const actorId = wake?.requestedByActorId ?? null;
const personName = wake?.requestedByActorType === "user" && actorId
? retryPersonNames.get(actorId) ?? activityActorFallbackName("user", actorId) : null;
return describeRunRetry(run, wake, personName);
};
const retryExhaustedReasonByRunId = new Map<string, string>();
for (const row of exhaustionRows) {
if (!row.message || retryExhaustedReasonByRunId.has(row.runId)) continue;
retryExhaustedReasonByRunId.set(row.runId, row.message);
}
const leaseByRunId = new Map<string, (typeof leaseRows)[number]>();
for (const row of leaseRows) {
if (row.lease.heartbeatRunId && !leaseByRunId.has(row.lease.heartbeatRunId)) {
leaseByRunId.set(row.lease.heartbeatRunId, row);
}
}
return runs.map((run) => {
const leaseRow = leaseByRunId.get(run.runId);
const leaseMetadata = leaseRow?.lease.metadata ?? null;
const workspacePath =
typeof leaseMetadata?.remoteCwd === "string" && leaseMetadata.remoteCwd.trim().length > 0
? leaseMetadata.remoteCwd
: typeof leaseMetadata?.remoteWorkspacePath === "string" && leaseMetadata.remoteWorkspacePath.trim().length > 0
? leaseMetadata.remoteWorkspacePath
: null;
return {
...run,
failureCause: normalizeRunFailureCause(run.failureCause),
resultJson: run.resultJson ? {
...run.resultJson,
...(Object.hasOwn(run.resultJson, "workspaceRestorePath") ? {
workspaceRestorePath: safeWorkspaceRestorePath(run.resultJson.workspaceRestorePath),
} : {}),
...(hasWorkspaceRestoreFailure(run.resultJson) ? {
...(savedPlan?.runId === run.runId ? { savedPlanRevisionId: savedPlan.revisionId } : {}),
} : {}),
} : null,
execution: executionByRunId.get(run.runId) ?? null,
environment: leaseRow
? {
id: leaseRow.environment.id,
name: leaseRow.environment.name,
driver: leaseRow.environment.driver,
}
: null,
environmentLease: leaseRow
? {
id: leaseRow.lease.id,
status: leaseRow.lease.status,
leasePolicy: leaseRow.lease.leasePolicy,
provider: leaseRow.lease.provider,
providerLeaseId: leaseRow.lease.providerLeaseId,
executionWorkspaceId: leaseRow.lease.executionWorkspaceId,
workspacePath,
failureReason: leaseRow.lease.failureReason,
cleanupStatus: leaseRow.lease.cleanupStatus,
acquiredAt: leaseRow.lease.acquiredAt,
releasedAt: leaseRow.lease.releasedAt,
}
: null,
retryExhaustedReason: retryExhaustedReasonByRunId.get(run.runId) ?? null,
retriedBy: retriedBy(run),
};
});
},
issuesForRun: async (runId: string, readCondition: SQL<boolean>) => {
const run = await db
.select({
companyId: heartbeatRuns.companyId,
issueId: heartbeatRuns.issueId,
})
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.then((rows) => rows[0] ?? null);
if (!run) return [];
const fromActivity = await db
.selectDistinctOn([issueIdAsText], {
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(activityLog)
.innerJoin(issues, eq(activityLog.entityId, issueIdAsText))
.where(
and(
eq(activityLog.companyId, run.companyId),
eq(activityLog.runId, runId),
eq(activityLog.entityType, "issue"),
eq(issues.companyId, run.companyId),
visibleIssueCondition(),
readCondition,
),
)
.orderBy(issueIdAsText);
const contextIssueId = run.issueId;
if (!contextIssueId) return fromActivity;
if (fromActivity.some((issue) => issue.issueId === contextIssueId)) return fromActivity;
const fromContext = await db
.select({
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(issues)
.where(
and(
eq(issues.companyId, run.companyId),
eq(issues.id, contextIssueId),
visibleIssueCondition(),
readCondition,
),
)
.then((rows) => rows[0] ?? null);
if (!fromContext) return fromActivity;
return [fromContext, ...fromActivity];
},
create: (data: typeof activityLog.$inferInsert) => db.transaction(async tx => {
if (data.entityType === "issue") await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, `task:${data.entityId}`);
const [row] = await tx.insert(activityLog).values(data).returning();
return row;
}),
};
}