Files
PaperClipAI/server/src/services/activity.ts
T
DottaandPaperclip c6ee96a3df fix: stop idle MCP event polling (#17036)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP task subscriptions and Dot mailboxes notify external clients
when work changes.
> - Their delivery workers queried SQL every two seconds, including when
no work existed.
> - Those queries prevent idle databases from becoming quiet across many
hosted instances.
> - Existing activity, mailbox, subscription, and delivery records
already preserve the work.
> - This pull request uses committed-write signals and saved retry
deadlines to drive delivery.
> - Idle subscriptions become quiet while startup, retry, drain, and
shutdown recovery remain supported.

## Linked Issues or Issue Description

Related sleep migrations: #15625, #17000, #17008. Searched open PRs and
issues for MCP event sleep/polling work; no duplicate found.

**What happened?**

Each public MCP and Dot event service started a two-second interval. It
checked settings, scanned subscriptions, created delivery receipts,
cleaned expired records, and attempted delivery even without changes.

**Expected behavior**

An empty or subscribed-but-idle service produces no recurring SQL or
network traffic. Committed events trigger prompt delivery. Only
unfinished work retains a retry deadline.

**Steps to reproduce**

Configure a public MCP origin and start the server. Observe recurring
settings queries with MCP disabled, or subscription/queue queries with
it enabled. No incoming task changes are required.

**Paperclip version or commit**

Initially inspected master at
`8b4ab65b1c3d863d282752e83307943720a5c85c`; rebased onto `c2daf906d9`
after master advanced. The only conflict was overlapping imports in the
heartbeat queue; both were preserved.

## What Changed

- Register separate MCP and Dot workers with the existing delivery
coordinator. Remove both unconditional intervals.
- Notify from committed activity, comment, mailbox, subscription, and
experimental-settings writes. Centralize comment insertion in one small
helper so normal callers do not manage recovery state.
- Preserve outer transaction ownership, uncertainty reconciliation, and
requests deferred during drain or standby.
- Restore delivery retry/abandoned-claim deadlines on startup. Scan
subscriptions in pages of 20; drain full source and delivery batches
without another external event. Scope subscription refreshes to their
subscription or shared Dot binding, issue activity to its task, comment
discovery to its company, and mailbox/assignment changes to their
binding. Retry deadlines do not rescan unchanged sources.
- Run expired-record cleanup opportunistically. Preserve on-use expiry
checks and the existing one-time automatic Dot challenge behavior.
- Add real PostgreSQL lifecycle coverage and guarded external test
database options for existing MCP/Dot integrations. Document producers
and the supported writer model.

## Verification

- Final head `e54fb34485921016ef988f7df9cca173a9ebdfc8`: all CI checks
passed (55 successful checks/statuses, one neutral review, two
intentionally skipped Storybook checks). Mergeable with no unresolved
review threads. Greptile: 5/5 on this exact commit; security scan
passed.
- All 284 affected tests passed in one local run after rebasing: 16
real-PostgreSQL lifecycle tests, 24 coordinator/scheduler/standby tests,
84 public MCP tests, 54 Dot runner tests, and 106 interaction-service
tests.
- Server TypeScript passed after refreshing generated Runner/plugin SDK
dependencies from master. Full workspace typecheck and build passed
before the rebase; final CI repeated typecheck and build successfully on
the rebased head.
- Coverage includes SQL/network quiet with configuration and history,
committed writes, outer commit/rollback, drain/standby, retry
restoration, reenable/unsubscribe, concurrent writes, shutdown,
source/delivery/subscription batch boundaries, Dot challenges and
reconnects, native broker fences, comment discovery, and
company-isolated scans.
- CI exposed an existing lock-order test whose insert proxy missed
nested transactions and waited indefinitely. The fixture now observes
nested inserts, preserves the lock assertion, and fails explicitly if
interception is missed. The complete 106-test interaction suite and the
formerly hanging CI shard pass. The warm-standby/settings test doubles
were also updated for the new coordinator/transaction boundaries.
- Earlier fixture failures (required timestamps, cleanup, external
database selection, async waits) were corrected. One existing OAuth
first-poll timing assertion and one E2E company-cleanup race passed on
single reruns; the final full CI run passed without a rerun. No OAuth or
company-deletion behavior was changed.
- The local full `pnpm test:run` attempt could not initialize embedded
PostgreSQL after five attempts. It was stopped; affected integrations
were verified with separate disposable PostgreSQL 17 databases, and
final CI supplied the full-suite result. Embedded-only local skips are
not counted as passes.

## Risks

- Notifications use one live server/database scope. Independent SQL
writers and separate live replicas do not trigger these process-local
workers. Startup recovers durable state; continuous multi-writer
discovery needs a notification/ingress contract.
- Idle physical cleanup waits for startup or another enabled sweep.
Subscription and authorization expiry remain enforced on use.
- Existing delivery claims, retry bounds, signatures, and receipt
deduplication remain in place. No new tables or migrations.
- Host sleep-safety checks remain conservative. MCP/Dot retry deadlines
are not yet covered by Paperclip Cloud's scheduled-wake contract. This
proves worker quiet, not deployed host suspension or fleet savings.

## Model Used

OpenAI Codex (GPT-6), with reasoning, local code execution, and tool
use. The runtime does not expose a more specific model version or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:55:09 -05:00

660 lines
27 KiB
TypeScript

import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { skillAccessService } from "./skill-access.js";
import { credentialAccessConnectionNameSql } from "./credential-access-visibility.js";
import { executionProjectionsForRuns } from "./execution-projection.js";
import { and, asc, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
activityLog,
agents,
documentRevisions,
documents,
environmentLeases,
environments,
heartbeatRunEvents,
heartbeatRuns,
issueComments,
issueDocuments,
issues,
issueWorkProducts,
workspaceOperations,
} from "@paperclipai/db";
import { hasWorkspaceRestoreFailure, safeWorkspaceRestorePath, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY } from "@paperclipai/shared";
import { logger } from "../middleware/logger.js";
import { visibleIssueCondition } from "./issue-visibility.js";
import { classifyRunLiveness } from "./run-liveness.js";
export interface ActivityFilters {
companyId: string;
agentId?: string;
entityType?: string;
entityId?: string;
limit?: number;
readCondition?: SQL<boolean>;
caseReadCondition?: SQL<boolean>;
}
const DEFAULT_ACTIVITY_LIMIT = 100;
const MAX_ACTIVITY_LIMIT = 500;
export function normalizeActivityLimit(limit: number | undefined) {
if (!Number.isFinite(limit)) return DEFAULT_ACTIVITY_LIMIT;
return Math.max(1, Math.min(MAX_ACTIVITY_LIMIT, Math.floor(limit ?? DEFAULT_ACTIVITY_LIMIT)));
}
export function activityService(db: Db) {
const scheduledLivenessBackfills = new Set<string>();
const issueIdAsText = sql<string>`${issues.id}::text`;
const summarizedUsageJson = sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.usageJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'inputTokens', coalesce(${heartbeatRuns.usageJson} -> 'inputTokens', ${heartbeatRuns.usageJson} -> 'input_tokens'),
'input_tokens', coalesce(${heartbeatRuns.usageJson} -> 'input_tokens', ${heartbeatRuns.usageJson} -> 'inputTokens'),
'outputTokens', coalesce(${heartbeatRuns.usageJson} -> 'outputTokens', ${heartbeatRuns.usageJson} -> 'output_tokens'),
'output_tokens', coalesce(${heartbeatRuns.usageJson} -> 'output_tokens', ${heartbeatRuns.usageJson} -> 'outputTokens'),
'cachedInputTokens', coalesce(
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cached_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
),
'cache_read_input_tokens', coalesce(
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens',
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
${heartbeatRuns.usageJson} -> 'cachedInputTokens'
),
'billingType', coalesce(${heartbeatRuns.usageJson} -> 'billingType', ${heartbeatRuns.usageJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.usageJson} -> 'billing_type', ${heartbeatRuns.usageJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd',
${heartbeatRuns.usageJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.usageJson} -> 'total_cost_usd',
${heartbeatRuns.usageJson} -> 'cost_usd',
${heartbeatRuns.usageJson} -> 'costUsd'
)
))
end
`.as("usageJson");
// A denied user needs the selected connection's name to repair their task.
// Other readers need the connection's existing human sharing permission.
const summarizedResultJson = (viewerUserId: string | null) => sql<Record<string, unknown> | null>`
case
when ${heartbeatRuns.resultJson} is null then null
else jsonb_strip_nulls(jsonb_build_object(
'conversationReset', ${heartbeatRuns.resultJson} -> 'conversationReset',
'configurationIncomplete', case
when ${heartbeatRuns.resultJson} -> 'configurationIncomplete' ->> 'reason' = 'workspace_base_ref_unresolved'
then jsonb_build_object('reason', 'workspace_base_ref_unresolved')
when ${heartbeatRuns.errorCode} = 'configuration_incomplete'
and (${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
or ${heartbeatRuns.error} = 'This credential is not shared with the responsible user')
then jsonb_build_object(
'selectionFailure', case when ${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
then 'ai_connection_credential_not_shared' end,
'credentialAccess', jsonb_build_object(
'connectionName', ${credentialAccessConnectionNameSql(viewerUserId)}
)
) end,
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
in ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
'workspaceRestorePath', case when length(${heartbeatRuns.resultJson} ->> 'workspaceRestorePath') <= 180
then ${heartbeatRuns.resultJson} -> 'workspaceRestorePath' end,
'finalResponseRecorded', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'finalResponseRecorded') = 'boolean'
then ${heartbeatRuns.resultJson} -> 'finalResponseRecorded' end,
'billingType', coalesce(${heartbeatRuns.resultJson} -> 'billingType', ${heartbeatRuns.resultJson} -> 'billing_type'),
'billing_type', coalesce(${heartbeatRuns.resultJson} -> 'billing_type', ${heartbeatRuns.resultJson} -> 'billingType'),
'costUsd', coalesce(
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd',
${heartbeatRuns.resultJson} -> 'total_cost_usd'
),
'total_cost_usd', coalesce(
${heartbeatRuns.resultJson} -> 'total_cost_usd',
${heartbeatRuns.resultJson} -> 'cost_usd',
${heartbeatRuns.resultJson} -> 'costUsd'
),
'stopReason', ${heartbeatRuns.resultJson} -> 'stopReason',
'effectiveTimeoutSec', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutSec',
'effectiveTimeoutMs', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutMs',
'timeoutConfigured', ${heartbeatRuns.resultJson} -> 'timeoutConfigured',
'timeoutSource', ${heartbeatRuns.resultJson} -> 'timeoutSource',
'timeoutFired', ${heartbeatRuns.resultJson} -> 'timeoutFired'
))
end
`.as("resultJson");
function countValue(value: unknown) {
const parsed = Number(value ?? 0);
return Number.isFinite(parsed) ? Math.max(0, Math.floor(parsed)) : 0;
}
function dateValue(value: unknown) {
if (value instanceof Date) return Number.isNaN(value.getTime()) ? null : value;
if (typeof value === "string" || typeof value === "number") {
const parsed = new Date(value);
return Number.isNaN(parsed.getTime()) ? null : parsed;
}
return null;
}
function latestDate(...values: unknown[]) {
let latest: Date | null = null;
for (const value of values) {
const parsed = dateValue(value);
if (!parsed) continue;
if (!latest || parsed.getTime() > latest.getTime()) latest = parsed;
}
return latest;
}
function asRecord(value: unknown): Record<string, unknown> | null {
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
return value as Record<string, unknown>;
}
function readNumber(value: unknown) {
return typeof value === "number" && Number.isFinite(value) ? value : null;
}
async function backfillMissingRunLivenessForIssue(companyId: string, issueId: string) {
const runs = await db
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
status: heartbeatRuns.status,
contextSnapshot: heartbeatRuns.contextSnapshot,
resultJson: heartbeatRuns.resultJson,
stdoutExcerpt: heartbeatRuns.stdoutExcerpt,
stderrExcerpt: heartbeatRuns.stderrExcerpt,
error: heartbeatRuns.error,
errorCode: heartbeatRuns.errorCode,
continuationAttempt: heartbeatRuns.continuationAttempt,
})
.from(heartbeatRuns)
.where(
and(
eq(heartbeatRuns.companyId, companyId),
isNull(heartbeatRuns.livenessState),
sql`${heartbeatRuns.status} not in ('queued', 'running')`,
or(
sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${issueId}`,
sql`exists (
select 1
from ${activityLog}
where ${activityLog.companyId} = ${companyId}
and ${activityLog.entityType} = 'issue'
and ${activityLog.entityId} = ${issueId}
and ${activityLog.runId} = ${heartbeatRuns.id}
)`,
),
),
)
.limit(20);
if (runs.length === 0) return;
const issue = await db
.select({
status: issues.status,
title: issues.title,
description: issues.description,
workMode: issues.workMode,
})
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
.then((rows) => rows[0] ?? null);
for (const run of runs) {
const context = asRecord(run.contextSnapshot);
const continuationAttempt =
readNumber(context?.continuationAttempt) ??
readNumber(context?.livenessContinuationAttempt) ??
run.continuationAttempt ??
0;
const [commentStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueComments.createdAt})`,
})
.from(issueComments)
.where(
and(
eq(issueComments.companyId, companyId),
eq(issueComments.issueId, issueId),
eq(issueComments.createdByRunId, run.id),
),
);
const [documentStats] = await db
.select({
count: sql<number>`count(*)::int`,
planCount: sql<number>`count(*) filter (where ${issueDocuments.key} = 'plan')::int`,
latestAt: sql<Date | null>`max(${documentRevisions.createdAt})`,
})
.from(documentRevisions)
.innerJoin(issueDocuments, eq(documentRevisions.documentId, issueDocuments.documentId))
.where(
and(
eq(documentRevisions.companyId, companyId),
eq(documentRevisions.createdByRunId, run.id),
eq(issueDocuments.companyId, companyId),
eq(issueDocuments.issueId, issueId),
sql`${issueDocuments.key} != ${ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY}`,
),
);
const [workProductStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${issueWorkProducts.createdAt})`,
})
.from(issueWorkProducts)
.where(
and(
eq(issueWorkProducts.companyId, companyId),
eq(issueWorkProducts.issueId, issueId),
eq(issueWorkProducts.createdByRunId, run.id),
),
);
const [workspaceOperationStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${workspaceOperations.startedAt})`,
})
.from(workspaceOperations)
.where(and(eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.heartbeatRunId, run.id)));
const [activityStats] = await db
.select({
count: sql<number>`count(*)::int`,
latestAt: sql<Date | null>`max(${activityLog.createdAt})`,
})
.from(activityLog)
.where(and(eq(activityLog.companyId, companyId), eq(activityLog.runId, run.id)));
const [eventStats] = await db
.select({
count: sql<number>`count(*) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))::int`,
latestAt: sql<Date | null>`max(${heartbeatRunEvents.createdAt}) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))`,
})
.from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, companyId), eq(heartbeatRunEvents.runId, run.id)));
const classification = classifyRunLiveness({
runStatus: run.status,
issue,
resultJson: asRecord(run.resultJson),
stdoutExcerpt: run.stdoutExcerpt,
stderrExcerpt: run.stderrExcerpt,
error: run.error,
errorCode: run.errorCode,
continuationAttempt,
evidence: {
issueCommentsCreated: countValue(commentStats?.count),
documentRevisionsCreated: countValue(documentStats?.count),
planDocumentRevisionsCreated: countValue(documentStats?.planCount),
workProductsCreated: countValue(workProductStats?.count),
workspaceOperationsCreated: countValue(workspaceOperationStats?.count),
activityEventsCreated: countValue(activityStats?.count),
toolOrActionEventsCreated: countValue(eventStats?.count),
latestEvidenceAt: latestDate(
commentStats?.latestAt,
documentStats?.latestAt,
workProductStats?.latestAt,
workspaceOperationStats?.latestAt,
activityStats?.latestAt,
eventStats?.latestAt,
),
},
});
await db
.update(heartbeatRuns)
.set({
livenessState: classification.livenessState,
livenessReason: classification.livenessReason,
continuationAttempt: classification.continuationAttempt,
lastUsefulActionAt: classification.lastUsefulActionAt,
nextAction: classification.nextAction,
updatedAt: new Date(),
})
.where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.livenessState)));
}
}
function scheduleRunLivenessBackfill(companyId: string, issueId: string) {
const key = `${companyId}:${issueId}`;
if (scheduledLivenessBackfills.has(key)) return;
scheduledLivenessBackfills.add(key);
void backfillMissingRunLivenessForIssue(companyId, issueId)
.catch((err: unknown) => {
logger.warn({ err, companyId, issueId }, "run liveness backfill failed");
})
.finally(() => {
scheduledLivenessBackfills.delete(key);
});
}
return {
list: async (filters: ActivityFilters) => {
const conditions = [eq(activityLog.companyId, filters.companyId), await skillAccessService(db).activityCondition(filters.companyId),
filters.caseReadCondition ?? sql<boolean>`${activityLog.entityType} <> 'case'`];
const limit = normalizeActivityLimit(filters.limit);
if (filters.agentId) {
conditions.push(eq(activityLog.agentId, filters.agentId));
}
if (filters.entityType) {
conditions.push(eq(activityLog.entityType, filters.entityType));
}
if (filters.entityId) {
conditions.push(eq(activityLog.entityId, filters.entityId));
}
return db
.select({ activityLog })
.from(activityLog)
.leftJoin(
issues,
and(
eq(activityLog.entityType, sql`'issue'`),
eq(activityLog.entityId, issueIdAsText),
),
)
.where(
and(
...conditions,
or(
sql`${activityLog.entityType} != 'issue'`,
and(visibleIssueCondition(), filters.readCondition ?? sql<boolean>`true`),
),
),
)
.orderBy(desc(activityLog.createdAt))
.limit(limit)
.then((rows) => rows.map((r) => r.activityLog));
},
forIssue: (issueId: string) =>
db
.select()
.from(activityLog)
.where(
or(
and(eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issueId)),
and(or(eq(activityLog.action, "project.created"), eq(activityLog.action, "company.skill_created")), sql`${activityLog.details}->>'sourceIssueId' = ${issueId}`,
sql`${activityLog.companyId} = (select company_id from issues where id = ${issueId})`),
),
)
.orderBy(desc(activityLog.createdAt)),
runsForIssue: async (companyId: string, issueId: string, viewerUserId: string | null = null) => {
scheduleRunLivenessBackfill(companyId, issueId);
const runs = await db
.select({
runId: heartbeatRuns.id,
runtimeMode: heartbeatRuns.runtimeMode,
wakeupRequestId: heartbeatRuns.wakeupRequestId,
status: heartbeatRuns.status,
agentId: heartbeatRuns.agentId,
adapterType: agents.adapterType,
startedAt: heartbeatRuns.startedAt,
finishedAt: heartbeatRuns.finishedAt,
createdAt: heartbeatRuns.createdAt,
invocationSource: heartbeatRuns.invocationSource,
responsibleUserId: heartbeatRuns.responsibleUserId,
errorCode: heartbeatRuns.errorCode,
error: sql<string | null>`case when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.errorCode} = 'native_provider_model_rejected'
then left(${heartbeatRuns.error}, 2000)
when ${heartbeatRuns.status} = 'failed'
and ${heartbeatRuns.error} = 'This credential is not shared with the responsible user'
then 'This credential is not shared with the responsible user' else null end`,
usageJson: summarizedUsageJson,
resultJson: summarizedResultJson(viewerUserId),
logBytes: heartbeatRuns.logBytes,
retryOfRunId: heartbeatRuns.retryOfRunId,
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
livenessState: heartbeatRuns.livenessState,
livenessReason: heartbeatRuns.livenessReason,
continuationAttempt: heartbeatRuns.continuationAttempt,
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
nextAction: heartbeatRuns.nextAction,
wakeCommentIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'wakeCommentIds'`,
queuedCommentSourceWakeIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'queuedCommentSourceWakeIds'`,
wakeCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'wakeCommentId'`,
contextCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'commentId'`,
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`,
})
.from(heartbeatRuns)
.innerJoin(
agents,
and(
eq(agents.id, heartbeatRuns.agentId),
eq(agents.companyId, heartbeatRuns.companyId),
),
)
.where(
and(
eq(heartbeatRuns.companyId, companyId),
or(
eq(heartbeatRuns.issueId, issueId),
sql`exists (
select 1
from ${activityLog}
where ${activityLog.companyId} = ${companyId}
and ${activityLog.entityType} = 'issue'
and ${activityLog.entityId} = ${issueId}
and ${activityLog.runId} = ${heartbeatRuns.id}
)`,
),
),
)
.orderBy(desc(heartbeatRuns.createdAt));
if (runs.length === 0) return runs;
const runIds = runs.map((run) => run.runId);
if (runIds.length === 0) return runs;
const exhaustionRowsQuery = db
.select({
runId: heartbeatRunEvents.runId,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
),
)
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id));
const leaseRowsQuery = db
.select({
lease: environmentLeases,
environment: {
id: environments.id,
name: environments.name,
driver: environments.driver,
},
})
.from(environmentLeases)
.innerJoin(environments, eq(environmentLeases.environmentId, environments.id))
.where(
and(
eq(environmentLeases.companyId, companyId),
inArray(environmentLeases.heartbeatRunId, runIds),
),
)
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
// Only stored, current plan revisions can support a saved-plan link.
// Do not trust an adapter's claim that it wrote a document.
const savedPlanQuery = runs.some((run) => hasWorkspaceRestoreFailure(run.resultJson))
? db.select({ revisionId: documentRevisions.id, runId: documentRevisions.createdByRunId })
.from(issueDocuments)
.innerJoin(documents, and(eq(documents.id, issueDocuments.documentId), eq(documents.companyId, companyId)))
.innerJoin(documentRevisions, and(eq(documentRevisions.id, documents.latestRevisionId), eq(documentRevisions.documentId, documents.id), eq(documentRevisions.companyId, companyId)))
.where(and(eq(issueDocuments.companyId, companyId), eq(issueDocuments.issueId, issueId), eq(issueDocuments.key, "plan")))
.limit(1)
: Promise.resolve([]);
const [exhaustionRows, leaseRows, executionByRunId, [savedPlan]] = await Promise.all([
exhaustionRowsQuery,
leaseRowsQuery,
executionProjectionsForRuns(db, companyId, runIds, new Date(), { retryDatabaseReads: true }),
savedPlanQuery,
]);
const retryExhaustedReasonByRunId = new Map<string, string>();
for (const row of exhaustionRows) {
if (!row.message || retryExhaustedReasonByRunId.has(row.runId)) continue;
retryExhaustedReasonByRunId.set(row.runId, row.message);
}
const leaseByRunId = new Map<string, (typeof leaseRows)[number]>();
for (const row of leaseRows) {
if (row.lease.heartbeatRunId && !leaseByRunId.has(row.lease.heartbeatRunId)) {
leaseByRunId.set(row.lease.heartbeatRunId, row);
}
}
return runs.map((run) => {
const leaseRow = leaseByRunId.get(run.runId);
const leaseMetadata = leaseRow?.lease.metadata ?? null;
const workspacePath =
typeof leaseMetadata?.remoteCwd === "string" && leaseMetadata.remoteCwd.trim().length > 0
? leaseMetadata.remoteCwd
: typeof leaseMetadata?.remoteWorkspacePath === "string" && leaseMetadata.remoteWorkspacePath.trim().length > 0
? leaseMetadata.remoteWorkspacePath
: null;
return {
...run,
resultJson: run.resultJson ? {
...run.resultJson,
...(Object.hasOwn(run.resultJson, "workspaceRestorePath") ? {
workspaceRestorePath: safeWorkspaceRestorePath(run.resultJson.workspaceRestorePath),
} : {}),
...(hasWorkspaceRestoreFailure(run.resultJson) ? {
...(savedPlan?.runId === run.runId ? { savedPlanRevisionId: savedPlan.revisionId } : {}),
} : {}),
} : null,
execution: executionByRunId.get(run.runId) ?? null,
environment: leaseRow
? {
id: leaseRow.environment.id,
name: leaseRow.environment.name,
driver: leaseRow.environment.driver,
}
: null,
environmentLease: leaseRow
? {
id: leaseRow.lease.id,
status: leaseRow.lease.status,
leasePolicy: leaseRow.lease.leasePolicy,
provider: leaseRow.lease.provider,
providerLeaseId: leaseRow.lease.providerLeaseId,
executionWorkspaceId: leaseRow.lease.executionWorkspaceId,
workspacePath,
failureReason: leaseRow.lease.failureReason,
cleanupStatus: leaseRow.lease.cleanupStatus,
acquiredAt: leaseRow.lease.acquiredAt,
releasedAt: leaseRow.lease.releasedAt,
}
: null,
retryExhaustedReason: retryExhaustedReasonByRunId.get(run.runId) ?? null,
};
});
},
issuesForRun: async (runId: string, readCondition: SQL<boolean>) => {
const run = await db
.select({
companyId: heartbeatRuns.companyId,
issueId: heartbeatRuns.issueId,
})
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId))
.then((rows) => rows[0] ?? null);
if (!run) return [];
const fromActivity = await db
.selectDistinctOn([issueIdAsText], {
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(activityLog)
.innerJoin(issues, eq(activityLog.entityId, issueIdAsText))
.where(
and(
eq(activityLog.companyId, run.companyId),
eq(activityLog.runId, runId),
eq(activityLog.entityType, "issue"),
eq(issues.companyId, run.companyId),
visibleIssueCondition(),
readCondition,
),
)
.orderBy(issueIdAsText);
const contextIssueId = run.issueId;
if (!contextIssueId) return fromActivity;
if (fromActivity.some((issue) => issue.issueId === contextIssueId)) return fromActivity;
const fromContext = await db
.select({
issueId: issues.id,
identifier: issues.identifier,
title: issues.title,
status: issues.status,
priority: issues.priority,
})
.from(issues)
.where(
and(
eq(issues.companyId, run.companyId),
eq(issues.id, contextIssueId),
visibleIssueCondition(),
readCondition,
),
)
.then((rows) => rows[0] ?? null);
if (!fromContext) return fromActivity;
return [fromContext, ...fromActivity];
},
create: (data: typeof activityLog.$inferInsert) => db.transaction(async tx => {
if (data.entityType === "issue") await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, `task:${data.entityId}`);
const [row] = await tx.insert(activityLog).values(data).returning();
return row;
}),
};
}