mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat agent answers questions about failed runs by reading the
task's run list (`GET /api/issues/{id}/runs`)
> - For a retry run, that read returned only raw invocation fields
(`on_demand`, `manual`) and an owner id. It did not say who started the
retry
> - In chat-eval cohort 13, the agent told the person "Paperclip retried
automatically" about a run that the board started with Retry
> - This pull request adds a plain-words `retriedBy` field to each retry
run, and tells the chat agent to use it
> - The benefit is that people get a true account of who retried their
work, with a name and no raw ids
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 ran OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017` on master `df1b12ad8e`.
In `runs-adapters.inspect-failed-run`, the reply said "Paperclip retried
automatically on a supported model". The successful run was a person's
manual Retry: `invocationSource: on_demand`, `triggerDetail: manual`,
wake reason `retry_failed_run`, and the responsible user was the board.
The agent's own reasoning hedged ("probably automatic"), because the run
read did not say who started the retry.
**Expected behavior**
The run reads that the agent uses say who started each retry: a person,
by display name, or Paperclip's automatic recovery, in plain words and
with no raw ids.
**Steps to reproduce**
1. Let a run fail on a task.
2. Press Retry on the failed run as a board user.
3. Read `GET /api/issues/{id}/runs`. Before this change, nothing in the
retry row says that a person pressed Retry.
## What Changed
- `runsForIssue` (`server/src/services/activity.ts`) reads the wake
request of each retry run. It adds `retriedBy` to each row:
- `{ kind: "person", personName, summary }` when a user wake with reason
`retry_failed_run` or `task_retry` started the run. The name comes from
the user record, or "Board" for the local board, with the same fallback
as activity actor names.
- `{ kind: "automatic", summary }` for an `automation` retry run.
- `null` for a run that is not a retry, or a retry whose starter is not
recorded. The read does not guess. It does not use `responsibleUserId`,
because automatic retries keep the original responsible user.
- The OpenAPI description of `GET /api/issues/{id}/runs` and the chat
agent directive tell the agent to use `retriedBy` and not to call a
retry automatic unless `retriedBy` says so.
- The UI `RunForIssue` type has the new optional field.
## Verification
- `cd server && npx vitest run src/__tests__/activity-service.test.ts
src/__tests__/agent-conversations.test.ts
src/__tests__/activity-routes.test.ts
src/__tests__/openapi-routes.test.ts` passes.
- Red/green: the new test "says who started each retry run" fails
without the service change (`expected undefined to be null`) and passes
with it. It covers a board Retry, a named member's Retry, an automatic
retry that keeps the board as responsible user, a retry with no recorded
wake, and no raw ids in the output.
- `npx tsc --noEmit -p server` passes.
- A narrow paid run of `runs-adapters.inspect-failed-run` (with the two
sibling cohort-13 PRs) ran on a combined branch. The results are in a PR
comment.
## Risks
- Low risk. The change adds one read-only field and one bounded query by
primary key on the wake requests of retry runs in the page.
- No persisted state changes.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
802 lines
37 KiB
TypeScript
802 lines
37 KiB
TypeScript
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { skillAccessService } from "./skill-access.js";
|
|
import { credentialAccessConnectionNameSql } from "./credential-access-visibility.js";
|
|
import { executionProjectionsForRuns } from "./execution-projection.js";
|
|
import { and, asc, desc, eq, inArray, isNull, or, sql, type SQL } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
activityLog,
|
|
agentWakeupRequests,
|
|
agents,
|
|
authUsers,
|
|
documentRevisions,
|
|
documents,
|
|
environmentLeases,
|
|
environments,
|
|
heartbeatRunEvents,
|
|
heartbeatRuns,
|
|
workspaceRestoreFailureCodesSql,
|
|
issueComments,
|
|
issueDocuments,
|
|
issues,
|
|
issueWorkProducts,
|
|
workspaceOperations,
|
|
} from "@paperclipai/db";
|
|
import { hasWorkspaceRestoreFailure, safeWorkspaceRestorePath, ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY, normalizeRunFailureCause, RUN_FAILURE_CAUSE_MAX_LENGTH } from "@paperclipai/shared";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { visibleIssueCondition } from "./issue-visibility.js";
|
|
import { classifyRunLiveness } from "./run-liveness.js";
|
|
import { missingRunLivenessCandidatesQuery } from "./run-liveness-backfill-query.js";
|
|
|
|
export interface ActivityFilters {
|
|
companyId: string;
|
|
agentId?: string;
|
|
entityType?: string;
|
|
entityId?: string;
|
|
limit?: number;
|
|
readCondition?: SQL<boolean>;
|
|
caseReadCondition?: SQL<boolean>;
|
|
}
|
|
|
|
const DEFAULT_ACTIVITY_LIMIT = 100;
|
|
const MAX_ACTIVITY_LIMIT = 500;
|
|
|
|
export function normalizeActivityLimit(limit: number | undefined) {
|
|
if (!Number.isFinite(limit)) return DEFAULT_ACTIVITY_LIMIT;
|
|
return Math.max(1, Math.min(MAX_ACTIVITY_LIMIT, Math.floor(limit ?? DEFAULT_ACTIVITY_LIMIT)));
|
|
}
|
|
|
|
const RUN_LIVENESS_BACKFILL_BATCH = 20;
|
|
// Runs finalize with a liveness classification. The backfill only repairs rows
|
|
// that missed it (older rows or a failed finalization), so a delay is safe.
|
|
const RUN_LIVENESS_CLEAN_TTL_MS = 10 * 60_000;
|
|
const RUN_LIVENESS_CLEAN_MAX_ENTRIES = 10_000;
|
|
|
|
const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
|
|
|
/** A person-facing name for an activity actor, so readers never see a raw id such as local-board. */
|
|
export function activityActorFallbackName(actorType: string, actorId: string) {
|
|
if (actorType === "user") return actorId === "local-board" ? "Board" : "A board member";
|
|
if (actorType === "agent") return "An agent";
|
|
if (actorType === "plugin") return "A plugin";
|
|
return "Paperclip";
|
|
}
|
|
|
|
/** Wake reasons recorded when a person presses Retry on a failed run or task. */
|
|
const PERSON_RETRY_WAKE_REASONS = new Set(["retry_failed_run", "task_retry"]);
|
|
|
|
export type RunRetriedBy = { kind: "person"; personName: string; summary: string } | { kind: "automatic"; summary: string };
|
|
|
|
/**
|
|
* Who started a retry run, in plain words. A run's own invocation fields do
|
|
* not say this: a person's Retry is an on-demand run whose responsible user
|
|
* may also own automatic retries. The run's wake request records whether a
|
|
* person pressed Retry; automatic recovery creates an automation run. Returns
|
|
* null for a run that is not a retry or whose starter is not recorded, rather
|
|
* than guessing.
|
|
*/
|
|
export function describeRunRetry(
|
|
run: { retryOfRunId: string | null; taskRetryRequestId?: string | null; invocationSource: string | null },
|
|
wake: { reason: string | null; requestedByActorType: string | null } | null,
|
|
personName: string | null,
|
|
): RunRetriedBy | null {
|
|
if (!run.retryOfRunId && !run.taskRetryRequestId) return null;
|
|
// The run's own start decides first: automatic recovery after a person's
|
|
// Task Retry inherits that Retry's request, but the person did not start it.
|
|
if (run.invocationSource === "automation") {
|
|
return { kind: "automatic", summary: "Paperclip retried the failed run automatically; no person pressed Retry." };
|
|
}
|
|
if (wake?.requestedByActorType === "user" && wake.reason && PERSON_RETRY_WAKE_REASONS.has(wake.reason)) {
|
|
const name = personName?.trim() || "A board member";
|
|
return { kind: "person", personName: name, summary: `${name} pressed Retry on the failed run. Paperclip did not retry it automatically.` };
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export function activityService(db: Db) {
|
|
const scheduledLivenessBackfills = new Set<string>();
|
|
// Issues whose last backfill pass left no missing liveness, keyed like
|
|
// `scheduledLivenessBackfills`. Values are the time the skip window ends.
|
|
// The map keeps insertion order, so the first entry is the oldest.
|
|
const livenessCleanUntil = new Map<string, number>();
|
|
function rememberLivenessClean(key: string) {
|
|
livenessCleanUntil.delete(key);
|
|
livenessCleanUntil.set(key, Date.now() + RUN_LIVENESS_CLEAN_TTL_MS);
|
|
while (livenessCleanUntil.size > RUN_LIVENESS_CLEAN_MAX_ENTRIES) {
|
|
const oldest = livenessCleanUntil.keys().next().value;
|
|
if (oldest === undefined) break;
|
|
livenessCleanUntil.delete(oldest);
|
|
}
|
|
}
|
|
const issueIdAsText = sql<string>`${issues.id}::text`;
|
|
const summarizedUsageJson = sql<Record<string, unknown> | null>`
|
|
case
|
|
when ${heartbeatRuns.usageJson} is null then null
|
|
else jsonb_strip_nulls(jsonb_build_object(
|
|
'inputTokens', coalesce(${heartbeatRuns.usageJson} -> 'inputTokens', ${heartbeatRuns.usageJson} -> 'input_tokens'),
|
|
'input_tokens', coalesce(${heartbeatRuns.usageJson} -> 'input_tokens', ${heartbeatRuns.usageJson} -> 'inputTokens'),
|
|
'outputTokens', coalesce(${heartbeatRuns.usageJson} -> 'outputTokens', ${heartbeatRuns.usageJson} -> 'output_tokens'),
|
|
'output_tokens', coalesce(${heartbeatRuns.usageJson} -> 'output_tokens', ${heartbeatRuns.usageJson} -> 'outputTokens'),
|
|
'cachedInputTokens', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
|
|
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
|
|
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
|
|
),
|
|
'cached_input_tokens', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
|
|
${heartbeatRuns.usageJson} -> 'cachedInputTokens',
|
|
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens'
|
|
),
|
|
'cache_read_input_tokens', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'cache_read_input_tokens',
|
|
${heartbeatRuns.usageJson} -> 'cached_input_tokens',
|
|
${heartbeatRuns.usageJson} -> 'cachedInputTokens'
|
|
),
|
|
'billingType', coalesce(${heartbeatRuns.usageJson} -> 'billingType', ${heartbeatRuns.usageJson} -> 'billing_type'),
|
|
'billing_type', coalesce(${heartbeatRuns.usageJson} -> 'billing_type', ${heartbeatRuns.usageJson} -> 'billingType'),
|
|
'costUsd', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'costUsd',
|
|
${heartbeatRuns.usageJson} -> 'cost_usd',
|
|
${heartbeatRuns.usageJson} -> 'total_cost_usd'
|
|
),
|
|
'cost_usd', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'cost_usd',
|
|
${heartbeatRuns.usageJson} -> 'costUsd',
|
|
${heartbeatRuns.usageJson} -> 'total_cost_usd'
|
|
),
|
|
'total_cost_usd', coalesce(
|
|
${heartbeatRuns.usageJson} -> 'total_cost_usd',
|
|
${heartbeatRuns.usageJson} -> 'cost_usd',
|
|
${heartbeatRuns.usageJson} -> 'costUsd'
|
|
)
|
|
))
|
|
end
|
|
`.as("usageJson");
|
|
// A denied user needs the selected connection's name to repair their task.
|
|
// Other readers need the connection's existing human sharing permission.
|
|
// A malformed present Stop receipt must stay unconfirmed, not disappear.
|
|
const summarizedResultJson = (viewerUserId: string | null) => sql<Record<string, unknown> | null>`
|
|
case
|
|
when ${heartbeatRuns.resultJson} is null then null
|
|
else jsonb_strip_nulls(jsonb_build_object(
|
|
'conversationReset', ${heartbeatRuns.resultJson} -> 'conversationReset',
|
|
'executionCancellation', case when ${heartbeatRuns.resultJson} ? 'executionCancellation'
|
|
then jsonb_build_object('state', case
|
|
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'executionCancellation') = 'object'
|
|
and ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' in ('requested', 'acknowledged')
|
|
then ${heartbeatRuns.resultJson} #>> '{executionCancellation,state}' else 'unknown' end) end,
|
|
'nativeCancellation', case
|
|
when jsonb_typeof(${heartbeatRuns.resultJson} -> 'nativeCancellation') = 'object'
|
|
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,schema}' = 'paperclip.native-cancellation.v1'
|
|
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,runId}' = ${heartbeatRuns.id}::text
|
|
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,companyId}' = ${heartbeatRuns.companyId}::text
|
|
and ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}' in ('pending', 'acknowledged')
|
|
then jsonb_build_object('schema', 'paperclip.native-cancellation.v1',
|
|
'runId', ${heartbeatRuns.id}, 'companyId', ${heartbeatRuns.companyId},
|
|
'dispatchState', ${heartbeatRuns.resultJson} #>> '{nativeCancellation,dispatchState}') end,
|
|
'configurationIncomplete', case
|
|
when ${heartbeatRuns.resultJson} -> 'configurationIncomplete' ->> 'reason' = 'workspace_base_ref_unresolved'
|
|
then jsonb_build_object('reason', 'workspace_base_ref_unresolved')
|
|
when ${heartbeatRuns.errorCode} = 'configuration_incomplete'
|
|
and (${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
|
|
or ${heartbeatRuns.error} = 'This credential is not shared with the responsible user')
|
|
then jsonb_build_object(
|
|
'selectionFailure', case when ${heartbeatRuns.resultJson} #>> '{configurationIncomplete,selectionFailure}' = 'ai_connection_credential_not_shared'
|
|
then 'ai_connection_credential_not_shared' end,
|
|
'credentialAccess', jsonb_build_object(
|
|
'connectionName', ${credentialAccessConnectionNameSql(viewerUserId)}
|
|
)
|
|
) end,
|
|
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
|
|
in (${workspaceRestoreFailureCodesSql})
|
|
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
|
|
'workspaceRestorePath', case when length(${heartbeatRuns.resultJson} ->> 'workspaceRestorePath') <= 180
|
|
then ${heartbeatRuns.resultJson} -> 'workspaceRestorePath' end,
|
|
'finalResponseRecorded', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'finalResponseRecorded') = 'boolean'
|
|
then ${heartbeatRuns.resultJson} -> 'finalResponseRecorded' end,
|
|
'billingType', coalesce(${heartbeatRuns.resultJson} -> 'billingType', ${heartbeatRuns.resultJson} -> 'billing_type'),
|
|
'billing_type', coalesce(${heartbeatRuns.resultJson} -> 'billing_type', ${heartbeatRuns.resultJson} -> 'billingType'),
|
|
'costUsd', coalesce(
|
|
${heartbeatRuns.resultJson} -> 'costUsd',
|
|
${heartbeatRuns.resultJson} -> 'cost_usd',
|
|
${heartbeatRuns.resultJson} -> 'total_cost_usd'
|
|
),
|
|
'cost_usd', coalesce(
|
|
${heartbeatRuns.resultJson} -> 'cost_usd',
|
|
${heartbeatRuns.resultJson} -> 'costUsd',
|
|
${heartbeatRuns.resultJson} -> 'total_cost_usd'
|
|
),
|
|
'total_cost_usd', coalesce(
|
|
${heartbeatRuns.resultJson} -> 'total_cost_usd',
|
|
${heartbeatRuns.resultJson} -> 'cost_usd',
|
|
${heartbeatRuns.resultJson} -> 'costUsd'
|
|
),
|
|
'stopReason', ${heartbeatRuns.resultJson} -> 'stopReason',
|
|
'effectiveTimeoutSec', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutSec',
|
|
'effectiveTimeoutMs', ${heartbeatRuns.resultJson} -> 'effectiveTimeoutMs',
|
|
'timeoutConfigured', ${heartbeatRuns.resultJson} -> 'timeoutConfigured',
|
|
'timeoutSource', ${heartbeatRuns.resultJson} -> 'timeoutSource',
|
|
'timeoutFired', ${heartbeatRuns.resultJson} -> 'timeoutFired'
|
|
))
|
|
end
|
|
`.as("resultJson");
|
|
|
|
function countValue(value: unknown) {
|
|
const parsed = Number(value ?? 0);
|
|
return Number.isFinite(parsed) ? Math.max(0, Math.floor(parsed)) : 0;
|
|
}
|
|
|
|
function dateValue(value: unknown) {
|
|
if (value instanceof Date) return Number.isNaN(value.getTime()) ? null : value;
|
|
if (typeof value === "string" || typeof value === "number") {
|
|
const parsed = new Date(value);
|
|
return Number.isNaN(parsed.getTime()) ? null : parsed;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function latestDate(...values: unknown[]) {
|
|
let latest: Date | null = null;
|
|
for (const value of values) {
|
|
const parsed = dateValue(value);
|
|
if (!parsed) continue;
|
|
if (!latest || parsed.getTime() > latest.getTime()) latest = parsed;
|
|
}
|
|
return latest;
|
|
}
|
|
|
|
function asRecord(value: unknown): Record<string, unknown> | null {
|
|
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
|
return value as Record<string, unknown>;
|
|
}
|
|
|
|
function readNumber(value: unknown) {
|
|
return typeof value === "number" && Number.isFinite(value) ? value : null;
|
|
}
|
|
|
|
async function backfillMissingRunLivenessForIssue(companyId: string, issueId: string) {
|
|
const runs = await missingRunLivenessCandidatesQuery(db, companyId, issueId, RUN_LIVENESS_BACKFILL_BATCH);
|
|
|
|
if (runs.length === 0) return 0;
|
|
|
|
const issue = await db
|
|
.select({
|
|
status: issues.status,
|
|
title: issues.title,
|
|
description: issues.description,
|
|
workMode: issues.workMode,
|
|
})
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
for (const run of runs) {
|
|
const context = asRecord(run.contextSnapshot);
|
|
const continuationAttempt =
|
|
readNumber(context?.continuationAttempt) ??
|
|
readNumber(context?.livenessContinuationAttempt) ??
|
|
run.continuationAttempt ??
|
|
0;
|
|
|
|
const [commentStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*)::int`,
|
|
latestAt: sql<Date | null>`max(${issueComments.createdAt})`,
|
|
})
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.companyId, companyId),
|
|
eq(issueComments.issueId, issueId),
|
|
eq(issueComments.createdByRunId, run.id),
|
|
),
|
|
);
|
|
|
|
const [documentStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*)::int`,
|
|
planCount: sql<number>`count(*) filter (where ${issueDocuments.key} = 'plan')::int`,
|
|
latestAt: sql<Date | null>`max(${documentRevisions.createdAt})`,
|
|
})
|
|
.from(documentRevisions)
|
|
.innerJoin(issueDocuments, eq(documentRevisions.documentId, issueDocuments.documentId))
|
|
.where(
|
|
and(
|
|
eq(documentRevisions.companyId, companyId),
|
|
eq(documentRevisions.createdByRunId, run.id),
|
|
eq(issueDocuments.companyId, companyId),
|
|
eq(issueDocuments.issueId, issueId),
|
|
sql`${issueDocuments.key} != ${ISSUE_CONTINUATION_SUMMARY_DOCUMENT_KEY}`,
|
|
),
|
|
);
|
|
|
|
const [workProductStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*)::int`,
|
|
latestAt: sql<Date | null>`max(${issueWorkProducts.createdAt})`,
|
|
})
|
|
.from(issueWorkProducts)
|
|
.where(
|
|
and(
|
|
eq(issueWorkProducts.companyId, companyId),
|
|
eq(issueWorkProducts.issueId, issueId),
|
|
eq(issueWorkProducts.createdByRunId, run.id),
|
|
),
|
|
);
|
|
|
|
const [workspaceOperationStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*)::int`,
|
|
latestAt: sql<Date | null>`max(${workspaceOperations.startedAt})`,
|
|
})
|
|
.from(workspaceOperations)
|
|
.where(and(eq(workspaceOperations.companyId, companyId), eq(workspaceOperations.heartbeatRunId, run.id)));
|
|
|
|
const [activityStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*)::int`,
|
|
latestAt: sql<Date | null>`max(${activityLog.createdAt})`,
|
|
})
|
|
.from(activityLog)
|
|
.where(and(eq(activityLog.companyId, companyId), eq(activityLog.runId, run.id)));
|
|
|
|
const [eventStats] = await db
|
|
.select({
|
|
count: sql<number>`count(*) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))::int`,
|
|
latestAt: sql<Date | null>`max(${heartbeatRunEvents.createdAt}) filter (where ${heartbeatRunEvents.eventType} not in ('lifecycle', 'adapter.invoke', 'error'))`,
|
|
})
|
|
.from(heartbeatRunEvents)
|
|
.where(and(eq(heartbeatRunEvents.companyId, companyId), eq(heartbeatRunEvents.runId, run.id)));
|
|
|
|
const classification = classifyRunLiveness({
|
|
runStatus: run.status,
|
|
issue,
|
|
resultJson: asRecord(run.resultJson),
|
|
stdoutExcerpt: run.stdoutExcerpt,
|
|
stderrExcerpt: run.stderrExcerpt,
|
|
error: run.error,
|
|
errorCode: run.errorCode,
|
|
continuationAttempt,
|
|
evidence: {
|
|
issueCommentsCreated: countValue(commentStats?.count),
|
|
documentRevisionsCreated: countValue(documentStats?.count),
|
|
planDocumentRevisionsCreated: countValue(documentStats?.planCount),
|
|
workProductsCreated: countValue(workProductStats?.count),
|
|
workspaceOperationsCreated: countValue(workspaceOperationStats?.count),
|
|
activityEventsCreated: countValue(activityStats?.count),
|
|
toolOrActionEventsCreated: countValue(eventStats?.count),
|
|
latestEvidenceAt: latestDate(
|
|
commentStats?.latestAt,
|
|
documentStats?.latestAt,
|
|
workProductStats?.latestAt,
|
|
workspaceOperationStats?.latestAt,
|
|
activityStats?.latestAt,
|
|
eventStats?.latestAt,
|
|
),
|
|
},
|
|
});
|
|
|
|
await db
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
livenessState: classification.livenessState,
|
|
livenessReason: classification.livenessReason,
|
|
continuationAttempt: classification.continuationAttempt,
|
|
lastUsefulActionAt: classification.lastUsefulActionAt,
|
|
nextAction: classification.nextAction,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.livenessState)));
|
|
}
|
|
return runs.length;
|
|
}
|
|
|
|
function scheduleRunLivenessBackfill(companyId: string, issueId: string) {
|
|
const key = `${companyId}:${issueId}`;
|
|
if (scheduledLivenessBackfills.has(key)) return;
|
|
const cleanUntil = livenessCleanUntil.get(key);
|
|
if (cleanUntil !== undefined) {
|
|
if (cleanUntil > Date.now()) return;
|
|
livenessCleanUntil.delete(key);
|
|
}
|
|
scheduledLivenessBackfills.add(key);
|
|
void backfillMissingRunLivenessForIssue(companyId, issueId)
|
|
.then((found) => {
|
|
// A partial batch means this pass classified every remaining candidate.
|
|
// Skip this issue for a while, so page reads do not repeat the lookup.
|
|
if (found < RUN_LIVENESS_BACKFILL_BATCH) rememberLivenessClean(key);
|
|
})
|
|
.catch((err: unknown) => {
|
|
logger.warn({ err, companyId, issueId }, "run liveness backfill failed");
|
|
})
|
|
.finally(() => {
|
|
scheduledLivenessBackfills.delete(key);
|
|
});
|
|
}
|
|
|
|
return {
|
|
list: async (filters: ActivityFilters) => {
|
|
const conditions = [eq(activityLog.companyId, filters.companyId), await skillAccessService(db).activityCondition(filters.companyId),
|
|
filters.caseReadCondition ?? sql<boolean>`${activityLog.entityType} <> 'case'`];
|
|
const limit = normalizeActivityLimit(filters.limit);
|
|
|
|
if (filters.agentId) {
|
|
conditions.push(eq(activityLog.agentId, filters.agentId));
|
|
}
|
|
if (filters.entityType) {
|
|
conditions.push(eq(activityLog.entityType, filters.entityType));
|
|
}
|
|
if (filters.entityId) {
|
|
conditions.push(eq(activityLog.entityId, filters.entityId));
|
|
}
|
|
|
|
return db
|
|
.select({ activityLog })
|
|
.from(activityLog)
|
|
.leftJoin(
|
|
issues,
|
|
and(
|
|
eq(activityLog.entityType, sql`'issue'`),
|
|
eq(activityLog.entityId, issueIdAsText),
|
|
),
|
|
)
|
|
.where(
|
|
and(
|
|
...conditions,
|
|
or(
|
|
sql`${activityLog.entityType} != 'issue'`,
|
|
and(visibleIssueCondition(), filters.readCondition ?? sql<boolean>`true`),
|
|
),
|
|
),
|
|
)
|
|
.orderBy(desc(activityLog.createdAt))
|
|
.limit(limit)
|
|
.then((rows) => rows.map((r) => r.activityLog));
|
|
},
|
|
|
|
inputFactsForIssue: async (companyId: string, issueId: string, commentIds: string[]) => {
|
|
if (!commentIds.length) return [];
|
|
const commentId = sql<string>`${activityLog.details}->>'commentId'`;
|
|
return db.selectDistinctOn([commentId, activityLog.action])
|
|
.from(activityLog)
|
|
.where(and(
|
|
eq(activityLog.companyId, companyId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(activityLog.entityId, issueId),
|
|
inArray(commentId, commentIds),
|
|
inArray(activityLog.action, ["issue.queued_comment_steered", "issue.comment_added"]),
|
|
sql`${activityLog.details}->>'duplicate' is distinct from 'true'`,
|
|
))
|
|
.orderBy(commentId, activityLog.action, asc(activityLog.createdAt), asc(activityLog.id));
|
|
},
|
|
|
|
/** Adds actorName to each row; agents and API readers name the actor without its raw id. */
|
|
withActorNames: async <T extends { actorType: string; actorId: string; companyId: string }>(rows: T[]): Promise<Array<T & { actorName: string }>> => {
|
|
// actorId is free text; only a UUID can name a row in agents.
|
|
const agentIds = [...new Set(rows.filter(row => row.actorType === "agent" && UUID_PATTERN.test(row.actorId)).map(row => row.actorId))];
|
|
const userIds = [...new Set(rows.filter(row => row.actorType === "user" && row.actorId !== "local-board").map(row => row.actorId))];
|
|
const [agentRows, userRows] = await Promise.all([
|
|
agentIds.length ? db.select({ id: agents.id, companyId: agents.companyId, name: agents.name }).from(agents).where(inArray(agents.id, agentIds)) : [],
|
|
userIds.length ? db.select({ id: authUsers.id, name: authUsers.name }).from(authUsers).where(inArray(authUsers.id, userIds)) : [],
|
|
]);
|
|
const agentNames = new Map(agentRows.map(row => [`${row.companyId}:${row.id}`, row.name]));
|
|
const userNames = new Map(userRows.filter(row => row.name?.trim()).map(row => [row.id, row.name!.trim()]));
|
|
return rows.map(row => ({ ...row, actorName: (row.actorType === "agent" ? agentNames.get(`${row.companyId}:${row.actorId}`)
|
|
: row.actorType === "user" ? userNames.get(row.actorId) : undefined) ?? activityActorFallbackName(row.actorType, row.actorId) }));
|
|
},
|
|
|
|
forIssue: (issueId: string, limit?: number) => {
|
|
const query = db
|
|
.select()
|
|
.from(activityLog)
|
|
.where(
|
|
or(
|
|
and(eq(activityLog.entityType, "issue"), eq(activityLog.entityId, issueId)),
|
|
and(or(eq(activityLog.action, "project.created"), eq(activityLog.action, "company.skill_created")), sql`${activityLog.details}->>'sourceIssueId' = ${issueId}`,
|
|
sql`${activityLog.companyId} = (select company_id from issues where id = ${issueId})`),
|
|
),
|
|
)
|
|
.orderBy(desc(activityLog.createdAt), desc(activityLog.id));
|
|
return limit === undefined ? query : query.limit(limit);
|
|
},
|
|
|
|
runsForIssue: async (
|
|
companyId: string,
|
|
issueId: string,
|
|
viewerUserId: string | null = null,
|
|
limit?: number,
|
|
selectedRunIds?: string[],
|
|
// Read-only callers (the chat bootstrap) skip the background liveness
|
|
// backfill: it writes to runs and outlives the request's own limits.
|
|
// The task page's own `/runs` read still schedules it.
|
|
options: { skipLivenessBackfill?: boolean } = {},
|
|
) => {
|
|
if (selectedRunIds && !selectedRunIds.length) return [];
|
|
if (!options.skipLivenessBackfill) scheduleRunLivenessBackfill(companyId, issueId);
|
|
const runsQuery = db
|
|
.select({
|
|
runId: heartbeatRuns.id,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
wakeupRequestId: heartbeatRuns.wakeupRequestId,
|
|
status: heartbeatRuns.status,
|
|
agentId: heartbeatRuns.agentId,
|
|
adapterType: agents.adapterType,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
invocationSource: heartbeatRuns.invocationSource,
|
|
responsibleUserId: heartbeatRuns.responsibleUserId,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
error: sql<string | null>`case when ${heartbeatRuns.status} = 'failed'
|
|
and ${heartbeatRuns.errorCode} = 'native_provider_model_rejected'
|
|
then left(${heartbeatRuns.error}, 2000)
|
|
when ${heartbeatRuns.status} = 'failed'
|
|
and ${heartbeatRuns.error} = 'This credential is not shared with the responsible user'
|
|
then 'This credential is not shared with the responsible user' else null end`,
|
|
usageJson: summarizedUsageJson,
|
|
resultJson: summarizedResultJson(viewerUserId),
|
|
// Bounded cause of a generic failure code; validated below, legacy rows have none.
|
|
failureCause: sql<string | null>`case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'failureCause') = 'string'
|
|
then left(${heartbeatRuns.resultJson} ->> 'failureCause', ${RUN_FAILURE_CAUSE_MAX_LENGTH + 1}) end`,
|
|
logBytes: heartbeatRuns.logBytes,
|
|
retryOfRunId: heartbeatRuns.retryOfRunId,
|
|
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
|
scheduledRetryAttempt: heartbeatRuns.scheduledRetryAttempt,
|
|
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
|
livenessState: heartbeatRuns.livenessState,
|
|
livenessReason: heartbeatRuns.livenessReason,
|
|
continuationAttempt: heartbeatRuns.continuationAttempt,
|
|
lastUsefulActionAt: heartbeatRuns.lastUsefulActionAt,
|
|
nextAction: heartbeatRuns.nextAction,
|
|
wakeCommentIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'wakeCommentIds'`,
|
|
queuedCommentSourceWakeIds: sql<string[] | null>`${heartbeatRuns.contextSnapshot} -> 'queuedCommentSourceWakeIds'`,
|
|
wakeCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'wakeCommentId'`,
|
|
contextCommentId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'commentId'`,
|
|
contextIssueId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`,
|
|
// A person's Task Retry links its saved request here instead of retryOfRunId.
|
|
taskRetryRequestId: sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'taskRetryRequestId'`,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.innerJoin(
|
|
agents,
|
|
and(
|
|
eq(agents.id, heartbeatRuns.agentId),
|
|
eq(agents.companyId, heartbeatRuns.companyId),
|
|
),
|
|
)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
sql`(not exists (select 1 from ${issues} where ${issues.id} = ${issueId}
|
|
and ${issues.companyId} = ${companyId} and ${issues.conversationAgentId} is not null)
|
|
or coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
|
|
${heartbeatRuns.contextSnapshot} ->> 'issueId', ${heartbeatRuns.contextSnapshot} ->> 'taskId') = ${issueId})`,
|
|
...(selectedRunIds ? [inArray(heartbeatRuns.id, selectedRunIds)] : []),
|
|
or(
|
|
eq(heartbeatRuns.issueId, issueId),
|
|
sql`exists (
|
|
select 1
|
|
from ${activityLog}
|
|
where ${activityLog.companyId} = ${companyId}
|
|
and ${activityLog.entityType} = 'issue'
|
|
and ${activityLog.entityId} = ${issueId}
|
|
and ${activityLog.runId} = ${heartbeatRuns.id}
|
|
)`,
|
|
),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id));
|
|
const runs = await (limit === undefined ? runsQuery : runsQuery.limit(limit));
|
|
|
|
if (runs.length === 0) return [];
|
|
const runIds = runs.map((run) => run.runId);
|
|
if (runIds.length === 0) return [];
|
|
|
|
const exhaustionRowsQuery = db
|
|
.select({
|
|
runId: heartbeatRunEvents.runId,
|
|
message: heartbeatRunEvents.message,
|
|
})
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
inArray(heartbeatRunEvents.runId, runIds),
|
|
eq(heartbeatRunEvents.eventType, "lifecycle"),
|
|
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
|
|
),
|
|
)
|
|
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id));
|
|
|
|
const leaseRowsQuery = db
|
|
.select({
|
|
lease: environmentLeases,
|
|
environment: {
|
|
id: environments.id,
|
|
name: environments.name,
|
|
driver: environments.driver,
|
|
},
|
|
})
|
|
.from(environmentLeases)
|
|
.innerJoin(environments, eq(environmentLeases.environmentId, environments.id))
|
|
.where(
|
|
and(
|
|
eq(environmentLeases.companyId, companyId),
|
|
inArray(environmentLeases.heartbeatRunId, runIds),
|
|
),
|
|
)
|
|
.orderBy(desc(environmentLeases.lastUsedAt), desc(environmentLeases.createdAt));
|
|
|
|
// Only stored, current plan revisions can support a saved-plan link.
|
|
// Do not trust an adapter's claim that it wrote a document.
|
|
const savedPlanQuery = runs.some((run) => hasWorkspaceRestoreFailure(run.resultJson))
|
|
? db.select({ revisionId: documentRevisions.id, runId: documentRevisions.createdByRunId })
|
|
.from(issueDocuments)
|
|
.innerJoin(documents, and(eq(documents.id, issueDocuments.documentId), eq(documents.companyId, companyId)))
|
|
.innerJoin(documentRevisions, and(eq(documentRevisions.id, documents.latestRevisionId), eq(documentRevisions.documentId, documents.id), eq(documentRevisions.companyId, companyId)))
|
|
.where(and(eq(issueDocuments.companyId, companyId), eq(issueDocuments.issueId, issueId), eq(issueDocuments.key, "plan")))
|
|
.limit(1)
|
|
: Promise.resolve([]);
|
|
const retryWakeId = (run: (typeof runs)[number]) => run.taskRetryRequestId && UUID_PATTERN.test(run.taskRetryRequestId)
|
|
? run.taskRetryRequestId : run.retryOfRunId ? run.wakeupRequestId : null;
|
|
const retryWakeIds = [...new Set(runs.flatMap(run => { const id = retryWakeId(run); return id ? [id] : []; }))];
|
|
const retryWakesQuery = retryWakeIds.length
|
|
? db.select({
|
|
id: agentWakeupRequests.id,
|
|
reason: agentWakeupRequests.reason,
|
|
requestedByActorType: agentWakeupRequests.requestedByActorType,
|
|
requestedByActorId: agentWakeupRequests.requestedByActorId,
|
|
}).from(agentWakeupRequests)
|
|
.where(and(eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.id, retryWakeIds)))
|
|
: Promise.resolve([]);
|
|
const [exhaustionRows, leaseRows, executionByRunId, [savedPlan], retryWakes] = await Promise.all([
|
|
exhaustionRowsQuery,
|
|
leaseRowsQuery,
|
|
executionProjectionsForRuns(db, companyId, runIds, new Date(), { retryDatabaseReads: true }),
|
|
savedPlanQuery,
|
|
retryWakesQuery,
|
|
]);
|
|
const retryWakeById = new Map(retryWakes.map(wake => [wake.id, wake]));
|
|
const retryPersonIds = [...new Set(retryWakes.flatMap(wake =>
|
|
wake.requestedByActorType === "user" && wake.requestedByActorId && wake.requestedByActorId !== "local-board" ? [wake.requestedByActorId] : []))];
|
|
const retryPersonNames = new Map((retryPersonIds.length
|
|
? await db.select({ id: authUsers.id, name: authUsers.name }).from(authUsers).where(inArray(authUsers.id, retryPersonIds))
|
|
: []).filter(row => row.name?.trim()).map(row => [row.id, row.name!.trim()]));
|
|
const retriedBy = (run: (typeof runs)[number]) => {
|
|
const wakeId = retryWakeId(run);
|
|
const wake = wakeId ? retryWakeById.get(wakeId) ?? null : null;
|
|
const actorId = wake?.requestedByActorId ?? null;
|
|
const personName = wake?.requestedByActorType === "user" && actorId
|
|
? retryPersonNames.get(actorId) ?? activityActorFallbackName("user", actorId) : null;
|
|
return describeRunRetry(run, wake, personName);
|
|
};
|
|
const retryExhaustedReasonByRunId = new Map<string, string>();
|
|
for (const row of exhaustionRows) {
|
|
if (!row.message || retryExhaustedReasonByRunId.has(row.runId)) continue;
|
|
retryExhaustedReasonByRunId.set(row.runId, row.message);
|
|
}
|
|
|
|
const leaseByRunId = new Map<string, (typeof leaseRows)[number]>();
|
|
for (const row of leaseRows) {
|
|
if (row.lease.heartbeatRunId && !leaseByRunId.has(row.lease.heartbeatRunId)) {
|
|
leaseByRunId.set(row.lease.heartbeatRunId, row);
|
|
}
|
|
}
|
|
|
|
return runs.map((run) => {
|
|
const leaseRow = leaseByRunId.get(run.runId);
|
|
const leaseMetadata = leaseRow?.lease.metadata ?? null;
|
|
const workspacePath =
|
|
typeof leaseMetadata?.remoteCwd === "string" && leaseMetadata.remoteCwd.trim().length > 0
|
|
? leaseMetadata.remoteCwd
|
|
: typeof leaseMetadata?.remoteWorkspacePath === "string" && leaseMetadata.remoteWorkspacePath.trim().length > 0
|
|
? leaseMetadata.remoteWorkspacePath
|
|
: null;
|
|
return {
|
|
...run,
|
|
failureCause: normalizeRunFailureCause(run.failureCause),
|
|
resultJson: run.resultJson ? {
|
|
...run.resultJson,
|
|
...(Object.hasOwn(run.resultJson, "workspaceRestorePath") ? {
|
|
workspaceRestorePath: safeWorkspaceRestorePath(run.resultJson.workspaceRestorePath),
|
|
} : {}),
|
|
...(hasWorkspaceRestoreFailure(run.resultJson) ? {
|
|
...(savedPlan?.runId === run.runId ? { savedPlanRevisionId: savedPlan.revisionId } : {}),
|
|
} : {}),
|
|
} : null,
|
|
execution: executionByRunId.get(run.runId) ?? null,
|
|
environment: leaseRow
|
|
? {
|
|
id: leaseRow.environment.id,
|
|
name: leaseRow.environment.name,
|
|
driver: leaseRow.environment.driver,
|
|
}
|
|
: null,
|
|
environmentLease: leaseRow
|
|
? {
|
|
id: leaseRow.lease.id,
|
|
status: leaseRow.lease.status,
|
|
leasePolicy: leaseRow.lease.leasePolicy,
|
|
provider: leaseRow.lease.provider,
|
|
providerLeaseId: leaseRow.lease.providerLeaseId,
|
|
executionWorkspaceId: leaseRow.lease.executionWorkspaceId,
|
|
workspacePath,
|
|
failureReason: leaseRow.lease.failureReason,
|
|
cleanupStatus: leaseRow.lease.cleanupStatus,
|
|
acquiredAt: leaseRow.lease.acquiredAt,
|
|
releasedAt: leaseRow.lease.releasedAt,
|
|
}
|
|
: null,
|
|
retryExhaustedReason: retryExhaustedReasonByRunId.get(run.runId) ?? null,
|
|
retriedBy: retriedBy(run),
|
|
};
|
|
});
|
|
},
|
|
|
|
issuesForRun: async (runId: string, readCondition: SQL<boolean>) => {
|
|
const run = await db
|
|
.select({
|
|
companyId: heartbeatRuns.companyId,
|
|
issueId: heartbeatRuns.issueId,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!run) return [];
|
|
|
|
const fromActivity = await db
|
|
.selectDistinctOn([issueIdAsText], {
|
|
issueId: issues.id,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
})
|
|
.from(activityLog)
|
|
.innerJoin(issues, eq(activityLog.entityId, issueIdAsText))
|
|
.where(
|
|
and(
|
|
eq(activityLog.companyId, run.companyId),
|
|
eq(activityLog.runId, runId),
|
|
eq(activityLog.entityType, "issue"),
|
|
eq(issues.companyId, run.companyId),
|
|
visibleIssueCondition(),
|
|
readCondition,
|
|
),
|
|
)
|
|
.orderBy(issueIdAsText);
|
|
|
|
const contextIssueId = run.issueId;
|
|
if (!contextIssueId) return fromActivity;
|
|
if (fromActivity.some((issue) => issue.issueId === contextIssueId)) return fromActivity;
|
|
|
|
const fromContext = await db
|
|
.select({
|
|
issueId: issues.id,
|
|
identifier: issues.identifier,
|
|
title: issues.title,
|
|
status: issues.status,
|
|
priority: issues.priority,
|
|
})
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, run.companyId),
|
|
eq(issues.id, contextIssueId),
|
|
visibleIssueCondition(),
|
|
readCondition,
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
|
|
if (!fromContext) return fromActivity;
|
|
return [fromContext, ...fromActivity];
|
|
},
|
|
|
|
create: (data: typeof activityLog.$inferInsert) => db.transaction(async tx => {
|
|
if (data.entityType === "issue") await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, `task:${data.entityId}`);
|
|
const [row] = await tx.insert(activityLog).values(data).returning();
|
|
return row;
|
|
}),
|
|
};
|
|
}
|