mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat run settlement records why a run failed, and the board shows that record. > - Runs that fail without a named code settle as `adapter_failed` (or `setup_failed`) with no cause. > - A user, an operator, and the restart chaos harness cannot tell what went wrong from such a run. > - Some run errors also show raw SQL statement text to users. > - This pull request records a bounded `failureCause` code for generic failures, shows it on the board, and keeps SQL text out of run errors. > - The benefit is that a generic failure now names its cause, for example "adapter failed (40001)", without leaking messages, SQL, or secrets. ## Linked Issues or Issue Description Refs #20385 (the nested-cause extraction this change reuses). Related: #20421 (restart chaos universal checks U1-U4). **What happened?** The restart chaos audit of 79 saved waves found 30 failed runs with `errorCode: "adapter_failed"` and a null or generic error. Every U3 ("diagnosable") warning was `adapter_failed`. Nobody could tell what failed. Some board run errors also showed raw SQL statement text from wrapped database errors (Drizzle `Failed query: ... params: ...`). **Expected behavior** A run that fails with a generic code names a short, safe cause. The board shows the cause next to the stop reason. Run error text never shows SQL statements or their parameters. **Steps to reproduce** Make an adapter throw a wrapped database error (an `Error` whose nested `cause` has `code: "40001"` and whose message has a Drizzle `Failed query:` statement). Before this change, the run settles as `adapter_failed` with no cause, and `heartbeat_runs.error` contains the SQL statement and parameters. The new tests in `server/src/services/heartbeat/run-completion.test.ts` reproduce this. ## What Changed - Runner: move the #20385 event-commit cause extraction into `safeErrorCauseCode` (`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and export it. The Runner's own diagnostics use the same function, unchanged in behavior. - Shared: add the `failureCause` contract (`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]` token of at most 80 characters, `normalizeRunFailureCause`, `readRunFailureCause`, and `withoutSqlStatementText`. - Server: add `server/src/services/heartbeat/run-failure-cause.ts`. Settlement records `resultJson.failureCause` only when the code is generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same. - Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`, else `provider_<category>`, else the first safe code in the error and up to three nested causes (`40001`, `ECONNRESET`), else the error class name, else `unknown_error`. - Adapter results without a code (`completeRun`): the error family, `signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`, `adapter_reported_error`, or `unknown_error`. - The failure log lines (`heartbeat execution failed`, `heartbeat execution setup failed`, and a new `heartbeat run failed` warning for generic adapter results) include `failureCause` and `errorCode`. - Server: replace SQL statement text in run error text with "Database query failed (<cause>)" at all three settlement paths, including the cancelled branch. The execution blocker's `runError`, the attention inbox (failed-run excerpt and metadata), and continuation summaries apply the same replacement on read, so older rows also stop showing SQL there. - Server: Retry admission and failed-preparation replay accept only a closed set of `resultJson` keys. `failureCause` is added to each of those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`, `cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is a diagnostic code with no execution or replay authority, so a failed-preparation run stays retryable. - Server: `runsForIssue` returns a validated top-level `failureCause`. It is null for older rows, non-string JSON values, and malformed values. - UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed (40001)". Older runs keep "adapter failed". The Storybook run-ledger fixture has a failed run with a cause. - Restart chaos harness: run samples keep `failureCause`. U3 treats a generic code with a specific `failureCause` as diagnosable. A bare `Error`, `unknown_error`, or the no-code fallback `adapter_reported_error` still counts as vague. U3 stays a warning. - Docs: `doc/run-log-events.md` and the harness README describe the field and the SQL replacement. ## Verification - Red before the fix: the four new `run-completion.test.ts` cases failed against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL text in `error`). - Green: - `server/src/services/heartbeat/run-completion.test.ts`: all tests passed (real embedded PostgreSQL). New cases cover a nested `40001` three levels deep for execution and setup failures, SQL and a secret parameter absent from `error` and `resultJson`, the bound (an 81-character code and a SQL "code" fall back to the class name), `exit_code_137` for a code-less adapter result, a cancelled result with SQL text, no cause for a named code, and the log line. On this loaded host the suite's `afterAll` database teardown hit its 30 s hook limit after every test passed (the known local teardown limit, also seen in #20385). CI runs the suite. - `server/src/services/heartbeat/run-failure-cause.test.ts` (4), `server/src/__tests__/activity-service.test.ts` (recorded, legacy, malformed, overlong, boolean, and number cause projections), `server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL row). - `packages/shared/src/run-failure-cause.test.ts` (6), `ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos `universal.test.ts` and `observer.test.ts`. - `packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`: 207 passed (the stall-diagnostic cases still read `40001` and `ECONNRESET`). - Typecheck: shared, runner, server (all layers), UI, restart chaos harness. `pnpm check:token-gates`: all gates clean. - `pnpm verify` returned exit 2 (global doc input changed); required CI supplies broad coverage. - CI on `6295f8d` found that the new key made failed-preparation runs ineligible for Retry (`chat-channels.integration.test.ts` saved board-message Retry got 409; `workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries). Both reproduced locally and pass after the allowlist fix, with the other 459 Retry cases in `chat-channels.integration.test.ts` and all 25 in `workspace-recovery-preparation-dispatch.test.ts`. - No chaos waves were run. Storybook was not opened in a browser; the fixture typechecks and the label is covered by the UI test. ## Risks - Low risk. `errorCode` values do not change, so retry, recovery, and comparison logic is unaffected. - No schema change and no migration. `failureCause` is a new optional `resultJson` key. Rows saved by the previous release have no key and render as before; a new row read by an old server is ignored. - The SQL replacement only acts on a Drizzle `Failed query:` marker, or on `$n` placeholder text with a statement shape. Ordinary error text does not change. When it acts, it also drops the text after the statement. - Code that checks `resultJson` against a closed key list must allow `failureCause`. I found and updated the four lists on the Retry path; a list I missed would refuse a Retry for a run with a generic failure. - This is the run log / instance database path. It adds no Telemetry event. - The server failure log line still logs the raw error message (unchanged behavior); only the board-facing text is changed. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file editing) and code execution for tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
449 lines
32 KiB
TypeScript
449 lines
32 KiB
TypeScript
import { isPreDispatchReviewWait, neverStartedDispatchCancellationPredicate } from "./pre-dispatch-review-wait.js";
|
|
import { canonicalExecutionUuid } from "./execution-identity.js";
|
|
import { preDispatchExecutionWaitRecoveryActionPredicate } from "./pre-dispatch-execution-wait.js";
|
|
import { legacyShutdownBeforeDispatchCondition } from "./legacy-shutdown-before-dispatch.js";
|
|
import { and, desc, eq, gt, inArray, not, or, sql, type SQL } from "drizzle-orm";
|
|
import { conversationRecoveryActionPredicate, getConversationOwnershipBlocker } from "./conversation-continuation.js";
|
|
import { z } from "zod";
|
|
import { agentWakeupRequests, chatConversations, chatEndpoints, environmentLeases, heartbeatRunEvents, heartbeatRuns, issueComments, issues, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, toolConnections, workspaceOperations, type Db } from "@paperclipai/db";
|
|
import { canContinueCancelledRun, readRunCancellation } from "./run-cancellation.js";
|
|
import { canRetryStoppedRun } from "./cancelled-native-startup.js";
|
|
import { queuedCommentIdsFromWakePayload } from "./issue-queued-comment-queue.js";
|
|
import { EXECUTION_RECONCILIATION_CAUSES, readRunFailureCause, withoutSqlStatementText, type ExecutionBlocker } from "@paperclipai/shared";
|
|
import { hasRequiredWorkspaceRecovery, LEGACY_WORKSPACE_RECOVERY_SCHEMA, nativeRunHasNoWorkspaceAccess } from "./workspace-restore-recovery-state.js";
|
|
import { unreconciledNativeExportPredicate } from "./native-runtime/native-workspace-reconstruction.js";
|
|
import { hasRecoveredCleanupStopReceipt, hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
|
|
import { completedWorkspaceRecoveryCondition, hasVerifiedWorkspaceRepair, workspaceRecoveryDisplay } from "./workspace-recovery-receipt.js";
|
|
import { nativeManualAdmissionHoldsExecution } from "./native-runtime/native-runner-ownership.js";
|
|
import { PROCESS_IDENTITY_RECORDED, PROCESS_START_REQUESTED } from "./native-local-process-stop.js";
|
|
|
|
const blockerContextKeys = ["issueId", "taskId", "source", "connectionIntentResolution",
|
|
"interactionKind", "interactionStatus", "mutation", "interactionId"] as const;
|
|
const blockerProfileKeys = ["adapterDispatch", "nativeExecutionInput", "nativeWorkspaceSync", "remoteWorkspace", "legacyWorkspaceRecoveryMigration", "finalizedWorkspaceRecoveryMigration"] as const;
|
|
const blockerContextColumn = sql<Record<string, unknown> | null>`case
|
|
when ${heartbeatRuns.contextSnapshot} is null then null else (
|
|
select coalesce(jsonb_object_agg(field.key, field.value), '{}'::jsonb)
|
|
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object'
|
|
then ${heartbeatRuns.contextSnapshot} else '{}'::jsonb end) as field(key, value)
|
|
where field.key in (${sql.join(blockerContextKeys.map(key => sql`${key}`), sql`, `)})
|
|
) end`;
|
|
const blockerProfileColumn = sql<Record<string, unknown> | null>`case
|
|
when ${heartbeatRuns.runnerProfileJson} is null then null else (
|
|
select coalesce(jsonb_object_agg(field.key, case field.key
|
|
when 'adapterDispatch' then jsonb_build_object('adapterType', field.value->'adapterType')
|
|
when 'nativeExecutionInput' then case when jsonb_typeof(field.value) = 'object' then (
|
|
select coalesce(jsonb_object_agg(input_field.key, input_field.value), '{}'::jsonb)
|
|
from jsonb_each(field.value) as input_field(key, value)
|
|
where input_field.key in ('schema', 'binding', 'provider', 'workspace')
|
|
) else field.value end
|
|
else field.value end), '{}'::jsonb)
|
|
from jsonb_each(case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object'
|
|
then ${heartbeatRuns.runnerProfileJson} else '{}'::jsonb end) as field(key, value)
|
|
where field.key in (${sql.join(blockerProfileKeys.map(key => sql`${key}`), sql`, `)})
|
|
) end`;
|
|
const blockerRunFields = {
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
status: heartbeatRuns.status,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
finishedAt: heartbeatRuns.finishedAt,
|
|
error: heartbeatRuns.error,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
issueId: heartbeatRuns.issueId,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
runtimeModeResolvedAt: heartbeatRuns.runtimeModeResolvedAt,
|
|
driverKind: heartbeatRuns.driverKind,
|
|
processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt,
|
|
sessionIdAfter: heartbeatRuns.sessionIdAfter,
|
|
controllerBootId: heartbeatRuns.controllerBootId,
|
|
controllerLeaseExpiresAt: heartbeatRuns.controllerLeaseExpiresAt,
|
|
executionStage: heartbeatRuns.executionStage,
|
|
resultJson: heartbeatRuns.resultJson,
|
|
contextSnapshot: blockerContextColumn,
|
|
runnerProfileJson: blockerProfileColumn,
|
|
};
|
|
|
|
/** Appended when the person can retry the stopped run on this task. */
|
|
export const EXECUTION_BLOCKER_PERSON_RETRY_ACTION =
|
|
"A person can use Retry on this task, or send it a new message, to continue once the previous execution has stopped. Agents cannot retry it for them.";
|
|
|
|
export async function hasPendingRunCleanup(db: Db, companyId: string, runId: string): Promise<boolean> {
|
|
if (await nativeManualAdmissionHoldsExecution(db, { companyId, runId })) return true;
|
|
// Recovery saved the files and re-stopped this sandbox before queueing its
|
|
// destroy. That destroy is resource cleanup, not a live owner of the run.
|
|
const recoveredStopped = (await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, runId),
|
|
sql`${environmentLeases.metadata}->'workspaceRecoveryCleanup' is not null`,
|
|
))).filter(hasRecoveredCleanupStopReceipt).map(lease => lease.id);
|
|
// A missing predecessor row owns nothing: deleting a run nulls its leases'
|
|
// run reference and removes its finalization coordinator. Failing closed on
|
|
// absence stranded the successor with no next action (invariants 3 and 7).
|
|
const [state] = await db.execute<{ pending: boolean }>(sql`SELECT EXISTS (SELECT 1 FROM ${heartbeatRuns}
|
|
WHERE ${heartbeatRuns.companyId} = ${companyId} AND ${heartbeatRuns.id} = ${runId} AND (
|
|
${heartbeatRuns.runnerProfileJson}->>'nativeWorkspaceFinalizationOwner' IS NOT NULL
|
|
OR EXISTS (SELECT 1 FROM ${environmentLeases}
|
|
WHERE ${environmentLeases.companyId} = ${companyId} AND ${environmentLeases.heartbeatRunId} = ${runId}
|
|
${recoveredStopped.length ? sql`AND ${environmentLeases.id}::text NOT IN (${sql.join(recoveredStopped.map(id => sql`${id}`), sql`, `)})` : sql``}
|
|
AND (( ${environmentLeases.releasedAt} IS NULL AND NOT COALESCE(
|
|
${environmentLeases.status} = 'retained' AND ${environmentLeases.leasePolicy} = 'reuse_by_environment'
|
|
AND ${environmentLeases.cleanupStatus} = 'success', false))
|
|
OR ${environmentLeases.status} IN ('active', 'pending_cleanup') OR ${environmentLeases.cleanupStatus} = 'failed'
|
|
OR COALESCE(${environmentLeases.metadata}->>'pendingCleanupInFlight', 'false') <> 'false'
|
|
OR (${heartbeatRuns.runtimeMode} = 'native' AND NOT COALESCE(
|
|
(${environmentLeases.status} IN ('released', 'expired', 'failed')
|
|
AND ${environmentLeases.releasedAt} IS NOT NULL AND (${environmentLeases.cleanupStatus} = 'success'
|
|
OR (${environmentLeases.cleanupStatus} IS NULL AND (
|
|
(${environmentLeases.provider} = 'local' AND ${environmentLeases.providerLeaseId} IS NULL)
|
|
OR (${environmentLeases.provider} = 'ssh' AND ${environmentLeases.metadata}->>'driver' = 'ssh'
|
|
AND ${environmentLeases.providerLeaseId} LIKE 'ssh://%')))))
|
|
OR (${environmentLeases.status} = 'retained' AND ${environmentLeases.leasePolicy} = 'reuse_by_environment'
|
|
AND ${environmentLeases.cleanupStatus} = 'success'), false))))))
|
|
OR EXISTS (SELECT 1 FROM ${nativeRunFinalizations}
|
|
WHERE ${nativeRunFinalizations.companyId} = ${companyId} AND ${nativeRunFinalizations.runId} = ${runId}
|
|
AND (${nativeRunFinalizations.leaseOwner} IS NOT NULL
|
|
OR ${nativeRunFinalizations.phase} NOT IN ('committed', 'applied', 'terminal_failure'))) AS pending`);
|
|
return state?.pending === true;
|
|
}
|
|
|
|
/** Resolved recovery bookkeeping can still carry an effective no-replay hold. */
|
|
export function executionBlockerPredicate() {
|
|
return and(
|
|
not(conversationRecoveryActionPredicate()!),
|
|
not(preDispatchExecutionWaitRecoveryActionPredicate()),
|
|
// Holds saved before shutdown certified pre-dispatch interruptions carry
|
|
// the same row proof; they have no unknown outcome to reconcile.
|
|
sql`not (${issueRecoveryActions.cause} = 'legacy_execution_requires_reconciliation' and exists (
|
|
select 1 from ${heartbeatRuns}
|
|
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
|
|
and ${heartbeatRuns.id} = ${canonicalExecutionUuid(sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`)}
|
|
and ${issueRecoveryActions.sourceIssueId} = ${canonicalExecutionUuid(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`)}
|
|
and ${legacyShutdownBeforeDispatchCondition()}
|
|
and coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'
|
|
and ${issueRecoveryActions.evidence}->'workspaceRestoreRecovery' is null
|
|
))`,
|
|
sql`not (${issueRecoveryActions.cause} = 'legacy_execution_requires_reconciliation' and exists (
|
|
select 1 from ${heartbeatRuns}
|
|
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
|
|
and ${heartbeatRuns.id} = ${canonicalExecutionUuid(sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`)}
|
|
and ${issueRecoveryActions.sourceIssueId} = ${canonicalExecutionUuid(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`)}
|
|
and (${heartbeatRuns.issueId} is null or ${heartbeatRuns.issueId} = ${issueRecoveryActions.sourceIssueId})
|
|
and ${neverStartedDispatchCancellationPredicate()}
|
|
and ${heartbeatRuns.errorCode} <> 'issue_continuation_waiting_on_review'
|
|
and coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'
|
|
and coalesce(${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema', '') <> ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}
|
|
))`,
|
|
or(unreconciledNativeExportPredicate(), and(
|
|
inArray(issueRecoveryActions.cause, [...EXECUTION_RECONCILIATION_CAUSES]),
|
|
or(inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
|
sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`),
|
|
)),
|
|
);
|
|
}
|
|
|
|
export function nativeExecutionRecoveryActionPredicate() {
|
|
return sql`exists (select 1 from ${heartbeatRuns}
|
|
where ${heartbeatRuns.companyId} = ${issueRecoveryActions.companyId}
|
|
and ${heartbeatRuns.id}::text = coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')
|
|
and ${heartbeatRuns.runtimeMode} = 'native')`;
|
|
}
|
|
|
|
// Exact runner startup lifecycle events (event-v3 schema): they open no turn.
|
|
// Every other runner-sourced event counts as provider work, so a new, unlisted
|
|
// or turn-scoped event (for example workspace.diff.recorded or a session goal)
|
|
// keeps the hold (fail closed). Server-side records without a source event
|
|
// (spans, process identity, lifecycle) count only when they name turn or
|
|
// provider work.
|
|
export const PROVIDER_STARTUP_EVENT_TYPES = ["harness.starting", "harness.ready", "harness.diagnostic", "harness.exited",
|
|
"session.starting", "session.started", "session.resuming", "session.resumed", "session.reconciled", "session.updated",
|
|
"session.capabilities.updated", "session.closed", "session.failed", "workspace.ready", "runtime.phase.changed",
|
|
"runtime.activity.recorded", "provider.notice.recorded", "auth.credentials.updated", "run.attached", "run.detached",
|
|
"mcp_app.discovered", "mcp_app.initializing", "mcp_app.ready", "mcp_app.resource.resolved",
|
|
"mcp_app.host_context.changed", "mcp_app.teardown", "mcp_app.failed"] as const;
|
|
const SERVER_TURN_WORK_EVENT_PREFIXES = ["turn.", "item.", "tool.", "semantic_tool.", "run.result."] as const;
|
|
const SERVER_TURN_WORK_EVENT_TYPES = ["provider.event", "provider.rpc_result"] as const;
|
|
|
|
const anyPrefix = (eventType: SQL, prefixes: readonly string[]) =>
|
|
sql.join(prefixes.map(prefix => sql`starts_with(${eventType}, ${prefix})`), sql` or `);
|
|
const anyType = (eventType: SQL, types: readonly string[]) =>
|
|
sql`${eventType} in (${sql.join(types.map(type => sql`${type}`), sql`, `)})`;
|
|
|
|
/** SQL truth for an event row (alias columns) that may prove provider turn work. */
|
|
export function providerTurnWorkEventCondition(eventType: SQL, sourceEventId: SQL) {
|
|
return sql`((${sourceEventId} is not null and not ${anyType(eventType, PROVIDER_STARTUP_EVENT_TYPES)})
|
|
or ${anyType(eventType, SERVER_TURN_WORK_EVENT_TYPES)} or ${anyPrefix(eventType, SERVER_TURN_WORK_EVENT_PREFIXES)})`;
|
|
}
|
|
|
|
/**
|
|
* A Stop that the native startup owner fenced before it published a provider
|
|
* handle proves that no provider turn ran. The run's own durable record must
|
|
* agree: no turn, tool or provider event and no native result. `stopped` is
|
|
* the SQL alias of the stopped run.
|
|
*/
|
|
export function stoppedBeforeProviderTurnCondition() {
|
|
return sql`(stopped.status = 'cancelled' and stopped.runtime_mode = 'native'
|
|
and stopped.result_json->'nativeCancellation'->>'schema' = 'paperclip.native-cancellation.v1'
|
|
and stopped.result_json->'nativeCancellation'->>'runId' = stopped.id::text
|
|
and stopped.result_json->'nativeCancellation'->>'scope' = 'run'
|
|
and stopped.result_json->'nativeCancellation'->>'dispatchState' = 'acknowledged'
|
|
and stopped.result_json->'nativeCancellation'->'beforeProviderTurn' = 'true'::jsonb
|
|
and not exists (select 1 from ${heartbeatRunEvents} work
|
|
where work.company_id = stopped.company_id and work.run_id = stopped.id
|
|
and ${providerTurnWorkEventCondition(sql`work.event_type`, sql`work.source_event_id`)})
|
|
and not exists (select 1 from ${nativeRunResults} result
|
|
where result.company_id = stopped.company_id and result.run_id = stopped.id))`;
|
|
}
|
|
|
|
export function interruptedPreparationBeforeProviderWorkCondition() {
|
|
return sql`(stopped.status = 'interrupted' and stopped.runtime_mode = 'legacy'
|
|
and stopped.runtime_mode_resolved_at is null and stopped.execution_stage = 'preparing'
|
|
and stopped.runner_profile_json#>>'{adapterDispatch,adapterType}' = 'paperclip_runner'
|
|
and stopped.runner_profile_json->'nativeExecutionInput' is null
|
|
and stopped.process_pid is null and stopped.process_group_id is null and stopped.process_started_at is null
|
|
and stopped.session_id_after is null and stopped.native_issue_id is null and stopped.native_session_id is null
|
|
and (stopped.controller_lease_expires_at is null or stopped.controller_lease_expires_at <= now())
|
|
and (stopped.runner_profile_json->'remoteWorkspace' is null or exists (
|
|
select 1 from jsonb_array_elements_text(case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
|
|
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end) saved(id)
|
|
join ${workspaceOperations} op on op.id::text = saved.id
|
|
where op.company_id = stopped.company_id and op.heartbeat_run_id = stopped.id and op.status = 'succeeded'
|
|
and op.metadata->>'snapshotId' = stopped.runner_profile_json#>>'{remoteWorkspace,snapshotId}'))
|
|
and coalesce(stopped.result_json#>'{executionRecovery,providerWorkStarted}', 'false'::jsonb) = 'false'::jsonb
|
|
and case when stopped.result_json->'cancellation' is null then stopped.error_code = 'server_shutdown_interrupted'
|
|
else stopped.result_json#>>'{cancellation,initiator,type}' = 'system'
|
|
and coalesce(stopped.result_json#>>'{cancellation,source}', '') <> 'operator'
|
|
and (stopped.result_json#>>'{cancellation,source}' = 'shutdown'
|
|
or stopped.result_json#>'{cancellation,expected}' = 'false'::jsonb) end
|
|
and not exists (select 1 from ${heartbeatRunEvents} work
|
|
where work.company_id = stopped.company_id and work.run_id = stopped.id
|
|
and (work.source_event_id is not null
|
|
or work.event_type in ('adapter.invoke', ${PROCESS_START_REQUESTED}, ${PROCESS_IDENTITY_RECORDED})
|
|
or ${providerTurnWorkEventCondition(sql`work.event_type`, sql`work.source_event_id`)}))
|
|
and not exists (select 1 from ${nativeRunFinalizations} coordinator
|
|
where coordinator.company_id = stopped.company_id and coordinator.run_id = stopped.id)
|
|
and not exists (select 1 from ${nativeRunResults} result
|
|
where result.company_id = stopped.company_id and result.run_id = stopped.id)
|
|
and not exists (select 1 from ${environmentLeases} lease
|
|
where lease.company_id = stopped.company_id and lease.heartbeat_run_id = stopped.id
|
|
and not coalesce(lease.released_at is not null and lease.cleanup_status = 'success'
|
|
and lease.status in ('released', 'expired', 'failed')
|
|
and coalesce(lease.metadata->>'pendingCleanupInFlight', 'false') = 'false'
|
|
and (lease.provider = 'local' or (
|
|
lease.metadata#>>'{remoteExecutionTermination,schema}' = 'paperclip.remote-termination.v1'
|
|
and lease.metadata#>>'{remoteExecutionTermination,companyId}' = lease.company_id::text
|
|
and lease.metadata#>>'{remoteExecutionTermination,runId}' = stopped.id::text
|
|
and lease.metadata#>>'{remoteExecutionTermination,leaseId}' = lease.id::text
|
|
and lease.metadata#>>'{remoteExecutionTermination,provider}' = lease.provider
|
|
and lease.metadata#>>'{remoteExecutionTermination,providerLeaseId}' = lease.provider_lease_id
|
|
and lease.metadata#>>'{remoteExecutionTermination,state}' in ('stopped', 'destroyed'))), false)))`;
|
|
}
|
|
|
|
/**
|
|
* A file-preservation hold binds a newer task owner only until automatic
|
|
* recovery saved the stopped run's files. That owner's turn is a fresh
|
|
* assignment, not a replay: the stopped run's unknown outcomes stay with its
|
|
* own agent, which keeps the hold. A Stop fenced before any provider turn, or a
|
|
* shutdown that interrupted preparation before any provider work, has no
|
|
* unknown outcome, so its hold releases for the same agent too. The proof is
|
|
* the durable completed-recovery receipt bound to this action, so later reuse
|
|
* of the workspace by the current owner (new epoch, writer or snapshot) cannot
|
|
* re-arm the hold.
|
|
*/
|
|
export function restoreHoldReleasedForCurrentOwnerPredicate() {
|
|
const sourceRunId = sql`coalesce(${issueRecoveryActions.evidence}->>'runId', ${issueRecoveryActions.evidence}->>'sourceRunId')`;
|
|
const operationIds = sql`case when jsonb_typeof(${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}') = 'array'
|
|
then ${issueRecoveryActions.evidence}#>'{systemRecovery,preservationOperationIds}' else '[]'::jsonb end`;
|
|
// JSON reads are NULL for other holds. Callers negate this predicate, so it
|
|
// must be strictly true or false: NOT NULL would hide unrelated blockers.
|
|
return sql`coalesce((${and(
|
|
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
|
|
sql`coalesce(${issueRecoveryActions.evidence}->>'workspaceRestoreFailure', '') <> 'restore_unsafe_archive'`,
|
|
sql`exists (select 1 from ${heartbeatRuns} stopped
|
|
join ${issues} task on task.company_id = stopped.company_id and task.id = ${issueRecoveryActions.sourceIssueId}
|
|
where stopped.company_id = ${issueRecoveryActions.companyId}
|
|
and stopped.id::text = ${sourceRunId}
|
|
and stopped.status in ('failed', 'timed_out', 'interrupted', 'cancelled')
|
|
and stopped.finished_at is not null
|
|
and task.assignee_agent_id is not null
|
|
and ((task.assignee_agent_id <> stopped.agent_id
|
|
and not coalesce(task.status = 'in_review'
|
|
and task.execution_state->>'status' = 'pending'
|
|
and task.execution_state->'currentParticipant'->>'type' = 'agent'
|
|
and task.execution_state->'currentParticipant'->>'agentId' = stopped.agent_id::text, false))
|
|
or ${stoppedBeforeProviderTurnCondition()} or ${interruptedPreparationBeforeProviderWorkCondition()}))`,
|
|
completedWorkspaceRecoveryCondition(sourceRunId),
|
|
// Copyback put the files back into the task workspace that the new owner
|
|
// reuses. A snapshot-only save needs an explicit fresh-workspace request
|
|
// to attach it, so that hold stays until such a request or a repair.
|
|
sql`not exists (select 1 from jsonb_array_elements_text(${operationIds}) saved(id)
|
|
join ${workspaceOperations} op on op.id::text = saved.id
|
|
where op.company_id = ${issueRecoveryActions.companyId} and op.phase <> 'workspace_recovery')`,
|
|
)}), false)`;
|
|
}
|
|
|
|
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { conversationResetCommentId?: string | null }): Promise<ExecutionBlocker | null> {
|
|
const [conversation] = await db.select({ agentId: issues.conversationAgentId,
|
|
boundaryId: issues.conversationBoundaryCommentId, status: issues.status,
|
|
assigneeAgentId: issues.assigneeAgentId, statusVersion: issues.statusVersion,
|
|
executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues).where(and(
|
|
eq(issues.companyId, companyId), eq(issues.id, issueId),
|
|
)).limit(1);
|
|
// Resetting model context cannot repair or recover a workspace. This hold
|
|
// survives conversation boundaries until the existing repair/reconciliation path clears it.
|
|
const [restoreHold] = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
executionBlockerPredicate(),
|
|
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
|
or(
|
|
unreconciledNativeExportPredicate(),
|
|
sql`${issueRecoveryActions.evidence}->>'workspaceRestoreFailure' = 'restore_unsafe_archive'`,
|
|
sql`${issueRecoveryActions.evidence}->'workspaceRestoreRecovery'->>'schema' = ${LEGACY_WORKSPACE_RECOVERY_SCHEMA}`,
|
|
),
|
|
)).orderBy(desc(issueRecoveryActions.updatedAt)).limit(1);
|
|
// A persisted user /new is an ordered context command, not a retry of uncertain work.
|
|
// The normal issue execution lock still serializes it behind any active turn.
|
|
if (!restoreHold && conversation?.agentId && options?.conversationResetCommentId) {
|
|
const [command] = await db.select().from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
eq(issueComments.id, options.conversationResetCommentId),
|
|
)).limit(1);
|
|
if (command?.authorUserId && !command.deletedAt && command.body.trim() === "/new") return null;
|
|
}
|
|
const [boundary] = conversation?.agentId && conversation.boundaryId
|
|
? await db.select({ createdAt: issueComments.createdAt }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
eq(issueComments.id, conversation.boundaryId),
|
|
)).limit(1) : [];
|
|
|
|
const ownership = await getConversationOwnershipBlocker(db, companyId, issueId);
|
|
if (ownership) return { ...ownership, recoveryActionId: null };
|
|
const [action] = restoreHold ? [restoreHold] : await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, companyId),
|
|
eq(issueRecoveryActions.sourceIssueId, issueId),
|
|
executionBlockerPredicate(),
|
|
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
|
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate()) : undefined,
|
|
)).orderBy(desc(issueRecoveryActions.updatedAt), desc(issueRecoveryActions.id)).limit(1);
|
|
if (!action) return null;
|
|
const parsedRunId = z.string().guid().safeParse(action.evidence.runId ?? action.evidence.sourceRunId);
|
|
const runId = parsedRunId.success ? parsedRunId.data : null;
|
|
const [run] = runId ? await db.select(blockerRunFields).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.id, runId),
|
|
)).limit(1) : [];
|
|
const withoutWorkspace = Boolean(run?.driverKind === "grok_bot_mcp" && nativeRunHasNoWorkspaceAccess(run));
|
|
const workspaceRepairRequired = Boolean(restoreHold && !withoutWorkspace && (!run || !await hasVerifiedWorkspaceRepair(db, run)));
|
|
const workspaceRecovery = workspaceRecoveryDisplay(action.evidence, runId);
|
|
const reconstructableExport = ["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"].includes(action.cause);
|
|
const exportSource = reconstructableExport ? readNativeWorkspaceSyncReference(run?.runnerProfileJson?.nativeWorkspaceSync) : null;
|
|
const [exportLease] = exportSource ? await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, companyId), eq(environmentLeases.heartbeatRunId, run!.id), eq(environmentLeases.id, exportSource.leaseId),
|
|
eq(environmentLeases.providerLeaseId, exportSource.providerLeaseId),
|
|
)).limit(1) : [];
|
|
const canReconstructWorkspace = Boolean(exportLease && hasRemoteTerminationReceipt(exportLease)
|
|
&& (exportLease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state === "destroyed"
|
|
&& !action.evidence.executionReconciliation && conversation && ["blocked", "in_progress"].includes(conversation.status)
|
|
&& !conversation.executionRunId && !conversation.checkoutRunId && conversation.assigneeAgentId === run?.agentId);
|
|
const reconstructionPending = Boolean((action.evidence.executionReconciliation as { workspaceReconstruction?: unknown } | undefined)?.workspaceReconstruction
|
|
&& action.evidence.continuationDelivery === "pending");
|
|
const queue = await db.select({ payload: agentWakeupRequests.payload }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), eq(agentWakeupRequests.status, "deferred_issue_execution"),
|
|
sql`${agentWakeupRequests.payload}->>'issueId' = ${issueId}`,
|
|
));
|
|
const savedIds = [...new Set(queue.flatMap(entry => {
|
|
const context = entry.payload?._paperclipWakeContext as Record<string, unknown> | undefined;
|
|
return [...queuedCommentIdsFromWakePayload(entry.payload), entry.payload?.commentId, context?.wakeCommentId]
|
|
.filter((id): id is string => z.string().guid().safeParse(id).success);
|
|
}))];
|
|
const saved = savedIds.length ? await db.select({ id: issueComments.id }).from(issueComments).where(and(
|
|
eq(issueComments.companyId, companyId), eq(issueComments.issueId, issueId),
|
|
inArray(issueComments.id, savedIds), eq(issueComments.authorType, "user"), sql`${issueComments.deletedAt} is null`,
|
|
sql`not exists (select 1 from (
|
|
select consumed.context_snapshot->>'wakeCommentId' as comment_id,
|
|
consumed.context_snapshot->'wakeCommentIds' as comment_ids
|
|
from ${heartbeatRuns} consumed where consumed.company_id = ${companyId}
|
|
and consumed.context_snapshot->>'issueId' = ${issueId}
|
|
and not (consumed.status = 'cancelled' and consumed.started_at is null)
|
|
offset 0
|
|
) consumed where consumed.comment_id = ${issueComments.id}::text
|
|
or consumed.comment_ids @> jsonb_build_array(${issueComments.id}::text))`,
|
|
)) : [];
|
|
const cancellation = run ? readRunCancellation(run.resultJson) : null;
|
|
const eligibleContinuation = Boolean(!restoreHold && run && conversation?.assigneeAgentId === run.agentId &&
|
|
!["done", "cancelled"].includes(conversation.status) && canContinueCancelledRun(run));
|
|
// A failed run whose saved work is verifiably recovered no longer needs the
|
|
// hold to protect its files. Offer the same Retry that admission accepts, so
|
|
// an infrastructure failure (such as a restart) is not left to a new message.
|
|
// An intentional Stop keeps "send a new message" as its only continuation.
|
|
const recoveredFailure = Boolean(restoreHold && run?.status === "failed" && !workspaceRepairRequired &&
|
|
workspaceRecovery?.status === "recovered");
|
|
const canRetry = Boolean((!restoreHold || recoveredFailure) && run && conversation?.assigneeAgentId === run.agentId &&
|
|
!["done", "cancelled"].includes(conversation.status) && await canRetryStoppedRun(db, run));
|
|
const runError = run && canRetry && isPreDispatchReviewWait(run)
|
|
? "Waiting for review; this continuation never started."
|
|
: cancellation?.reason ?? (run?.status === "cancelled"
|
|
? "Execution was cancelled; its source was not recorded."
|
|
// Rows written before SQL text was kept out of run errors still carry it.
|
|
: run?.error ? withoutSqlStatementText(run.error, readRunFailureCause(run.resultJson)) : run?.error);
|
|
const [chatBinding] = eligibleContinuation || canRetry ? await db.select({
|
|
state: chatConversations.state, endpointStatus: chatEndpoints.status,
|
|
connectionStatus: toolConnections.status, connectionEnabled: toolConnections.enabled,
|
|
}).from(chatConversations).leftJoin(chatEndpoints, and(
|
|
eq(chatEndpoints.companyId, companyId), eq(chatEndpoints.id, chatConversations.endpointId),
|
|
)).leftJoin(toolConnections, and(
|
|
eq(toolConnections.companyId, companyId), eq(toolConnections.id, chatEndpoints.connectionId),
|
|
)).where(and(
|
|
eq(chatConversations.companyId, companyId), eq(chatConversations.issueId, issueId),
|
|
)).limit(1) : [];
|
|
let nextAction = action.nextAction;
|
|
// Retry is a person's control on this task. Say so, so an agent reading this
|
|
// record points the person to it instead of re-running the work itself.
|
|
if (canRetry && !eligibleContinuation) nextAction += ` ${EXECUTION_BLOCKER_PERSON_RETRY_ACTION}`;
|
|
if (chatBinding && !restoreHold) {
|
|
if (chatBinding.state === "endpoint_removed" || !chatBinding.endpointStatus || chatBinding.endpointStatus === "archived") {
|
|
nextAction = "This chat connection was removed. Inspect the stopped run and create a new task to continue the work.";
|
|
} else if (chatBinding.state === "unavailable" || !["active", "verifying"].includes(chatBinding.endpointStatus) ||
|
|
chatBinding.connectionStatus !== "active" || !chatBinding.connectionEnabled) {
|
|
nextAction = "This chat connection is unavailable. Restore access in Apps or create a new task to continue the work.";
|
|
} else {
|
|
nextAction = "Send a new chat message to continue this conversation.";
|
|
}
|
|
} else if (!restoreHold && run?.status === "cancelled" && !eligibleContinuation && !canRetry && action.cause === "legacy_execution_requires_reconciliation") {
|
|
nextAction += " Inspect the run before sending a new message to request continuation.";
|
|
}
|
|
|
|
if (withoutWorkspace && (restoreHold || run?.status === "cancelled") && !run?.resultJson?.externalWorkspaceInspection) {
|
|
nextAction = run?.driverKind === "grok_bot_mcp"
|
|
? "Inspect this Bot in Grok Bot, then clear the recovery hold and verify the connection in the agent's Runtime settings before assigning more work."
|
|
: "Inspect the stopped work in the external agent and its connection before requesting continuation.";
|
|
}
|
|
|
|
if (run?.resultJson?.externalWorkspaceInspection && !restoreHold) {
|
|
nextAction = "Verify the Grok connection, then send a new message to continue. The stopped work will not be replayed.";
|
|
}
|
|
|
|
return {
|
|
recoveryActionId: action.id,
|
|
runId,
|
|
// A stopped reviewer can differ from the task owner who receives the work back.
|
|
agentId: run?.agentId ?? null,
|
|
cause: action.cause,
|
|
nextAction: (workspaceRecovery?.status === "recovered" && canRetry && !chatBinding
|
|
? "Saved work recovered. Retry or send a message to continue." : workspaceRecovery?.message) ?? (reconstructionPending ? "Continuation from saved sources is queued and waiting for execution checks." : nextAction),
|
|
...(workspaceRecovery ? { workspaceRecovery } : {}),
|
|
runStatus: run?.status ?? null,
|
|
runError: runError?.slice(0, 1024) ?? null,
|
|
canContinue: eligibleContinuation && !chatBinding,
|
|
canRetry: canRetry && !chatBinding,
|
|
...(restoreHold ? { workspaceRepairRequired, canRecordWorkspaceRepair: workspaceRepairRequired && hasRequiredWorkspaceRecovery(action.evidence) } : {}),
|
|
savedMessageCount: saved.length,
|
|
...(canReconstructWorkspace ? { canReconstructWorkspace, workspaceRecoveryStatusVersion: conversation!.statusVersion } : {}),
|
|
...(reconstructionPending ? { workspaceReconstructionPending: true } : {}),
|
|
};
|
|
}
|