Files
PaperClipAI/server/src/services/heartbeat/runtime-execution.ts
T
DottaandClaude Opus 5.5 340157d1d6 perf(recovery): scope run lifecycle recovery hints to the run's task graph (#21552)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs issue recovery queues in the background. Two of
them, `stranded-assignments` and `assigned-work-invariant`, find
assigned tasks that have no live run and restart them.
> - Every run dispatch and every run state change sent a recovery hint
for the whole company. On each hint these queues reload every eligible
open task of the company and check each one again, at most every 30
seconds per queue.
> - With live runs the hint comes all the time, so the queues never
stop. Their cost is (run activity) x (open-task backlog). On a hosted
instance with about 1,400 open tasks and 5 live runs, the server main
thread stayed at 87-92% of a core.
> - A run event can only change the tasks of its own task graph. This
pull request makes run and task producers name that graph, and keeps one
slow full pass as a backstop.
> - The benefit is that a run event costs a few checks, whatever the
size of the backlog, and a missed hint still delays recovery by at most
five minutes.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The issue recovery queues in `server/src/services/recovery/work.ts`
(`stranded-assignments`, `assigned-work-invariant`,
`resolved-dependencies`, `stale-issue-locks`) and the producers that
wake them on run dispatch, run start, run state change, run finalization
and accounting settlement.

**Subsystem affected**

Server background recovery (`server/src/services/recovery/`,
`server/src/services/scoped-deadline-work.ts`, the notification helpers
in `server/src/services/*-notifications.ts`).

**Current behavior**

`notifyRunDispatch`, `notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` sent `notifyIssueRecovery(tx, {
companyId })`. Most run lifecycle paths call one of them, often two or
three in the same transaction (for example `notifyIssueReviewState`).
Some other run paths sent the company hint directly: the wake status
update, the execution-control deadline writes, legacy run
terminalization, scheduled retries, accounting settlement, task checkout
and task document writes. Each company hint makes each task queue reload
all eligible open tasks of the company, 100 per page, and check each
task again. A company hint is reconstructed at most every 30 seconds per
queue, so one run event per 30 seconds is enough to keep a full pass
running all the time. `issue-recovery-deadlines.test.ts` pinned this: a
company hint re-checks every eligible task.

**Proposed behavior**

Design. For each task queue, a run event of run R on task T can change:

| Queue | What the check reads | What one run event can change |
| --- | --- | --- |
| `stranded-assignments` | T's own runs, wakes, handoffs, questions,
recovery actions, goal, assignee eligibility | T (its runs). The parent
of T and the tasks T blocks when the event also writes T's status. |
| `assigned-work-invariant` | the same plus open blockers and open
children | T, the parent of T (open children), the tasks T blocks (open
blockers) |
| `resolved-dependencies` | blocked tasks whose blockers are all done |
the tasks T blocks |
| `stale-issue-locks` | tasks whose checkout or execution run is R | T |

The assignee's agent status (`running`, `idle`, `error`) does not change
whether the agent is invokable
(`packages/shared/src/agent-eligibility.ts`), and a freed concurrency
slot starts the agent's next queued run through the
`queued-run-dispatch` queue, which keeps its company hint. So the
agent's other tasks are not in the set.

- `notifyIssueGraphRecovery(tx, { companyId, runId?, issueId? })` sends
`{ companyId, graphRunId?, graphIssueId? }` to the four task queues. It
runs no query in the producer transaction.
- When a task queue loads that hint, it resolves the graph with one
bounded, indexed query (`resolveIssueGraph`): the task of the run
(`issue_id`, `native_issue_id`, or the saved `issueId`/`taskId`,
validated and cast so the primary key is used), its parent, and the
tasks it blocks. The four queues share the result for 2 seconds. With no
task or more than 100 tasks, the queue turns the hint into the plain
company hint, so task-less runs coalesce under its 30-second gate
instead of each starting its own full pass. The lookup runs after
commit, so it sees the written state. Graph hints do not wait for the
30-second reconstruction gate (they load at most 100 tasks); hints for
the same run coalesce while dirty, and the lookup is shared across the
four queues, so it runs about once per pass instead of once per producer
call. An earlier round ran the lookup in every producer transaction: 34
lookups/s at 1,500 open tasks and 70/s at 5,000.
- The three company-level queues (`orphan-blockers`,
`active-recovery-actions`, `task-watchdogs`) keep the company hint. They
hold one entry per company or per watchdog, not per open task.
- `notifyRunDispatch`, `notifyScheduledRetryDispatch`,
`notifyNativeStatusReassessment` and
`notifyExternalConversationSettlement` use it when the scope names a run
or a task. A scope with only a company keeps the company hint.
- Run paths that knew the run or the task but sent only the company now
pass it. This includes three `writeExecutionControl` deadline writes,
legacy run terminalization (`notifyExecutionRunWork`) and native safe
replacement. Those helpers also forward the scope to
`notifyLiveRunRecovery` and `notifyLiveResourceRecovery`, so their hints
to the live-run queues now carry the run id.
- Recovery action changes (`issue-recovery-actions.ts`: create,
supersede, update, exhaust, resolve) name their source task. A recovery
action is a fact of its source task for these queues. The company-level
`active-recovery-actions` queue still gets the company hint. In the
fixture, failing runs create and resolve recovery actions every few
seconds, and that alone kept a full pass running.
- `writeExecutionControl` and `notifyExecutionRunWork` send the
attention retention hint for a task through
`notifyIssueAttentionRetentionWork` (the task, its parents and the tasks
it blocks), a superset of the single-task hint they sent before. A
recovery action of a blocker can change the blocker attention of its
dependents, so the narrower recovery action scope needs this.
- `promoteDeferredWorkIntents` scopes `lock_released` with a task to
that task, and `accounting_settled` to the run when the company has no
active hard-stop budget policy. Only a reached hard stop can refuse work
because of accounting (`evaluateAdmissionInTransaction`), so without
such a policy the settlement of one run cannot admit another task. The
other hold releases keep the company hint.

Backstop. Each scoped hint also sends `{ companyId, backstop: true }` to
the same four queues. Nothing else arms it. The queue runs it as a full
company pass, but not before 5 minutes after the last full pass of that
queue for that company or of the startup pass. It is a scheduler
deadline (`nextWakeAt` reports it), but it holds no idle intent and does
not mark the queue unreconciled. So:

- Startup still reconstructs every company.
- A failed scoped load retries with the existing back-off (30 seconds to
5 minutes). The backstop is pending in any case.
- A graph with no task or more than 100 tasks loads the whole company at
once.
- A task that a scoped hint missed is checked at most
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes) after the event. While runs
continue, each queue runs at most one full pass per company per 5
minutes. With no activity, nothing is scheduled.

Why arming on activity only cannot leave a stranded task undetected when
the instance then goes quiet. A task becomes stranded only through a
write or through a run that stops without a write:

1. The write names the task's graph (run and task producers). The scoped
pass checks the task at once.
2. The write sends a company hint (pause, budget, agent and company
configuration, tree holds, the producers listed below). The company pass
checks it within the 30-second gate, as before.
3. The write is a run event whose graph does not contain the task. That
same event armed the backstop. The backstop deadline stays scheduled
while the instance is quiet and checks the task at most 5 minutes later.
4. A run stops without any write (process lost, controller gone). The
existing deadline paths find it: `stale-issue-locks` waits for the
stored controller lease, and `active-run-silence` and
`orphan-active-leases` act on their own deadlines. Each of them
terminalizes the run, which is a run write and falls under 1 or 3. The
new test `restarts a task whose run vanished without a final event once
its lease lapses` covers this with the production backstop.
5. The process restarts or a quiet instance is woken by a restart. The
startup pass reconstructs every company.
6. A check that found a task waiting (grace, back-off, monitor) returns
its own deadline. That entry keeps its deadline whatever hints arrive.

Five minutes is the existing upper bound of recovery back-off in this
area: the maximum retry delay of a failed scoped-deadline check
(`scoped-deadline-work.ts`), and `DEFERRED_WAKE_PENDING_BOUND_MS`, the
time `assigned-work-sweep.ts` already trusts a deferred wake before it
calls a task stranded.

Producers that keep the company hint, and why. They can change
eligibility of tasks outside one task graph, or I could not show that
their effect stays in the graph. None of them fires once per run event
in the fixture.

- Eligibility for many tasks: company status (`companies.ts`), budgets
(`budgets.ts`), agent lifecycle and configuration (`agent-lifecycle.ts`,
`agent-configuration.ts`), instance settings (`instance-settings.ts`,
`*`), tree holds (`issue-tree-control.ts`), company deletion and import
(`company-deletion.ts`, `company-portability.ts`), and the hold releases
`pause_lifted`, `tree_hold_released`, `scheduling_resumed` and
`reconciler_backstop`. `accounting_settled` also keeps it when the
company has an active hard-stop budget policy.
- Task writes that can move a task between graphs or change eligibility:
task creation and a task update that changes the parent in `issues.ts`
(other task updates and checkout releases now name the task), document
delete and restore paths in `documents.ts`, task watchdogs
(`task-watchdogs.ts`), system and workspace recovery
(`system-recovery.ts`, `task-retry.ts`,
`workspace-recovery-capture-policy.ts`,
`workspace-recovery-stop-proof.ts`).
- Conversation and chat paths: `agent-conversations.ts`, the daily-cap
release in `heartbeat/queue.ts`, the Telegram attachment failure,
`chat-attachment-reuse.ts`, `native-runner-file-handoff.ts`,
`notifyChatRunMilestones` without a task or run (company authority
change), and `notifyNativeStatusReassessment` from `access.ts`.
- `writeExecutionControl` and `notifyExecutionRunWork` callers that pass
only a company.

**Reason and benefit**

Background work must not grow with the backlog of open tasks. The checks
a run event causes now depend on its task graph, not on the company.

## What Changed

- `server/src/services/issue-recovery-notifications.ts`:
`notifyIssueGraphRecovery`, `resolveIssueGraph`, the scope matcher and
decoder, `ISSUE_RECOVERY_MAX_ISSUE_IDS` (100) and
`ISSUE_RECOVERY_BACKSTOP_MS` (5 minutes).
- `server/src/services/recovery/work.ts`: the task queues resolve a
graph hint when they load it (shared for 2 seconds), remember the last
full pass per company, and gate the backstop hint on it.
`registerIssueRecoveryWork` takes an optional `backstopMs`.
- `server/src/services/recovery/assigned-work-sweep.ts`: `listSuspects`
accepts `issueIds`.
- `server/src/services/scoped-deadline-work.ts`: an optional
`backstop(scope)` that returns the earliest time of a backstop hint. A
pending backstop hint keeps its deadline but does not count as pending
work or as unreconciled.
- Producers: `issue-recovery-actions.ts`,
`execution-control-notifications.ts`, `run-dispatch-notifications.ts`,
`native-status-reassessment-notifications.ts`,
`external-conversation-settlement-notifications.ts`,
`work-intent-outbox.ts`, `run-cost-accounting.ts`,
`heartbeat/run-lifecycle.ts`, `heartbeat/runtime-execution.ts`,
`heartbeat/retries.ts`, `heartbeat.ts`, `legacy-execution-recovery.ts`,
`issues.ts` (checkout), `documents.ts` (task document writes),
`modules/wake-queue/adapters/postgres.ts` (recovery runs for a task),
`heartbeat/queue.ts` (wake enqueue), `recovery/service.ts`,
`native-runtime/native-provider-capacity-retry.ts`,
`native-runtime/native-safe-replacement.ts`,
`modules/run-dispatch/adapters/workspace-busy-wait.ts` (scheduled
retries).
- `bounds subsidiary scans for100 assignment candidates`: a resolved
recovery action now adds one scoped load per task queue (bound 8 to 11),
and the coalescing check sends one company hint before the burst of 100,
because the first full pass after the resolve is now a backstop pass
with its own key.
- `checks only blocked tasks whose blockers are done`: uses the shared
recording harness and runs out the backstop before its quiet check,
because the blocker's status change is now a graph hint (its 300
dependents overflow to a company load).
- Tests. `issue-recovery.test-fixture.ts` and
`live-recovery-work.test.ts` register the queues with `backstopMs: 0`,
so their quiet checks keep the earlier timing. The new tests use the
production value.

## Verification

New tests. I saw each red test fail before the change.

- `issue-recovery-deadlines.test.ts` `re-checks only the tasks a run
lifecycle event can affect, whatever the backlog` (red before): a run of
the agent on a task with a parent and a dependent, 1,000 backlog tasks
of another agent, and a second open task of the same agent. Seven real
producers (`notifyNativeStatusReassessment`, `notifyExecutionRunWork`,
`notifyRunDispatch`, `promoteDeferredWorkIntents` lock release and
accounting settlement, recovery action create and resolve) check exactly
the 3 tasks of the graph. The recovery action steps were red before the
third commit. With 4,000 more backlog tasks the event issues the same
number of statements. Before the change it checked 1,004 tasks.
- `resolves a run's task graph through index lookups, not the backlog`
(red before the second commit, from a Greptile finding): with 2,000
backlog tasks the graph lookup read 2,005 issue rows
(`pg_stat_xact_user_tables`) because it compared `id::text`. It now
validates and casts the saved id, reads fewer than 50 rows, and returns
exactly the run's graph.
- `finds a stranded task outside the affected set within the backstop
bound` (red before): a task inserted without a hint is not checked by
the run event, and is checked at most 5 minutes after the last full
pass. After 20 quiet minutes, a run event checks a new unrelated task
within 5 seconds, because the last full pass is older than 5 minutes.
Then the queues are quiet.
- `keeps the company pass for hold releases that can admit any task`
(guard, passes before and after): accounting settlement with a hard-stop
policy and a lifted pause still check every task.
- `falls back to the company pass when a run affects more tasks than a
hint names` (guard): a run task that blocks 100 more tasks checks the
whole company.
- `limits an ordinary comment to its issue among one thousand open
assignments`: the old final assertion (a company hint checks all 1,000
tasks) stays, and the test now also runs out the backstop and ends
quiet.
- `issue-recovery-assignments.test.ts`, real heartbeat, real producers,
production backstop (guards; they also pass with the old company hint):
a task whose run failed, a task whose run the system cancelled, a run
that vanished without a final event and is found when its controller
lease lapses, and an agent's queued task that starts when its previous
run ends. Each one recovers within 15 seconds, well before the backstop.
- `scoped-deadline-work.test.ts` `keeps a backstop hint's deadline
without holding the worker busy` (red before).

Suites, each file alone, in `server/` (issue recovery, live recovery,
recovery action and attention suites against dedicated local `_test`
databases): `issue-recovery-deadlines` (13/13),
`issue-recovery-assignments` (7/7), `issue-recovery-work-lifecycle`
(14/14), `scoped-deadline-work` (24/24), `live-recovery-work` (51/51),
`assigned-work-invariant` (32/32), `assigned-work-sweep-round-trips`
(2/2), `saved-work-recovery-overdue` (4/4),
`issue-recovery-dependencies` (5/5), `issue-recovery-conversations`
(3/3), `issue-recovery-actions` (112/112), `attention-retention-work`
(55/55). `recovery-deadline-acceptance` skips without its external
database. `pnpm check:module-boundaries` passes.

Local benchmark. The real server ran against PostgreSQL with
`log_statement=all` and instrumented counters (not committed). Fixture:
1 company, 60 agents, open tasks at 150 / 1,500 / 5,000 (52% in review,
39% blocked behind open blockers, 3% in progress, 2% to do), with
process agents that fail at once so recovery re-dispatches runs all the
time. The tables are analyzed after loading. Each window is 180 s,
starting 150 s after the server is ready, so each "after" window
contains one backstop pass. Base is `master` at 54c3616ad9; after is
this branch at dcd170e29f. The last commit (graph hints without the
30-second gate, task updates and checkout releases scoped) is covered by
the tests below but not re-measured. The machine was shared (load
average 9-16, one base window at 45). Run churn differs between runs, so
the table gives counts per run event (runs created in the window).

| Open tasks | Runs created (/s) base → after | Stranded checks per run
base → after | Assigned-work checks per run base → after |
`agent_task_sessions` reads (/s, 4 shapes, stranded check) | Statements
per run event base → after |
| --- | --- | --- | --- | --- | --- |
| 150 | 0.08 → 0.14 | 45.1 → 6.2 | 2.00 → 0.96 | 3.75 → 1.27 | 2,260 →
1,257 |
| 1,500 | 1.09 → 1.00 | 22.0 → 3.9 | 1.24 → 1.53 | 26.5 → 7.2 | 1,313 →
1,060 |
| 5,000 | 1.12 → 1.99 | 20.4 → 1.6 | 4.10 → 0.91 | 29.3 → 8.1 | 1,400 →
829 |

New statements: the task graph lookup (`with seed(id) ...`) at 0.8 / 9.3
/ 18.5 per second, and scoped loads of the task queues (`select "id",
"company_id" from "issues" ...` grew from 0.7 / 6.6 / 7.8 to 2.9 / 20.6
/ 34.4 per second, each a lookup of at most 100 ids). Stale-lock checks
grew from 0 / 50 / 52 to 22 / 190 / 445 per window: each run event now
checks the locks of its own task at once instead of waiting for the next
full pass. In an earlier round the graph lookup ran in every producer
transaction: 34/s at 1,500 and 70/s at 5,000.

With no activity the server issues no statements before and after this
change (quiet checks in `issue-recovery-deadlines`,
`issue-recovery-assignments`, `live-recovery-work`).

## Risks

- A task outside the graph of the run whose state the event changes is
now found by the backstop, at most 5 minutes later, instead of within 30
seconds. One known case: from #21427, `assigned-work-invariant` counts a
held recovery wake as pending while its agent has queued recovery runs.
When the last one starts, the continuation queue delivers the held wake;
the invariant check of that task waits for the backstop.
- A producer that names a task but changes something wider would now
miss the wider effect. Producers that change eligibility for many tasks
(pause, budget, agent and company configuration, tree holds) still send
the company hint.
- The graph is resolved after commit, from the written state. A parent
change is not a run event and keeps its company hint.
- The live-recovery and issue-recovery suites opt out of the backstop
deadline with `backstopMs: 0`.
- The execution-control deadline writes and legacy run terminalization
now pass the run id. `writeExecutionControl` also forwards that scope to
the live-run queues (`active-run-silence`, `native-process-sessions` and
others), so those hints now name the run instead of the company. Those
queues already handle run-scoped hints from native finalization.
- Test-only option: `registerIssueRecoveryWork(..., { backstopMs })`.
The shared recovery fixtures pass `0` so their quiet checks keep the
earlier timing; the new tests use the production value.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 01:29:22 -06:00

1344 lines
66 KiB
TypeScript

import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
import { readContextCompactionRequest } from "../context-compaction.js";
import { remoteWorkspaceFailureCause } from "../remote-workspace-transport.js";
import { readNativePostResultFailureDiagnostic } from "../native-runtime/native-post-result-failure.js";
import { acceptedNativeAdapterResult } from "../native-runtime/native-accepted-result-settlement.js";
import { NativeStartupCleanupPendingError } from "../native-runtime/native-startup-cleanup.js";
import { NativeControllerRecoveryWaitingError } from "../native-runtime/native-controller-recovery.js";
import { describeFailedAttachment } from "../native-runtime/remote-warm-session-recovery.js";
import { usesNativeUsageDelivery } from "../native-runtime/native-usage-accounting.js";
import { INTERACTION_DELIVERY_DENIED, preparedRunInteractionDeliveryDenied } from "../agent-interaction-delivery.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import { readRemoteWorkspaceAuthority, saveRemoteWorkspace } from "../remote-workspaces.js";
import { environmentService } from "../environments.js";
import { writeExecutionControl } from "../execution-control-notifications.js";
import { inspectPluginWorkspaceForFinalization } from "../native-runtime/native-plugin-workspace-finalization.js";
import { AsyncLocalStorage } from "node:async_hooks";
import { drainRunPreparationPhases, measureRunPhase } from "../execution-preparation-timing.js";
import { parseNativeSessionGoalControl } from "./scheduling.js";
import { isHeartbeatRunTerminalStatus } from "./run-lifecycle.js";
import {
PAPERCLIP_WAKE_PAYLOAD_KEY,
configuredPaperclipApiBaseUrl,
buildPaperclipRuntimeMcpServers,
createAdapterRuntimeToolAccess,
paperclipApiBaseUrl,
createAdapterRuntimeMcpAccess,
createManagedMcpRunConfig,
revokeHeartbeatRunGatewayTokens,
} from "./run-preparation.js";
import {
WorkspaceValidationFailure,
isWorkspaceValidationFailure,
fingerprintFinalizeWorkspaceBranchValidation,
attachPaperclipSessionMetadataToSessionParams,
} from "./workspaces.js";
import { recordLegacyWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
import { configuredEnvironmentProjection } from "../../vendor/paperclip-runner/index.js";
import { observabilitySecretValues } from "@paperclipai/adapter-utils/agent-observability";
import { buildAgentIdentityEnv } from "@paperclipai/adapter-utils/server-utils";
import { externalObjectService } from "../external-objects.js";
import { isAgentDirectoryCopy } from "../agent-directory-working-copies.js";
import { restoreNativeWorkspaceBestEffort } from "../native-runtime/native-workspace-best-effort.js";
import {
withNativeWorkspaceFinalizationOwnership,
NativeWorkspaceFinalizationBusyError,
NativeWorkspaceFinalizationOwnershipLostError,
type NativeWorkspaceFinalizationOwnership,
} from "../native-runtime/native-workspace-finalization-ownership.js";
import { createRunUsageRecorder, readAcceptedRunUsageCheckpoint } from "../usage-receipts.js";
import { applyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-result";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { isConversation } from "../agent-conversations.js";
import { withAdapterExecutionPhase } from "@paperclipai/adapter-utils/execution-phase";
import { remoteExecutionHasStopped } from "../remote-execution-termination.js";
import { registerAdapterExecutionControl } from "../adapter-execution-control.js";
import { resolveManagedOpenAiBilling } from "@paperclipai/adapter-utils";
import { and, eq, isNull, sql } from "drizzle-orm";
import { heartbeatRuns, nativeRunFinalizations, workspaceOperations } from "@paperclipai/db";
import { conflict } from "../../errors.js";
import { getStartupTraceContext } from "../../instrumentation.js";
import { logger } from "../../middleware/logger.js";
import {
buildNativeProviderEnvironment,
executePaperclipNativeSession,
finalizeNativeRun,
isRunnerIngressAuthorized,
materializeLegacyQuestionResponseWakeProjection,
NativeCancellationPendingRecoveryError,
NativeControllerDetachedForRestartError,
recordNativeFinalizationFailure,
} from "../native-runtime/index.js";
import {
buildNativeHeartbeatPreparationSpans,
buildNativeWakeIngressSpan,
} from "../native-runtime/native-run-trace.js";
import { PROVIDER_TRACE_MAX_BYTES } from "../provider-trace-store.js";
import type { AdapterExecutionResult } from "../../adapters/index.js";
import { createLocalAgentJwt } from "../../agent-auth-jwt.js";
import { parseObject, asNumber } from "../../adapters/utils.js";
import { NativeRunnerOwnershipUnverifiedError } from "../native-runtime/native-runner-ownership.js";
import {
ensureGitWorktreeBranchCoherent,
formatManagedGitWorktreeBranchInspection,
inspectManagedGitWorktreeBranch,
} from "../workspace-runtime.js";
import {
blockRunnerGoalRecovery,
failRunnerGoalAction,
isRunnerGoalActionCompleted,
settleLiveRunnerGoalBeforeInterrupt,
} from "../runner-goals.js";
import type { Db, agents, executionWorkspaces } from "@paperclipai/db";
import type { AdapterExecutionContext, ServerAdapterModule } from "../../adapters/index.js";
import type { HeartbeatRuntimeSelectionInput, HeartbeatRuntimeSelectionResult } from "./runtime-selection.js";
import type { createHeartbeatRunState } from "./run-state.js";
import type { createHeartbeatLifecycle } from "./run-lifecycle.js";
import type { createHeartbeatRunControl } from "./run-control.js";
import type { createHeartbeatQueue } from "./queue.js";
import { secretOutputForms, type createAgentIdentityRedactor } from "../agent-identity-redaction.js";
import { createRunOutputSecretStore } from "../run-secret-redaction.js";
import type { createAdapterExecutionControl } from "../adapter-execution-control.js";
import type { executionWorkspaceService } from "../execution-workspaces.js";
import type { workspaceOperationService } from "../workspace-operations.js";
import type { environmentRunOrchestrator } from "../environment-run-orchestrator.js";
import type { instanceSettingsService } from "../instance-settings.js";
import type { EffectiveRunSessionConfigMetadata } from "./workspaces.js";
import type { ProviderResourceDisposition } from "../environment-runtime.js";
import type { NativeExecutionInput, NativeSessionBackend } from "../../vendor/paperclip-runner/index.js";
import type { PluginWorkerManager } from "../plugin-worker-manager.js";
import { resolvePluginWorkspaceFinalization } from "../plugin-workspaces.js";
import type { RealizedExecutionWorkspace } from "../workspace-runtime.js";
import type { reserveWarmNativeInstructionDirectory } from "../native-runtime/index.js";
type Selection = HeartbeatRuntimeSelectionInput;
type Selected = Extract<HeartbeatRuntimeSelectionResult, { selected: true }>;
type NativeSessionInput = Parameters<typeof executePaperclipNativeSession>[0];
type RunState = ReturnType<typeof createHeartbeatRunState>;
type Lifecycle = ReturnType<typeof createHeartbeatLifecycle>;
/** Dispatches a prepared run; admission and terminal run status remain owned by the executor. */
export interface HeartbeatRuntimeExecutionInput {
run: typeof heartbeatRuns.$inferSelect;
agent: typeof agents.$inferSelect;
options: {
nativeSessionBackendFactory?: (execution: NativeExecutionInput, context?: { recoveryPurpose: "execution" | "stop_cleanup" }) => NativeSessionBackend;
pluginWorkerManager?: PluginWorkerManager;
};
task: Pick<Selection["task"], "issueContext" | "context" | "executionContinuation"> & {
issueRef: (NonNullable<Selection["task"]["issueRef"]> & { projectId: string | null }) | null;
issueId: string | null;
taskKey: string | null;
};
runtime: Pick<Selected, "nativeExecution" | "nativeRunnerInstanceId" | "pluginTaskDescription" | "toolRefreshOnResume" | "getNativeFreshSessionHandoff"> & {
nativeRuntimeResolution: Selection["nativeRuntimeResolution"];
adapter: ServerAdapterModule;
runtimeForAdapter: AdapterExecutionContext["runtime"];
getFreshSessionHandoff: AdapterExecutionContext["getFreshSessionHandoff"];
runOptions: Selection["recovery"];
};
workspace: {
persistedExecutionWorkspace: typeof executionWorkspaces.$inferSelect | null;
executionWorkspace: RealizedExecutionWorkspace;
executionTarget: Selection["workspace"]["executionTarget"];
workspaceOperationRecorder: ReturnType<ReturnType<typeof workspaceOperationService>["createRecorder"]>;
nativeWorkspaceSync: Selected["nativeWorkspaceSync"];
workspaceRestoreSource: Parameters<typeof recordLegacyWorkspaceRestoreFailure>[3] | null;
remoteExecution: unknown;
};
config: Pick<Selection["config"], "runtimeConfig" | "resolvedConfig" | "managedAiRuntime"> & {
resolvedInstanceSettings: { experimental: Pick<Awaited<ReturnType<ReturnType<typeof instanceSettingsService>["getExperimental"]>>, "enableWorkspaceBranchReconcileForward" | "enableWorkspaceDirtyQuarantineRepair"> };
configuredTaskEnvironment: Parameters<typeof configuredEnvironmentProjection>[0];
/** Names of configured entries resolved from secrets; values never leave the projection. */
secretKeys?: ReadonlySet<string>;
adapterEnv: Parameters<typeof buildNativeProviderEnvironment>[0];
agentIdentity: AdapterExecutionContext["agentIdentity"];
observability?: AdapterExecutionContext["observability"];
runtimeEnv: Selection["runtimeEnv"];
useHostGitHub: boolean;
githubSelection: { configured: boolean };
};
session: {
runtimeSessionParamsForAdapter: Record<string, unknown> | null;
configuredModel: string | null;
sessionConfigMetadata: EffectiveRunSessionConfigMetadata;
goalCheckpointSession: { current: { params: Record<string, unknown>; displayId: string } | null };
};
instructions: {
// Instruction callbacks can replace the copy and receipt while a provider is running.
readonly instructionCopy: Selection["config"]["instructionCopy"];
getInstructionSave: () => Record<string, unknown> | null;
nativeInstructionWorkingCopy: () => NativeSessionInput["instructionWorkingCopy"];
collectStoppedInstructions: AdapterExecutionContext["onProviderStopped"];
nativeInstructionReservation: Awaited<ReturnType<typeof reserveWarmNativeInstructionDirectory>>;
releaseInstructionCopy: () => Promise<void>;
};
trace: {
providerTraceCapture: { path: string } | null;
nativeRunnerPreparationSpans: NonNullable<NativeSessionInput["preparationSpans"]>;
attestedQuestionResponseAtMs: Parameters<typeof buildNativeWakeIngressSpan>[0]["attestedQuestionResponseAtMs"];
attemptStartedAtMs: number;
environmentAcquireStartedAtMs: number;
environmentRealizeEndedAtMs: number;
};
control: {
executionControl: ReturnType<typeof createAdapterExecutionControl>;
executionPhaseContext: Parameters<typeof withAdapterExecutionPhase>[0];
dispatchResolvedInteractionContinuationWithAtomicGate: <T>(dispatch: (markDispatchStarted: () => void) => Promise<T>) => Promise<
{ dispatched: true; resultPromise: Promise<T> } | { dispatched: false }
>;
};
output: Pick<AdapterExecutionContext, "onLog"> & {
onPersistedNativeLog?: (cursor: number, chunk: string) => Promise<void>;
onAdapterEvent: NonNullable<AdapterExecutionContext["onEvent"]>;
onAdapterMeta: AdapterExecutionContext["onMeta"];
identityRedactor: ReturnType<typeof createAgentIdentityRedactor>;
appendIdentityRedactedLog: AdapterExecutionContext["onLog"];
};
services: Pick<RunState, "upsertTaskSession" | "getRun"> &
Pick<Lifecycle, "persistRunProcessMetadata" | "recordCurrentHeartbeatRunRuntimeProgress"> &
Pick<ReturnType<typeof createHeartbeatRunControl>, "cancelRunInternal"> &
Pick<ReturnType<typeof createHeartbeatQueue>, "enqueueWakeup" | "dispatchPendingNativeStatusWakeups"> & {
executionWorkspacesSvc: Pick<ReturnType<typeof executionWorkspaceService>, "getStoredById">;
instanceSettings: Pick<ReturnType<typeof instanceSettingsService>, "getExperimental">;
envOrchestrator: Pick<ReturnType<typeof environmentRunOrchestrator>, "releaseForRun">;
};
// Teardown must observe these updates immediately, including when execution later throws.
effects: {
onUsageCaptureReady: (persistFailure: () => Promise<void>) => void;
onNativeDispatchStarted: () => void;
onLegacyAdapterEntered: () => void;
onWorkspaceRestoreFailure: (evidence: Record<string, unknown>) => void;
onProviderResourceDisposition: (disposition: ProviderResourceDisposition) => void;
onNativeWorkspaceFinalizeScheduled: () => void;
onNativeSessionResumeScheduled: () => void;
onNativeOwnershipHeld: () => void;
};
}
export type HeartbeatRuntimeExecutionResult = { dispatched: false } | {
dispatched: true;
adapterResult: AdapterExecutionResult;
};
export class NativeSessionResumeScheduledError extends Error {
constructor(readonly original: unknown) {
super("Native session recovery has been scheduled for the same run.");
this.name = "NativeSessionResumeScheduledError";
}
}
export class NativeWorkspaceFinalizeScheduledError extends Error {
constructor(
readonly original: unknown,
readonly terminalFailure: boolean,
readonly reasonCode:
"workspace_sync_out_failed" | "workspace_sync_out_unrecoverable",
) {
super("Native workspace finalization recovery has been scheduled.");
this.name = "NativeWorkspaceFinalizeScheduledError";
}
}
function readNonEmptyString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
export async function executeHeartbeatRuntime(db: Db, input: HeartbeatRuntimeExecutionInput): Promise<HeartbeatRuntimeExecutionResult> {
const { run, agent, options } = input;
const { issueRef, issueContext, context, issueId, taskKey, executionContinuation } = input.task;
const { nativeRuntimeResolution, nativeExecution, nativeRunnerInstanceId, adapter, runtimeForAdapter, getNativeFreshSessionHandoff, getFreshSessionHandoff, runOptions } = input.runtime;
const { persistedExecutionWorkspace, executionWorkspace, executionTarget, workspaceOperationRecorder, nativeWorkspaceSync, workspaceRestoreSource, remoteExecution } = input.workspace;
const remoteWorkspaceAuthority = readRemoteWorkspaceAuthority(persistedExecutionWorkspace?.metadata);
const { runtimeConfig, resolvedConfig, resolvedInstanceSettings, configuredTaskEnvironment, secretKeys, adapterEnv, agentIdentity, observability, managedAiRuntime, runtimeEnv, useHostGitHub, githubSelection } = input.config;
const { runtimeSessionParamsForAdapter, configuredModel, sessionConfigMetadata, goalCheckpointSession } = input.session;
const { nativeInstructionWorkingCopy, collectStoppedInstructions, nativeInstructionReservation, releaseInstructionCopy } = input.instructions;
const { providerTraceCapture, nativeRunnerPreparationSpans, attestedQuestionResponseAtMs, attemptStartedAtMs, environmentAcquireStartedAtMs, environmentRealizeEndedAtMs } = input.trace;
const { executionControl, executionPhaseContext, dispatchResolvedInteractionContinuationWithAtomicGate } = input.control;
const dispatchWithTiming: HeartbeatRuntimeExecutionInput["control"]["dispatchResolvedInteractionContinuationWithAtomicGate"] = async dispatch => {
if (await preparedRunInteractionDeliveryDenied(db, run)) {
await input.services.cancelRunInternal(run.id, "Queued agent input is no longer permitted.", {
errorCode: INTERACTION_DELIVERY_DENIED, suppressImmediateRecovery: true,
});
return { dispatched: false };
}
const providerDispatch = AsyncLocalStorage.bind(dispatch);
return measureRunPhase("execution.dispatch_admission", () => dispatchResolvedInteractionContinuationWithAtomicGate(providerDispatch));
};
const { identityRedactor, appendIdentityRedactedLog, onLog, onAdapterEvent, onAdapterMeta } = input.output;
const { executionWorkspacesSvc, instanceSettings, envOrchestrator, upsertTaskSession, getRun, cancelRunInternal, enqueueWakeup, persistRunProcessMetadata, recordCurrentHeartbeatRunRuntimeProgress, dispatchPendingNativeStatusWakeups } = input.services;
const localAgentJwtScope =
issueRef?.workMode === "skill_test"
? { kind: "skill_test" as const, issueId: issueRef.id }
: { kind: "standard" as const };
const authToken =
nativeRuntimeResolution.kind === "legacy" &&
adapter.supportsLocalAgentJwt
? createLocalAgentJwt(
agent.id,
agent.companyId,
agent.adapterType,
run.id,
run.responsibleUserId,
localAgentJwtScope,
)
: null;
if (
nativeRuntimeResolution.kind === "legacy" &&
adapter.supportsLocalAgentJwt &&
!authToken
) {
logger.warn(
{
companyId: agent.companyId,
agentId: agent.id,
runId: run.id,
adapterType: agent.adapterType,
},
"local agent jwt secret missing or invalid; running without injected PAPERCLIP_API_KEY",
);
}
let adapterFinalizeOutcome: "succeeded" | "failed" | null = null;
let preserveRunGatewayTokens = false;
let pluginWorkspaceFinalization: Record<string, unknown> | null = null;
let pluginWorkspaceFinalizationPlan: Awaited<ReturnType<typeof resolvePluginWorkspaceFinalization>>;
let pluginWorkspaceFinalizationRecord: typeof persistedExecutionWorkspace;
let pluginWorkspacePlacementFailed = false;
const inspectFinalizeWorkspaceBranch = async (
workspaceRecord: typeof persistedExecutionWorkspace,
plan: Awaited<ReturnType<typeof resolvePluginWorkspaceFinalization>>,
) => {
pluginWorkspaceFinalization = workspaceRecord
? await inspectPluginWorkspaceForFinalization(db, options.pluginWorkerManager, workspaceRecord, agent.id, run.id, { taskLease: null, plan })
: null;
if (pluginWorkspaceFinalization || readRemoteWorkspaceAuthority(workspaceRecord?.metadata)
|| workspaceRecord?.strategyType !== "git_worktree") return null;
const worktreePath =
readNonEmptyString(workspaceRecord.providerRef) ??
readNonEmptyString(workspaceRecord.cwd) ??
readNonEmptyString(executionWorkspace.worktreePath) ??
readNonEmptyString(executionWorkspace.cwd);
const expectedBranchName =
readNonEmptyString(workspaceRecord.branchName) ??
readNonEmptyString(executionWorkspace.branchName);
if (!worktreePath || !expectedBranchName) return null;
const inspection = await inspectManagedGitWorktreeBranch({
worktreePath,
expectedBranchName,
});
return { workspaceRecord, inspection };
};
const recordWorkspaceFinalize = async (
status: "succeeded" | "failed",
metadata?: Record<string, unknown>,
restore?: () => Promise<void>,
) => {
if (adapterFinalizeOutcome) return;
const workspaceRecord = pluginWorkspaceFinalizationRecord ??
((status === "succeeded" || persistedExecutionWorkspace?.providerType === "plugin") && persistedExecutionWorkspace?.id
? await executionWorkspacesSvc.getStoredById(agent.companyId, persistedExecutionWorkspace.id).catch(error => {
if (persistedExecutionWorkspace.providerType === "plugin") pluginWorkspacePlacementFailed = true;
throw error;
})
: persistedExecutionWorkspace);
const plan = pluginWorkspaceFinalizationPlan ??= workspaceRecord
? await resolvePluginWorkspaceFinalization(db, workspaceRecord, { runId: run.id, agentId: agent.id })
.catch(error => { pluginWorkspacePlacementFailed = true; throw error; })
: undefined;
if (plan) pluginWorkspaceFinalizationRecord = workspaceRecord;
const pluginWorkspaceTaskLease = plan?.kind === "finalize" ? plan.taskLease : undefined;
await workspaceOperationRecorder.recordOperation({
phase: "workspace_finalize", cwd: executionWorkspace.cwd,
metadata: { adapterType: agent.adapterType, executionTargetKind: executionTarget?.kind ?? "local",
...(pluginWorkspaceTaskLease ? { pluginWorkspaceTaskLease } : {}),
...(restore ? { owningService: "native_workspace_finalizer" } : {}), ...metadata },
run: async () => {
await restore?.();
let finalizeBranchMetadata: Record<string, unknown> | null = null;
let finalizeBranchRepairMetadata: Record<string, unknown> | null =
null;
if (status === "succeeded") {
const branchInspection = await inspectFinalizeWorkspaceBranch(workspaceRecord, plan);
if (branchInspection) {
let inspection = branchInspection.inspection;
const initialManagedGitWorktreeBranch =
formatManagedGitWorktreeBranchInspection(inspection);
if (
!inspection.valid &&
inspection.reasonCode === "branch_mismatch" &&
inspection.repoRoot
) {
let repairedExpectedBranchName = inspection.expectedBranchName;
try {
const coherence = await ensureGitWorktreeBranchCoherent({
db,
repoRoot: inspection.repoRoot,
worktreePath: inspection.worktreePath,
expectedBranchName: inspection.expectedBranchName,
actualBranchName: inspection.actualBranchName,
sourceIssue: issueRef
? {
id: issueRef.id,
identifier: issueRef.identifier,
title: issueRef.title,
workMode: issueRef.workMode,
}
: null,
executionWorkspaceId: branchInspection.workspaceRecord.id,
heartbeatRunId: run.id,
enableWorkspaceBranchReconcileForward:
resolvedInstanceSettings.experimental
.enableWorkspaceBranchReconcileForward,
enableWorkspaceDirtyQuarantineRepair:
resolvedInstanceSettings.experimental
.enableWorkspaceDirtyQuarantineRepair,
persistForwardReconcile: false,
reconcileOperationPhase: "workspace_finalize",
recorder: workspaceOperationRecorder,
});
if (
coherence.branchName &&
coherence.branchName !==
branchInspection.workspaceRecord.branchName
) {
repairedExpectedBranchName = coherence.branchName;
executionWorkspace.branchName = coherence.branchName;
executionWorkspace.warnings.push(...coherence.warnings);
}
} catch (repairErr) {
const workspaceValidationFailure =
isWorkspaceValidationFailure(repairErr) ? repairErr : null;
finalizeBranchMetadata = {
executionWorkspaceId: branchInspection.workspaceRecord.id,
...initialManagedGitWorktreeBranch,
};
finalizeBranchRepairMetadata = {
attempted: true,
succeeded: false,
initial: initialManagedGitWorktreeBranch,
reason:
repairErr instanceof Error
? repairErr.message
: String(repairErr),
};
await workspaceOperationRecorder.recordOperation({
phase: "workspace_finalize",
cwd: executionWorkspace.cwd,
metadata: {
adapterType: agent.adapterType,
executionTargetKind: executionTarget?.kind ?? "local",
...metadata,
managedGitWorktreeBranch: finalizeBranchMetadata,
managedGitWorktreeBranchRepair:
finalizeBranchRepairMetadata,
...(workspaceValidationFailure?.resultJson
? {
workspaceValidation:
workspaceValidationFailure.resultJson
.workspaceValidation ??
workspaceValidationFailure.resultJson,
}
: {}),
},
run: async () => ({
status: "failed",
stderr: `Managed git worktree branch check failed: ${repairErr instanceof Error ? repairErr.message : String(repairErr)}\n`,
}),
});
adapterFinalizeOutcome = "failed";
throw repairErr;
}
const repairedInspection =
await inspectManagedGitWorktreeBranch({
worktreePath: inspection.worktreePath,
expectedBranchName: repairedExpectedBranchName,
repoRoot: inspection.repoRoot,
});
finalizeBranchRepairMetadata = {
attempted: true,
succeeded: repairedInspection.valid,
initial: initialManagedGitWorktreeBranch,
repaired:
formatManagedGitWorktreeBranchInspection(
repairedInspection,
),
};
inspection = repairedInspection;
}
const managedGitWorktreeBranch =
formatManagedGitWorktreeBranchInspection(inspection);
finalizeBranchMetadata = {
executionWorkspaceId: branchInspection.workspaceRecord.id,
...managedGitWorktreeBranch,
};
if (!inspection.valid) {
const workspaceValidationFingerprint =
fingerprintFinalizeWorkspaceBranchValidation({
issueId: issueRef?.id ?? null,
executionWorkspaceId: branchInspection.workspaceRecord.id,
inspection: managedGitWorktreeBranch,
});
await workspaceOperationRecorder.recordOperation({
phase: "workspace_finalize",
cwd: executionWorkspace.cwd,
metadata: {
adapterType: agent.adapterType,
executionTargetKind: executionTarget?.kind ?? "local",
...metadata,
managedGitWorktreeBranch: finalizeBranchMetadata,
...(finalizeBranchRepairMetadata
? {
managedGitWorktreeBranchRepair:
finalizeBranchRepairMetadata,
}
: {}),
},
run: async () => ({
status: "failed",
stderr: `Managed git worktree branch check failed: ${inspection.reason ?? "unknown branch mismatch"}\n`,
}),
});
adapterFinalizeOutcome = "failed";
throw new WorkspaceValidationFailure(
`Execution workspace ${branchInspection.workspaceRecord.id} expected git worktree branch "${inspection.expectedBranchName}" at "${inspection.worktreePath}", but ${inspection.reason ?? "the checked-out branch could not be verified"}. Record a sanctioned execution-workspace branch transition or restore the workspace branch before completing the run.`,
{
workspaceValidation: {
reason: "git_worktree_branch_incoherence",
fingerprint: workspaceValidationFingerprint,
adapterType: agent.adapterType,
issueId: issueRef?.id ?? null,
issueIdentifier: issueRef?.identifier ?? null,
persistedExecutionWorkspaceId:
branchInspection.workspaceRecord.id,
executionWorkspaceCwd: executionWorkspace.cwd,
managedGitWorktreeBranch: finalizeBranchMetadata,
},
},
);
}
}
}
return { status, metadata: {
...(pluginWorkspaceFinalization ? { pluginWorkspace: pluginWorkspaceFinalization } : {}),
...(finalizeBranchMetadata ? { managedGitWorktreeBranch: finalizeBranchMetadata } : {}),
...(finalizeBranchRepairMetadata ? { managedGitWorktreeBranchRepair: finalizeBranchRepairMetadata } : {}),
} };
},
});
// Only mark the outcome after the row landed, so a transient write
// failure on the succeeded path can still be recovered by recording
// finalize=failed from the catch path below.
adapterFinalizeOutcome = status;
};
const producerOwnsUsage = usesNativeUsageDelivery({
runnerd: nativeRuntimeResolution.kind === "native" && agent.adapterType === "paperclip_runner",
remote: executionTarget?.kind === "remote", provider: nativeExecution?.provider,
});
const usageRecorder = producerOwnsUsage ? undefined : await measureRunPhase("execution.usage_startup", () =>
createRunUsageRecorder(db, { companyId: run.companyId, runId: run.id, adapterType: agent.adapterType }));
input.effects.onUsageCaptureReady(usageRecorder?.persistFailure ?? (async () => {}));
let adapterResult: AdapterExecutionResult;
const runGoalControlRequestId = readNonEmptyString(
context.goalControlRequestId,
);
try {
if (nativeRuntimeResolution.kind === "native") {
if (!nativeExecution || !nativeRunnerInstanceId)
throw new Error("native_runtime_selection_not_persisted");
const expectedNativeMcpDigest =
"runtimeContext" in nativeExecution &&
nativeExecution.runtimeContext.mcp.bindingId
? nativeExecution.runtimeContext.mcp.digest
: null;
const nativeMcpServers = "homeMode" in nativeExecution && nativeExecution.homeMode === "preconfigured" ? [] : await measureRunPhase("execution.mcp_handoff", () => buildPaperclipRuntimeMcpServers({
db,
agent,
runId: run.id,
expectedAssignmentDigest: expectedNativeMcpDigest,
}));
if ("runtimeContext" in nativeExecution) {
if (nativeMcpServers.length > 1)
throw new Error(
"native MCP realization must produce one aggregate gateway",
);
const server = nativeMcpServers[0] ?? null;
const digest = server?.connectionId.startsWith("assignment:")
? server.connectionId.slice("assignment:".length)
: null;
if (digest && digest !== expectedNativeMcpDigest) {
throw new Error("native MCP assignment digest mismatch");
}
}
const nativeMcpServer = nativeMcpServers[0] ?? null;
let sessionGoalControl = parseNativeSessionGoalControl(
context.runnerGoalControl,
);
if (runGoalControlRequestId && !sessionGoalControl) {
throw new Error("session_goal_control_payload_invalid");
}
// A hard restart replays the heartbeat context, not a new user
// action. Do not repeat a completed create/replace/edit (which
// could reactivate or clear a goal that finished while detached).
const completedGoalControl =
sessionGoalControl !== null &&
taskKey !== null &&
(await isRunnerGoalActionCompleted(
db,
{
companyId: agent.companyId,
agentId: agent.id,
issueId: taskKey,
},
sessionGoalControl.requestId,
));
if (completedGoalControl) sessionGoalControl = null;
// Goal checkpoints already publish their task session. Goal-less turns
// need the same durable pointer before cleanup can delay host completion.
const taskSessionPublication = taskKey && !sessionGoalControl &&
context.resumeSessionGoalHeartbeat !== true && !completedGoalControl
? {
taskKey,
adapterType: agent.adapterType,
sessionParamsJson: attachPaperclipSessionMetadataToSessionParams(
{ ...runtimeSessionParamsForAdapter, cwd: executionWorkspace.cwd },
configuredModel,
sessionConfigMetadata,
),
}
: undefined;
const nativeDispatchAtMs = Date.now();
const runCreatedAtMs = run.createdAt.getTime();
const runStartedAtMs = (run.startedAt ?? run.createdAt).getTime();
const wakeComments = Array.isArray(
parseObject(context.paperclipWake).comments,
)
? (parseObject(context.paperclipWake).comments as unknown[])
: [];
const wakeIngressSpan = buildNativeWakeIngressSpan({
runCreatedAtMs,
wakeComments,
attestedQuestionResponseAtMs,
});
if (wakeIngressSpan)
nativeRunnerPreparationSpans.unshift(wakeIngressSpan);
// Named preparation phases (workspace, configuration, runner staging)
// that ran before dispatch, as run-log spans under task.prepare.
nativeRunnerPreparationSpans.push(...drainRunPreparationPhases(run.id).map(phase => ({
name: phase.name, startedAtMs: phase.startedAtMs, endedAtMs: phase.endedAtMs,
...(phase.outcome === "failed" ? { outcome: "failed" as const } : {}),
})));
nativeRunnerPreparationSpans.push(
...buildNativeHeartbeatPreparationSpans({
runCreatedAtMs,
runStartedAtMs,
attemptStartedAtMs,
environmentAcquireStartedAtMs,
environmentRealizeEndedAtMs,
nativeDispatchAtMs,
}),
);
const outputSecretStore = createRunOutputSecretStore(db);
await outputSecretStore.registerAll(run.companyId, run.id,
identityRedactor.secretSources.filter((value) => secretOutputForms([value]).length > 0));
identityRedactor.addSecretValues(await outputSecretStore.valuesForRun(run.companyId, run.id));
const stopCleanup = runOptions.nativeRestartRecovery?.kind === "reattach_remote_runner"
&& await readClaimedNativeStopRecovery(db, run, runOptions.nativeLeaseOwner);
const guardedDispatch =
await dispatchWithTiming(
(markDispatchStarted) => {
return executePaperclipNativeSession({
casesEnabled: context.paperclipCasesEnabled === true,
db,
execution: nativeExecution,
observability,
getFreshSessionHandoff: getNativeFreshSessionHandoff,
refreshTools: context.refreshTools === true,
toolRefreshOnResume: input.runtime.toolRefreshOnResume,
conversationMode: isConversation(issueContext),
turnTimeoutMs: Math.max(0, asNumber(runtimeConfig.timeoutSec, 0)) * 1_000,
runnerInstanceId: nativeRunnerInstanceId,
pluginTaskDescription: input.runtime.pluginTaskDescription,
leaseOwner: runOptions.nativeLeaseOwner,
restartRecovery: runOptions.nativeRestartRecovery,
onExecutionLeaseAcquired: runOptions.onNativeExecutionLeaseAcquired,
backend:
stopCleanup
? options.nativeSessionBackendFactory?.(nativeExecution, { recoveryPurpose: "stop_cleanup" })
: options.nativeSessionBackendFactory?.(nativeExecution),
useRunnerd: agent.adapterType === "paperclip_runner",
adapterType: agent.adapterType,
sessionGoalControl,
sessionCompaction: readContextCompactionRequest(context),
resumeSessionGoalHeartbeat:
context.resumeSessionGoalHeartbeat === true ||
completedGoalControl,
taskSessionPublication,
onGoalCheckpoint: async (snapshot) => {
if (!taskKey) return;
const params =
attachPaperclipSessionMetadataToSessionParams(
{
...runtimeSessionParamsForAdapter,
sessionId: snapshot.identity.sessionId,
cwd: executionWorkspace.cwd,
},
configuredModel,
sessionConfigMetadata,
)!;
const displayId =
snapshot.providerSessionId ?? snapshot.sessionId;
await upsertTaskSession({
companyId: agent.companyId,
agentId: agent.id,
adapterType: agent.adapterType,
taskKey,
sessionParamsJson: params,
sessionDisplayId: displayId,
lastRunId: run.id,
lastError: null,
});
goalCheckpointSession.current = { params, displayId };
},
onLog,
onPersistedNativeLog: input.output.onPersistedNativeLog,
onEvent: onAdapterEvent,
instructionWorkingCopy: nativeInstructionWorkingCopy(),
onUsage: usageRecorder ? async receipt => { await usageRecorder.capture(receipt); } : undefined,
preparationSpans: nativeRunnerPreparationSpans,
// Bootstrap with executable/home discovery while keeping
// configured provider values and the server-selected
// workspace boundary authoritative.
managedGitHub: !useHostGitHub && githubSelection.configured,
billingIdentity: managedAiRuntime ? { provider: managedAiRuntime.attribution.provider, biller: managedAiRuntime.attribution.provider === "openai" ? resolveManagedOpenAiBilling(managedAiRuntime.config.managedAiRouting)?.biller ?? managedAiRuntime.attribution.provider : managedAiRuntime.attribution.provider, billingType: managedAiRuntime.attribution.method === "subscription" ? "subscription_included" : "metered_api" } : undefined,
managedAiCredentialIdentity: managedAiRuntime?.identity,
managedAiAccountIdentity: managedAiRuntime?.sessionIdentity,
managedAiCredentialWriteback: managedAiRuntime?.credentialWriteback,
managedAiCredentialHome: managedAiRuntime ? String((managedAiRuntime.config.env as Record<string, unknown>).CODEX_HOME) : undefined,
outputSecrets: [...identityRedactor.secretSources, ...observabilitySecretValues(observability)],
dotWorkspaceRoot: nativeExecution.provider.kind === "openai_dot" && resolvedConfig.dotWorkspaceAccess === true ? executionWorkspace.cwd : undefined,
runnerEnvironment: {
...configuredEnvironmentProjection(configuredTaskEnvironment, secretKeys ?? []),
...buildNativeProviderEnvironment(
adapterEnv,
process.env,
executionWorkspace.cwd,
),
...buildAgentIdentityEnv(agentIdentity),
...(input.instructions.instructionCopy && isAgentDirectoryCopy(input.instructions.instructionCopy) ? { AGENT_HOME: input.instructions.instructionCopy.executionRoot } : {}),
...(nativeMcpServer
? {
PAPERCLIP_NATIVE_MCP_NAME: nativeMcpServer.name,
PAPERCLIP_NATIVE_MCP_URL: nativeMcpServer.url,
PAPERCLIP_NATIVE_MCP_TOKEN: nativeMcpServer.token,
}
: {}),
...(providerTraceCapture
? {
PAPERCLIP_PROVIDER_TRACE_PATH:
providerTraceCapture.path,
PAPERCLIP_PROVIDER_TRACE_MAX_BYTES: String(
PROVIDER_TRACE_MAX_BYTES,
),
}
: {}),
},
onTaskDispatched: markDispatchStarted,
runnerExecutionTarget: executionTarget,
runnerIngressAuthorized: isRunnerIngressAuthorized(
nativeRuntimeResolution,
),
runnerPublicUrl:
runtimeEnv.PAPERCLIP_RUNNER_PUBLIC_URL?.trim() || null,
runnerCaBundlePath:
runtimeEnv.PAPERCLIP_RUNNER_CA_BUNDLE_PATH?.trim() ||
null,
runnerRemoteBinaryPath:
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH?.trim() ||
null,
runnerRemoteCodexPath:
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH?.trim() ||
null,
runnerRemoteCodexNpmSpec:
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_CODEX_NPM_SPEC?.trim() ||
null,
runnerRemoteProviderPackPath:
runtimeEnv.PAPERCLIP_RUNNER_REMOTE_PROVIDER_PACK_PATH?.trim() ||
null,
stopTaskForReassignment: async (target) => {
await settleLiveRunnerGoalBeforeInterrupt(db, target);
if (!target.runId) return;
const prior = await getRun(target.runId);
if (!prior || prior.companyId !== target.companyId || prior.agentId !== target.agentId) {
throw conflict("Reassignment run binding changed");
}
const stopped = await cancelRunInternal(target.runId, "Cancelled for task reassignment", {
errorCode: "issue_reassigned", suppressImmediateRecovery: true,
resultJson: { reassignmentStopConfirmed: true },
});
if (stopped && ["running", "queued", "scheduled_retry"].includes(stopped.status)) {
throw conflict("The previous run did not stop; reassignment was not applied");
}
},
enqueueWakeup,
syncIssueExternalObjects: externalObjectService(db, {
pluginWorkerManager: options.pluginWorkerManager,
enabled: async () => (await instanceSettings.getExperimental()).enableExternalObjects === true,
}).syncIssueSafely,
onSpawn: async (meta) => {
markDispatchStarted();
await persistRunProcessMetadata(run.id, { ...meta, targetKind: executionTarget?.kind ?? "local" });
},
});
},
);
if (!guardedDispatch.dispatched) return { dispatched: false };
input.effects.onNativeDispatchStarted();
adapterResult = await guardedDispatch.resultPromise;
} else {
const interactionId = readNonEmptyString(context.interactionId);
const legacyQuestionResponse =
issueRef &&
interactionId &&
readNonEmptyString(context.interactionKind) ===
"ask_user_questions" &&
readNonEmptyString(context.interactionStatus) === "answered"
? await materializeLegacyQuestionResponseWakeProjection({
db,
companyId: agent.companyId,
issueId: issueRef.id,
runId: run.id,
agentId: agent.id,
interactionId,
})
: null;
// Do not write the answer projection back to `context`: legacy
// adapters need it in their prompt, but the authoritative answers
// remain on the interaction instead of being duplicated in the
// heartbeat run snapshot.
const adapterContext: Record<string, unknown> = {
...context,
...(legacyQuestionResponse
? {
[PAPERCLIP_WAKE_PAYLOAD_KEY]: {
...parseObject(context[PAPERCLIP_WAKE_PAYLOAD_KEY]),
questionResponse: legacyQuestionResponse,
},
}
: {}),
};
const runtimeTools = createAdapterRuntimeToolAccess({
agentId: agent.id,
companyId: agent.companyId,
runId: run.id,
responsibleUserId: run.responsibleUserId,
});
if (!runtimeTools) {
logger.warn(
{
companyId: agent.companyId,
agentId: agent.id,
runId: run.id,
},
"runtime connection tools could not be delivered",
);
}
const runtimeMcpServers = await buildPaperclipRuntimeMcpServers({
db,
agent,
runId: run.id,
});
const runtimeToolDelivery =
adapter.runtimeToolDelivery ?? "invocation_context";
if (runtimeTools && runtimeToolDelivery === "native_mcp") {
runtimeMcpServers.unshift({
name: "Paperclip connections",
url: runtimeTools.mcpEndpoint,
token: runtimeTools.bearerToken,
connectionId: "paperclip-runtime-tools",
});
}
if (authToken && configuredPaperclipApiBaseUrl() && issueRef) {
runtimeMcpServers.unshift({ name: "Paperclip projects", url: `${paperclipApiBaseUrl()}/api/mcp/project-tools`,
token: authToken, connectionId: "paperclip-project-tools" });
}
const runtimeMcp = createAdapterRuntimeMcpAccess(runtimeMcpServers);
if (runtimeTools && runtimeToolDelivery === "invocation_context") {
adapterContext.paperclipRuntimeTools = runtimeTools;
}
const managedMcpConfig = await measureRunPhase("execution.mcp_handoff", () => createManagedMcpRunConfig({
db,
agent,
runId: run.id,
config: runtimeConfig,
projectId: issueRef?.projectId ?? null,
issueId: issueRef?.id ?? null,
}));
if (managedMcpConfig) {
adapterContext.paperclipManagedMcp = managedMcpConfig;
}
const guardedDispatch =
await dispatchWithTiming(
(markDispatchStarted) => {
input.effects.onLegacyAdapterEntered();
return withAdapterExecutionPhase(executionPhaseContext, "adapter_execution", () => adapter.execute({
getFreshSessionHandoff,
agentIdentity,
observability,
runId: run.id,
agent,
runtime: runtimeForAdapter,
config: runtimeConfig,
context: adapterContext,
executionContinuation: executionContinuation ?? null,
runtimeCommandSpec:
adapter.getRuntimeCommandSpec?.(runtimeConfig) ?? null,
executionTarget,
executionTransport: remoteExecution
? {
remoteExecution: remoteExecution as unknown as Record<
string,
unknown
>,
}
: undefined,
runtimeMcp,
runtimeTools,
onLog,
onMeta: onAdapterMeta,
onEvent: onAdapterEvent,
onUsage: usageRecorder ? async receipt => { await usageRecorder.capture(receipt); } : undefined,
onExecutionPhase: executionControl.phases.enter,
startupTraceContext: getStartupTraceContext(),
onRuntimeProgress: async (progress) => {
await recordCurrentHeartbeatRunRuntimeProgress(
run,
progress,
issueId,
);
},
onProviderStopped: collectStoppedInstructions,
onDispatch: markDispatchStarted,
signal: executionControl.controller.signal,
...(executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" ? {
stopRemoteStartup: async () => {
// Scope comes from the running host invocation, never agent
// config. Keep adapter ownership until setup has unwound.
if (!executionControl.controller.signal.aborted) {
throw new Error("Remote startup stop requires a cancelled run");
}
const release = await envOrchestrator.releaseForRun({
heartbeatRunId: run.id,
companyId: agent.companyId,
agentId: agent.id,
status: "released",
providerResourceDisposition: "stop_and_retain",
cancelActiveWork: true,
});
if (release.errors.length || !await remoteExecutionHasStopped(db, agent.companyId, run.id)) {
throw new Error("Could not verify remote startup stopped");
}
},
} : {}),
onCancellationReady: async () => {
await registerAdapterExecutionControl(run.id, executionControl);
const current = await getRun(run.id);
if (!current || isHeartbeatRunTerminalStatus(current.status)) {
executionControl.controller.abort(new Error("Run stopped before provider startup"));
}
},
onSpawn: async (meta) => {
markDispatchStarted();
await persistRunProcessMetadata(run.id, {
pid: meta.pid,
processGroupId:
"processGroupId" in meta &&
typeof meta.processGroupId === "number"
? meta.processGroupId
: null,
startedAt: meta.startedAt,
// A sandbox runner's spawn metadata names a sandbox PID.
targetKind: executionTarget?.kind ?? "local",
});
},
authToken: authToken ?? undefined,
}));
},
);
if (!guardedDispatch.dispatched) return { dispatched: false };
adapterResult = await guardedDispatch.resultPromise;
}
adapterResult = identityRedactor.redact(adapterResult);
if (run.runtimeMode === "legacy" && hasWorkspaceRestoreFailure(adapterResult.resultJson)
&& executionTarget?.kind === "remote" && executionTarget.transport === "sandbox") {
const requiredWorkspaceRestoreEvidence = {
workspaceRestoreFailure: adapterResult.resultJson!.workspaceRestoreFailure,
...(adapterResult.resultJson?.workspaceRestoreDiagnostic ? { workspaceRestoreDiagnostic: adapterResult.resultJson.workspaceRestoreDiagnostic } : {}),
};
input.effects.onWorkspaceRestoreFailure(requiredWorkspaceRestoreEvidence);
// Retention is the fallback even if recording this receipt fails.
input.effects.onProviderResourceDisposition("stop_and_retain");
if (workspaceRestoreSource === null) throw new Error("legacy_workspace_restore_source_unavailable");
await recordLegacyWorkspaceRestoreFailure(db, run, requiredWorkspaceRestoreEvidence, workspaceRestoreSource);
}
for (const stream of ["stdout", "stderr"] as const) {
const tail = identityRedactor.finish(stream);
if (tail) await appendIdentityRedactedLog(stream, tail);
}
const instructionSave = input.instructions.getInstructionSave();
if (instructionSave) adapterResult.resultJson = { ...adapterResult.resultJson, instructionSave };
if (parseObject(adapterResult.executionRecovery).providerWorkStarted !== false) {
if (usageRecorder) {
const captured = await measureRunPhase("execution.usage", () => usageRecorder.complete(adapterResult));
adapterResult = { ...adapterResult, ...captured, usageComplete: captured.complete };
} else {
const accepted = await measureRunPhase("execution.usage", () => readAcceptedRunUsageCheckpoint(db, { companyId: run.companyId, runId: run.id }));
adapterResult = { ...adapterResult, ...accepted, usageComplete: accepted?.complete ?? false,
...(!accepted ? { costStatus: "unpriced" as const } : {}) };
}
} else {
// Stop may already own the terminal result. Preserve its metadata
// while durably recording the proof needed to release admission.
await db.update(heartbeatRuns).set({ costAccountingPending: true,
usageJson: sql`coalesce(${heartbeatRuns.usageJson}, '{}'::jsonb) || '{"accountingProviderWorkStarted":false}'::jsonb`,
}).where(and(eq(heartbeatRuns.id, run.id), isNull(heartbeatRuns.costAccountedAt)));
}
adapterResult = applyWorkspaceRestoreFailure(adapterResult);
// A returned result can include a failed restore. Keep the workspace
// barrier closed until required files have been restored.
// If recording the barrier itself fails, propagate as a run failure
// rather than silently leaving dependents stranded behind a missing
// finalize row.
const completeWorkspace = async (ownership?: NativeWorkspaceFinalizationOwnership) => {
try {
if (remoteWorkspaceAuthority && persistedExecutionWorkspace) {
if (!ownership || !executionTarget) throw new Error("remote_workspace_ownership_required");
const current = await executionWorkspacesSvc.getStoredById(agent.companyId, persistedExecutionWorkspace.id);
const authority = readRemoteWorkspaceAuthority(current?.metadata);
const lease = authority?.location
? await environmentService(db).getLeaseById(authority.location.leaseId) : null;
if (!current || !lease || lease.companyId !== run.companyId || lease.heartbeatRunId !== run.id) {
throw new Error("workspace_sync_out_unrecoverable");
}
await recordWorkspaceFinalize("succeeded", { owningService: "native_workspace_finalizer", storage: "remote" },
async () => { await saveRemoteWorkspace({ db, workspace: current, runId: run.id, lease,
target: executionTarget, assertOwned: ownership.assertHeld }); });
}
if (nativeWorkspaceSync) {
const exported = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, run.companyId),
eq(workspaceOperations.heartbeatRunId, run.id),
eq(workspaceOperations.phase, "workspace_finalize"),
eq(workspaceOperations.status, "succeeded"),
)).limit(1);
if (exported.length) adapterFinalizeOutcome = "succeeded";
else await recordWorkspaceFinalize(hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded", undefined,
async () => { await measureRunPhase("execution.workspace_finalize", () => restoreNativeWorkspaceBestEffort({
db, runId: run.id, assertOwnership: ownership?.assertHeld,
restore: () => nativeWorkspaceSync!.restoreWorkspace(ownership?.assertHeld),
})); });
}
await ownership?.assertHeld();
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
.update(heartbeatRuns)
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
),
), { companyId: run.companyId, runId: run.id });
const workspaceFinalizeStatus = hasWorkspaceRestoreFailure(adapterResult.resultJson) ? "failed" : "succeeded";
await recordWorkspaceFinalize(workspaceFinalizeStatus);
if (adapterResult.nativeFinalization) {
adapterResult.nativeFinalization.workspaceFinalizeStatus =
workspaceFinalizeStatus;
try {
const finalized = await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
db,
runId: run.id,
workspaceFinalizeStatus,
preserveProviderAttempt: Boolean(nativeWorkspaceSync || remoteWorkspaceAuthority),
}));
await dispatchPendingNativeStatusWakeups({
companyId: run.companyId,
});
if (finalized.phase === "committed") {
await nativeWorkspaceSync?.cleanup();
}
} catch (finalizeErr) {
logger.warn(
{ err: finalizeErr, runId: run.id },
"native result persisted but finalization did not apply; the reconciliation loop will retry",
);
}
}
} catch (error) {
if (ownership) {
await ownership.assertHeld();
await recordWorkspaceFinalize("failed");
}
throw error;
}
};
if (nativeWorkspaceSync || remoteWorkspaceAuthority) {
const owned = await measureRunPhase("execution.workspace_ownership", () => withNativeWorkspaceFinalizationOwnership({
db, companyId: run.companyId, runId: run.id,
}, completeWorkspace));
if (!owned.acquired) throw new NativeWorkspaceFinalizationBusyError();
} else {
await completeWorkspace();
}
} catch (adapterErr) {
if (adapterErr instanceof NativeCancellationPendingRecoveryError) {
// Durable cancellation is settled by the outer recovery handler;
// it does not imply a failed workspace or a persisted run result.
throw adapterErr;
}
if (adapterErr instanceof NativeWorkspaceFinalizationBusyError
|| adapterErr instanceof NativeWorkspaceFinalizationOwnershipLostError) {
input.effects.onNativeWorkspaceFinalizeScheduled();
throw adapterErr;
}
if (adapterErr instanceof NativeControllerDetachedForRestartError) {
// Preserve the provider and its run for the new controller. This
// also keeps generic teardown from terminalizing/releasing its lease.
input.effects.onNativeSessionResumeScheduled();
preserveRunGatewayTokens = true;
throw adapterErr;
}
if (adapterErr instanceof NativeControllerRecoveryWaitingError) {
input.effects.onNativeSessionResumeScheduled();
throw new NativeSessionResumeScheduledError(adapterErr);
}
if (adapterErr instanceof NativeRunnerOwnershipUnverifiedError) {
input.effects.onNativeOwnershipHeld();
throw adapterErr;
}
if (nativeRuntimeResolution.kind === "native" && (nativeWorkspaceSync?.reference.state === "prepared" || remoteWorkspaceAuthority?.dirty)
&& !(nativeExecution && "access" in nativeExecution.workspace && nativeExecution.workspace.access === "none")) {
input.effects.onProviderResourceDisposition("stop_and_retain");
}
try {
await writeExecutionControl(db, DELIVERY_QUEUES.executionFinalization, tx => tx
.update(heartbeatRuns)
.set({ executionControlDeadlineAt: new Date(Date.now() + 60_000) })
.where(
and(
eq(heartbeatRuns.id, run.id),
eq(heartbeatRuns.status, "running"),
),
), { companyId: run.companyId, runId: run.id });
} catch (writeError) {
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
throw writeError;
}
if (adapterErr instanceof NativeStartupCleanupPendingError) throw adapterErr;
if (
issueRef &&
context.resumeSessionGoalHeartbeat === true &&
!runGoalControlRequestId
) {
await blockRunnerGoalRecovery(
db,
{
companyId: run.companyId,
issueId: issueRef.id,
agentId: agent.id,
adapterType: agent.adapterType,
},
"provider_session_goal_recovery_failed",
).catch(() => undefined);
}
if (issueRef && runGoalControlRequestId) {
await failRunnerGoalAction(
db,
{
companyId: run.companyId,
issueId: issueRef.id,
agentId: agent.id,
adapterType: agent.adapterType,
},
runGoalControlRequestId,
adapterErr instanceof Error
? identityRedactor.redact(adapterErr.message)
: "session_goal_control_failed",
).catch(() => undefined);
}
const nativeResumeScheduled =
nativeRuntimeResolution.kind === "native"
? await db
.select({
phase: nativeRunFinalizations.phase,
resultId: nativeRunFinalizations.resultId,
})
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id))
.limit(1)
.then(
(rows) =>
rows[0]?.phase === "retryable_failure" &&
rows[0]?.resultId === null,
)
: false;
if (nativeResumeScheduled) {
input.effects.onNativeSessionResumeScheduled();
const failedAttachment = describeFailedAttachment(adapterErr instanceof Error ? adapterErr.message : null);
if (failedAttachment) await Promise.resolve(onLog("stderr", `[paperclip] ${failedAttachment}\n`)).catch(() => undefined);
throw new NativeSessionResumeScheduledError(adapterErr);
}
// Adapter (or its restore finally) threw — or the finalize record
// write itself threw. Either way the workspace may be in a partial
// state. Best-effort record finalize=failed so the dependent readiness
// check keeps the gate closed instead of waking on stale local state,
// and surface the original error to the caller.
// The control code can be a fencing label; the closed diagnostic says
// what actually failed after an accepted result (no provider text).
const postResultFailure = readNativePostResultFailureDiagnostic(adapterErr);
// A runner that could not prove suspension after an accepted result left a
// local workspace untouched: there is nothing to copy back. Settle the
// accepted result instead of failing the run over runner bookkeeping.
// Rebuild the accepted result before writing any finalize status: only a
// proven, complete accepted result may record workspace finalization as
// succeeded.
let acceptedResult = nativeRuntimeResolution.kind === "native"
&& postResultFailure?.category === "runner_settlement_incomplete"
&& !nativeWorkspaceSync && !remoteWorkspaceAuthority
? await (async () => {
const resultId = await db.select({ resultId: nativeRunFinalizations.resultId }).from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id)).limit(1).then(rows => rows[0]?.resultId ?? null);
return resultId ? acceptedNativeAdapterResult(db, {
runId: run.id, companyId: run.companyId, resultId,
model: nativeExecution?.provider.model ?? null, driverKind: nativeExecution?.session.driverKind ?? "unknown",
postResultFailure: postResultFailure!,
}) : null;
})().catch((readErr: unknown) => {
logger.warn({ err: readErr, runId: run.id }, "accepted native result could not be read; keeping the failure path");
return null;
})
: null;
let settleAcceptedResult = acceptedResult !== null;
try {
await recordWorkspaceFinalize(settleAcceptedResult ? "succeeded" : "failed", settleAcceptedResult
? { runnerSettlementDiagnostic: postResultFailure }
: {
errorMessage: identityRedactor.redact(
adapterErr instanceof Error
? adapterErr.message
: String(adapterErr),
),
...(postResultFailure ? { failureDiagnostic: postResultFailure } : {}),
});
} catch (recordErr) {
logger.warn(
{
err: recordErr,
runId: run.id,
executionWorkspaceId: persistedExecutionWorkspace?.id ?? null,
},
"failed to record workspace_finalize=failed operation; dependents may remain gated",
);
}
// Settle only when the workspace finalization record itself succeeded. A
// failed branch or plugin workspace check keeps the failure path.
if (adapterFinalizeOutcome !== "succeeded") { settleAcceptedResult = false; acceptedResult = null; }
if (nativeRuntimeResolution.kind === "native") {
const proposedResult = await db
.select({ resultId: nativeRunFinalizations.resultId })
.from(nativeRunFinalizations)
.where(eq(nativeRunFinalizations.runId, run.id))
.limit(1)
.then((rows) => rows[0]?.resultId ?? null);
if (proposedResult && (nativeWorkspaceSync || remoteWorkspaceAuthority || pluginWorkspacePlacementFailed ||
(pluginWorkspaceFinalizationPlan?.kind === "finalize" && pluginWorkspaceFinalizationPlan.taskLease))) {
const workspaceFailureMessage =
adapterErr instanceof Error ? adapterErr.message : "";
const unrecoverable =
workspaceFailureMessage ===
"workspace_sync_out_unrecoverable" ||
workspaceFailureMessage.includes("daytona_sandbox_not_found");
const failure = await recordNativeFinalizationFailure({
db,
runId: run.id,
error: new Error(
unrecoverable
? "native_workspace_sync_out_unrecoverable"
: "native_workspace_sync_out_failed",
),
projectRunStatus: true,
failureScope: "workspace",
permanent: unrecoverable,
failureCause: postResultFailure
? `native_post_result_failure: category=${postResultFailure.category}`
: remoteWorkspaceFailureCause(adapterErr),
});
input.effects.onNativeWorkspaceFinalizeScheduled();
throw new NativeWorkspaceFinalizeScheduledError(
adapterErr,
failure.phase === "terminal_failure",
unrecoverable
? "workspace_sync_out_unrecoverable"
: "workspace_sync_out_failed",
);
}
if (settleAcceptedResult && acceptedResult) {
// Finalize exactly as a returned result would, then let the ordinary
// completion path settle the run, session, accounting and agent.
await onLog("stderr", "[paperclip] runner settlement was incomplete after the result was accepted; finalizing the accepted result\n")
.catch(() => undefined);
try {
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
db, runId: run.id, workspaceFinalizeStatus: "succeeded",
}));
await dispatchPendingNativeStatusWakeups({ companyId: run.companyId });
} catch (finalizeErr) {
logger.warn(
{ err: finalizeErr, runId: run.id },
"native result persisted but finalization did not apply; the reconciliation loop will retry",
);
}
return { dispatched: true, adapterResult: acceptedResult };
}
try {
await measureRunPhase("execution.native_finalize", () => finalizeNativeRun({
db,
runId: run.id,
workspaceFinalizeStatus: "failed",
}));
await dispatchPendingNativeStatusWakeups({
companyId: run.companyId,
});
} catch (finalizeErr) {
logger.warn(
{ err: finalizeErr, runId: run.id },
"native result could not be marked workspace_failed; the reconciliation loop will retry persisted results",
);
}
}
throw adapterErr;
} finally {
try {
if (!preserveRunGatewayTokens) await revokeHeartbeatRunGatewayTokens({
db,
companyId: agent.companyId,
runId: run.id,
});
} catch (revokeErr) {
logger.warn(
{ err: revokeErr, runId: run.id, companyId: agent.companyId },
"failed to revoke heartbeat-run MCP gateway tokens",
);
}
await nativeInstructionReservation?.release();
await withAdapterExecutionPhase(executionPhaseContext, "instruction_cleanup", releaseInstructionCopy);
}
return { dispatched: true, adapterResult };
}